Server device, system, and method for processing information

Abstract

To provide a server device, system, and method for processing information allowing for performing more efficiently the confirming operation of face images, a great deal of which have been collectively registered or which are automatically registered by a system.SOLUTION: A server device in an embodiment comprises a face-image registration-control section, a detecting section, an appropriateness-degree calculating section, and a display control section. The face-image registration-control section accepts a registration of a face image including a human face. The detecting section detects feature points of the face image and detects an area showing a human face from the face image based on the feature points. The appropriateness-degree calculating section calculates an appropriateness degree of the face image to be registered in a face lexicon based on information detected from the area showing the human face. The display control section displays on a display device first display information to confirm the face image in a lower order of the appropriateness degree.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] An embodiment of the present invention relates to a server device, an information processing system, and an information processing method. [Background technology]

[0002] In a face recognition system, a face image showing the face of a target person, features generated from the face image, and the name, etc. must be registered in advance as a dictionary. In order to achieve high-quality face recognition, it is desirable to register face images that match the characteristics of the face recognition engine, so face image registration conditions are set. However, particularly in applications such as generating meta information for video content, it takes a lot of time to prepare face images of people to be recognized in advance, and the quality of the face images varies widely, so the dictionary conditions for registration are designed to have a certain degree of leeway so that face images can be registered.

[0003] For face images that do not satisfy the conditions for dictionary registration, such as the size and image quality of the face, and the inability to detect the face, dictionary registration fails and the accuracy of face recognition is not affected. However, by providing a margin for the conditions for face images, in rare cases, face images that are correctly registered may have faces that are tilted or have extremely poor image quality, or images other than faces may be registered. This not only makes it impossible to recognize the person in question, but also frequently leads to overdetection and erroneous recognition of the background, particularly when an image other than a face is erroneously registered, significantly affecting the overall accuracy of face recognition performed using the dictionary. For this reason, a confirmation process after registering a face image is essential. For example, in face recognition of video content such as broadcast programs, hundreds to tens of thousands of face images may be registered in the dictionary, so they are registered in the dictionary in bulk. However, visual confirmation of a large number of images places a heavy burden on the operator and takes time, so there has been a demand for reducing the burden of the confirmation process. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2021-157649 A Summary of the Invention [Problem to be solved by the invention]

[0005] The problem to be solved by the present invention is to provide a server device, an information processing system, and an information processing method that can more efficiently perform the confirmation work of face images after batch registration of a large number of face images or after automatic registration by the system. [Means for solving the problem]

[0006] The server device of the embodiment includes a face image registration control unit, a detection unit, an appropriateness calculation unit, and a display control unit. The face image registration control unit accepts registration of a face image including a person's face. The detection unit detects feature points of the face image, and detects an area showing the person's face from the face image based on the feature points. The appropriateness calculation unit calculates the appropriateness of the face image to be registered in a face dictionary based on information detected from the area showing the person's face. The display control unit displays, on a display device, first display information that confirms the face images in order of decreasing appropriateness. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an example of a device configuration of an information processing system according to the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a functional configuration of the client apparatus according to the first embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of a functional configuration of the server device according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a face dictionary table according to the first embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of a face information table according to the first embodiment. [Figure 6] FIG. 6 is a diagram showing Example 1 of the display information according to the first embodiment. [Figure 7]FIG. 7 is a diagram showing a second example of the display information according to the first embodiment. [Figure 8] FIG. 8 is a sequence diagram illustrating an example of an information processing method according to the first embodiment. [Figure 9] FIG. 9 is a sequence diagram illustrating an example of an information processing method according to the first embodiment. [Figure 10] FIG. 10 is a diagram for explaining the effects of the first embodiment. [Figure 11] FIG. 11 is a diagram showing an example of display information according to a modified example of the first embodiment. [Figure 12] FIG. 12 is a diagram illustrating an example of a hardware configuration of the server device according to the first and second embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of a server device, an information processing system, and an information processing method will be described in detail with reference to the accompanying drawings.

[0009] (First embodiment) First, an example of the device configuration of the information processing system according to the first embodiment will be described.

[0010] [Example of device configuration] 1 is a diagram showing an example of a device configuration of an information processing system 100 according to the first embodiment. The information processing system 100 according to the first embodiment includes a client device 1 and a server device 2.

[0011] In the client device 1, an API client and a Web client that communicate with a Web API (Application Programming Interface) of the server device 2 and a Web application, etc., run. The communication between the client device 1 and the server device 2 is performed using, for example, a Web API and a socket communication I / F.

[0012] For example, the client device 1 may be a business PC, a broadcasting device, a non-linear editing system, an archive system, a material management system, a CG (Computer Graphics) system, a contract management system, or the like.

[0013] For example, the server device 2 receives input video (an example of content data) via a digital video interface such as SDI or HDMI (registered trademark). For example, the server device 2 performs real-time analysis in a broadcasting station to instantly identify people included in the input video from multiple relay points using a face dictionary registered in advance.

[0014] Also, for example, the server device 2 accepts input video by uploading a video file from the client device 1.

[0015] Furthermore, the server device 2 displays on the client device 1 display information that assists in confirmation for adding, changing, or deleting a face image to be registered in the face dictionary.

[0016] [Example of functional configuration] 2 is a diagram showing an example of a functional configuration of the client device 1 according to the first embodiment. The client device 1 according to the first embodiment includes an input unit 11, a display unit 12, a communication unit 13, and a storage unit .

[0017] The input unit 11 accepts operation input from the user for real-time display information transmitted from the server device 2. It also requests the server device 2 to perform face recognition processing on the video content file stored in the memory unit 14 using an API provided by the server device 2, and accepts operation input from the user for offline display information transmitted from the server device 2. The display unit 12 displays the display information transmitted from the server device 2. The communication unit 13 communicates with the server device 2 by wired communication or wireless communication. The memory unit 14 stores information used by the client device 1.

[0018] 3 is a diagram showing an example of a functional configuration of the server device 2 according to the first embodiment. The server device 2 according to the first embodiment includes a face image registration control unit 21, a content registration control unit 22, a detection unit 23, an appropriateness calculation unit 24, an extraction unit 25, a storage control unit 26, a display control unit 27, a similarity calculation unit 28, and an output control unit 29.

[0019] The face image registration control unit 21 receives person registration information for registering information on a person to be identified in the information processing system 100 in the face dictionary DB from the client device 1 or the like. The person registration information includes the face image, name, properties, and the like of the person. The face image is an image including the face of the person to be registered in the face dictionary DB. The name is information for identifying the person, such as the name of the person appearing in the face image. The properties are supplementary information, such as the name of the group if the person is active in a group, a path for identifying the face image file, and the like.

[0020] The number of entries in the face dictionary DB prepared in advance for face recognition varies depending on the purpose. For example, in the case of an event related to national politics, the number of entries in the face dictionary DB may range from several hundred to several thousand. For example, in the case of adding meta information to archived video of a broadcasting station or the like, the number of entries in the face dictionary DB may reach tens of thousands. In these cases, it takes time for a person to manually register each entry one by one on the screen of an application for registering face information. Therefore, the content registration control unit 22 accepts multiple person registration information in a lump sum using a list such as a csv (Comma Separated Value) file.

[0021] The content registration control unit 22 receives content registration information including content data for which face recognition is to be performed, such as video content and image files. When the content data is video content, the content registration control unit generates continuous still images from the video content. The continuous still images may be generated by cutting out at a preset interval, such as 5 fps (one image per 0.2 seconds).

[0022] The detection unit 23 receives a face image of the person registration information from the face image registration control unit 21, and receives a still image cut out from the video content of the content registration information from the content registration control unit 22. The detection unit 23 detects feature points of the face image included in the person registration information, and detects an area showing the person's face from the face image based on the feature points. Similarly, the detection unit 23 detects feature points of the face of a person shown in a still image cut out from content data included in the content registration information, and detects an area showing the person's face shown in a still image cut out from the content data based on the feature points.

[0023] The shape of the area showing the person's face may be any shape, for example, a rectangular area. The number of people to be detected may be one or more for one extracted still image, and the number of people to be detected and the size of the area of ​​the face to be detected may be set in advance.

[0024] The suitability calculation unit 24 refers to an area indicating a face detected from the face image of the person registration information, and calculates the suitability of the face image as a face image to be registered in the face dictionary DB. For example, the suitability is expressed as a value from 0 to 100, with a higher value indicating a higher suitability. Also, for example, the suitability is expressed as a value from 0.0 to 1.0, with a higher value indicating a higher suitability.

[0025] An appropriate face image is one that can recognize the person himself / herself and does not erroneously recognize the background, etc. due to overdetection. For example, a face image in which the face is not tilted is appropriate as a face image to be registered in the face dictionary DB. Also, for example, a face image with good image quality is appropriate as a face image to be registered in the face dictionary DB. Also, for example, an image other than a face is inappropriate as a face image to be registered in the face dictionary DB.

[0026] In this embodiment, since the calculation of the suitability is included in the registration process of the face dictionary DB, it is desirable for the processing time to be as short as possible, particularly in applications where hundreds to tens of thousands of dictionaries are registered.

[0027] Specifically, the appropriateness calculation unit 24 calculates the appropriateness based on the feature points of the eyes, nose, and mouth detected from the area showing the person's face. For example, the appropriateness is calculated based on five feature points (two points for both eyes, one point for the nose, and two points for the corners of the mouth) of the eyes, nose, and mouth corners (corners of the mouth) identified from the area showing the person's face. Specifically, for example, the appropriateness is calculated based on the average, minimum, maximum, etc. of the feature point detection scores. Also, for example, the appropriateness is calculated based on the number of detected feature points.

[0028] For example, the closer to the front a face is captured, the more useful it is for identifying a person, so the closer to the front a face is captured, the higher the appropriateness is calculated to be.Also, the more areas where the face cannot be identified, for example because it is hidden by a mask or sunglasses, the lower the appropriateness is calculated to be.

[0029] The extraction unit 25 extracts facial feature amounts indicating features of a face from an area indicating a face detected from a face image. Similarly, the extraction unit 25 extracts facial feature amounts indicating features of a face from an area indicating a face detected from content data.

[0030] The storage control unit 26 controls the storage of the person registration information registered in the face dictionary DB and the facial feature amount of the person extracted by the extraction unit 25, and controls the storage of the content registration information registered in the content DB.

[0031] The display control unit 27 controls the display of display information displayed on the client device. For example, the display control unit 27 displays, on the client device, display information that displays the results of collectively registering face images in a face dictionary DB in order of people with low suitability.

[0032] The similarity calculation unit 28 compares the feature amount of the face image registered in the face dictionary DB with the feature amount of the face of a person included in the content data stored in the content DB, and calculates the similarity amount.

[0033] The output control unit 29 controls the output of an analysis result (output information) including, for example, information included in the face dictionary DB, information included in the content DB, and the similarity calculated by the similarity calculation unit 28. This analysis result is used, for example, to identify an individual included in content data such as video broadcast by a broadcast station.

[0034] The face clustering processing unit 30 aggregates face images of people appearing in the content by person based on the face feature amount extracted by the extraction unit 25. From the result of this processing, a representative face image of the cluster is extracted based on the suitability calculated by the suitability calculation unit 24. The representative face image of the cluster is used to create a list of performers appearing in the content, such as a video broadcast by a broadcasting station. In addition, for a person not registered in the face dictionary, this representative face image may be used as an image for registration in the face dictionary.

[0035] [Example of face dictionary table structure] FIG. 4 is a diagram showing an example of a face dictionary table of the first embodiment. The face dictionary table of the first embodiment includes a dictionary ID, a dictionary name, the number of registrations, and an update date and time. The dictionary ID is identification information for identifying a dictionary. The dictionary name is the name of the face dictionary. The number of registrations is the number of pieces of face information registered in the face dictionary. The update date and time is the date and time when the dictionary was updated. Multiple face dictionary DBs can be registered, and they may be switched for use for each video content or image file for which face recognition is to be performed.

[0036] 5 is a diagram showing an example of a face information table according to the first embodiment. The face information table according to the first embodiment includes one or more face information records. Face information registered in the face dictionary is managed by the face information record.

[0037] The face information record includes the following items: face ID, name, and update date and time. The face ID is identification information for identifying the face information record. The name is the name of a person identified by the face ID, etc. The update date and time is the date and time when the data identified by the face ID was updated.

[0038] In addition, in principle, one face image record per person is linked to the face ID of a face information record, but two or more face image records may be linked. Each face image record includes the following items: face ID, image ID, error, appropriateness, face feature amount, file path, and update date and time. The face ID is identification information linked to the face ID of the face information record. The image ID is identification information that identifies the face image.

[0039] The error is information indicating an error when a face image registered in the face dictionary DB does not satisfy the conditions of the face dictionary DB or when some error occurs during face recognition processing in the face detection unit, extraction unit, etc. For example, a csv error indicates that when person registration information is accepted in a lump sum as a list such as a csv file, the suitability calculated by the suitability calculation unit 24 is lower than a predetermined threshold value for the face image (e.g., an unclear image, etc.). The suitability indicates the suitability of the dictionary registration of the face image.

[0040] The facial feature amount records the facial feature amount extracted by the extraction unit 25. The file path is information indicating the location of a path for referencing a facial image file. The update date and time is the date and time when the facial image record was created or updated.

[0041] [Example of display information] Fig. 6 is a diagram showing Example 1 of display information in the first embodiment. The display information in Fig. 6 shows an example of a confirmation screen for a face image of face information registered in a face dictionary DB.

[0042] The face information with face IDs 000012 and 000010 displayed in the error image area 101 on the first and second lines indicates data in which an error has occurred (a CSV error in the example of FIG. 6). The error image area 101 displays all error information that has occurred at the time the confirmation screen is displayed, and prompts the user to perform an operation to resolve the error. Once the error has been resolved, the face information is displayed in the normal image area 103. The error image area 101 is not displayed if no error has occurred or if all errors have been resolved.

[0043] The face information with face IDs 000013 and 000009 displayed in the attention image area 102 on the third and fourth lines indicates data that requires visual confirmation of the registered face image because the appropriateness is less than the preset appropriateness threshold of 70. As shown in FIG. 6, the attention image area 102 displays face images whose appropriateness is equal to or less than the threshold. The attention image area 102 is not displayed when all face images whose appropriateness exceeds the appropriateness threshold are registered. Also, for example, even if the appropriateness of an image is less than the threshold, if it is found to be suitable as an image to be registered in the face dictionary DB as a result of visual confirmation, there is no need to re-register the face image, and the face information may be displayed in the normal face image area 103 after visual confirmation.

[0044] The face information with the face ID of 000011 displayed in the normal image area 103 on the fifth row indicates normal data with an appropriateness of 94.2. As shown in Fig. 6, the normal image area 103 displays a face image with an appropriateness higher than a threshold value.

[0045] As shown in Fig. 6, the display control unit 27 first displays data in which an error has occurred in the topmost error image area 101 in a first display color, then displays data requiring attention and confirmation in a second display color different from the first display color in a warning image area 102 below the error image area 101, and then displays normal data in a third display color in a normal image area 103 below the warning image area 102. The warning image area 102 and the normal image area 103 are displayed in descending order of appropriateness in each area. This allows the user to immediately grasp the face images that require attention and their priority, thereby making it possible to more efficiently carry out the confirmation work after registering the face images.

[0046] Fig. 7 is a diagram showing a second example of display information in the first embodiment. The display information in Fig. 6 shows an example of a confirmation screen for a plurality of face images registered in a face ID. At least one face image is registered for each person in the face dictionary. For a person with a plurality of face images registered in the face dictionary, the display control unit 27 displays first display information including a representative face image with a lower suitability.

[0047] If even one of the multiple face images contains an error, the error is displayed as shown in an error image area 201. The face image in which the error occurs (representative face image) does not satisfy the registration conditions of the face dictionary DB, so the face image is not displayed.

[0048] Furthermore, when the display control unit 27 accepts a selection operation of the face information of face ID 000008 displayed in the error image area 201, it displays display information 301 (second display information) for confirming the face image registered with the face ID 000008. In the example of the display information 301, there is no registered image due to the error.

[0049] Since the number of images of the face information of face ID 000006 displayed in the display area 202 is 2, two face images are registered for face ID 000006. When multiple face images are registered, the display control unit 27 displays an image with a lower suitability (representative face image) and the suitability as shown in the display area 202. Furthermore, when the display control unit 27 accepts a selection operation of the face information of face ID 000006 displayed in the display attention image area 202, it displays display information 302 (second display information) confirming the two face images registered for the face ID 000006. In the display information 302, multiple face images of the same person are displayed in order of decreasing suitability. Furthermore, data with a lower suitability may be displayed in a second display color.

[0050] The user deletes the image as necessary and registers a new face image that satisfies the registration conditions in the face dictionary DB, and the face data moves from the warning image area 202 to the normal image area. That is, when the storage control unit 26 receives a designation of a face image to be deleted and registration of a new face image from the user via the display information 302, the storage control unit 26 deletes the face image to be deleted from the face dictionary DB and replaces the face image to be deleted with the new face image, thereby registering the new face image in the face dictionary DB.

[0051] The designation of the facial image to be deleted and the registration of a new facial image may be accepted through the display information 301 and 302 (second display information) shown in FIG. 7, or may be accepted directly from the display information (first display information) shown in FIG. 6.

[0052] [Examples of information processing methods] 8 and 9 are sequence diagrams showing an example of the information processing method of the first embodiment. First, the face image registration control unit 21 registers the above-mentioned list of person registration information (for example, the above-mentioned CSV file) (step S1), and inputs the list to the detection unit 23 and the storage control unit 26.

[0053] Next, the detection unit 23 judges whether each of the face images included in the person registration information registered in step S1 satisfies the dictionary registration conditions (step S2). For example, the dictionary registration conditions include a condition for checking whether the image includes a face. Specifically, the dictionary registration conditions are conditions related to the size of the face, the clarity of the face (e.g., resolution), and the direction of the face (e.g., the accuracy of the face calculated from the proximity of the eye feature points). It is desirable to provide the dictionary registration conditions with a margin so that they can handle a wide variety of face image qualities, so that the minimum conditions suitable for dictionary registration are set. This reduces the effort required for advance preparation of face images for dictionary registration and enables the dictionary registration of more people's face images.

[0054] In the case where priority is given to reducing the work required after dictionary registration over dealing with a wide variety of face image qualities by providing some leeway in the dictionary registration conditions, the suitability calculated by the suitability calculation unit 24 may be used as the dictionary registration conditions. A threshold value may be set for the suitability, and the condition for dictionary registration may be that a face has been detected and that the suitability is equal to or greater than the threshold.

[0055] If the facial image does not satisfy the dictionary registration conditions (step S2, No), the display control unit 27 generates error display information to be displayed in, for example, the error image area 101 shown in FIG. 6 described above (step S9).

[0056] If the face image satisfies the dictionary registration condition (Yes at step S2), the detection unit 23 detects feature points of the face image, and detects an area showing a person's face from the face image based on the feature points (step S3).

[0057] Next, the suitability calculation unit 24 refers to the area indicating the face detected from the face image in step S3, calculates the suitability of the face image to be registered in the face dictionary (step S4), and registers the suitability in the face dictionary (step S5).

[0058] Next, the extraction unit 25 refers to the area indicating the face detected from the face image in step S3, extracts a facial feature amount indicating the characteristics of the face (step S6), and registers the facial feature amount in a face dictionary (step S7).

[0059] Meanwhile, the storage control unit 26 judges whether or not information other than the face image included in the person registration information generated in step S1 (e.g., the number of characters, etc.) satisfies the dictionary registration condition (step S8). In the judgment in step S8, for example, it is judged whether or not the information registered in each field of the record of the face dictionary DB is equal to or less than the number of characters allowed in each field, whether or not prohibited characters are included, etc.

[0060] If the information other than the face image does not satisfy the dictionary registration conditions (step S8, No), the display control unit 27 generates error display information to be displayed in, for example, the error image area 101 shown in FIG. 6 described above (step S9).

[0061] Next, the display control unit 27 rearranges the display order of the face images included in the list of person registration information generated in step S1 based on the appropriateness calculated in step S4 and the error display information generated in step S9 (step S10).

[0062] Specifically, when an error occurs in the facial image (step S11, Yes), the display control unit 27 displays the error display information generated in step S9 in the front row of the display information (for example, the top row like the error image area 101 shown in the above-mentioned FIG. 6) in a first display color that prompts a warning (step S12). When the facial image registration control unit 21 receives a facial image re-registration operation from a user who has referred to the display information displayed in step S12, it re-registers the facial image (step S17), and the process returns to step S2.

[0063] Next, when the suitability of the face image is equal to or lower than the threshold (Yes in step S13), the display control unit 27 displays the face images in ascending order of suitability in a second display color for prompting confirmation, for example, as in the warning image area 102 shown in Fig. 6 described above, following the error display in step S12 (step S14). When the face image registration control unit 21 receives a face image re-registration operation from the user who has referred to the display information displayed in step S14, it re-registers the face image (step S17), and the process returns to step S2.

[0064] Next, if the face image has a suitability higher than the threshold value (step S13, No), after the warning display in step S15, the display control unit 27 displays the face images in order of decreasing suitability in a third display color indicating that they are normal, for example, as in the normal image area 103 shown in Figure 6 described above (step S16).

[0065] As described above, in the server device 2 of the first embodiment, the face image registration control unit 21 accepts registration of a face image including a person's face. The detection unit 23 detects feature points of the face image, and detects an area showing the person's face from the face image based on the feature points. The suitability calculation unit 24 calculates the suitability of the face image to be registered in the face dictionary based on information detected from the area showing the person's face. Then, the display control unit 27 displays first display information on the client device 1 (an example of a display device) that confirms the face images in order of decreasing suitability.

[0066] As a result, according to the first embodiment, the user can check the registered images narrowed down by the system according to their priority, so that the checking work after the registration of the face image can be performed more efficiently.

[0067] 10 is a diagram for explaining the effect of the first embodiment. It is preferable to be able to calculate an index capable of accurately evaluating an image unsuitable for dictionary registration, such as a face image with extremely poor image quality that may cause erroneous person recognition, and to be able to determine whether or not to register the image in the dictionary based on the index, but image quality is related to complex elements such as the face direction, image quality, and the degree of face concealment, and in the application of this embodiment in which the face dictionary conditions are given some leeway, the current AI (Artificial Intelligence) technology is not yet capable of automatically determining whether or not the image can be registered, and visual confirmation of the face image is required.

[0068] As shown in Fig. 10, in the past, since it was not clear where in the registered images the image to be confirmed was displayed, it was necessary to visually confirm all registered images, but according to the above-mentioned first embodiment, confirmation of registered images with lower suitability is prioritized, and confirmation of registered images with higher suitability (normal image area 401 with suitability of 0.8 or more in the example of Fig. 10) can be lowered in priority or omitted. This can reduce the burden of confirmation work on the user after registering face images.

[0069] For example, in the past, when tens of thousands of facial images needed to be registered, it was sometimes necessary to visually review the tens of thousands of facial images after registration; however, by visualizing images that are suitable for dictionary registration based on their suitability, it is possible to narrow down the images that require confirmation and set priorities.

[0070] As a secondary effect, this can assist the user in checking the face image after registration, further relaxing the dictionary registration conditions regarding the quality of the face image to be registered in the face dictionary. This also reduces the burden on the user who prepares the face image. In other words, even in cases where it is difficult to obtain images suitable for the face dictionary (for example, in the case of face recognition of video content such as television), dictionary registration can be permitted with reduced errors, and the face dictionary can be developed using face images with fewer restrictions, providing face recognition over a wider range.

[0071] In addition, since inappropriate image registration can be eliminated, it is possible to prevent a significant decrease in accuracy of the entire system caused by inappropriate images being mixed in. Even if an inappropriate image is mixed in, the cause can be quickly traced by the above-mentioned error display (for example, the error image area 101 in FIG. 6) and the sort display based on the appropriateness (for example, the confirmation area including the caution image area 102 and the normal image area 103 in FIG. 6).

[0072] (Modification of the first embodiment) Next, a modified example of the first embodiment will be described. In the description of the modified example, the same parts as those in the first embodiment will be omitted, and only the differences from the first embodiment will be described.

[0073] For example, when using face recognition to add meta information indicating the cast of a television program to the television program, it is necessary to perform face recognition over a long period of time from the person's childhood to the present. In the modified example, in order to handle such a case, the face dictionary DB stores face images of the same person from multiple periods (ages).

[0074] As a method of updating the face image by the memory control unit 26, for example, a method of updating the face image when the recognition accuracy is lower than the average through one video, regardless of which registered face image is used, is considered. In this case, when the recognition accuracy is lower than the average, the display control unit 27 selects a representative face image from the image of the person recognized from the video, and automatically registers it in the face dictionary DB. The suitability may be used to select this representative face image, or the suitability may be used to present the user with confirmation of the representative face image selected by some means. This allows the user to continue using high-precision face recognition with minimal confirmation work, due to the automatic update of the dictionary and the efficiency of the confirmation work after the update.

[0075] For example, in a modified example, when it is not possible to identify all the performers in a past television program, a method is used in which all the performers are identified by clustering the faces of people shown in the video.

[0076] Fig. 11 is a diagram showing an example of display information according to a modified example of the first embodiment. In the example of Fig. 11, the face clustering processing unit 30 aggregates (clusters) face images of people included in the content data based on the facial features of the people (e.g., performers in a TV program) included in the content data.

[0077] Based on the list of performers obtained by clustering, performers to be recognized are registered in a face dictionary DB. The suitability may be used to select this representative face image 401, or the representative face image 401 selected by some means may be presented to the user for confirmation using the suitability.

[0078] For example, when the suitability is used to select the representative face image 401, the suitability calculation unit 24 calculates the suitability of the face images aggregated by person by the face clustering processing unit 30. The display control unit 27 displays display information (third display information) that displays a face image with a higher suitability among the face images aggregated by person as the representative face image 401. Then, the display control unit 27 accepts the registration of the selected representative face image 401 in the face dictionary DB by a selection operation of the representative face image 401 in the third display information (display information 402). Note that the display control unit 27 may automatically register a face image with a higher suitability among the face images aggregated by person as the representative face image in the face dictionary without, for example, displaying the third display information or going through the above selection operation. According to this, it is possible to efficiently register a representative image of a person not registered in the dictionary among the people included in the content data by clustering in the face dictionary without prior preparation of a face image for dictionary registration.

[0079] Second embodiment Next, a second embodiment will be described. In the description of the second embodiment, the description of the same as in the first embodiment will be omitted, and only the points different from the first embodiment will be described. In the second embodiment, a case where management such as addition and updating of face images used for face authentication in an entrance / exit management system or the like is performed will be described.

[0080] In the server device 2 of the second embodiment, the face dictionary DB stores face images used for face authentication in an entrance / exit management system, etc. Then, the display control unit 27 controls the display of a confirmation screen similar to the display information (FIG. 6) of the first embodiment described above, thereby allowing a user such as an administrator to confirm the face image.

[0081] In the second embodiment, the similarity calculation unit 28 calculates the similarity between a face image in the face dictionary DB and a captured image of a person's face captured during face authentication.

[0082] In the case of face authentication in an entrance / exit management system or the like, it is expected that the similarity between the face image in the face dictionary DB and the captured image of the person may decrease due to aging of the person, etc. For example, the similarity calculation unit 28 may further store the history of the calculated similarity in the face dictionary DB so that the trend of the similarity over time can be grasped.

[0083] Then, when the similarity is on a downward trend and falls below a predetermined threshold, for example when replacing the facial image with the latest captured image of the person used in the similarity calculation unit 28, the display control unit 27 may use the suitability to select the most suitable representative facial image for dictionary registration, or may use the suitability to present the representative facial image selected by some means to the user for confirmation.

[0084] Finally, an example of the hardware configuration of the server device 2 in the first and second embodiments will be described. Note that the example of the hardware configuration of the client device 1 is the same as the example of the hardware configuration of the server device 2.

[0085] [Example of hardware configuration] 12 is a diagram showing an example of a hardware configuration of the server device 2 in the first and second embodiments. The server device 2 includes a processor 501, a main storage device 502, an auxiliary storage device 503, a display device 504, an input device 505, and a communication device 506. The processor 501, the main storage device 502, the auxiliary storage device 503, the display device 504, the input device 505, and the communication device 506 are connected via a bus 510.

[0086] It is not necessary for the server device 2 to have some of the above configurations. For example, if the server device 2 can use the input function and display function of an external device, the server device 2 does not need to have the display device 504 and the input device 505.

[0087] The processor 501 executes a program read from the auxiliary storage device 503 to the main storage device 502. The main storage device 502 is a memory such as a read only memory (ROM) and a random access memory (RAM). The auxiliary storage device 503 is a hard disk drive (HDD), a memory card, or the like.

[0088] The display device 504 is, for example, a liquid crystal display. The input device 505 is an interface for operating the server device 2. The display device 504 and the input device 505 may be realized by a touch panel or the like having a display function and an input function. The communication device 506 is an interface for communicating with other devices such as the client device 1.

[0089] For example, the program executed by the server device 2 is provided as a computer program product in the form of a file in an installable or executable format, recorded on a computer-readable storage medium such as a memory card, a hard disk, a CD-RW, a CD-ROM, a CD-R, a DVD-RAM, or a DVD-R.

[0090] Also, for example, the program executed by the server device 2 may be stored on a computer connected to a network such as the Internet, and may be provided by being downloaded via the network.

[0091] Also, for example, the program executed by the server device 2 may be provided via a network such as the Internet without being downloaded. Specifically, for example, it may be configured as an ASP (Application Service Provider) type cloud service.

[0092] Also, for example, the program of the server device 2 may be provided by being pre-installed in a ROM or the like.

[0093] The program executed by the server device 2 has a modular configuration including functions that can be realized by the program among the above-mentioned functional configurations. As for each function, as actual hardware, the processor 501 reads the program from a storage medium and executes it, and the above-mentioned functional blocks are loaded onto the main storage device 502. In other words, the above-mentioned functional blocks are generated on the main storage device 502.

[0094] Note that some or all of the above-described functions may be realized by hardware such as an integrated circuit (IC) instead of by software.

[0095] Furthermore, each function may be realized using a plurality of processors 501. In this case, each processor 501 may realize one of the functions, or may realize two or more of the functions.

[0096] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0097] 1. Client device 11 Input section 12 Display section 13. Communications Department 14 Storage section 2. Server equipment 21 Face image registration control unit 22 Content registration control section 23 Detection unit 24 Appropriateness Calculation Section 25 Extraction part 26 Memory control unit 27 Display control section 28 Similarity calculation unit 29 Output control section 100 Information Processing Systems 501 Processor 502 Main storage 503 Auxiliary storage device 504 Display device 505 Input Device 506 Communication Equipment 510 Bus

Claims

1. a face image registration control unit that accepts registration of a face image including a person's face; a detection unit that detects feature points of the face image and detects an area showing a person's face from the face image based on the feature points; an appropriateness calculation unit that calculates an appropriateness of a face image to be registered in a face dictionary based on information detected from an area showing the person's face; a display control unit that displays, on a display device, first display information that identifies the face images in descending order of suitability; A server device comprising:

2. The face image registration control unit accepts a plurality of face images at once, the display control unit displays the plurality of face images on the display device in descending order of the appropriateness. The server device according to claim 1 .

3. The detection unit determines whether the face image satisfies a dictionary registration condition, the first display information includes an error image area for displaying an error of a face image not satisfying the dictionary registration condition, and a confirmation area for confirming the face images in order of decreasing suitability, 3. The server device according to claim 1 or 2.

4. the confirmation region includes a caution image region in which a face image having a suitability level equal to or lower than a threshold is displayed, and a normal image region in which the suitability level is higher than the threshold, The display control unit displays the normal image area next to the attention image area. The server device according to claim 3.

5. the display control unit displays the error image area in a first display color that prompts a warning, displays the caution image area in a second display color that prompts a need for confirmation, and displays the normal image area in a third display color that indicates that the image is a normal face image. The server device according to claim 4.

6. At least one face image is registered for each person in the face dictionary; The first display information includes, for a person whose face images are registered in the face dictionary, a representative face image having a lower suitability.

3. The server device according to claim 1 or 2.

7. when the representative face image is selected, the display control unit further displays second display information for confirming a plurality of face images of the same person including the representative face image; the second display information displays a plurality of face images of the same person in order of decreasing suitability. The server device according to claim 6.

8. the appropriateness calculation unit calculates the appropriateness based on feature points of eyes, nose, and mouth detected from an area showing the person's face; 3. The server device according to claim 1 or 2.

9. a similarity calculation unit that compares features of a face image registered in the face dictionary with features of a face of a person included in content data to calculate a similarity; an output control unit that outputs output information including information indicating the similarity and information indicating a person identified by the similarity; The server device according to claim 1 or 2, further comprising:

10. the content data is an input video image input to the server device in real time, the similarity calculation unit compares features of the face image registered in the face dictionary with a face of a person included in the input video and calculates a similarity; The server device according to claim 9.

11. a face clustering processing unit that aggregates face images of people included in the content data for each person based on a facial feature amount of the person included in the content data, the appropriateness calculation unit calculates appropriateness of the face images aggregated for each person; the display control unit displays a face image having a higher suitability among the face images aggregated for each person as a representative face image, and further displays third display information for accepting registration of the selected representative face image in the face dictionary by a selection operation of the representative face image. The server device according to claim 9.

12. a storage control unit that, when receiving designation of a face image to be deleted and registration of a new face image from a user via the first display information, deletes the face image to be deleted from the face dictionary and replaces the face image to be deleted with the new face image, thereby registering the new face image in the face dictionary; The server device according to claim 1 or 2, further comprising:

13. The detection unit determines whether the face image satisfies a dictionary registration condition for confirming whether the face image is an image including a face, the appropriateness calculation unit detects feature points of the face image that satisfies the dictionary registration condition, and detects an area showing a person's face from the face image that satisfies the dictionary registration condition based on the feature points.

3. The server device according to claim 1 or 2.

14. A client device and a server device are provided, The server device includes: a face image registration control unit that accepts registration of a face image including a person's face; a detection unit that detects feature points of the face image and detects an area showing a person's face from the face image based on the feature points; an appropriateness calculation unit that calculates an appropriateness of a face image to be registered in a face dictionary based on information detected from an area showing the person's face; a display control unit that displays, on the client device, first display information that confirms the face images in order of decreasing suitability; The client device includes: A display unit that displays the first display information; an input unit that receives an operation input for the first display information from a user; An information processing system comprising:

15. A step in which a server device accepts registration of a face image including a person's face; a detection unit configured to detect feature points of the face image and detect a region showing a face of a person from the face image based on the feature points; the server device includes an appropriateness calculation unit that calculates an appropriateness of a face image to be registered in a face dictionary based on information detected from an area showing the person's face; a display control unit that displays, on a display device, first display information that identifies the face images in order of decreasing suitability; An information processing method comprising:

Images