Face tracking device and program

Abstract

To provide a face tracking apparatus which improves accuracy by merging a tracking processing result with a face recognition processing result.SOLUTION: A video analysis processing unit outputs information on motion prediction result of a face region by executing processing of motion prediction of the face region included in a video. A face recognition processing unit extracts a feature quantity of the face region included in the video, and executes face recognition processing of the face region on the basis of the feature quantity, to output first position information which is position information of the face region and a first pair of person identification information, which is information for identifying a person as a result of face recognition processing, and a recognition score corresponding to the person identification information. A tracking processing unit determines a track (tracking information) in a time direction of the face region. The track has second position information which is position information of the face region, and a second pair of person identification information of the face region and recognition score.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a face tracking device and a program. [Background technology]

[0002] Facial recognition technology, which identifies people by recognizing their faces in images, has become more accurate with the recent advances in AI (artificial intelligence) technology. Facial recognition technology has reached the point where it can be put to practical use in the security field.

[0003] Patent Document 1 describes face recognition technology. According to Patent Document 1, face recognition technology basically involves collecting face images for learning, calculating facial features for recognizing differences between people based on the images, and recognizing faces based on the similarity of the facial features.

[0004] The application of face recognition technology to video images, as described above, has led to the spread of efforts to efficiently add metadata such as personal identification information to video images, primarily among broadcasting companies. When extending image processing to video processing with time connections, object region tracking is a useful process.

[0005] Patent Document 2 describes a procedure for using the results of a tracking process for face recognition.

[0006] Patent Document 3 employs a tracking process that utilizes template matching with a previously prepared image.

[0007] Various proposals have been made so far regarding general object tracking technology in videos that does not specify the target as a human face. Representative examples of general tracking processing are described in Patent Document 4 and Patent Document 5.

[0008] Patent Document 4 describes a process for tracking an object by clarifying the correspondence between regions according to the similarity of feature amounts of object regions detected in different frames.

[0009] Patent Document 5 describes a statistical processing method called a particle filter. The tracking processing method described in Patent Document 5 aims to improve the reliability of tracking by weighting the correspondence between regions using a particle filter.

[0010] In recent years, deep learning has been introduced into the field of tracking technology, and the tracking accuracy has been greatly improved by using deep learning.

[0011] Non-Patent Document 1 describes many tracking processing techniques using deep learning processing. Non-Patent Document 2, which is also cited in Non-Patent Document 1, also describes tracking processing using deep learning processing. [Prior art documents] [Patent documents]

[0012] [Patent Document 1] JP 2017-033372 A [Patent Document 2] JP 2014-191479 A [Patent Document 3] JP 2014-021896 A [Patent Document 4] International Publication WO2018 / 163243 A1 [Patent Document 5] JP 2019-153112 A [Non-patent literature]

[0013] [Non-Patent Document 1] Gioele Ciaparrone,Francisco Luque Sanchez,Siham Tabik,Luigi Troiano,Roberto Tagliaferri,Francisco Herrera,Deep Learning in video multi-object tracking:A survey,ArXiv:1907.12740,2019,[Retrieved January 1, 2021],Internet<URL:https: / / arxiv.org / pdf / 1907.12740.pdf> [Non-Patent Document 2] Nicolai Wojke, Alex Bewley, Dietrich Paulus, Simple Online and Realtime Tracking with a Deep Association Metric, arXiv:1703.07402, 2017, [Retrieved January 1, 2021], Internet<URL:https: / / arxiv.org / pdf / 1703.07402.pdf> Summary of the Invention [Problem to be solved by the invention]

[0014] However, each of the background technologies has problems.

[0015] The tracking processes described in Patent Documents 2 and 3 are merely incidental processes to face recognition processing, and there is no semantic connection between the recognition results and tracking. The technologies in Patent Documents 2 and 3 have a problem in that they do not allow easy identification of people in a video based on the output of face recognition processing.

[0016] The tracking techniques described in Non-Patent Document 1 and Non-Patent Document 2 are intended to obtain track information of identical object regions. Non-Patent Document 1 and Non-Patent Document 2 do not describe fusion processing of advanced semantic information between the recognition result such as face recognition and the tracking result.

[0017] In addition, the conventional technology has a problem in that it cannot handle blurred areas or scene changes in the video. The present invention aims to realize consistent face recognition in the section where a person appears in the video.

[0018] The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide a face tracking device and program that can provide highly semantic information, such as a person's name, in the track information by combining the tracking processing results with the face recognition processing results. [Means for solving the problem]

[0019] [1] In order to solve the above problems, a face tracking device according to one embodiment of the present invention includes a video analysis processing unit that performs a motion prediction process for a face area included in a video and outputs information on the motion prediction result of the face area; a face recognition processing unit that extracts features of the face area included in the video and performs face recognition processing of the face area based on the features, thereby outputting first position information that is position information of the face area and a first set that is a set of person identification information that is information for identifying a person as a result of the face recognition processing and a recognition score corresponding to the person identification information; and a tracking processing unit that obtains a track of the face area in the time direction. the track has second position information which is position information of the face area, and a second set which is a set of personal identification information of the face area and a recognition score, and the tracking processing unit determines the second position information after the face area has moved based on information of a movement prediction result of the face area passed from the video analysis processing unit, and updates the track based on the first set passed from the face recognition processing unit and the second set held by the track, and also based on a relationship between the first position information and the second position information, thereby assigning new second position information and new second set to the track.

[0020] [2] Moreover, one aspect of the present invention is that, in the above-mentioned face tracking device, the face tracking device performs processing by the video analysis processing unit more frequently than processing by the face recognition processing unit, the video analysis processing unit accumulates information on the motion prediction results which are the result of multiple rounds of the motion prediction processing, and the tracking processing unit performs processing based on the accumulated information on the motion prediction results passed from the video analysis processing unit.

[0021] [3] Also, one aspect of the present invention is that in the face tracking device described above, the video analysis processing unit further detects a scene change in the video, and when the scene change is detected, the tracking processing unit resets the track to an initial state.

[0022] [4] Furthermore, in one aspect of the present invention, in the face tracking device described above, the video analysis processing unit further determines whether the face area is a blurred area, and for the face area that is a blurred area, the tracking processing unit sets the value of the recognition score to a minimum value (worst score value) when updating the track of the face area.

[0023] [5] Furthermore, in one aspect of the present invention, in the face tracking device described above, the face recognition processing unit is capable of accessing information associating the feature amount when the face region is a blurred region with special personal identification information associated with the feature amount of the blurred region, and the video analysis processing unit determines that the face region is the blurred region when the face recognition processing unit identifies the special personal identification information associated with the feature amount of the blurred region as a result of performing face recognition processing on the face region.

[0024] [6] Also, in one aspect of the present invention, in the face tracking device described above, the tracking processing unit updates the track based on a degree of overlap between the first position information and the second position information.

[0025] [7] In one aspect of the present invention, the face tracking device further includes an output shaping processing unit that shapes and outputs data of the set of tracks by adding a pair of information of the person identification information and the recognition score to the track information.

[0026] [8] In one aspect of the present invention, in the face tracking device of [7] above, the face tracking device further includes a search processing unit that acquires a search key from the person identification information, a person's name string associated with the person identification information, or a thumbnail image associated with the person identification information, and searches the data of the set of tracks output by the output formatting processing unit based on the acquired search key, thereby identifying the track associated with specific person identification information.

[0027] [9] Moreover, one aspect of the present invention includes a video analysis processing unit that performs a motion prediction process for a face region included in a video and outputs information on a motion prediction result of the face region; a face recognition processing unit that extracts features of the face region included in the video and performs face recognition processing on the face region based on the features, thereby outputting first position information that is position information of the face region and a first set that is a set of person identification information that is information for identifying a person as a result of the face recognition processing and a recognition score corresponding to the person identification information; and a tracking processing unit that obtains a track of the face region in the time direction, the track being the position information of the face region. and a second set which is a pair of personal identification information of the face area and a recognition score, and the tracking processing unit determines the second position information after the face area has moved based on information of a movement prediction result of the face area passed from the video analysis processing unit, and updates the track based on the first set passed from the face recognition processing unit and the second set held by the track, and also based on the relationship between the first position information and the second position information, thereby assigning new second position information and new second set to the track. Effect of the Invention

[0028] According to the present invention, it is possible to identify tracks and perform face recognition in new frames based on both information on feature quantities as an image of a face region and information on tracks (tracking of face regions across multiple frames) that take into account position, movement, etc. This is expected to improve both tracking accuracy and face recognition accuracy. [Brief description of the drawings]

[0029] [Figure 1] 1 is a block diagram showing a schematic functional configuration of a face tracking device according to an embodiment of the present invention; [Diagram 2] 4 is a schematic diagram showing the temporal relationship between video analysis processing and tracking processing in the face tracking device according to the embodiment. FIG. [Diagram 3] 10 is a schematic diagram showing a method for constructing a face database for blur determination processing integrated with face recognition processing in the embodiment. FIG. [Figure 4] 13 is a flowchart showing the procedure of a tracking process performed by a tracking processing unit in the embodiment. [Diagram 5] FIG. 11 is a schematic diagram showing an example of a time transition of a recognition score for a certain track (a face region track) in the embodiment. [Figure 6] 4 is a schematic diagram showing an example of a data structure obtained as a result of a tracking process performed by the face tracking device of the embodiment. FIG. [Figure 7] 10 is a schematic diagram showing an example of an information path for extracting a scene of video content in which a person appears, starting from a person's name or thumbnail image according to the embodiment. FIG. [Figure 8] 8 is a schematic diagram showing an example of a GUI (Graphical User Interface) for the video scene search described above (FIG. 7) in the embodiment. FIG. [Figure 9] 2 is a block diagram showing an example of the internal configuration of the face tracking device according to the embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] Next, an embodiment of the present invention will be described with reference to the drawings. The face tracking device of this embodiment realizes technical elements such as face recognition of people who appear continuously in time, a robust face area tracking method that can handle occlusion and temporary disappearance of people, and a robust method that can handle changes such as scene changes and motion blur in a video. This makes it possible to provide highly semantic information such as a person's name to the track information based on the result of face recognition. In addition, the face tracking device of this embodiment outputs data of the recognition process result to facilitate person search targeting the video. This makes it easy to develop application software that utilizes the face recognition result. For this purpose, the face tracking device of this embodiment outputs the process result in a data structure that integrates tracking process and face recognition process.

[0031] In this embodiment, a face is tracked based on temporally consecutive images (in other words, frame images that come before and after in time) while taking into consideration the temporal relationship. Furthermore, face tracking processing and face recognition processing are integrated. In this way, the face tracking device of this embodiment uses information related to face tracking (information on the position of the face area and information on predicting the movement of the face area) to improve the accuracy of the face recognition processing. Furthermore, the face tracking device of this embodiment uses information related to the face recognition processing (similarity of features of the face area (distance between features)) to improve the accuracy of face tracking.

[0032] FIG. 1 is a block diagram showing a schematic functional configuration of a face tracking device according to this embodiment. As shown in the figure, the face tracking device 1 includes an image acquisition unit 10, an image analysis processing unit 20, a face database 30, a face recognition processing unit 40, a tracking processing unit 50, and an output shaping processing unit 60. Each of these functional units can be realized, for example, by a computer and a program. Each functional unit also has a storage unit as necessary. The storage unit is, for example, a variable in the program or a memory allocated by the execution of the program. Also, non-volatile storage units such as a magnetic hard disk drive or a solid state drive (SSD) may be used as necessary. Also, at least a part of the functions of each functional unit may be realized as a dedicated electronic circuit rather than a program.

[0033] The face tracking device 1 inputs an image, recognizes a face contained in the image, and outputs data that integrates the face recognition result with the image information. The data output by the face tracking device 1 has a predetermined data structure. The functions of each part that constitutes the face tracking device 1 are as follows.

[0034] The video acquisition unit 10 acquires a video of a processing target (face tracking target) from outside. Here, "external" may be another device or a recording medium for recording data. The video is a sequence of continuous frame images at a predetermined frame rate (e.g., 30 frames per second, 60 frames per second, etc.).

[0035] The video analysis processing unit 20 analyzes the video acquired by the video acquisition unit 10. Specifically, the video analysis processing unit 20 performs motion prediction processing (extraction of motion parameter values) of objects, people, etc. between frame images constituting the video, performs scene change analysis processing, and performs blur detection for each region within the frame images. The video analysis processing unit 20 supplies the results of the video analysis processing to the tracking processing unit 50.

[0036] In other words, the video analysis processing unit 20 performs a process of predicting the movement of a face region included in a video, and outputs information on the result of predicting the movement of the face region (information on a model of the movement prediction).

[0037] The face database 30 is a database that stores a person ID, which is information for identifying a person, and feature amounts of face images in association with each other. The database itself is realized using existing technology. The database may use, for example, a magnetic disk device or a semiconductor storage device as a storage means. The feature amounts registered in the face database 30 are information obtained by machine learning processing or the like based on a large amount of collected face images. In other words, the face database 30 stores information on face recognition models.

[0038] In addition to the person ID, the face database 30 may also hold attribute information of the person (information such as genre, such as "politician" or "actor").

[0039] The face recognition processing unit 40 detects a face area in the video acquired by the video acquisition unit 10, and outputs a result of face recognition by comparing a feature amount calculated from the face area with a feature amount stored in the face database 30. The face recognition processing unit 40 outputs a person ID, which is information for identifying a person, as a result of the face recognition processing. Alternatively, the face recognition processing unit 40 may output a score for each person ID corresponding to a specific face area in the video. This score indicates the likelihood that a face included in the face area is the face of a person identified by the person ID. The face recognition processing itself can be performed using existing technology.

[0040] In other words, the face recognition processing unit 40 extracts features of the face area contained in the video, and performs face recognition processing of the face area based on the features, thereby outputting position information of the face area (referred to as first position information) and a set (referred to as the first set) of person identification information, which is information that identifies a person as a result of the face recognition processing, and a recognition score corresponding to the person identification information.

[0041] The tracking processing unit 50 utilizes the video analysis processing results from the video analysis processing unit 20 to perform tracking processing of the face area in the frame image output by the face recognition processing unit 40. In this way, the tracking processing unit 50 obtains a face area track that is continuous in time. Here, "continuous in time" means continuous between adjacent frame images. This improves the accuracy of face recognition.

[0042] That is, the tracking processing unit 50 obtains a track of the face area in the time direction. Here, the track is data of tracking information (trajectory information). The track has position information of the face area (called second position information) and a set of personal identification information and a recognition score of the face area (called the second set). The tracking processing unit 50 obtains the second position information after the face area moves based on the information of the movement prediction result of the face area passed from the video analysis processing unit 20, and updates the track based on the first set passed from the face recognition processing unit and the second set originally held by the track, and also based on the relationship (degree of overlap) between the first position information and the second position information. As a result, the tracking processing unit assigns new second position information and a new second set (a set of personal identification information and a recognition score) to the track.

[0043] The output shaping processor 60 outputs the tracking result data in a form shaped into a predetermined data structure. The structure of the data output by the output shaping processor 60 is a structure that is easily usable by a video interaction system such as a video search process, and includes face recognition result information and information related to the video. The data structure output by the output shaping processor 60 enables a function (application software, etc.) that uses the output data to effectively realize operations such as searches using people in the video as a key.

[0044] That is, the output shaping processor 60 shapes and outputs the data resulting from the tracking process passed from the tracking processor 50. As an example, the output shaping processor 60 shapes and outputs data of a set of tracks in which information on a pair of person identification information (person ID) and a recognition score is added to the track information. There may be multiple pairs of person identification information and recognition score.

[0045] The face tracking device 1 produces effects that could not be achieved by conventional techniques through the processing of the video analysis processing unit 20, the tracking processing unit 50, and the output shaping processing unit 60. Next, more detailed processing contents of each of the video analysis processing unit 20, the tracking processing unit 50, and the output shaping processing unit 60 will be described.

[0046] The video analysis processing unit 20 detects scene changes in the video, predicts global motion between frame images and local motion for each region, and determines whether a region is blurred. The results of these video analysis processes are supplied to the tracking processing unit 50 and the output shaping processing unit 60 together with the results of the face recognition processing unit 40, and are used to improve the accuracy of face recognition.

[0047] The detection of a scene change performed by the video analysis processing unit 20 is a process of detecting a break where the contents shown in the video change significantly. When a scene change occurs, it is necessary to reset the tracking process of the face in the video performed by the tracking processing unit 50. In order to detect a scene change, the video analysis processing unit 20 calculates the degree of correlation between the previous and next frame images. In a situation where a scene change does not occur, the degree of correlation between the previous and next frame images is maintained at a level equal to or higher than a predetermined threshold. When a scene change occurs, the degree of correlation between the previous and next frame images falls significantly below the above threshold. The video analysis processing unit 20 judges whether a scene change has occurred based on whether the degree of correlation falls below this threshold. This scene change detection itself can be performed using existing technology. The video analysis processing unit 20 outputs information indicating whether a scene change has occurred between a frame image at time (t-1) and a frame image at time t (where t is an integer value). The video analysis processing unit 20 conveys the result of the scene change detection to the tracking processing unit 50, for example, by outputting the value of the variable is_sc_change. is_sc_change=1 indicates "there is a scene change." is_sc_change=0 means "no scene change".

[0048] The motion prediction process performed by the video analysis processing unit 20 predicts the degree of movement of an object or person captured in an image between previous and subsequent frame images. The video analysis processing unit 20 passes the predicted value of the degree of movement to the tracking processing unit 50. This allows the tracking processing unit 50 to perform motion compensation processing according to the predicted value passed from the video analysis processing unit 20, thereby improving the accuracy of the tracking process.

[0049] In addition, the video analysis processing unit 20 predicts two types of motion in the motion prediction process: global motion (such as panning the shooting camera), which is the motion of the entire screen, and local motion, which is the motion specific to the detected face area. To predict global motion, the video analysis processing unit 20 calculates parameters that model the global motion of the entire screen based on the degree of correlation between frame images. A representative example of a global motion model is the affine model. The affine model itself is a model that can be processed by existing technology. Parameters for global motion are expressed by a matrix called globalM. Motion compensation using this matrix globalM can be calculated using the following equation (1).

[0050] p(t)=glabalM·p(t-1) ···(1)

[0051] In this equation (1), p(t-1) is a vector of two-dimensional coordinates (horizontal and vertical coordinates within the frame image) that represent a position on the frame image at time (t-1). Similarly, p(t) is a vector of two-dimensional coordinates that represent a position on the frame image at time t. The matrix globalM is a 2-row, 2-column matrix that represents the global motion from time (t-1) to time t.

[0052] Meanwhile, for local motion prediction, the video analysis processing unit 20 predicts facial motion within the target face region. To predict local motion, the video analysis processing unit 20 uses the framework of a Kalman filter. The processing procedure for predicting local motion is also disclosed in, for example, the above-mentioned Non-Patent Document 2. The model of local motion prediction differs for each region. The model of local motion for region r is expressed as kf(r). Motion compensation for region r from time (t-1) to time t using the prediction function kf(r).pred of the Kalman filter can be calculated using the following formula (2).

[0053] r(t)=kf(r).pred(r(t-1)) ···(2)

[0054] In this embodiment, the video analysis processing unit 20 performs more accurate motion prediction than the conventional method by combining the above global motion prediction and local motion prediction. The motion prediction from time (t-1) to time (t) by combining the global motion prediction and the local motion prediction is expressed by the following formula (3).

[0055] r(t)=integrateM(r(t-1))=kf(r).pred(globalM·r(t-1)) (3)

[0056] In equation (3), integrateM represents a motion prediction model that integrates global motion prediction and local motion prediction.

[0057] In order to appropriately adjust the overall processing load of the face tracking device 1, the face recognition process (the process of the face recognition processing unit 40), which takes a relatively long processing time, may be performed at a low rate. In other words, the face tracking device 1 can perform the motion prediction process at a relatively high frequency and the face recognition process at a relatively low frequency. If the frequency of the motion prediction process is performed at a low rate in accordance with the frequency of the face recognition process, the accuracy of the motion prediction may be reduced. However, as described above, the face tracking device 1 of this embodiment can perform the motion prediction process at a relatively high rate while performing the face recognition process at a relatively low rate. This makes it possible to maintain high accuracy of the motion prediction while reducing the load on the device for the face recognition process. In other words, the face tracking device 1 of this embodiment can maintain the accuracy of the motion prediction while suppressing processing costs by differentiating the rate of the face recognition process from the rate of the motion prediction process.

[0058] In other words, the face tracking device 1 performs the video analysis processing by the video analysis processing unit 20 more frequently than the face recognition processing by the face recognition processing unit 40. The video analysis processing unit 20 then accumulates information on the motion prediction result (the amount of motion possessed by the motion prediction model) which is the result of multiple motion prediction processes. The tracking processing unit 50 then performs processing based on the accumulated information on the motion prediction result passed from the video analysis processing unit 20.

[0059] The rate of the tracking process (the process of the tracking processing unit 50) and the process for data output (the process of the output shaping processing unit 60) is matched to the rate of the face recognition process.

[0060] FIG. 2 is a schematic diagram showing the temporal relationship between video analysis processing and tracking processing in the face tracking device 1. As shown in the figure, the video processed by the face tracking device 1 is a sequence of frame images. The video is a sequence of frame images at a rate of, for example, 30 FPS or 60 FPS. "FPS" stands for "frames per second." The video analysis processing unit 20 performs video analysis processing on each of these frame images and outputs the video analysis results. Meanwhile, the tracking processing unit 50 performs face tracking processing on a predetermined number of consecutive frame images that are appropriately determined. When the tracking processing unit 50 performs tracking processing, the video analysis results output by the video analysis processing unit 20 are used.

[0061] The video analysis process in Fig. 2 includes the above-mentioned motion prediction process. The tracking process is executed at the same frequency as the face recognition process, using the results of the face recognition process. When the processing is performed in the time relationship shown in Fig. 2, one face recognition process and one tracking process correspond to multiple motion prediction processes. At this time, the face tracking device 1 stores motion parameters obtained by the motion prediction process by the video analysis processing unit 20.

[0062] That is, the video analysis processing unit 20 obtains and outputs a video analysis result at time t based on a frame image at time (t-1) and a frame image at time t. Similarly, the video analysis processing unit 20 obtains a video analysis result based on two temporally adjacent frame images. That is, the video analysis processing unit 20 outputs a motion prediction parameter M1 (global motion prediction parameter) as a result of the first video analysis process. Similarly, the video analysis processing unit 20 outputs a motion prediction parameter Mk (global motion prediction parameter) as a result of the kth (k is a positive integer) video analysis process. The video analysis processing unit 20 accumulates the motion prediction parameters Mk (k=1, 2, . . .) for each time and calculates a global motion prediction parameter GlobalM for the tracking process. The video analysis processing unit 20 also accumulates a local motion prediction model for each face region r in the same way and calculates a local motion prediction parameter for the tracking process. The video analysis processing unit 20 passes these accumulated motion prediction parameters to the tracking processing unit 50 at the timing of the tracking process.

[0063] Similarly, the video analysis processing unit 20 can perform scene change detection processing at a high frequency. In this case, as a result of the first video analysis processing, is_sc_change_1 (the value is 0 (no scene change) or 1 (scene change) as described above) is obtained. Similarly, the video analysis processing unit 20 obtains is_sc_change_k as a result of the kth (k is a positive integer) video analysis processing. Then, the video analysis processing unit 20 obtains is_sc_change for tracking processing based on the determination result is_sc_change_k (k=1, 2, . . .) of the presence or absence of a scene change as a result of each video analysis processing. Note that if is_sc_change_k=1 in any one of k=1, 2, . . ., the video analysis processing unit 20 sets is_sc_change=1. Also, if is_sc_change=0 in all of k=1, 2, . . ., the video analysis processing unit 20 sets is_sc_change=0. The video analysis processing unit 20 passes the value of is_sc_change, which is the accumulated scene change determination result, to the tracking processing unit 50 at the timing of tracking processing. When is_sc_change=1, the tracking processing unit 50 resets the motion prediction parameters. Once a scene change is detected and the motion prediction parameters are reset, the video analysis processing unit 20 may stop all motion prediction processing until the timing of the next face recognition processing (and tracking processing).

[0064] The blur judgment performed by the video analysis processing unit 20 is to judge whether or not each face area in the video to be processed is a blurred area. The video analysis processing unit 20 detects the face area even when the face area is blurred. If face recognition processing is performed based on the video of the blurred area, the recognition accuracy will be poor. In other words, even if face recognition processing is performed on a blurred face image, the recognition accuracy is low and there is a high possibility that an incorrect person will be recognized. This leads to a decrease in the face recognition accuracy of the entire face tracking device 1. In order to avoid such factors that deteriorate accuracy, the face tracking device 1 of this embodiment performs blur judgment for each area (determines whether or not the area is blurred), and does not perform face recognition processing if the area is a blurred area. However, even if the face tracking device 1 does not perform face recognition processing, it passes information on the detection result of the face area to the tracking processing unit 50.

[0065] The video analysis processing unit 20 determines whether or not the face region r is a blurred region, and sets the determination result as the value of the variable is_blur(r). is_blur(r)=1 indicates that the region r is determined to be a blurred image. is_blur(r)=0 indicates that the region r is determined not to be a blurred image. The video analysis processing unit 20 determines whether or not each face region is a blurred region.

[0066] As an example, the video analysis processing unit 20 determines the presence or absence of blur by analyzing the edge components contained in the image of the face region r. This method itself can be realized using conventional technology. Specifically, the video analysis processing unit 20 detects the edge components in the face region r, and determines that there is "blur" (is_blur(r)=1) if the sharpness of the edge components is below a predetermined level (the image of that part does not have high frequency components). Also, the video analysis processing unit 20 determines that there is "no blur" (is_blur(r)=0) if the sharpness of the edge components in the face region r is above a predetermined level (the image of that part has high frequency components).

[0067] The video analysis processing unit 20 may determine the presence or absence of a blurred image by another method. In this method, the video analysis processing unit 20 determines the presence or absence of a blurred image at the same time that the face recognition processing unit 40 performs face recognition processing. That is, when this method is used, the face database 30 associates the feature amount of an image extracted from a large number of blurred face images with the person ID of the blurred image and stores it in advance. Thereby, when the feature amount of the face area r that is the target of the face recognition processing is compared with the feature amount held by the face database 30, it is indicated that the face area r is a blurred area. That is, the determination of the presence or absence of a blurred image by the video analysis processing unit 20 and the face recognition processing by the face recognition processing unit 40 can be performed as an integrated process. That is, it is possible to reduce the amount of calculation of the entire face tracking device 1.

[0068] FIG. 3 is a schematic diagram showing a method of constructing a face database for blur determination processing integrated with face recognition processing. As shown in the figure, in this method, a large number of blurred face images are prepared. Then, the face tracking device 1 extracts feature amounts of these blurred face images. The process of extracting image feature amounts can be performed using the functions of the face recognition processing unit 40. Then, the face tracking device 1 associates the obtained data of blurred face feature amounts with a person ID representing the blurred face, and registers it in the face database 30. It should be noted that it is sufficient to assign one person ID representing a blurred face to the blurred faces of an unspecified number of people.

[0069] That is, the video analysis processing unit 20 judges whether or not the face region is a blurred region. Based on the judgment result, the tracking processing unit 50 sets the recognition score value to the minimum value (worst score value) for the face region that is a blurred region when updating the track of the face region. One method of blur judgment is as follows. The face recognition processing unit 40 is previously made accessible to information (such as the face database 30) that associates the feature amount when the face region is a blurred region with the special person ID identification information associated with the feature amount of the blurred region. When the face recognition processing unit 40 identifies the special person ID associated with the feature amount of the blurred region as a result of performing face recognition processing on the face region, the video analysis processing unit 20 can judge that the face region is a blurred region.

[0070] Details of the tracking process by the tracking processing unit 50 are as follows. The tracking processing unit 50 performs a process of determining tracks of face areas of the same person from the correspondence on the time axis using the results output by the face recognition processing unit 40 and the results output by the video analysis processing unit 20. If the tracks of the face areas of the same person can be identified in the video by this process of the tracking processing unit 50, even if the reliability of the face recognition processing result is low in a frame image at a certain time, if a reliable face recognition processing result can be obtained in a frame image at another time belonging to the same track, then highly accurate face recognition can be achieved by replacement.

[0071] Fig. 4 is a flowchart showing the procedure of the tracking process by the tracking processing unit 50. The following description will be given along with this flowchart. Note that this flowchart shows the procedure of the tracking process at a certain point in time. In other words, the tracking processing unit 50 executes the process shown in Fig. 4 every time the tracking process is performed.

[0072] The data on which the tracking processing unit 50 processes at time t is as follows. The tracking processing unit 50 acquires output signals (data) from the face recognition processing unit 40 and the video analysis processing unit 20. The face recognition processing unit 40 passes to the tracking processing unit 50 the detected face area (position information within the frame image), the recognition score which is the result of the face recognition processing, the person ID which is the recognition result, and the feature amount of the face area. The video analysis processing unit 20 passes to the tracking processing unit 50 a scene change detection signal is_sc_change and an integrated movement prediction parameter integrateM of the face area.

[0073] The tracing processing unit 50 also holds a set of tracks that have been traced at the previous tracing process, i.e., at time (t-1). The set of tracks at time (t-1) is as shown in the following formula (4). In formula (4), n_Tr is the number of tracks included in the set of tracks.

[0074] Tr(t-1) = [tr[0](t-1), tr[1](t-1), …, tr[n_Tr-1](t-1)] ···(4)

[0075] Each track tr[k](t-1) (0≦k≦n_Tr-1) is a data structure, and the elements of the structure include the following bbox, feat, recog_ids, and recog_scores. The bbox is rectangular area information (position information within a frame image) of a face area corresponding to a track at time (t-1). feat is the feature amount of the face region in the track at time (t-1). recog_ids is a candidate of a person ID of a face recognition result corresponding to the track at time (t-1). There may be a plurality of candidates of a person ID. The candidates of a person ID are arranged, for example, in descending order of score. recog_scores is the score value corresponding to the person ID candidate in the track at time (t-1), and is sorted in the same order as the person IDs are arranged (for example, from highest to lowest score).

[0076] The notation for accessing elements of the structure is as follows. For example, the bbox (rectangular area information) for the entire set of tracks at time (t-1) is expressed as Tr(t-1).bbox. For example, the bbox for an individual track at time (t-1) is expressed as tr[k](t-1).bbox, where k is the index value of an individual track within the set of tracks, and 0≦k≦n_Tr-1.

[0077] Before the first process of the tracking processing unit 50 (i.e., at time 0), the track set is initialized. That is, initialization is performed so that Tr(0)=[] (empty set). When the track is reset due to a scene change, the track set is also reset to an empty set.

[0078] The face detection area and face recognition result information at time t that the tracking processing unit 50 receives from the face recognition processing unit 40 is as shown in the following formula (5). In formula (5), n_Det is the number of face detection areas (i.e., the number of face recognitions) in the face recognition process at time t.

[0079] Det(t) = [det[0](t), det[1](t), …, det[n_Det-1](t)] ···(5)

[0080] As in the case of tracks, each face detection area det[k](t) (0≦k≦n_Det-1) is a data structure, and the elements of the structure include the following bbox, feat, recog_ids, and recog_scores. bbox is rectangular area information (position information within a frame image) of the face detection area at time t. feat is the feature amount of the face detection area at time t. recog_ids is a candidate of a person ID of a face recognition result corresponding to the face detection area at time t. There may be a plurality of candidates of a person ID. The candidates of a person ID are arranged, for example, in descending order of score. recog_scores is a score value corresponding to the person ID candidates in the face detection area at time t, and is sorted in the same order as the person IDs are arranged (for example, in descending order of score).

[0081] The notation for accessing elements of the structure is as follows. For example, the bbox (rectangular area information) for the entire set of face detection areas at time t is expressed as Det(t).bbox. For example, the bbox for an individual face detection area at time t is expressed as det[k](t-1).bbox, where k is the index value of an individual face detection area within the set of face detection areas, and 0≦k≦n_Det-1.

[0082] The data handled by the tracking processing unit 50 further includes a matrix representing the degree of overlap between rectangular regions and a matrix representing the distance between feature amounts.

[0083] The matrix representing the degree of overlap between rectangular regions is expressed as IoU(bboxA, bboxB). IoU stands for "Intersection over Union" (the ratio of the intersection of the regions to the union of the regions). The tracking processing unit 50 obtains the degree of overlap between the rectangular region of the track (above bboxA) and the rectangular region of the face detection region (above bboxB), and uses it for tracking processing. This matrix is ​​a two-dimensional matrix with a size of n_bboxA×n_bboxB. Here, n_bboxA is the number of regions of bboxA, and n_bboxB is the number of regions of bboxB. When the value of an element of this matrix is ​​1, the corresponding regions completely match. When the value of the element is a positive value (0 or more and 1 or less), the smaller the value, the smaller the degree of overlap between the corresponding regions. When the value of the element is a negative value, the corresponding regions do not have any overlap at all, and the larger the absolute value of the value, the greater the distance between the two regions.

[0084] The matrix representing the distance between the features is expressed as Dis(FeatA, FeatB). The tracking processing unit 50 obtains the distance between the feature of the rectangular area of ​​the track (FeatA above) and the feature of the rectangular area of ​​the face detection area (FeatB above), and uses it for tracking processing. This matrix is ​​a two-dimensional matrix with a size of n_FeatA×n_FeatB. Here, n_FeatA is the number of features of FeatA (same as the number of rectangular areas of the track), and n_FeatB is the number of features of FeatB (same as the number of rectangular areas of the face detection area). The value of the distance is 0 or a positive value. If the two features are the same, the distance between the two features is 0. The more dissimilar the two features are, the greater the distance between the features.

[0085] The tracking processing unit 50 performs tracking processing based on the above data. In other words, the tracking processing performed by the tracking processing unit 50 is reduced to a problem of associating a track set Tr(t-1) at time (t-1) with a set Det(t) of face detection areas at time t. The tracking processing unit 50 performs the process shown in FIG. 4 once each time time passes (for example, from time (t-1) to time t), and by repeating this process, the data of the track set Tr(t) is updated sequentially.

[0086] The processing procedure of the tracking processing unit 50 at time t is as follows. First, in step S11, the tracking processing unit 50 judges whether or not a scene change has occurred since the previous tracking processing based on information passed from the video analysis processing unit 20. The presence or absence of a scene change is indicated by the value of the variable is_sc_change. If a scene change has occurred, that is, if the value of is_sc_change is 1 (step S11: YES), the process jumps to step S21. If no scene change has occurred, that is, if the value of is_sc_change is 0 (step S11: NO), the process proceeds to the next step S12, with the track set at time (t-1) as the current track set (existTr=Tr(t-1)).

[0087] Next, in step S12, the tracking processing unit 50 performs a motion correction process. That is, the tracking processing unit 50 performs a motion correction process using the integrated motion prediction model integrateM passed from the video analysis processing unit 20 for each track in the current track set existTr.bbox.

[0088] Next, in step S13, the tracking processing unit 50 calculates the overlapping degree of the regions. Specifically, the tracking processing unit 50 expands the rectangular region obtained as a result of the motion compensation process in step S12 by sc times. This expansion is for covering the motion compensation error that may be included in the motion compensation process. The value of the expansion magnification sc is an appropriately determined value equal to or greater than 1.0. For example, 1.5≦sc≦2.5 may be used. In the present embodiment, as an example, sc=2.0 is used. This expansion does not change the center position of the rectangular region, and the size of the rectangular region becomes sc times. Then, the tracking processing unit calculates the overlapping degree between the expanded rectangular region and Det(t).bbox passed from the face recognition processing unit 40. Then, the tracking processing unit 50 performs threshold processing on each element of the overlapping degree matrix (the above-mentioned IoU) with a predetermined threshold value to create a threshold-processed matrix GateMtx. That is, in the thresholded matrix GateMtx, regions having a degree of overlap below a certain threshold are treated as regions that do not overlap at all with each other.

[0089] That is, the tracking processing unit 50 performs processing for updating the track based on the degree of overlap between the first position information and the second position information.

[0090] Next, in step S14, the tracking processing unit 50 compares the feature amount of the face area in the current track with the feature amount of the face area at time t passed from the face recognition processing unit 40. That is, the tracking processing unit 50 obtains a matrix DisMtx that represents the distance between the feature amount ExistTr.feat in the current track and the feature amount Det(t).feat passed from the face recognition processing unit 40.

[0091] Next, in step S15, the tracking processing unit 50 performs a region filtering process. That is, the tracking processing unit uses the GateMtx calculated in step S13 to filter out invalid components included in the matrix DisMtx calculated in step S14. In other words, if the face region of the track and the face region of the face detection region are treated as regions that do not overlap with each other at all as a result of the threshold process in step S13, the tracking processing unit 50 does not associate the track with the face detection region.

[0092] Next, in step S16, the tracking processing unit 50 determines the correspondence between the face area track up to time (t-1) and the face detection area at time t based on DisMtx obtained as a result of step S15. In other words, the tracking processing unit 50 performs correspondence between the track up to time (t-1) and the face detection area at time t based on the degree of overlap between the areas (however, threshold processing has been performed) and the distance between the feature amounts of each area. In this step, the tracking processing unit 50 obtains the above correspondence using, for example, the Hungarian algorithm. The Hungarian algorithm itself is an existing technology. In other words, the tracking processing unit 50 attempts to obtain correspondence between the number (tr_no) of the face area track and the number (det_no) of the face detection area. The tracking processing unit 50 branches the processing of the face area as follows according to the obtained correspondence. If the face area track and the face detection area correspond (matched_pairs), the process proceeds to step S17. If there is no face area track corresponding to the face detection area (unmatched_dets), proceed to step S18. If there is no face detection area corresponding to the face area track (unmatched_tracks), proceed to step S19. Note that the above matched_pairs is a set of combinations of face area track numbers and face detection area numbers (matched_pairs ∈ {(tr_no, det_no)}). Also, the above unmatched_dets is a set of face detection area numbers (unmatched_dets ∈ {det_no}). Also, unmatched_tracks is a set of face area track numbers (unmatched_tracks ∈ {tr_no}).

[0093] The tracking processing unit 50 performs the condition branch in step S16 for each area (for each matched_pair, for each unmatched_det, and for each unmatched_tracks). Then, the tracking processing unit 50 performs the processes in the following steps S17, S18, and S19 for each area classified in step S16.

[0094] Next, in step S17, the tracking processing unit 50 adds information on the relevant face detection area (Detection) to the corresponding existing face area track (Track).

[0095] Next, in step S18, the tracking processing unit 50 registers a new face area track (Track) based on the corresponding face detection area (unmatched detection). That is, the tracking processing unit 50 adds the face detection area as a new track to the existing track set.

[0096] Next, in step S19, the tracking processing unit 50 registers the corresponding face area track (unmatched track) as a lost track or deleted track. Specifically, the tracking processing unit 50 may mark an existing track as lost when there is no face detection area corresponding to the existing track, and if this lost state continues for a predetermined period, may mark the track as terminated (deleted) and exclude it from the target of subsequent tracking processing.

[0097] When the processes of steps S17, S18, and S19 are completed for all the regions, the tracking processing unit 50 updates the Tracks in step S20. That is, the tracking processing unit 50 updates the tracks based on the correspondence determined in step S16, and generates a track set Tr(t) at time t.

[0098] When updating the track, if the region tr[i](t) is a blurred face region (is_blur=1), the tracking processing unit 50 forcibly sets the recognition score of the region to the minimum value (usually 0.0). That is, the tracking processing unit 50 sets tr[i](t).recog_score=0.0 for such a region.

[0099] On the other hand, if the process proceeds to step S21 (if the determination in step S11 indicates that a scene change has occurred), the tracking processor resets the track. That is, the track at time t is set as Tr(t)=[] (empty set). In other words, in this case, the track information up to Tr(t-1) will no longer be used.

[0100] That is, when a scene change is detected, the tracking processing unit 50 resets the track to the initial state.

[0101] 4 described above, the tracking processing unit 50 outputs a plurality of face area tracks having information on the face recognition result. The tracking processing unit 50 passes the data of the plurality of face area tracks to the output shaping processing unit 60.

[0102] The output shaping processor 60 shapes and outputs the data of the result of the tracking process passed from the tracking processor 50. Specifically, the output shaping processor 60 shapes the results of the face recognition process for each frame image of the track output by the tracking processor 50 (the track after updating by the tracking process in FIG. 4) so ​​that they are output in descending order of the recognition score. Taking the track tr[k] (k is an index value for a specific track) at time t as an example, it is as follows.

[0103] Up to time (t-1), the track contains the following data related to face recognition: tr[k](t-1).recog_scores is a list of the recognition scores up to time (t-1). Here, the recognition scores are arranged in descending order. tr[k](t-1).recog_ids is a sequence of person IDs corresponding to the above tr[k](t-1).recog_scores. In other words, tr[k](t-1).recog_ids is a sequence of person IDs (recognition results) sorted in descending order of recognition scores up to the time (t-1) of the track.

[0104] Here, assume that the face detection area det[j] at time t is added to the track tr[k]. Then, det[j].recog_scores and det[j].recog_ids, which are the face recognition results for the face detection area det[j], are also added to the track tr[k] as information on the recognition result. At this time, in addition to the information on the recognition score and person ID (recognition result) originally held by the track tr[k], the newly added information is also sorted in descending order of the recognition score. This sorting is performed by the output shaping processing unit 60. This process is expressed by the following formulas (6) and (7).

[0105] tr[k](t).recog_scores = sort([tr[k](t-1).recog_scores+det[j].recog_scores],topK) ···(6) tr[k}(t).recog_ids = argsort([tr[k](t-1).recog_ids+det[j].recog_ids],topK) ···(7)

[0106] The process sort(listA,topK) shown in formula (6) represents a process of rearranging only the topK elements of the value list listA in order of their values ​​(here, in descending order). The operator "+" in formula (6) is an operator that concatenates lists. In other words, formula (6) represents a sort process for a list formed by concatenating the recognition score list tr[k](t-1).recog_scores and the recognition score list det[j].recog_scores.

[0107] Moreover, the process argsort(listA,topK) shown in formula (7) represents a process of sorting only the topK sentences in listA in descending order using the corresponding recog_scores (recognition scores) as a sort key. Here again, the operator "+" is an operator that concatenates lists. In other words, formula (7) represents a process of sorting the list of person IDs tr[k](t-1).recog_ids and the list of person IDs det[j].recog_ids in the order corresponding to formula (6).

[0108] In this manner, in this embodiment, the process of updating a track is repeated so that the maximum recognition score for a certain person ID remains over the time axis for a certain track. This makes it possible to perform face recognition for a track based on the recognition result for a time period in which a certain face region has a high score, without being influenced by the recognition result for a time period in which a certain face region has a low score (including a time period in which the face region is blurred).

[0109] That is, the face tracking device 1 holds and updates the face recognition result data (person ID and recognition score) sorted in descending order of the recognition score for each track. This makes it possible to obtain stable recognition results for people appearing in the video to be processed.

[0110] 5 is a schematic diagram showing an example of the time transition of the recognition score for a certain track. As shown in the figure, this track has person IDs of person A and person B as candidates for the recognition result. For this track, the score for person A is 0.5 at time t1 and 0.8 at time t2. On the other hand, the score for person B is 0.6 at time t1 and 0.6 at time t2.

[0111] Such a score occurs, for example, when the correct answer for face recognition for this track is "Person A," and at time t1, Person A is looking sideways, causing the recognition score for "Person A" to drop to 0.5, and at time t2, Person A is looking forward, causing the recognition score to rise to 0.8. In other words, in this case, at time t1, the score for Person A, which should be the correct answer, is lower than the score for Person B, which is the incorrect answer. <t2である。

[0112] In the above case, the data held by the track at time t1 is as follows: tr(t1).recog_socres = [0.6,05] tr(t1).recog_ids = [B,A] At time t2, the track contains the following data: tr(t2).recog_scores = [0.8,0.5] tr(t2).recog_ids = [A,B]

[0113] The recognition score at time t1 and the recognition score at time t2 are integrated as follows: tr.recog_score = [0.8,0.6] tr.recog_ids = [A,B] In other words, as a recognition result of tracking through time, the face tracking device 1 preferentially uses the state of person A when he / she is facing forward, which has a higher score. As a result, the face tracking device outputs person A, which is the correct answer, as a higher recognition result than person B. In this way, the face tracking device 1 uses the person with the highest recognition score during the period of tracking through time as the recognition result. This enables the face tracking device 1 to realize a more reliable recognition process.

[0114] As described above, when the blur detection result is_blur is 1, the face tracking device 1 forcibly sets the face recognition score of the track at that time to 0.0. In other words, the recognition result when the face area of ​​the track is blurred does not affect the calculation of the recognition score of the entire track through time.

[0115] As described above, the face tracking device 1 tracks a face appearing in a video and identifies the person ID of the face. This allows the face tracking device 1 to obtain and output data on candidate person IDs and scores associated with the person IDs for each track.

[0116] FIG. 6 is a schematic diagram showing an example of a data structure obtained as a result of the tracking process by the face tracking device 1. As shown in the figure, the face tracking device 1 can output a set of data having a track as an entity for a specific video content. The primary key of the track shown in the figure is, for example, a track ID. The video content to be processed can have a unique content ID. In other words, one content ID is associated with multiple (multiple) tracks. The attributes of the track include a start time (start frame), an end time (end frame), a recognition score (list), and a person ID (list). Each of the start time and the end time is a relative time in the video content. The start time corresponds to the start frame. The end time corresponds to the end frame. The accuracy of the start time and the end time of each track depends on the frequency of the tracking process (see FIG. 2). In other words, by performing the tracking process sufficiently frequently (for example, once per second or once per 0.5 seconds, etc.), the accuracy of the start time and the end time of the track is sufficient for practical use. However, there is no restriction on the frequency of the tracking process. The recognition score is a list of score values ​​corresponding to the person ID. The recognition score is a list of data in which scores are sorted in descending order. The person ID is a list of people who appear in the video. The person IDs are sorted in descending order of their corresponding recognition scores. In this way, by searching the data of a set of tracks using the person ID as a key, it is possible to obtain the scene (start time and end time) in the video in which the relevant person appears with a specified accuracy.

[0117] As shown in FIG. 6, the person ID is identification information for uniquely identifying a person. The person ID is associated with data of one or more person names (notations). The person ID is also associated with data of one or more thumbnail images. In other words, it is possible to determine the person ID from the notation of the person's name. Also, when a thumbnail image is selected, it is possible to determine the person ID.

[0118] FIG. 7 is a schematic diagram showing an example of an information path for extracting a scene of a video content in which a person appears, starting from the person's name or thumbnail image. As described with reference to FIG. 6, for example, when a person's name is input as a character string or a specific thumbnail image is selected from a large number of thumbnail images (images of the person's face) presented, it is possible to identify a person ID. Once a person ID is obtained, it is possible to identify one or more sets of information on the content ID of the video content corresponding to the person ID and its relative position (start time (start frame) and end time (end frame)) by searching the data of the set of tracks shown in FIG. 6. This makes it possible to present a video scene corresponding to the person ID to a user, etc.

[0119] FIG. 8 is a schematic diagram showing an example of a GUI (Graphical User Interface) for searching for a video scene described in FIG. 7. In other words, the figure shows an example of a GUI of a video search system based on a face recognition result. This video search system can search for a video using track data obtained as a result of processing by the face tracking device 1 of this embodiment. The screen shown in the figure is a display screen of a PC, a tablet terminal, a mobile terminal such as a smartphone, etc. In the figure, 701 is a video display area. Also, 702 is a person name display area. Also, 703 is a thumbnail image display area. Also, 704 is a scene specific information display area. A user of this system can select and click (or tap) one of the person names (character strings) displayed in the person name display area 702 or one of the thumbnail images (face images) displayed in the thumbnail image display area 703. As a result, the video search system can extract a track corresponding to the person, as shown in FIG. 7. Then, based on the extracted track information, the video search system displays a specific scene (relative position) of a specific video content in the video display area 701. If there are multiple tracks related to the selected person, for example, the video search system may select a track having a high score for the person based on the value of the recognition score.

[0120] The face tracking device 1 itself may be provided with a search processor to perform the video search shown in FIG. 8. In this case, the search processor acquires a search key from a GUI screen or the like, which is one of a person ID (person identification information), a character string representing a person's name associated with the person ID, and a thumbnail image associated with the person ID. The search processor searches the data of a set of tracks output by the output shaping processor 60 based on the acquired search key, thereby identifying one or more tracks associated with a specific person ID. The track identified in this way has a content ID and information on a relative position within the video content. Therefore, the search processor can acquire a video scene identified by the track from a storage device or the like, decode it, and display it as a video on the screen (video display area 701).

[0121] FIG. 9 is a block diagram showing an example of the internal configuration of a face tracking device. The face tracking device 1 can be realized using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions included in a program read from the RAM 902 or the like. The central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations according to each instruction. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. Note that RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with an external input / output device or the like. The input / output devices 904 and 905 are input / output devices. Input / output devices 904 and 905 exchange data with the central processing unit 901 via an input / output port 903. A bus 906 is a common communication path used inside the computer. For example, the central processing unit 901 reads and writes data from and to a RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port via the bus 906.

[0122] At least some of the functions of the face tracking device 1 can be realized by a computer. In that case, a program for realizing the functions may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the functions. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to a portable medium such as a flexible disk, an optical magnetic disk, a ROM, a CD-ROM, a DVD-ROM, a USB memory, and a storage device such as a hard disk built into a computer system. In other words, the term "computer-readable recording medium" may be a non-transitory computer-readable recording medium. The term "computer-readable recording medium" may also include a medium that temporarily and dynamically holds a program, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. The above program may be a program for realizing some of the functions described above, or may be a program that can realize the functions described above in combination with a program already recorded in the computer system.

[0123] Although the embodiment has been described above, the present invention can also be embodied in the following modified examples.

[0124] [First Modification] For example, the face recognition processing by the face recognition processing unit 40 may be performed using a machine learning means such as a neural network. In this case, the image feature amount for each person ID held in the face database 30 is represented and held as a set of internal parameter values ​​in the neural network. In other words, information on the relationship between the image feature amount and the person ID is embedded as a parameter value of each node in the neural network.

[0125] [Second modified example] In the above embodiment, the face area is tracked from the past side to the future side of the video (in the forward direction) and the track is updated. This direction may be reversed. That is, the face tracking device 1 may track the face area from the future side to the past side of the video (in the reverse direction) and update the track. The face tracking device 1 may also perform a combination of forward tracking and reverse tracking. This makes it possible to identify multiple tracks in a video file from start to finish and to provide recognition results for the tracks.

[0126] As described above, according to this embodiment (including its modified examples, the same applies below), by combining face recognition processing of a person appearing in a video with processing for tracking a face area, both face recognition accuracy and tracking accuracy are improved. The face tracking device 1 of this embodiment can provide consistent recognition results for areas in a video where a person appears over time.

[0127] In addition, this embodiment utilizes the detection result of a scene change (cut) during face tracking processing. That is, in this embodiment, when a scene change is detected, the track that was being tracked is initialized. This allows appropriate tracking to be performed even when a scene change is included in a long video, for example. In other words, it is possible to prevent confusion in processing.

[0128] Furthermore, in this embodiment, in order to deal with blurred images, which are a factor in reducing the accuracy of the recognition process, a process for detecting blurred images is actively performed, and when a blurred image is detected, the recognition score is set to the lowest value, thereby reducing the impact of blur on recognition.

[0129] In addition, in this embodiment, the rates (frequencies) of the two types of processing are changed and these processes are combined. Specifically, the video analysis process by the video analysis processing unit 20 and the recognition and tracking processes by the face recognition processing unit 40 and tracking processing unit 50 can be performed at different rates. As a result, in this embodiment, motion predictions that are the results of the two types of motion prediction processes are accumulated and used during tracking processing. This makes it possible to perform the necessary processes at a high rate (frequency) while reducing the processing load on the entire device.

[0130] Furthermore, this embodiment outputs data in units of the trajectory (track) of the face area, and assigns recognition result data to the track. By outputting data in this format, this embodiment is able to provide the most likely recognition result (single or multiple) for each track. Such output data is suitable for searching for video scenes in which a person appears, using the person ID as a key. This makes it easy to perform video search starting from a person. This embodiment also presents an example of a GUI for such video search.

[0131] In other words, this embodiment makes it possible to perform face recognition of people appearing in a video in a consistent format that also responds to changes over time. By devising a data structure that allows the recognition results to be easily used to add metadata to the target video, it becomes easy to develop a search system that can effectively search for characters.

[0132] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Industrial Applicability]

[0133] The present invention can be used, for example, in video content production businesses, video content distribution businesses, video content management businesses, video content search service businesses, etc. However, the scope of use of the present invention is not limited to the examples given here. [Explanation of symbols]

[0134] 1. Face Tracking Device 10. Video Acquisition Unit 20 Video analysis processing section 30 Face Database 40 Face recognition processing unit 50 Tracking processing unit 60 Output Formatter 701 Video display area 702 Person name display area 703 Thumbnail image display area 704 Scene specific information display area 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus

Claims

1. a video analysis processing unit that performs a process of predicting the movement of a face area included in a video and outputs information on the result of predicting the movement of the face area; a face recognition processing unit that extracts features of the face area included in the video, and performs face recognition processing on the face area based on the features, thereby outputting first position information that is position information of the face area, and a first set that is a set of person identification information that is information for identifying a person as a result of the face recognition processing, and a recognition score corresponding to the person identification information; a tracking processing unit for obtaining a track of the face region in a time direction; Equipped with the track has second position information which is position information of the face region, and a second set which is a set of person identification information of the face region and a recognition score, the tracking processing unit obtains the second position information after the face area has moved based on information of a motion prediction result of the face area delivered from the video analysis processing unit, and updates the track based on the first set delivered from the face recognition processing unit and the second set held by the track, and also based on a relationship between the first position information and the second position information, thereby providing the track with new second position information and new second set; The image analysis processing unit further determines whether the face region is a blurred region, the tracking processing unit, with respect to the face region which is the blurred region, sets the value of the recognition score to a minimum value when updating the track of the face region, the face recognition processing unit is capable of accessing information associating the feature amount when the face region is a blurred region with special person identification information associated with the feature amount of the blurred region; When the face recognition processing unit identifies special person identification information associated with the feature amount of the blurred region as a result of performing face recognition processing on the face region, the image analysis processing unit determines that the face region is the blurred region. Face tracker.

2. the face tracking device performs the processing by the video analysis processing unit more frequently than the processing by the face recognition processing unit; The video analysis processing unit accumulates information on the motion prediction result, which is a result of a plurality of times of the motion prediction processing, The tracking processing unit performs processing based on the accumulated information of the motion prediction result delivered from the video analysis processing unit. The face tracking device of claim 1 .

3. The video analysis processing unit further detects a scene change in the video, The tracking processing unit resets the track to an initial state when the scene change is detected.

3. A face tracking device according to claim 1 or 2.

4. The tracking processing unit updates the track based on a degree of overlap between the first position information and the second position information. A face tracking device according to any one of claims 1 to 3.

5. an output shaping processing unit that shapes and outputs data of the set of tracks in a form in which information of the tracks is added with information of a pair of the person identification information and the recognition score; The face tracking device according to claim 1 , further comprising:

6. a search processing unit that acquires a search key selected from the person identification information, a character string representing a person's name associated with the person identification information, and a designation of a thumbnail image associated with the person identification information, and searches the data of the set of tracks output by the output shaping processing unit based on the acquired search key, thereby identifying the track associated with specific person identification information; The face tracking device of claim 5 further comprising:

7. A program for causing a computer to function as a face tracking device described in any one of claims 1 to 6.

Images