Person identification device and its program

Abstract

To provide a person identification device with highly reliable identification results. [Solution] The person identification device 2 comprises a face region detection unit 12, a face region identification unit 13, a body region detection unit 14, a body region identification unit 15, a correspondence relationship processing unit 23 that associates face regions and body regions based on the degree of overlap, a graph generation unit 24 that generates a graph in which face regions and body regions become nodes and edges represent the degree of overlap, a grouping processing unit 31 that groups face regions and body regions by performing connected component analysis on the graph, and an identification result output unit 41 that outputs the registered person with the highest identification score in the same group as the identification result. It is equipped with.

Description

Technical Field

[0001] The present invention relates to a person identification device and its program.

Background Art

[0002] Person identification technology is a technology for identifying and specifying a person in a video. With the recent progress of AI (Artificial Intelligence) technology, it has become more accurate and has reached the stage of practical application in the fields of security and personal authentication.

[0003] Person identification technology mainly identifies a person based on face identification processing, and a method for tracking a person in a video to improve the identification accuracy is known (for example, Patent Document 1). In the method described in Patent Document 1, the same face area is tracked in the time direction, and the most probable identification result is output within that range. The method described in Patent Document 1 can identify a person stably and with high accuracy within the range where the same face area can be tracked. On the other hand, in the method described in Patent Document 1, if the tracking of the face area fails once, it is necessary to perform face identification processing again. Therefore, even if there is information about a person (person information) in the vicinity, that person information cannot be reflected in the identification result.

[0004] Therefore, a multi-modal person identification technology has been proposed that extracts person information not only through face identification processing but also through other modalities and reflects it in the identification result (Patent Document 2, Non-Patent Document 1). The method described in Patent Document 2 fuses the tracking results of the face area and the person area, and calculates and determines the overlapping degree of both areas. In addition, in the method described in Non-Patent Document 1, the voice section information of the speaker is also fused, and the temporal overlap of the processing results of each modal signal is obtained.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

[0006] [Non-Patent Document 1] “Face, Body, Voice: Video Person-Clustering with Multiple Modalities”, Andrew Brown, Vicky Kalogeiton, Andrew Zisserman,2021 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] In the aforementioned multimodal conventional methods, the conditions for determining whether fusion is possible become complex, resulting in the identification results being highly dependent on those conditions. Consequently, the aforementioned multimodal conventional methods suffer from a problem of reduced reliability in their identification results.

[0008] Therefore, the object of the present invention is to provide a person identification device and program therefor that have high reliability in identification results. [Means for solving the problem]

[0009] To solve the aforementioned problems, the person identification device according to the present invention is a person identification device that identifies a target person using the face region and body region of the target person included in an image, and comprises a face region detection unit, a face region identification unit, a body region detection unit, a body region identification unit, a correspondence processing unit, a graph generation unit, a grouping processing unit, and an identification result output unit.

[0010] In this configuration, the face region detection unit detects and tracks the face region from the video. The face region identification unit calculates an identification score for the face region of a pre-registered person from the face region detected by the face region detection unit. The body area detection unit detects and tracks the body area from the video. The body region identification unit calculates an identification score for the body region of a registered person from the body region detected by the body region detection unit. The correspondence processing unit calculates the degree of overlap between the face region and the body region for each frame of the video, and then associates the face region with the body region based on the calculated degree of overlap.

[0011] The graph generation unit generates a graph in which the associated face and body regions become nodes, and the edges connecting the nodes represent the degree of overlap between the face and body regions. The grouping processing unit groups the face region and body region by performing connected component analysis on the graph generated by the graph generation unit. The identification result output unit outputs the registered person with the highest identification score among the identification scores calculated from the face and body regions included in the same group as the identification result.

[0012] In other words, the person recognition device reflects the identification results of different modals, namely the face region and the body region, into a graph structure and fuses them, eliminating the need to determine whether or not fusion is possible under complex conditions, thereby improving reliability.

[0013] Furthermore, the present invention can also be realized by a program that causes a computer to function as the aforementioned person identification device. [Effects of the Invention]

[0014] According to the present invention, the reliability of the identification results can be increased. [Brief explanation of the drawing]

[0015] [Figure 1] This diagram illustrates a multimodal person identification technology, which is a premise of the present invention. [Figure 2] This is a block diagram showing the configuration of a person identification device according to an embodiment. [Figure 3] This diagram illustrates the processing content of the correspondence relationship processing unit in an embodiment. [Figure 4]This is a diagram for explaining the positional relationship between the face region and the body region in an embodiment. [Figure 5] This is a diagram for explaining the generation of the IOU matrix in an embodiment. [Figure 6] This is a diagram for explaining the association between the face region and the body region in an embodiment. [Figure 7] This is a diagram showing an example of the association information in an embodiment. [Figure 8] This is a diagram for explaining the generation of a graph in an embodiment. [Figure 9] This is a diagram for explaining an output example of the identification result in an embodiment. [Figure 10] This is a flowchart showing the operation of a person identification device according to an embodiment.

Embodiments for Carrying Out the Invention

[0016] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, each of the embodiments described below is for embodying the technical idea of the present invention, and the present invention is not limited to the following unless specifically described. Also, the same means may be denoted by the same reference numerals, and the description may be omitted.

[0017] [Overview of Multimodal Person Identification Technology] Referring to FIG. 1, the overview of multimodal person identification processing will be described. Person identification processing is a process of comparing the feature amounts of a person to be identified in a video with the feature amounts of a plurality of registered persons registered in advance, and outputting the registered person whose feature amounts are similar as an identification result.

[0018] Feature extraction can utilize the results of signal processing from multiple modals, such as features of a person's body and face regions, features of a person's speech, and information about a person extracted from text describing the video. A modal refers to the type of signal used as the basis for calculating features. For example, modals could include a face region representing a person's face, a body region representing a person's entire body, a person's voice, or text about a person (a person description).

[0019] Recently, multimodal methods that compare features across modals have been attracting attention. The person recognition device 1 fuses the signal processing results of multiple modals into a graph structure, effectively fuses the multimodal signals using graph signal processing, and outputs the final recognition result. As shown in Figure 1, the person recognition device 1 comprises a modal signal processing unit 10, a graph construction unit 20, a grouping processing unit 30, and a recognition result output unit 40.

[0020] The modal signal processing unit 10 processes signals with temporal extension related to person identification in the video across multiple modals. In the example shown in Figure 1, the modal signal processing unit 10 consists of modal 1 processing unit 101, modal 2 processing unit 102, ..., modal K processing unit 10 K It includes (where K is an integer of 2 or more). In other words, the modal signal processing unit 10 supports K types of modals such as face region, body region, voice, and person description text, and can be implemented using known methods such as face recognition processing, person recognition processing, speaker recognition, and natural language processing.

[0021] The graph construction unit 20 uses the signal processing results from the modal signal processing unit 10 to construct a graph in which the processing results of each modal are represented as nodes and the relationships between each modal as edges. The connections between nodes are made sequentially over time. The graph construction unit 20 comprises a correspondence processing unit 21 and a graph generation unit 22. The correspondence processing unit 21 associates the signal processing results of each modal, and the graph generation unit 22 generates a graph from the results of the association.

[0022] The grouping processing unit 30 extracts and groups the fused portions of each modal by performing connected component analysis on the graph. This makes it possible to obtain the result of fusing information of the same person across modals across space and time.

[0023] The identification result output unit 40 calculates the optimal identification result based on the processing results of each modal that has become part of the same group through the fusion process. This allows for the calculation of an optimal identification result with high accuracy. Since the calculation of the optimal identification result is performed using the feature quantities of each modal as input, more complex processing such as further deep learning processing is also possible. The simplest method for calculating the identification result is to select the maximum identification score for each modal.

[0024] Furthermore, by setting constraints for graph construction, such as prohibiting actions to ensure spatial and temporal consistency, and guaranteeing the maintenance of feature consistency within the same fusion group, the robustness and stability of the processing can be improved.

[0025] (Embodiment) [Configuration of the person identification device] Referring to Figure 2, the configuration of the person identification device 2 according to this embodiment will be described in detail. Person identification device 2 is configured by applying person identification device 1 shown in Figure 1 to two types of modals: the face region and the body region of a person.

[0026] The person identification device 2 identifies a person to be identified using the face region and body region of the person to be identified contained in the video. In this embodiment, the video to be identified is input to the person identification device 2.

[0027] As shown in Figure 2, the person identification device 2 comprises a face region detection unit 12, a face region identification unit 13, a body region detection unit 14, a body region identification unit 15, a correspondence processing unit 23, a graph generation unit 24, a grouping processing unit 31, and an identification result output unit 41.

[0028] The face region detection unit 12 detects and tracks face regions from video footage. Specifically, the face region detection unit 12 applies general face region detection and tracking processing to the video footage input to the person identification device 2. For example, deep learning can be used for the face region detection and tracking processing. The face region detection unit 12 also assigns a unique identifier to the face region of the detected person to be identified. Hereafter, the face region identifier may be referred to as "fid". The face region detection unit 12 outputs the face region of the detected person to be identified to the face region identification unit 13 and the correspondence relationship processing unit 23.

[0029] The face region identification unit 13 calculates an identification score for a pre-registered person from the face region detected by the face region detection unit 12. Specifically, the face region identification unit 13 calculates general feature quantities for the face regions of multiple registered people and registers (stores) them in advance. The face region identification unit 13 also calculates the same feature quantities for the face region of the person to be identified, which is input from the face region detection unit 12, as it does for the face region of the registered person. Next, the face region identification unit 13 calculates the similarity of the face regions between the registered person and the person to be identified as an identification score. The face region identification unit 13 outputs the calculated identification score to the identification result output unit 41.

[0030] The body region detection unit 14 detects and tracks body regions from video footage. Specifically, the body region detection unit 14 performs general body region detection and tracking processing on the video footage input to the person identification device 2. For example, deep learning can be used for the body region detection and tracking processing. The body region detection unit 14 also assigns a unique identifier to the body region of the detected person to be identified. Hereafter, the body region identifier may be referred to as "bid". The body region detection unit 14 outputs the body region of the detected person to be identified to the body region identification unit 15 and the correspondence relationship processing unit 23.

[0031] The body region identification unit 15 calculates an identification score for the body region of a registered person from the body region detected by the body region detection unit 14. Specifically, the body region identification unit 15 pre-registers (stores) general feature quantities for the body regions of multiple registered people. The body region identification unit 15 also calculates the same feature quantities for the body region of the person to be identified, which is input from the body region detection unit 14, as it does for the body region of the registered person. Next, the face region identification unit 13 calculates the similarity of the body regions between the registered person and the person to be identified as an identification score. The body region identification unit 15 outputs the calculated identification score to the identification result output unit 41.

[0032] The face region detection unit 12 and the face region identification unit 13 correspond to the modal 1 processing unit 101 in Figure 1. The body region detection unit 14 and the body region identification unit 15 correspond to the modal 2 processing unit 102 in Figure 1.

[0033] The correspondence processing unit 23 calculates the degree of overlap between the face region and the body region for each frame of the video, and associates the face region and the body region based on the calculated degree of overlap. In other words, the correspondence processing unit 23 associates the face region and the body region of the same person detected in each frame and outputs the association information (fid, bid) to the graph generation unit 24. In this association, the degree of overlap between the face region and the body region, IOU (Intersection over Union), is calculated, and if the IOU is greater than a preset threshold, it is determined that a correspondence exists between the two regions.

[0034] <Mapping between face region and body region> Refer to Figures 3 to 6 to explain in detail the correspondence between the face region and the body region. As shown in Figure 3, the multiple face regions detected by the face region detection unit 12 are designated as face-bboxes, and the group of identifiers for each face region is designated as fids. The multiple body regions detected by the body region detection unit 14 are designated as body-bboxes, and the group of identifiers for each body region is designated as bids. The face candidate regions that predict face regions are designated as pred-bboxes. Additionally, one face region is designated as face-bbox, one body region as body-bbox, and one face candidate region as pred-bbox.

[0035] The correspondence processing unit 23 determines the pred-bboxes in body-bboxes where face-bboxes are expected to exist (step S1 in Figure 3). Typically, face-bboxes are located at the top of body-bboxes. As shown in Figures 4(a) and 4(b), the pred-bbox is defined as a rectangular region shifted upward by the vertical length raito_h of body-bbox from the bottom edge of body-bbox and shifted to the right by the horizontal length raito_w of body-bbox from the left edge of body-bbox.

[0036] Furthermore, if the horizontal dimension raito_w of body-bbox is longer than the vertical dimension raito_h, it is assumed that the person is lying on their side. In this case, two rectangular regions are created by shifting horizontally by raito_w in the opposite direction from the left and vertically by raito_h, and these two regions are designated as pred-bbox1 and pred-bbox2.

[0037] The correspondence processing unit 23 calculates IOUs between pred-bboxes and face-bboxes and generates the calculation result as an IOU matrix (step S2 in Figure 3). The IOU matrix is ​​a matrix whose elements are the IOUs of both regions for all combinations of fid and bid. The correspondence processing unit 23 then manipulates the IOU matrix to find the combinations of (fid, bid) that can be matched (step S3 in Figure 3).

[0038] The following steps 1-4 show how to obtain correspondence information (fid, bid) from the IOU matrix. Step 1: From the combinations of face and body regions, exclude combinations that fall below the first threshold pre-set by IOU. Step 2: From the excluded combinations, further exclude any combinations where both the highest IOU and the second highest IOU for each face region within the same body region are above a predetermined second threshold. Step 3: From the excluded combinations, select the face region combination that has the highest IOU for the same body region. Step 4: From the selected combinations, further select the combination of body regions that has the maximum IOU for the same face region.

[0039] If the body region is determined to be oriented horizontally and there are two candidate face regions, the combination that maximizes the IOU between each candidate face region and the body region should be selected.

[0040] The remaining (fid,bid) combinations from steps 1-4 are output as mapping information, along with the IOU value (iou_val) for each combination, as {(fid,bid):iou_val}. Additionally, any fid,bid combinations that could not be mapped are output as {(fid,-1):-1,(-1,bid):-1}.

[0041] Refer to Figures 5 and 6 to explain specific examples of correspondence. As shown in Figure 5, assume that in a given frame, fids={fid1,fid2,fid3} and bids={bid1,bid2,bid3} were detected. Also, as shown in Figure 5, fid1 overlaps with bid1 and bid2, fid2 overlaps with bid1 to bid3, and fid3 overlaps with bid2 and bid3. Assume that the IOU matrix shown in Figure 5 was generated considering the degree of overlap between bid1 to bid3 and fid1 to bid3.

[0042] For example, in this IOU matrix, the IOU between bid1 and fid1 is 0.9, and the IOU between bid1 and fid2 is 0.1. This indicates that most of the face region of fid1 overlaps with the body region of bid1, and the remaining portion of the face region of fid1 overlaps with the body region of bid2.

[0043] As shown in Figures 6(a) to (e), the correspondence processing unit 23 transforms the IOU matrix according to steps 1 to 4. As a result, as shown in Figure 6(f), correspondence can be made from the combinations of fids and bids included in the final IOU matrix.

[0044] In step 1, we remove combinations from the IOU matrix in Figure 6(a) where the IOU is less than the first threshold (e.g., 0.3). In Figure 6(a), (fid2,bid1)=0.1 and (fid1,bid2)=0.1 are less than the first threshold = 0.3, so we remove these two combinations. Thus, as shown in Figure 6(b), (fid2,bid1)=0 and (fid1,bid2)=0.

[0045] In step 2, from the IOU matrix in Figure 6(b), we exclude combinations where both the largest IOU and the second highest IOU for the same bid are above the second threshold (e.g., 0.6). In this step 2, false positives increase when the facial regions of different people are close together, such as in a meeting, so these combinations are excluded from the mapping. In Figure 6(b), (fid2,bid2)=0.8 and (fid3,bid2)=0.6 are above the second threshold of 0.6, so these two combinations are excluded. Therefore, as shown in Figure 6(c), (fid2,bid2)=0 and (fid3,bid2)=0.

[0046] In step 3, we select the combination of fids that maximizes the IOU for the same bid from the IOU matrix in Figure 6(c). In Figure 6(c), (fid2,bid3)=0.4 and (fid3,bid3)=0.8, so we select only the latter combination that yields the maximum value. Therefore, as shown in Figure 6(d), (fid2,bid3)=0, while (fid3,bid3) remains at 0.8.

[0047] In step 4, we further select the bid combination that maximizes the IOU for the same fid from the IOU matrix in Figure 6(d). Note that since there are no combinations that match step 4, the IOU matrices in Figure 6(d) and Figure 6(e) are identical.

[0048] Then, as shown in Figure 6(f), the combination information {(fid,bid):iou_val} is obtained from the IOU matrix in Figure 6(e). As shown in Figure 6(f), it is set to {(fid1,bid1):0.9, (fid2,-1):-1, (-1,bid2):-1, (fid3,bid3):0.8}.

[0049] Returning to Figure 2, we will continue the explanation of the person identification device 2. The graph generation unit 24 generates a graph in which the associated face region and body region become nodes, and the edges connecting the nodes represent the degree of overlap between the face region and the body region. In other words, the graph generation unit 24 generates a graph that spans time from the {(fid,bid):iou_val} associated by the correspondence processing unit 23.

[0050] The graph generation unit 24 can generate a graph by using the associated fid and bid as nodes and connecting both nodes with edges weighted by iou_val. In this case, if there is no association (fid,-1):-1,(-1,bid):-1, the graph generation unit 24 simply adds a node for either fid or bid, without connecting the two nodes with an edge. The graph generation unit 24 performs this process in each frame to generate a graph for modal fusion.

[0051] <Graph generation> Refer to Figures 7 and 8 to explain the graph generation process in detail. As shown in Figure 7, at time t1 the person is facing sideways, at time t2 the person is facing backward, and at time t3 the person is facing forward. Therefore, the body region is detected at times t1 to t3. The face region is detected at times t1 and t3, but not at time t2. Now, let's assume bid=25 at times t1 to t3. Let fid=31 at time t1 and fid=32 at time t2.

[0052] In this case, the correspondence information is fid=31 and bid=25 at times t1 and t3, and only bid=25 at time t2. Furthermore, the IOUs for bid and fid are calculated to be 0.8 at time t1 and 0.9 at time t3. As a result, the correspondence information for times t1 to t3 is as follows:

[0053] Time t1:{(31,25):0.8} Time t2:{(-1,25):-1} Time t3:{(32,25):0.9}

[0054] Figure 8 shows a graph generated from the mapping information at times t1 to t3. This graph for modal fusion is sometimes referred to as MergG. In MergG, the mapped bid and fid are treated as nodes, and they are connected by edges with the iou_val of the nodes as weights. Specifically, at time t1, the nodes fid=31 and bid=25 are connected by an edge with the weight IOU=0.8. At time t3, the nodes fid=31 and bid=25 are connected by an edge with the weight IOU=0.9. In other words, this graph shows that there is no face region at time t2, while the body region continuously exists from time t1 to t3.

[0055] Returning to Figure 2, we will continue the explanation of the person identification device 2. The grouping processing unit 31 groups the face region and body region by performing connected component analysis on the graph generated by the graph generation unit 24. The aforementioned MergG reflects information spanning time in its graph structure, and by performing connected component analysis on the graph, individual fid and bid can be grouped.

[0056] <Grouping> The grouping will be explained in detail below. In the example shown in Figure 8, the grouping unit 31 groups the nodes {fid31, fid32, bid25} by performing connected component analysis on the fid and bid spanning times t1 to t3. The grouping unit 31 then considers the grouped set of nodes as a modally fused group and assigns a new mergid (e.g., 30) to it. The mergid represents an identifier for each group.

[0057] The grouping processing unit 31 groups the face region and body region for each cut of the video. In this embodiment, the grouping processing unit 31 detects cuts in the video using a general cut point detection method. The grouping of fids and bids via connected component analysis of the graph is fixed based on the video delimiter signal. The video delimiter signal is a signal that indicates a break (cut point) in the video. A mergid is assigned to the set of fids and bids that have been fixed at the cut point. As shown in the following equation, mergid can be treated as identification information that represents the fids and bids belonging to that group. mergid={fid,bid|fid∈fids,bid∈bids}

[0058] Furthermore, the graph generation unit 24 restricts the connection of nodes by using constraint conditions to determine whether or not node connections are possible, in order to suppress false detections in complex situations such as situations where people are densely packed together during graph generation. Specifically, the graph generation unit 24 uses the following conditions as constraint conditions for connecting nodes: that there is only one face region and one body region in the same frame, and that the similarity between the associated face region and body region is less than or equal to a pre-set third threshold.

[0059] This constraint consists of the following two conditions: (1) and (2). Condition (1): In the grouped set of mergids, there is only one fid or bid existing at the same time. Condition (2): The features of the connected modal should not differ significantly from the features of the same modal in the group.

[0060] Condition (1) means that the IDs of each grouped modal do not exist simultaneously at the same time. When connecting the associated fid and bid, it is determined whether the same ID exists at the same time and in the same modal among the previously grouped {fids} and {bids}, and connections are restricted. If the same ID exists, the connection of the (fid, bid) combination with the lower iou_val is disconnected, and the (fid, bid) combination with the higher iou_val is connected.

[0061] Condition (2) means that the similarity between the features of each modal calculated from the previously grouped fids and bids and the features of the modal ID to be newly connected is determined, and if the features are not similar, that ID is not connected. This condition adds the fids features f-feats and bids features b-feats as attribute information to the grouped mergid. The decision of whether or not to connect the node is made by comparing the similarity with the features of the newly connected identical modal ID to a predetermined threshold. The similarity determination with the features of the identical modal ID is performed by calculating the similarity with f-feats if the candidate ID is a fid, and with b-feats if it is a bid. By using such constraints, node connections with fewer errors can be achieved.

[0062] The identification result output unit 41 outputs the registered person with the highest identification score among the identification scores calculated from the face and body regions included in the same group as the identification result. In other words, the identification result output unit 41 integrates the identification scores of fid and bid belonging to mergid, which are grouped for each video segment, and calculates the optimal identification result. At this time, the identification result output unit 41 refers to the identification scores input from the face region identification unit 13 and the body region identification unit 15.

[0063] <Integration of identification results> The following explains in detail how the identification results are integrated. Let f_func(fid) be the face region recognition process for the fid representing the group, b_func(bid) be the body region recognition process for the bid representing the group, and m_func(megrid) be the integration process for mergid.

[0064] Generally, the integration process can be expressed by the following formula. m_func(megrid)=m_func({f_func(fid),b_func(bid)})

[0065] Here, {·} indicates that processing is applied to the identification results from each element of the modal belonging to mergid. Both f_func() and b_func() are expected to perform modal-specific processing such as feature extraction and classification score calculation. Alternatively, m_func() may be used to apply more complex processing, such as deep learning using the features of each modal. It is assumed that both f_func() and b_func() output classification scores for their respective modals. In this case, m_func() selects the optimal score from the classification scores of fid and bid belonging to mergid.

[0066] Below, we consider the case where mergid30 = {fid31, fid32, bid25}, as shown in Figure 8. For each fid and bid, the identification score for each registered person from person1 to person4 is as shown in Table 1 below.

[0067] [Table 1]

[0068] The function m_func(·) simply adopts the highest identification score for each registered person. The results are shown in Table 1 below, and the most probable person identification result can be obtained when mergid=30.

[0069] [Table 2]

[0070] In this way, by integrating the identification processing results from each modal merged with mergid, it is possible to achieve person identification processing that is more reliable than the identification results from individual modals and can cover a longer period of time.

[0071] The output of the identification result by the identification result output unit 41 will be explained in detail below. By utilizing the identification results of the person identification device 2, a GUI with excellent visibility and operability can be provided. As shown in Figure 9, mergid is added to the face and body regions within the frame image to improve visibility. In the example in Figure 9, mergid=15 is added to the face and body regions of the man on the left, and mergid=16 is added to the face and body regions of the woman on the right.

[0072] [Operation of the person identification device] Referring to Figure 10, the operation of the person identification device 2 will be explained. As shown in Figure 10, in step S10, the face region detection unit 12 detects and tracks the face region from the video. In step S11, the face region identification unit 13 calculates an identification score from the face region detected in step S10.

[0073] In step S12, the body region detection unit 14 detects and tracks the body region from the video. In step S13, the body region identification unit 15 calculates an identification score from the body region detected in step S13.

[0074] In step S14, the correspondence processing unit 23 associates the face region with the body region based on the degree of overlap between the face region and the body region. In step S15, the graph generation unit 24 generates a graph in which the associated face region and body region become nodes and edges represent the degree of overlap between the face region and the body region.

[0075] In step S16, the grouping processing unit 31 groups the nodes of the face region and the body region by performing connected component analysis on the graph generated in step S15. In step S17, the identification result output unit 41 determines the registered person with the highest identification score among the identification scores calculated from the face region and body region included in the same group as the identification result.

[0076] [Effects / Effects] As described above, the person identification device 2 reflects the identification results of different modals, the face region and the body region, in a graph structure and fuses them, eliminating the need to determine whether or not fusion is possible under complex conditions, thereby improving reliability.

[0077] The person identification device 2 can recognize people appearing in video footage, even in response to changes over time. For example, by adding the identification results as metadata to the video, it is possible to efficiently search for people.

[0078] (modified version) In the embodiments described above, the person identification device was described as an independent piece of hardware, but the present invention is not limited thereto. For example, the present invention can also be implemented using a program that causes hardware resources such as a CPU, memory, and hard disk of a computer to function as the person identification device described above. This program may be distributed via a communication line, or it may be written to a recording medium such as a CD-ROM or flash memory and distributed. [Explanation of Symbols]

[0079] 1,2 Person identification device 10 Modal signal processing unit 12 Face region detection unit 13 Face region identification unit 14 Body area detection unit 15 Body area identification unit 20 Graph Construction Section 21,23 Correspondence Processing Unit 22,24 Graph generation section 30,31 Grouping Processing Unit 40,41 Identification Result Output Section

Claims

1. A person identification device that identifies a person to be identified using the face region and body region of the person to be identified contained in the video, A face region detection unit that detects and tracks the face region from the aforementioned video, A face region identification unit calculates an identification score for the face region of a pre-registered person from the face region detected by the face region detection unit, A body region detection unit that detects and tracks the body region from the aforementioned video, A body region identification unit calculates an identification score for the body region of the registered person from the body region detected by the body region detection unit, A correspondence processing unit calculates the degree of overlap between the face region and the body region for each frame of the video, and associates the face region and the body region based on the calculated degree of overlap, A graph generation unit generates a graph in which the corresponding face region and body region become nodes, and the edges connecting the nodes represent the degree of overlap between the face region and the body region. A grouping processing unit performs connected component analysis on the graph generated by the graph generation unit to group the face region and the body region, An identification result output unit outputs the registered person with the highest identification score among the identification scores calculated from the face region and body region included in the same group as the identification result. A person identification device characterized by comprising the following features.

2. The aforementioned correspondence processing unit, From the combinations of the face region and the body region, combinations whose degree of overlap is less than a preset first threshold are excluded. From the excluded combinations, further combinations are excluded in which, for each face region relating to the same body region, both the maximum overlap and the second highest overlap are equal to or greater than a predetermined second threshold. From the excluded combinations, select the combination of face regions that has the greatest degree of overlap with the same body region. From the selected combinations, further select the combination of body regions that has the greatest degree of overlap with the same face region. The person identification device according to claim 1, characterized in that it associates the selected combination of face region and body region.

3. The grouping processing unit sets the following constraints for connecting the nodes: The person identification device according to claim 1, characterized in that only one of the face region and one of the body region exist in the same frame, and the similarity between the associated face region and body region is less than or equal to a predetermined third threshold.

4. The person identification device according to claim 1, characterized in that the grouping processing unit groups the face region and the body region for each cut of the video.

5. A program for operating a computer as a person identification device according to any one of claims 1 to 4.

Images