Tracking program, tracking method and information processing device

Abstract

An information processing device 100 identifies each head region of a person from a plurality of images photographed by a plurality of cameras, respectively. The information processing device 100 identifies a combination of head regions corresponding to the same person on the basis of the positions of the head regions identified from the plurality of images. The information processing device 100 identifies the position of the head of the person in a three-dimensional space on the basis of the positions of the combination of head regions of the same person in a two-dimensional space and parameters set for the plurality of cameras, respectively.

Description

[Technical field]

[0001] The present invention relates to a tracking program and the like. [Background technology]

[0002] There is a technology for tracking the position of a person in three dimensions using images captured by multiple cameras. FIG. 25 is a diagram for explaining an example of a person tracking result. In FIG. 25, image M1 is an image captured by camera c1. Image M2 is an image captured by camera c2. Image M3 is an image captured by camera c3. A person in three dimensions is tracked by repeating a process of calculating the position of the person in three dimensions (world coordinate system) based on the position of the same person in each image frame included in images M1 to M3 and the parameters of cameras c1 to c3.

[0003] For example, person 1-1 in video M1 and person 2-1 in video M2 are the same person, and the result of tracking this person in three dimensions is the trajectory tra1. Person 1-2 in video M1, person 2-2 in video M2, and person 3-2 in video M3 are the same person, and the result of tracking this person in three dimensions is the trajectory tra2.

[0004] Person 1-3 in video M1, person 2-3 in video M2, and person 3-3 in video M3 are the same person, and the result of tracking these persons in three dimensions is a trajectory tra3. Person 1-4 in video M1, person 2-4 in video M2, and person 3-4 in video M3 are the same person, and the result of tracking these persons in three dimensions is a trajectory tra4. Person 1-5 in video M1, person 2-5 in video M2, and person 3-5 in video M3 are the same person, and the result of tracking these persons in three dimensions is a trajectory tra5.

[0005] Here, we will explain Prior Art 1 and 2, which use images captured by multiple cameras to track the position of a person in three dimensions.

[0006] 26 is a diagram for explaining conventional technique 1. An apparatus that executes conventional technique 1 is denoted as conventional apparatus 10. Conventional apparatus 10 includes a single MOT (Multiple Object Tracking) 11, a three-dimensional trajectory calculation unit 12, and an association processing unit 13.

[0007] When the single MOT 11 receives the two-dimensional area information 1a, 1b, 1c (other two-dimensional area information), it generates two-dimensional trajectory information 2a, 2b, 2c (other two-dimensional trajectory information).

[0008] The two-dimensional area information 1a is the two-dimensional coordinates (2d bboxes) of the area of ​​the person extracted from the video (consecutive image frames) captured by camera c1. The two-dimensional area information 1b is the two-dimensional coordinates of the area of ​​the person extracted from the video captured by camera c2. The two-dimensional area information 1c is the two-dimensional coordinates of the area of ​​the person extracted from the video captured by camera c3.

[0009] The two-dimensional trajectory information 2a is trajectory information calculated by following the continuous two-dimensional area information 1a. The two-dimensional trajectory information 2b is trajectory information calculated by following the continuous two-dimensional area information 1b. The two-dimensional trajectory information 2c is trajectory information calculated by following the continuous two-dimensional area information 1c.

[0010] When the three-dimensional trajectory calculation unit 12 receives the two-dimensional trajectory information 2a, 2b, 2c (other two-dimensional trajectory information), it calculates three-dimensional trajectory information 3a, 3b, 3c (other three-dimensional trajectory information) based on the parameters of the cameras c1 to c3. The three-dimensional trajectory calculation unit 12 converts the two-dimensional person's trajectory into a three-dimensional person's trajectory, assuming that the Z-axis coordinate of the person's feet is 0 (Z=0).

[0011] For example, the three-dimensional trajectory calculation unit 12 calculates three-dimensional trajectory information 3a based on the two-dimensional trajectory information 2a. The three-dimensional trajectory calculation unit 12 calculates three-dimensional trajectory information 3b based on the two-dimensional trajectory information 2b. The three-dimensional trajectory calculation unit 12 calculates three-dimensional trajectory information 3c based on the two-dimensional trajectory information 2c.

[0012] The association processing unit 13 performs association based on the three-dimensional trajectory information 3a, 3b, 3c (other three-dimensional trajectory information) to generate the three-dimensional trajectory information 4. For example, the association processing unit 13 calculates the Euclidean distance or the like of each trajectory from the three-dimensional trajectory information 3a, 3b, 3c, and associates the three-dimensional trajectory information 3a, 3b, 3c based on the Euclidean distance to generate the three-dimensional trajectory information 4.

[0013] The conventional device 10 repeatedly executes the above process to track the position of the person in three dimensions.

[0014] 27 is a diagram for explaining the conventional technique 2. An apparatus that executes the conventional technique 2 is denoted as a conventional apparatus 20. The conventional apparatus 20 includes an association processing unit 21 and an MOT 22.

[0015] The association processing unit 21 generates three-dimensional posture information 6 based on the two-dimensional posture information 5a, 5b, 5c (other two-dimensional posture information). The two-dimensional posture information 5a is information on the posture of a person extracted from the video (successive image frames) captured by the camera c1, and includes information on joint positions, etc. The two-dimensional posture information 5b is information on the posture of a person extracted from the video captured by the camera c2, and includes information on joint positions, etc. The two-dimensional posture information 5c is information on the posture of a person extracted from the video captured by the camera c3, and includes information on joint positions, etc.

[0016] The association processing unit 21 associates the two-dimensional posture information 5a, 5b, 5c (other two-dimensional posture information) based on the distance, similarity, etc. between the epipolar line and the person identified from the two-dimensional posture information 5a, 5b, 5c, and generates three-dimensional posture information 6. The three-dimensional posture information 6 is information on the posture of the person in three dimensions, and includes information such as the joint positions of the person.

[0017] The MOT 22 generates three-dimensional trajectory information 7 based on the three-dimensional posture information 6. The three-dimensional trajectory information 7 is information on the trajectory of a three-dimensional person.

[0018] The conventional device 20 repeatedly executes the above process to track the position of the person in three dimensions. [Prior art documents] [Non-patent literature]

[0019] [Non-Patent Document 1] Yuhang He et al “Multi-Target Multi-Camera Tracking by Tracklet-to-Target Assignment” IEEE TRANSACTIONS ON IMAGE PROCESSING,VOL.29,2020 [Non-Patent Document 2] He Chen et al “Multi-person 3D Pose Estimation in Crowded Scenes Based on Multi-View Geometry” [Non-Patent Document 3] Long Chen et al “Cross-View Tracking for Multi-Human 3D Pose Estimation at over 100 FPS” arXiv:2003.03972v3 [cs.CV] 29 Jul 2021 [Non-Patent Document 4] Junting Dong et al “Fast and Robust Multi-Person 3D Pose Estimation and Tracking from Multiple Views”JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 [Non-Patent Document 5] Yifu Zhang et al “VoxelTrack: Multi-Person 3D Human Pose Estimation and Tracking in the Wild” arXiv:2108.02452v1 [cs.CV] 5 Aug 2021 Summary of the Invention [Problem to be solved by the invention]

[0020] However, the above-mentioned conventional technology has a problem in that it is not possible to track the three-dimensional position of a person.

[0021] FIG. 28 is a diagram for explaining the problems of the conventional technology. In the conventional technology, in order to calculate one piece of three-dimensional trajectory information from one piece of two-dimensional trajectory information, it is assumed that the Z-axis coordinate of the feet of person P1 is 0 (Z=0), and the three-dimensional coordinates of person P1 are calculated. For example, the three-dimensional coordinates of person P1 are calculated from the area A1 of the person included in image Im1 of camera c1. Also, the three-dimensional coordinates of person P1 are calculated from the area A2 of the person included in image Im2 of camera c2.

[0022] For this reason, when person P1 is positioned on a platform or the like, the Z-axis coordinate of person P1's feet is not 0 (Z≠0), so the three-dimensional coordinates of person P1 cannot be calculated with high accuracy, and tracking fails.

[0023] In addition, with conventional technology, when multiple people are crowded together, the people tend to overlap, so it is sometimes impossible to match areas of the same person in images captured by different cameras. Furthermore, when multiple people are crowded together, occlusion occurs, making it difficult to see the people's feet on the screen, making it difficult to calculate the three-dimensional position of the people.

[0024] According to one aspect, the present invention has an object to provide a tracking program, a tracking method, and an information processing device that are capable of tracking a three-dimensional position of a person with high accuracy. [Means for solving the problem]

[0025] In the first proposal, the computer is made to execute the following process. The computer identifies a person's head region from each of the multiple images captured by each of the multiple cameras. The computer identifies a set of head regions corresponding to the same person based on the positions of the head regions identified from the multiple images. The computer identifies the three-dimensional position of the person's head based on the two-dimensional positions of the set of head regions of the same person and parameters set for each of the multiple cameras. Effect of the Invention

[0026] It is possible to track a person's three-dimensional coordinates. [Brief description of the drawings]

[0027] [Figure 1] FIG. 1 is a diagram illustrating an example of a system according to the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a configuration of the information processing device according to the first embodiment. [Diagram 3] FIG. 3 is a diagram showing an example of the data structure of the video DB. [Figure 4] FIG. 4 is a diagram showing an example of a detection result of the head region. [Diagram 5] FIG. 5 is a diagram for explaining an example of the process of the single MOT. [Figure 6]FIG. 6 is a diagram for explaining the process of the first interpolation unit. [Figure 7] FIG. 7 is a diagram (1) for explaining the processing of the association processing unit. [Figure 8] FIG. 8 is a diagram (2) for explaining the processing of the association processing unit. [Figure 9] FIG. 9 is a diagram (3) for explaining the processing of the association processing unit. [Figure 10] FIG. 10 is a diagram (4) for explaining the processing of the association processing unit. [Figure 11] FIG. 11 is a diagram (1) for explaining the processing of the calculation processing unit. [Figure 12] FIG. 12 is a diagram (2) for explaining the processing of the calculation processing unit. [Figure 13] FIG. 13 is a diagram for explaining the process of the second interpolation unit. [Figure 14] FIG. 14 is a flowchart of the processing procedure of the information processing device according to the first embodiment. [Figure 15] FIG. 15 is a diagram illustrating an example of a system according to the second embodiment. [Figure 16] FIG. 16 is a diagram for explaining the process of the information processing device according to the second embodiment. [Figure 17] FIG. 17 is a diagram showing the result of association according to the conventional technique. [Figure 18] FIG. 18 is a diagram showing an example of an association error in the prior art. [Figure 19] FIG. 19 is a diagram (1) showing the result of association by the information processing device of the second embodiment. [Figure 20] FIG. 20 is a diagram (2) showing the result of association by the information processing device of the second embodiment. [Figure 21] FIG. 21 is a diagram illustrating an example of a configuration of an information processing device according to the second embodiment. [Figure 22] FIG. 22 is a diagram for explaining the process of the association processor 250 according to the second embodiment. [Figure 23]FIG. 23 is a flowchart of a processing procedure of the information processing device according to the second embodiment. [Figure 24] FIG. 24 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the information processing apparatus of the embodiment. [Diagram 25] FIG. 25 is a diagram for explaining an example of a person tracking result. [Figure 26] FIG. 26 is a diagram for explaining the prior art 1. In FIG. [Figure 27] FIG. 27 is a diagram for explaining the prior art 2. In FIG. [Figure 28] FIG. 28 is a diagram for explaining the problem with the conventional technology. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] Hereinafter, embodiments of a tracking program, a tracking method, and an information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to these embodiments. EXAMPLES

[0029] Fig. 1 is a diagram showing an example of a system according to the present embodiment 1. As shown in Fig. 1, this system includes a plurality of cameras c1, c2, and c3, a data acquisition device 60, and an information processing device 100. The cameras c1 to c3 and the data acquisition device 60 are connected to each other via a network 50. Although the cameras c1 to c3 are shown in Fig. 1, the system according to the present embodiment 1 may further include other cameras.

[0030] Cameras c1 to c3 are cameras that capture video images of the inside of stores such as convenience stores and supermarkets. Cameras c1 to c3 transmit video data to data acquisition device 60. In the following description, the video data will be referred to as "video data." In the following description, when there is no need to distinguish between cameras c1 to c3, they will be referred to simply as "cameras."

[0031] Video data includes multiple image frames in a time series. Each image frame is assigned a frame number in ascending chronological order. One image frame is a still image captured by a camera at a certain timing.

[0032] The data acquisition device 60 receives video data from the cameras c1 to c3 and registers the received video data in a video DB (Data Base) 65. The video DB 65 is set in the information processing device 100 by a user or the like. Note that, in the first embodiment, the information processing device 100 is described as being offline as an example, but the information processing device 100 may be connected to the network 50 and the video data may be transmitted directly from the cameras c1 to c3 to the information processing device 100.

[0033] The information processing device 100 is a device that generates three-dimensional trajectory information by tracking a person in three dimensions based on each image frame (video data) registered in the video DB 65.

[0034] For example, the information processing device 100 identifies the region of a person's head and the epipolar line from each image frame registered in the video DB 65. In the following description, the region of a person's head is referred to as a "head region."

[0035] The information processing device 100 identifies a set of head regions corresponding to the same person based on the head region, epipolar line, and distance identified from each image frame, and calculates the three-dimensional coordinates of the person's head based on the identified set of head regions. The information processing device 100 generates three-dimensional trajectory information related to the person's head region by repeatedly executing this process.

[0036] Since cameras are usually installed in high places, even if multiple people are crowded together, the head area is not easily affected by occlusion, and most cameras can capture the head areas of multiple people. Therefore, compared to the conventional technology where area information of the whole body of a person is used, the head area is less likely to be lost, and the position of the person (position of the head area) can be stably tracked. Furthermore, since the information processing device 100 extracts only the head area, it can reduce calculation costs and increase processing speed compared to the conventional technology where area information and posture of the whole body of a person are specified.

[0037] Furthermore, the information processing device 100 according to the present embodiment identifies a set of head regions corresponding to the same person based on the head region, epipolar line, and distance of the person identified from each image frame. This makes it possible to prevent head regions of different people from being identified as the same set, and to track the three-dimensional position of the person with high accuracy.

[0038] Next, an example of the configuration of the information processing device 100 shown in FIG. 1 will be described. FIG. 2 is a diagram showing an example of the configuration of the information processing device according to the first embodiment. As shown in FIG. 1, the information processing device 100 has a video DB 65, a head region identification unit 110, a single MOT 111, a first interpolation unit 112, an association processing unit 113, a calculation processing unit 114, and a second interpolation unit 115. The head region identification unit 110, the single MOT 111, the first interpolation unit 112, the association processing unit 113, the calculation processing unit 114, and the second interpolation unit 115 are realized by a control unit such as a CPU. The head region identification unit 110 is an example of a first identification unit. The association processing unit 113 is an example of a second identification unit. The calculation processing unit 114 is an example of a third identification unit.

[0039] The video DB 65 is a DB that stores video data captured by cameras c1, c2, c3, etc. FIG. 3 is a diagram showing an example of the data structure of the video DB. As shown in FIG. 3, the video DB 65 stores camera identification information and video data in association with each other. The camera identification information is information that uniquely identifies a camera. For example, cam1 indicates camera c1. cam2 indicates camera c2. cam3 indicates camera c3. The video data is video data captured by the corresponding camera. The video data includes image frames in chronological order, and is assigned frame numbers.

[0040] Returning to the explanation of Fig. 2, head region identification unit 110 analyzes each image frame captured by each camera and registered in video DB 65 to identify each person's head region, and outputs two-dimensional region information 8a, 8b, and 8c in which the two-dimensional coordinates of the person's head region are set.

[0041] For example, the head region identification unit 110 uses a detection model that has been machine-learned. The detection model is a machine-learning model that detects the head region of a person included in an image frame when a time-series image frame included in video data is input. A person ID for identifying the person is assigned to the person detected from the image frame. The detection model is realized by an open source machine-learning model or the like.

[0042] Fig. 4 is a diagram showing an example of a head region detection result. As shown in Fig. 4, head regions HA1a, HA1b, HA1c, HA1d, and HA1e of each person are detected from an image frame frame1 captured by a certain camera. The head region identification unit 110 sets the two-dimensional coordinates (2d bboxes) of the head regions HA1a to HA1e of each person, a person ID, and the like, and generates two-dimensional region information.

[0043] The head region identifying unit 110 generates two-dimensional region information 8a based on each image frame captured by camera c1 and outputs it to the single MOT 111. The head region identifying unit 110 generates two-dimensional region information 8b based on each image frame captured by camera c2 and outputs it to the single MOT 111. The head region identifying unit 110 generates two-dimensional region information 8c based on each image frame captured by camera c3 and outputs it to the single MOT 111. Although not shown in the figure, the head region identifying unit 110 may further generate two-dimensional region information based on each image frame captured by another camera.

[0044] When the single MOT 111 receives the two-dimensional area information 8a, 8b, and 8c, it generates two-dimensional trajectory information 9a, 9b, and 9c. The single MOT 111 outputs the two-dimensional trajectory information 9a, 9b, and 9c to the first interpolation unit 112.

[0045] FIG. 5 is a diagram for explaining an example of processing of the single MOT. The image frames frame k-1, frame k, and frame k+1 shown in FIG. 5 are consecutive image frames captured by the same camera, and correspond to two-dimensional area information identified from each image frame. The image frame frame k-1 includes head areas HA1a, HA1b, and HA1c. The image frame frame k includes head areas HA2a, HA2b, and HA2c. The image frame frame k includes head areas HA3a, HA3b, and HA3c. The single MOT 111 identifies the head area of ​​the same person based on the distance of the head area in each image frame.

[0046] If the head regions HA1a, HA2a, and HA3a belong to the same person, the single MOT 111 links the head regions HA1a, HA2a, and HA3a. If the head regions HA1b, HA2b, and HA3b belong to the same person, the single MOT 111 links the head regions HA1b, HA2b, and HA3b. If the head regions HA1c, HA2c, and HA3c belong to the same person, the single MOT 111 links the head regions HA1a, HA2a, and HA3a.

[0047] The single MOT 111 executes the process shown in FIG. 5 to generate two-dimensional trajectory information from each two-dimensional area information corresponding to each image frame captured by the same camera.

[0048] In addition, the single MOT111 can generate two-dimensional trajectory information from each two-dimensional region information by using the technology described in the non-patent document (Ramana Sundararaman et al “Tracking Pedestrian Heads in Dense Crowd” arXiv:2103.13516v1 [cs.CV] 24 Mar 2021).

[0049] Returning to the explanation of Fig. 2, the first interpolation unit 112 receives the two-dimensional trajectory information 9a-9c, and when there is a gap or the like in the head region in a certain image frame, the first interpolation unit 112 interpolates the head region using information on the preceding and succeeding head regions. The first interpolation unit 112 outputs the interpolated two-dimensional trajectory information 9a-9c to the association processing unit 113.

[0050] Fig. 6 is a diagram for explaining the processing of the first interpolation unit. In the example shown in Fig. 6, a head region is not detected in an image frame k before interpolation. Note that head regions HA1a, HA1b, and HA1c are detected in an image frame k-1 before interpolation. Head regions HA3a, HA3b, and HA3c are detected in an image frame k+1 before interpolation.

[0051] The first interpolation unit 112 interpolates the head region HA2a of the image frame frame k based on the two-dimensional coordinates of the head region HA1a of the image frame frame k-1 and the two-dimensional coordinates of the head region HA3a of the image frame frame k+1. The first interpolation unit 112 interpolates the head region HA2b of the image frame frame k based on the two-dimensional coordinates of the head region HA1b of the image frame frame k-1 and the two-dimensional coordinates of the head region HA3b of the image frame frame k+1. The first interpolation unit 112 interpolates the head region HA2c of the image frame frame k based on the two-dimensional coordinates of the head region HA1c of the image frame frame k-1 and the two-dimensional coordinates of the head region HA3c of the image frame frame k+1.

[0052] By performing the above process by the first interpolation unit 112, head areas HA2a, HA2b, and HA2c of the image frame frame k are set after the interpolation.

[0053] Returning to the explanation of Fig. 2, the association processing unit 113 associates head regions corresponding to the same person among the head regions in the image frames captured by different cameras based on the two-dimensional trajectory information 9a to 9c. Figs. 7 to 10 are diagrams for explaining the processing of the association processing unit.

[0054] First, FIG. 7 will be described. In FIG. 7, an image frame Im10-1 and an image frame Im10-2 are used. The image frame 10-1 is an image frame included in video data captured by a camera c1. The image frame 10-1 includes a head region HA10 of a person. The height of the head region HA10 is defined as "h 1 " and width "w 1 "

[0055] The image frame 10-2 is an image frame included in the video data captured by the camera c2. The image frame 10-2 includes a head area HA11 of a person. The height of the head area HA11 is defined as "h 2 " and width "w 2 "

[0056] The image frame 10-1 and the image frame 10-2 are assumed to be image frames captured at the same timing, for example, the frame number of the image frame 10-1 and the frame number of the image frame 10-2 are assumed to be the same.

[0057] The association processing unit 113 calculates the parameters of the cameras c1 and c2 and the center coordinate x 2 Based on the above, the epipolar line l(x 2 Identify the epipolar line l(x 2 ,0) on the center coordinate x of the head area HA11 2 This means that it includes.

[0058] The association processing unit 113 calculates the center coordinate x of the head area HA10 on the image frame Im10-1. 1 and the epipolar line l(x 2 ) and the distance d(l(x 2 ),x 1 The association processing unit 113 calculates the distance d(l(x 2 ),x 1 ) to ((w 1 +h 1 ) / 2) to perform scale adjustment and calculate the epipolar distance between the head region HA10 and the head region HA11.

[0059] The association processing unit 113 calculates the parameters of the cameras c1 and c2 and the center coordinate x 1 Based on the above, the epipolar line l(x 1 ) is identified. 1 ) The center coordinate x of the head area HA10 is 1 The association processing unit 113 calculates the distance d(l(x 1 ),x 2 ) to ((w 2 +h 2 ) / 2) to perform scale adjustment and calculate the epipolar distance between the head region HA10 and the head region HA11.

[0060] The association processing unit 113 executes the above process for each head region included in the image frames captured by different cameras, and calculates the epipolar distance of each head region.

[0061] We now move on to the description of Fig. 8. In Fig. 8, the epipolar distance for each head region is shown by the matrix MA. For example, head regions HA10-1 and HA11-1 are head regions of a person in an image frame captured by camera c1. Head regions HA10-2 and HA11-2 are head regions of a person in an image frame captured by camera c2. The epipolar distance for the same head region is "0.0".

[0062] The association processing unit 113 vertically scans the epipolar distances set in the matrix MA, identifies the smallest epipolar distance among the epipolar distances excluding the epipolar distance "0.0" corresponding to the same image, and identifies a pair of head regions corresponding to the same person based on the identification result. In the following description, the smallest epipolar distance among the epipolar distances excluding the epipolar distance "0.0" is referred to as the "minimum epipolar distance."

[0063] 8, the minimum epipolar distance is the epipolar distance "0.2" obtained from the pair of head region HA10-1 and head region HA10-2. Therefore, association processing unit 113 associates the pair of head region HA10-1 and head region HA10-2 as head regions of the same person.

[0064] 8, the minimum epipolar distance is the epipolar distance "0.1" obtained from the pair of head region HA11-1 and head region HA11-2. Therefore, association processing unit 113 associates the pair of head region HA11-1 and head region HA11-2 as head regions of the same person.

[0065] The association processing unit 113 repeatedly executes the above process for each head region included in each image frame of the two-dimensional trajectory information 9a to 9c, thereby identifying a set of head regions corresponding to the same person.

[0066] We now move on to the description of FIG. 9. The description will be given using image frames Im20-1 and Im20-2 with frame number "k-1". For example, image frame Im20-1 is an image frame captured by camera c1. From image frame Im20-1, person head regions HA1a, HA1b, and HA1c are identified.

[0067] Image frame Im20-2 is an image frame captured by camera c2. From image frame Im20-2, a person's head regions HA1x, HA1y, and HA1z are identified. In image frame Im20-2, epipolar line l1a corresponding to head region HA1a is identified. In image frame Im20-2, epipolar line l1b corresponding to head region HA1b is identified. In image frame Im20-2, epipolar line l1c corresponding to head region HA1c is identified.

[0068] The association processing unit 113 calculates the epipolar distance and associates the head regions of the same person. For example, the association processing unit 113 associates the head region HA1a with the head region HA1x as the head region of the same person. The association processing unit 113 associates the head region HA1b with the head region HA1y as the head region of the same person. The association processing unit 113 associates the head region HA1c with the head region HA1z as the head region of the same person.

[0069] Next, a description will be given using image frames Im21-1 and Im21-2 with frame number "k". Image frame Im21-1 is an image frame captured by camera c1. Person head areas HA2a, HA2b, and HA2c are identified from image frame Im21-1.

[0070] Image frame Im21-2 is an image frame captured by camera c2. From image frame Im21-2, a person's head regions HA2x, HA2y, and HA2z are identified. In image frame Im21-2, epipolar line l2a corresponding to head region HA2a is identified. In image frame Im21-2, epipolar line l2b corresponding to head region HA2b is identified. In image frame Im21-2, epipolar line l2c corresponding to head region HA2c is identified.

[0071] The association processing unit 113 calculates the epipolar distance and associates the head regions of the same person. For example, the association processing unit 113 associates the head region HA2a with the head region HA2x as the head region of the same person. The association processing unit 113 associates the head region HA2b with the head region HA2y as the head region of the same person. The association processing unit 113 associates the head region HA2c with the head region HA2z as the head region of the same person.

[0072] Next, a description will be given using image frames Im22-1 and Im22-2 with frame number "k+1". Image frame Im22-1 is an image frame captured by camera c1. Person head areas HA3a, HA3b, and HA3c are identified from image frame Im22-1.

[0073] Image frame Im22-2 is an image frame captured by camera c2. From image frame Im22-2, a person's head regions HA3x, HA3y, and HA3z are identified. In image frame Im22-2, epipolar line l3a corresponding to head region HA3a is identified. In image frame Im22-2, epipolar line l3b corresponding to head region HA3b is identified. In image frame Im22-2, epipolar line l3c corresponding to head region HA3c is identified.

[0074] We now move on to the explanation of Fig. 10. In the same manner as the process explained in Fig. 8, the association processing unit 113 creates matrices MA1, MA2, and MA3 for each frame in frames k-1, k, and k+1, and associates them as head regions of the same person based on the minimum epipolar distance. An example is shown below. Fig. 10 will be explained using matrices MA1, MA2, and MA3 based on successive image frames. Matrix MA1 is identified based on the image frame with frame number "k-1". Matrix MA2 is identified based on the image frame with frame number "k". Matrix MA3 is identified based on the image frame with frame number "k+1".

[0075] The matrix MA1 will now be described. Head areas HA10-1 and HA11-1 are the head areas of people in the image frame captured by camera c1. Head areas HA10-2 and HA11-2 are the head areas of people in the image frame captured by camera c2.

[0076] In the 0th row of matrix MA1, the minimum epipolar distance is the epipolar distance "0.2" obtained from the pair of head region HA10-1 and head region HA10-2, so association processing unit 113 associates the pair of head region HA10-1 and head region HA10-2 as head regions of the same person.

[0077] In the first row of matrix MA1, the minimum epipolar distance is the epipolar distance "0.1" obtained from the pair of head region HA11-1 and head region HA11-2. Therefore, association processing unit 113 associates the pair of head region HA11-1 and head region HA11-2 as head regions of the same person.

[0078] The matrix MA2 will now be described. Head areas HA12-1 and HA13-1 are the head areas of people in the image frame captured by camera c1. Head areas HA12-2 and HA13-2 are the head areas of people in the image frame captured by camera c2.

[0079] The association processing unit 113 determines that the minimum epipolar distance in the 0th row of the matrix MA2 is the epipolar distance "0.1" obtained from the pair of the head region HA12-1 and the head region HA12-2. The association processing unit 113 determines that the minimum epipolar distance in the 1st row of the matrix MA2 is the epipolar distance "0.2" obtained from the pair of the head region HA13-1 and the head region HA13-2. The association processing unit 113 associates the pair of the head region HA12-1 and the head region HA12-2, and the pair of the head region HA12-2 and the head region HA12-3 as the head regions of the same person.

[0080] Matrix MA3 will now be described. Head areas HA14-1 and HA15-1 are head areas of people in the image frame captured by camera c1. Head areas HA14-2 and HA15-2 are head areas of people in the image frame captured by camera c2.

[0081] In the 0th row of matrix MA3, the minimum epipolar distance is the epipolar distance "0.2" obtained from the pair of head region HA14-1 and head region HA14-2, so association processing unit 113 associates the pair of head region HA14-1 and head region HA14-2 as head regions of the same person.

[0082] In the first row of matrix MA3, the minimum epipolar distance is the epipolar distance "0.3" obtained from the pair of head region HA15-1 and head region HA15-2, so association processor 113 associates the pair of head region HA15-1 and head region HA15-2 as head regions of the same person.

[0083] 7 to 10, the association processing unit 113 associates head regions corresponding to the same person among the head regions in the image frames captured by different cameras based on the two-dimensional trajectory information 9a to 9c. The association processing unit 113 outputs the information on the associated head regions to the calculation processing unit 114.

[0084] The calculation processing unit 114 calculates the three-dimensional coordinates of the person's head region from the two-dimensional coordinates of the associated head region using the camera parameters and triangulation. Figures 11 and 12 are diagrams for explaining the processing of the calculation processing unit.

[0085] 11 will be described. Image frame Im19-1 is an image frame captured by camera c1. Person's head regions HA1a and HA1b are identified from image frame Im19-1. Person's head regions HA1x and HA1y are identified from image frame Im19-2. For example, it is assumed that head region HA1a and head region HA1x are associated with each other by the processing of association processing unit 113 described above. It is assumed that head region HA1b and head region HA1y are associated with each other.

[0086] The calculation processing unit 114 calculates the three-dimensional coordinates of the head of person P1 by triangulation based on the two-dimensional coordinates of head region HA1a and the two-dimensional coordinates of head region HA1x. The calculation processing unit 114 calculates the three-dimensional coordinates of the head of person P2 by triangulation based on the two-dimensional coordinates of head region HA1b and the two-dimensional coordinates of head region HA1y. The calculation processing unit repeatedly executes the above process for the image frame of each frame number.

[0087] We now move on to the description of FIG. 12. The description will be given using image frames Im20-1 and Im20-2 with frame number "k-1." For example, image frame Im20-1 is an image frame captured by camera c1. Person's head regions HA1a, HA1b, and HA1c are identified from image frame Im20-1. Person's head regions HA1x, HA1y, and HA1z are identified from image frame Im20-2. It is assumed that the head region HA1a and the head region HA1x are associated with each other by the processing of association processing unit 113. It is assumed that the head region HA1b and the head region HA1y are associated with each other. It is assumed that the head region HA1c and the head region HA1z are associated with each other.

[0088] The calculation processing unit 114 calculates the three-dimensional coordinates of person P1 by triangulation based on the two-dimensional coordinates of head region HA1a and the two-dimensional coordinates of head region HA1x. The calculation processing unit 114 calculates the three-dimensional coordinates of person P2 by triangulation based on the two-dimensional coordinates of head region HA1b and the two-dimensional coordinates of head region HA1y. The calculation processing unit 114 calculates the three-dimensional coordinates of person P3 by triangulation based on the two-dimensional coordinates of head region HA1c and the two-dimensional coordinates of head region HA1z.

[0089] A description will be given using image frames Im21-1 and Im21-2 with frame number "k." For example, image frame Im21-1 is an image frame captured by camera c1. Person head regions HA2a, HA2b, and HA2c are identified from image frame Im21-1. Person head regions HA2x, HA2y, and HA2z are identified from image frame Im21-2. It is assumed that the head region HA2a and the head region HA2x are associated with each other through processing by association processing unit 113. It is assumed that the head region HA2b and the head region HA2y are associated with each other. It is assumed that the head region HA2c and the head region HA2z are associated with each other.

[0090] The calculation processing unit 114 calculates the three-dimensional coordinates of person P1 by triangulation based on the two-dimensional coordinates of head region HA2a and the two-dimensional coordinates of head region HA2x. The calculation processing unit 114 calculates the three-dimensional coordinates of person P2 by triangulation based on the two-dimensional coordinates of head region HA2b and the two-dimensional coordinates of head region HA2y. The calculation processing unit 114 calculates the three-dimensional coordinates of person P3 by triangulation based on the two-dimensional coordinates of head region HA2c and the two-dimensional coordinates of head region HA2z.

[0091] An explanation will be given using image frames Im22-1 and Im22-2 with frame number "k+1". For example, image frame Im22-1 is an image frame captured by camera c1. Person head regions HA3a, HA3b, and HA3c are identified from image frame Im22-1. Person head regions HA3x, HA3y, and HA3z are identified from image frame Im22-2. It is assumed that the head region HA3a and the head region HA3x are associated with each other by the processing of association processing unit 113. It is assumed that the head region HA3b and the head region HA3y are associated with each other. It is assumed that the head region HA3c and the head region HA3z are associated with each other.

[0092] The calculation processing unit 114 calculates the three-dimensional coordinates of person P1 by triangulation based on the two-dimensional coordinates of head region HA3a and the two-dimensional coordinates of head region HA3x. The calculation processing unit 114 calculates the three-dimensional coordinates of person P2 by triangulation based on the two-dimensional coordinates of head region HA3b and the two-dimensional coordinates of head region HA3y. The calculation processing unit 114 calculates the three-dimensional coordinates of person P3 by triangulation based on the two-dimensional coordinates of head region HA3c and the two-dimensional coordinates of head region HA3z.

[0093] By performing the above process, the calculation processing unit 114 calculates the trajectories (three-dimensional trajectory information 15) of the three-dimensional coordinates of the head regions of the persons P1, P2, and P3 from the image frame numbers of the respective frame numbers. The calculation processing unit 114 outputs the three-dimensional trajectory information 15 to the second interpolation unit 115.

[0094] Returning to the explanation of Fig. 2, the second interpolation unit 115 receives the three-dimensional trajectory information 15, and when there is a gap or the like in the three-dimensional coordinates of the head region in a certain image frame, the second interpolation unit 115 interpolates the head region using information on the three-dimensional coordinates of the preceding and following head regions. The second interpolation unit 115 outputs the interpolated three-dimensional trajectory information 15.

[0095] Fig. 13 is a diagram for explaining the processing of the second interpolation unit. In the example shown in Fig. 13, in the three-dimensional trajectory information before the interpolation, the three-dimensional coordinates of the person P1 are specified in frame number k-1, the three-dimensional coordinates of the person P1 are not specified in frame number k, and the three-dimensional coordinates of the person P1 are specified in frame number k+1.

[0096] The second interpolation unit 115 performs interpolation by calculating the coordinates between the three-dimensional coordinates of person P1 in frame number k-1 and the three-dimensional coordinates of person P1 in frame number k+1 as the three-dimensional coordinates of person P1 in frame number k.

[0097] Next, an example of a processing procedure of the information processing device 100 according to the first embodiment will be described. Fig. 14 is a flowchart showing the processing procedure of the information processing device according to the first embodiment. The head region identifying unit 110 of the information processing device 100 acquires each image frame captured by each camera from the video DB 65 (step S101). The head region identifying unit 110 identifies a person's head region from the image frame and generates two-dimensional region information (step S102).

[0098] The single MOT 111 of the information processing device 100 generates two-dimensional trajectory information based on the two-dimensional region information (step S103). When a head region to be interpolated exists, the first interpolation unit 112 of the information processing device 100 executes an interpolation process on the two-dimensional trajectory information (step S104).

[0099] The association processing unit 113 of the information processing device 100 calculates the epipolar distance based on the two-dimensional trajectory information and associates head regions corresponding to the same person (step S105). The calculation processing unit 114 of the information processing device 100 generates three-dimensional trajectory information by triangulation based on the two-dimensional coordinates of the pair of head regions corresponding to the same person (step S106).

[0100] If a head region to be interpolated exists, the second interpolation unit 115 of the information processing device 100 executes an interpolation process on the three-dimensional trajectory information (step S107). The information processing device 100 outputs the three-dimensional trajectory information (step S108).

[0101] Next, the effect of the information processing device 100 according to the first embodiment will be described. The information processing device 100 identifies a set of head regions corresponding to the same person based on the head region, epipolar line, and distance identified from each image frame, and calculates the three-dimensional coordinates of the person's head based on the identified set of head regions. The information processing device 100 generates three-dimensional trajectory information related to the person's head region by repeatedly executing this process.

[0102] Since the cameras are installed at high places, even if multiple people are crowded together, the head region is not easily affected by occlusion, and most cameras can capture the head regions of multiple people. Since the information processing device 100 identifies the head region of a person, it is less likely to lose the head region and can stably track the position of the person (position of the head region) compared to the case where area information of the person's entire body is used as in the conventional technology. Furthermore, since the information processing device 100 extracts only the head region, it is possible to reduce calculation costs and increase processing speed compared to the case where area information and posture of the person's entire body are identified as in the conventional technology.

[0103] The information processing device 100 identifies a set of head regions corresponding to the same person based on the head region, epipolar line, and distance of each person identified from each image frame. This makes it possible to prevent head regions of different people from being identified as the same set, and to track the three-dimensional position of a person with high accuracy.

[0104] When calculating the epipolar distance, the information processing device 100 adjusts the scale of the epipolar distance based on the size of the head region included in each image frame. This makes it possible to appropriately associate head regions corresponding to the same person even if the distance between the person and each camera is different. EXAMPLES

[0105] Fig. 15 is a diagram showing an example of a system according to the second embodiment. As shown in Fig. 15, this system includes a plurality of cameras c1, c2, and c3, and an information processing device 200. The cameras c1 to c3 and the information processing device 200 are connected to each other via a network 50. Although the cameras c1 to c3 are shown in Fig. 15, the system according to the second embodiment may further include other cameras.

[0106] The cameras c1 to c3 are cameras that capture images of the inside of stores such as convenience stores and supermarkets. The cameras c1 to c3 transmit image data to the information processing device 200. The information processing device 200 receives the image data from the cameras c1 to c3 online, and outputs three-dimensional trajectory information. The information processing device 200 can also register the received image data in the image DB 65.

[0107] The information processing device 200 sequentially acquires image frames from the cameras c1 to c3, and calculates three-dimensional trajectory information for each preset window (sliding window). The information processing device 200 associates the three-dimensional trajectory information of each window to generate three-dimensional trajectory information of a person.

[0108] 16 is a diagram for explaining the process of the information processing device according to the embodiment 2. The information processing device 200 sets a window w1, a window w2, and a window w3. Here, the explanation will be given using the windows w1 to w3, but the windows w4 and w5 following the window w3 are also included.

[0109] Adjacent windows have some overlapping image frames, and the section between windows w1 to w3 is defined as n frames, where n=60, for example.

[0110] The information processing device 200 divides an image frame into a plurality of windows of short sections. The information processing device 200 executes processes corresponding to the single MOT 111, the first interpolation unit 112, the association processing unit 113, the calculation processing unit 114, and the second interpolation unit 115 described in the first embodiment on the image frame of the short section, and generates three-dimensional trajectory information for each short section. The information processing device 200 generates three-dimensional trajectory information (w1) by integrating the three-dimensional trajectory information for each short section of the window w1.

[0111] The information processing device 200 also performs the same process on the windows w2 and w3 as on the window w1, thereby generating three-dimensional trajectory information (w2) and three-dimensional trajectory information (w3).

[0112] The information processing device 200 inputs the three-dimensional trajectory information (w1) and the three-dimensional trajectory information (w2) to the association processing unit 250-1. The association processing unit 250-1 generates three-dimensional trajectory information (w1&w2) by associating the three-dimensional trajectory information (w1) and the three-dimensional trajectory information (w2), and outputs the three-dimensional trajectory information (w1&w2) to the association processing unit 250-2.

[0113] The information processing device 200 inputs the three-dimensional trajectory information (w3) to the association processing unit 250-2. The association processing unit 250-2 generates three-dimensional trajectory information (w1&w2&w3) by associating the three-dimensional trajectory information (w1&w2) with the three-dimensional trajectory information (w3), and outputs the three-dimensional trajectory information (w1&w2) to the subsequent association processing unit.

[0114] Each association processing unit of the information processing device 200 repeatedly executes the above process, thereby generating information in which the three-dimensional trajectories of each window are associated with each other.

[0115] Here, the conventional device 20 of Prior Art 2 is compared with the information processing device 200. The method of the conventional device 20 is a method (Single-frame Multi-view Data Association method) in which areas of the same person included in one image frame captured by each camera are associated with each other.

[0116] Fig. 17 is a diagram showing the result of association by the conventional technology. For convenience of explanation, Fig. 17 uses the head region of a person for explanation, but in actual conventional technology, information on the region of the entire body, posture, etc. is used. In Fig. 17, image frame Im30-1 is an image captured by camera c1. Person regions A1-1, A1-2, A1-3, and A1-4 are detected from image frame Im30-1.

[0117] Image frame Im30-2 is an image captured by camera c2. Person areas A2-1 and A2-2 are detected from image frame Im30-2. Image frame Im30-3 is an image captured by camera c3. Person area A3-1 is detected from image frame Im30-3.

[0118] The frame numbers of image frames Im30-1, Im30-2, and Im30-3 are assumed to be "k." In conventional device 20, areas A1-1, A2-1, and A3-1 are associated with each other by the Single-frame Multi-view Data Association method. In conventional device 20, when the above association is repeatedly performed for each image frame of each frame number, an error in the association may occur.

[0119] Fig. 18 is a diagram showing an example of an association error in the prior art. In Fig. 18, image frames Im40-1, Im40-2, Im40-3, Im40-4, and Im40-5 have the same frame number.

[0120] Image frame Im40-1 is an image captured by camera c1. Person areas A1-0, A1-1, A1-2, A1-3, A1-4, and A1-7 are detected from image frame Im40-1.

[0121] Image frame Im40-2 is an image captured by camera c2. Person areas A2-0, A2-1, and A2-2 are detected from image frame Im40-2.

[0122] Image frame Im40-3 is an image captured by camera c3. Person areas A3-0, A3-1, A3-2, A3-3, A3-4, A3-5, and A3-7 are detected from image frame Im40-3.

[0123] Image frame Im40-4 is an image captured by camera c4. Person areas A4-0, A4-1, A4-2, A4-3, and A4-6 are detected from image frame Im40-4.

[0124] Image frame Im40-5 is an image captured by camera c5. Person areas A5-0, A5-1, A5-2, A5-3, A5-4, and A5-5 are detected from image frame Im40-5.

[0125] In the prior art, when areas of the same person in each image frame are associated with each other, areas A1-0, A2-0, A3-0, A4-0, and A5-0 are associated with each other. Areas A1-1, A2-1, A3-1, A4-1, and A5-1 are associated with each other. Areas A1-2, A2-2, A3-2, A4-2, and A5-2 are associated with each other. Areas A1-3, A3-3, A4-3, and A5-3 are associated with each other. Areas A1-4, A3-4, and A5-4 are associated with each other. Areas A3-5 and A5-5 are associated with each other. Areas A4-6 and A5-6 are associated with each other.

[0126] Here, an error has occurred in the association between areas A1-4, A3-4, and A5-4. The correct association is A1-4, A3-5 (A3-4 is incorrect), and A5-5 (A5-4 is incorrect).

[0127] 19 and 20 are diagrams showing the results of association by the information processing device of the second embodiment. Fig. 19 will be described. The frame numbers of image frames Im35-1, Im35-2, and Im35-3 are assumed to be "k". The frame numbers of image frames Im35-4, Im35-5, and Im35-6 are assumed to be "k+△".

[0128] Image frame Im35-1 is an image captured by camera c1. Person head regions A1-1, A1-2, A1-3, and A1-4 are detected from image frame Im35-1.

[0129] Image frame Im35-2 is an image captured by camera c2. Person head regions A2-1 and A2-2 are detected from image frame Im35-2.

[0130] Image frame Im35-3 is an image captured by camera c3. A person's head area A3-1 is detected from image frame Im35-3.

[0131] Image frame Im35-4 is an image captured by camera c1. Person head regions A4-1, A4-2, and A4-3 are detected from image frame Im35-4.

[0132] Image frame Im35-5 is an image captured by camera c2. Person head regions A5-1, A5-2, and A5-3 are detected from image frame Im35-2.

[0133] Image frame Im35-6 is an image captured by camera c3. Person head regions A6-1 and A6-2 are detected from image frame Im35-6.

[0134] When the information processing device 200 executes the process described in FIG. 16 to associate head regions between different frame numbers, head regions A1-1, A2-1, A3-1, A4-1, A5-1, and A6-1 are associated with each other. Head regions A1-2, A2-2, A4-2, A5-2, and A6-2 are associated with each other. Head regions A1-3, A4-3, and A5-3 are associated with each other. The information processing device 200 sets a window and repeats the above association, thereby preventing the occurrence of an association error as described in FIG. 18.

[0135] Now, let us move on to the description of Fig. 20. In Fig. 20, image frames Im45-1, Im45-2, Im45-3, Im45-4, and Im45-5 have the same frame number.

[0136] Image frame Im45-1 is an image captured by camera c1. Person head regions A1-1, A1-2, A1-3, A1-4, A1-5, A1-6, and A1-7 are detected from image frame Im45-1.

[0137] Image frame Im45-2 is an image captured by camera c2. Person head regions A2-1, A2-2, and A2-4 are detected from image frame Im45-2.

[0138] Image frame Im45-3 is an image captured by camera c3. Person head regions A3-1, A3-2, A3-3, A3-4, A3-5, A3-6, and A3-7 are detected from image frame Im45-3.

[0139] Image frame Im45-4 is an image captured by camera c4. Person head regions A4-1, A4-2, A4-3, A4-4, and A4-5 are detected from image frame Im45-4.

[0140] Image frame Im45-5 is an image captured by camera c5. Person head regions A5-1, A5-2, A5-3, A5-4, A5-5, A5-6, and A5-7 are detected from image frame Im45-5.

[0141] When the information processing device 200 associates the areas of the same person for each person in each image frame, areas A1-1, A2-1, A3-1, A4-1, and A5-1 are associated with each other. Areas A1-2, A2-2, A3-2, A4-2, and A5-2 are associated with each other. Areas A1-3, A3-3, A4-3, and A5-3 are associated with each other. Areas A1-6, A3-6, and A5-6 are associated with each other. Areas A1-7, A3-7, and A5-7 are associated with each other. The associations shown in FIG. 20 are correct associations.

[0142] As shown in the results of the association for the conventional device 20 described in FIGS. 17 and 18 and the results of the association for the information processing device 200 described in FIGS. 19 and 20, according to the information processing device 200, it is possible to improve tracking accuracy by setting windows with overlapping sections and performing association.

[0143] Next, an example of the configuration of the information processing device 200 shown in FIG. 15 will be described. FIG. 21 is a diagram showing an example of the configuration of the information processing device according to the second embodiment. As shown in FIG. 21, the information processing device 200 has a window generating unit 65A, a head region identifying unit 110, a single MOT 111, a first interpolation unit 112, an association processing unit 113, a calculation processing unit 114, and a second interpolation unit 115. The information processing device 200 has a communication unit 210, a window control unit 220, and an association processing unit 250. The head region identifying unit 110, the single MOT 111, the first interpolation unit 112, the association processing unit 113, the calculation processing unit 114, and the second interpolation unit 115 are realized by a control unit such as a CPU. In addition, the communication unit 210, the window control unit 220, and the association processing unit 250 are also realized by a control unit such as a CPU.

[0144] In Fig. 21, the explanations regarding the head region identification unit 110, single MOT 111, first interpolation unit 112, association processing unit 113, calculation processing unit 114, and second interpolation unit 115 are the same as the explanations of each processing unit described in Fig. 2. The window generation unit 65A generates a frame for each window from the frames received from the camera.

[0145] The communication unit 210 receives video data from the cameras c1 to c3 (other cameras), and outputs the received video data to the window generating unit 65A.

[0146] 16, in cooperation with the head region identification unit 110, the single MOT 111, the first interpolation unit 112, the association processing unit 113, the calculation processing unit 114, and the second interpolation unit 115. The window control unit 220 generates three-dimensional trajectory information (w1, w2, . . .) of each window, and outputs the three-dimensional trajectory information (w1, w2, . . .) of each window to the association processing unit 250.

[0147] The association processing unit 250 executes processing corresponding to the association processing units 250-1, 250-2, etc. shown in Fig. 16. For example, the association processing unit 250 generates three-dimensional trajectory information (w1&w2) by associating three-dimensional trajectory information (w1) with three-dimensional trajectory information (w2). The association processing unit 250 generates three-dimensional trajectory information (w1&w2&w3) by associating three-dimensional trajectory information (w1&w2) with three-dimensional trajectory information (w3).

[0148] Fig. 22 is a diagram for explaining the processing of the association processing unit 250 according to the second embodiment. As shown in Fig. 22, a window w includes a three-dimensional trajectory w1-1 of the head region of person A and a three-dimensional trajectory w1-2 of the head region of person B. A window w+1 includes a three-dimensional trajectory w2-1 of the head region of person C, a three-dimensional trajectory w2-2 of the head region of person D, and a three-dimensional trajectory w2-3 of the head region of person E.

[0149] The association processing unit 250 calculates the Euclidean distance between the three-dimensional trajectory w1-1 and the three-dimensional trajectories w2-1, w2-2, and w2-3, identifies a set of three-dimensional trajectories whose Euclidean distance is less than a threshold, and associates them. For example, the three-dimensional trajectory w1-1 and the three-dimensional trajectory w2-1 are associated with each other and integrated into one three-dimensional trajectory.

[0150] The association processing unit 250 calculates the Euclidean distance between the three-dimensional trajectory w1-2 and the three-dimensional trajectories w2-1, w2-2, and w2-3, identifies a set of three-dimensional trajectories whose Euclidean distance is less than a threshold, and associates them. For example, the three-dimensional trajectory w1-2 and the three-dimensional trajectory w2-2 are associated with each other and integrated into one three-dimensional trajectory.

[0151] For example, the association processing unit 250 calculates the Euclidean distance based on formula (1). In addition, the association processing unit 250 may associate each three-dimensional trajectory using a cost matrix shown in formula (2) and a Boolean matrix shown in formula (3).

[0152]

number

number

number

[0153] Next, a description will be given of an example of a processing procedure of the information processing device 200 according to the embodiment 2. Fig. 23 is a flowchart showing the processing procedure of the information processing device according to the embodiment 2. The communication unit 210 of the information processing device 200 starts receiving video data from the camera (step S201).

[0154] The window control unit 220 of the information processing device 200 sets a window of a predetermined interval, and cooperates with the head region identification unit 110, the single MOT 111, the first interpolation unit 112, the association processing unit 113, the calculation processing unit 114, and the second interpolation unit 115 to sequentially generate 3D trajectory information for each window (step S202).

[0155] The association processing unit 250 of the information processing device 200 associates the three-dimensional information based on the Euclidean distance of the three-dimensional trajectory information of each window (step S203). The association processing unit 250 outputs the three-dimensional trajectory information (step S204).

[0156] Next, the effect of the information processing device 200 according to the second embodiment will be described. The information processing device 200 sequentially acquires image frames registered in the video DB 65, calculates three-dimensional trajectory information for each preset window, and generates three-dimensional trajectory information of a person by associating the three-dimensional trajectory information of each window. This prevents errors from occurring in associating the head regions of each image frame, and enables accurate generation of three-dimensional trajectory information for each person.

[0157] Next, an example of a hardware configuration of a computer that realizes the same functions as the information processing device 100 (200) shown in the above embodiment will be described. Fig. 24 is a diagram showing an example of a hardware configuration of a computer that realizes the same functions as the information processing device of the embodiment.

[0158] 24, the computer 300 has a CPU 301 that executes various arithmetic processes, an input device 302 that accepts data input from a user, and a display 303. The computer 300 also has a communication device 304 that transmits and receives data to and from cameras c1-c1 (other cameras) and external devices via a wired or wireless network, and an interface device 305. The computer 300 also has a RAM 306 that temporarily stores various information, and a hard disk device 307. The devices 201-207 are connected to a bus 208.

[0159] The hard disk device 307 has a head region identification program 307a, a trajectory information calculation program 307b, a window processing program 307c, and an association processing program 307d. The CPU 301 reads out each of the programs 307a to 307d and develops them in the RAM 306.

[0160] The head region identification program 307a functions as a head region identification process 306a. The trajectory information calculation program 307b functions as a trajectory information calculation process 306b. The window processing program 307c functions as a window processing process 306c. The association program 307d functions as an association processing process 306d.

[0161] The processing of the head region identification process 306a corresponds to the processing of the head region identification unit 110. The processing of the trajectory information calculation process 306b corresponds to the processing of the single MOT 111, the first interpolation unit 112, the association processing unit 113, the calculation processing unit 114, and the second interpolation unit 115. The processing of the window processing process 306c corresponds to the processing of the window control unit 220. The processing of the association processing process 306d corresponds to the processing of the association processing unit 250.

[0162] It is not necessary that each of the programs 307a to 307d is stored in the hard disk device 307 from the beginning. For example, each program may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 300. Then, the computer 300 may read and execute each of the programs 307a to 307d. [Explanation of symbols]

[0163] 65 Video DB 100,200 Information processing equipment 110 Head region identification part 111 Single MOT 112 First Interpolation Section 113 Association processing unit 114 Calculation processing unit 115 Second Interpolation Section 220 Window control section 250 Association processing unit

Claims

1. Identifying a person's head region from each of a plurality of images captured by each of a plurality of cameras; calculating a value based on an epipolar line for each pair of head regions based on the positions of the head regions identified from the plurality of images, and identifying the pair of head regions for which the value is a minimum value other than zero as a pair of head regions corresponding to the same person; Identifying the position of the person's head in three dimensions based on the two-dimensional positions of the set of head regions of the same person and the parameters set for each of the multiple cameras. A tracking program that causes a computer to execute a process.

2. The tracking program according to claim 1, characterized in that the process of identifying a pair of areas corresponding to the heads of the same person includes a process of identifying whether the head area included in the first image and the head area included in the second image are head areas corresponding to the same person based on the distance between an epipolar line included in the first image that corresponds to a head area included in the second image and the head area identified from the first image.

3. The tracking program according to claim 2, further comprising causing the computer to execute a process of correcting the distance based on the size of the head region included in the first image and the size of the head region included in the second image.

4. The tracking program according to claim 1, further comprising a process for estimating a three-dimensional position of a person's head at a second timing between the first timing and the third timing, based on a three-dimensional position of the person's head identified based on a plurality of images taken by each of a plurality of cameras at a first timing and a third timing.

5. The tracking program according to claim 1, characterized in that the process of identifying the position of the person's head in the three-dimensional space further causes the computer to execute a process of identifying trajectory information of the position of the person's head in the three-dimensional space for each window section including consecutive image frames, and associating the trajectory information of each window section.

6. Identifying a person's head region from each of a plurality of images captured by each of a plurality of cameras; calculating a value based on an epipolar line for each pair of head regions based on the positions of the head regions identified from the plurality of images, and identifying the pair of head regions for which the value is a minimum value other than zero as a pair of head regions corresponding to the same person; Identifying the position of the person's head in three dimensions based on the two-dimensional positions of the set of head regions of the same person and the parameters set for each of the multiple cameras. A tracking method, the processing of which is performed by a computer.

7. a first identification unit that identifies a head region of a person from a plurality of images captured by each of a plurality of cameras; a second identification unit that calculates a value based on an epipolar line for each pair of head regions based on the positions of the head regions identified from the plurality of images, and identifies a pair of head regions for which the value is a minimum value other than zero as a pair of head regions corresponding to the same person; a third identification unit that identifies a three-dimensional position of the person's head based on two-dimensional positions of a set of head regions of the same person and parameters set for each of the multiple cameras; 13. An information processing device comprising:

Images