Tracking device

Abstract

This device provides a highly accurate tracking system that avoids misidentification when tracking people using images from multiple cameras. [Solution] The tracking device 10 of the present invention tracks images of the same person using the feature quantities of person images acquired by multiple imaging devices 20, and determines that the images of the same person are obtained by multiple imaging devices 20 by performing a matching process using the similarity of the feature quantities of the person images in combinations of tracking results from two imaging devices. In the matching process, the target of the matching process is limited using a threshold for the similarity of the feature quantities and a threshold for the distance between the coordinates of the person images, and the maximum flow problem of the similarity of the feature quantities in combinations of tracking results is solved.

Description

Technical Field

[0001] The present invention relates to a tracking technique for tracking a person using camera images.

Background Art

[0002] Techniques for tracking moving persons using images captured by cameras have been proposed for purposes such as analyzing the behavior of persons in an office and tracking suspicious individuals. In Patent Document 1, a moving body tracking system is proposed that tracks a moving body by extracting feature quantities of the moving body from images captured by each of a plurality of cameras, and tracks the moving body by linking the tracking results obtained from the plurality of camera images.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of Invention

Problems to be Solved by the Invention

[0004] In Patent Document 1, a moving body can be tracked by linking tracking results from images captured by a plurality of cameras. When tracking a person using images captured by multiple cameras, it is necessary to identify images of the same person across the images captured by the multiple cameras. However, depending on the installation position of the cameras and the shooting angle, it may not be possible to identify the images of the same person, leading to a problem of erroneous recognition where images of a different person are mistakenly recognized as belonging to the same person.

[0005] The present invention is made to solve such a problem, and an object thereof is to provide a highly accurate tracking device free from erroneous recognition when tracking a person using images captured by a plurality of cameras.

Means for Solving the Problems

[0006] To achieve this objective, the tracking device of the present invention is a tracking device that tracks the position of a person in a predetermined area using image information acquired by a plurality of imaging devices installed in the area, and comprises: a person detection unit that detects a person's image based on the image information acquired by each of the plurality of imaging devices; a tracking processing unit that acquires tracking information of the same person's image by tracking the person's image using the feature information of the person's image for each imaging device; and a matching processing unit that determines whether the person's image is of the same person by performing a matching process of the tracking information using the similarity of the feature information of the person's image in a combination of the tracking information from two of the imaging devices. The tracking processing unit assigns a first identification information, which is the identification information of the person image for each imaging device, to each piece of tracking information, and acquires the tracking information including the coordinate information of the person image in the area and the feature information. The matching processing unit excludes combinations of tracking information having a similarity below a predetermined threshold and combinations of tracking information where the distance between the coordinate information is greater than a predetermined threshold from the matching process, and performs the matching process of the tracking information by solving the maximum flow problem related to the similarity of the combinations of tracking information, and assigns a second identification information, which is the identification information in the area, to the combinations of tracking information that are determined to be the person images of the same person.

[0007] Furthermore, in another configuration example of the tracking device according to the present invention, the matching processing unit assigns the second identification information, which is the identification information for the area, to the tracking information that has not been assigned the second identification information in a combination of tracking information that has been determined to be the person image of the same person, and excludes from the matching processing combination of tracking information that does not match the second identification information in a combination of tracking information that has been assigned the second identification information.

[0008] Furthermore, in another configuration example of the tracking device according to the present invention, the matching processing unit is: In one imaging device, the tracking information to which the second identification information has been assigned is not assigned the second identification information. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a highly accurate tracking device that does not misrecognize people when tracking them using images captured by multiple cameras. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 is a diagram illustrating a tracking system using a tracking device according to an embodiment of the present invention. [Figure 2] Figure 2 is a block diagram showing the configuration of a tracking device according to an embodiment of the present invention. [Figure 3] Figure 3 is a diagram illustrating the tracking process in a tracking system according to an embodiment of the present invention. [Figure 4] Figure 4 is a diagram illustrating the tracking process of a person's image in an embodiment of the present invention. [Figure 5] Figure 5 is a diagram illustrating the tracking information matching process in an embodiment of the present invention. [Figure 6] Figure 6 is a diagram illustrating the tracking information matching process in an embodiment of the present invention. [Figure 7] Figure 7 is a diagram illustrating the tracking information matching process in an embodiment of the present invention. [Figure 8] Figure 8 is a diagram illustrating the tracking information matching process in an embodiment of the present invention. [Figure 9] Figure 9 is a diagram illustrating the assignment of a global ID in the matching process according to an embodiment of the present invention. [Figure 10]FIG. 10 is a diagram for explaining assignment of global IDs in matching processing according to an embodiment of the present invention. [Figure 11] FIG. 11 is a diagram for explaining tracking information matching processing according to the embodiment of the present invention. [Figure 12] FIG. 12 is a diagram for explaining tracking information matching processing according to the embodiment of the present invention. [Figure 13] FIG. 13 is a diagram for explaining tracking information matching processing according to the embodiment of the present invention. [Figure 14] FIG. 14 is a diagram showing an operation flowchart of matching processing of the tracking device according to the embodiment of the present invention. [Description of Embodiments]

[0011] Next, an embodiment of the present invention will be described with reference to the drawings. However, the present invention can be implemented in many different aspects, and should not be construed as being limited to the embodiments of the present invention described below.

[0012] <Overview of Tracking System> Referring to FIG. 1, a tracking system 1 to which a tracking device according to an embodiment of the present invention is applied will be described. FIG. 1 is a diagram for explaining the tracking system using the tracking device according to the embodiment of the present invention.

[0013] The tracking system 1 using the tracking device of the present embodiment includes imaging devices (cameras) (20-A to 20-D) installed at predetermined positions in a predetermined area 100 such as an office, and a tracking device 10 that tracks the positions of persons (2-1 to 2-6) in the area 100 using image information acquired by the imaging devices (cameras) (20-A to 20-D). The tracking device 10 and the imaging devices (cameras) (20-A to 20-D) are connected via a communication network NW.

[0014] <Configuration of Tracking Device> The tracking device 10 includes, as main circuit configurations, a control unit 11, a storage unit 15, an input unit 16, a display unit 17, and an I / F unit 18.

[0015] <Configuration of Control Unit>

[0016] The control unit 11 includes a human detection unit 12 that detects human images of persons (2-1 to 2-6) based on image information acquired by each of the plurality of imaging devices (20-A to 20-D), a tracking processing unit 13 that acquires tracking information of human images of the same person by tracking the human images using feature quantity information of the human images for each of the imaging devices (20-A to 20-D), and a matching processing unit 14 that determines whether the human images are of the same person by performing matching processing on the tracking information using the similarity of the feature quantity information of the human images in a combination of tracking information from two imaging devices.

[0017] The control unit 11 has a CPU and its peripheral circuits, and is configured to perform human image detection processing, tracking processing for tracking human images, and tracking information matching processing by causing the CPU to cooperate with programs. Programs for implementing various processing units that perform the human image detection processing, the tracking processing for tracking human images, and the tracking information matching processing are stored in the storage unit 15 in advance.

[0018] <Human Detection Unit> The human detection unit 12 detects human images of persons (2-1 to 2-6) based on image information acquired by each of the plurality of imaging devices (20-A to 20-D).

[0019] In the human detection processing by the human detection unit 12, human images are detected from the image information acquired by the imaging devices (20-A to 20-D). As the algorithm for performing the human detection processing, various existing algorithms such as an object detection algorithm based on deep learning can be used.

[0020] <Tracking Processing Unit> The tracking processing unit 13 acquires tracking information for the same person's image by tracking the person's image using the feature information of the person's image for each imaging device (20-A to 20-D).

[0021] The tracking process in the tracking processing unit 13 tracks the person image using the feature information of the person image detected by the person detection unit 12. Various existing algorithms, such as object tracking algorithms based on deep learning, can be used as the algorithm for performing the person tracking process.

[0022] In the tracking process of the tracking processing unit 13, the feature quantities in the image frame of the image acquired by the imaging device are used to track the person's image between the current image frame and the previous image frame. If tracking is successful, that is, if the person's image can be tracked, tracking information (tracklet) for the person's image is obtained.

[0023] The tracking information includes local coordinates in the images from the imaging devices (20-A to 20-D), global coordinates indicating the position in a predetermined area 100, feature quantities of the person image, a local ID (first identification information) which is the identification information of the person image for each imaging device, and a global ID (second identification information) which is the identification information in a predetermined area 100.

[0024] The global coordinates indicating the position information in a predetermined area 100 can be obtained by using the position information and captured images of the imaging devices (20-A to 20-D) in the predetermined area 100, and performing a coordinate transformation on the local coordinates in the images of the imaging devices (20-A to 20-D).

[0025] The local ID (first identification information), which is the identification information of the person image for each imaging device, is assigned to the new tracking information (tracklet) acquired by each imaging device (20-A to 20-D).

[0026] The global ID (second identification information), which is the identification information for a predetermined area 100, is the identification information that is assigned to the tracking information when a match is achieved in the matching processing unit 14 and it is determined that the tracking information belongs to the same person. In the tracking processing unit 13, if tracking is possible from the previous image frame, the global ID (second identification information) of the previous image frame is inherited.

[0027] <Matching Processing Unit> The matching processing unit 14 determines whether the images are of the same person by performing a tracking information matching process using the similarity of the feature information of the person's image in the combination of tracking information from the two imaging devices.

[0028] In this embodiment, the tracking information matching process is performed by solving a maximum flow problem related to the similarity of feature information included in the tracking information for a combination of tracking information from two imaging devices. That is, in this embodiment, the tracking information matching process is performed by searching for a combination of tracking information that maximizes the sum of the similarity values ​​of the feature information for the combination of tracking information.

[0029] In this case, if the matching process is performed by solving the maximum flow problem, the accuracy of the matching may decrease because combinations of tracking information with low similarity or tracking information with coordinates far apart in a given area are selected.

[0030] Therefore, the matching processing unit 14 of this embodiment is configured to exclude combinations of tracking information having a similarity below a predetermined threshold from the matching process, and to exclude combinations of tracking information where the distance between coordinate information is greater than a predetermined threshold from the matching process.

[0031] By excluding combinations of tracking information with low similarity or those with large coordinate distances from the matching process, it is possible to prevent misrecognitions that occur when solving the maximum flow problem, where images of different people are mistakenly identified as the same person, thereby improving the accuracy of the matching process.

[0032] The matching processing unit 14 assigns a global ID (second identification information), which is area identification information, to combinations of tracking information that, as a result of the matching process, are determined to be images of the same person. If tracking was possible from the previous image frame, the tracking processing unit 13 inherits the global ID (second identification information) from the previous image frame.

[0033] A common global ID (second identification information) is assigned to images of the same person from multiple imaging devices. Furthermore, this global ID (second identification information) is passed between image frames in the image information of each imaging device, making it possible to track a person's image across multiple imaging devices.

[0034] <Storage section> The memory unit 15 consists of a storage device such as a semiconductor memory or a hard disk, and is configured to store the program that executes the person detection process, tracking process, and matching process in the control unit 11, as well as the results of the person detection process, tracking process, and matching process.

[0035] The input unit 16 is for inputting the contents of the tracking information and various parameters used in tracking and matching processing, such as the similarity threshold and coordinate information threshold when performing matching processing.

[0036] The display unit 17 is for displaying the results of detecting human images in the imaging device (20-A to 20-D), tracking information acquired through tracking, and the results of the tracking information matching process.

[0037] [I / F section] The I / F unit 18 is configured to acquire image information from the imaging devices (20-A to 20-D) via the communication network NW. The image information acquired by the imaging devices (20-A to 20-D) is obtained via the communication network NW and used for person detection processing, tracking processing, and matching processing in the tracking device 10.

[0038] <Tracking process in a tracking system> Figure 3 is a diagram illustrating the tracking process in the tracking system 1 of an embodiment of the present invention. The tracking device 10 of this embodiment is configured to perform a person image detection process, a tracking process to track the person image, and a tracking information matching process.

[0039] <Person detection process> In the person detection process, a person's image is detected based on image information acquired by each of the multiple imaging devices (20-A to 20-D). For example, in the tracking system 1 shown in Figures 1 and 2, a rectangular image of a person is detected from the image information acquired by cameras #A to #D.

[0040] The person detection process detects human images from image information acquired by the imaging devices (20-A to 20-D). Various existing algorithms, such as object detection algorithms based on deep learning, can be used as the algorithm for performing the person detection process.

[0041] <Tracking process> The tracking process obtains tracking information for the same person's image by tracking the person's image using the feature information of the rectangular image of the person for each camera from Camera #A to Camera #D.

[0042] The tracking process tracks the person image using the feature information of the person image detected by the person detection process. Various existing algorithms, such as deep learning-based object tracking algorithms, can be used as the algorithm for performing the person tracking process.

[0043] In the tracking process, the features of the image frame acquired by the imaging device are used to track the person's image between the current and previous image frames. If tracking is successful, that is, if the same person's image can be tracked, tracking information (tracklet) for that person's image is obtained.

[0044] The tracking information includes local coordinates in the image from the imaging device, global coordinates indicating the position in a predetermined area 100, feature quantities of the person image, a local ID (first identification information) which is the identification information of the person image for each imaging device, and a global ID (second identification information) which is the identification information in the predetermined area 100.

[0045] The global coordinates indicating the position information in a predetermined area 100 can be obtained by using the position information of the imaging device in the predetermined area 100 and the captured image, and performing a coordinate transformation on the local coordinates in the image of the imaging device.

[0046] The local ID (first identification information), which is the identification information of the person image for each imaging device, is assigned to the new tracking information (tracklet) acquired by each imaging device.

[0047] The global ID (second identification information), which is the identification information for a predetermined area 100, is the identification information that is assigned to the tracking information when a match is achieved in the matching process and it is determined that the tracking information belongs to the same person. In the tracking process, if tracking is possible from the previous image frame, the global ID (second identification information) of the previous image frame is inherited.

[0048] <Matching process> In the matching process, the similarity of the feature information of the person's image is used to perform a matching process on the tracking information of the combination of tracking information from two imaging devices, thereby determining whether the images are of the same person.

[0049] In this embodiment, tracking information matching is performed by solving a maximum flow problem related to the similarity of feature information included in the tracking information for combinations of tracking information from two imaging devices. That is, in this embodiment, tracking information matching is performed by searching for a combination of tracking information that maximizes the sum of the similarity values ​​of the feature information for combinations of tracking information. For example, in the tracking system 1 shown in Figures 1 and 2, matching is performed by solving a maximum flow problem for combinations of tracklet information acquired by cameras #A to #D.

[0050] In the matching process, a global ID (second identification information), which is area-specific identification information, is assigned to combinations of tracking information that are determined to be images of the same person as a result of the matching process. If tracking is possible from the previous image frame during the tracking process, the global ID (second identification information) from the previous image frame is inherited.

[0051] A common global ID (second identification information) is assigned to images of the same person from multiple imaging devices. Furthermore, this global ID (second identification information) is passed between image frames in the image information of each imaging device, making it possible to track images of a person using multiple imaging devices.

[0052] <Tracking process for human images> Figure 4 is a diagram illustrating the tracking process of a person's image in an embodiment of the present invention. Figure 4 is an example of the result of tracking a rectangular image of a person with camera #A.

[0053] In Figure 4, the rectangular image with local ID:0 and the rectangular image with local ID:1 are detected in the image frame at time t=0. In the image frame at time t=1, the rectangular image with local ID:2 and the rectangular image with local ID:3 are detected.

[0054] In Figure 4, when tracking a person's image using feature information between the image information of the image frame at time t=0 and the image information of the image frame at time t=1, it was determined that the rectangular image with local ID:0 and the rectangular image with local ID:2 are of the same person, and that the rectangular image with local ID:1 and the rectangular image with local ID:3 are of the same person.

[0055] <Matching process for tracking information> <Application of the Maximum Flow Problem> Figure 5 is a diagram illustrating the tracking information matching process in an embodiment of the present invention. Figure 5 shows an example of the result of matching the tracking information of camera #A and the tracking information of camera #B by solving the maximum flow problem.

[0056] In the example shown in Figure 5, tracking information (tracklets) for local ID:0 and local ID:1 is detected in camera #A, and tracking information (tracklets) for local ID:2 and local ID:3 is detected in camera #B.

[0057] As shown in the similarity table in Figure 5, the feature similarity between the tracking information of camera #A and camera #B was calculated, and the feature similarity between camera #A's ID:0 and camera #B's ID:2 was "0.2", and the feature similarity between camera #A's ID:0 and camera #B's ID:3 was "0.8". Similarly, the feature similarity between camera #A's ID:1 and camera #B's ID:2 was "0.7", and the feature similarity between camera #A's ID:1 and camera #B's ID:3 was "0.3". For feature similarity, for example, COS similarity can be used, but other similarity measures may also be used.

[0058] As shown in the similarity graph in Figure 5, when solving the maximum flow problem with the tracking information from camera #A and the tracking information from camera #B, the combination of feature similarity "0.8" between camera #A's ID:0 and camera #B's ID:3, and feature similarity "0.7" between camera #A's ID:1 and camera #B's ID:2, is the combination that yields the highest total similarity.

[0059] In this case, matching the tracking information between camera #A and camera #B results in the determination that ID:0 in camera #A and ID:3 in camera #B are the same person, and ID:1 in camera #A and ID:2 in camera #B are the same person.

[0060] Figure 6 is a diagram illustrating the tracking information matching process in an embodiment of the present invention. Figure 6 shows an example of the result of matching the tracking information of camera #A and the tracking information of camera #B by solving the maximum flow problem.

[0061] In the example in Figure 6, camera #A detects tracking information (tracklets) for local ID:0 and local ID:1. However, camera #A cannot detect the person behind local ID:1 due to overlapping of person images.

[0062] In the example in Figure 6, tracking information (tracklets) for local ID:2 and local ID:3 is detected in camera #B. In camera #B, due to overlapping of person images, the person behind the person with local ID:2 is not detected.

[0063] As shown in the similarity table in Figure 6, the feature similarity between the tracking information of camera #A and camera #B was calculated, and the feature similarity between camera #A's ID:0 and camera #B's ID:2 was "0.2", and the feature similarity between camera #A's ID:0 and camera #B's ID:3 was "0.8". Similarly, the feature similarity between camera #A's ID:1 and camera #B's ID:2 was "0.2", and the feature similarity between camera #A's ID:1 and camera #B's ID:3 was "0.1".

[0064] As shown in the similarity graph in Figure 6, when solving the maximum flow problem with the tracking information from camera #A and the tracking information from camera #B, the combination of feature similarity "0.8" between camera #A's ID:0 and camera #B's ID:3, and feature similarity "0.2" between camera #A's ID:1 and camera #B's ID:2, is the combination that maximizes the total similarity.

[0065] In this case, as a result of matching the tracking information between camera #A and camera #B, camera #A's ID:0 and camera #B's ID:3 are determined to be the same person, and camera #A's ID:1 and camera #B's ID:2 are determined to be the same person despite their low similarity.

[0066] In the matching processes illustrated in Figures 5 and 6, the correct matching result is obtained in the example in Figure 5. On the other hand, in the example in Figure 6, the correct matching result is not obtained because there is overlapping of the person images in the image information. In Figure 6, the person behind ID:1 in camera #A and the person with ID:2 are the same person, and the person with ID:1 in camera #A and the person behind ID:2 are the same person.

[0067] Thus, when applying the maximum flow problem to combinations of tracking information to search for a combination of tracking information that maximizes the sum of the similarity values ​​of the feature information, the matching process may be reduced by selecting combinations of tracking information with low similarity.

[0068] <Introducing a similarity threshold> In Figure 7, by introducing a similarity threshold in the matching process shown in Figure 6, misrecognition caused by applying the maximum flow problem to combinations of tracking information is prevented.

[0069] In Figure 7, a similarity threshold of 0.5 is introduced in the matching process shown in Figure 6, excluding combinations of tracking information with a similarity score of 0.5 or less from the matching process.

[0070] In Figure 7, by excluding tracking information combinations with a similarity score of 0.5 or less from the matching process, it is possible to avoid the misidentification where ID:0 in camera #A and ID:3 in camera #B are determined to be the same person, but ID:1 in camera #A and ID:2 in camera #B are determined to be the same person.

[0071] Thus, by introducing a similarity threshold in the matching process, misrecognition caused by applying the maximum flow problem can be avoided. If it is possible to prevent identifying people who are not the same person as the same person, then in subsequent image frame matching processes, overlaps of people in the images will be resolved, making it possible to obtain correct judgment results.

[0072] <Introducing a threshold for distance between coordinates> Figures 6 and 7 illustrate how introducing a similarity threshold during the matching process can improve its accuracy. While introducing a similarity threshold can improve the accuracy of the matching process, depending on the state of the image acquired by the imaging device, misrecognition may not be prevented even with a similarity threshold.

[0073] Using Figures 8 and 9, we will explain how a threshold is introduced for the distance between coordinate positions when performing the matching process in this embodiment. In this embodiment, by introducing a threshold for the distance between coordinate positions when performing the matching process, misrecognition of people when applying the maximum flow problem is prevented.

[0074] In the example shown in Figure 8, tracking information (tracklet) ID:0 for camera #A, tracking information (tracklet) ID:1 for camera #B, tracking information (tracklet) ID:2 for camera C, and tracking information (tracklet) ID:3 for camera C are obtained in global coordinates.

[0075] Here, in the example in Figure 8, we assume that ID:0, ID:1, and ID:2 are the same person, but ID:3 is a different person. In Figure 8, the numbers in the circles are the local IDs of the tracking information (tracklets).

[0076] In the example in Figure 8, the distances between ID:0 and ID:1, and between ID:0 and ID:2 are below the threshold, so the combinations of ID:0 and ID:1, and ID:0 and ID:2 are included in the matching. On the other hand, the distances between ID:0 and ID:3, between ID:1 and ID:2, and between ID:1 and ID:3 are greater than the threshold, so the combinations of ID:0 and ID:3, ID:1 and ID:2, and ID:1 and ID:3 are excluded from the matching because the distances are too great and the probability of them being the same person is low.

[0077] Thus, by introducing a threshold for the distance between global coordinate locations, in addition to the threshold for feature similarity, when deciding whether to include an object in the matching process, it becomes possible to prevent misrecognition of individuals when applying the maximum flow problem.

[0078] <Assignment of Global ID> The assignment of a global ID in the matching process of an embodiment of the present invention will be explained using Figures 9 and 10. Figure 9 is a diagram illustrating the assignment of a global ID in the matching process of an embodiment of the present invention. The global ID (second identification information) is identification information that is assigned to tracking information when a match is achieved in the matching process and it is determined that the tracking information belongs to the same person.

[0079] In the example in Figure 9, similar to Figure 8, tracking information (tracklet) ID:0 for camera #A, tracking information (tracklet) ID:1 for camera #B, tracking information (tracklet) ID:2 for camera C, and tracking information (tracklet) ID:3 are obtained in global coordinates. Here, in Figure 9, similar to Figure 8, it is assumed that ID:0, ID:1, and ID:2 are the same person, but ID:3 is a different person. In Figure 9, the numbers in the circles are the local IDs of the tracking information (tracklet).

[0080] If we introduce a threshold for the distance between global coordinate positions to determine whether to include them in the matching, the distances between ID:0 and ID:1, ID:0 and ID:2, and ID:1 and ID:3 are below the threshold, so the combinations of ID:0 and ID:1, ID:0 and ID:2, and ID:1 and ID:3 will be included in the matching. On the other hand, the distances between ID:0 and ID:3, and ID:1 and ID:2 are greater than the threshold, so the combinations of ID:0 and ID:3, and ID:1 and ID:2 will be excluded from the matching.

[0081] Thus, when a threshold is introduced for the distance between global coordinate locations in a combination of tracking information to determine whether to include it in a matching process, if the distance between the coordinates of the tracking information is small, the combination of ID:1 and ID:3 may be included in the matching process even though they are different people. In such cases, it may be possible to exclude them from the matching process using a feature similarity threshold, but depending on the similarity value, they may be determined to be the same person, and as a result, a global ID may be assigned to them.

[0082] Figure 10 illustrates the assignment of global IDs when matching the combinations of ID:0 and ID:1, ID:0 and ID:2, and ID:1 and ID:3 in the example from Figure 9. Note that the following explanation assumes that tracking information of other individuals could not be excluded from the matching target due to the similarity threshold.

[0083] In Figure 10 (1), if the matching target is the combination of ID:0 of ​​camera #A and ID:1 of camera #B, then ID:0 and ID:1 are determined to be the same person. In this case, since local ID:0 and local ID:1 do not have a global ID assigned to them, a common global ID:0 is assigned to them.

[0084] In Figure 10 (2), if the matching target is the combination of ID:0 of ​​camera #A and ID:2 of camera #C, then ID:0 and ID:2 are determined to be the same person. In this case, since local ID:2 does not have a global ID assigned to it, it is assigned the global ID:0 which is common to local ID:0 and local ID:1.

[0085] In Figure 10 (3), if the matching target is the combination of ID:1 of camera #B and ID:3 of camera #C, even if solving the maximum flow problem determines that ID:1 and ID:3 are the same person, in Figure 10 (2), the global ID:0 is already assigned to local ID:2 of camera #C, so the global ID:0 common to ID:0, ID:1, and ID:2 is not assigned.

[0086] In this way, by preventing the same global information from being assigned to different tracking information (tracklets) in the image data of a single camera, it is possible to avoid assigning duplicate global IDs that occur when solving the maximum flow problem. By preventing the assignment of duplicate global IDs, it becomes possible to improve the accuracy of matching using global IDs.

[0087] <Matching process and assignment of global ID> Figure 11-13 illustrates the tracking information matching process and global ID assignment in an embodiment of the present invention.

[0088] Figure 11 is a diagram illustrating the tracking information matching process in an embodiment of the present invention. In the example in Figure 11, tracking information (tracklets) ID:0, ID:1, and ID:2 of camera #A are acquired, and tracking information (tracklets) ID:3, ID:4, and ID:5 of camera #B are acquired.

[0089] The number on the left within the circle is the local ID, and the number on the right is the global ID inherited from the matching result of the previous image frame. Camera #A's local ID:0 is assigned global ID:1, and Camera #B's local ID:4 is assigned global ID:1.

[0090] In the example in Figure 11, camera #A's local ID:0 and camera #B's local ID:4 are assigned the same global ID:1. Furthermore, if the distance between the coordinates of camera #A's local ID:0 and camera #B's local ID:4 is less than or equal to a threshold, the camera will be subject to matching processing using the maximum flow problem.

[0091] On the other hand, based on the global ID, the combination of camera #A's local ID:0 and camera #B's local ID:4 was given priority, so the combinations of camera #A's local ID:0 and other local IDs of camera #B, as well as the combinations of camera #A's local IDs:1 and ID:2 and camera #B's local ID:4, were excluded from the matching process.

[0092] In Figure 11, by prioritizing the selection of tracking information (tracklets) combinations based on a global ID, and then introducing similarity thresholds and distance thresholds between coordinates for the remaining tracking information (tracklets) combinations, it is possible to limit the tracking information (tracklets) combinations to be matched, thereby preventing misrecognition of individuals when applying the maximum flow problem.

[0093] Figure 12 is a diagram illustrating the tracking information matching process in an embodiment of the present invention. In the example in Figure 12, tracking information (tracklets) ID:0, ID:1, and ID:2 of camera #A are acquired, and tracking information (tracklets) ID:3, ID:4, and ID:5 of camera #B are acquired.

[0094] The number on the left within the circle is the local ID, and the number on the right is the global ID inherited from the matching result of the previous image frame. Camera #A's local ID:0 is assigned global ID:1, and Camera #B's local ID:4 is assigned global ID:2.

[0095] In the example in Figure 12, camera #A's local ID:1 and camera #B's local ID:4 are assigned different global IDs. Therefore, the combination of camera #A's local ID:0 and camera #B's local ID:4 is excluded from the matching process.

[0096] In Figure 12, by excluding combinations of tracking information (tracklets) assigned different global IDs, and then introducing similarity thresholds and distance thresholds between coordinates for the remaining combinations of tracking information (tracklets), it is possible to limit the combinations of tracking information (tracklets) to be matched, thereby preventing misrecognition of individuals when applying the maximum flow problem.

[0097] Figure 13 is a diagram illustrating the tracking information matching process in an embodiment of the present invention. In the example in Figure 13, tracking information (tracklets) ID:0, ID:1, and ID:2 of camera #A are acquired, and tracking information (tracklets) ID:3, ID:4, and ID:5 of camera #B are acquired.

[0098] The number on the left within the circle is the local ID, and the number on the right is the global ID inherited from the matching result of the previous image frame. Camera #A's local ID:0 is assigned global ID:1, and Camera #B's local ID:4 is assigned global ID:1.

[0099] In the example in Figure 13, camera #A's local ID:0 and camera #B's local ID:4 are assigned the same global ID:1. Furthermore, if the distance between the coordinates of camera #A's local ID:0 and camera #B's local ID:4 is less than or equal to a threshold, the camera will be subject to matching processing using the maximum flow problem.

[0100] On the other hand, based on the global ID, the combination of camera #A's local ID:0 and camera #B's local ID:4 was given priority, so the combinations of camera #A's local ID:0 and other local IDs of camera #B, as well as the combinations of camera #A's local IDs:1 and ID:2 and camera #B's local ID:4, were excluded from the matching process.

[0101] In the left diagram of Figure 13, an example of the result when the maximum flow problem is applied to the combination of tracking information (tracklets) that was matched is shown in the right diagram of Figure 13. As a result of solving the maximum flow problem, the combination of local ID:1 of camera #A and local ID:5 of camera #B was determined to be the same person and was assigned global ID:2. Also, the combination of local ID:2 of camera #A and local ID:3 of camera #B was determined to be the same person and was assigned global ID:3.

[0102] <Flowchart of the matching process> Figure 14 is a diagram showing the operation flowchart of the matching process of the tracking device according to an embodiment of the present invention.

[0103] The matching processing unit uses the tracking information (tracklets) obtained through the tracking process to create a similarity graph consisting of combinations of tracking information (tracklets) between the cameras to be matched (step S1-1).

[0104] The matching processing unit selects, in the similarity graph, combinations of tracking information (tracklets) with matching global IDs, where the distance between global coordinates is less than or equal to a threshold, as the target of the matching process (step S1-2).

[0105] The matching processing unit discards combinations of tracking information (tracklets) with global IDs that do not match in the similarity graph, and excludes them from the matching process (step S1-3).

[0106] The matching processing unit discards combinations of tracking information (tracklets) in the similarity graph whose distance on the global coordinate system is greater than a threshold, and excludes them from the matching process (step S1-4).

[0107] The matching processing unit discards combinations of tracking information (tracklets) in the similarity graph whose similarity is below a threshold, and excludes them from the matching process (step S1-5).

[0108] The matching processing unit performs the matching process by solving the maximum flow problem for the combinations of tracking information (tracklets) limited by the above-described process in the similarity graph (steps S1-6).

[0109] The matching processing unit assigns a global ID to combinations of tracking information that are determined to be images of the same person (steps S1-7).

[0110] [Effects of the embodiments of the present invention] The tracking device of this embodiment is a tracking device that tracks the position of a person in a predetermined area using image information acquired by multiple imaging devices installed in that area. It detects a person's image from the image information acquired by each of the multiple imaging devices, and tracks the person's image using the feature information of the person's image for each imaging device to acquire tracking information of the same person. It is configured to determine whether the images are of the same person by performing a tracking information matching process using the similarity of the feature information of the person's image in the combination of tracking information from two imaging devices.

[0111] The tracking device of this embodiment is configured to assign first identification information, which is the identification information of a person image for each imaging device, to each piece of tracking information, acquire tracking information including coordinate information and feature information of a person image in an area, exclude combinations of tracking information having a similarity below a predetermined threshold and combinations of tracking information where the distance between coordinate information is greater than a predetermined threshold from the matching process, perform a matching process of tracking information by solving a maximum flow problem related to similarity in the combinations of tracking information, and assign second identification information, which is the identification information for an area, to combinations of tracking information that are determined to be person images of the same person.

[0112] By excluding combinations of tracking information with a similarity below a predetermined threshold and combinations of tracking information where the distance between coordinate information is greater than a predetermined threshold from the matching process, it is possible to prevent misrecognition that occurs when solving the maximum flow problem, where images of different people are mistakenly identified as the same person, thereby improving the accuracy of the matching.

[0113] Furthermore, in the tracking device of this embodiment, in combinations of tracking information determined to be images of the same person, the second identification information, which is area identification information, may be added to tracking information that does not have the second identification information attached, and in combinations of tracking information to which the second identification information is attached, combinations of tracking information to which the second identification information does not match may be excluded from the matching process.

[0114] In combinations of tracking information to which a second identification information is assigned, by excluding combinations of tracking information to which the second identification information does not match from the matching process, the matching target can be limited using the second identification information. This prevents misrecognition that occurs when solving the maximum flow problem and improves the accuracy of the matching.

[0115] Furthermore, in the tracking device of this embodiment, it is possible to avoid assigning the second identification information to tracking information that has already been assigned the second identification information in a single imaging device.

[0116] By configuring an imaging device to not assign a second identification information to tracking information that already has this information, it becomes possible to prevent the assignment of redundant second identification information that occurs when solving the maximum flow problem, and to improve the accuracy of matching by utilizing the second identification information.

[0117] [Expansion of the embodiment] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments. Various modifications to the configuration and details of the present invention can be made within the scope of the present invention as can be understood by those skilled in the art. Furthermore, each embodiment can be arbitrarily combined and implemented without contradiction. [Explanation of Symbols]

[0118] 1...Tracking system, 10...Tracking device, 11...Control unit, 12...Person detection unit, 13...Tracking processing unit, 14...Matching processing unit, 15...Storage unit, 16...Input unit, 17...Display unit, 18...I / F unit, 20-A~20-D...Imaging device (camera), 100...Area, NW...Communication network.

Claims

1. A tracking device that tracks the position of a person in a predetermined area using image information acquired by multiple imaging devices installed in the area, A person detection unit detects a person's image based on the image information acquired by each of the multiple imaging devices, Each imaging device includes a tracking processing unit that acquires tracking information for the same person's image by tracking the person's image using the characteristic information of the person's image, A matching processing unit determines whether the images of the same person are obtained by performing a matching process on the tracking information in a combination of the tracking information of the two imaging devices, using the similarity of the feature information of the person images. Equipped with, The aforementioned tracking processing unit For each piece of tracking information, a first identification information is assigned, which is the identification information of the person image for each imaging device. The tracking information is then acquired, which includes the coordinate information of the person image in the area and the feature information. The matching processing unit is Combinations of tracking information having a similarity below a predetermined threshold and combinations of tracking information where the distance between them is greater than a predetermined threshold are excluded from the matching process. In the combination of tracking information, the matching process of the tracking information is performed by solving the maximum flow problem related to the similarity. For the combination of tracking information that has been determined to be the same person's image, a second identification information, which is identification information for the area, is assigned. Tracking device.

2. The matching processing unit is In the combination of tracking information that has been determined to be the same person's image, the second identification information, which is the identification information for the area, is added to the tracking information that has not been assigned the second identification information. In the combination of tracking information to which the second identification information is assigned, combinations of tracking information to which the second identification information does not match are excluded from the matching process. The tracking device according to claim 1.

3. The matching processing unit is In one imaging device, the tracking information to which the second identification information has been assigned is not assigned the second identification information. The tracking device according to claim 2.

Images