Information processor, method for processing information, information processing program, and information processing system

Abstract

To provide a technique of tracking an object more precisely.SOLUTION: The position of an object in a space is predicted in a case where an object is tracked by using an image acquired from a camera, and first positional information of the object is acquired on the basis of the predicted position of the object, and second positional information of the object is determined on the basis of the first positional information and the image acquired by the camera.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an object tracking device, an object tracking system, an object tracking method, a display control device, an object detection device, a program, and a recording medium. [Background technology]

[0002] In recent years, systems that use multiple cameras and the like to track people have been developed. For example, a moving object tracking system described in Patent Document 1 uses multiple in-camera tracking means that track people in a distributed manner for each camera. The system tracks moving objects in cooperation with the multiple in-camera tracking means. Patent Document 2 also describes a method for tracking the same object captured by multiple imaging units based on the tracking results of each object.

[0003] As a related technique, Patent Document 3 describes a method for early excluding an object that does not need to be tracked from the tracking targets.

[0004] Moreover, Patent Document 4 describes an apparatus for detecting a moving object in an image captured by one imaging means. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2004-72628 A [Patent Document 2] Special Publication No. 2009-510541 [Patent Document 3] International Publication No. 2013 / 012091 [Patent Document 4] JP 2006-202047 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the technology described in Patent Document 1 or 2, for example, when a camera is located far away from an object, the tracking accuracy of the object (moving body) in the image captured by the camera may be low. In this case, the technology described in Patent Document 1 or 2 may be affected by the tracking accuracy of the camera, and may not be able to integrate the tracking results of the object, or even if integration is possible, the detection accuracy of the object position required during integration may be low.

[0007] The present invention has been made in view of the above-mentioned problems, and has an object to provide a technique capable of tracking an object with higher accuracy. [Means for solving the problem]

[0008] An object tracking device according to one embodiment of the present invention comprises a plurality of detection means which detect an object from output information of sensors and output the detection results, and an integrated tracking means which tracks the object based on the plurality of detection results output by each of the plurality of detection means and generates tracking information of the object expressed in a common coordinate system, wherein the integrated tracking means outputs the generated tracking information to each of the plurality of detection means, and the detection means detects the object based on the tracking information.

[0009] An object tracking system according to one embodiment of the present invention comprises a sensor and an object tracking device that receives output information consisting of information acquired by the sensor, the object tracking device comprising a plurality of detection means that detect the object from the output information and output the detection results, and an integrated tracking means that tracks the object based on the plurality of detection results output by each of the plurality of detection means and generates tracking information of the object expressed in a common coordinate system, the integrated tracking means outputs the generated tracking information to each of the plurality of detection means, and the detection means detects the object based on the tracking information.

[0010] An object tracking method according to one aspect of the present invention detects an object from output information of a sensor, outputs detection results, tracks the object based on the outputted multiple detection results, generates tracking information for the object expressed in a common coordinate system, and outputs the generated tracking information, and detects the object based on the tracking information.

[0011] A display control device according to one embodiment of the present invention is a display control device that displays display data on a display device, wherein the display data indicates a search range in which an object is searched for based on object tracking information from among output information of a sensor, the search range being expressed in an individual coordinate system specific to the sensor that outputs the output information, and the object tracking information is information indicating the result of tracking the object based on the detection result of the object detected within the search range in the output information of each of a plurality of sensors.

[0012] An object detection device according to one embodiment of the present invention detects an object from sensor output information based on tracking information expressed in a common coordinate system, the tracking information indicating a tracking result of an object tracked based on multiple detection results output from each of multiple object detection devices.

[0013] Note that the scope of the present invention also includes a computer program for implementing the above-mentioned devices, object tracking system, or object tracking method by a computer, and a computer-readable storage medium on which the computer program is stored. Effect of the Invention

[0014] According to the present invention, an object can be tracked with higher accuracy. [Brief description of the drawings]

[0015] [Figure 1] 1 is a functional block diagram showing an example of a functional configuration of an object tracking device according to a first embodiment of the present invention. [Diagram 2] 1 is a diagram showing an example of a schematic overall configuration of an object tracking system according to a first embodiment of the present invention. [Diagram 3] FIG. 11 is a diagram for explaining a process of associating a target with a tracker. [Figure 4] 2 is a functional block diagram showing an example of a functional configuration of a detection unit of the object tracking device according to the first embodiment of the present invention. FIG. [Diagram 5] 2 is a functional block diagram showing an example of a functional configuration of an object detection unit in a detection unit of the object tracking device according to the first embodiment of the present invention. FIG. [Figure 6] 2 is a functional block diagram showing an example of a functional configuration of an integrated tracking unit of the object tracking device according to the first embodiment of the present invention. FIG. [Figure 7] 3A to 3C are diagrams for explaining a sequential tracking process of an object performed by an integrated tracking unit according to the first embodiment of the present invention; [Figure 8] 5 is a flowchart showing an example of the flow of an object tracking process of the object tracking device according to the first embodiment of the present invention. [Figure 9] FIG. 11 is a functional block diagram showing an example of a functional configuration of an object tracking device according to a second embodiment of the present invention. [Figure 10] FIG. 11 is a functional block diagram showing an example of a functional configuration of a detection unit of an object tracking device according to a second embodiment of the present invention. [Figure 11] FIG. 11 is a diagram for explaining an application example of an object tracking device according to a second embodiment of the present invention. [Figure 12] FIG. 11 is a diagram for explaining an application example of an object tracking device according to a second embodiment of the present invention. [Figure 13] FIG. 11 is a diagram for explaining an application example of an object tracking device according to a second embodiment of the present invention. [Figure 14] FIG. 11 is a diagram for explaining an application example of an object tracking device according to a second embodiment of the present invention. [Figure 15] FIG. 13 is a functional block diagram showing an example of a functional configuration of a detection unit of an object tracking device according to a third embodiment of the present invention. [Figure 16] FIG. 13 is a functional block diagram showing an example of a functional configuration of an object detection unit in a detection unit of an object tracking device according to a third embodiment of the present invention. [Figure 17] FIG. 13 is a functional block diagram showing an example of a functional configuration of an integrated tracking unit of an object tracking device according to a fourth embodiment of the present invention. [Figure 18] 13 is a diagram for explaining a collective tracking process of objects performed by an integrated tracking unit according to a fourth embodiment of the present invention; FIG. [Figure 19] FIG. 1 is a diagram illustrating an example of a hardware configuration of a computer (information processing device) capable of implementing each embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] <First embodiment> A first embodiment of the present invention will be described in detail with reference to the drawings. First, the overall configuration of an object tracking system (also simply referred to as a system) of the present invention will be described with reference to FIG. 2. FIG. 2 is a diagram showing an example of a schematic overall configuration of an object tracking system 1 according to this embodiment. As shown in FIG. 2, the object tracking system 1 according to this embodiment includes an object tracking device 10, a plurality of cameras (20-1 to 20-N (N is a natural number)), and one or more display devices 30. In this embodiment, when the plurality of cameras (20-1 to 20-N) are not distinguished from one another or when they are referred to collectively, they are referred to as cameras 20.

[0017] The object tracking device 10, the camera 20, and the display device 30 are connected to each other so as to be able to communicate with each other via a network 40. The display device 30 does not have to be included in the object tracking system 1. The display device 30 may also be configured to be directly connected to the object tracking device 10 without going through the network 40.

[0018] The camera 20 functions as a sensor that detects an object. In the present embodiment, the case where the camera 20 is used as a sensor that detects an object will be described as an example, but the present invention is not limited to this. The sensor is not limited to a camera, and may be anything capable of position measurement, such as a radio wave sensor. Also, a sensor that includes a mixture of multiple sensors, such as a sensor that combines a radio wave sensor and a camera, may be used. By using the camera 20 as a sensor, the object tracking device 10 can more suitably acquire visual information, such as color.

[0019] In addition, in this embodiment, the information acquired by the sensor is described as being an image captured by a camera, but if the sensor is a radio wave sensor, the information acquired by the sensor is radio waves acquired by the radio wave sensor.

[0020] The object tracking device 10 is a device that tracks an object included in images captured by each of a plurality of cameras 20. The functional configuration of the object tracking device 10 will be described with reference to another drawing.

[0021] The display device 30 displays the results of tracking of the object by the object tracking device 10. The display device 30 may display the video captured by the camera 20. The display device 30 may also display other information such as flow line information.

[0022] (Object Tracking Device 10) Next, the functions of the object tracking device 10 will be described. Fig. 1 is a functional block diagram showing an example of the functional configuration of the object tracking device 10 according to the present embodiment. As shown in Fig. 1, the object tracking device 10 includes a plurality of detection units (100-1 to 100-N) and an integrated tracking unit 200. In this embodiment, when the plurality of detection units (100-1 to 100-N) are not distinguished from one another or are referred to collectively, they are referred to as detection unit 100.

[0023] (Detection unit 100) The detection unit 100 detects an object from the output information of the camera 20 based on tracking information of an object in a frame preceding a frame in which the object is to be detected, which is tracking information output from the integrated tracking unit 200 described later. Here, in this embodiment, the output information of the camera 20 refers to video data showing a video image captured by the camera 20.

[0024] In this embodiment, it is assumed that the multiple detection units 100 and the multiple cameras 20 are associated one-to-one. For example, the detection unit 100-1 detects an object from a video captured by the camera 20-1, and the detection unit 100-2 detects an object from a video captured by the camera 20-2. Note that this embodiment is not limited to this, and for example, the detection unit 100-1 may detect an object from a video captured by the camera 20-N.

[0025] Furthermore, there is no need for a one-to-one correspondence between the cameras 20 and the detection units 100. For example, the detection unit 100-1 may detect an object from images captured by each of the multiple cameras 20.

[0026] The operation of the detection unit 100 will be described below. The detection unit 100 receives from the camera 20 video data showing the video captured by the camera 20 (hereinafter referred to as camera video). In FIG. 1, video data showing the video captured by the camera 20-n (n is 1 to N) is described as camera video (n). Here, the camera video may be video captured by the camera 20 such as a surveillance camera and acquired in real time, or may be video captured by the camera 20 that is temporarily stored in a storage unit (not shown) and later decoded (or played back). This video data includes time information showing the time of capture.

[0027] Furthermore, the detection unit 100 receives tracking information of an object in a previous frame from the integrated tracking unit 200. The previous frame may be the frame immediately before the frame (current frame) in which an object is to be detected, or may be a frame a predetermined number of frames before the current frame. The previous frame may be one or multiple. When the detection unit 100 detects an object for the first frame, there is no tracking information of the object in the previous frame, so the detection unit 100 does not receive (does not use) tracking information of the object in the previous frame.

[0028] The detection unit 100 detects objects (called object detection) from the received camera video using the camera video and object tracking information in the previous frame. As described above, when performing object detection on the first frame, the detection unit 100 performs the detection without using the object tracking information in the previous frame. In the following, the detected object is called a target. That is, the object detection result (also called an object detection result, or simply a detection result) is a collection of targets.

[0029] The object detection result includes, for example, information representing the position of the target, information representing the size of the target, etc., for each target. Specifically, the object detection result includes, for example, information of a circumscribing rectangle of an area (target area) occupied by the target in a frame in a video in which the object is detected, coordinate values ​​of the center of gravity of the target area, information indicating the width of the target, information indicating the height of the target, etc. Note that the object detection result is not limited to this. For example, the object detection result may include coordinate values ​​of the topmost end and bottommost end of the target area instead of or in addition to the coordinate values ​​of the center of gravity of the target area. It is sufficient that the object detection result includes information representing the position, size, etc. of the target for each target.

[0030] In this embodiment, the object detection result includes, for each target, the coordinate value of the bottom end of the target and information indicating the circumscribing rectangle of the target. The coordinate value of the bottom end of the target indicates the coordinate value of the point where the object contacts the floor surface (ground) and / or the coordinate value of the middle point of the bottom side of the circumscribing rectangle of the object. In addition, the coordinate value of the bottom end of the target may be the coordinate value of the feet when the object is a person.

[0031] Then, the detection unit 100 converts the coordinate values ​​included in the object detection result into coordinate values ​​in a common coordinate system defined within the space captured by the multiple cameras 20 (capture space), and sets the converted coordinate values ​​as the object detection result.

[0032] In addition, the object detection result may include information representing the shape of the target in addition to the above information. That is, the object detection result may include silhouette information representing the target area. Here, the silhouette information is information for distinguishing between pixels inside the target area and pixels outside the target area, and may be, for example, image information in which the pixel values ​​inside are set to 255 and the pixel values ​​outside are set to 0, or a value obtained by extracting a shape descriptor (shape feature amount) as standardized in MPEG-7 from the silhouette shape. The object detection result may also include a feature amount of the object's appearance. For example, the object detection result may also include a feature amount of the object's color, pattern, shape, and the like.

[0033] Furthermore, the object detection result may include, for each target, information describing the likelihood (target likelihood information) that indicates the certainty (accuracy) of object detection. The target likelihood information is information necessary for calculating the likelihood of a target, and is information related to the accuracy of object detection, such as the score value at the time of object detection, the distance from the camera of the detected object, and the size of the detected object. Furthermore, the detection unit 100 may calculate the likelihood of the target itself, and use the calculated likelihood as the target likelihood information.

[0034] Then, the detection unit 100 outputs the object detection result to the joint tracking unit 200.

[0035] (Integrated Tracking Unit 200) The integrated tracking unit 200 receives the detection results output from each of the detection units 100. The integrated tracking unit 200 then tracks the object based on each of the detection results. Specifically, the integrated tracking unit 200 tracks the object (performs object tracking) using the object detection results for one or more objects detected by each of the detection units 100 from the video captured by the camera 20 linked to the detection unit 100. The integrated tracking unit 200 then generates a tracking result of the object (object tracking result) expressed in a common coordinate system. In this way, the integrated tracking unit 200 integrates the object detection results detected by each detection unit 100 from the video captured by the camera 20 linked to the detection unit 100, and performs object tracking. Therefore, the object tracking performed by the integrated tracking unit 200 is also called object integrated tracking.

[0036] Hereinafter, information for each object generated as an object tracking result is called a tracker. In other words, the tracker includes information on the tracked object (object tracking result), such as information indicating the position of the tracked object and a motion model of the object, but the present invention is not limited to this. Note that the position of the tracked object is the position of the object prior to the current time (past), and is therefore also called the past position of the object.

[0037] In other words, object tracking can be regarded as a process of associating objects between frames by associating a target detected by object detection with a tracker generated before the detection of the target. This object tracking will be described with reference to FIG. 3. FIG. 3 is a diagram for explaining the process of associating targets and trackers by the integrated tracking unit 200. As shown in FIG. 3, the number of targets is M, and the number of trackers is K (M and K are integers equal to or greater than 0). The integrated tracking unit 200 associates the M targets with the K trackers. When associating targets with trackers, the integrated tracking unit 200 first predicts the current position of the object from the past position of the object indicated by information included in the tracker, and then associates the targets with the trackers using an index indicating the relevance between the targets and the trackers.

[0038] In other words, the integrated tracking unit 200 predicts the position of an object in the current frame based on the position of the object detected in the previous frame and the motion model of the object calculated and stored for each tracker. This can be done using various existing methods such as a method using a Kalman filter or a method using a particle filter.

[0039] Then, the integrated tracking unit 200 associates the object tracking result (tracker) in the previous frame with the object (target) included in the detection result, based on, for example, the following information (1) to (3). (1) The proximity of the object’s position in the current frame predicted by the tracker to the target’s position (2) Similarity of appearance features between the target and the object tracked by the tracker (3) The likelihood of the target and the tracker This matching process can be reduced to a cost minimization problem in a bipartite graph as shown in Fig. 3. Therefore, the integrated tracking unit 200 can solve this problem using an algorithm such as the Hungarian method.

[0040] In Fig. 3, the case where the target and the tracker are associated with each other is shown by using arrows. In other words, it is shown that the topmost target is associated with the topmost tracker.

[0041] If there is a target that does not correspond to a tracker, the integrated tracking unit 200 judges whether the target can be regarded as a newly appeared object. If the integrated tracking unit 200 judges that the target is highly likely to have newly appeared, it adds a new tracker related to the target. In FIG. 3, the target indicated by the symbol m (referred to as target m) is assumed to be a target that does not correspond to a tracker. At this time, the integrated tracking unit 200 judges whether the target m can be regarded as a newly appeared object, and if so, creates a new tracker related to the target m.

[0042] On the other hand, if there is a tracker that does not correspond to the target, the integrated tracking unit 200 judges whether the tracker is information about an object that has disappeared from the shooting space. Then, if there is a high possibility that the tracker is information about an object that has disappeared from the shooting space, the integrated tracking unit 200 deletes the tracker. In FIG. 3, the tracker indicated by the symbol k (referred to as tracker k) is a tracker that does not correspond to the target. At this time, the integrated tracking unit 200 judges whether the tracker k is information about the disappeared object, and deletes the tracker k if the tracker k is information about the disappeared object.

[0043] The integrated tracking unit 200 repeats these processes on a frame-by-frame basis to track objects. The integrated tracking unit 200 assigns a unique ID (identifier) ​​common to all cameras 20 to the tracker, and manages the tracking results (trackers) using this ID. The integrated tracking unit 200 also includes in the tracker a value that evaluates the likelihood of the tracking result (hereinafter referred to as the tracker's likelihood (weight)) as a tracker parameter. The position indicated by the information indicating the position of the tracked object included in the tracker, and the most recent position of the object, is referred to as the tracker position. The size of the object at this time is also referred to as the size of the tracker.

[0044] Furthermore, the integrated tracking unit 200 updates the information indicating the position of each tracker and the information of the likelihood of the tracker based on the result of the association. The information of the position of the tracker is information expressed in a common coordinate system defined in the shooting space photographed by the multiple cameras 20. The information expressed in this common coordinate system is a coordinate value in the common coordinate system. For example, when the multiple cameras 20 are cameras installed in a store, the coordinate value in this common coordinate system is a coordinate system indicating the position of the floor in the real world. In contrast, the coordinate system unique to each camera 20 is called the individual coordinate system of the camera 20. This individual coordinate system is a coordinate system on the photographed image of the camera 20. Hereinafter, the information of the position expressed in the common coordinate system will be described as the coordinate value of the common coordinate system. Furthermore, the information of the position expressed in the individual coordinate system of the camera 20 will be described as the coordinate value of the individual coordinate system of the camera 20.

[0045] Then, the integrated tracking unit 200 generates a tracker whose information has been updated based on the result of the association as a new object tracking result.The integrated tracking unit 200 outputs information indicating the position and / or size of the tracker, and information on the likelihood of the tracker, etc., from the generated object tracking result (tracker), as information indicating the tracking result of object tracking (tracking information).

[0046] The tracking information includes coordinate values ​​of each tracker in a common coordinate system as information indicating the position of each tracker. The tracking information is fed back to the detection unit 100. That is, the detection unit 100 receives the tracking information and uses the tracking information for object detection in subsequent frames.

[0047] The integrated tracking unit 200 may be configured to output the object tracking result including the above tracking information and other information of the tracker to the detection unit 100.

[0048] In this way, the object tracking device 10 according to the present embodiment performs object tracking by using images captured by each of the multiple cameras and integrating the object detection results for the images captured by each of the multiple cameras 20. Then, the object tracking device 10 feeds back the obtained tracking information to object detection in the next frame.

[0049] In this way, the object tracking device 10 detects an object from the video using the tracking result for the previous frame. For example, when there is an object that is not visible from one camera 20 but is visible from another camera 20, the object tracking device 10 uses the tracking result for the object that is not visible from the one camera 20 for object detection in the video of the one camera 20. This allows the detection unit 100 to suitably detect the object when the same object appears within the range visible from the one camera 20. Therefore, the object tracking device 10 can perform object tracking for the object with high accuracy.

[0050] Therefore, the object tracking device 10 can improve the object detection accuracy compared to when the tracking result for the previous frame is not used. Also, since the object tracking device 10 performs object tracking using all of the detection results with high detection accuracy, the accuracy of the tracking result obtained as a whole is improved.

[0051] (Details of the detection unit 100) Next, the function of each unit of the object tracking device 10 will be described in more detail with reference to Fig. 4 to Fig. 8. Fig. 4 is a functional block diagram showing an example of a more detailed functional configuration of the detection unit 100 of the object tracking device 10 according to the present embodiment. As shown in Fig. 4, the detection unit 100 includes an object detection unit 110, a common coordinate conversion unit (second conversion unit) 120, and an individual coordinate conversion unit (first conversion unit) 130. Note that in Fig. 4, the camera video (n) (n is 1 to N) received by the detection unit 100 is simply referred to as the camera video.

[0052] The individual coordinate conversion unit 130 receives tracking information output from the integrated tracking unit 200 from the integrated tracking unit 200. The individual coordinate conversion unit 130 then converts the coordinate values ​​of the common coordinate system of each tracker included in the tracking information into coordinate values ​​on the frame captured by each camera 20 (i.e., coordinate values ​​expressed in the individual coordinate system specific to each camera 20). When the coordinate values ​​of the common coordinate system are (X, Y, Z) and the coordinate values ​​of the individual coordinate system of the camera 20 are expressed as (x, y), the individual coordinate conversion unit 130 obtains the coordinate values ​​(x, y) of the individual coordinate system of the camera 20 linked to the detection unit 100 including the individual coordinate conversion unit 130 from the coordinate values ​​(X, Y, Z) of the common coordinate system of the tracker. At this time, it is preferable that the individual coordinate conversion unit 130 obtains at least camera parameters representing the camera position, attitude, etc. of the camera 20 linked to the detection unit 100 by calibration. As a result, the individual coordinate conversion unit 130 converts the coordinate values ​​of the common coordinate system into coordinate values ​​of the individual coordinate system of the camera 20 using the obtained camera parameters.

[0053] The camera parameters may be stored in a storage unit (not shown) in the detection unit 100, or may be stored in a storage area in the individual coordinate conversion unit 130. In the latter case, the individual coordinate conversion unit 130 may be configured to supply the camera parameters to the common coordinate conversion unit 120.

[0054] For example, suppose that the object is a person, and the information indicating the position of the tracker is coordinate values ​​indicating the position of the feet and the position of the top of the head of the person. The coordinate values ​​of the feet and the top of the head are (X0, Y0, 0) and (X0, Y0, H) (H represents the height of the person), respectively. Also, suppose that the camera 20 linked to the detection unit 100 equipped with the individual coordinate conversion unit 130 is the camera 20-1.

[0055] At this time, the individual coordinate conversion unit 130 uses the camera parameters related to the camera 20-1 to determine the foot position (x0, y0) and the head position (x1, y1) on the frame captured by the camera 20-1. If the information indicating the tracker position includes information indicating a circumscribing rectangle, the value indicating the width of the circumscribing rectangle has previously been determined by converting the coordinate values ​​into coordinate values ​​of the common coordinate system using the camera parameters. Therefore, the individual coordinate conversion unit 130 may use a value converted again using the camera parameters as the width of the circumscribing rectangle.

[0056] In addition, not all of the objects for which the tracker shows the tracking result are visible from one camera 20, and may exist outside the field of view of this camera 20. Therefore, when the information indicating the position of the tracker included in the tracking information is information regarding an object that is not visible from the camera 20 linked to the detection unit 100, the individual coordinate conversion unit 130 cannot obtain the coordinate value of the object in the individual coordinate system of the camera 20. Therefore, the individual coordinate conversion unit 130 may not convert the coordinate value of the common coordinate system of the tracker related to such an object that is not visible outside the angle of view of the camera 20 into the coordinate value of the individual coordinate system. In this case, the individual coordinate conversion unit 130 may register in advance in a storage unit (not shown) or the like for each camera 20 the range of coordinate values ​​of the common coordinate system that are visible from each camera 20, and determine whether each object is within this range. In addition, the individual coordinate conversion unit 130 may actually convert into coordinate values ​​of the individual coordinate system, and when an abnormal value indicating the outside of the area monitored by the linked camera 20 is obtained or a value cannot be obtained, determine that the object whose coordinate value has been converted is an object that is not visible from the camera 20.

[0057] Then, the individual coordinate conversion unit 130 converts the coordinate values ​​of the common coordinate system of the tracker included in the tracking information output from the integrated tracking unit 200 into coordinate values ​​of the individual coordinate system of the camera 20 linked to the detection unit 100, and outputs the result (tracking information) to the object detection unit 110. In other words, the individual coordinate conversion unit 130 converts the tracking information expressed in the common coordinate system into tracking information expressed in the individual coordinate system, and outputs the converted tracking information to the object detection unit 110. Hereinafter, when simply described as "coordinate values ​​in the individual coordinate system," it indicates the coordinate values ​​in the individual coordinate system of the camera 20 linked to the detection unit 100.

[0058] The object detection unit 110 receives camera video from a camera 20 linked to the detection unit 100 including the object detection unit 110. The object detection unit 110 also receives tracking information converted into coordinate values ​​in an individual coordinate system from the individual coordinate conversion unit 130. Then, the object detection unit 110 detects an object from the received camera video based on the tracking information.

[0059] Then, object detection unit 110 generates a detection result. Object detection unit 110 outputs the generated detection result to common coordinate conversion unit 120. Note that this detection result is a detection result expressed in a separate coordinate system.

[0060] The configuration of object detection unit 110 will be described in more detail with reference to Fig. 5. Fig. 5 is a functional block diagram showing an example of the functional configuration of object detection unit 110 according to this embodiment. As shown in Fig. 5, object detection unit 110 includes recognition-type object detection unit (first object detection unit) 111 and search range setting unit 112.

[0061] The search range setting unit 112 receives the tracking information converted into coordinate values ​​of the individual coordinate system from the individual coordinate conversion unit 130. Then, the search range setting unit 112 uses the tracking information converted into coordinate values ​​of the individual coordinate system to determine an area (search range) for detecting an object in the current frame. That is, the search range setting unit 112 predicts the position of an object in the current frame based on the tracking information consisting of the tracking result of the previous frame converted into coordinate values ​​of the individual coordinate system. Then, the search range setting unit 112 determines a detection range in which to search for an object from the predicted position. This search range is also called a detection range of an object.

[0062] Here, the tracking information received by the search range setting unit 112 is the result of object tracking in a past frame (also called the past tracking result) when viewed from the time of the frame currently being processed. Therefore, the search range setting unit 112 predicts the movement of each object and predicts the position of each object in the current frame. Hereinafter, this predicted object position is called the predicted position. Then, the search range setting unit 112 sets the vicinity of this predicted position as the search range of the object.

[0063] It is preferable that the search range setting unit 112 predicts the movement of each object using a motion model for each object calculated from past tracking results. For example, when the position of an object has not changed in the tracking results of the past few frames (or even two frames), the search range setting unit 112 determines that the object is stationary, and sets the position of the object obtained in the tracking results as the predicted position. In addition, when the tracking results of the past few frames show that the object is moving, the search range setting unit 112 may assume that the object is moving at a constant speed, and calculate the predicted position taking into account the time difference from the past frame.

[0064] The tracking information may include the tracking results of the past few frames used by the search range setting unit 112 when predicting the movement of each object. Also, the tracking information may include a motion model of the object obtained from the tracking results of the past few frames.

[0065] This predicted position may be included in the tracking information. That is, the integrated tracking unit 200 may include a value obtained by a Kalman filter or a particle filter during object tracking as a predicted position in the tracking information.

[0066] Also, for example, at the outer edge of the range that camera 20 can capture, a new object may appear in the angle of view of camera 20. Also, if the location captured by camera 20 includes a place such as an entrance / exit, a new object may appear in the angle of view of camera 20. Therefore, it is preferable that search range setting unit 112 also includes these areas (the outer edge of the frame and / or the entrance / exit part) on the frame included in the video captured by camera 20 in the object search range.

[0067] The search range setting unit 112 outputs information indicating the set search range for the object (search range information) to the recognition-type object detection unit 111.

[0068] The recognition-type object detection unit 111 receives search range information from the search range setting unit 112. Based on the received search range information, the recognition-type object detection unit 111 detects an object from a camera image input to the recognition-type object detection unit 111. The recognition-type object detection unit 111 temporarily stores frames of the input camera image in a storage means such as a buffer in the recognition-type object detection unit 111, and upon receiving search range information, applies this information to execute object detection processing. Specifically, the recognition-type object detection unit 111 performs object detection using a classifier that has learned image features of objects for an area (search range) indicated by the search range information.

[0069] For example, when the object is a person, the recognition object detection unit 111 detects the person by applying a classifier that has learned a characteristic part of the person (for example, the head or the upper body). The recognition object detection unit 111 may use a classifier that has learned the whole person as the classifier. The recognition object detection unit 111 may use various types of classifiers as this classifier. For example, the recognition object detection unit 111 may use a classifier obtained by learning images of the head, the upper body, and the whole person of the person using a CNN (Convolutional Neural Network). The recognition object detection unit 111 may extract features using HOG (Histogram Of Gaussian) or the like, and may use a classifier such as SVM (Support Vector Machine) or GLVQ (Generalised Learning Vector Quantization). In addition to the above, the recognition object detection unit 111 may use various existing recognition-based detection methods.

[0070] In this way, the recognition-type object detection unit 111 according to this embodiment detects an object within the search range set by the search range setting unit 112. That is, the recognition-type object detection unit 111 performs object detection within the search range narrowed down by the search range setting unit 112 using the tracking result in the previous frame. Therefore, the recognition-type object detection unit 111 does not perform object detection in a range in a frame where the object is unlikely to exist, and therefore unnecessary false detections can be reduced. Furthermore, the recognition-type object detection unit 111 can speed up the object detection process.

[0071] Furthermore, the search range in which the recognition-type object detection unit 111 performs object detection may include not only the area indicated by the search range information, but also an area determined by silhouette information obtained by background difference, etc. Furthermore, the recognition-type object detection unit 111 may set the intersection between the area determined by silhouette information and the area specified by the object search range information as the area in which object detection is performed (search range).

[0072] Then, the recognition-type object detection unit 111 generates a result of the object detection (detection result) and outputs the detection result to the common coordinate transformation unit 120. At this time, the coordinate values ​​of the object included in the detection result are coordinate values ​​in the individual coordinate system.

[0073] Returning to Fig. 4, the function of the common coordinate conversion unit 120 of the detection unit 100 will be described. The common coordinate conversion unit 120 receives the object detection result expressed in the individual coordinate system from the object detection unit 110. Then, the individual coordinate conversion unit 130 converts the coordinate values ​​of the individual coordinate system included in the received detection result into coordinate values ​​of the common coordinate system. This allows the common coordinate conversion unit 120 to generate information for integrating the detected positions of the object for each camera 20.

[0074] Specifically, the common coordinate conversion unit 120 converts the coordinate values ​​of the individual coordinate system included in the object detection result into coordinate values ​​of the common coordinate system, using the camera parameters of the camera 20 linked to the detection unit 100 including the common coordinate conversion unit 120. For example, when the coordinates of the bottom end of the object on the frame captured by the camera 20 are (x0, y0), the common coordinate conversion unit 120 converts this to the coordinates of the common coordinate system (X0, Y0, 0). Here, since the ground is a plane of Z=0, the component in the Z axis direction is 0. In addition, when the coordinates of the top end of the object are (x1, y1) (here, it is assumed that the top end of the object is directly above (vertically above) the bottom end of the object), the height of the object is H. At this time, the common coordinate conversion unit 120 converts the coordinates (x1, y1) of the top end of this object into the coordinates of the common coordinate system (X0, Y0, H). The common coordinate conversion unit 120 finds the height of the object by searching for H that satisfies this. In this manner, common coordinate conversion unit 120 determines the coordinate values ​​(X, Y, Z) in the common coordinate system for each detected object.

[0075] When H is known, the common coordinate transformation unit 120 may use the known value as is.

[0076] The common coordinate transformation unit 120 outputs the detection result including the coordinate values ​​after the coordinate transformation (coordinate values ​​in the common coordinate system) to the integrated tracking unit 200. In other words, the common coordinate transformation unit 120 outputs the detection result expressed in the common coordinate system to the integrated tracking unit 200. Note that the common coordinate transformation unit 120 may include, for each of one or more targets (objects) included in the detection result, information such as silhouette information and feature amounts of the object's appearance characteristics (color, pattern, shape, etc.) as information about the object. Then, the common coordinate transformation unit 120 may output the detection result including this information to the integrated tracking unit 200.

[0077] (Details of the Integrated Tracking Unit 200) Next, the functional configuration of the integrated tracking unit 200 will be described in more detail with reference to Fig. 6. Fig. 6 is a functional block diagram showing an example of a more detailed functional configuration of the integrated tracking unit 200 of the object tracking device 10 according to this embodiment. As shown in Fig. 6, the integrated tracking unit 200 includes a prediction unit 210, a storage unit 220, an association unit 230, and an update unit 240. Note that the integrated tracking unit 200 in this embodiment is also called a sequential tracking unit because it sequentially tracks the video from each camera 20 on a camera-by-camera basis.

[0078] The sequential object tracking (also called sequential object tracking or sequential integrated tracking) performed by the integrated tracking unit 200 will be described with reference to FIG. 7. FIG. 7 is a diagram for explaining the sequential object tracking process performed by the integrated tracking unit 200 according to this embodiment. FIG. 7 shows an example of the timing of image acquisition by each of the cameras A, B, and C when the number of cameras is three. In FIG. 7, the horizontal axis indicates the time axis, and the further to the right, the later in time it is. As shown in FIG. 7, the camera A acquires images at times t1, t5, and t8. Similarly, the camera B acquires images at times t2, t4, t6, and t9, and the camera C acquires images at times t3 and t7.

[0079] 7, the times (timestamps) of frames (images) captured by each camera 20 are not necessarily the same for all cameras 20, and are usually different. Also, the frame intervals may differ for each camera 20, and may be non-uniform even for the same camera 20.

[0080] Then, the detection unit 100 performs detection in chronological order from the camera images output asynchronously between the cameras 20 , and outputs the detection results to the integrated tracking unit 200 .

[0081] The integrated tracking unit 200 according to this embodiment performs sequential tracking processing in order from the image with the earliest time. That is, in the case of FIG. 7, the integrated tracking unit 200 first performs integration between multiple cameras using the object detection result for the image of camera A at time t1, and performs object tracking. After that, the integrated tracking unit 200 performs integration between multiple cameras using the object detection results in the order of camera B at time t2, camera C at time t3, camera B at time t4, and so on, and performs object tracking. At this time, since not all objects are visible from any of the cameras 20, the integrated tracking unit 200 performs object tracking for objects that are likely to be visible from each camera 20.

[0082] Returning to FIG. 6, each component of the integrated tracking unit 200 will be described.

[0083] The memory unit 220 stores tracker information that is associated with an object (target) included in the detection result received by the integrated tracking unit 200. The tracker information stored in the memory unit 220 is managed by the update unit 240 using the tracker ID. The tracker information is, for example, information about an object whose tracking result is included in the tracker, parameters including the tracker likelihood, etc., but the present invention is not limited to this. The object information includes information indicating the past position of the object, a motion model of the object, etc., but the present invention is not limited to this, and the object information may include information included in the object detection result described above.

[0084] 6, the storage unit 220 is described as being built into the integrated tracking unit 200, but the present invention is not limited to this. The storage unit 220 may be provided in the object tracking device 10 separately from the integrated tracking unit 200. The storage unit 220 may be realized by a storage device or the like separate from the object tracking device 10.

[0085] When the storage unit 220 is not built into the integrated tracking unit 200, the storage unit 220 may be configured to store data and the like used in the object tracking device 10. For example, the storage unit 220 may store, for example, camera images captured by the cameras 20, camera parameters of each camera 20, the range of coordinate values ​​of a common coordinate system visible by each camera 20, and the like.

[0086] The prediction unit 210 predicts the position of an object in the current frame by referring to the storage unit 220. Specifically, the prediction unit 210 predicts the current position of the object based on a motion model of the object, using the tracking result (tracker) of the object in the previous frame. Here, information indicating the object's position is expressed in a common coordinate system.

[0087] In addition, the motion model of the object used by the prediction unit 210 to predict the position may be stored in the storage unit 220, or may be calculated by the prediction unit 210 using the tracking result before predicting the position of the object.

[0088] For example, prediction processing such as a Kalman filter or a particle filter can be applied to predict the position of an object by the prediction unit 210. Alternatively, the prediction unit 210 may simply calculate the speed of the object from the tracking results for the past few times, assume uniform linear motion, predict the amount of movement from the position in the previous frame from the speed, and add the amount of movement to the position in the previous frame to predict the current position.

[0089] Then, the prediction unit 210 outputs the prediction result to the association unit 230.

[0090] The association unit 230 receives the detection results output from each of the detection units 100. In FIG. 6, detection result (n) (n is 1 to N) indicates the detection result output from the detection unit 100-n. The association unit 230 also receives the prediction result from the prediction unit 210. Then, the association unit 230 refers to the storage unit 220 and uses the prediction result to associate the target included in the detection result with the tracker.

[0091] The matching unit 230 finds the combination that will give the highest degree of accuracy in the overall matching. The likelihood that a certain target m and a certain tracker k will be matched is the product of the likelihoods Pm and ηk of the target m and tracker k, respectively, and the likelihood qkm that indicates the possibility that the two are the same object. Therefore, the matching unit 230 calculates this value for each pair of target and tracker, and finds the combination that will give the highest degree of accuracy overall.

[0092] Here, the target likelihood (first likelihood) is a value that indicates the certainty (accuracy) of object detection. The accuracy of object detection depends on the size of the object to be detected (called a detected object) on the screen (on the frame), the distance from the camera 20 to the detection position of the object, how the object appears from the camera 20, and the like.

[0093] For example, if the detected object is small and the size of the detected object is close to the limit of the size that can be detected, the accuracy of object detection will be low. Also, if the size of the detected object deviates from the apparent size of the object assumed by the camera parameters, the accuracy of object detection will be low. Also, if the detected position of the object is far from the camera 20 or the lighting conditions for the area where the object exists are poor and the object is difficult to detect, the accuracy of object detection will be low. Also, if the data used to train the classifier differs significantly from the actual appearance (for example, the angle is different), the accuracy of object detection will be low.

[0094] The association unit 230 calculates the likelihood of the target by reflecting such characteristics. Specifically, the association unit 230 calculates the likelihood of the target by using the likelihood information of the target included in the detection result received from the detection unit 100. Note that, when the detection unit 100 calculates the likelihood of the target and includes the calculated likelihood as the likelihood information of the target in the detection result, the association unit 230 may use the likelihood included in the likelihood information of the target as it is. Note that the likelihood of the target does not need to reflect all the items described here, and may reflect only the main factors.

[0095] The tracker likelihood (second likelihood) is a value that indicates the certainty (accuracy) of object tracking. The accuracy of object tracking changes depending on the tracking result of object tracking in the previous frame. For example, a tracker that is reliably associated with a target in the tracking result up to the previous (past) frame of the current frame can be said to have a high accuracy of object tracking, and a tracker that is not so associated can be said to have a low accuracy of object tracking. Therefore, the association unit 230 may change the likelihood based on the result of whether or not the target and the tracker are associated in each frame, and may increase the likelihood of the tracker when the target and the tracker are associated, and may decrease the likelihood of the tracker when the target and the tracker are not associated.

[0096] In addition, in this case, if the tracker is located far from the camera 20, it is considered that the error in the position of the tracker will be large. As a result, such a tracker is difficult to associate with an object (target) included in the detection result. For this reason, the association unit 230 may change the ratio at which the likelihood of the tracker is changed according to the distance between the position of the tracker and the camera 20. Furthermore, the association unit 230 may change the ratio at which the likelihood of the tracker is changed according to the angle (depression angle or elevation angle) between the horizontal plane including the camera 20 and the line of sight when the camera 20 sees the object for which the tracking result is indicated by the tracker.

[0097] For example, when an object whose tracking result is indicated by a tracker (hereinafter referred to as a tracker's object) is close to the camera 20 and the depression angle of the camera 20 with respect to the object is greater than a predetermined angle, the accuracy of the object's position is high. Therefore, the tracker and the target are easily associated. Therefore, in such a case, the association unit 230 increases the ratio at which the tracker's likelihood is changed.

[0098] Also, for example, when the tracker object is far from the camera 20 and the depression angle of the camera 20 with respect to the object is shallower than a predetermined angle, the size of the object in the frame captured by the camera 20 becomes small. Also, a slight positional deviation on the image becomes a large deviation in the real space. Therefore, the accuracy of the detected position of the object is likely to be low. Therefore, in such a case, the association unit 230 reduces the ratio at which the tracker likelihood is changed. In this way, the association unit 230 calculates the tracker likelihood.

[0099] As described above, the association unit 230 can preferentially reflect the detection result detected by the camera 20 close to the tracker's object in the tracker's likelihood, thereby improving the tracking accuracy overall. Note that it is not necessary to reflect all the items described here in the tracker's likelihood, and only the main factors may be reflected.

[0100] Moreover, the likelihood qkm, which indicates the identity between the target m and the tracker k, indicates the degree of certainty that the two are identical. When the object indicated by the target and the object of the tracker are the same object, the positions of the target and the object of the tracker are likely to be close to each other. Therefore, the association unit 230 changes the likelihood according to the distance between the target and the object of the tracker. In other words, the association unit 230 may increase the value of the likelihood qkm when the distance between the target m and the object of the tracker k is close, and may decrease the value of the likelihood qkm when the distance is farther.

[0101] In this case, if the target m is far from the camera 20 or the depression angle of the camera 20 with respect to the target m is shallower than a predetermined angle, the accuracy of the position of the target m is likely to be low. Therefore, the association unit 230 may obtain the distance between the target and the object of the tracker using the Mahalanobis distance that takes into account the error (ambiguity) included in the detected position of the target, instead of calculating the distance using a simple Euclidean distance. In addition to the above method, the association unit 230 may control the degree of change in the likelihood qkm according to the distance, taking into account the ambiguity. That is, when the ambiguity is larger, the association unit 230 makes the change in the likelihood qkm smaller according to the distance between the target m and the object of the tracker k. In this way, the association unit 230 reduces the influence of the positional deviation of the target on the association.

[0102] Furthermore, the matching unit 230 may also take into account the similarity in appearance between the target and the tracker. That is, the matching unit 230 may extract features such as the color, pattern, and shape of the target and tracker objects, evaluate the similarity between them, and obtain the likelihood qkm.

[0103] For example, the association unit 230 may calculate the color histograms of the objects for both the target and the tracker objects, evaluate the similarity between them based on the overlap of the color histograms, and reflect the similarity in the likelihood qkm. Note that the likelihood qkm representing the identity of the target and the tracker does not need to reflect all the above-mentioned items, as with the tracker likelihood ηk and the target likelihood Pm, and may reflect only the main factors.

[0104] Furthermore, the association unit 230 may calculate each of the above likelihoods while taking into consideration the case where an object is not detected because it is outside the angle of view of the camera 20 or is occluded by another object. This allows the integrated tracking unit 200 to track an object with high accuracy even when the object is not detected or is outside the angle of view.

[0105] As described above, the problem of calculating each likelihood and finding the correspondence between the target and the tracker that maximizes each likelihood as a whole can be reduced to an assignment problem (which target is associated with which tracker) that minimizes the cost by converting each likelihood into a cost using a monotonically non-increasing function. This assignment problem can be efficiently calculated using a method such as the Hungarian method.

[0106] Then, the association unit 230 outputs the association result to the update unit 240. This association result includes information indicating which target and tracker are associated with each other, and each of the likelihoods including at least the tracker likelihood.

[0107] In this embodiment, the association unit 230 performs association using both the likelihood of the target and the likelihood of the tracker, but may perform association using either one of the likelihoods.

[0108] The update unit 240 updates the information of the tracker. Then, the update unit 240 generates this tracker as a new object tracking result. Specifically, the update unit 240 receives the result of the association from the association unit 230. Then, the update unit 240 calculates the current position of the object of the tracker based on this result. Then, the update unit 240 updates the tracker information stored in the storage unit 220. The information to be updated is, for example, the position and / or size of the object whose tracking result is included in the tracker, the motion model of the object, and parameters such as the likelihood of the tracker, but the present invention is not limited to this. The update unit 240 may update information that has been updated among the information stored in the storage unit 220.

[0109] First, the calculation of the current position of the tracker object by the update unit 240 will be described. The update unit 240 calculates the current position of the tracker object taking into consideration the accuracy of the target position. For example, the update unit 240 weights the predicted position of the object predicted by using the tracker and the detected position of the target associated with the tracker, and calculates the current position of the tracker object. In this case, the update unit 240 may control the weight depending on the accuracy of the target position.

[0110] For example, if a target is located far away from the camera 20 and the depression angle of the camera 20 with respect to the target is shallow, the accuracy of the target position is likely to be low. In such a case, the update unit 240 assigns a smaller weight to the target position.

[0111] On the other hand, if the target is located close to the camera 20 and the depression angle of the camera 20 with respect to the target is greater than a predetermined value, the accuracy of the target position is assumed to be high. In such a case, the update unit 240 assigns a larger weight to the target position.

[0112] The update unit 240 uses the predicted position and the weighted position to calculate the current position of the tracker object.

[0113] In this way, by the update unit 240 determining the weight for the target position, the prediction result of the object detection position by the camera 20 closer to the target is more strongly reflected. Therefore, the object tracking device 10 can improve the prediction accuracy of the object position.

[0114] Then, the update section 240 updates the most recent position of the object, which is included in the information about the object stored in the storage section 220, to the calculated current position of the object.

[0115] Next, updating of the tracker likelihood performed by the update unit 240 will be described.

[0116] When a tracker object is located far away from the camera 20, the size of the object becomes small, so it becomes difficult for the detection unit 100 to detect such an object.

[0117] In addition, a case where the recognition-type object detection unit 111 of the detection unit 100 performs recognition-type object detection when an object included in a frame looks different from an object used for learning will be described. When an object included in a frame looks different from an object used for learning, for example, the depression angle of the camera 20 with respect to the object, which is assumed from the position of the object in the tracker, is significantly different from the depression angle of the camera 20 with respect to the object used for learning. In such a case, it becomes difficult for the recognition-type object detection unit 111 of the detection unit 100 to detect the object included in the frame.

[0118] In such a difficult-to-detect situation, an object contained in a frame may go undetected, meaning that there may be no target associated with the tracker associated with this object.

[0119] Therefore, the update unit 240 keeps small changes in the likelihood of trackers of objects that are not associated with a target and are in a situation where the object is difficult to detect.

[0120] In this way, the update unit 240 can suppress the influence on tracking when an object is difficult to detect, and can largely reflect the detection result of a camera where the object is easy to detect in the tracker likelihood. Then, when tracking an object in the next frame, the association unit 230 performs association based on the tracker likelihood, so that the integrated tracking unit 200 can further improve the accuracy of object tracking.

[0121] Then, among the tracker likelihoods stored in the storage unit 220, the likelihood of the tracker calculated by the association unit 230 and the likelihood of the tracker in which the change is suppressed to a small value are updated.

[0122] Next, the updating of the motion model of the tracker object by the update unit 240 will be described. For example, a case will be described in which the integrated tracking unit 200 performs object tracking by predicting the position of the object using a Kalman filter. In this case, the update unit 240 substitutes the position coordinates of the tracker object associated with the target as the detected position coordinates into the update equation of the state variable of the Kalman filter stored in the storage unit 220, and updates the state of the Kalman filter.

[0123] In addition to the above, the update unit 240 updates other parameters of the tracker stored in the storage unit 220.

[0124] For example, the object itself may change its posture. For example, if the object is a person, the apparent height of the object may change when the person crouches or bends. In this way, when there is a change in the size of the object in addition to the motion model, the update unit 240 updates the information stored in the storage unit 220 that has been changed.

[0125] Furthermore, if the tracker parameters include the probability of the existence of the tracker's object, a weight indicating the reliability of the tracking result, etc., this weight changes according to the result of the association by the association unit 230. Therefore, the update unit 240 updates the parameters such as the weight.

[0126] Moreover, the update unit 240 creates and deletes a tracker as an update of the tracker. First, the update unit 240 judges whether or not a target not associated with a tracker exists after the association process by the association unit 230. This target not associated with a tracker may be an object that has newly appeared within the range photographed by the camera 20. Therefore, when a target not associated with a tracker exists, the update unit 240 judges whether or not this target can be regarded as an object that has newly appeared within the range.

[0127] That is, when there is a target that does not correspond to a tracker, the update unit 240 evaluates the probability that this target exists. Then, the update unit 240 judges whether this probability is equal to or greater than a predetermined value. Then, when this probability is equal to or greater than a predetermined value, the update unit 240 judges that the object indicated by this target is an object that has newly appeared within the above range. Then, the update unit 240 creates a new tracker related to the object (target) that has been judged to be an object that has newly appeared within the above range.

[0128] Furthermore, the update unit 240 judges whether or not there is a tracker that does not correspond to the target. This tracker that does not correspond to the target may be a tracker related to an object that has disappeared (moved from within the range to outside the range) within the range captured by the camera 20. Therefore, when a tracker that does not correspond to the target exists, the update unit 240 judges whether or not this target can be regarded as a tracker related to an object that has disappeared from the range.

[0129] That is, when a tracker that does not correspond to a target exists, the update unit 240 evaluates the probability that an object related to this tracker exists. Then, the update unit 240 judges whether this probability is below a predetermined value. Then, when this probability is below a predetermined value, the update unit 240 judges that the object related to this tracker is an object that has disappeared from the above-mentioned range. Then, the update unit 240 deletes the tracker related to the object that has been judged to be an object that has disappeared from the above-mentioned range.

[0130] The probability that an object exists is determined by the likelihood of the tracker. That is, if there is a tracker that does not correspond to the target, the update unit 240 decreases the likelihood of this tracker. Then, if the value of the likelihood of the tracker falls below a predetermined threshold, the update unit 240 deletes the tracker.

[0131] The update unit 240 then generates the trackers that are finally left as the tracking result of the object in this frame. The update unit 240 then outputs, among these, information indicating the position of the tracker and information regarding the size of the object of the tracker as information indicating the tracking result of the object tracking (tracking information).

[0132] In this way, according to the object tracking device 10 of this embodiment, object tracking is performed in order of the time information included in the camera images output from each camera 20, starting from the oldest. Objects detected from the video data of each of the multiple cameras 20 are detected by integrating the most accurate detection results from each camera 20 and using a tracking result that reflects this detection result and past tracking results. This improves the tracking accuracy of object tracking performed by the integrated tracking unit 200 of the object tracking device 10.

[0133] Next, the flow of the object tracking process of the object tracking device 10 according to the present embodiment will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of the flow of the object tracking process of the object tracking device 10 according to the present embodiment.

[0134] As shown in FIG. 8, first, the recognition-type object detection unit 111 of the detection unit 100 receives a camera image from the camera 20 linked to the detection unit 100 including the object detection unit 110 (step S81).

[0135] Detection unit 100 checks whether the received camera video frame is the first frame output from camera 20 (step S82), and if it is the first frame (YES in step S82), proceeds to the process of step S85.

[0136] If the received camera image frame is not the first frame (NO in step S82), the individual coordinate conversion unit 130 converts the tracking information for the previous frame output from the integrated tracking unit 200 into tracking information expressed in an individual coordinate system (step S83).

[0137] Then, search range setting section 112 of object detection section 110 uses the tracking information converted in step S83 to set a search range for the object in the current frame (step S84).

[0138] Then, the recognition-type object detection unit 111 detects an object from the received camera image (step S85).

[0139] Next, the individual coordinate conversion unit 130 converts the detection result by the recognition-type object detection unit 111 into a detection result expressed in a common coordinate system (step S86).

[0140] Next, the prediction unit 210 of the integrated tracking unit 200 predicts the position of the object in the current frame using the tracker information (step S87).

[0141] Then, the association unit 230 associates the object (target) included in the detection result with the tracker (step S88).

[0142] Next, the update unit 240 updates the information of the tracker, such as the position of the object in the tracker and the motion model of the object (step S89).

[0143] Furthermore, the update unit 240 creates and / or deletes a tracker (step S90). Then, the object tracking device 10 repeats this process until no more frames are input to the detection unit 100.

[0144] (effect) As described above, the object tracking device 10 according to this embodiment can track an object with higher accuracy. This is because the detection unit 100 detects an object from the output information of the camera 20 based on tracking information for the output information before the output information (video frame). Then, the integrated tracking unit 200 tracks the object based on multiple detection results output by each detection unit 100, and generates tracking information of the object expressed in a common coordinate system.

[0145] For example, if there is an object that is not visible from one camera 20 but is visible from another camera 20, the object tracking device 10 uses the tracking result for the object that is not visible from the one camera 20 for object detection in the video of the one camera 20. This allows the detection unit 100 to suitably detect the object when the same object appears within the range visible from the one camera 20. Therefore, the object tracking device 10 can perform object tracking for the object with high accuracy.

[0146] Therefore, the object tracking device 10 can improve the object detection accuracy compared to when the tracking result for the previous frame is not used. Also, since the object tracking device 10 performs object tracking using all of the detection results with high detection accuracy, the accuracy of the tracking result obtained as a whole is improved.

[0147] In this way, the object tracking device 10 according to the present embodiment can extract the movement line of a person or the like moving across the areas photographed by each of the multiple cameras 20. As a result, the tracking results by the object tracking device 10 can be used to analyze the behavior of customers moving around in a store, for example, and can be used as basic information for marketing or changing the layout of the store. In addition, the tracking results can be used to detect people wandering between areas for security purposes.

[0148] In addition, since the search range setting unit 112 uses the tracking result to set a search range for object detection in the camera video, the recognition-type object detection unit 111 can reduce unnecessary false positives. In addition, the recognition-type object detection unit 111 can speed up the object detection process.

[0149] Furthermore, the integrated tracking unit 200 performs object tracking using the likelihood of the target and / or the likelihood of the tracker, so that the object tracking device 10 can obtain a more reliable object tracking result. Furthermore, the object tracking device 10 performs an object search using the object tracking result obtained in this manner, so that the object tracking device 10 can further improve the accuracy of object tracking. As a result, the object tracking device 10 can improve the accuracy of object tracking overall.

[0150] <Second embodiment> Next, a second embodiment of the present invention will be described with reference to the drawings. For convenience of explanation, the same reference numerals are used for members having the same functions as those included in the drawings described in the first embodiment, and the explanations thereof will be omitted.

[0151] The object tracking system 2 according to this embodiment is configured to include an object tracking device 50 instead of the object tracking device 10 of the object tracking system 1 according to the first embodiment described with reference to Fig. 2. Other system configurations of the object tracking system 2 are similar to those of the object tracking system 1 shown in Fig. 2, and therefore will not be described.

[0152] (Object Tracking Device 50) The function of the object tracking device 50 will be described with reference to Fig. 9. Fig. 9 is a functional block diagram showing an example of the functional configuration of the object tracking device 50 according to the present embodiment. As shown in Fig. 9, the object tracking device 50 includes a plurality of detection units (100-1 to 100-N), an integrated tracking unit 200, and a display control unit 300. Note that in this embodiment, similar to the above-described first embodiment, when the plurality of object detection units (100-1 to 100-N) are not distinguished from one another or when they are referred to collectively, they are referred to as detection units 100.

[0153] The display control unit 300 controls the image (video) to be displayed on the display device 30. Specifically, the display control unit 300 generates display data by converting the tracking information output by the integrated tracking unit 200 into data that can be displayed on the display device 30, and transmits the display data to the display device 30. The tracking information output by the integrated tracking unit 200 is expressed in a common coordinate system. Therefore, the display control unit 300 generates display data that can be displayed on the display device 30 in the common coordinate system.

[0154] At this time, it is preferable that the integrated tracking unit 200 outputs information required for displaying the movement line of the object on the display device 30 (for example, information indicating the past positions of the object of the tracker) as tracking information to the display control unit 300. This tracking information may be the same as the information fed back to the detection unit 100, or it may be different.

[0155] The display device 30 then displays the received display data on the screen, thereby enabling the object tracking device 50 to present the tracking results to the user.

[0156] Furthermore, the detection unit 100 of the object tracking device 50 according to this embodiment generates display data by converting the search range set by the search range setting unit 112 of the object detection unit 110 into data that can be displayed on the display device 30, and transmits the display data to the display device 30. The search range information output by the search range setting unit 112 is expressed in an individual coordinate system. Therefore, the display control unit 300 generates display data that can be displayed on the display device 30 in the individual coordinate system, using the camera parameters of the camera 20 linked to the detection unit 100 that output the search range information.

[0157] The display device 30 then displays the received display data on the screen. This allows the object tracking device 50 to present the search range to the user using the object search range information output from each detection unit 100.

[0158] Note that there may be a plurality of display devices 30. For example, the display device 30 may receive display data displayed in a common coordinate system and display data displayed in an individual coordinate system at different display devices 30, and each display device 30 may display the received display data on its screen. The display device 30 may also be configured to divide a display area of ​​one screen and display a plurality of display data on the screen. In this way, the method of displaying display data in the display device 30 according to this embodiment is not particularly limited.

[0159] Moreover, the display control unit 300 may be configured to be provided in the detection unit 100. Fig. 10 is a functional block diagram showing an example of a functional configuration of the detection unit 100 of the object tracking device 50 according to the present embodiment. As shown in Fig. 10, the detection unit 100 includes an object detection unit 110, a common coordinate conversion unit 120, an individual coordinate conversion unit 130, and a display control unit 150. Similarly to the object detection unit 110 shown in Fig. 5, the object detection unit 110 includes a recognition-type object detection unit 111 and a search range setting unit 112.

[0160] 10 outputs search range information indicating the set search range to the display control unit 150. The display control unit 150 receives the search range information output from the search range setting unit 112, and generates display data by converting the search range information into data that can be displayed on the display device 30, similar to the display control unit 300. The search range information output by the search range setting unit 112 is expressed in an individual coordinate system. Therefore, the display control unit 150 generates display data that can be displayed on the display device 30 in the individual coordinate system, using the camera parameters of the camera 20 linked to the detection unit 100 that includes the display control unit 150.

[0161] Then, the display control unit 150 transmits the generated display data to the display device 30. Then, the display device 30 displays the received display data on the screen.

[0162] (Examples) Application examples of the object tracking device 50 according to the present embodiment will be described with reference to Figures 11 to 14. Figures 11 to 14 are diagrams for explaining application examples of the object tracking device 50 according to the present embodiment.

[0163] First, Fig. 11 is a diagram showing an example of a room in which shelves R1 and R2 and multiple cameras (A to F) are installed, viewed from the direction opposite to the direction of gravity. As shown in Fig. 11, the horizontal direction of Fig. 11 is the X-axis in a common coordinate system, and the vertical direction is the Y-axis. Shelves R1 and R2 are installed side by side on the X-axis so that their longitudinal directions are parallel to the Y-axis direction.

[0164] Camera A is installed in a position close to the entrance of this room. In this embodiment, it is assumed that the entire room is photographed by cameras A to F. That is, as shown in Fig. 11, the indoor space in which multiple cameras (A to F) are installed becomes the photographed space. Moreover, cameras A to F photograph a common location.

[0165] Fig. 12 is a diagram showing an example of images captured by each of camera A and camera B. The upper diagram in Fig. 12 shows a frame of an image captured by camera A, and the lower diagram shows a frame of an image captured by camera B. Coordinate values ​​in these frames are expressed by coordinate values ​​in individual coordinate systems for each camera.

[0166] Note that the object tracking device 50 in this embodiment may be configured to display the video captured by the camera 20 on the display device 30.

[0167] As shown in Fig. 12, the image captured by camera A includes person C1. In addition, since camera A is installed near the entrance, the image includes the entrance. In addition, the image captured by camera B includes person C1 and person C2.

[0168] When viewed from camera A, person C2 is hidden behind shelf R1. Therefore, at the time of the image of Fig. 12, person C2 is an object that is not visible to camera A. If these image frames are the first frames of the images captured by camera A and camera B, there is no tracking information in the previous frame, so object tracking device 50 detects objects from these frames and generates tracking information.

[0169] Then, using the tracking information expressed in the individual coordinate system, search range setting unit 112 sets a search range for the object for the next frame of the video captured by camera A. Similarly, using the tracking information expressed in the individual coordinate system, search range setting unit 112 sets a search range for the object for the next frame of the video captured by camera B.

[0170] Fig. 13 is a diagram showing an example of a search range displayed on display device 30. The upper diagram in Fig. 13 is a diagram showing an example of a search range for an object for a frame output from camera A, and the lower diagram is a diagram showing an example of a search range for an object for a frame output from camera B.

[0171] As shown in the upper diagram of FIG. 13, the search range setting unit 112 obtains a search range A1 from the position of the person C1 in the upper diagram of FIG. 12. The search range setting unit 112 also obtains an area of ​​an entrance / exit part to the room as a search range N1. The search range setting unit 112 also obtains the outer edge of the frame as search ranges N2 and N3. The search range setting unit 112 then outputs information summarizing the obtained search ranges A1, N1 to N3 as search range information to the recognition-type object detection unit 111 and the display control unit 150 or the display control unit 300. The display control unit 150 or the display control unit 300 then converts the search range indicated by the search range information on the display device 30 into display data that can be displayed on the screen, and transmits the converted data to the display device 30.

[0172] The display device 30, which has received the display data from the display control unit 150 or the display control unit 300, displays the search range on the screen as shown in the upper diagram of FIG.

[0173] Next, the lower diagram of FIG. 13 will be described. As shown in the lower diagram of FIG. 13, the search range setting unit 112 obtains search ranges B1 and B2 from the positions of the person C1 and the person C2 in the lower diagram of FIG. 12, respectively. The search range setting unit 112 also obtains the outer edge of the frame as search ranges N4 to N7. Then, the search range setting unit 112 outputs information summarizing the obtained search ranges B1, B2, and N4 to N7 as search range information to the recognition-type object detection unit 111 and the display control unit 150 or the display control unit 300. Then, the display control unit 150 or the display control unit 300 converts the search range indicated by this search range information on the display device 30 into display data that can be displayed on the screen, and transmits it to the display device 30.

[0174] The display device 30, which has received the display data from the display control unit 150 or the display control unit 300, displays the search range on the screen as shown in the lower diagram of FIG.

[0175] The display device 30 may display the search range in a different manner for each area. For example, the display device 30 may display the search range for an already detected object and the search range for the outer edge of the frame in different colors.

[0176] Then, the integrated tracking unit 200 generates trackers for the persons C1 and C2 detected in the subsequent frames. Then, the integrated tracking unit 200 outputs information required to display the respective movement lines of the persons C1 and C2 to the display control unit 300 as tracking information.

[0177] The display control unit 300 receives the tracking information from the integrated tracking unit 200 , converts the tracking information into display data that can be displayed on the display device 30 , and transmits the display data to the display device 30 .

[0178] Then, the display device 30 displays the display data received from the display control unit 300 on the screen. FIG. 14 is a diagram showing an example in which the display device 30 displays on a screen (display screen) the movement lines indicating the respective tracking results of the person C1 and the person C2. As shown in FIG. 14, in this application example, the display screen displays the tracking results of the objects on the XY plane in the common coordinate system. In FIG. 14, the movement line of the person C1 is displayed by a solid line, and the movement line of the person C2 is displayed by a dashed line. In this way, the display device 30 can display the tracking results of the objects on the screen.

[0179] <Third embodiment> Next, a third embodiment of the present invention will be described with reference to the drawings. For convenience of explanation, the same reference numerals are used for members having the same functions as those included in the drawings described in the first and second embodiments, and the explanations thereof will be omitted.

[0180] The object tracking device 10 according to this embodiment is configured to include a detection unit 400 instead of the detection unit 100 of the object tracking device 10 shown in Fig. 1. The configuration of this detection unit 400 will be described with reference to Fig. 15. Fig. 15 is a functional block diagram showing an example of the functional configuration of the detection unit 400 of the object tracking device 10 according to this embodiment.

[0181] The detection unit 400 includes an object detection unit 140 instead of the object detection unit 110 of the detection unit 100 shown in Fig. 4 and Fig. 5. The detection unit 400 further includes a storage unit 160. That is, the detection unit 400 according to the present embodiment includes the object detection unit 140, the common coordinate transformation unit 120, the individual coordinate transformation unit 130, and the storage unit 160, as shown in Fig. 15.

[0182] In this embodiment, a configuration in which the object detection unit 140 is provided instead of the object detection unit 110 of the detection unit 100 of the object tracking device 10 according to the first embodiment will be described as an example. Note that the present invention is not limited to this, and the object tracking device 50 according to the second embodiment may be configured to provide the object detection unit 140 instead of the object detection unit 110 of the detection unit 100. In other words, the detection unit 100 according to this embodiment may be configured to output data to be displayed to the display control unit 150 or the display control unit 300.

[0183] The storage unit 160 stores camera parameters for each camera 20, which are used when converting the coordinate system. Furthermore, the storage unit 160 stores information indicating the range of coordinate values ​​in the common coordinate system, which is used when the individual coordinate conversion unit 130 checks whether or not the coordinate values ​​in the common coordinate system included in the tracking information are included in the range captured by the associated camera 20. Furthermore, the storage unit 160 may store video captured by the camera 20. Note that this video may be stored temporarily.

[0184] 15, the storage unit 160 is described as being built into the detection unit 400, but the present invention is not limited to this. The storage unit 160 may be provided in the object tracking device 10 separately from the detection unit 400. The storage unit 160 may be realized by a storage device or the like separate from the object tracking device 10.

[0185] Next, a detailed functional configuration of the object detection unit 140 of the detection unit 400 will be described with reference to Fig. 16. Fig. 16 is a functional block diagram showing an example of the functional configuration of the object detection unit 140 of the detection unit 400 according to this embodiment. As shown in Fig. 16, the object detection unit 140 includes a recognition-type object detection unit (first object detection means) 141, a non-recognition-type object detection unit (second object detection means) 142, a detection parameter update unit 143, and a detection result integration unit 144.

[0186] In this embodiment, object detection using a dictionary (classifier) ​​or the like is referred to as “recognition-based object detection.” On the other hand, object detection without using a classifier or the like is referred to as “non-recognition-based object detection.”

[0187] The recognition-type object detection unit 141 detects an object from a camera image input to the recognition-type object detection unit 141. The recognition-type object detection unit 141 detects an object from the entire frame. Note that the recognition-type object detection unit 141 may detect an object based on search range information output from a detection parameter update unit 143 described later, as indicated by a dashed line in Fig. 16. At this time, the recognition-type object detection unit 141 detects an object in the same manner as the recognition-type object detection unit 111 described in the first embodiment.

[0188] Furthermore, when search range information is not output from the detection parameter update unit 143, the recognition-type object detection unit 141 may perform object detection using another criterion instead of performing object detection for the entire frame. For example, the recognition-type object detection unit 141 may use silhouette information to perform object detection only for an area where a silhouette is present and its surrounding area.

[0189] The recognition-type object detection unit 141 outputs the detection result of the object detection to the detection result integration unit 144 as a first detection result.

[0190] In addition, the recognition object detection unit 141 may extract the appearance features of the object at this point in preparation for object detection by the non-recognition object detection unit 142 described later. The appearance features of the object include information such as the color, pattern, and shape of the object, but the present invention is not limited thereto. The recognition object detection unit 141 extracts these feature amounts as the appearance features of the object. At this time, the area used in the object detection by the recognition object detection unit 141 and the area used in the object detection by the non-recognition object detection unit 142 may not be the same. For example, when the object is a person, the object detection by the recognition object detection unit 141 detects the head, and the object detection by the non-recognition object detection unit 142 detects up to the clothing area. At this time, the recognition object detection unit 141 extracts the feature amounts of the appearance features of the object so as to include the clothing area. Then, the recognition object detection unit 141 may output the extracted feature amounts as template information together with information indicating the area used for extraction (extraction area information). Furthermore, the recognition-type object detection unit 141 may store the feature amount itself inside the recognition-type object detection unit 141 and output only information for identifying the feature amount.

[0191] The detection parameter update unit 143 receives the tracking information expressed in the individual coordinate system from the individual coordinate transformation unit 130. Then, the detection parameter update unit 143 uses this tracking information to obtain parameters (called detection parameters) necessary for object detection. These detection parameters are parameters required for object detection processing. The detection parameters include, for example, a predicted position (prediction area) of the object in the current frame, a search range to which object detection is applied, a size of the template used in template matching, and a feature amount (template information) of the template of a target previously associated with a tracker. Note that the detection parameters do not need to include all of this information, and only need to include parameters necessary for object detection by the non-recognition type object detection unit 142. The detection parameters may also include information of a target associated with a tracker in the object tracking result in the previous frame.

[0192] For example, for a target (object) detected in the previous frame and associated with a tracker, the detection parameter update unit 143 determines the position where the object exists in the current frame as a predicted position based on tracking information of the object. This prediction process is similar to the prediction process of the predicted position in the search range setting unit 112 according to the first embodiment. The detection parameter update unit 143 may determine an area including this predicted position as a predicted area.

[0193] Also, for example, the detection parameter update unit 143 may determine a range in which object detection by template matching is applied, centered on the prediction region, and include this range as a search range for objects included in the detection parameters.

[0194] The detection parameter update unit 143 obtains the above detection parameters for each object included in the tracking information. Then, the detection parameter update unit 143 updates the obtained detection parameters as detection parameters to be used in the object detection process. Then, the detection parameter update unit 143 outputs the detection parameters to the non-recognized object detection unit 142.

[0195] The detection parameter update unit 143 may obtain the search range of the object by using the tracking information converted into coordinate values ​​in the individual coordinate system, similarly to the search range setting unit 112 of the detection unit 100 according to the above-described first embodiment. Then, the detection parameter update unit 143 may output search range information indicating the obtained search range of the object to the recognition-type object detection unit 141.

[0196] The non-recognizable object detection unit 142 receives the detection parameters from the detection parameter update unit 143. Then, based on the received detection parameters, the non-recognizable object detection unit 142 detects an object from the camera video input to the non-recognizable object detection unit 142. Unlike the recognition object detection unit 141, this non-recognizable object detection unit 142 detects an object based on the similarity of the appearance of an object detected in a previous frame.

[0197] That is, when an object is detected in the previous frame, the non-recognition type object detection unit 142 stores the image features of that area (or a partial image of the detected area itself) as a template.The non-recognition type object detection unit 142 then performs object detection by checking whether an area similar to the stored template exists in the current frame by template matching.The image features used in this case may be, for example, information on color patterns and distribution, distribution information on edges and brightness gradients, or features obtained by combining these.

[0198] The detection parameters used when detecting an object in the non-recognition type object detection unit 142 are controlled by the detection parameters output from the detection parameter update unit 143. Specifically, the non-recognition type object detection unit 142 performs object detection by template matching on the predicted position (prediction area) of the object predicted by the detection parameter update unit 143 and its vicinity. That is, the non-recognition type object detection unit 142 sets a search range for template matching centered on the predicted object existence range (prediction area) and performs template matching on the periphery. In addition, at this time, the non-recognition type object detection unit 142 may also take into consideration that the apparent size of the object changes due to the movement of the object's position. This change can be calculated by using the camera parameters. Therefore, the non-recognition type object detection unit 142 may calculate the change in the size of the object and reflect it in the template before performing template matching.

[0199] Furthermore, the information on the template for which the non-recognition-type object detection unit 142 performs template matching may be a feature amount extracted by the recognition-type object detection unit 141 in the object detection process in the previous frame.

[0200] In this way, the non-recognition type object detection unit 142 performs object detection based on the tracking results for the objects tracked by the integrated tracking unit 200, thereby improving the detection accuracy compared to when the tracking results are not used.

[0201] Then, the non-recognition-type object detection unit 142 outputs the detection result of the object detection to the detection result integration unit 144 as a second detection result.

[0202] The detection result integration unit 144 receives a first detection result from the recognition-type object detection unit 141. The detection result integration unit 144 also receives a second detection result from the non-recognition-type object detection unit 142. Then, the detection result integration unit 144 integrates the first detection result and the second detection result. Then, the detection result integration unit 144 outputs the integrated result to the common coordinate transformation unit 120 as the detection result of the object detection in the object detection unit 140.

[0203] There may be an object that is included in both the first detection result and the second detection result, and an object that is included in only one of them. Therefore, the detection result integration unit 144 associates and integrates the objects included in the first detection result and the second detection result. For example, the degree of overlap of the object regions may be used for this association.

[0204] That is, the detection result integration unit 144 calculates the overlap ratio between object regions (for example, the overlap ratio of object circumscribing rectangles), and if this is greater than a predetermined value, matches the object included in the first detection result with the object included in the second detection result.

[0205] Furthermore, the detection result integration unit 144 may formulate the problem as a graph problem with the overlap ratio of the areas between the objects as a weight, and perform correspondence between the objects. For example, the detection result integration unit 144 converts the overlap ratio into a cost using a monotonically non-increasing function, and then performs correspondence between the objects by calculating the optimal correspondence using the Hungarian method or the like.

[0206] The detection result integration unit 144 may merge at this point those results for which the value used in the matching (e.g., overlap ratio or cost) is greater than a predetermined value. Alternatively, the detection result integration unit 144 may not merge at this point, but may generate information indicating the matching. The detection result integration unit 144 may then output the detection result obtained by combining the first detection result and the second detection result with the information indicating the matching as the detection result of the object detection unit 140, and may perform tracking using the information regarding the matching during integrated tracking.

[0207] Furthermore, the non-recognition type object detection unit 142 outputs a second detection result based on the object tracking result for the previous frame. Therefore, the second detection result may be generated later than the first detection result. In such a case, the detection result integration unit 144 temporarily stores the first detection result in a storage means such as a buffer in the detection result integration unit 144 or in the storage unit 160. Then, the detection result integration unit 144 may integrate both results at the time of receiving the second detection result for the frame corresponding to the frame for which the first detection result is generated.

[0208] As described above, the detection unit 400 of the object tracking device 10 according to this embodiment outputs, as a detection result, a result obtained by integrating the result of object detection by the recognition-type object detection unit 141 (first detection result) and the result of object detection by the non-recognition-type object detection unit 142 (second detection result). At this time, the non-recognition-type object detection unit 142 detects an object by performing template matching based on the tracking result for the object tracked by the integrated tracking unit 200. This allows the detection unit 400 to further improve the accuracy of object detection compared to a case where only object detection by identifying an object (recognition-type object detection) is performed.

[0209] Therefore, the object tracking device 10 can track the object with higher accuracy.

[0210] <Fourth embodiment> Next, a fourth embodiment of the present invention will be described with reference to the drawings. For convenience of explanation, the same reference numerals are used for members having the same functions as those included in the drawings described in the above-mentioned embodiments, and the explanations thereof will be omitted.

[0211] The object tracking device 10 according to the present embodiment is configured to include an integrated tracking unit 500 instead of the integrated tracking unit 200 of the object tracking device 10 shown in FIG. 1. The configuration of the integrated tracking unit 500 will be described with reference to FIG. 17. FIG. 17 is a functional block diagram showing an example of the functional configuration of the integrated tracking unit 500 of the object tracking device 10 according to the present embodiment. As shown in FIG. 17, the integrated tracking unit 500 includes a buffer unit 510, a prediction unit 210, a storage unit 220, an association unit 530, and an update unit 240.

[0212] In this embodiment, an example of a configuration in which an integrated tracking unit 500 is provided instead of the integrated tracking unit 200 of the object tracking device 10 according to the first embodiment will be described. Note that the present invention is not limited to this, and the object tracking device 50 according to the second embodiment may be provided with the integrated tracking unit 500 instead of the integrated tracking unit 200. In other words, the integrated tracking unit 500 according to this embodiment may be configured to output data to be displayed to the display control unit 300.

[0213] Moreover, the detection unit that outputs the detection result to the integrated tracking unit 500 according to this embodiment may be the detection unit 400 described in the third embodiment.

[0214] The buffer unit 510 is a means for temporarily storing the detection results expressed in a common coordinate system output from the detection unit 100. Among the data (detection results) buffered in the buffer unit 510, data in which the time information included in the camera video in which the detection was performed is within a predetermined period is acquired by the association unit 530. This predetermined period is a periodic period. The association unit 530 performs object tracking using one or more detection results acquired in a certain period. In this way, the integrated tracking unit 500 in this embodiment is also called a collective tracking unit because it performs object tracking using images from multiple cameras among the images from each camera 20.

[0215] The object tracking (also called collective integrated tracking) performed by the integrated tracking unit 500 will be described with reference to FIG. 18. FIG. 18 is a diagram for explaining the collective tracking process of objects performed by the integrated tracking unit 500 according to this embodiment. As in FIG. 7, FIG. 18 shows an example of the timing of acquiring images by each of the cameras A, B, and C when the number of cameras is three. In FIG. 18, the horizontal axis indicates the time axis, and the further to the right, the later in time it is. As shown in FIG. 18, the camera A acquires images at times t1, t5, and t8. Similarly, the camera B acquires images at times t2, t4, t6, and t9, and the camera C acquires images at times t3 and t7.

[0216] Then, the images acquired at each of these timings are subjected to object detection in order. For convenience of explanation, the following description will be given assuming that the times shown in FIG. 18 are substantially the same as the times when the object detection results are output. In other words, time t1 is the time when the detection results for the frames of the video captured by camera A are output from detection unit 100 and buffered in buffer unit 510.

[0217] Moreover, the time axis at the bottom of FIG. 18 indicates an example of a periodic period.

[0218] The association unit 530 acquires, from among one or more detection results buffered in the buffer unit 510, a detection result in which the time information included in the camera video that was the subject of object detection is within a predetermined period. As described above, this predetermined period is a periodic period. In addition, in this embodiment, the buffered time and the camera video time are considered to be the same. Therefore, it can be said that the association unit 530 acquires one or more detection results buffered in the buffer unit 510 at a predetermined period.

[0219] Specifically, the association unit 530 first acquires the detection result buffered in the first period T1. That is, the association unit 530 acquires the detection results buffered at times t1, t2, and t3. The detection result buffered at time t1 is the detection result for the frame of the video captured by camera A. The detection result buffered at time t2 is the detection result for the frame of the video captured by camera B, and the detection result buffered at time t3 is the detection result for the frame of the video captured by camera C. Therefore, the association unit 530 acquires from the buffer unit 510 a plurality of detection results for the frames of the video captured by each of the plurality of cameras 20, which are the detection results buffered in the buffer unit 510 within a predetermined period (T1 in this case). Then, the association unit 530 performs object tracking using the acquired detection results.

[0220] Similarly, during periods T2, T3, and T4, the association unit 530 acquires the detection results buffered within these periodic periods and performs object tracking.

[0221] In this embodiment, a configuration is described in which association unit 530 acquires data (multiple detection results) buffered in buffer unit 510 from buffer unit 510 at a predetermined cycle, but association unit 530 may be configured to receive this data from buffer unit 510 at a predetermined cycle. In other words, buffer unit 510 may have a function of outputting this data to association unit 530 at a predetermined cycle.

[0222] Returning to FIG. 17, the association unit 530 of the integrated tracking unit 500 will be further described.

[0223] The matching unit 530 obtains the distance between the targets from the positions of the targets included in each of the obtained detection results, and matches targets that are close to each other. At this time, the matching unit 530 performs matching by a method such as the Hungarian method using the distance between the targets. The matching unit 530 may also use the similarity of the appearance features of the targets at the same time, in addition to the distance between the targets. For example, targets that are close to each other and have similar colors are likely to be the same object. Therefore, the matching unit 530 may perform matching using such features. Note that the features for determining the similarity of the appearance features are not limited to colors, and may be, for example, the pattern of the targets.

[0224] Then, the matching unit 530 integrates the detection results for the targets that are matched with each other. That is, after matching the targets, the matching unit 530 uses the detection results of each of the matched targets to determine the position of the object. At this time, the matching unit 530 may evaluate the likelihood and / or accuracy of the predicted position of each target, and the position with the maximum accuracy may be determined as the position of the object.

[0225] Furthermore, the association unit 530 may weight the position of each target based on the accuracy of the predicted position determined by the angle (depression angle or elevation angle) of the camera 20 with respect to each target and the distance from the camera 20 to the target, etc. Then, the association unit 530 may calculate statistics such as an average value from the weighted positions, and may regard the position indicated by the calculated statistics as the position of the object.

[0226] Then, the matching unit 530 sets the obtained object position as the target position for the period in which the detection result was obtained. The matching unit 530 performs matching using this target position in the same manner as the matching unit 230 according to the first embodiment. The matching process and subsequent processes by the integrated tracking unit 500 are similar to the processes by the integrated tracking unit 200 described in the first embodiment, and therefore will not be described here.

[0227] Furthermore, the association unit 530 may associate each target with a tracker and integrate the targets before associating the targets with each other. That is, when there are multiple targets associated with the same tracker, the association unit 530 performs a merging process between these targets. In this case, the association unit 530 may prioritize merging targets with higher likelihood and / or higher accuracy of predicted position. In this way, the association unit 530 may evaluate the detection results based on these pieces of information and integrate targets associated with the same tracker.

[0228] As described above, the object tracking device 10 according to the present embodiment performs object tracking using all of the detection results for the camera images captured by each camera 20 within a predetermined period of time. This allows the object tracking device 10 to prioritize object search results among the multiple cameras 20, making it easier to apply the process of tracking objects.

[0229] Also, for example, when the frame rates of all the cameras 20 are stable and the same, by setting a predetermined period according to the frame interval, the frames of all the cameras 20 are included in this predetermined period. Therefore, the object tracking device 10 according to this embodiment can perform object tracking for frames for all the cameras 20. This allows the object tracking device 10 to simultaneously evaluate detection results for objects that are simultaneously visible from the multiple cameras 20, so that the reliability of the detection results can be directly reflected in tracking.

[0230] <Fifth embodiment> A fifth embodiment of the present invention will be described below. In this embodiment, a minimum configuration for solving the problems of the present invention will be described.

[0231] The object tracking device 10 according to this embodiment has a similar configuration to the object tracking device 10 shown in FIG. 1 and described in the first embodiment, and therefore will be described with reference to FIG.

[0232] 1, the object tracking device 10 according to this embodiment includes a plurality of detection units (100-1 to 100-N) (N is a natural number) and an integrated tracking unit 200. In this embodiment, when the plurality of detection units (100-1 to 100-N) are not distinguished from one another or are referred to collectively, they are referred to as detection unit 100.

[0233] Each of the multiple detection units 100 detects an object from output information output from a sensor. Note that in FIG. 1, the sensor is a camera and the output information of the sensor is described as a camera image (image data), but the sensor is not limited to a camera. Specifically, the detection unit 100 detects an object based on tracking information output from the integrated tracking unit 200. The detection unit 100 outputs the detection result to the integrated tracking unit 200.

[0234] The integrated tracking unit 200 tracks one or more objects indicated by the detection results based on the detection results output by each of the multiple detection units (100-1 to 100-N). The integrated tracking unit 200 then generates tracking information of the object expressed in a common coordinate system as the tracking result. The integrated tracking unit 200 then outputs the tracking information to each of the multiple detection units (100-1 to 100-N).

[0235] In this manner, the detection unit 100 of the object tracking device 10 according to this embodiment detects an object from the output information output from the sensor, based on the tracking result for the object tracked by the integrated tracking unit 200 .

[0236] In this way, the object tracking device 10 detects an object from a video using the tracking result for the previous frame, and therefore can improve the object detection accuracy compared to when the tracking result is not used. Also, since the object tracking device 10 performs object tracking using all of the object detection results for the video captured by each of the cameras 20, the tracking accuracy of the object tracking device 10 can be improved compared to when object tracking is performed independently for each camera. Also, since the object tracking device 10 performs object tracking using all of the detection results with high detection accuracy, it can track objects with higher accuracy.

[0237] In the above-described embodiments, the object tracking device 10 includes the detection unit (100, 400) and the integrated tracking unit (200, 500), but the detection unit and the integrated tracking unit may be realized by separate devices. In other words, the detection unit (100, 400) may be realized as an object detection device, and the integrated tracking unit (200, 500) may be realized as an integrated tracking device, each of which is a separate device. The display control unit 300 may also be realized by a display control device separate from the object tracking device 50. This display control device may be built into the display device 30. Sixth embodiment A sixth embodiment of the present invention will be described. The object tracking device 10 according to this embodiment has the same configuration as the object tracking device 10 shown in FIG. 1 described in the first embodiment, and therefore will be described with reference to FIG. 1. The object tracking device 10 according to this embodiment has a configuration in which the functions described below are further added to the object tracking device 10 according to the first embodiment, but the present invention is not limited to this. The object tracking device 10 according to this embodiment can also be applied to the object tracking devices according to the second to fifth embodiments described above.

[0238] In this embodiment, the integrated tracking unit 200 further acquires information on the appearance of the object and includes the acquired information in the tracking information. Then, the detection unit 100 controls the detection of the object using the information on the appearance of each object included in the tracking information output from the integrated tracking unit 200.

[0239] This information about how the object appears (hereinafter, "appearance information") is information about how each object appears when viewed from the position of each camera 20, and is determined by the position of the object on each tracker.

[0240] For example, consider a case where an object is in front of (on the side of camera 20) another object when the object and another object are viewed from a camera 20. In this case, the object behind (the other object) is likely to be hidden by the object in front (the object), making it invisible to the camera 20. In this case, the integrated tracking unit 200 includes information indicating such overlapping of objects as visibility information in the tracker (tracking result) related to the other object, and outputs the tracking result.

[0241] Next, a specific operation of each part of the object tracking device 10 according to the present embodiment will be described.

[0242] The integrated tracking unit 200 stores information about the arrangement of the cameras 20, for example, in the storage unit 220 shown in Fig. 6. The information about the arrangement of the cameras 20 is, for example, information indicating where the cameras 20 are arranged, and in what direction they are photographing. The integrated tracking unit 200 may also include, as the information about the arrangement of the cameras 20, information about the position and direction of the lighting in the shooting space, information about the characteristics of the lighting, and information about the lighting conditions, such as which positions in the shooting space are bright or dark. The integrated tracking unit 200 may also hold directional information about the shooting space as the information about the arrangement of the cameras 20.

[0243] Similar to the integrated tracking unit 200 in each of the above-mentioned embodiments, the integrated tracking unit 200 predicts the movement of the object represented by each tracker in the current frame and determines the position of the tracker by matching the target with the tracker.

[0244] Then, the joint tracking unit 200 predicts the position of the object indicated by each tracker on the captured image captured by each camera 20, based on the determined tracker position and information on the movement of each tracker.

[0245] Then, the integrated tracking unit 200 refers to information about the arrangement of the cameras 20, and predicts how each object at the predicted position will look when viewed from each camera 20. In other words, the integrated tracking unit 200 predicts whether or not objects indicated by the trackers of each of the multiple cameras 20 will overlap with each other at the next timing of image capture.

[0246] Then, when the integrated tracking unit 200 determines, based on the predicted appearance, that objects overlap in an image captured by a certain camera 20, it generates information (appearance information) indicating that the objects may not appear to overlap.

[0247] For example, the integrated tracking unit 200 determines whether the predicted object is between a line segment connecting a camera 20 and a predicted object. If the predicted object is between the line segment, the object is highly likely to overlap with the other object. Therefore, the integrated tracking unit 200 obtains information on the hidden (overlapping) object and the degree of overlap (likelihood) as visibility information for the object.

[0248] The integrated tracking unit 200 may then include this determination result as visibility information in the tracking result for this object.

[0249] Then, the integrated tracking unit 200 includes the generated visibility information in tracking information of an object that is likely to become invisible in an image captured by a certain camera 20. At this time, the visibility information preferably includes information indicating the camera 20 that is likely to make the object invisible.

[0250] Then, the integrated tracking unit 200 outputs the tracking information including the appearance information to each detection unit 100. The integrated tracking unit 200 may output the tracking information including the appearance information to a detection unit 100 associated with a camera 20 (a certain camera 20) where the object is likely to be overlapped and not visible. The integrated tracking unit 200 may output the tracking information not including the appearance information to an object tracking device 10 associated with another camera 20.

[0251] In addition, if it is known that the way the lighting hits an object changes depending on its position, causing the color and brightness of the object to change, the integrated tracking unit 200 may include information in the tracking information that describes the change in appearance of the object depending on its position.

[0252] For example, if the way the light hits the object is determined by the position of the object, the integrated tracking unit 200 may predict the way the light hits the object from the position and include information such as whether the light will get brighter, darker, or the color will change in the tracking information for each tracker.

[0253] Furthermore, when the position of the light in the space in which the object is placed is known, the integrated tracking unit 200 determines whether or not the shadow of the object or another object placed in this environment overlaps with another object, based on the relationship between the light and the object. If there is a possibility of the shadows overlapping, the integrated tracking unit 200 calculates the possibility (likelihood) of the shadows overlapping for the object with which the shadows overlap, and includes this in the tracking information.

[0254] Even in the case of a moving object like the sun, the integrated tracking unit 200 may obtain the position of the sun from the time and information on the direction of the site, predict the direction in which a shadow will be cast, and consider the impact on the appearance of the object. For example, the integrated tracking unit 200 obtains the current position of the sun from time information, and predicts in which direction a shadow will be cast based on the information on the direction. Then, when there is a possibility that the shadow of another object will be cast, the integrated tracking unit 200 may calculate the possibility (likelihood) that the shadows will overlap, and include this in the tracking information.

[0255] Next, the operation of the detection unit 100 will be described. The detection unit 100 detects an object based on the tracking information, as in each of the above-mentioned embodiments. At this time, the detection unit 100 of the object tracking device 10 according to this embodiment controls the detection of the object using information on the visibility of each object contained in the tracking information. Specifically, the detection unit 100 does not detect an object that is likely to be invisible because it overlaps with another object. For example, the detection unit 100 does not set a search range for this object that is likely to be invisible.

[0256] If the tracking information includes information that changes in brightness or color, the detection unit 100 may perform detection by correcting the effect of the lighting when searching. For example, in a dark area, the detection unit 100 may perform detection after correcting the pixel values ​​of the area to be brighter.

[0257] Furthermore, if there is a change in color tone, the detection unit 100 may take the change in color tone into consideration when updating the matching parameters (i.e., the detection parameters described above) used in template matching and correct the color information of the parameters. Furthermore, if there is a large change in color tone, the detection unit 100 may not use the color information. Furthermore, the detection unit 100 may perform matching by lowering the weight of color information among the template features and increasing the weight of other features such as edges.

[0258] Furthermore, when the detection section 100 updates the detection parameters used in the object detection process, if it is determined that there is a high possibility that the objects are overlapping, the detection parameters may not be updated.

[0259] As described above, the detection unit 100 controls object detection based on the visibility information included in the tracking information. This allows the detection unit 100 to reduce the process of detecting invisible objects and the process of updating parameters such as template matching. This allows the detection unit 100 to reduce the possibility of erroneous detection and erroneous parameter updates.

[0260] Similarly, when it is highly likely that the lighting conditions have changed, the detection unit 100 may perform control to correct the effect of the change and update the parameters, or may perform control not to update the parameters.

[0261] In addition, when a plurality of detection algorithms can be switched, the object tracking device 10 may use a detector that is more resistant to overlapping as the detection unit 100. Specifically, the object tracking device 10 may use a detector that normally detects the entire head, but when there is overlapping, a detector that detects only a part of the head, not the entire head, may be used. This allows the object tracking device 10 to use a simple detector under normal circumstances, and to use a more detailed detector when there is a possibility of overlapping. This allows the object tracking device 10 to perform highly accurate detection while maintaining efficiency. Similarly, when the lighting conditions change, the object tracking device 10 may control detection using a detector (feature amount) that is highly robust against the lighting conditions.

[0262] As described above, in the object tracking device 10 according to this embodiment, the integrated tracking unit 200 determines how an object appears using information from other cameras as well. Therefore, the integrated tracking unit 200 can determine how an object appears with high accuracy even when it is difficult to determine the appearance using only a certain camera 20, such as when objects are captured overlapping each other from the certain camera 20. The integrated tracking unit 200 can then feed back this result to the detection unit 100. This allows the detection unit 100 to reduce erroneous detection of objects. The integrated tracking unit 200 then uses this detection result to track objects, thereby improving the tracking accuracy of the object.

[0263] <Hardware configuration example> Here, an example of a hardware configuration capable of realizing the object tracking device (10, 50) according to each of the above-mentioned embodiments will be described. The above-mentioned object tracking device (10, 50) may be realized as a dedicated device, or may be realized using a computer (information processing device).

[0264] FIG. 19 is a diagram illustrating an example of a hardware configuration of a computer (information processing device) capable of implementing each embodiment of the present invention.

[0265] The hardware of the information processing device (computer) 700 shown in FIG. 19 includes a CPU (Central Processing Unit) 11, a communication interface (I / F) 12, an input / output user interface 13, a ROM (Read Only Memory) 14, a RAM (Random Access Memory) 15, a storage device 17, and a drive device 18 for a computer-readable storage medium 19, which are connected via a bus 16. The input / output user interface 13 is a man-machine interface such as a keyboard as an example of an input device and a display as an output device. The communication interface 12 is a general communication means for the devices according to the above-mentioned embodiments (FIGS. 1 and 9) to communicate with an external device via a communication network 600. In this hardware configuration, the CPU 11 controls the overall operation of the information processing device 700 that realizes the object tracking device (10, 50) according to each embodiment.

[0266] The present invention described using the above-mentioned embodiments as examples is achieved, for example, by supplying a program (computer program) capable of realizing the processes described in the above-mentioned embodiments to an information processing device 700 shown in Fig. 19, and then reading and executing the program in a CPU 11. Note that such a program may be, for example, a program capable of realizing various processes described in the flowchart (Fig. 8) referred to in the description of the above-mentioned embodiments, or each unit (each block) shown in the device in the block diagrams shown in Figs. 1, 4 to 6, 9, and 15 to 17.

[0267] The programs supplied to the information processing device 700 may be stored in a readable / writable temporary storage memory (15) or a non-volatile storage device (17) such as a hard disk drive. That is, in the storage device 17, the program group 17A is, for example, a program capable of realizing the functions of each unit shown in the object tracking device (10, 50) in each of the above-mentioned embodiments. In addition, the various stored information 17B is, for example, the object tracking results, camera images, camera parameters, and the range of coordinate values ​​of a common coordinate system visible by each camera 20 in each of the above-mentioned embodiments. However, when implementing the programs in the information processing device 700, the constituent units of the individual program modules are not limited to the division of each block shown in the block diagrams (FIGS. 1, 4 to 6, 9, 15 to 17), and may be appropriately selected by a person skilled in the art when implementing the programs.

[0268] In the above case, the method of supplying the program to the device can be a currently common method, such as installing the program in the device via a computer-readable recording medium (19) such as a CD (Compact Disk)-ROM or a flash memory, or downloading the program from an external source via a communication line (600) such as the Internet. In such a case, the present invention can be considered to be constituted by the code (program group 17A) constituting the computer program or the storage medium (19) in which the code is stored.

[0269] The present invention has been described above as an example of application to the exemplary embodiments described above. However, the technical scope of the present invention is not limited to the scope described in each of the above-mentioned embodiments. It is clear to those skilled in the art that various modifications or improvements can be made to the embodiments. In such cases, new embodiments with such modifications or improvements may also be included in the technical scope of the present invention. This is clear from the matters described in the claims. [Explanation of symbols]

[0270] 1. Object Tracking System 2. Object Tracking System 10 Object Tracking Device 100 Detector 110 Object detection unit 111 Recognition-based object detection unit 112 Search range setting section 120 Common coordinate conversion unit 130 Individual coordinate conversion section 140 Object Detection Unit 141 Recognition-based object detection unit 142 Non-cognitive object detection unit 143 Detection parameter update unit 144 Detection result integration unit 150 Display control unit 160 Storage section 200 Integrated Tracking Unit 210 Prediction Department 220 Storage section 230 Mapping section 240 Update Department 300 Display control unit 400 Detector 500 Integrated Tracking Unit 510 Buffer section 530 Mapping section 20 Camera 30 Display device 40 Network 50 Object Tracking Device

Claims

1. A process of acquiring a first feature representing an object appearing in a first image based on a first image captured by a first camera; A process of acquiring a second feature representing the object shown in a second image based on a second image captured by a second camera; obtaining a third feature representative of the object based on the first feature and the second feature; displaying, on a display device, the third feature in a three-dimensional space representing a space photographed by the first camera and the second camera; An information processing device that performs the above.

2. The third feature represents a position of the object. The information processing device according to claim 1 .

3. The position of the object is a position in the three-dimensional space. The information processing device according to claim 2 .

4. The third characteristic represents a size of the object. The information processing device according to claim 1 .

5. The process of acquiring the first feature and the process of acquiring the second feature include a process of detecting an object from an image using a neural network. The information processing device according to claim 1 .

6. A neural network for detecting the object is associated with each camera. The information processing device according to claim 5 .

7. The process of displaying the third characteristic includes a process of displaying the three-dimensional space from a viewpoint looking down from above. The information processing device according to claim 1 .

8. The object includes a moving object. The information processing device according to claim 1 .

9. Moving objects include people. The information processing device according to claim 7.

10. acquiring a first feature representing an object appearing in a first image based on the first image captured by a first camera; acquiring a second feature representing the object appearing in a second image based on the second image captured by a second camera; obtaining a third feature representative of the object based on the first feature and the second feature; displaying, on a display device, the third feature in a three-dimensional space representing a space photographed by the first camera and the second camera; Information processing methods.

11. On the computer, A process of acquiring a first feature representing an object appearing in a first image based on a first image captured by a first camera; A process of acquiring a second feature representing the object shown in a second image based on a second image captured by a second camera; obtaining a third feature representative of the object based on the first feature and the second feature; displaying, on a display device, the third feature in a three-dimensional space representing a space photographed by the first camera and the second camera; An information processing program that executes the above.

12. A process of acquiring a first feature representing an object appearing in a first image based on a first image captured by a first camera; A process of acquiring a second feature representing the object shown in a second image based on a second image captured by a second camera; obtaining a third feature representative of the object based on the first feature and the second feature; displaying, on a display device, the third feature in a three-dimensional space representing a space photographed by the first camera and the second camera; An information processing system that performs the above.

Images