Information processing device, information processing method, information processing program, and information processing system

Abstract

To provide a technique of tracking an object more precisely.SOLUTION: The position of an object in a space is predicted in a case where an object is tracked by using an image acquired from a camera, and first positional information of the object is acquired on the basis of the predicted position of the object, and second positional information of the object is determined on the basis of the first positional information and the image acquired by the camera.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an object tracking device, an object tracking system, an object tracking method, a display control device, an object detection device, a program, and a recording medium. [Background technology]

[0002] In recent years, systems have been developed that use multiple cameras and the like to track people. For example, a moving object tracking system described in Patent Document 1 uses multiple in-camera tracking means that track people in a distributed manner for each camera. The system tracks moving objects by linking the multiple in-camera tracking means. Furthermore, Patent Document 2 describes a method for tracking the same object captured by multiple image capture units based on the tracking results of each individual object.

[0003] As a related technique, Patent Document 3 describes a method for early excluding an object that does not need to be tracked from the tracking target.

[0004] Furthermore, Patent Document 4 describes an apparatus for detecting a moving object in an image captured by one imaging means. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2004-72628 [Patent Document 2] Special Publication No. 2009-510541 [Patent Document 3] International Publication No. 2013 / 012091 [Patent Document 4] Japanese Patent Application Laid-Open No. 2006-202047 Summary of the Invention [Problem to be solved by the invention]

[0006] However, with the technology described in Patent Document 1 or 2, for example, if a camera is located far away from an object, the tracking accuracy of the object (moving body) in the image captured by this camera may be low. In this case, the technology described in Patent Document 1 or 2 may be affected by the tracking accuracy of the camera, and may not be able to integrate the tracking results of the object, or even if integration is possible, the detection accuracy of the object's position required during integration may be low.

[0007] The present invention has been made in view of the above-mentioned problems, and has an object to provide a technology that can track an object with higher accuracy. [Means for solving the problem]

[0008] An object tracking device according to one aspect of the present invention comprises a plurality of detection means that detect an object from output information from sensors and output the detection results, and an integrated tracking means that tracks the object based on the plurality of detection results output by each of the plurality of detection means and generates tracking information for the object expressed in a common coordinate system, wherein the integrated tracking means outputs the generated tracking information to each of the plurality of detection means, and the detection means detects the object based on the tracking information.

[0009] An object tracking system according to one aspect of the present invention comprises a sensor and an object tracking device that receives output information consisting of information acquired by the sensor, the object tracking device comprising a plurality of detection means that detect the object from the output information and output the detection results, and an integrated tracking means that tracks the object based on the plurality of detection results output by each of the plurality of detection means and generates tracking information for the object expressed in a common coordinate system, the integrated tracking means outputting the generated tracking information to each of the plurality of detection means, and the detection means detects the object based on the tracking information.

[0010] An object tracking method according to one aspect of the present invention detects an object from output information of a sensor, outputs the detection results, tracks the object based on the output detection results, generates tracking information for the object expressed in a common coordinate system, and outputs the generated tracking information, and detects the object based on the tracking information.

[0011] A display control device according to one aspect of the present invention is a display control device that displays display data on a display device, wherein the display data indicates a search range for searching for an object based on object tracking information from among the output information of a sensor, the search range being expressed in an individual coordinate system specific to the sensor that outputs the output information, and the object tracking information is information indicating the results of tracking the object based on the detection results of the object detected within the search range in the output information of each of a plurality of sensors.

[0012] An object detection device according to one aspect of the present invention detects an object from sensor output information based on tracking information expressed in a common coordinate system, the tracking information indicating the tracking results of an object tracked based on multiple detection results output from each of multiple object detection devices.

[0013] Note that the scope of the present invention also includes a computer program for implementing the above-described devices, object tracking system, or object tracking method by a computer, and a computer-readable storage medium on which the computer program is stored. [Effects of the Invention]

[0014] According to the present invention, it is possible to track an object with higher accuracy. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a functional block diagram showing an example of a functional configuration of an object tracking device according to a first embodiment of the present invention. [Figure 2] 1 is a diagram showing an example of a schematic overall configuration of an object tracking system according to a first embodiment of the present invention. [Figure 3] FIG. 10 is a diagram for explaining a process of associating a target with a tracker. [Figure 4] 2 is a functional block diagram showing an example of the functional configuration of a detection unit of the object tracking device according to the first embodiment of the present invention. FIG. [Figure 5] 2 is a functional block diagram showing an example of the functional configuration of an object detection unit in a detection unit of the object tracking device according to the first embodiment of the present invention. FIG. [Figure 6] 2 is a functional block diagram showing an example of the functional configuration of an integrated tracking unit of the object tracking device according to the first embodiment of the present invention. FIG. [Figure 7] 3A to 3C are diagrams for explaining the sequential tracking process of an object performed by an integrated tracking unit according to the first embodiment of the present invention; [Figure 8] 5 is a flowchart showing an example of the flow of an object tracking process of the object tracking device according to the first embodiment of the present invention. [Figure 9] FIG. 10 is a functional block diagram showing an example of a functional configuration of an object tracking device according to a second embodiment of the present invention. [Figure 10] FIG. 10 is a functional block diagram showing an example of the functional configuration of a detection unit of an object tracking device according to a second embodiment of the present invention. [Figure 11] FIG. 10 is a diagram for explaining an application example of an object tracking device according to a second embodiment of the present invention. [Figure 12] FIG. 10 is a diagram for explaining an application example of an object tracking device according to a second embodiment of the present invention. [Figure 13] FIG. 10 is a diagram for explaining an application example of an object tracking device according to a second embodiment of the present invention. [Figure 14] FIG. 10 is a diagram for explaining an application example of an object tracking device according to a second embodiment of the present invention. [Figure 15] FIG. 11 is a functional block diagram showing an example of the functional configuration of a detection unit of an object tracking device according to a third embodiment of the present invention. [Figure 16] FIG. 11 is a functional block diagram showing an example of the functional configuration of an object detection unit in a detection unit of an object tracking device according to a third embodiment of the present invention. [Figure 17] FIG. 10 is a functional block diagram showing an example of the functional configuration of an integrated tracking unit of an object tracking device according to a fourth embodiment of the present invention. [Figure 18] 13A and 13B are diagrams for explaining a collective tracking process of objects performed by an integrated tracking unit according to a fourth embodiment of the present invention. [Figure 19] FIG. 1 is a diagram illustrating an example of a hardware configuration of a computer (information processing device) capable of implementing each embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0016] First Embodiment A first embodiment of the present invention will be described in detail with reference to the drawings. First, the overall configuration of an object tracking system (also simply referred to as a system) of the present invention will be described with reference to FIG. 2. FIG. 2 is a diagram showing an example of a schematic overall configuration of an object tracking system 1 according to this embodiment. As shown in FIG. 2, the object tracking system 1 according to this embodiment includes an object tracking device 10, a plurality of cameras (20-1 to 20-N (N is a natural number)), and one or more display devices 30. In this embodiment, when the plurality of cameras (20-1 to 20-N) are not distinguished from one another or when they are referred to collectively, they are referred to as cameras 20.

[0017] The object tracking device 10, the camera 20, and the display device 30 are connected to each other so that they can communicate with each other via a network 40. The display device 30 does not have to be included in the object tracking system 1. The display device 30 may also be configured to be directly connected to the object tracking device 10 without going through the network 40.

[0018] The camera 20 functions as a sensor that detects an object. In this embodiment, an example will be described in which the camera 20 is used as a sensor that detects an object, but the present invention is not limited to this. The sensor is not limited to a camera, and any sensor capable of positioning, such as a radio wave sensor, may be used. A combination of multiple sensors, such as a sensor that combines a radio wave sensor and a camera, may also be used. By using the camera 20 as a sensor, the object tracking device 10 can more suitably acquire visual information such as color.

[0019] In addition, in this embodiment, the information acquired by the sensor is explained as being an image captured by a camera, but if the sensor is a radio wave sensor, the information acquired by the sensor is radio waves acquired by the radio wave sensor.

[0020] The object tracking device 10 is a device that tracks an object included in images captured by each of a plurality of cameras 20. The functional configuration of the object tracking device 10 will be described with reference to another drawing.

[0021] The display device 30 displays the results of object tracking by the object tracking device 10. The display device 30 may also display images captured by the camera 20. The display device 30 may also display other information such as flow line information.

[0022] (Object Tracking Device 10) Next, the functions of the object tracking device 10 will be described. Fig. 1 is a functional block diagram showing an example of the functional configuration of the object tracking device 10 according to this embodiment. As shown in Fig. 1, the object tracking device 10 includes a plurality of detection units (100-1 to 100-N) and an integrated tracking unit 200. In this embodiment, when the plurality of detection units (100-1 to 100-N) are not distinguished from one another or when they are referred to collectively, they are referred to as detection units 100.

[0023] (Detection unit 100) The detection unit 100 detects an object from the output information of the camera 20 based on tracking information of an object in a frame preceding a frame in which the object is to be detected, which is tracking information output from the integrated tracking unit 200 (described later). Here, in this embodiment, the output information of the camera 20 refers to video data showing the video captured by the camera 20.

[0024] In this embodiment, it is assumed that the multiple detection units 100 and the multiple cameras 20 are associated one-to-one. For example, detection unit 100-1 detects an object from a video captured by camera 20-1, and detection unit 100-2 detects an object from a video captured by camera 20-2. Note that this embodiment is not limited to this, and for example, detection unit 100-1 may detect an object from a video captured by camera 20-N.

[0025] Furthermore, there does not have to be a one-to-one correspondence between the cameras 20 and the detection units 100. For example, the detection unit 100-1 may detect an object from the video images captured by each of the multiple cameras 20.

[0026] The operation of the detection unit 100 will be described below. The detection unit 100 receives from the camera 20 video data (hereinafter referred to as camera video) showing video captured by the camera 20. In FIG. 1, video data showing video captured by the camera 20-n (n is 1 to N) is described as camera video (n). Here, the camera video may be video captured by a camera 20 such as a surveillance camera and acquired in real time, or video captured by the camera 20 that is temporarily stored in a storage unit (not shown) and then decoded (or played back) later. This video data includes time information showing the time of capture.

[0027] The detection unit 100 also receives tracking information of an object in a previous frame from the integrated tracking unit 200. The previous frame may be the frame immediately before the frame (current frame) in which an object is to be detected, or may be a frame a predetermined number of frames before the current frame. The previous frame may be one or multiple. When the detection unit 100 detects an object in the first frame, there is no tracking information of the object in the previous frame, so the detection unit 100 does not receive (does not use) the tracking information of the object in the previous frame.

[0028] The detection unit 100 detects objects (called object detection) from the received camera video using the camera video and object tracking information from the previous frame. As described above, when performing object detection on the first frame, the detection unit 100 performs the detection without using the object tracking information from the previous frame. Hereinafter, the detected object will be called a target. That is, the object detection result (also called an object detection result, or simply a detection result) will be a set of targets.

[0029] The object detection result includes, for each target, information indicating the position of the target, information indicating the size of the target, etc. Specifically, the object detection result includes, for example, information on the circumscribing rectangle of the area occupied by the target (target area) in the frame of the video in which the object is detected, the coordinate values of the center of gravity of the target area, information indicating the width of the target, information indicating the height of the target, etc. Note that the object detection result is not limited to this. For example, the object detection result may include the coordinate values of the top and bottom of the target area instead of or in addition to the coordinate values of the center of gravity of the target area. It is sufficient that the object detection result includes information indicating the position, size, etc. of the target for each target.

[0030] In this embodiment, the object detection result will be described as including, for each target, the coordinate value of the bottom edge of the target and information indicating the circumscribing rectangle of the target. The coordinate value of the bottom edge of the target indicates the coordinate value of the point where the object contacts the floor (ground) and / or the coordinate value of the midpoint of the bottom side of the circumscribing rectangle of the object. Furthermore, if the object is a person, the coordinate value of the bottom edge of the target may be the coordinate value of the feet.

[0031] Then, the detection unit 100 converts the coordinate values included in the object detection result into coordinate values in a common coordinate system defined within the space (shooting space) photographed by the multiple cameras 20, and sets the converted coordinate values as the object detection result.

[0032] In addition to the above-mentioned information, the object detection result may also include information representing the shape of the target. That is, the object detection result may also include silhouette information representing the target area. Here, silhouette information is information that distinguishes between pixels inside the target area and pixels outside the target area, and may be, for example, image information in which the pixel values inside are set to 255 and the pixel values outside are set to 0, or a value extracted from the silhouette shape of a shape descriptor (shape feature) such as that standardized by MPEG-7. The object detection result may also include features of the object's appearance. For example, the object detection result may also include features such as the color, pattern, and shape of the object.

[0033] Furthermore, the object detection result may include, for each target, information describing likelihood (target likelihood information) that indicates the certainty (accuracy) of object detection. The target likelihood information is information necessary for calculating the likelihood of the target, and is information related to the accuracy of object detection, such as the score value at the time of object detection, the distance from the camera of the detected object, and the size of the detected object. Alternatively, the detection unit 100 may calculate the likelihood of the target itself and use the calculated likelihood as the target likelihood information.

[0034] Then, the detection unit 100 outputs the object detection result to the joint tracking unit 200.

[0035] (Integrated Tracking Unit 200) The integrated tracking unit 200 receives detection results output from each of the detection units 100. The integrated tracking unit 200 then tracks objects based on these detection results. Specifically, the integrated tracking unit 200 tracks one or more objects (performs object tracking) using object detection results for the objects detected by each of the detection units 100 from video captured by the camera 20 linked to the detection unit 100. The integrated tracking unit 200 then generates object tracking results (object tracking results) expressed in a common coordinate system. In this way, the integrated tracking unit 200 integrates the object detection results detected by each of the detection units 100 from video captured by the camera 20 linked to the detection unit 100, and performs object tracking. For this reason, the object tracking performed by the integrated tracking unit 200 is also referred to as object integrated tracking.

[0036] Hereinafter, the information for each object generated as an object tracking result will be referred to as a tracker. In other words, the tracker includes, as information about the tracked object (object tracking result), information indicating the position of the tracked object, a motion model of the object, etc., but the present invention is not limited to this. Note that the position of the tracked object is the position of the object before (in the past) the current time, and therefore is also referred to as the past position of the object.

[0037] In other words, object tracking can be considered as a process of associating objects between frames by associating targets detected by object detection with trackers generated before the detection of the targets. This object tracking will be described with reference to FIG. 3. FIG. 3 is a diagram for explaining the process of associating targets with trackers by the integrated tracking unit 200. As shown in FIG. 3, the number of targets is M and the number of trackers is K (M and K are integers equal to or greater than 0). The integrated tracking unit 200 associates these M targets with the K trackers. When associating targets with trackers, the integrated tracking unit 200 first predicts the current position of the object from the object's past position indicated by information included in the tracker, and then associates the targets with the trackers using an index indicating the relationship between the targets and trackers.

[0038] That is, the integrated tracking unit 200 predicts the position of the object in the current frame based on the position of the object detected in the previous frame and the motion model of the object calculated and stored for each tracker. This can be done using various existing methods, such as a method using a Kalman filter or a method using a particle filter.

[0039] Then, the integrated tracking unit 200 associates the object tracking result (tracker) in the previous frame with the object (target) included in the detection result, based on, for example, the following information (1) to (3). (1) The proximity of the object's position in the current frame predicted by the tracker to the target position. (2) Similarity of appearance features between the target and the object tracked by the tracker (3) The likelihood of the target and the tracker This matching process can be reduced to a cost minimization problem for a bipartite graph as shown in Fig. 3. Therefore, the integrated tracking unit 200 can solve this problem using an algorithm such as the Hungarian method.

[0040] In Figure 3, arrows are used to indicate when targets and trackers are associated with each other, i.e., the top target is associated with the top tracker.

[0041] If there is a target that does not correspond to a tracker, the integrated tracking unit 200 determines whether the target can be considered a newly appeared object. If the integrated tracking unit 200 determines that there is a high possibility that the target has newly appeared, it adds a new tracker related to the target. In FIG. 3, the target indicated by the symbol m (referred to as target m) is assumed to be a target that does not correspond to a tracker. In this case, the integrated tracking unit 200 determines whether target m can be considered a newly appeared object, and if so, creates a new tracker related to target m.

[0042] On the other hand, if there is a tracker that does not correspond to the target, the integrated tracking unit 200 determines whether the tracker is information about an object that has disappeared from the imaging space. Then, if there is a high possibility that the tracker is information about an object that has disappeared from the imaging space, the integrated tracking unit 200 deletes the tracker. In FIG. 3, it is assumed that the tracker indicated by the symbol k (referred to as tracker k) is a tracker that does not correspond to the target. In this case, the integrated tracking unit 200 determines whether tracker k is information about the disappeared object, and deletes tracker k if tracker k is information about the disappeared object.

[0043] The integrated tracking unit 200 performs object tracking by repeating these processes on a frame-by-frame basis. The integrated tracking unit 200 assigns each tracker a unique ID (identifier) that is common to all cameras 20, and manages the tracking results (trackers) using this ID. The integrated tracking unit 200 also includes in the tracker a value that evaluates the likelihood of the tracking result (hereinafter referred to as the tracker likelihood (weight)) as a tracker parameter. The position indicated by the information indicating the position of the tracked object, which is included in the tracker, and the most recent position of the object is called the tracker position. The size of the object at this time is also called the tracker size.

[0044] Furthermore, the integrated tracking unit 200 updates information indicating the position of each tracker, information on the likelihood of the tracker, etc., based on the result of the association. The tracker position information is information expressed in a common coordinate system defined within the shooting space captured by the multiple cameras 20. The information expressed in this common coordinate system is a coordinate value in the common coordinate system. For example, if the multiple cameras 20 are cameras installed in a store, the coordinate value in this common coordinate system is a coordinate system indicating the position of a floor in the real world. In contrast, the coordinate system unique to each camera 20 is called the individual coordinate system of that camera 20. This individual coordinate system is a coordinate system on the image captured by the camera 20. Hereinafter, the position information expressed in the common coordinate system will be described as coordinate values in the common coordinate system. Furthermore, the position information expressed in the individual coordinate system of the camera 20 will be described as coordinate values in the individual coordinate system of the camera 20.

[0045] The integrated tracking unit 200 then generates a tracker whose information has been updated based on the association result as a new object tracking result.The integrated tracking unit 200 then outputs, from the generated object tracking result (tracker), information indicating the position and / or size of the tracker, information on the likelihood of the tracker, etc., as information indicating the tracking result of object tracking (tracking information).

[0046] This tracking information includes coordinate values of each tracker in a common coordinate system as information indicating the position of each tracker. This tracking information is fed back to the detection unit 100. That is, the detection unit 100 receives this tracking information and uses it for object detection in subsequent frames.

[0047] The integrated tracking unit 200 may be configured to output the object tracking result including the tracking information and other information of the tracker to the detection unit 100.

[0048] In this way, the object tracking device 10 according to this embodiment performs object tracking by using images captured by each of the multiple cameras and integrating the object detection results for the images captured by each of the multiple cameras 20. The object tracking device 10 then feeds back the obtained tracking information for object detection in the next frame.

[0049] In this way, the object tracking device 10 detects an object from a video using the tracking result for the previous frame. For example, if there is an object that is invisible from one camera 20 but visible from another camera 20, the object tracking device 10 uses the tracking result for the object that is invisible from the one camera 20 to detect the object in the video of the one camera 20. This allows the detection unit 100 to suitably detect the object when the same object appears within the range visible from the one camera 20. Therefore, the object tracking device 10 can accurately track the object.

[0050] Therefore, object tracking device 10 can improve the object detection accuracy compared to when not using the tracking results for the previous frame. Also, since object tracking device 10 tracks objects using all of the detection results with high detection accuracy, the accuracy of the tracking results obtained as a whole is improved.

[0051] (Details of the detection unit 100) Next, the function of each unit of object tracking device 10 will be described in more detail with reference to Fig. 4 to Fig. 8. Fig. 4 is a functional block diagram showing an example of a more detailed functional configuration of detection unit 100 of object tracking device 10 according to this embodiment. As shown in Fig. 4, detection unit 100 includes object detection unit 110, common coordinate conversion unit (second conversion unit) 120, and individual coordinate conversion unit (first conversion unit) 130. Note that in Fig. 4, camera video (n) (n is 1 to N) received by detection unit 100 is simply referred to as camera video.

[0052] The individual coordinate conversion unit 130 receives tracking information output from the integrated tracking unit 200 from the integrated tracking unit 200. Then, the individual coordinate conversion unit 130 converts the coordinate values of the common coordinate system of each tracker included in the tracking information into coordinate values on the frame captured by each camera 20 (i.e., coordinate values expressed in the individual coordinate system specific to each camera 20). When the coordinate values of the common coordinate system are (X, Y, Z) and the coordinate values of the individual coordinate system of the camera 20 are expressed as (x, y), the individual coordinate conversion unit 130 calculates the coordinate values (x, y) of the individual coordinate system of the camera 20 linked to the detection unit 100 including the individual coordinate conversion unit 130 from the coordinate values (X, Y, Z) of the common coordinate system of the tracker. At this time, it is preferable that the individual coordinate conversion unit 130 calculates at least camera parameters representing the camera position, attitude, etc. of the camera 20 linked to the detection unit 100 by calibration. As a result, the individual coordinate transformation unit 130 transforms the coordinate values of the universal coordinate system into coordinate values of the individual coordinate system of the camera 20 using the obtained camera parameters.

[0053] The camera parameters may be stored in a storage unit (not shown) within the detection unit 100, or may be stored in a storage area within the individual coordinate transformation unit 130. In the latter case, the individual coordinate transformation unit 130 may be configured to supply the camera parameters to the common coordinate transformation unit 120.

[0054] For example, suppose the object is a person, and the information indicating the position of the tracker is coordinate values indicating the position of the person's feet and the position of the top of the head. The coordinate values of the foot position and the coordinate values of the top of the head are (X0, Y0, 0) and (X0, Y0, H) (H represents the height of the person), respectively. Also, suppose the camera 20 linked to the detection unit 100 equipped with the individual coordinate transformation unit 130 is camera 20-1.

[0055] At this time, the individual coordinate transformation unit 130 uses the camera parameters for camera 20-1 to determine the foot position (x0, y0) and the head position (x1, y1) on the frame captured by camera 20-1. If the information indicating the tracker position includes information indicating a circumscribing rectangle, the value indicating the width of the circumscribing rectangle has previously been determined by converting the coordinate values into coordinate values in a universal coordinate system using the camera parameters. Therefore, the individual coordinate transformation unit 130 may use a value converted again using the camera parameters as the width of the circumscribing rectangle.

[0056] Furthermore, not all of the objects indicated by the tracking results of a tracker are visible from a single camera 20, and may exist outside the field of view of that camera 20. Therefore, if the information indicating the tracker's position included in the tracking information is information about an object that is invisible to the camera 20 linked to the detection unit 100, the individual coordinate conversion unit 130 cannot determine the coordinate values of the object in the individual coordinate system of the camera 20. Therefore, the individual coordinate conversion unit 130 may not convert the coordinate values of the tracker's common coordinate system for such an object that is invisible outside the angle of view of the camera 20 into coordinate values in the individual coordinate system. In this case, the individual coordinate conversion unit 130 may pre-register the range of coordinate values in the common coordinate system visible to each camera 20 in a storage unit (not shown) for each camera 20 and determine whether each object is within this range. Furthermore, the individual coordinate conversion unit 130 may actually convert the coordinate values into coordinate values in the individual coordinate system, and if an abnormal value indicating outside the area monitored by the linked camera 20 is obtained or no value can be determined, determine that the object whose coordinate values have been converted is an object that is invisible to the camera 20.

[0057] Then, the individual coordinate conversion unit 130 converts the coordinate values in the common coordinate system of the tracker included in the tracking information output from the integrated tracking unit 200 into coordinate values in the individual coordinate system of the camera 20 linked to the detection unit 100, and outputs the result (tracking information) to the object detection unit 110. In other words, the individual coordinate conversion unit 130 converts the tracking information expressed in the common coordinate system into tracking information expressed in the individual coordinate system, and outputs the converted tracking information to the object detection unit 110. Hereinafter, when simply written as "coordinate values in the individual coordinate system," it refers to the coordinate values in the individual coordinate system of the camera 20 linked to the detection unit 100.

[0058] The object detection unit 110 receives camera images from a camera 20 linked to the detection unit 100 that includes the object detection unit 110. The object detection unit 110 also receives tracking information converted into coordinate values in an individual coordinate system from the individual coordinate conversion unit 130. Then, the object detection unit 110 detects an object from the received camera images based on the tracking information.

[0059] Then, object detection unit 110 generates a detection result and outputs the generated detection result to common coordinate transformation unit 120. Note that this detection result is expressed in a separate coordinate system.

[0060] The configuration of object detection unit 110 will be described in more detail with reference to Fig. 5. Fig. 5 is a functional block diagram showing an example of the functional configuration of object detection unit 110 according to this embodiment. As shown in Fig. 5, object detection unit 110 includes a recognition-type object detection unit (first object detection unit) 111 and a search range setting unit 112.

[0061] The search range setting unit 112 receives the tracking information converted into coordinate values in the individual coordinate system from the individual coordinate conversion unit 130. Then, using this tracking information converted into coordinate values in the individual coordinate system, the search range setting unit 112 determines an area (search range) for detecting an object in the current frame. That is, the search range setting unit 112 predicts the position of the object in the current frame based on the tracking information consisting of the tracking result of the previous frame converted into coordinate values in the individual coordinate system. Then, the search range setting unit 112 determines a detection range in which to search for the object from the predicted position. This search range is also called the object detection range.

[0062] Here, the tracking information received by the search range setting unit 112 is the result of object tracking in a past frame (also referred to as a past tracking result) when viewed from the time of the frame currently being processed. Therefore, the search range setting unit 112 predicts the movement of each object and predicts the position of each object in the current frame. Hereinafter, this predicted object position will be referred to as a predicted position. Then, the search range setting unit 112 sets the vicinity of this predicted position as the search range for the object.

[0063] It is preferable that the search range setting unit 112 predicts the movement of each object using a motion model for each object calculated from past tracking results. For example, if the position of the object has not changed in the tracking results of the past few frames (or even two frames), the search range setting unit 112 determines that the object is stationary and sets the position of the object obtained in the tracking results as the predicted position. Furthermore, if the tracking results of the past few frames show that the object is moving, the search range setting unit 112 may assume that the object is moving at a constant speed and calculate the predicted position taking into account the time difference from the past frame.

[0064] The tracking information may include the tracking results of the past few frames that are used by the search range setting unit 112 when predicting the movement of each object. Also, the tracking information may include a motion model of the object obtained from the tracking results of the past few frames.

[0065] This predicted position may also be included in the tracking information. That is, the integrated tracking unit 200 may include a value obtained by a Kalman filter or a particle filter during object tracking as a predicted position in the tracking information.

[0066] Furthermore, for example, at the outer edge of the range that camera 20 can capture, there is a possibility that a new object will appear in the angle of view of camera 20. Furthermore, if the location captured by camera 20 includes a place such as an entrance or exit, there is also a possibility that a new object will appear in the angle of view of camera 20. Therefore, it is preferable that search range setting unit 112 also include these areas (the outer edge of the frame and / or the entrance or exit part) on the frame included in the video captured by camera 20 in the object search range.

[0067] The search range setting unit 112 outputs information indicating the set search range for the object (search range information) to the recognition-type object detection unit 111.

[0068] The recognition-type object detection unit 111 receives search range information from the search range setting unit 112. Based on the received search range information, the recognition-type object detection unit 111 detects objects from camera images input to the recognition-type object detection unit 111. The recognition-type object detection unit 111 temporarily stores frames of the input camera images in a storage means such as a buffer within the recognition-type object detection unit 111, and upon receiving search range information, applies this information to perform object detection processing. Specifically, the recognition-type object detection unit 111 performs object detection for the area (search range) indicated by the search range information using a classifier that has learned image features of objects.

[0069] For example, if the object is a person, the recognition-type object detection unit 111 applies a classifier that has been trained on characteristic parts of a person (e.g., the head or upper body) to detect the person. Alternatively, the recognition-type object detection unit 111 may use a classifier that has been trained on the entire person as the classifier. The recognition-type object detection unit 111 can use various classifiers as this classifier. For example, the recognition-type object detection unit 111 can use a classifier obtained by training images of the head, upper body, and entire body of a person using a convolutional neural network (CNN). Alternatively, the recognition-type object detection unit 111 may perform feature extraction using a histogram of Gaussian (HOG) or the like, and use a classifier such as a support vector machine (SVM) or generalized learning vector quantization (GLVQ). In addition to the above, the recognition-type object detection unit 111 can use various existing recognition-based detection methods.

[0070] In this way, the recognition-type object detection unit 111 according to this embodiment detects an object within the search range set by the search range setting unit 112. That is, the recognition-type object detection unit 111 performs object detection within the search range narrowed by the search range setting unit 112 using the tracking result in the previous frame. Therefore, the recognition-type object detection unit 111 does not perform object detection in a range in a frame where the object is unlikely to exist, thereby reducing unnecessary false detections. Furthermore, the recognition-type object detection unit 111 can speed up the object detection process.

[0071] Furthermore, the search range in which the recognition-type object detection unit 111 performs object detection may include not only the area indicated by the search range information but also an area determined by silhouette information obtained by background subtraction, etc. Furthermore, the recognition-type object detection unit 111 may set the intersection of the area determined by silhouette information and the area specified by the object search range information as the area in which object detection is performed (search range).

[0072] Then, the recognition-type object detection unit 111 generates a result of the object detection (detection result) and outputs the detection result to the common coordinate transformation unit 120. At this time, the coordinate values of the object included in the detection result are coordinate values in the individual coordinate system.

[0073] Returning to FIG. 4 , the function of the common coordinate transformation unit 120 of the detection unit 100 will be described. The common coordinate transformation unit 120 receives object detection results expressed in individual coordinate systems from the object detection unit 110. The individual coordinate transformation unit 130 then transforms the coordinate values of the individual coordinate systems included in the received detection results into coordinate values of a common coordinate system. This allows the common coordinate transformation unit 120 to generate information for integrating the detected positions of objects for each of the cameras 20.

[0074] Specifically, the common coordinate transformation unit 120 transforms coordinate values in the individual coordinate system included in the object detection result into coordinate values in the common coordinate system using camera parameters of the camera 20 linked to the detection unit 100 including the common coordinate transformation unit 120. For example, when the coordinates of the bottom end of an object in a frame captured by the camera 20 are (x0, y0), the common coordinate transformation unit 120 transforms this into coordinates (X0, Y0, 0) in the common coordinate system. Here, the ground is considered to be a plane where Z = 0, so the component in the Z-axis direction is 0. Furthermore, when the coordinates of the top end of the object are (x1, y1) (here, it is assumed that the top end of the object is directly above (vertically above) the bottom end of the object), the height of the object is defined as H. In this case, the common coordinate transformation unit 120 transforms the coordinates (x1, y1) of the top end of this object into coordinates (X0, Y0, H) in the common coordinate system. The common coordinate transformation unit 120 finds H that satisfies this to determine the height of the object. In this way, the common coordinate transformation unit 120 obtains the coordinate values (X, Y, Z) in the common coordinate system for each detected object.

[0075] If H is known, the common coordinate transformation unit 120 may use the known value as is.

[0076] The common coordinate transformation unit 120 outputs the detection result, including the coordinate values after coordinate transformation (coordinate values in the common coordinate system), to the integrated tracking unit 200. In other words, the common coordinate transformation unit 120 outputs the detection result expressed in the common coordinate system to the integrated tracking unit 200. Note that the common coordinate transformation unit 120 may include, for each of one or more targets (objects) included in the detection result, information about the object, such as silhouette information and feature quantities of the object's appearance characteristics (color, pattern, shape, etc.). The common coordinate transformation unit 120 may then output the detection result, including this information, to the integrated tracking unit 200.

[0077] (Details of the integrated tracking unit 200) Next, the functional configuration of the integrated tracking unit 200 will be described in more detail with reference to Fig. 6. Fig. 6 is a functional block diagram showing an example of a more detailed functional configuration of the integrated tracking unit 200 of the object tracking device 10 according to this embodiment. As shown in Fig. 6, the integrated tracking unit 200 includes a prediction unit 210, a storage unit 220, an association unit 230, and an update unit 240. Note that the integrated tracking unit 200 in this embodiment is also called a sequential tracking unit because it sequentially tracks the images from each camera 20 on a camera-by-camera basis.

[0078] The sequential object tracking (also referred to as sequential object tracking or sequential integrated tracking) performed by the integrated tracking unit 200 will be described with reference to FIG. 7. FIG. 7 is a diagram for explaining the sequential object tracking process performed by the integrated tracking unit 200 according to this embodiment. FIG. 7 shows an example of the timing at which images are acquired by each of camera A, camera B, and camera C when there are three cameras. In FIG. 7, the horizontal axis represents the time axis, with the further to the right, the later the time. As shown in FIG. 7, camera A acquires images at times t1, t5, and t8. Similarly, camera B acquires images at times t2, t4, t6, and t9, and camera C acquires images at times t3 and t7.

[0079] 7, the times (timestamps) of frames (images) acquired by each camera 20 are not necessarily the same for all cameras 20, and are usually different. Furthermore, the frame intervals may also differ for each camera 20, and may also be uneven even for the same camera 20.

[0080] The detection unit 100 then performs detection in chronological order from the camera images output asynchronously between the cameras 20 , and outputs the detection results to the integrated tracking unit 200 .

[0081] The integrated tracking unit 200 according to this embodiment performs sequential tracking processing in chronological order, starting with the image captured at the earliest possible time. In other words, in the case of FIG. 7 , the integrated tracking unit 200 first performs object tracking by integrating the object detection results across multiple cameras using the object detection results for the image captured by camera A at time t1. After this, the integrated tracking unit 200 then performs object tracking by integrating the object detection results across multiple cameras using the object detection results for camera B at time t2, camera C at time t3, camera B at time t4, and so on, in that order. Since not all objects are visible from all cameras 20, the integrated tracking unit 200 performs object tracking for objects that are likely to be visible from each camera 20.

[0082] Returning to FIG. 6, each part of the integrated tracking unit 200 will be described.

[0083] The memory unit 220 stores tracker information associated with objects (targets) included in the detection results received by the integrated tracking unit 200. The tracker information stored in the memory unit 220 is managed by the update unit 240 using the tracker ID. The tracker information includes, for example, information about objects whose tracking results are included in the tracker, parameters including the likelihood of the tracker, etc., but the present invention is not limited to this. The information about the object includes information indicating the past positions of the object, a motion model of the object, etc., but the present invention is not limited to this. The information about the object may also include information included in the object detection results described above.

[0084] 6, the storage unit 220 is described as being built into the integrated tracking unit 200, but the present invention is not limited to this. The storage unit 220 may be provided in the object tracking device 10 separately from the integrated tracking unit 200. The storage unit 220 may also be realized by a storage device or the like separate from the object tracking device 10.

[0085] When the storage unit 220 is not built into the integrated tracking unit 200, the storage unit 220 may be configured to store data used in the object tracking device 10. For example, the storage unit 220 may store camera images captured by the cameras 20, camera parameters of each camera 20, the range of coordinate values of a common coordinate system visible by each camera 20, and the like.

[0086] The prediction unit 210 predicts the position of an object in the current frame by referring to the storage unit 220. Specifically, the prediction unit 210 predicts the current position of the object based on a motion model of the object, using the tracking result (tracker) of the object in the previous frame. Here, information indicating the position of the object is expressed in a universal coordinate system.

[0087] Furthermore, the motion model of the object used by the prediction unit 210 to predict the position may be one stored in the storage unit 220, or may be one calculated by the prediction unit 210 using the tracking results before predicting the position of the object.

[0088] For example, prediction processing such as a Kalman filter or a particle filter can be applied to predict the position of an object by prediction unit 210. Alternatively, prediction unit 210 may simply calculate the velocity of the object from the tracking results of the past few times, assume uniform linear motion, predict the amount of movement from the position in the previous frame from the velocity, and add this to the position in the previous frame to predict the current position.

[0089] Then, the prediction unit 210 outputs the prediction result to the association unit 230.

[0090] The associating unit 230 receives the detection results output from each of the detecting units 100. In FIG. 6, detection result (n) (n is 1 to N) indicates the detection result output from the detecting unit 100-n. The associating unit 230 also receives the prediction result from the predicting unit 210. Then, the associating unit 230 refers to the storage unit 220 and uses the prediction result to associate the target included in the detection result with the tracker.

[0091] The association unit 230 determines the combination that will result in the highest overall association accuracy. The likelihood that a target m and a tracker k will be associated is calculated by multiplying the likelihoods Pm and ηk of target m and tracker k, respectively, by the likelihood qkm that indicates the possibility that the two are the same object. Therefore, the association unit 230 calculates this value for each pair of target and tracker, and determines the combination that results in the highest overall association accuracy.

[0092] Here, the target likelihood (first likelihood) is a value that indicates the probability (accuracy) of object detection. The accuracy of object detection depends on the size of the object to be detected (called the detected object) on the screen (on the frame), the distance from the camera 20 to the detected position of the object, how the object appears from the camera 20, etc.

[0093] For example, if the detected object is small and its size is close to the limit of what can be detected, the accuracy of object detection will be low. Furthermore, if the size of the detected object deviates from the apparent size of the object assumed by the camera parameters, the accuracy of object detection will be low. Furthermore, if the detected position of the object is far from the camera 20 or if the lighting conditions for the area where the object exists are poor, making it difficult to detect the object, the accuracy of object detection will be low. Furthermore, if the data used to train the classifier differs significantly from the actual appearance (for example, if the angle is different), the accuracy of object detection will be low.

[0094] The association unit 230 calculates the likelihood of a target by reflecting such characteristics. Specifically, the association unit 230 calculates the likelihood of a target by using likelihood information of the target included in the detection result received from the detection unit 100. Note that, if the detection unit 100 calculates the likelihood of a target and includes the calculated likelihood as likelihood information of the target in the detection result, the association unit 230 may use the likelihood included in the likelihood information of the target as is. Note that the likelihood of a target does not need to reflect all of the items described here, and may reflect only major factors.

[0095] The likelihood (second likelihood) of a tracker is a value that represents the reliability (accuracy) of object tracking. The accuracy of object tracking varies depending on the tracking result of object tracking in the previous frame. For example, a tracker that is reliably associated with a target in the tracking results up to the previous (past) frame before the current frame can be said to have a high accuracy of object tracking, while a tracker that is not closely associated can be said to have a low accuracy of object tracking. Therefore, the association unit 230 can change the likelihood based on the result of whether or not the target and tracker are associated in each frame, increasing the likelihood of the tracker if there is association, and decreasing the likelihood of the tracker if there is no association.

[0096] Furthermore, in this case, if the tracker is located far from the camera 20, it is considered that the error in the tracker's position will be large. As a result, it will be difficult to associate such a tracker with an object (target) included in the detection result. For this reason, the association unit 230 may change the rate at which the likelihood of the tracker is changed depending on the distance between the tracker's position and the camera 20. Furthermore, the association unit 230 may change the rate at which the likelihood of the tracker is changed depending on the angle (depression angle or elevation angle) formed between the horizontal plane including the camera 20 and the line of sight when the camera 20 sees the object for which the tracking result is indicated by the tracker.

[0097] For example, if an object whose tracking result is indicated by a tracker (hereinafter referred to as the tracker's object) is close to the camera 20 and the depression angle of the camera 20 with respect to the object is greater than a predetermined angle, the accuracy of the object's position is high. Therefore, the tracker and the target are easily associated. Therefore, in such a case, the association unit 230 increases the rate at which the tracker's likelihood is changed.

[0098] Furthermore, for example, if the tracker's object is far from the camera 20 and the depression angle of the camera 20 relative to the object is shallower than a predetermined angle, the size of the object in the frame captured by the camera 20 will be small. Also, a slight positional deviation in the image will result in a large deviation in real space. Therefore, the accuracy of the detected position of the object is likely to be low. Therefore, in such a case, the association unit 230 reduces the rate at which the tracker's likelihood is changed. In this way, the association unit 230 calculates the tracker's likelihood.

[0099] As described above, the association unit 230 can preferentially reflect the detection results detected by the camera 20 closest to the tracker's object in the tracker's likelihood, thereby improving the overall tracking accuracy. Note that it is not necessary to reflect all of the items described here in the tracker's likelihood, and it is also possible to reflect only the main factors.

[0100] Furthermore, the likelihood qkm, which indicates the identity between target m and tracker k, represents the degree of certainty that they are the same. When the object indicated by the target and the object of the tracker are the same object, the positions of the target and the object of the tracker are likely to be close to each other. Therefore, the association unit 230 changes the likelihood according to the distance between the target and the object of the tracker. In other words, the association unit 230 increases the value of the likelihood qkm when the distance between target m and the object of tracker k is close, and decreases the value of the likelihood qkm when the distance is farther.

[0101] In this case, if the target m is far from the camera 20 or if the depression angle of the camera 20 relative to the target m is shallower than a predetermined angle, the accuracy of the position of the target m is likely to be low. Therefore, the association unit 230 may calculate the distance between the target and the tracker's object using the Mahalanobis distance, which takes into account the error (ambiguity) contained in the detected position of the target, rather than simply calculating the distance using the Euclidean distance. In addition to the above method, the association unit 230 may also control the degree of change in the likelihood qkm according to the distance, taking into account the ambiguity. That is, when the ambiguity is large, the association unit 230 reduces the change in the likelihood qkm according to the distance between the target m and the tracker's object. In this way, the association unit 230 reduces the impact of a positional shift of the target on the association.

[0102] Furthermore, the association unit 230 may also take into consideration the similarity in appearance between the target and the tracker. That is, the association unit 230 may extract features such as the color, pattern, and shape of the target and tracker objects, evaluate the similarity between them, and calculate the likelihood qkm.

[0103] For example, the association unit 230 may calculate color histograms of the objects for both the target and the tracker, evaluate the similarity between them based on the overlap of the color histograms, or the like, and reflect this in the likelihood qkm. Note that, like the tracker likelihood ηk and the target likelihood Pm, the likelihood qkm representing the identity of the target and the tracker does not need to reflect all of the above-mentioned items, and may reflect only the main factors.

[0104] Furthermore, the association unit 230 may calculate each of the likelihoods described above taking into consideration the case where an object is not detected because it has gone out of the angle of view of the camera 20 or is occluded by another object. This allows the integrated tracking unit 200 to track an object with high accuracy even when the object is not detected or has gone out of the angle of view.

[0105] As described above, the problem of calculating each likelihood and finding the correspondence between the target and the tracker that maximizes each likelihood overall can be reduced to an assignment problem (which target should be associated with which tracker) that minimizes the cost by converting each likelihood into a cost using a monotonically non-increasing function. This assignment problem can be efficiently calculated using a method such as the Hungarian algorithm.

[0106] Then, the association unit 230 outputs the association result to the update unit 240. This association result includes information indicating which target and tracker are associated with each other, and the likelihoods including at least the likelihood of the tracker.

[0107] In this embodiment, the association unit 230 performs association using both the likelihood of the target and the likelihood of the tracker, but association may be performed using either one of the likelihoods.

[0108] The update unit 240 updates the tracker information. Then, the update unit 240 generates this tracker as a new object tracking result. Specifically, the update unit 240 receives the association result from the association unit 230. Then, the update unit 240 calculates the current position of the object of the tracker based on this result. Then, the update unit 240 updates the tracker information stored in the storage unit 220. The information to be updated is, for example, the position and / or size of the object whose tracking result is included in the tracker, a motion model of the object, and parameters such as the likelihood of the tracker, but the present invention is not limited to this. The update unit 240 only needs to update information that has been updated among the information stored in the storage unit 220.

[0109] First, the calculation of the current position of the tracker object by the update unit 240 will be described. The update unit 240 calculates the current position of the tracker object taking into account the accuracy of the target position. For example, the update unit 240 may weight the predicted position of the object predicted using the tracker and the detected position of the target associated with the tracker, and calculate the current position of the tracker object. In this case, the update unit 240 may control the weight depending on the accuracy of the target position.

[0110] For example, if a target is located far from the camera 20 and the depression angle of the camera 20 relative to the target is shallow, the accuracy of the target position is likely to be low. In such a case, the update unit 240 assigns a smaller weight to the target position.

[0111] On the other hand, if the target is located close to the camera 20 and the depression angle of the camera 20 relative to the target is greater than a predetermined value, the accuracy of the target position is assumed to be high. In such a case, the update unit 240 assigns a larger weight to the target position.

[0112] The update unit 240 uses the predicted position and the weighted position to calculate the current position of the tracker object.

[0113] In this way, by the update unit 240 determining the weight for the target position, the prediction result of the object detection position by the camera 20 closer to the target is more strongly reflected. Therefore, the object tracking device 10 can improve the prediction accuracy of the object position.

[0114] Then, the update unit 240 updates the most recent position of the object, which is included in the information about the object stored in the storage unit 220, to the calculated current position of the object.

[0115] Next, the update of the tracker likelihood performed by the update unit 240 will be described.

[0116] If the tracker object is located far away from the camera 20, the size of the object will be small, making it difficult for the detection unit 100 to detect such an object.

[0117] Next, a case will be described in which the recognition-type object detection unit 111 of the detection unit 100 performs recognition-type object detection when an object included in a frame looks different from the look of an object used for learning. A case in which an object included in a frame looks different from the look of an object used for learning is, for example, a case in which the depression angle of the camera 20 with respect to the object, which is assumed from the position of the object in the tracker, is significantly different from the depression angle of the camera 20 with respect to the object used for learning. In such a case, it becomes difficult for the recognition-type object detection unit 111 of the detection unit 100 to detect the object included in the frame.

[0118] In such a situation where it is difficult to detect an object, an object contained in a frame may go undetected, and in this case, there may be no target associated with the tracker associated with this object.

[0119] Therefore, the update unit 240 keeps small the change in likelihood of the tracker of the object in a situation where the object is difficult to detect, among the trackers not associated with the target.

[0120] In this way, the update unit 240 can reduce the impact on tracking when an object is difficult to detect, and can make the detection result from a camera where the object is easy to detect reflected more significantly in the tracker likelihood. Then, when tracking an object in the next frame, the association unit 230 performs association based on this tracker likelihood, so the integrated tracking unit 200 can further improve the accuracy of object tracking.

[0121] Then, of the likelihoods of the trackers stored in the storage unit 220, the likelihood of the tracker calculated by the association unit 230 and the likelihood of the tracker for which the change has been suppressed to a small value are updated.

[0122] Next, the updating of the motion model of the tracker object by the update unit 240 will be described. For example, a case will be described in which the integrated tracking unit 200 performs object tracking by predicting the position of the object using a Kalman filter. In this case, the update unit 240 substitutes the position coordinates of the tracker object associated with the target as the detected position coordinates into the update equation for the state variables of the Kalman filter stored in the storage unit 220, and updates the state of the Kalman filter.

[0123] In addition to the above, the update unit 240 also updates other parameters of the tracker stored in the storage unit 220.

[0124] For example, the object itself may change its posture. For example, if the object is a person, the apparent height of the object may change when the person crouches or bends. In this way, when there is a change in the size of the object in addition to the motion model, the update unit 240 updates the information stored in the storage unit 220 that has been changed.

[0125] Furthermore, if the tracker parameters include the probability that the tracker object exists, weights that represent the reliability of the tracking result, etc., these weights change depending on the result of the association by the association unit 230. Therefore, the update unit 240 updates these parameters such as the weights.

[0126] Furthermore, the update unit 240 updates the trackers by creating or deleting them. First, the update unit 240 determines whether or not there is a target that does not correspond to a tracker after the association process by the association unit 230. This target that does not correspond to a tracker may be an object that has newly appeared within the range photographed by the camera 20. Therefore, when there is a target that does not correspond to a tracker, the update unit 240 determines whether or not this target can be considered as an object that has newly appeared within the range.

[0127] That is, when there is a target that does not correspond to a tracker, the update unit 240 evaluates the probability that this target exists. Then, the update unit 240 determines whether this probability is equal to or greater than a predetermined value. Then, if this probability is equal to or greater than the predetermined value, the update unit 240 determines that the object indicated by this target is an object that has newly appeared within the range. Then, the update unit 240 creates a new tracker related to the object (target) that has been determined to be an object that has newly appeared within the range.

[0128] Furthermore, the update unit 240 determines whether or not there is a tracker that does not correspond to the target. This tracker that does not correspond to the target may be a tracker related to an object that has disappeared (moved from within the range to outside the range) within the range photographed by the camera 20. Therefore, if there is a tracker that does not correspond to the target, the update unit 240 determines whether or not this target can be considered to be a tracker related to an object that has disappeared from within the range.

[0129] That is, when a tracker that does not correspond to a target exists, the update unit 240 evaluates the probability that an object related to this tracker exists. Then, the update unit 240 determines whether this probability is below a predetermined value. Then, if this probability is below the predetermined value, the update unit 240 determines that the object related to this tracker is an object that has disappeared from the above-mentioned range. Then, the update unit 240 deletes the tracker related to the object that has been determined to be an object that has disappeared from the above-mentioned range.

[0130] The probability that an object exists is determined by the likelihood of the tracker. That is, if there is a tracker that does not correspond to the target, the update unit 240 decreases the likelihood of this tracker. Then, if the likelihood value of the tracker falls below a predetermined threshold, the update unit 240 deletes the tracker.

[0131] The update unit 240 then generates the tracker that is finally left as the tracking result of the object in this frame.The update unit 240 then outputs, among these, information indicating the position of the tracker, information regarding the size of the object of the tracker, etc. as information indicating the tracking result of the object tracking (tracking information).

[0132] As described above, according to the object tracking device 10 of this embodiment, object tracking is performed in chronological order based on the time information included in the camera images output from each camera 20. Objects detected from the image data of each of the multiple cameras 20 are detected by integrating the most accurate detection results from each camera 20 and using a tracking result that reflects this detection result and past tracking results. This improves the tracking accuracy of object tracking performed by the integrated tracking unit 200 of the object tracking device 10.

[0133] Next, the flow of the object tracking process of the object tracking device 10 according to this embodiment will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of the flow of the object tracking process of the object tracking device 10 according to this embodiment.

[0134] As shown in FIG. 8, first, the recognition-type object detection unit 111 of the detection unit 100 receives a camera image from the camera 20 linked to the detection unit 100 including the object detection unit 111 (step S81).

[0135] The detection unit 100 checks whether the received camera video frame is the first frame output from the camera 20 (step S82), and if it is the first frame (YES in step S82), the process proceeds to step S85.

[0136] If the received camera image frame is not the first frame (NO in step S82), the individual coordinate conversion unit 130 converts the tracking information for the previous frame output from the integrated tracking unit 200 into tracking information expressed in the individual coordinate system (step S83).

[0137] Then, the search range setting unit 112 of the object detection unit 110 uses the tracking information converted in step S83 to set a search range for the object in the current frame (step S84).

[0138] Then, the recognition-type object detection unit 111 detects an object from the received camera image (step S85).

[0139] Next, the individual coordinate transformation unit 130 transforms the detection result by the recognition-type object detection unit 111 into a detection result expressed in a common coordinate system (step S86).

[0140] Next, the prediction unit 210 of the integrated tracking unit 200 predicts the position of the object in the current frame using the tracker information (step S87).

[0141] Then, the association unit 230 associates the object (target) included in the detection result with the tracker (step S88).

[0142] Next, the update unit 240 updates the tracker information such as the position of the object in the tracker and the motion model of the object (step S89).

[0143] Furthermore, the update unit 240 generates and / or deletes a tracker (step S90).The object tracking device 10 then repeats this process until no more frames are input to the detection unit 100.

[0144] (effect) As described above, the object tracking device 10 according to this embodiment can track objects with higher accuracy. This is because the detection unit 100 detects an object from the output information of the camera 20 based on tracking information for the output information previous to the output information (video frame). Then, the integrated tracking unit 200 tracks the object based on multiple detection results output by each detection unit 100, and generates tracking information for the object expressed in a common coordinate system.

[0145] For example, if there is an object that is invisible from one camera 20 but visible from another camera 20, the object tracking device 10 uses the tracking results for the object that is invisible from the camera 20 for object detection in the video of the camera 20. This allows the detection unit 100 to suitably detect the object when the same object appears within the range visible from the camera 20. Therefore, the object tracking device 10 can accurately track the object.

[0146] Therefore, object tracking device 10 can improve the object detection accuracy compared to when not using the tracking results for the previous frame. Also, since object tracking device 10 tracks objects using all of the detection results with high detection accuracy, the accuracy of the tracking results obtained as a whole is improved.

[0147] In this way, the object tracking device 10 according to this embodiment can extract the movement path of a person or the like who moves across areas photographed by each of the multiple cameras 20. As a result, the tracking results by the object tracking device 10 can be used, for example, to analyze the behavior of customers moving around in a store and serve as basic information for marketing or changing the store layout. In addition, the tracking results can be used to detect people wandering between areas for security purposes.

[0148] Furthermore, since the search range setting unit 112 uses the tracking result to set a search range for object detection in the camera image, the recognition-type object detection unit 111 can reduce unnecessary false detections. Furthermore, the recognition-type object detection unit 111 can speed up the object detection process.

[0149] Furthermore, the integrated tracking unit 200 performs object tracking using the likelihood of the target and / or the likelihood of the tracker, which allows the object tracking device 10 to obtain more reliable object tracking results. Furthermore, the object tracking device 10 performs object search using the object tracking results obtained in this manner, which allows the object tracking device 10 to further improve the accuracy of object tracking overall. As a result, the object tracking device 10 can improve the accuracy of object tracking.

[0150] <Second embodiment> Next, a second embodiment of the present invention will be described with reference to the drawings. For the sake of convenience, components having the same functions as those included in the drawings described in the first embodiment will be given the same reference numerals, and their description will be omitted.

[0151] The object tracking system 2 according to this embodiment is configured to include an object tracking device 50 instead of the object tracking device 10 of the object tracking system 1 according to the first embodiment described using Fig. 2. Other system configurations of the object tracking system 2 are similar to those of the object tracking system 1 shown in Fig. 2, and therefore description thereof will be omitted.

[0152] (Object tracking device 50) The function of the object tracking device 50 will be described with reference to Fig. 9. Fig. 9 is a functional block diagram showing an example of the functional configuration of the object tracking device 50 according to this embodiment. As shown in Fig. 9, the object tracking device 50 includes a plurality of detection units (100-1 to 100-N), an integrated tracking unit 200, and a display control unit 300. Note that in this embodiment, as in the first embodiment described above, when the plurality of object detection units (100-1 to 100-N) are not to be distinguished from one another or when they are referred to collectively, they are referred to as detection units 100.

[0153] The display control unit 300 controls the image (video) to be displayed on the display device 30. Specifically, the display control unit 300 generates display data by converting the tracking information output by the integrated tracking unit 200 into data that can be displayed on the display device 30, and transmits the display data to the display device 30. The tracking information output by the integrated tracking unit 200 is expressed in a common coordinate system. Therefore, the display control unit 300 generates display data that can be displayed on the display device 30 in the common coordinate system.

[0154] At this time, it is preferable that the integrated tracking unit 200 outputs information necessary for displaying the movement line of the object on the display device 30 (for example, information indicating the past positions of the tracker object) as tracking information to the display control unit 300. This tracking information may be the same as the information fed back to the detection unit 100, or it may be different.

[0155] The display device 30 then displays the received display data on the screen, thereby enabling the object tracking device 50 to present the tracking results to the user.

[0156] Furthermore, detection unit 100 of object tracking device 50 according to this embodiment converts the search range set by search range setting unit 112 of object detection unit 110 into data that can be displayed on display device 30, generates display data, and transmits the display data to display device 30. The search range information output by search range setting unit 112 is expressed in an individual coordinate system. Therefore, display control unit 300 generates display data that can be displayed on display device 30 in the individual coordinate system, using camera parameters of camera 20 linked to detection unit 100 that output the search range information.

[0157] The display device 30 then displays the received display data on the screen. This allows the object tracking device 50 to present the search range to the user using the object search range information output from each detection unit 100.

[0158] It should be noted that there may be a plurality of display devices 30. For example, the display devices 30 may be configured such that display data displayed in a common coordinate system and display data displayed in individual coordinate systems are received by different display devices 30, and each display device 30 displays the received display data on its screen. Furthermore, the display device 30 may be configured to divide the display area of one screen and display a plurality of pieces of display data on the screen. In this way, the method of displaying display data in the display device 30 according to this embodiment is not particularly limited.

[0159] 5, the object detection unit 110 includes a recognition-type object detection unit 111 and a search range setting unit 112. The display control unit 300 may be provided within the detection unit 100. Fig. 10 is a functional block diagram showing an example of the functional configuration of the detection unit 100 of the object tracking device 50 according to the present embodiment. As shown in Fig. 10, the detection unit 100 includes an object detection unit 110, a common coordinate transformation unit 120, an individual coordinate transformation unit 130, and a display control unit 150. The object detection unit 110 includes a recognition-type object detection unit 111 and a search range setting unit 112, similar to the object detection unit 110 shown in Fig. 5.

[0160] 10 outputs search range information indicating the set search range to the display control unit 150. The display control unit 150 receives the search range information output from the search range setting unit 112 and, similar to the display control unit 300, converts the search range information into data that can be displayed on the display device 30 to generate display data. The search range information output by the search range setting unit 112 is expressed in an individual coordinate system. Therefore, the display control unit 150 generates display data that can be displayed on the display device 30 in the individual coordinate system using camera parameters of the camera 20 linked to the detection unit 100 that includes the display control unit 150.

[0161] Then, the display control unit 150 transmits the generated display data to the display device 30. Then, the display device 30 displays the received display data on the screen.

[0162] (Application example) Application examples of the object tracking device 50 according to this embodiment will be described with reference to Figures 11 to 14. Figures 11 to 14 are diagrams for explaining application examples of the object tracking device 50 according to this embodiment.

[0163] First, Fig. 11 is a diagram showing an example of a room where shelves R1 and R2 and multiple cameras (A to F) are installed, viewed from the direction opposite to the direction of gravity. As shown in Fig. 11, the horizontal direction of Fig. 11 is the X-axis in a common coordinate system, and the vertical direction is the Y-axis. Shelves R1 and R2 are installed side by side on the X-axis so that their longitudinal directions are parallel to the Y-axis direction.

[0164] Camera A is installed in a position close to the entrance of this room. In this embodiment, it is assumed that the entire room is photographed by cameras A to F. That is, as shown in FIG. 11, the indoor space in which multiple cameras (A to F) are installed becomes the photographed space. Furthermore, cameras A to F photograph the same location.

[0165] Fig. 12 is a diagram showing an example of images captured by each of camera A and camera B. The upper diagram in Fig. 12 shows a frame of the image captured by camera A, and the lower diagram shows a frame of the image captured by camera B. The coordinate values in these frames are expressed as coordinate values in the individual coordinate system for each camera.

[0166] The object tracking device 50 in this embodiment may be configured to display the video captured by the camera 20 on the display device 30.

[0167] As shown in Figure 12, the video captured by camera A includes person C1. Furthermore, because camera A is installed near the entrance, the video also includes the entrance. Furthermore, the video captured by camera B includes person C1 and person C2.

[0168] When viewed from camera A, person C2 is hidden behind shelf R1. Therefore, at the time of the video in Fig. 12, person C2 is an object that is not visible to camera A. If these video frames are the first frames of the video captured by camera A and camera B, there is no tracking information in the previous frame, so object tracking device 50 detects the object from these frames and generates tracking information.

[0169] Then, using the tracking information expressed in the individual coordinate system, search range setting unit 112 sets a search range for the object for the next frame of the video captured by camera A. Similarly, using the tracking information expressed in the individual coordinate system, search range setting unit 112 sets a search range for the object for the next frame of the video captured by camera B.

[0170] Fig. 13 is a diagram showing an example of a search range displayed on display device 30. The upper diagram in Fig. 13 is a diagram showing an example of a search range for an object for a frame output from camera A, and the lower diagram is a diagram showing an example of a search range for an object for a frame output from camera B.

[0171] As shown in the upper diagram of FIG. 13, the search range setting unit 112 determines a search range A1 from the position of person C1 in the upper diagram of FIG. 12. The search range setting unit 112 also determines the area of the entrance / exit to the room as search range N1. The search range setting unit 112 also determines the outer edges of the frame as search ranges N2 and N3. The search range setting unit 112 then outputs information summarizing the determined search ranges A1, N1 to N3 as search range information to the recognition-type object detection unit 111 and the display control unit 150 or the display control unit 300. The display control unit 150 or the display control unit 300 then converts the search range indicated by this search range information into display data that can be displayed on the screen of the display device 30 and transmits the converted data to the display device 30.

[0172] The display device 30 receives the display data from the display control unit 150 or the display control unit 300 and displays the search range on the screen as shown in the upper diagram of FIG.

[0173] Next, the lower diagram of Fig. 13 will be described. As shown in the lower diagram of Fig. 13, the search range setting unit 112 determines search ranges B1 and B2 from the positions of person C1 and person C2, respectively, in the lower diagram of Fig. 12. The search range setting unit 112 also determines the outer edges of the frame as search ranges N4 to N7. The search range setting unit 112 then outputs information summarizing the determined search ranges B1, B2, and N4 to N7 as search range information to the recognition-type object detection unit 111 and the display control unit 150 or the display control unit 300. The display control unit 150 or the display control unit 300 then converts the search range indicated by this search range information into display data that can be displayed on the screen of the display device 30, and transmits the converted data to the display device 30.

[0174] The display device 30 receives the display data from the display control unit 150 or the display control unit 300 and displays the search range on the screen as shown in the lower diagram of FIG.

[0175] The display device 30 may display the search range in a different manner for each region. For example, the display device 30 may display the search range for an already detected object and the search range for the outer edge of the frame in different colors.

[0176] The integrated tracking unit 200 then generates trackers for the persons C1 and C2 detected in subsequent frames, and outputs information necessary for displaying the respective flow lines of the persons C1 and C2 to the display control unit 300 as tracking information.

[0177] The display control unit 300 receives the tracking information from the integrated tracking unit 200 , converts the tracking information into display data that can be displayed on the display device 30 , and transmits the display data to the display device 30 .

[0178] The display device 30 then displays the display data received from the display control unit 300 on the screen. FIG. 14 is a diagram showing an example in which the display device 30 displays on the screen (display screen) flow lines indicating the tracking results of each of the persons C1 and C2. As shown in FIG. 14, in this application example, the display screen displays the tracking results of the objects on the XY plane in the common coordinate system. In FIG. 14, the flow line of the person C1 is displayed as a solid line, and the flow line of the person C2 is displayed as a dashed line. In this way, the display device 30 can display the tracking results of the objects on the screen.

[0179] <Third embodiment> Next, a third embodiment of the present invention will be described with reference to the drawings. For the sake of convenience, components having the same functions as those included in the drawings described in the first and second embodiments will be designated by the same reference numerals, and their description will be omitted.

[0180] The object tracking device 10 according to this embodiment is configured to include a detection unit 400 instead of the detection unit 100 of the object tracking device 10 shown in Fig. 1. The configuration of this detection unit 400 will be described with reference to Fig. 15. Fig. 15 is a functional block diagram showing an example of the functional configuration of the detection unit 400 of the object tracking device 10 according to this embodiment.

[0181] 4 and 5, detection unit 400 includes object detection unit 140 instead of object detection unit 110. Detection unit 400 also includes storage unit 160. That is, detection unit 400 according to this embodiment includes object detection unit 140, common coordinate transformation unit 120, individual coordinate transformation unit 130, and storage unit 160, as shown in FIG.

[0182] In this embodiment, an example configuration will be described in which the object detection unit 110 of the detection unit 100 of the object tracking device 10 according to the first embodiment is replaced with an object detection unit 140. Note that the present invention is not limited to this, and the object tracking device 50 according to the second embodiment may be configured to include the object detection unit 140 instead of the object detection unit 110 of the detection unit 100. In other words, the detection unit 100 according to this embodiment may be configured to output data to be displayed to the display control unit 150 or the display control unit 300.

[0183] The storage unit 160 stores camera parameters for each camera 20 that are used when converting the coordinate system. Furthermore, the storage unit 160 stores information indicating the range of coordinate values in the common coordinate system that the individual coordinate conversion unit 130 uses when checking whether or not the coordinate values in the common coordinate system included in the tracking information are included in the range captured by the associated camera 20. The storage unit 160 may also store video captured by the camera 20. Note that this video may be stored temporarily.

[0184] 15, an example will be described in which the storage unit 160 is built into the detection unit 400, but the present invention is not limited to this. The storage unit 160 may be provided in the object tracking device 10 separately from the detection unit 400. Furthermore, the storage unit 160 may be realized by a storage device or the like separate from the object tracking device 10.

[0185] Next, a detailed functional configuration of object detection unit 140 of detection unit 400 will be described with reference to Fig. 16. Fig. 16 is a functional block diagram showing an example of the functional configuration of object detection unit 140 of detection unit 400 according to this embodiment. As shown in Fig. 16, object detection unit 140 includes a recognition-type object detection unit (first object detection means) 141, a non-recognition-type object detection unit (second object detection means) 142, a detection parameter update unit 143, and a detection result integration unit 144.

[0186] In this embodiment, object detection using a dictionary (classifier) or the like is called "recognition-based object detection." On the other hand, object detection without using a classifier or the like is called "non-recognition-based object detection."

[0187] The recognition-type object detection unit 141 detects objects from camera images input to the recognition-type object detection unit 141. The recognition-type object detection unit 141 detects objects in the entire frame. Note that, as indicated by the dashed line in Fig. 16, the recognition-type object detection unit 141 may perform object detection based on search range information output from a detection parameter update unit 143, which will be described later. In this case, the recognition-type object detection unit 141 performs object detection using a method similar to that of the recognition-type object detection unit 111 described in the first embodiment.

[0188] Furthermore, when search range information is not output from the detection parameter update unit 143, the recognition-type object detection unit 141 may perform object detection using a different criterion rather than performing object detection on the entire frame. For example, the recognition-type object detection unit 141 may use silhouette information to perform object detection only on the area where the silhouette is located and its surrounding area.

[0189] The recognition-type object detection unit 141 outputs the detection result of the object detection to the detection result integration unit 144 as a first detection result.

[0190] Furthermore, the recognition-type object detection unit 141 may extract appearance features of the object at this stage in preparation for object detection by the non-recognition-type object detection unit 142, which will be described later. Appearance features of an object include information such as the color, pattern, and shape of the object, but the present invention is not limited to this. The recognition-type object detection unit 141 extracts these feature amounts as appearance features of the object. In this case, the area used for object detection by the recognition-type object detection unit 141 and the area used for object detection by the non-recognition-type object detection unit 142 may not be the same. For example, if the object is a person, the recognition-type object detection unit 141 may detect the head, and the non-recognition-type object detection unit 142 may detect up to the clothing area. In this case, the recognition-type object detection unit 141 extracts feature amounts of the appearance features of the object so as to include the clothing area. The recognition-type object detection unit 141 may then output the extracted feature amounts as template information together with information indicating the area used for extraction (extraction area information). Furthermore, the recognition-type object detection unit 141 may store the feature amount itself inside the recognition-type object detection unit 141 and output only information for identifying the feature amount.

[0191] The detection parameter update unit 143 receives tracking information expressed in the individual coordinate system from the individual coordinate transformation unit 130. Then, the detection parameter update unit 143 uses this tracking information to determine parameters (called detection parameters) necessary for object detection. These detection parameters are parameters required for object detection processing. The detection parameters include, for example, a predicted position (prediction area) of the object in the current frame, a search range in which object detection is applied, the size of the template used in template matching, and template feature amounts (template information) of a target previously associated with a tracker. Note that the detection parameters do not need to include all of this information, and it is sufficient that they include parameters necessary for object detection by the non-recognition-type object detection unit 142. The detection parameters may also include information about a target associated with a tracker in the object tracking result in the previous frame.

[0192] For example, for a target (object) detected in the previous frame and associated with a tracker, the detection parameter update unit 143 determines the position where the object exists in the current frame as a predicted position based on tracking information of the object. This prediction process is similar to the prediction process of the predicted position in the search range setting unit 112 according to the first embodiment. Note that the detection parameter update unit 143 may determine an area including this predicted position as a predicted area.

[0193] Furthermore, for example, the detection parameter update unit 143 may determine a range in which object detection by template matching is applied, with the prediction region at the center, and include this range as a search range for objects included in the detection parameters.

[0194] The detection parameter update unit 143 obtains the above detection parameters for each object included in the tracking information. Then, the detection parameter update unit 143 updates the obtained detection parameters as detection parameters to be used in the object detection process. Then, the detection parameter update unit 143 outputs the detection parameters to the non-recognizable object detection unit 142.

[0195] Note that the detection parameter update unit 143 may determine the search range of the object by using the tracking information converted into coordinate values in the individual coordinate system, similar to the search range setting unit 112 of the detection unit 100 according to the first embodiment described above. Then, the detection parameter update unit 143 may output search range information indicating the determined search range of the object to the recognition-type object detection unit 141.

[0196] The non-recognizable object detection unit 142 receives the detection parameters from the detection parameter update unit 143. Then, based on the received detection parameters, the non-recognizable object detection unit 142 detects objects from the camera video input to the non-recognizable object detection unit 142. Unlike the recognition-type object detection unit 141, the non-recognizable object detection unit 142 detects objects based on the similarity of the appearance of objects detected in the previous frame.

[0197] That is, when an object is detected in the previous frame, the non-recognition type object detection unit 142 stores the image features of that area (or the partial image of the detected area itself) as a template.The non-recognition type object detection unit 142 then performs object detection by checking, through template matching, whether an area similar to this stored template exists in the current frame.The image features used in this case may include, for example, information on color patterns and distribution, distribution information on edges and brightness gradients, or features obtained by combining these.

[0198] The detection parameters used when the non-recognizable object detection unit 142 detects an object are controlled by the detection parameters output from the detection parameter update unit 143. Specifically, the non-recognizable object detection unit 142 performs object detection by template matching on the predicted position (prediction area) of the object predicted by the detection parameter update unit 143 and its vicinity. That is, the non-recognizable object detection unit 142 sets a search range for template matching centered on the predicted object existence range (prediction area) and performs template matching on the periphery thereof. In addition, at this time, the non-recognizable object detection unit 142 may also take into account changes in the apparent size of the object due to movement of the object's position. This change can be calculated using camera parameters. Therefore, the non-recognizable object detection unit 142 may calculate the change in the object's size and reflect it in the template before performing template matching.

[0199] Furthermore, the information on the template for template matching performed by the non-recognition-type object detection unit 142 may be the feature amount extracted by the recognition-type object detection unit 141 in the object detection process in the previous frame.

[0200] In this way, the non-recognition type object detection unit 142 performs object detection based on the tracking results for the objects tracked by the integrated tracking unit 200, thereby improving the detection accuracy compared to when the tracking results are not used.

[0201] Then, the non-recognition-type object detection unit 142 outputs the detection result of the object detection to the detection result integration unit 144 as a second detection result.

[0202] The detection result integration unit 144 receives a first detection result from the recognition-type object detection unit 141. The detection result integration unit 144 also receives a second detection result from the non-recognition-type object detection unit 142. The detection result integration unit 144 then integrates the first detection result and the second detection result. The detection result integration unit 144 then outputs the integrated result to the common coordinate transformation unit 120 as the detection result of the object detection in the object detection unit 140.

[0203] There may be objects that are included in both the first and second detection results, and objects that are included in only one of them. Therefore, the detection result integration unit 144 associates and integrates the objects that are included in the first and second detection results. For example, this association can be performed using the degree of overlap of the object areas.

[0204] That is, the detection result integration unit 144 calculates the overlap ratio between object regions (for example, the overlap ratio of object circumscribing rectangles), and if this is greater than a predetermined value, it associates the object included in the first detection result with the object included in the second detection result.

[0205] Alternatively, the detection result integration unit 144 may formulate the problem as a graph problem in which the overlap ratio of the areas between the objects is used as a weight, and perform correspondence between the objects. For example, the detection result integration unit 144 converts the overlap ratio into a cost using a monotonically non-increasing function, and then calculates the optimal correspondence using the Hungarian method or the like, thereby performing correspondence between the objects.

[0206] The detection result integrating unit 144 may merge at this point any matching results for which the value used in the matching (for example, overlap ratio or cost) is greater than a predetermined value. Alternatively, the detection result integrating unit 144 may not merge at this point, but may instead generate information indicating that the results are matched. The detection result integrating unit 144 may then output the detection result obtained by combining the first and second detection results with the information indicating that the results are matched as the detection result of the object detection unit 140, and may perform tracking using the information regarding the matching during integrated tracking.

[0207] Furthermore, the non-recognition-type object detection unit 142 outputs a second detection result based on the object tracking result for the previous frame. Therefore, the second detection result may be generated later than the first detection result. In such a case, the detection result integration unit 144 temporarily stores the first detection result in a storage means such as a buffer within the detection result integration unit 144 or in the storage unit 160. Then, upon receiving the second detection result for a frame corresponding to the frame for which the first detection result is generated, the detection result integration unit 144 may integrate both results.

[0208] As described above, the detection unit 400 of the object tracking device 10 according to this embodiment outputs, as a detection result, a result obtained by integrating the result of object detection by the recognition-type object detection unit 141 (first detection result) and the result of object detection by the non-recognition-type object detection unit 142 (second detection result). At this time, the non-recognition-type object detection unit 142 detects an object by performing template matching based on the tracking result of the object tracked by the integrated tracking unit 200. This allows the detection unit 400 to further improve the accuracy of object detection compared to when only performing object detection by identifying the object (recognition-type object detection).

[0209] Therefore, the object tracking device 10 can track the object with higher accuracy.

[0210] <Fourth embodiment> Next, a fourth embodiment of the present invention will be described with reference to the drawings. For the sake of convenience, components having the same functions as those included in the drawings described in the above-mentioned embodiments will be designated by the same reference numerals, and their description will be omitted.

[0211] The object tracking device 10 according to this embodiment is configured to include an integrated tracking unit 500 instead of the integrated tracking unit 200 of the object tracking device 10 shown in FIG. 1. The configuration of this integrated tracking unit 500 will be described with reference to FIG. 17. FIG. 17 is a functional block diagram showing an example of the functional configuration of the integrated tracking unit 500 of the object tracking device 10 according to this embodiment. As shown in FIG. 17, the integrated tracking unit 500 includes a buffer unit 510, a prediction unit 210, a storage unit 220, an association unit 530, and an update unit 240.

[0212] In this embodiment, an example configuration will be described in which an integrated tracking unit 500 is provided instead of the integrated tracking unit 200 of the object tracking device 10 according to the first embodiment. Note that the present invention is not limited to this, and the object tracking device 50 according to the second embodiment may be provided with the integrated tracking unit 500 instead of the integrated tracking unit 200. In other words, the integrated tracking unit 500 according to this embodiment may be configured to output data to be displayed to the display control unit 300.

[0213] Furthermore, the detection unit that outputs the detection result to the integrated tracking unit 500 according to this embodiment may be the detection unit 400 described in the third embodiment.

[0214] The buffer unit 510 is a means for temporarily storing the detection results expressed in a common coordinate system output from the detection unit 100. Of the data (detection results) buffered in the buffer unit 510, data for which the time information included in the camera video in which the detection was performed falls within a predetermined period is acquired by the association unit 530. This predetermined period is a periodic period. The association unit 530 then performs object tracking using one or more detection results acquired at a certain period. In this way, the integrated tracking unit 500 in this embodiment is also referred to as a collective tracking unit, because it performs object tracking using video from multiple cameras among the video from each camera 20.

[0215] The object tracking (also referred to as collective integrated tracking) performed by the integrated tracking unit 500 will be described with reference to FIG. 18. FIG. 18 is a diagram for explaining the collective object tracking process performed by the integrated tracking unit 500 according to this embodiment. As with FIG. 7, FIG. 18 shows an example of the timing at which images are acquired by each of camera A, camera B, and camera C when there are three cameras. In FIG. 18, the horizontal axis indicates the time axis, with the further to the right, the later in time it is. As shown in FIG. 18, camera A acquires images at times t1, t5, and t8. Similarly, camera B acquires images at times t2, t4, t6, and t9, and camera C acquires images at times t3 and t7.

[0216] Then, object detection is performed sequentially on the images acquired at each of these timings. For convenience of explanation, the following description will be given assuming that the times shown in FIG. 18 are substantially the same as the times at which the object detection results are output. In other words, time t1 is the time at which the detection results for the frames of the video captured by camera A are output from detection unit 100 and buffered in buffer unit 510.

[0217] The time axis at the bottom of FIG. 18 shows an example of a periodic period.

[0218] The association unit 530 acquires, from one or more detection results buffered in the buffer unit 510, a detection result for which time information included in the camera video that was the target of object detection falls within a predetermined period. As described above, this predetermined period is a periodic period. In this embodiment, the buffered time and the camera video time are considered to be the same. Therefore, it can be said that the association unit 530 acquires one or more detection results buffered in the buffer unit 510 at a predetermined period.

[0219] Specifically, the association unit 530 first acquires the detection results buffered during the first period T1. That is, the association unit 530 acquires the detection results buffered at times t1, t2, and t3. The detection result buffered at time t1 is the detection result for the frame of the video captured by camera A. The detection result buffered at time t2 is the detection result for the frame of the video captured by camera B, and the detection result buffered at time t3 is the detection result for the frame of the video captured by camera C. Therefore, the association unit 530 acquires from the buffer unit 510 a plurality of detection results buffered in the buffer unit 510 within a predetermined period (T1 in this case), which are for the frames of the videos captured by each of the plurality of cameras 20. Then, the association unit 530 performs object tracking using the acquired detection results.

[0220] Similarly, during periods T2, T3, and T4, the association unit 530 also acquires the detection results buffered within these periodic periods and performs object tracking.

[0221] In the present embodiment, a configuration will be described in which association unit 530 acquires data (plurality of detection results) buffered in buffer unit 510 from buffer unit 510 at predetermined intervals, but association unit 530 may also be configured to receive this data from buffer unit 510 at predetermined intervals. In other words, buffer unit 510 may have a function of outputting this data to association unit 530 at predetermined intervals.

[0222] Returning to FIG. 17, the association unit 530 of the integrated tracking unit 500 will be further described.

[0223] The association unit 530 calculates the distance between targets from the positions of the targets included in each of the acquired detection results, and associates targets that are close to each other. At this time, the association unit 530 performs the association using a method such as the Hungarian algorithm, using the distance between the targets. The association unit 530 may also use the similarity in the appearance characteristics of the targets in addition to the distance between the targets. For example, targets that are close to each other and have similar colors are likely to be the same object. Therefore, the association unit 530 may perform the association using such characteristics. Note that the characteristics used to determine the similarity in appearance characteristics are not limited to color, and may be, for example, the pattern of the target.

[0224] The association unit 530 then integrates the detection results for the associated targets. That is, after associating the targets, the association unit 530 determines the position of the object using the detection results for each of the associated targets. In this case, the association unit 530 may evaluate the likelihood and / or accuracy of the predicted position of each target, and determine the position with the highest accuracy as the position of the object.

[0225] Furthermore, the association unit 530 may weight the position of each target based on the accuracy of the predicted position determined by the angle (depression angle or elevation angle) of the camera 20 relative to each target and the distance from the camera 20 to the target, etc. Then, the association unit 530 may calculate a statistic such as an average value from the weighted positions, and may regard the position indicated by the calculated statistic as the position of the object.

[0226] The association unit 530 then sets the determined object position as the target position for the period in which the detection result was obtained. The association unit 530 uses this target position to perform association in the same way as the association unit 230 according to the first embodiment. The association process and subsequent processes by the integrated tracking unit 500 are similar to the processes by the integrated tracking unit 200 described in the first embodiment, and therefore will not be described here.

[0227] Furthermore, the association unit 530 may associate each target with a tracker and integrate the targets before associating the targets with each other. That is, when there are multiple targets associated with the same tracker, the association unit 530 performs a merging process between these targets. In this case, the association unit 530 may prioritize merging targets with higher likelihood and / or higher accuracy of predicted position. In this way, the association unit 530 may evaluate the detection results based on this information and integrate targets associated with the same tracker.

[0228] As described above, the object tracking device 10 according to this embodiment performs object tracking using all of the detection results for the camera images captured by each camera 20 within a predetermined period of time. This allows the object tracking device 10 to prioritize object search results among the multiple cameras 20, making it easier to apply the object tracking process.

[0229] Furthermore, for example, if the frame rates of all cameras 20 are stable and the same, by setting a predetermined period according to the frame interval, frames from all cameras 20 are included in this predetermined period. Therefore, the object tracking device 10 according to this embodiment can perform object tracking for frames from all cameras 20. This allows the object tracking device 10 to simultaneously evaluate detection results for objects that are simultaneously visible from multiple cameras 20, and therefore allows the reliability of the detection results to be more directly reflected in tracking.

[0230] <Fifth embodiment> A fifth embodiment of the present invention will be described below, which is a minimum configuration that solves the problems of the present invention.

[0231] The object tracking device 10 according to this embodiment has the same configuration as the object tracking device 10 shown in FIG. 1 and described in the first embodiment, and therefore will be described with reference to FIG.

[0232] 1, the object tracking device 10 according to this embodiment includes a plurality of detection units (100-1 to 100-N) (N is a natural number) and an integrated tracking unit 200. In this embodiment, when the plurality of detection units (100-1 to 100-N) are not distinguished from one another or are referred to collectively, they are referred to as detection units 100.

[0233] Each of the multiple detection units 100 detects an object from output information output from a sensor. Note that in FIG. 1, the sensor is depicted as a camera and the sensor output information is depicted as camera video (video data), but the sensor is not limited to a camera. Specifically, the detection unit 100 detects an object based on tracking information output from the integrated tracking unit 200. The detection unit 100 outputs the detection result to the integrated tracking unit 200.

[0234] Based on the multiple detection results output by the multiple detection units (100-1 to 100-N), the integrated tracking unit 200 tracks each of one or more objects indicated by the detection results.The integrated tracking unit 200 then generates tracking information of the objects expressed in a common coordinate system as the tracking results.The integrated tracking unit 200 then outputs this to each of the multiple detection units (100-1 to 100-N).

[0235] In this way, the detection unit 100 of the object tracking device 10 according to this embodiment detects an object from the output information output from the sensor, based on the tracking result for the object tracked by the integrated tracking unit 200.

[0236] In this way, object tracking device 10 detects an object from a video using tracking results for a previous frame, thereby improving object detection accuracy compared to when the tracking results are not used. Also, because object tracking is performed using all of the object detection results for the video captured by each of cameras 20, object tracking device 10 can improve tracking accuracy compared to when object tracking is performed independently for each camera. Also, because object tracking device 10 performs object tracking using all of the detection results with high detection accuracy, it can track objects with higher accuracy.

[0237] In the above-described embodiments, the object tracking device 10 has been described as including a detection unit (100, 400) and an integrated tracking unit (200, 500), but the detection unit and the integrated tracking unit may be realized as separate devices. In other words, the detection unit (100, 400) may be realized as an object detection device, and the integrated tracking unit (200, 500) may be realized as an integrated tracking device, each as a separate device. Furthermore, the display control unit 300 may also be realized as a display control device separate from the object tracking device 50. This display control device may be built into the display device 30. Sixth Embodiment A sixth embodiment of the present invention will be described. Object tracking device 10 according to this embodiment has the same configuration as object tracking device 10 shown in FIG. 1 described in the first embodiment, and therefore will be described with reference to FIG. 1. Note that object tracking device 10 according to this embodiment has a configuration in which the functions described below are added to object tracking device 10 according to the first embodiment, but the present invention is not limited to this. Object tracking device 10 according to this embodiment can also be applied to the object tracking devices according to the second to fifth embodiments described above.

[0238] In this embodiment, the integrated tracking unit 200 further acquires information about the appearance of the object and includes the acquired information in the tracking information. The detection unit 100 then controls object detection using the information about the appearance of each object included in the tracking information output from the integrated tracking unit 200.

[0239] This information about how an object appears (hereinafter referred to as appearance information) is information about how each object appears when viewed from the position of each camera 20, and is determined by the position of the object on each tracker.

[0240] For example, consider a case where an object is in front of (on the camera 20 side of) another object when the object is viewed from a certain camera 20. In this case, the object behind (the other object) is likely to be hidden by the object in front (the certain object) and become invisible to the camera 20. In this case, the integrated tracking unit 200 includes information representing such overlap between objects as visibility information in the tracker (tracking result) for the other object, and outputs the tracking result.

[0241] Next, specific operations of each part of the object tracking device 10 according to this embodiment will be described.

[0242] The integrated tracking unit 200 stores information about the placement of each camera 20, for example, in the storage unit 220 shown in FIG. 6 . The information about the placement of the cameras 20 is, for example, information indicating where each camera 20 is located and in what direction it is capturing images. The integrated tracking unit 200 may also include, as the information about the placement of the cameras 20, information about the position and direction of lighting in the capturing space, information about the lighting characteristics, and information about lighting conditions such as which positions in the capturing space are bright or dark. The integrated tracking unit 200 may also hold directional information about the capturing space as information about the placement of the cameras 20.

[0243] Similar to the integrated tracking unit 200 according to each of the above-described embodiments, the integrated tracking unit 200 predicts the movement of the object indicated by each tracker in the current frame and determines the position of the tracker by associating the target with the tracker.

[0244] The integrated tracking unit 200 then predicts the position of the object indicated by each tracker on the captured image taken by each camera 20, based on the calculated tracker positions and information on the movement of each tracker.

[0245] Then, the integrated tracking unit 200 refers to information about the placement of each camera 20, and predicts how each object at the predicted position will look (appear) when viewed from each camera 20. In other words, the integrated tracking unit 200 predicts whether or not the objects indicated by the trackers of each of the multiple cameras 20 will overlap at the next timing of capturing an image.

[0246] Then, when the integrated tracking unit 200 determines, based on the predicted appearance, that objects overlap in an image captured by a certain camera 20, it generates information (appearance information) indicating that the objects may not appear to overlap.

[0247] For example, the integrated tracking unit 200 determines whether a predicted object is located between a line segment connecting a camera 20 and a predicted object. If the predicted object is located between the line segment, there is a high possibility that the object overlaps with the other object. Therefore, the integrated tracking unit 200 obtains information about the hidden (overlapping) object and the degree of overlap (likelihood) as visibility information for the object.

[0248] The integrated tracking unit 200 may then include this determination result as visibility information in the tracking result of this object.

[0249] The integrated tracking unit 200 then includes the generated visibility information in tracking information for an object that is likely to disappear from an image captured by a certain camera 20. In this case, the visibility information preferably includes information indicating the camera 20 from which the object is likely to disappear.

[0250] The integrated tracking unit 200 then outputs the tracking information including the appearance information to each detection unit 100. The integrated tracking unit 200 may also output the tracking information including the appearance information to a detection unit 100 associated with a camera 20 (a certain camera 20) where there is a high possibility that the object is overlapping and cannot be seen. The integrated tracking unit 200 may also output the tracking information not including the appearance information to an object tracking device 10 associated with another camera 20.

[0251] In addition, if it is known that the lighting conditions change depending on the object's position, causing the color and brightness of the object to change, the integrated tracking unit 200 may include information in the tracking information that describes the change in appearance depending on the object's position.

[0252] For example, if the lighting conditions depend on the object's position, the integrated tracking unit 200 may predict the lighting conditions from the object's position and include information such as whether the lighting conditions will become brighter, darker, or change in color in the tracking information for each tracker.

[0253] Furthermore, if the position of the light in the space where the object is placed is known, the integrated tracking unit 200 determines whether the shadow of the object or another object placed in this environment overlaps with another object, based on the relationship between the light and the object. If there is a possibility of overlapping shadows, the integrated tracking unit 200 calculates the possibility (likelihood) of the shadows overlapping for the object with which the shadows overlap, and includes this in the tracking information.

[0254] Furthermore, even in the case of a moving object such as the sun, the integrated tracking unit 200 may determine the position of the sun from information on the time and the direction of the site, predict the direction in which a shadow will be cast, and consider the impact on the appearance of the object. For example, the integrated tracking unit 200 may determine the current position of the sun from time information and, in combination with direction information, predict the direction in which a shadow will be cast. Then, when there is a possibility that a shadow will be cast by another object, the integrated tracking unit 200 may calculate the possibility (likelihood) of the shadows overlapping and include this in the tracking information.

[0255] Next, the operation of the detection unit 100 will be described. The detection unit 100 detects objects based on tracking information, as in the above-described embodiments. At this time, the detection unit 100 of the object tracking device 10 according to this embodiment controls object detection using information on the visibility of each object included in the tracking information. Specifically, the detection unit 100 does not detect objects that are likely to be invisible because they overlap with other objects. For example, the detection unit 100 does not set a search range for these objects that are likely to be invisible.

[0256] If the tracking information contains information that changes in brightness or color, the detection unit 100 may correct the effect of the lighting when searching and then perform detection. For example, in a dark area, the detection unit 100 may correct the pixel values of that area to be brighter before performing detection.

[0257] Furthermore, if the color tone changes, the detection unit 100 may take the color tone change into account when updating the matching parameters used in template matching (i.e., the detection parameters described above) and correct the color information of the parameters. Furthermore, if the color tone change is significant, the detection unit 100 may not use the color information. Furthermore, the detection unit 100 may perform matching by lowering the weight of color information among the template features and increasing the weight of other features such as edges.

[0258] Furthermore, when the detection unit 100 updates the detection parameters used in the object detection process, if it is determined that there is a high possibility that the objects are overlapping, the detection parameters may not be updated.

[0259] As described above, the detection unit 100 controls object detection based on the appearance information included in the tracking information. This allows the detection unit 100 to reduce the processing required to detect invisible objects and to update parameters such as template matching. This allows the detection unit 100 to reduce the possibility of false detection and erroneous parameter updates.

[0260] Similarly, when it is highly likely that the lighting conditions have changed, the detection unit 100 may perform control to correct the effect of the change and update the parameters, or may perform control not to update the parameters.

[0261] Furthermore, when multiple detection algorithms can be switched, the object tracking device 10 may use a detector as the detection unit 100 that is more resistant to overlapping. Specifically, the object tracking device 10 may normally use a detector that detects the entire head, but when overlapping occurs, a detector that detects only a portion of the head rather than the entire head may be used. This allows the object tracking device 10 to normally use a simple detector, and when there is a possibility of overlapping, to use a more detailed detector. This enables the object tracking device 10 to perform highly accurate detection while maintaining efficiency. Similarly, when lighting conditions change, the object tracking device 10 may control detection using a detector (feature) that is highly robust against the lighting conditions.

[0262] As described above, in the object tracking device 10 according to this embodiment, the integrated tracking unit 200 determines the appearance of an object using information from other cameras as well. Therefore, the integrated tracking unit 200 can determine the appearance with high accuracy even when it is difficult to determine the appearance using a certain camera 20 alone, such as when objects overlap each other in an image captured by that camera 20. The integrated tracking unit 200 can then feed back this result to the detection unit 100. This allows the detection unit 100 to reduce false detection of objects. The integrated tracking unit 200 then uses this detection result to track objects, thereby improving the accuracy of object tracking.

[0263] <Hardware configuration example> Here, an example of a hardware configuration capable of realizing the object tracking device (10, 50) according to each of the above-described embodiments will be described. The above-described object tracking device (10, 50) may be realized as a dedicated device, or may be realized using a computer (information processing device).

[0264] FIG. 19 is a diagram illustrating an example of the hardware configuration of a computer (information processing device) capable of implementing each embodiment of the present invention.

[0265] The hardware of the information processing device (computer) 700 shown in FIG. 19 includes a CPU (Central Processing Unit) 11, a communication interface (I / F) 12, an input / output user interface 13, a ROM (Read Only Memory) 14, a RAM (Random Access Memory) 15, a storage device 17, and a drive device 18 for a computer-readable storage medium 19, all of which are connected via a bus 16. The input / output user interface 13 is a man-machine interface such as a keyboard, which is an example of an input device, and a display, which is an output device. The communication interface 12 is a general communication means for the devices according to the above-described embodiments (FIGS. 1 and 9) to communicate with external devices via a communication network 600. In this hardware configuration, the CPU 11 controls the overall operation of the information processing device 700 that realizes the object tracking device (10, 50) according to the respective embodiments.

[0266] The present invention, which has been described using the above-mentioned embodiments as examples, is achieved, for example, by supplying a program (computer program) capable of realizing the processes described in the above-mentioned embodiments to information processing device 700 shown in Fig. 19, and then reading and executing the program in CPU 11. Note that such a program may be, for example, a program capable of realizing the various processes described in the flowchart (Fig. 8) referred to in the description of the above-mentioned embodiments, or a program capable of realizing each unit (each block) shown in the device in the block diagrams shown in Figs. 1, 4 to 6, 9, and 15 to 17.

[0267] The programs provided to the information processing device 700 may be stored in a readable / writable temporary memory (15) or a non-volatile storage device (17) such as a hard disk drive. That is, in the storage device 17, the program group 17A may be programs capable of implementing the functions of the various units shown in the object tracking devices (10, 50) in the above-described embodiments. The various stored information 17B may be, for example, the object tracking results, camera images, camera parameters, and the range of coordinate values in a common coordinate system visible by each camera 20 in the above-described embodiments. However, when implementing the programs in the information processing device 700, the constituent units of the individual program modules are not limited to the divisions of the blocks shown in the block diagrams (FIGS. 1, 4 to 6, 9, 15 to 17), and may be appropriately selected by those skilled in the art.

[0268] In the above case, the method of supplying the program to the device can be a currently common method, such as installing the program in the device via a computer-readable recording medium (19) such as a CD (Compact Disk)-ROM or flash memory, or downloading the program from an external source via a communication line (600) such as the Internet. In such a case, the present invention can be considered to be constituted by the code (program group 17A) constituting the computer program or the storage medium (19) on which the code is stored.

[0269] The present invention has been described above as an example of application to the exemplary embodiments described above. However, the technical scope of the present invention is not limited to the scope described in the above-mentioned embodiments. It is clear to those skilled in the art that various modifications and improvements can be made to such embodiments. In such cases, new embodiments incorporating such modifications or improvements may also fall within the technical scope of the present invention. This is clear from the matters described in the claims. [Explanation of symbols]

[0270] 1. Object Tracking System 2. Object Tracking System 10 Object Tracking Device 100 Detector 110 Object detection unit 111 Recognition-type object detection unit 112 Search range setting section 120 Common coordinate transformation unit 130 Individual coordinate transformation unit 140 Object detection unit 141 Recognition-based object detection unit 142 Non-recognition object detection unit 143 Detection parameter update unit 144 Detection result integration unit 150 Display control unit 160 Storage section 200 Integrated Tracking Unit 210 Prediction Department 220 Storage section 230 Mapping section 240 Update Department 300 Display control unit 400 Detector 500 Integrated Tracking Unit 510 Buffer section 530 Mapping section 20 Camera 30 Display device 40 Network 50 Object Tracking Device

Claims

1. A process of acquiring, based on a first image captured by a first camera, a first feature representing an object appearing in the first image; A process of acquiring, based on a second image captured by a second camera, a second feature representing the object appearing in the second image; obtaining a third feature representing the object based on the first feature and the second feature; a process of displaying the third feature in a three-dimensional space representing a space photographed by the first camera and the second camera on a display device; and the process of acquiring the first feature and the process of acquiring the second feature include a process of detecting an object from an image using a neural network; A neural network for detecting the object is associated with each camera. Information processing device.

2. the third feature represents a position of the object; The information processing device according to claim 1 .

3. The position of the object is a position in the three-dimensional space. The information processing device according to claim 2 .

4. the third characteristic represents a size of the object; The information processing device according to claim 1 .

5. the process of displaying the third characteristic includes a process of displaying the three-dimensional space from a viewpoint looking down from above. The information processing device according to claim 1 .

6. The object includes a moving object. The information processing device according to claim 1 .

7. Moving objects include people. The information processing device according to claim 6 .

8. acquiring, based on a first image captured by a first camera, a first feature representing an object appearing in the first image; acquiring, based on a second image captured by a second camera, a second feature representing the object appearing in the second image; obtaining a third feature representative of the object based on the first feature and the second feature; displaying the third feature in a three-dimensional space representing a space photographed by the first camera and the second camera on a display device; detecting an object from an image using a neural network in acquiring the first feature and the second feature; A neural network for detecting the object is associated with each camera. Information processing methods.

9. The third feature represents a position of the object. The information processing method according to claim 8.

10. The position of the object is a position within the three-dimensional space. The information processing method according to claim 9.

11. The third feature represents the size of the object. The information processing method according to any one of claims 8 to 10.

12. The process of displaying the third feature includes a process of displaying the three-dimensional space from a perspective looking down from above. The information processing method according to any one of claims 8 to 11.

13. The object includes a moving object. The information processing method according to any one of claims 8 to 12.

14. The moving object includes a person. The information processing method according to claim 13.

15. On the computer, A process of acquiring, based on a first image captured by a first camera, a first feature representing an object appearing in the first image; A process of acquiring, based on a second image captured by a second camera, a second feature representing the object appearing in the second image; obtaining a third feature representing the object based on the first feature and the second feature; a process of displaying the third feature in a three-dimensional space representing a space photographed by the first camera and the second camera on a display device; Execute the process of acquiring the first feature and the process of acquiring the second feature include a process of detecting an object from an image using a neural network; A neural network for detecting the object is associated with each camera. Information processing program.

16. A process of acquiring, based on a first image captured by a first camera, a first feature representing an object appearing in the first image; A process of acquiring, based on a second image captured by a second camera, a second feature representing the object appearing in the second image; obtaining a third feature representing the object based on the first feature and the second feature; a process of displaying the third feature in a three-dimensional space representing a space photographed by the first camera and the second camera on a display device; and the process of acquiring the first feature and the process of acquiring the second feature include a process of detecting an object from an image using a neural network; A neural network for detecting the object is associated with each camera. Information processing system.

Images