Tracking system, tracking method and tracking program

Abstract

To provide a tracking system capable of enhancing the tracking precision when the same target is tracked by plural cameras.SOLUTION: A tracking system for tracking the same target with plural cameras comprises a first camera 100 for capturing a first picture, a second camera 200 for capturing a second picture which is captured at substantially the same angle of view as the first camera 100, and a processor 80. The processor 80 performs first tracking processing for tracking a target on the first picture, second tracking processing for tracking the target on the second picture, reliability degree calculation processing for calculating a first reliability degree as the reliability degree of a target obtained by the first tracking processing, and a second reliability degree as the reliability of the target obtained by the second tracking processing, and control processing for comparing the first reliability degree and the second reliability degree with each other to specify tracking processing having a higher reliability degree, and controlling one tracking processing based on the specified other tracking processing result.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a tracking system, a tracking method, and a tracking program for tracking the same object using multiple cameras. [Background technology]

[0002] Various methods have been proposed for tracking an object when the user specifies an object in an input image sequence or video.

[0003] Patent Document 1 describes a moving object detection device that detects moving objects from moving images. The device described in Patent Document 1 removes false optical flows by using the optical flows of moving objects in current and past image frames.

[0004] Also known is a method of tracking an object using multiple cameras. Patent Document 2 describes a monitoring system that tracks and monitors a moving object by linking multiple cameras. In the monitoring system described in Patent Document 2, multiple monitoring cameras exchange characteristic information of the tracking object between each other, and each monitoring camera tracks the tracking object using the characteristic information.

[0005] Patent Document 3 describes an image processing device that processes visible light images to detect and track objects that have entered a target area. A visible light camera and a far-infrared camera are connected to the image processing device described in Patent Document 3, and both cameras capture images of the target area in which objects are to be detected from approximately the same angle and at approximately the same imaging magnification. Furthermore, when there is an image area in the visible light image that is not suitable for object detection, the image processing device described in Patent Document 3 uses the far-infrared background image to generate a far-infrared background subtraction image and detects the object being captured.

[0006] Incidentally, Patent Document 4 describes a target detection device that detects targets at high speed. The target detection device described in Patent Document 4 includes detectors that detect far-infrared images and near-infrared images that are input through a lens. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent No. 3810657 [Patent Document 2] Japanese Patent Application Laid-Open No. 2003-324720 [Patent Document 3] Japanese Patent Application Laid-Open No. 2014-157452 [Patent Document 4] Japanese Patent Application Publication No. 6-174828 Summary of the Invention [Problem to be solved by the invention]

[0008] When tracking an object using a single camera, as in the device described in Patent Document 1, there are environments in which tracking cannot be performed properly depending on the characteristics of the camera. For example, when tracking using a single visible light camera, the color of the object can be considered an effective feature for tracking. However, when the tracking environment is dark or other environments where color differences are difficult to detect, it is difficult to track the object using a single visible light camera. Furthermore, when tracking using a single far-infrared camera, for example, while features can be obtained by temperature regardless of day or night, tracking the object is difficult if other objects with the same temperature are present.

[0009] In addition, in the surveillance system described in Patent Document 2, when a surveillance camera device loses sight of a tracking target, it transmits extracted feature information to other surveillance camera devices, and the other surveillance camera devices start transmitting video if the transmitted feature information matches. However, since the shooting positions and angle of view of each surveillance camera device are usually different, it is difficult to say that the feature information received from other surveillance camera devices can be used with high accuracy for subsequent tracking.

[0010] On the other hand, it is possible to improve the reliability of tracking by tracking an object using multiple cameras, as in the image processing device described in Patent Document 3. However, when tracking using multiple cameras simultaneously, such as stereo vision, there are problems with differences in the angle of view and differences in appearance due to parallax. Ideally, if the angle of view is the same, the position of the object does not change between cameras, making processing very easy. However, since the angle of view varies depending on subtle differences in the size, focal length, lens distortion, etc. of the camera sensor, it is difficult to eliminate the difference in the angle of view when using multiple cameras.

[0011] Furthermore, even if the angle of view is adjusted to be approximately the same by using cameras of the same model number, parallax cannot be eliminated in principle as long as multiple cameras are used. If image processing is performed without taking into account the difference in appearance due to parallax and assuming that the angle of view is approximately the same, the tracking target cannot be accurately captured because this assumption is unreasonable, resulting in a problem of reduced tracking accuracy.

[0012] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide a tracking system, a tracking method, and a tracking program that can improve the tracking accuracy when tracking the same object with multiple cameras. [Means for solving the problem]

[0013] The tracking system according to the present invention is a tracking system for tracking the same object using a plurality of cameras, and includes a processing unit that receives images from at least two or more cameras, and the processing unit: visible light a first detection process for detecting an object in a first image captured by the camera; Infrared a second detection process for detecting an object in a second image captured by the camera; visible light Camera and Infrared Based on the position of the object in the image of a specific frame captured by each camera, visible light Camera and Infraredand a control process for controlling a tracking process for tracking an object on an image captured by at least one of the cameras.

[0015] The tracking method according to the present invention comprises: visible light a first detection process for detecting an object in a first image captured by the camera; Infrared a second detection process for detecting an object in a second image captured by the camera; visible light Camera and Infrared Based on the position of the object in the image of a specific frame captured by each camera, visible light Camera and Infrared and a control process for controlling a tracking process for tracking an object on an image captured by at least one of the cameras.

[0016] The tracking program according to the present invention includes: visible light a first detection process for detecting an object in a first image captured by the camera; Infrared a second detection process for detecting an object in a second image captured by the camera; and visible light Camera and Infrared Based on the position of the object in the image of a specific frame captured by each camera, visible light Camera and Infrared The control process is characterized by executing a control process for controlling a tracking process for tracking an object on an image captured by at least one of the cameras. [Effects of the Invention]

[0017] According to the present invention, it is possible to improve the tracking accuracy when tracking the same object with multiple cameras. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a block diagram illustrating one embodiment of a tracking system in accordance with the present invention. [Figure 2]FIG. 10 is an explanatory diagram showing an example in which two videos are displayed side by side. [Figure 3] FIG. 10 is an explanatory diagram showing an example of displaying a tracking result. [Figure 4] FIG. 1 is an explanatory diagram showing an example of the operation of the tracking system. [Figure 5] FIG. 10 is a block diagram illustrating a modification of the tracking system according to the present invention. [Figure 6] 1 is a block diagram showing an overview of a tracking system according to the present invention; [Figure 7] FIG. 2 is a block diagram showing another overview of the tracking system according to the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0019] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. In the present invention, an object refers to any person or object to be tracked.

[0020] 1 is a block diagram showing an embodiment of a tracking system according to the present invention. The tracking system of this embodiment tracks the same object using multiple cameras, and includes a first camera 10, a second camera 20, a processing unit 30, an output unit 40, and a storage unit 50.

[0021] In the following description, an example is given in which the tracking system has two cameras (first camera 10 and second camera 20), but the number of cameras the tracking system has is not limited to two and may be three or more.

[0022] In this embodiment, the first camera 10 and the second camera 20 are used to track the same object. Specifically, one camera captures an image having substantially the same angle of view as the other camera. In the following description, the image captured by the first camera will be referred to as the first image, and the image captured by the second camera will be referred to as the second image. The second image is captured with substantially the same angle of view as the first image.

[0023] The positional relationship between the first camera 10 and the second camera 20 may be set so that a target object can be tracked by stereoscopic vision. Here, the positional relationship refers to the position, orientation, and magnification of the cameras that are set so that two images are captured with approximately the same angle of view, as described above.

[0024] Due to the nature of tracking the same object using multiple cameras, it is theoretically impossible to eliminate parallax. Therefore, the angle of view of one camera is used as the reference and the angle of view of each of the other cameras (image range, size at a certain distance, etc.) is adjusted so that they are approximately the same. This setting is the same even when there are three or more cameras.

[0025] The cameras may be set using a device that fixes their relative positions. Furthermore, as long as their relative positions do not change, the installation locations of the first camera 10 and the second camera 20 do not need to be fixed. For example, these cameras may be attached to the same moving object and move, changing the angle of view from the initial state.

[0026] Each camera is set so that the target object is present within its field of view when tracking begins. However, the target object does not necessarily need to be present within its field of view at any time other than when tracking begins. First camera 10 and second camera 20 input the captured images to processing unit 30.

[0027] The cameras used in this embodiment may be any type as long as they are capable of capturing an image of an object. Specifically, the first camera 10 and the second camera 20 may be realized as cameras equipped with the same type of sensor, or may be realized as cameras equipped with different types of sensors. For example, the first camera may be a visible light camera, and the second camera may be a far-infrared camera. Note that the first camera 10 and the second camera 20 are not limited to visible light cameras and far-infrared cameras, and may be, for example, near-infrared cameras.

[0028] Furthermore, the frame rates of the first camera 10 and the second camera 20 may be the same or different. For example, the first camera 10 may capture high-resolution images at a lower frame rate than the second camera 20, and the second camera 20 may capture low-resolution images at a higher frame rate than the first camera 10.

[0029] The storage unit 50 stores information used for various processes by the processing unit 30. The storage unit 50 is realized by, for example, a magnetic disk device.

[0030] The output unit 40 displays the video input to the processing unit 30. For example, the output unit 40 may display two videos taken at the same time side by side so that they can be compared. The output unit 40 is realized by, for example, a display device or a touch panel device. FIG. 2 is an explanatory diagram showing an example of two videos displayed side by side. In the example shown in FIG. 2, the video taken by the visible light camera is displayed on the left, and the video taken by the far-infrared camera is displayed on the right.

[0031] The processing unit 30 inputs an image captured by the first camera 10 (i.e., a first image) and an image captured by the second camera 20 (i.e., a second image). The processing unit 30 receives a user's designation of an object in one of the input images. In this embodiment, the processing unit 30 receives the designation of an object in the first image.

[0032] The object is specified by selecting a range that includes the object on the first image. The method of specifying the object is arbitrary. For example, if the output unit 40 is realized by a display device, the object may be specified by a range selection operation using a pointing device on the first image displayed on the output unit 40. Furthermore, for example, if the output unit 40 is realized by a touch panel, the object may be specified by a range selection operation by the user on the first image displayed on the output unit 40.

[0033] When an object in the first image is specified, the processing unit 30 specifies a specified range that includes the object as a search range. Hereinafter, this search range may be referred to as a first search range. In addition, the processing unit 30 specifies a range in the second image in which the same object exists as a search range. Hereinafter, this search range may be referred to as a second search range.

[0034] Here, the search range is the area where the object detection process is performed. Since the detection process is a process of searching for and detecting an object that has the same appearance as the learned tracking target, the search range can also be said to be the range on the image where the search is performed. Note that the search is a process in which all grids obtained by dividing the image into predetermined sizes (for example, 5 pixels x 5 pixels, 10 pixels x 10 pixels, etc.) are used as candidates, and an object that has the same appearance as the learned tracking target is searched for among these candidates and determined as the detection result.

[0035] As described above, parallax exists between the first and second images. Therefore, the processing unit 30 determines the second search range by taking the parallax (misalignment) into account. For example, the processing unit 30 targets a rectangular area specified in the first image, compares the frame immediately before the specified time with the frame at the specified time, and estimates the magnitude and direction of the object's movement. The processing unit 30 may estimate the magnitude and direction of the movement using, for example, optical flow. Next, the processing unit 30 determines a rectangle in the second image that is located at the same position as the rectangle specified in the first image. As with the first image, the processing unit 30 compares the frame immediately before the specified time with the frame at the specified time, and estimates the magnitude and direction of the object's movement. The processing unit 30 then determines the second search range by estimating that a location where the estimated size and direction of the object are similar is where the specified object is located in the second image.

[0036] In this way, the search range in one video is determined based on the magnitude and direction of the movement of the object in the other video, allowing the object to be identified while taking into account any discrepancies between the two videos. Furthermore, by automatically determining the object in the other video, even if the object is not specified, the effort required to specify the object can be reduced. In other words, in this embodiment, the user only needs to specify the object in one video, and the object in the other video is automatically estimated. Therefore, despite the tracking technology using multiple heterogeneous cameras, the effort required for input does not increase compared to the case of a single camera. Thereafter, the processing unit 30 determines the search range for the object in parallel from the video captured by each camera and performs tracking processing.

[0037] After identifying each search range, the processing unit 30 calculates the reliability of the object for each video at regular frame intervals. This reliability is a similarity indicating the likelihood of the object, and is calculated for each video based on tracking information from the start of tracking. Here, the tracking information from the start of tracking includes each image showing the object acquired during tracking and information indicating each position of the object (hereinafter referred to as a template). A template is created for each video (for each camera used for shooting).

[0038] The processing unit 30 calculates the reliability of the object based on the degree of similarity in appearance when compared with a template, and the degree of proximity and likelihood of the object's position estimated according to its movement. At this time, the processing unit 30 may calculate the reliability of the object in each frame while constantly updating the object template. The processing unit 30 may also learn an object model (appearance model, movement model) based on images constantly acquired during search and information indicating the movement of the object, and calculate the reliability of the object using the learned model.

[0039] The processing unit 30 may update the object template based on the reliability calculated for each image, or may update the object template based on the relative reliability (relative reliability) when comparing the reliability calculated for each image. For example, if the calculated reliability is lower than the reliability calculated for other images, the processing unit 30 may lower (or set to 0) the update weight for the object identified in that image. Also, for example, if all the reliability levels are high but the relative reliability is not high, the processing unit 30 may lower (or set to 0) the update weight for the object in this case as well.

[0040] Any method can be used as long as it can calculate the reliability of an object based on information acquired in the tracking process of the object in each video (specifically, tracking information from the start of tracking). In other words, the reliability is a degree of likelihood that the object is a tracking object, and it can be calculated using an equation that determines that the higher the value, the higher the similarity.

[0041] Next, the processing unit 30 reflects the position and size of the object in one tracking process determined to have high reliability in setting the search range for the object in the other tracking process. Specifically, the processing unit 30 compares the reliability calculated from each video and identifies the video and object for which the highest reliability was calculated at the same time. The processing unit 30 then determines the search range for the other videos other than the video for which the highest reliability was calculated, using information indicating the range (position and size) of the identified object. In other words, it can be said that the processing unit 30 shares the information of the object for which the highest reliability was calculated with all other videos. Thereafter, the tracking process continues within the search range determined for each video.

[0042] The processing unit 30 may change the method for determining the search range depending on the calculated reliability and the tracking result of the previous frame. Specifically, the processing unit 30 may change the method for determining the search range depending on whether a tracking result of the previous frame exists or whether the reliability is high or low. For example, if a tracking result of the previous frame does not exist or if the reliability is low, the processing unit 30 searches for the tracking target within the entire image as the search range. At this time, the processing unit 30 may identify the target from within each entire image using only templates whose size is close to the size of the target identified in the most recent past tracking result.

[0043] In this way, by narrowing the search range of each camera image and searching for and adopting the candidate with the highest reliability within that range, the search time can be shortened and the tracking result can be adjusted. Note that if the camera image of the shared party does not have a reliability above a certain level or if there is no matching template, the processing unit 30 may adopt the shared tracking information as is.

[0044] In this embodiment, since the object is tracked complementarily using multiple cameras, parallax occurs between the cameras. By utilizing this parallax, the processing unit 30 may determine the search range using the depth from the camera to the object.

[0045] When determining the search range using depth, the user first performs calibration to calculate in advance the distance to the target, based on knowledge of the degree of deviation that will occur within a certain depth range, and stores the calculation results in the storage unit 50. That is, the storage unit 50 stores the deviation (difference) corresponding to the distance to the target and the depth range calculated through calibration. The processing unit 30 calculates the depth to the target based on the deviation observed during actual tracking, and determines the search range by estimating the movement range of the target in the image of the current frame from the depth calculated in the image of the previous frame. Narrowing the search range in this way makes it possible to apply a template to only a portion of the image, thereby improving the speed of the tracking process.

[0046] Alternatively, the storage unit 50 may store a correspondence between the depth range and the search range. In this case, the processing unit 30 may calculate the depth range of the target object based on the deviation observed during actual tracking, and determine a search range to be set for another tracking process from the calculated depth range based on the correspondence stored in the storage unit 50.

[0047] After determining the search range, the processing unit 30 may perform tracking processing (target object detection processing) again and may also display the tracking results at any time, for example, by surrounding the target object with a rectangle on the video displayed on the output unit 40. Displaying such tracking results makes it possible to visually track the target object.

[0048] At this time, the processing unit 30 may display the tracking results on the video with higher reliability in a specific manner. For example, the processing unit 30 may display the object on the video with higher reliability surrounded by a rectangle with a thick frame in a dark color, and the object on the video with lower reliability surrounded by a rectangle with a thin frame in a light color (or not surrounded by any other object).

[0049] Fig. 3 is an explanatory diagram showing an example of a display of tracking results. The example shown in Fig. 3 shows that, as a result of performing tracking processing on the video shown in Fig. 2, the reliability of the tracking results from the video captured by the second camera was consistently higher than that of the video captured by the first camera. This makes it possible to visually understand which video's tracking results of the object are being used to perform the tracking processing.

[0050] The processing unit 30 is realized by a CPU of a computer that operates according to a program (tracking program). For example, the program may be stored in the storage unit 50, and the CPU may read the program and operate as the processing unit 30 according to the program. The processing unit 30 may also be realized by dedicated hardware.

[0051] Next, the operation of the tracking system of this embodiment will be described. Fig. 4 is an explanatory diagram showing an example of the operation of the tracking system of this embodiment. First, image adjustment is performed in advance between the image captured by the first camera 10 (i.e., the first image) and the image captured by the second camera 20 (i.e., the second image) (step S11). Specifically, the first camera 10 and the second camera 20 are adjusted so that they can capture images with approximately the same angle of view.

[0052] After the adjustment, the processing unit 30 receives a user's designation of the object in the first image (step S12). When the object in the first image is designated, the processing unit 30 specifies a search range in the first image (first search range) and also specifies a search range in the second image (second search range) (step S13).

[0053] Once the search range in each video has been identified, the processing unit 30 tracks the object in each video (steps S14 and S15) and calculates the reliability of the tracked object at regular frame intervals (steps S16 and S17). The processing unit 30 compares the reliability calculated for each video and identifies the video and object with the highest reliability calculated (step S18).

[0054] The processing unit 30 controls the other tracking process based on the identified tracking process result (steps S19 and S20). Specifically, the processing unit 30 determines the search range by adjusting the tracking result, and repeats the processes from step S14 to step S15 onward for the image of the next frame.

[0055] As described above, in this embodiment, the first camera 10 captures a first video, and the second camera 20 captures a second video with approximately the same angle of view as the first camera. The processing unit 30 included in the tracking system then executes a first tracking process for tracking an object in the first video and a second tracking process for tracking the object in the second video. The processing unit 30 also executes a reliability calculation process for calculating the reliability of the object obtained in the first tracking process (first reliability) and the reliability of the object obtained in the second tracking process (second reliability). The processing unit 30 then compares the first reliability with the second reliability to identify a tracking process with a higher reliability, and executes a control process to control the other tracking process based on the results of the identified tracking process. This improves the tracking accuracy when tracking the same object with multiple cameras.

[0056] Furthermore, in this embodiment, the misalignment in appearance (parallax) that occurs when using multiple cameras, such as stereoscopic vision, is taken into consideration, and the search range is fine-tuned (corrected) by sharing the tracking results, thereby enabling more robust tracking of the target object.

[0057] Furthermore, in this embodiment, the reliability of the object is calculated independently for each image, and tracking information of the object identified from the image with high reliability is reflected in the other images, regardless of whether the sensors or frame rates of the cameras used for tracking are the same. Therefore, since the object can be tracked complementarily using different types of cameras, it is possible to improve tracking accuracy. In other words, in this invention, since tracking is performed based on the tracking information with the highest reliability among multiple cameras, it can be said that the scene in which each camera has advantageous characteristics is automatically selected and tracked.

[0058] Specifically, in this embodiment, images from multiple cameras of the same or different types (for example, a far-infrared camera and a visible light camera) with approximately the same angle of view are used to simultaneously track an object, and the tracking information from the more reliable side is shared with the less reliable side to correct the tracking information. For example, when cameras with different types of sensors are combined, it becomes possible to track scenes where one type of camera is weak, while the other type of camera complements it.

[0059] Furthermore, in this embodiment, since it is sufficient to combine existing single cameras and adjust the angle of view of each camera to be approximately the same, there is no need for a special lens mechanism as described in Patent Document 4. Furthermore, as long as the positional relationship between the cameras (relationships such as camera position, orientation, and magnification) is not changed, the target object can be tracked even if the cameras are moved during tracking. Therefore, the tracking system of this embodiment can also be applied to a mobile camera system.

[0060] Next, a modified example of this embodiment will be described. In the example shown in Fig. 1, the tracking system includes a processing unit 30, which performs tracking processing and reliability calculation processing on both the image input from the first camera 10 (first image) and the image input from the second camera (second image).

[0061] In this modification, a configuration will be described in which the processing of the images captured by each camera is realized by separate means. Fig. 5 is a block diagram showing a modification of the tracking system according to the present invention. The tracking system illustrated in Fig. 5 includes a first camera 10, a second camera 20, a tracking unit 31, a tracking unit 32, a reliability calculation unit 33, a reliability calculation unit 34, a control unit 35, an output unit 40, and a storage unit 50.

[0062] The contents of first camera 10, second camera 20, output unit 40, and storage unit 50 are the same as those in the above embodiment, and therefore detailed description thereof will be omitted. In addition, the contents performed by tracking unit 31, tracking unit 32, reliability calculation unit 33, reliability calculation unit 34, and control unit 35 are the same as those performed by processing unit 30 in the above embodiment.

[0063] Specifically, tracking unit 31 and reliability calculation unit 33 perform processing on the video (first video) captured by first camera 10. Tracking unit 32 and reliability calculation unit 34 perform processing on the video (second video) captured by second camera 20. Note that the processing performed by tracking unit 31 and the processing performed by tracking unit 32, and the processing performed by reliability calculation unit 33 and the processing performed by reliability calculation unit 34 are the same except for the target video or image.

[0064] The tracking unit 31 tracks the object on the first video (first tracking process). The tracking unit 32 tracks the object on the second video (second tracking process). The method of tracking the object is the same as the method performed by the processing unit 30 described above.

[0065] The reliability calculation unit 33 calculates the reliability (first reliability) of the object obtained in the first tracking process. The reliability calculation unit 34 calculates the reliability (second reliability) of the object obtained in the second tracking process. The reliability calculation method is the same as the method performed by the processing unit 30 described above.

[0066] The control unit 35 compares the first reliability with the second reliability, identifies a tracking process with a higher reliability, and controls the other tracking process based on the results of the identified tracking process. Specifically, the control unit 35 reflects the position and size of the object in one tracking process determined to have a higher reliability in setting the search range for the object in the other tracking process. Note that the method of reflecting the search range setting is the same as the method performed by the processing unit 30 described above.

[0067] The tracking unit 31, the tracking unit 32, the reliability calculation unit 33, the reliability calculation unit 34, and the control unit 35 are realized by the CPU of a computer that operates according to a program (communication information reference program). Furthermore, the tracking unit 31, the tracking unit 32, the reliability calculation unit 33, the reliability calculation unit 34, and the control unit 35 may each be realized by dedicated hardware.

[0068] Also, for example, the first camera 10, the tracking unit 31, and the reliability calculation unit 33 may be configured as an integrated tracking device, and the second camera 20, the tracking unit 32, and the reliability calculation unit 34 may be configured as an integrated tracking device. Then, each tracking device may perform each process in response to an instruction from the control unit 35.

[0069] This configuration also makes it possible to improve the tracking accuracy when tracking the same object with multiple cameras.

[0070] Next, an overview of the present invention will be described. Fig. 6 is a block diagram showing an overview of a tracking system according to the present invention. The tracking system according to the present invention is a tracking system that tracks the same object (e.g., a tracking target) using multiple cameras, and includes a first camera 100 (e.g., first camera 10) that captures a first video, a second camera 200 (e.g., second camera 20) that captures a second video captured at approximately the same angle of view as the first camera 100, and a processing unit 80 (e.g., processing unit 30).

[0071] The processing unit 80 executes a first tracking process for tracking an object on a first image, a second tracking process for tracking an object on a second image, a reliability calculation process for calculating a first reliability which is the reliability of the object obtained in the first tracking process and a second reliability which is the reliability of the object obtained in the second tracking process, and a control process for comparing the first reliability and the second reliability to identify a tracking process with a higher reliability, and controlling the other tracking process based on the results of the identified tracking process.

[0072] Such a configuration can improve the tracking accuracy when tracking the same object with multiple cameras.

[0073] Here, the first camera 100 and the second camera 200 may be realized by cameras equipped with different types of sensors. When each camera is realized by cameras equipped with different types of sensors, the content of the images captured by each camera will be significantly different. However, in the present invention, the reliability of each captured image is calculated, and the tracking process of the other image is controlled based on the tracking process result (e.g., search range) of the image for which a higher reliability is calculated. Therefore, it becomes possible to track scenes that are difficult for one type of camera by complementing them with the other type of camera.

[0074] Specifically, the first camera 100 may be a visible light camera, and the second camera 200 may be a far-infrared camera. In this case, it is possible to track the object in a particularly complementary manner, making it possible to make better use of the characteristics of each camera.

[0075] Furthermore, the first camera 100 may capture high-resolution video at a lower frame rate than the second camera 200, and the second camera 200 may capture low-resolution video at a higher frame rate than the first camera 100.

[0076] Furthermore, the processing unit 80 may reflect the position and size of the object in one tracking process determined to be highly reliable in setting the search range for the object in the other tracking process. That is, since the search range for the other tracking process is determined based on the tracking results of the highly reliable object, tracking can be performed while correcting the search range even when multiple cameras with different angles of view are used. This makes it possible to track the same object with high accuracy. In other words, in the present invention, by reflecting the results of one tracking process, the search range for the other image can be set to accommodate the difference in appearance that occurs when multiple cameras are used.

[0077] The tracking system may also include a storage unit (e.g., storage unit 50) that stores the depth range and the search range in association with each other. The processing unit 80 may then refer to the storage unit and specify the search range from the depth range of the object calculated based on the difference observed during tracking.

[0078] Furthermore, the first camera 100 and the second camera 200 may be attached to the same moving body and track the same object. In the present invention, as long as both cameras are installed so that they can capture images with approximately the same angle of view, the angle of view does not need to match exactly, and therefore the accuracy of the tracking process performed when both cameras are attached to the same moving body can also be improved.

[0079] Furthermore, the processing unit 80 may update the template information of the object used to calculate the reliability at regular frame intervals, and calculate the reliability of each frame using the updated template information.

[0080] Fig. 7 is a block diagram showing another outline of the tracking system according to the present invention. The tracking system shown in Fig. 7 is a tracking system for tracking the same object using multiple cameras, and includes a first camera 100 for capturing a first image, a second camera 200 for capturing a second image captured at approximately the same angle of view as the first camera 100, a first tracking unit 81 (e.g., tracking unit 31, processing unit 30) for tracking the object on the first image, a second tracking unit 82 (e.g., tracking unit 32, processing unit 30) for tracking the object on the second image, and a reliability of the object obtained by the first tracking unit 81. The tracking system includes a first reliability calculation unit 83 (e.g., reliability calculation unit 33, processing unit 30) that calculates a first reliability, which is the reliability of the object obtained by the second tracking unit 82, a second reliability calculation unit 84 (e.g., reliability calculation unit 34, processing unit 30) that calculates a second reliability, which is the reliability of the object obtained by the second tracking unit 82, and a control unit 85 (e.g., control unit 35, processing unit 30) that compares the first reliability and the second reliability to identify a tracking process with a higher reliability, and controls the other tracking process based on the result of the identified tracking process.

[0081] Such a configuration can also improve the tracking accuracy when tracking the same object with multiple cameras. [Explanation of symbols]

[0082] 10 First Camera 20 Second Camera 30 Processing section 31,32 Tracking Section 33,34 Reliability calculation unit 35 Control Unit 40 Output section 50 Storage section

Claims

1. A tracking system that tracks the same object using multiple cameras, A processing unit is provided which receives images from at least two cameras, The processing unit a first detection process for detecting an object in a first image captured by the visible light camera; a second detection process for detecting the object on a second image captured by the infrared camera; and a control process for controlling a tracking process for tracking the object on images captured by at least one of the visible light camera and the infrared camera after the time at which the specific frame is captured, based on the position of the object shown in an image of the specific frame captured by each of the visible light camera and the infrared camera. A tracking system characterized by:

2. Furthermore, the processing unit executes a control process for controlling a tracking process for tracking the object within a search range set based on the position of the object obtained from the first detection process and the second detection process. The tracking system of claim 1 .

3. A first detection process for detecting an object in a first image captured by a visible light camera; a second detection process for detecting the object on a second image captured by the infrared camera; and a control process for controlling a tracking process for tracking the object on images captured by at least one of the visible light camera and the infrared camera after the time at which the specific frame is captured, based on the position of the object shown in an image of the specific frame captured by each of the visible light camera and the infrared camera. A tracking method characterized by:

4. Furthermore, the control process controls a tracking process for tracking the object within a search range set based on the positions of the object obtained from the first detection process and the second detection process. The tracking method according to claim 3.

5. On the computer, a first detection process for detecting an object in a first image captured by the visible light camera; a second detection process for detecting the object in a second image captured by the infrared camera; and a control process for controlling a tracking process for tracking the object on images captured by at least one of the visible light camera and the infrared camera after the time at which the specific frame is captured, based on the position of the object shown in the image of the specific frame captured by each of the visible light camera and the infrared camera; A program to execute.

6. On the computer, Furthermore, the control process controls a tracking process for tracking the object within a search range set based on the positions of the object obtained from the first detection process and the second detection process. The program according to claim 5.

Images