Tracking system, tracking method and tracking program
The tracking system improves accuracy by using multiple visible light cameras with controlled parallax and adjusted viewing angles, enhancing reliability through shared tracking information across diverse camera types and conditions.
Patent Information
- Application Number
- JP2021204825
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-12-17
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2035-07-28
AI Technical Summary
Existing tracking systems using multiple cameras face challenges in maintaining accurate tracking due to differences in viewing angles, parallax, and environmental conditions such as darkness or temperature, leading to decreased tracking accuracy.
A tracking system that utilizes multiple visible light cameras with controlled parallax and adjusts their angles of view to match, employing a processing unit to detect and track objects across multiple images, calculating reliability and sharing tracking information to improve accuracy.
Enhances tracking accuracy by compensating for parallax and environmental limitations, allowing robust tracking across different camera types and conditions, particularly in stereo viewing scenarios.
Smart Images

Figure 0007593638000001 
Figure 0007593638000002 
Figure 0007593638000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a tracking system, a tracking method, and a tracking program for tracking the same object with multiple cameras. [Background technology]
[0002] Various methods have been proposed for tracking an object specified by a user in an input image sequence or video.
[0003] Patent Document 1 describes a moving object detection device that detects moving objects from video images. The device described in Patent Document 1 removes false optical flows by using the optical flows of moving objects in current and past image frames.
[0004] Also, a method of tracking an object using multiple cameras is known. Patent Document 2 describes a monitoring system that tracks and monitors a moving object by linking multiple cameras. In the monitoring system described in Patent Document 2, multiple monitoring cameras exchange characteristic information of the tracking object between each other, and each monitoring camera tracks the tracking object using the characteristic information.
[0005] Patent Document 3 describes an image processing device that processes a visible light image to detect and track an object that has entered a target area. A visible light camera and a far-infrared camera are connected to the image processing device described in Patent Document 3, and both cameras capture the target area in which the object is to be detected at approximately the same angle and approximately the same imaging magnification. Furthermore, when there is an image area in the visible light image that is not suitable for object detection, the image processing device described in Patent Document 3 generates a far-infrared background difference image using a far-infrared background image to detect the captured object.
[0006] Incidentally, a target detection device that detects targets at high speed is described in Patent Document 4. The target detection device described in Patent Document 4 includes detectors that detect far-infrared images and near-infrared images input through a lens. [Prior art documents] [Patent documents]
[0007] [Patent Document 1] Patent No. 3810657 [Patent Document 2] JP 2003-324720 A [Patent Document 3] JP 2014-157452 A [Patent Document 4] Japanese Patent Application Publication No. 6-174828 Summary of the Invention [Problem to be solved by the invention]
[0008] When tracking an object using a single camera as in the device described in Patent Document 1, there are environments in which tracking cannot be performed properly depending on the characteristics of the camera. For example, when tracking using a single visible camera, the color of the object can be said to be an effective feature for tracking. However, when the tracking environment is darkness or other environments where color differences are difficult to find, it is difficult to track the object using a single visible camera. Also, for example, when tracking using a single far-infrared camera, features can be obtained by temperature regardless of day or night, but tracking the object is difficult if other objects with the same temperature are present.
[0009] In addition, in the surveillance system described in Patent Document 2, when a surveillance camera device loses sight of a tracking target, it transmits extracted feature information to other surveillance camera devices, and the other surveillance camera devices start transmitting video if the transmitted feature information matches. However, since the shooting positions and angle of view of each surveillance camera device are usually different, it is difficult to say that the feature information received from other surveillance camera devices can be used with high accuracy for subsequent tracking.
[0010] On the other hand, it is possible to improve the reliability of tracking by tracking an object using multiple cameras, as in the image processing device described in Patent Document 3. However, when tracking using multiple cameras simultaneously, such as stereo vision, there are problems with differences in the angle of view and differences in appearance due to parallax. Ideally, if the angle of view is the same, the position of the object does not change between cameras, making processing very easy. However, since the angle of view differs due to subtle differences in the size and focal length of the camera sensor, lens distortion, etc., it is difficult to eliminate the difference in the angle of view caused by using multiple cameras.
[0011] Even if the angle of view is adjusted to be approximately the same by using cameras of the same model number, parallax cannot be eliminated in principle when multiple cameras are used. If image processing is performed without considering the difference in appearance due to parallax and assuming that the angle of view is approximately the same, the tracking target cannot be captured accurately because this assumption is unreasonable, resulting in a problem of reduced tracking accuracy.
[0012] Therefore, an object of the present invention is to provide a tracking system, a tracking method, and a tracking program that can improve the tracking accuracy when tracking the same object with multiple cameras. [Means for solving the problem]
[0013] A tracking system according to the present invention is a tracking system that tracks the same object with a plurality of cameras, and includes a processing unit that inputs images from at least two or more visible light cameras, and the processing unit performs a first detection process to detect the object in a first image captured by a first visible light camera, a second detection process to detect the object in a second image captured by a second visible light camera that captures images at an angle of view with parallax from the first visible light camera, and a second detection process to detect the object in a second image captured by at least one of the first visible light camera and the second visible light camera after the time the specific frame was captured, based on the position of the object shown in an image of a specific frame captured by each of the first visible light camera and the second visible light camera. Same angle of viewand a control process for controlling a tracking process for tracking an object on an image.
[0015] The tracking method according to the present invention includes a first detection process for detecting an object in a first image captured by a first visible light camera, a second detection process for detecting an object in a second image captured by a second visible light camera that captures an image at an angle of view with parallax from the first visible light camera, and a second detection process for detecting an object in a second image captured by at least one of the first visible light camera and the second visible light camera after the time the specific frame was captured, based on the position of the object shown in an image of a specific frame captured by each of the first visible light camera and the second visible light camera. Same angle of view and a control process for controlling a tracking process for tracking an object on an image.
[0016] The tracking program according to the present invention causes a computer to perform a first detection process of detecting an object in a first image captured by a first visible light camera, a second detection process of detecting an object in a second image captured by a second visible light camera that captures an image at an angle of view with parallax from the first visible light camera, and a third detection process of detecting an object in a second image captured by at least one of the first visible light camera and the second visible light camera after the time when the specific frame was captured, based on the position of the object shown in an image of a specific frame captured by each of the first visible light camera and the second visible light camera. Same angle of view The present invention is characterized in that a control process is executed that controls a tracking process for tracking an object on an image. Effect of the Invention
[0017] According to the present invention, it is possible to improve the tracking accuracy when tracking the same object with a plurality of cameras. [Brief description of the drawings]
[0018] [Figure 1] FIG. 1 is a block diagram illustrating one embodiment of a tracking system in accordance with the present invention. [Diagram 2] FIG. 11 is an explanatory diagram showing an example in which two videos are displayed side by side. [Diagram 3] FIG. 13 is an explanatory diagram showing an example of displaying a tracking result. [Figure 4] FIG. 1 is an explanatory diagram showing an example of the operation of the tracking system. [Diagram 5] FIG. 13 is a block diagram showing a modified example of a tracking system according to the present invention. [Figure 6] 1 is a block diagram showing an overview of a tracking system according to the present invention; [Figure 7] FIG. 2 is a block diagram showing another overview of the tracking system according to the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0019] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the present invention, an object refers to any person or object to be tracked.
[0020] 1 is a block diagram showing an embodiment of a tracking system according to the present invention. The tracking system of this embodiment tracks the same object using multiple cameras, and includes a first camera 10, a second camera 20, a processing unit 30, an output unit 40, and a storage unit 50.
[0021] In the following description, a tracking system having two cameras (a first camera 10 and a second camera 20) is exemplified, but the number of cameras provided in the tracking system is not limited to two and may be three or more.
[0022] In this embodiment, the first camera 10 and the second camera 20 are used to track the same object. Specifically, one camera captures an image having substantially the same angle of view as the other camera. In the following description, the image captured by the first camera is referred to as the first image, and the image captured by the second camera is referred to as the second image. The second image is captured with substantially the same angle of view as the first image.
[0023] The positional relationship between the first camera 10 and the second camera 20 may be set so that a target object can be tracked by stereoscopic vision. Here, the positional relationship means the positions, orientations, and magnifications of the cameras set so that two images are captured with approximately the same angle of view as described above.
[0024] Due to the nature of tracking the same object using multiple cameras, it is theoretically impossible to eliminate parallax. Therefore, the angle of view of one camera is used as the standard and the angle of view of each of the other cameras (image range, size at a certain distance, etc.) is adjusted so that they are almost the same. This setting is the same even when there are three or more cameras.
[0025] Each camera may be set using a device that fixes the relative positions of the cameras. Furthermore, as long as the relative positions of the cameras do not change, the installation locations of the first camera 10 and the second camera 20 do not need to be fixed. For example, these cameras may be attached to the same moving object and move, changing the angle of view from the initial state.
[0026] In addition, each camera is set so that the target object is present within the angle of view when tracking starts. However, the target object does not need to be present within the angle of view at any time other than when tracking starts. The first camera 10 and the second camera 20 input the captured images to the processing unit 30.
[0027] The camera used in this embodiment may be any camera capable of photographing an object. Specifically, the first camera 10 and the second camera 20 may be realized by cameras equipped with the same type of sensor, or may be realized by cameras equipped with different types of sensors. For example, the first camera may be a visible light camera, and the second camera may be a far-infrared camera. Note that the form of the first camera 10 and the second camera 20 is not limited to a visible light camera or a far-infrared camera, and may be, for example, a near-infrared camera.
[0028] The frame rates of the first camera 10 and the second camera 20 may be the same or different. For example, the first camera 10 may capture high-resolution images at a lower frame rate than the second camera 20, and the second camera 20 may capture low-resolution images at a higher frame rate than the first camera 10.
[0029] The storage unit 50 stores information used for various processes by the processing unit 30, which will be described later. The storage unit 50 is realized by, for example, a magnetic disk device.
[0030] The output unit 40 displays the video input to the processing unit 30. For example, the output unit 40 may display two videos taken at the same time side by side so that they can be compared. The output unit 40 is realized by, for example, a display device or a touch panel device. FIG. 2 is an explanatory diagram showing an example in which two videos are displayed side by side. In the example shown in FIG. 2, the video taken by the visible light camera is displayed on the left side, and the video taken by the far-infrared camera is displayed on the right side.
[0031] The processing unit 30 inputs an image captured by the first camera 10 (i.e., a first image) and an image captured by the second camera 20 (i.e., a second image). The processing unit 30 accepts a designation of an object in one of the input images from a user. In this embodiment, the processing unit 30 accepts the designation of an object in the first image.
[0032] The object is specified by selecting a range including the object on the first image. The method of specifying the object is arbitrary. For example, when the output unit 40 is realized by a display device, the object may be specified by a range selection operation using a pointing device on the first image displayed on the output unit 40. Also, for example, when the output unit 40 is realized by a touch panel, the object may be specified by a range selection operation by the user on the first image displayed on the output unit 40.
[0033] When an object on the first image is specified, the processing unit 30 specifies a specified range including the object as a search range. Hereinafter, this search range may be referred to as a first search range. In addition, the processing unit 30 specifies a range in the second image in which the same object exists as a search range. Hereinafter, this search range may be referred to as a second search range.
[0034] Here, the search range is an area where the object detection process is performed. Since the detection process is a process of searching for and detecting an object that has the same appearance as the learned tracking target, the search range can also be said to be the range on the image where the search is performed. Note that the search is a process in which all the lattices obtained by dividing the image into a predetermined size (for example, 5 pixels x 5 pixels, 10 pixels x 10 pixels, etc.) are treated as candidates, and an object that has the same appearance as the learned tracking target is searched for among the candidates and determined as the detection result.
[0035] As described above, there is a parallax between the first image and the second image. Therefore, the processing unit 30 specifies the second search range taking the parallax (displacement) into consideration. For example, the processing unit 30 compares a frame one frame before the specified time with a frame at the specified time for a rectangular range specified in the first image, and estimates the magnitude and direction of the movement of the object. The processing unit 30 may estimate the magnitude and direction of the movement using, for example, optical flow. Next, the processing unit 30 specifies a rectangle in the second image at the same position as the rectangle specified in the first image. The processing unit 30 compares the specified rectangle in the second image with the frame one frame before the specified time, as in the first image, and estimates the magnitude and direction of the movement of the object. Then, the processing unit 30 specifies the second search range by estimating that a location where the estimated size and direction of the object are similar is a location where the object specified in the second image is present.
[0036] In this way, since the search range in one image is specified based on the size and direction of the movement of the object in the other image, the object can be specified taking into consideration the discrepancy between the two images. In addition, the effort of specifying the object can be reduced by automatically specifying the object in the other image in which the object is not specified. That is, in this embodiment, the user only needs to specify the object in one image, and the object in the other image is automatically estimated, so that it can be said that the effort required for input does not increase compared to the case of a single camera, despite the tracking technology using multiple heterogeneous cameras. Thereafter, the processing unit 30 determines the search range of the object in parallel from the images captured by each camera and performs tracking processing.
[0037] When the processing unit 30 specifies each search range, it calculates the reliability of the object for each video at regular frame intervals. This reliability is a similarity indicating the likelihood of the object, and is calculated for each video based on tracking information from the start of tracking. Here, the tracking information from the start of tracking includes each image showing the object acquired during tracking, and information showing each position of the object (hereinafter referred to as a template). A template is created for each video (for each camera that captures the image).
[0038] The processing unit 30 calculates the reliability of the object based on the degree of similarity of the appearance when compared with the template, and the degree of closeness and plausibility of the object's position estimated according to the movement. At this time, the processing unit 30 may calculate the reliability of the object in each frame while updating the object's template as needed. The processing unit 30 may also learn an object model (appearance model, movement model) based on images acquired as needed during search and information indicating the movement of the object, and calculate the reliability of the object using the learned model.
[0039] The processing unit 30 may update the object template based on the reliability calculated for each image, or may update the object template based on the relative reliability (relative reliability) when comparing the reliability calculated for each image. For example, when the calculated reliability is lower than the reliability calculated for other images, the processing unit 30 may lower (or set to 0) the update weight of the object identified in that image. Also, for example, when both reliability are high and the relative reliability is not high, the processing unit 30 may lower (or set to 0) the update weight of the object in this case as well.
[0040] Any method can be used as long as it can calculate the reliability of an object based on information acquired in the tracking process of the object on each video (specifically, tracking information from the start of tracking). In other words, the reliability is a degree indicating the likelihood of the object being tracked, and it may be calculated using a formula that indicates that the higher the value, the higher the similarity.
[0041] Next, the processing unit 30 reflects the position and size of the object in one tracking process that is determined to have high reliability in the setting of the search range of the object in the other tracking process. Specifically, the processing unit 30 compares the reliability calculated from each video, and identifies the video and object for which the highest reliability is calculated at the same time. Then, the processing unit 30 determines the search range of the other videos other than the video for which the highest reliability is calculated, using information indicating the range (position and size) of the identified object. In other words, it can be said that the processing unit 30 shares the information of the object for which the highest reliability is calculated with all other videos. Thereafter, the tracking process continues in the search range determined for each video.
[0042] The processing unit 30 may change the method of determining the search range depending on the calculated reliability and the tracking result of the previous frame. Specifically, the processing unit 30 may change the method of determining the search range depending on whether the tracking result of the previous frame exists or not, and whether the reliability is high or low. For example, when the tracking result of the previous frame does not exist or when the reliability is low, the processing unit 30 searches for the tracking target with the entire image as the search range. At this time, the processing unit 30 may identify the target from within each entire image using only templates whose size is close to the size of the target identified in the most recent past tracking result.
[0043] In this way, by narrowing the search range of each camera image and searching for and adopting the candidate with a certain level of reliability or higher within that range, it is possible to shorten the search time while adjusting the tracking result. Note that, if the shared side camera image does not have a certain level of reliability or a matching template, the processing unit 30 may adopt the shared tracking information as is.
[0044] In this embodiment, since the object is tracked complementarily by the multiple cameras, parallax occurs in each camera. By utilizing this parallax, the processing unit 30 may determine the search range using the depth from the camera to the object.
[0045] When determining the search range using the depth, first, the user actually performs calibration to calculate in advance the knowledge that a certain depth range will cause a certain degree of deviation with respect to the distance to the object, and stores the calculation result in the storage unit 50. That is, the storage unit 50 stores the deviation (difference) according to the distance to the object and the depth range calculated by calibration. The processing unit 30 calculates the depth to the object based on the deviation observed during actual tracking, and specifies the search range by estimating the movement range of the object on the image of the current frame from the depth calculated in the image of the previous frame. By narrowing the search range in this way, it becomes possible to apply the template to only a part of the range on the image, and therefore the tracking process speed can be improved.
[0046] Alternatively, the storage unit 50 may store the depth range and the search range in association with each other. In this case, the processing unit 30 may calculate the depth range of the target object based on the deviation observed during actual tracking, and determine the search range to be set for another tracking process from the calculated depth range based on the correspondence stored in the storage unit 50.
[0047] After determining the search range, the processing unit 30 may perform tracking processing (detection processing of the target object) again and may also display the tracking results at any time, for example, by surrounding the target object with a rectangle on the image displayed on the output unit 40. By displaying such tracking results, it becomes possible to visually track the target object.
[0048] At this time, the processing unit 30 may display the tracking results on the video with higher reliability in a specific manner. For example, the processing unit 30 may display the object on the video with higher reliability surrounded by a rectangle with a thick frame in a dark color, and the object on the video with lower reliability surrounded by a rectangle with a thin frame in a light color (or not surrounded by a rectangle).
[0049] Fig. 3 is an explanatory diagram showing an example of a display of a tracking result. In the example shown in Fig. 3, as a result of performing a tracking process on the video shown in Fig. 2, the reliability of the tracking result from the video captured by the second camera was continuously higher than that of the video captured by the first camera. In this way, it is possible to visually grasp which video's tracking result of the object is being used to perform the tracking process.
[0050] The processing unit 30 is realized by a CPU of a computer that operates according to a program (tracking program). For example, the program may be stored in the storage unit 50, and the CPU may read the program and operate as the processing unit 30 according to the program. The processing unit 30 may also be realized by dedicated hardware.
[0051] Next, the operation of the tracking system of this embodiment will be described. Fig. 4 is an explanatory diagram showing an example of the operation of the tracking system of this embodiment. First, image adjustment is performed in advance between the image captured by the first camera 10 (i.e., the first image) and the image captured by the second camera 20 (i.e., the second image) (step S11). Specifically, the first camera 10 and the second camera 20 are adjusted so that they can capture images with approximately the same angle of view.
[0052] After the adjustment, the processing unit 30 accepts a user's designation of the object on the first image (step S12). When the object on the first image is designated, the processing unit 30 specifies a search range on the first image (first search range) and specifies a search range on the second image (second search range) (step S13).
[0053] Once the search ranges on each image are specified, the processing unit 30 tracks the object on each image (steps S14 and S15) and calculates the reliability of the tracked object at regular frame intervals (steps S16 and S17). The processing unit 30 compares the reliability calculated for each image and identifies the image and object for which the highest reliability was calculated (step S18).
[0054] The processing unit 30 controls the other tracking process based on the identified tracking process result (steps S19 and S20). Specifically, the processing unit 30 determines the search range by adjusting the tracking result, and repeats the processes of steps S14 and S15 onward for the image of the next frame.
[0055] As described above, in this embodiment, the first camera 10 captures a first image, and the second camera 20 captures a second image captured at approximately the same angle of view as the first camera. Then, the processing unit 30 provided in the tracking system executes a first tracking process for tracking an object on the first image and a second tracking process for tracking an object on the second image. The processing unit 30 also executes a reliability calculation process for calculating the reliability (first reliability) of the object obtained by the first tracking process and the reliability (second reliability) of the object obtained by the second tracking process. Then, the processing unit 30 compares the first reliability with the second reliability to identify a tracking process with a higher reliability, and executes a control process for controlling the other tracking process based on the result of the identified one of the tracking processes. Thus, it is possible to improve the tracking accuracy when tracking the same object with multiple cameras.
[0056] Furthermore, in this embodiment, the misalignment in appearance (parallax) that occurs when multiple cameras, such as stereoscopic vision, is taken into account, and the search range is fine-tuned (corrected) by sharing the tracking results, thereby enabling more robust tracking of the target object.
[0057] In addition, in this embodiment, the reliability of the object is calculated independently for each image, and the tracking information of the object identified from the image with high reliability is reflected in the other images, so that it does not matter whether the sensor or frame rate of the camera used for tracking is the same. Therefore, since the object can be tracked complementarily with different types of cameras, it is possible to improve the tracking accuracy. In other words, in the present invention, since tracking is performed based on the most reliable tracking information among multiple cameras, it can be said that the scene in which each camera has an advantageous characteristic is automatically selected and tracked.
[0058] Specifically, in this embodiment, images from multiple cameras of the same or different types (e.g., a far-infrared camera and a visible light camera) with approximately the same angle of view are used to simultaneously track an object, while the tracking information from the high-reliability side is shared with the low-reliability side to correct the tracking information. For example, when cameras including different types of sensors are combined, it becomes possible to track scenes where one type of camera is weak by using the other type of camera to complement it.
[0059] In addition, in this embodiment, since it is sufficient to combine existing single cameras and adjust the angle of view of each camera to be approximately the same, there is no need for a special lens mechanism as described in Patent Document 4. Also, as long as the positional relationship between the cameras (relationships such as camera position, orientation, and magnification) is not changed, the target object can be tracked even if the cameras are moved during tracking. Therefore, the tracking system of this embodiment can also be applied to a moving camera system.
[0060] Next, a modified example of this embodiment will be described. In the example shown in Fig. 1, the tracking system includes a processing unit 30, which performs tracking processing and reliability calculation processing on both an image (first image) input from a first camera 10 and an image (second image) input from a second camera.
[0061] In this modification, a configuration in which the processing of the images captured by each camera is realized by separate means will be described. Fig. 5 is a block diagram showing a modification of the tracking system according to the present invention. The tracking system illustrated in Fig. 5 includes a first camera 10, a second camera 20, a tracking unit 31, a tracking unit 32, a reliability calculation unit 33, a reliability calculation unit 34, a control unit 35, an output unit 40, and a storage unit 50.
[0062] The contents of the first camera 10, the second camera 20, the output unit 40, and the storage unit 50 are the same as those of the above embodiment, and therefore detailed description thereof will be omitted. In addition, the contents performed by the tracking unit 31, the tracking unit 32, the reliability calculation unit 33, the reliability calculation unit 34, and the control unit 35 are the same as those performed by the processing unit 30 of the above embodiment.
[0063] Specifically, the tracking unit 31 and the reliability calculation unit 33 perform processing on the video (first video) captured by the first camera 10. The tracking unit 32 and the reliability calculation unit 34 perform processing on the video (second video) captured by the second camera 20. The processing performed by the tracking unit 31 and the processing performed by the tracking unit 32, and the processing performed by the reliability calculation unit 33 and the processing performed by the reliability calculation unit 34 are the same except that the target videos or images are different.
[0064] The tracking unit 31 tracks the object on the first video (first tracking process). The tracking unit 32 tracks the object on the second video (second tracking process). The method of tracking the object is the same as the method performed by the processing unit 30 described above.
[0065] The reliability calculation unit 33 calculates the reliability (first reliability) of the object obtained in the first tracking process. The reliability calculation unit 34 calculates the reliability (second reliability) of the object obtained in the second tracking process. The reliability calculation method is the same as the method performed by the processing unit 30 described above.
[0066] The control unit 35 compares the first reliability with the second reliability, identifies a tracking process with a higher reliability, and controls the other tracking process based on the results of the identified tracking process. Specifically, the control unit 35 reflects the position and size of the object in one tracking process determined to have a higher reliability in the setting of the search range of the object in the other tracking process. The method of reflecting the setting of the search range is the same as the method performed by the processing unit 30 described above.
[0067] The tracking unit 31, the tracking unit 32, the reliability calculation unit 33, the reliability calculation unit 34, and the control unit 35 are realized by a CPU of a computer that operates according to a program (communication information reference program). Furthermore, the tracking unit 31, the tracking unit 32, the reliability calculation unit 33, the reliability calculation unit 34, and the control unit 35 may each be realized by dedicated hardware.
[0068] Also, for example, the first camera 10, the tracking unit 31, and the reliability calculation unit 33 may be configured as an integrated tracking device, and the second camera 20, the tracking unit 32, and the reliability calculation unit 34 may be configured as an integrated tracking device. Then, each tracking device may perform each process in response to an instruction from the control unit 35.
[0069] This configuration also makes it possible to improve the tracking accuracy when tracking the same object with multiple cameras.
[0070] Next, an overview of the present invention will be described. Fig. 6 is a block diagram showing an overview of a tracking system according to the present invention. The tracking system according to the present invention is a tracking system that tracks the same object (e.g., a tracking target) with a plurality of cameras, and includes a first camera 100 (e.g., first camera 10) that captures a first image, a second camera 200 (e.g., second camera 20) that captures a second image captured with approximately the same angle of view as the first camera 100, and a processing unit 80 (e.g., processing unit 30).
[0071] The processing unit 80 executes a first tracking process for tracking an object on a first image, a second tracking process for tracking an object on a second image, a reliability calculation process for calculating a first reliability which is the reliability of the object obtained by the first tracking process and a second reliability which is the reliability of the object obtained by the second tracking process, and a control process for comparing the first reliability and the second reliability to identify a tracking process having a higher reliability and controlling the other tracking process based on the result of the identified tracking process.
[0072] Such a configuration can improve the tracking accuracy when tracking the same object with multiple cameras.
[0073] Here, the first camera 100 and the second camera 200 may be realized by cameras equipped with different types of sensors. When each camera is realized by a camera equipped with a different type of sensor, the contents of the images captured by each camera are significantly different. However, in the present invention, the reliability of each captured image is calculated, and the tracking process of the other image is controlled based on the tracking process result (e.g., search range) of the image for which the higher reliability is calculated. Therefore, it becomes possible to track a scene in which one type of camera is poor at tracking while complementing it with the other type of camera.
[0074] Specifically, the first camera 100 may be a visible light camera, and the second camera 200 may be a far-infrared camera. In this case, it is possible to track the object in a particularly complementary manner, and it is possible to make the most of the characteristics of each camera.
[0075] In addition, the first camera 100 may capture high-resolution video at a lower frame rate than the second camera 200, and the second camera 200 may capture low-resolution video at a higher frame rate than the first camera 100.
[0076] Furthermore, the processing unit 80 may reflect the position and size of the object in one tracking process that is determined to have high reliability in the setting of the search range of the object in the other tracking process. That is, since the search range of the other tracking process is determined from the tracking result of the object with high reliability, even when multiple cameras with different angles of view are used, the search range can be corrected while tracking. Therefore, it becomes possible to track the same object with high accuracy. That is, in the present invention, it can be said that the search range of the other image can be set to absorb the difference in appearance on the image that occurs when multiple cameras are used by reflecting the result of one tracking process.
[0077] The tracking system may also include a storage unit (e.g., storage unit 50) that stores the depth range and the search range in association with each other. The processing unit 80 may then refer to the storage unit and specify the search range from the depth range of the object calculated based on the difference observed during tracking.
[0078] Furthermore, the first camera 100 and the second camera 200 may be attached to the same moving body and track the same object. In the present invention, as long as both cameras are installed so that they can capture images with approximately the same angle of view, the angle of view does not need to match exactly, and therefore the accuracy of the tracking process performed when both cameras are attached to the same moving body can be improved.
[0079] Furthermore, the processing unit 80 may update the template information of the object used to calculate the reliability at regular frame intervals, and calculate the reliability of each frame using the updated template information.
[0080] Fig. 7 is a block diagram showing another outline of the tracking system according to the present invention. The tracking system shown in Fig. 7 is a tracking system for tracking the same object with a plurality of cameras, and includes a first camera 100 for capturing a first image, a second camera 200 for capturing a second image captured at substantially the same angle of view as the first camera 100, a first tracking unit 81 (e.g., tracking unit 31, processing unit 30) for tracking the object on the first image, a second tracking unit 82 (e.g., tracking unit 32, processing unit 30) for tracking the object on the second image, and a reliability of the object obtained by the first tracking unit 81. The tracking system includes a first reliability calculation unit 83 (e.g., reliability calculation unit 33, processing unit 30) that calculates a first reliability, which is the reliability of the object obtained by the second tracking unit 82, a second reliability calculation unit 84 (e.g., reliability calculation unit 34, processing unit 30) that calculates a second reliability, which is the reliability of the object obtained by the second tracking unit 82, and a control unit 85 (e.g., control unit 35, processing unit 30) that compares the first reliability and the second reliability, identifies a tracking process having a higher reliability, and controls the other tracking process based on the result of the identified tracking process.
[0081] Such a configuration can also improve the tracking accuracy when tracking the same object with multiple cameras. [Explanation of symbols]
[0082] 10 First Camera 20 Second Camera 30 Processing section 31,32 Tracking Department 33,34 Reliability calculation unit 35 Control section 40 Output section 50 Storage section
Claims
1. A tracking system for tracking the same object using multiple cameras, comprising: A processing unit is provided for receiving images from at least two or more visible light cameras, The processing unit includes: a first detection process for detecting an object in a first image captured by a first visible light camera; a second detection process for detecting the object on a second image captured by a second visible light camera that captures an image at an angle of view having parallax with respect to the first visible light camera; and a control process for controlling a tracking process for tracking the object on an image having the same angle of view captured by at least one of the first visible light camera and the second visible light camera after a time when the specific frame is captured, based on a position of the object shown in an image of a specific frame captured by each of the first visible light camera and the second visible light camera. A tracking system comprising:
2. Furthermore, the processing unit executes a control process for controlling a tracking process for tracking the object within a search range set based on the position of the object obtained from the first detection process and the second detection process. The tracking system of claim 1 .
3. a first detection process for detecting an object in a first image captured by a first visible light camera; a second detection process for detecting the object on a second image captured by a second visible light camera that captures an image at an angle of view having parallax with respect to the first visible light camera; and a control process for controlling a tracking process for tracking the object on an image having the same angle of view captured by at least one of the first visible light camera and the second visible light camera after a time when the specific frame is captured, based on a position of the object shown in an image of a specific frame captured by each of the first visible light camera and the second visible light camera. A tracking method comprising:
4. Furthermore, in the control process, a tracking process is controlled to track the object within a search range set based on the positions of the object obtained from the first detection process and the second detection process. The tracking method according to claim 3.
5. On the computer, a first detection process for detecting an object in a first image captured by a first visible light camera; a second detection process for detecting the object on a second image captured by a second visible light camera that captures an image at an angle of view having parallax with respect to the first visible light camera; and a control process for controlling a tracking process for tracking the object on an image having the same angle of view captured by at least one of the first visible light camera and the second visible light camera after a time when the specific frame is captured, based on a position of the object captured in an image of a specific frame captured by each of the first visible light camera and the second visible light camera. A program for executing the above.
6. On the computer, Furthermore, the control process controls a tracking process for tracking the object within a search range set based on the positions of the object obtained from the first detection process and the second detection process. The program according to claim 5.
Citation Information
Patent Citations
Target detecting device
JP1994174828A
Supervisory system and supervisory camera
JP2003324720A
Moving body tracking system using a plurality of cameras and its method
JP2004072628A
Image processing apparatus, image processing method, and image processing program
JP2014157452A
MOTIONING OBJECT DETECTION METHOD AND APPARATUS
JP3810657B2
