Object tracking device and object tracking method

Abstract

The present invention provides an object tracking device and an object tracking method. The object tracking device stores, for each part, information of a region image representing the object in a first frame image that is one of a plurality of frame images constituting the motion image, and stores a notable part of the object. When the object is lost, the recovery unit of the object tracking device (1) estimates the region where the parts of the object exist when the notable part exists at the maximum position for each maximum position of the visual saliency map corresponding to the second frame image that is the frame image in which the object is lost, and (2) calculates a similarity score based on the similarity of the region images of each part of the first frame image and the second frame image, and determines that the object exists at the maximum position in the second frame image where the similarity score is the largest.

Description

Technical Field

[0001] The present invention relates to a recovery technology when tracking of an object is lost. Background Art

[0002] Object tracking, which tracks an object detected in a certain frame of a moving image (time-series image), is an important technology in the field of computer vision.

[0003] As one of the object tracking methods, a method called KCF (Kernelized Correlation Filter) is known as a correlation filter model. KCF regards the area of ​​the tracking object initially provided as the correct answer, and performs online learning and tracking. As time goes by, noise may be generated in the area of ​​the tracking object (image area outside the tracking object), and drift to the background may occur. In this way, the situation where the tracking object is lost and the tracking cannot be continued normally is called tracking failure or loss.

[0004] Tracking loss is a problem that occurs in any tracking method, not just the tracking method using the correlation filter model. When tracking loss occurs, it is desirable to regain the object being tracked and continue tracking. Re-searching for the object being tracked and continuing tracking after tracking loss is called tracking recovery.

[0005] Patent document 1 proposes a recovery process for tracking loss. Specifically, in Patent document 1, the template image is continuously updated during tracking, and when tracking loss occurs, the initial template image is used for searching, and the search result is used as the tracking object, thereby recovering from the loss.

[0006] However, in the fields of building automation (BA) and factory automation (FA), application software is required that automatically measures the "number" / "position" / "movement line" of people by image sensors, and optimizes and controls lighting, air conditioning and other equipment. In the above-mentioned applications, in order to obtain image information in the widest range possible, an ultra-wide-angle camera equipped with a fisheye lens (fisheye lens) is mostly used (referred to as fisheye camera, omnidirectional camera, panoramic camera, etc., which have the same meaning. The term "fisheye camera" is used in this specification). In addition, in the above-mentioned applications, in order to obtain image information in the widest range possible, a camera installed at a high place such as a ceiling is configured so that the camera's viewpoint is a bird's-eye view. In the camera of this configuration, the viewpoint for photographing a person is a front view, a side view or a back view when the person is located at the periphery of the image, and a bird's-eye view when the person is located at the center of the image. In addition, in the camera of this configuration, when located at the periphery, the entire image of the subject can be seen, and on the other hand, in the bird's-eye view located in the center, the head and shoulders are photographed, but the torso and lower body are blocked.

[0007] The appearance of the object being photographed is distorted due to the position within the shooting plane, so the image captured by the fisheye camera is deformed. In addition, even if a camera other than a fisheye camera is used, when the camera's viewpoint is set to look down, the appearance of the object being tracked may change or be blocked due to the position of the object. In addition, considering that the frame rate is low in an environment with limited processing power such as embedded devices, there is a particularity that the amount of movement and feature quantity of the object between frames changes greatly.

[0008] From the above situation, the conventional method such as Patent Document 1 may not be able to restore with high accuracy.

[0009] Prior art literature

[0010] Patent Literature

[0011] Patent Document 1: (Japanese) Patent Publication No. 2014-036429 Summary of the invention

[0012] Technical problem to be solved by the invention

[0013] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide a restoration technology for object tracking with higher accuracy than before.

[0014] Technical solutions for solving technical problems

[0015] In order to achieve the above object, the present invention adopts the following structure.

[0016] A first aspect of the present invention is an object tracking device, comprising:

[0017] Storage unit;

[0018] A tracking unit that tracks an object in a moving image;

[0019] a recovery unit for recovering the object when the object is lost;

[0020] The storage unit stores, for each part, a region image of the object in a first frame image that is one of a plurality of frame images constituting the motion image, and stores a part of the object where a visual saliency map corresponding to the first frame image takes a maximum value,

[0021] The recovery unit,

[0022] When the object is lost, for each maximum position of a visual saliency map corresponding to a second frame image as a frame image in which the object is lost, (1) an area where parts of the object exist when it is assumed that a part of the object that takes a maximum value in the first frame image exists at the maximum position is estimated, and (2) a similarity score is calculated based on the similarity of the regional images of each part of the first frame image and the second frame image,

[0023] It is determined that the object exists at a maximum value position where the similarity score is the largest in the second frame image.

[0024] The object to be tracked, i.e., the "object", can be any object, and a human body, a human face, an animal, a vehicle, etc. are examples thereof. The "parts" of the object can be distinguished according to the object, and the number is not particularly limited. For example, parts with different visual salience can be defined as parts. When the object is a human body, it can be distinguished into three parts: the head, the upper body, and the lower body, but it can also be distinguished into more or less than three parts. "Visual salience" is an indicator of the degree of easy attention, and can be determined based on, for example, features such as brightness, changes in brightness over time, color, and direction.

[0025] Although it is assumed that the size and shape of the object change depending on the position in the image, it can be expected that the position where the visual saliency takes the maximum value (hereinafter also referred to as the salient part) does not change. Therefore, the maximum value position in the visual saliency map can be regarded as a candidate for the position where the salient part exists. In addition, when the condition that a specified part exists at a specified position in the image is given, it is also possible to determine where other parts exist in the image. Therefore, the restoration unit calculates the similarity score based on the similarity of the regional image of each part of the first frame image and the second frame image for each maximum value position of the visual saliency map, and determines that there is an object at the position with the largest similarity score. As a result, even if the size and shape of the object in the image have changed, restoration can be performed with good accuracy.

[0026] In order to perform the above-mentioned restoration, the storage unit stores information of the regional image representing the object in the first frame image for each part, and pre-stores the part (salient part) where the visual saliency takes a maximum value in the first frame image. The information representing the regional image of each part can be any information as long as the image similarity can be calculated based on it, for example, it can be the regional image itself, or it can be an image feature quantity representing the characteristics of the regional image. In addition, the salient part is obtained as the part located at the position where the visual saliency map corresponding to the first frame image takes a maximum value. It should be noted that when there are multiple maxima in the object, the part to which the position with the maximum value among the multiple maxima or the position obtained based on the center of gravity or weighted sum belongs is taken as the salient part.

[0027] The similarity of the regional image of each part can also be calculated based on the similarity of the image feature quantity, for example. As examples of image feature quantities, color histograms and brightness histograms can be cited. The final similarity score can also be calculated by integrating the similarities of the image feature quantities. When integrating, for example, weights corresponding to the visible proportions of each part in the second frame image can also be used. In addition, different weights can be used for salient parts and other parts (hereinafter also referred to as non-salient parts) to integrate image similarities. By calculating the similarity score in this way, a more accurate restoration can be performed.

[0028] In addition, when a part of the object exists at a specified position in the image, the method for finding the position of other parts is not particularly limited. For example, the objects arranged at various positions can be photographed by prior measurement or simulation, and the positions of the various parts at this time can be stored. In addition, the object can be modeled with a specified shape (such as a cylinder), and when the specified part is located at the specified position, it can be determined at any time by calculation. It should be noted that not only the position of each part, but also the size or shape of each part, and the visible proportion of each part can be stored in advance or calculated at any time.

[0029] The first frame image may be a frame image before the second frame image, but is preferably an image close to the frame at which tracking of the object is started. Therefore, the first frame image may also be a frame image after tracking of the object is started or a frame image next to it. The earlier the frame image is after tracking is started, the less likely it is to be mixed with noise, so the accuracy of restoration is improved.

[0030] The tracking algorithm of the tracking unit of the present invention is not particularly limited, but examples thereof include methods using correlation filters such as KCF, MOSSE, DCF, DSST, CSR-DCF, Staple, and methods using Mean-Shift and template matching. In tracking using correlation filters, when the maximum value of an indicator calculated by the correlation filter is smaller than a threshold value, it can be determined that the object is lost. In Mean-Shift and template matching, when an indicator obtained using a color histogram or the like is outside an allowable range, it can be determined that the object is lost. Specifically, when the indicator is similarity and is smaller than a threshold value, and when the indicator is distance and is larger than a threshold value, it can be determined that the object is lost.

[0031] In addition, the image to be processed in the present invention may be a fisheye image obtained by a fisheye camera. A "fisheye camera" is a camera equipped with a fisheye lens, and is a camera that can shoot at an ultra-wide angle compared to an ordinary camera. An omnidirectional camera, a panoramic camera, and a fisheye camera are all types of ultra-wide-angle cameras, and they all have the same meaning. A fisheye camera can be set to look down on the detection object area from above the detection object area. Although the optical axis of the fisheye camera is usually set vertically downward, the optical axis of the fisheye camera can also be tilted relative to the vertical direction. Because the distortion of the fisheye image is large, the feature changes of the object between frames are large, especially in the image with a low frame rate, and it often drifts to the background. In addition, when the optical axis of the camera is set vertically downward, the viewpoint of the object being photographed changes due to the position of the object in the image, so, especially in the image with a low frame rate, the object is greatly deformed, tracking often fails, and recovery is also difficult. However, according to the present invention, in the above-mentioned fisheye image, even if the optical axis of the camera is set vertically downward, it is possible to recover from the tracking failure with good accuracy. In particular, the image to be processed in the present invention is not limited to the fisheye image, but may be a normal image (an image with little distortion, an image with a high frame rate).

[0032] A second aspect of the present invention is a tracking loss recovery method, which is performed by an object tracking device when the tracked object is lost. The tracking loss recovery method executes:

[0033] a storing step of storing, for each part, a regional image of the object in a first frame image that is one of a plurality of frame images constituting the motion image, and storing a part of the object where a visual saliency map corresponding to the first frame image takes a maximum value;

[0034] A calculation step, wherein when the object is lost, for each maximum position of a visual saliency map corresponding to a second frame image as a frame image in which the object is lost, (1) an area where the parts of the object exist when it is assumed that the part of the object that takes a maximum value in the first frame image exists at the maximum position, and (2) a similarity score is calculated based on the similarity of the regional images of each part of the first frame image and the second frame image;

[0035] A determination step is performed to determine that the object exists at a maximum position in the second frame image where the similarity score is the largest.

[0036] The present invention can be regarded as an object tracking device having at least a part of the above-mentioned device, and can also be regarded as an image processing device and a monitoring system. In addition, the present invention can also be regarded as an object tracking method, an image processing method, and a monitoring method including at least a part of the above-mentioned processing. In addition, the present invention can also be regarded as a program for implementing the method, or a storage medium storing the program in a non-transient state. It should be noted that the above-mentioned devices and processes can be combined with each other as much as possible to constitute the present invention.

[0037] Effects of the Invention

[0038] According to the present invention, when a tracking object is lost, it is possible to restore it with higher accuracy than before. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a diagram showing an application example of the human body tracking device of the present invention.

[0040] Figure 2 It is a diagram showing the structure of a monitoring system including a human body tracking device.

[0041] Figure 3 It is a diagram for explaining the part information table stored in the storage unit.

[0042] Figure 4 It is a flow chart of the overall processing performed by the human body tracking device.

[0043] Figure 5 is a flowchart of the learning process.

[0044] Figure 6 is a flowchart of the tracking process.

[0045] Figure 7 is a flowchart of the recovery process.

[0046] Figure 8 It is the data flow diagram of the recovery process.

[0047] Fig. 9A is a diagram illustrating a color histogram of a regional image of each part in an initial frame (second frame). Fig. 9 B is a diagram showing a visual saliency map corresponding to an initial frame (second frame).

[0048] Fig.10 A is a diagram showing a visual saliency map corresponding to a frame (Nth frame) where tracking loss has occurred.

[0049] Fig.10 B is a diagram for explaining a method of obtaining a similarity score for each local maximum value in a frame (Nth frame) where tracking loss has occurred. DETAILED DESCRIPTION

[0050] <Application Examples>

[0051] Reference Figure 1 , describing an application example of the object tracking device of the present invention. A human tracking device 1 is a device that analyzes a fisheye image obtained by a fisheye camera 10 installed above a tracking target area 11 (e.g., a ceiling 12, etc.), and detects / tracks a person 13 existing in the tracking target area 11. The human tracking device 1 detects, identifies, tracks, etc. a person 13 passing through the tracking target area 11 in, for example, an office or a factory. Figure 1 In the example, the areas of the four human bodies detected from the fisheye image are represented by bounding boxes. The detection result of the human body tracking device 1 is output to an external device, for example, to be used in counting the number of people, controlling various equipment such as lighting and air conditioning, monitoring suspicious persons, and analyzing movement lines.

[0052] In this application example, a local optimization tracking algorithm is used as an object tracking algorithm. In this algorithm, an image of a partial area of ​​the object to be tracked is learned, and the position of the area having the same characteristics as the object is specified, thereby tracking. Because the vicinity of the object is also a learning object, in the case of complex and changing backgrounds, noise may be generated over time, causing tracking failure.

[0053] This application example is to restore tracking with good accuracy when tracking is lost. Specifically, the human body tracking device 1 estimates candidate areas for each position that takes a maximum value in the visual saliency map, calculates the similarity score between the candidate area and the object, and estimates the maximum position with the largest similarity score as the position of the object. At this time, the human body tracking device 1 obtains the position of other parts when a certain part exists at the position for each position on the image. In addition, the human body tracking device 1 stores the part with the highest significance in tracking (salient part). When tracking is lost, the human body tracking device 1 estimates that a salient part exists at the maximum position of the visual saliency map, and can also estimate the positions of other parts. In this way, since the position of each part can be appropriately estimated, the final similarity score is calculated based on the similarity of the area where each part exists. In this way, high-precision restoration can be achieved.

[0054] <Monitoring system>

[0055] Reference Figure 2 , describing an embodiment of the present invention. Figure 2 1 is a block diagram showing the configuration of a monitoring system to which a human body tracking device according to an embodiment of the present invention is applied. The monitoring system 2 includes a fisheye camera 10 and a human body tracking device 1 .

[0056] [Fisheye Camera]

[0057] The fisheye camera 10 is a photographing device having an optical system including a fisheye lens and a photographing element (image sensor such as CCD or CMOS). Figure 1 As shown, the fisheye camera 10 can be set on the ceiling 12 of the tracking target area 11 in a state where the optical axis is vertically downward, and can shoot all-round (360-degree) images of the tracking target area 11. The fisheye camera 10 is connected to the human tracking device 1 by wire (USB cable, LAN cable, etc.) or wireless (WiFi, etc.), and the image data captured by the fisheye camera 10 is extracted into the human tracking device 1. The image data can be any one of a monochrome image and a color image, and the resolution, frame rate, and format of the image data are arbitrary. In this embodiment, it is assumed that a color (RGB) image extracted at 10fps (10 images per second) is used.

[0058] [Human tracking device]

[0059] The human body tracking device 1 has one or more processors, a main storage device, an auxiliary storage device, a communication device, an input device, and an output device as its main hardware components. The processor executes the following various processes by executing a computer program. It should be noted that a part or all of the processes can also be performed by a dedicated hardware circuit.

[0060] The human body tracking device 1 of the present embodiment includes an image input unit 20 , a human body detection unit 21 , a tracking unit 22 , a learning unit 23 , a storage unit 24 , a visual saliency map calculation unit 25 , a restoration unit 26 , and an output unit 27 .

[0061] The image input unit 20 has a function of extracting image data from the fisheye camera 10 . The extracted image data is transferred to the human body detection unit 21 and the tracking unit 22 .

[0062] (Human body detection unit)

[0063] The human body detection unit 21 has a function of detecting a human body from a fisheye image using an algorithm for detecting a human body. The human body detected by the human body detection unit 21 is the object of tracking processing by the tracking unit 22. It should be noted that the human body detection unit 21 can only detect new people in the image, or exclude people near the location where the person to be tracked exists from the objects of detection processing. In addition, the human body detection unit 21 can detect people in the entire image at a constant time interval or frame interval, and then the tracking unit 22 performs tracking processing.

[0064] (Tracking Department)

[0065] The tracking unit 22 is a functional unit that tracks the tracking object in the moving image, and specifies the position of the tracking object person in the current frame image. The tracking unit 22 first uses the area including the detection position of the human body detection unit 21 as the target area, and specifies the position of an object having the same characteristics as the person detected from the target area. Then, the vicinity of the position specified by the tracking unit 22 for the previous frame image is used as the target area, and the position of the tracking object person is specified from the current frame image.

[0066] The tracking unit 22 includes a feature quantity extraction unit 101 , a response map generation unit 102 , and a loss determination unit 103 .

[0067] The feature extraction unit 101 extracts image features from the target area. The feature extraction unit 101 extracts one or both of the features related to the shape and the features related to the color as image features. An example of the feature related to the shape is HOG (Histogram of Gradient), and an example of the feature related to the color is a color histogram and a color names feature.

[0068] The response map generation unit 102 generates a response map (likelihood map) indicating the probability of the existence of the tracking object for each position in the target area, using the feature amount extracted from the input image and the correlation filter stored in the storage unit 24 .

[0069] The loss determination unit 103 determines whether tracking loss has occurred. For example, the loss determination unit 103 determines that tracking loss has occurred when the maximum likelihood of the response graph is smaller than a threshold value TH1. The threshold value TH1 can be appropriately set according to system requirements. It should be noted that when it can be inferred that the tracked object has moved out of the shooting range, the loss determination unit 103 may not determine that tracking loss has occurred, but may determine that frame out has occurred.

[0070] When the maximum likelihood of the response map is larger than the threshold value TH1, the tracking unit 22 determines that a human body to be tracked exists at the position of the maximum likelihood.

[0071] It should be noted that the tracking unit 22 may also perform tracking based on an evaluation based on color features instead of, or in addition to, the evaluation based on the shape features using the correlation filter. For example, the tracking unit 22 may also generate a foreground likelihood map based on the learned color features, and find the position of the tracked object based on the map. Alternatively, the tracking unit 22 may also find the position of the tracked object based on a composite map of the response map of the correlation filter and the foreground likelihood map. The feature quantity of color information includes a color histogram, a color names feature quantity, and the like.

[0072] (Learning Department)

[0073] The learning unit 23 learns the features of the human body of the tracking object based on the image of the human body detected by the human body detection unit 21 or specified by the tracking unit 22, and stores the learning results in the storage unit 24. Here, the learning unit 23 obtains the correlation filter for evaluation based on the shape feature. The learning unit 23 learns for each frame, and reflects the learning results obtained from the current frame in the past learning results with a predetermined coefficient and updates them. It should be noted that when the evaluation is based on the color feature in the tracking process, the learning unit 23 can also use the feature quantity of color information such as the color histogram and the color names feature quantity.

[0074] (Storage Department)

[0075] The storage unit 24 stores the learning results obtained by the learning unit 23. The storage unit 24 also stores the used feature quantities (HOG, color histogram), parameters of each feature quantity, and hyper parameters of the learning process and the tracking process, such as the learning coefficient.

[0076] The storage unit 24 stores information of a region image indicating a tracking target in a frame image at an early stage of tracking for each region, and stores salient regions.

[0077] In the present embodiment, it is assumed that the parts of the human body are divided into three parts: the head, the upper body, and the lower body. Therefore, the storage unit 24 stores information representing the regional images of the head, the upper body, and the lower body. Here, the information representing the regional image can be the regional image itself, or it can be an image feature quantity (such as a color histogram) obtained from the regional image of each part. It should be noted that the process of dividing the image of the human body into the regional images of each part can be performed by a known image segmentation process. Alternatively, it is also possible to assume the human body ratio and perform regional segmentation of each part according to the ratio.

[0078] The salient part is the part of the tracking target where the visual saliency map corresponding to the frame image at the initial stage of tracking has a maximum value. The visual saliency map can be calculated by the visual saliency map calculation unit 25. By comparing the position of the maximum value of the visual saliency map with the area of ​​each part of the input image, the salient part can be grasped. The storage unit 24 stores the salient part thus obtained. In this embodiment, whether any part of the head, upper body, or lower body is a salient part is stored.

[0079] It should be noted that the "frame image at the beginning of tracking" in this embodiment refers to the second frame image after the start of tracking, in other words, the frame image initially processed by the tracking unit 22. Since the less time passes after the start of tracking, the less noise is mixed in, and the first frame that can obtain the time change of brightness when significance is required is the second frame, the second frame image is used. It should be noted that the first frame image, in other words, the frame image in which the human body detection unit 21 performs human body detection, or the frame images after the third frame can also be taken as the object to acquire and store the above information. The "frame image at the beginning of tracking" is equivalent to the first frame image of one of the multiple frame images constituting the motion image to be processed in the present invention.

[0080] The storage unit 24 also associates and stores the position on the image captured by the fisheye camera 10, the position and size of other parts of the human body when a part of the human body is present at the position, and the visible ratio of each part. In this specification, the above information is referred to as part information, and the table storing the part information is referred to as a part information table.

[0081] Figure 3 3 is a diagram for explaining the part information table 300. As shown in the figure, the part information table 300 stores the center position / size / visible ratio for each part of the head / upper body / lower body. The center position is, for example, the center when the part is represented by a rectangular area. The size is the size of the rectangular area in the vertical and horizontal directions. The visible ratio indicates the degree to which each part is visible, and the visible area ratio when observing the human body directly is 100%.

[0082] exist Figure 3In the figure, the part information table 300 is described as including only one record, but in fact, it includes records for a large number of positions in the image. By referring to the part information table 300, when any part (one part) of the head, upper body, and lower body is located at a specified position, the position of other parts, the size of each part, and the visible ratio of each part can be obtained.

[0083] The method for making the part information table 300 is briefly described. The first method is to analyze the image obtained by actual measurement to find the center position, size, and visible ratio of each part. The second method is to find it by simulation or calculation. For example, modeling can be performed to represent each part of the human body with cylinders of different sizes, and the positions and sizes of the cylindrical models photographed by the fisheye camera 10 when assuming that the human body is located at each position. The visible ratio is calculated as the ratio of the area of ​​the designated part photographed when assuming that there is only a cylindrical model of the designated part and the area of ​​the designated part photographed when all cylindrical models exist.

[0084] (Visual saliency map calculation unit)

[0085] The visual saliency map calculation unit 25 (hereinafter also referred to as the saliency map calculation unit 25) calculates a saliency map based on the input image. Visual saliency is an index that can be obtained for a partial area or each pixel and indicates the degree to which it is easy to attract attention. Visual saliency can be determined by a known method of mathematically modeling the vision of a living body based on features such as brightness, temporal changes in brightness, color, and direction.

[0086] (Restoration Department)

[0087] The restoration unit 26 restores the tracking object when the tracking object is lost. The restoration unit 26 is for the purpose of restoring the tracking object when the tracking is lost, and estimates the position of the tracking object based on the current frame image through a process different from that of the tracking unit 22. The restoration unit 26 includes an area estimation unit 112, a similarity score calculation unit 113, and a reset unit 114. The details of the restoration unit 26 and its sub-function units will be described in detail later together with the description of the flowchart, so only a brief description is given here.

[0088] The restoration unit 26 performs restoration processing on the frame image (current frame image; second frame image) for which the loss determination unit 103 determines that the tracking target has been lost.

[0089] The region estimation unit 112 estimates the region where each part of the tracking object in the current frame image exists based on the maximum value position of the visual saliency map corresponding to the current frame image and the salient part of the tracking object. More specifically, the region estimation unit 112 assumes that there is a salient part at the maximum value position of the visual saliency map, and estimates the region (center position and size) of other parts with reference to the part information table stored in the storage unit 24. When there are multiple maximum values ​​in the visual saliency map, the region estimation unit 112 estimates the region for each maximum value position.

[0090] The similarity score calculation unit 113 calculates the similarity score between the region estimated by the region estimation unit 112 and the human body region in the initial frame. The similarity score calculation unit 113 calculates the similarity between the region image of each part of the current frame image and the initial frame image, performs weighted average on the similarity of each part, and calculates the final similarity score. The weight factor can be determined, for example, based on whether the part is a significant part or a non-significant part, or can be determined based on the visible ratio of the part.

[0091] When the maximum similarity score value is greater than the threshold value TH2, the reset unit 114 determines that the tracking target exists at the position with the maximum similarity score, and notifies the result to the tracking unit 22. In this way, when the restoration process is successful, the tracking unit 22 can continue tracking.

[0092] (Output section)

[0093] The output unit 27 has a function of outputting information such as fisheye images, detection results / tracking results, etc. to an external device. For example, the output unit 27 can display information on a display as an external device, forward information to a computer as an external device, or send information or control signals to a lighting device, air conditioner, or FA device as an external device.

[0094] (Hardware Structure)

[0095] The human body tracking device 1 may be composed of, for example, a computer having a CPU (processor), memory, storage, etc. In this case, Figure 2 The structure shown is realized by loading the program stored in the memory into the internal memory and executing the program in the CPU. The computer may be a general-purpose computer such as a personal computer, a server computer, a tablet terminal, a smart phone, or an embedded computer such as a vehicle-mounted computer. Alternatively, it may be composed of an ASIC, an FPGA, etc. Figure 2 All or part of the structure shown. Or it can also be achieved through cloud computing, distributed computing Figure 2 All or part of the structure shown.

[0096] <Overall treatment>

[0097] Figure 4 This is an overall flow chart of the human body tracking process performed by the monitoring system 2. Figure 4 , explaining the overall process of human body tracking processing.

[0098] In execution Figure 4 Before the process of the flowchart of , hyper parameters for learning and tracking are set in the human body tracking device 1. Examples of hyper parameters include feature quantities to be used, parameters of each feature quantity, and learning coefficients. The input hyper parameters are stored in the storage unit 24.

[0099] In step S101, the image input unit 20 inputs a frame of fisheye image from the fisheye camera 10. At this time, although a planar expansion image in which the distortion of the fisheye image is corrected can be produced for subsequent processing, in the monitoring system 2 of this embodiment, the fisheye image is directly applied (with distortion maintained) to the detection and tracking processing.

[0100] In step S102, the human body detection unit 21 performs human body detection based on the input image. In step S104, the tracking unit 22 sets the detected human body area as the target area for tracking. The target area is an area where the area where the person of the tracking object exists and its surroundings are combined, and is an area where the possibility of the person of the tracking object existing is high. The target area can also be said to be an area that is processed by the tracking unit 22. In the present embodiment, the initial position of the person of the tracking object is detected by the human body detection unit 21, but it can also be detected by other methods such as input by the user.

[0101] Next, the processing of steps S104 to S116 is repeatedly performed. When the termination condition is satisfied in the termination determination of step S104, the processing is terminated. The termination condition may be, for example, the exit of the frame of the person being tracked or the end of the moving image.

[0102] In step S105 , the visual saliency map calculation unit 25 generates a visual saliency map based on the input image. The generated visual saliency map is stored in the storage unit 24 .

[0103] In step S106, it is determined whether the current frame is the initial image. Here, the initial image refers to a frame image to which the initial position of the tracking target person is assigned, and generally refers to a frame image in which the tracking target person is detected by the human body detection unit 21. If the current frame is the initial image (S106-YES), the process proceeds to step S115, and if it is not the initial image (S106-NO), the process proceeds to step S107.

[0104] In step S107, it is determined whether the current frame is the second image. Here, the second image refers to the next frame image of the frame image to which the initial position of the person to be tracked is assigned. If the current frame is the second image (S107-YES), the process proceeds to step S108, and if it is not the second image (S107-NO), the process proceeds to step S111.

[0105] In step S108 , the learning unit 23 obtains a position in the target region where the saliency is maximized in the visual saliency map.

[0106] In step S109, the learning unit 23 determines which part of the tracking target corresponds to the position where the saliency reaches the maximum value, and stores the part as a salient part in the storage unit 24. At this point, when the target area is not segmented (segmented) for each part, the human body tracking device 1 performs the segmentation process in step S109.

[0107] In step S110, the learning unit 23 calculates a color histogram for each part of the human body to be tracked, and stores it in the storage unit 24. It should be noted that although the color histogram is stored here, other image feature quantities may be calculated / stored when the similarity of the regional images between frames can be compared.

[0108] In step S111, the tracking unit 22 performs tracking processing. For details of the tracking processing, refer to Figure 6 , which will be described later, is a process of applying a correlation filter to an input image and determining whether a target area exists at a position where the response (likelihood) is maximum.

[0109] In step S112, the loss determination unit 103 determines whether tracking loss has occurred. When the maximum value of the response (likelihood) is smaller than the threshold value TH1, it can be determined that tracking loss has occurred. When tracking loss has occurred (S112-YES), the process proceeds to step S113, and when tracking loss has not occurred (S112-NO), the process proceeds to step S115.

[0110] In step S113, the restoration unit 26 performs restoration processing. For details of the restoration processing, refer to Figure 7 Etc., described later.

[0111] In step S114, it is determined whether the restoration process is successful. If the restoration is successful (S114-YES), the process proceeds to step S115 to continue the tracking process, and if the restoration fails (S114-NO), the tracking process ends.

[0112] In step S115, the learning unit 23 performs a learning process of the correlation filter. For details of the learning process, refer to Figure 5 , which will be described later.

[0113] In step S116, the image input unit 20 inputs the fisheye image of the next frame from the fisheye camera 10. After step S116, the process returns to step S104, and the above-mentioned process is repeated until the tracking end condition is satisfied.

[0114] In this way, the tracking process S111 specifies the position of the person to be tracked for each frame to achieve tracking. In addition, the tracking process of this embodiment includes a recovery process using information stored in an initial frame when tracking is lost.

[0115] <Learning Process>

[0116] Figure 5 This is a flowchart showing the details of the learning process of step S115. Figure 5 , the learning process is explained.

[0117] The learning unit 23 first cuts out the target area from the current frame image (S201). The target area is an area including the foreground area and the background area of ​​the tracking target person. The foreground area is the area where the tracking target person exists, and the background area is the area where the tracking target person does not exist. The size of the background area is determined according to the size of the foreground area. For example, the size of the background area is determined so that the size of the foreground area is a specified ratio (for example, 1 / 3) of the overall size of the target area. It should be noted that because the target area is updated at the end of the tracking process so that the center is the position of the tracking target person ( Figure 6 Therefore, the center of the target area is the same as the center position of the person being tracked.

[0118] The learning unit 23 obtains the brightness feature and the HOG feature as the image feature of the target area (S202). The HOG feature is a feature that makes the brightness gradient direction of the local area a histogram, and is used as a feature representing the shape / contour of the object. Here, the HOG feature is used, but other feature that represents the shape / contour of the object, such as the LBP feature, the SHIFT feature, and the SURF feature, may also be used. In addition, a brightness feature may be used instead of the brightness feature. It should be noted that when the brightness feature and the HOG feature have been obtained in the tracking process, there is no need to re-obtain the above-mentioned feature. Because the above-mentioned feature can be obtained in the form of an image, the feature is also referred to as a feature image in this specification.

[0119] The learning unit 23 obtains a correlation filter that makes the response have a peak at the center of the target area (S203). Specifically, after extracting the HOG feature, a filter that has a peak only at the center and is closest to the ideal response is obtained with respect to the correlation of the feature itself, thereby obtaining a correlation filter. In the case of using a method of calculating a correlation filter in Fourier space as a tracking algorithm of a correlation filter model, the feature image may be multiplied by a window function.

[0120] When the current learning is the initial learning (S204-YES), the correlation filter generated in step S203 is directly stored in the storage unit 24. On the other hand, when the current learning is the second or later learning (S204-NO), the process proceeds to step S205. In step S205, the learning unit 23 synthesizes the correlation filter obtained in the previous time (the correlation filter stored in the storage unit 24) and the correlation filter obtained in step S203 this time, thereby obtaining a new correlation filter, and stores it in the storage unit 24.

[0121] <Tracking Process>

[0122] Figure 6 This is a flowchart showing the details of the tracking process of step S111. Figure 6 , the tracking process is explained.

[0123] The tracking unit 22 cuts out the last target area from the current frame image (S301). In the last tracking process, the target area is updated so that the center of the target area is the position of the person to be tracked in the last frame image.

[0124] The feature extraction unit 101 extracts brightness feature and HOG feature as feature of the target area (S302). Since the above feature can be obtained in the form of an image, the feature is also referred to as a feature image in this specification. The brightness feature image has the same resolution as the frame image, but the HOG feature image obtains the feature in each unit (for example, every 3×3 pixels), so its resolution is lower than that of the frame image.

[0125] The response map generation unit 102 obtains a response map (likelihood map) based on the HOG feature image in the target area and the correlation filter stored in the storage unit 24 (S303). The tracking unit 22 updates the position corresponding to the maximum value of the response map to the center position of the next target area (S304). In addition, the tracking unit 22 updates the size of the target area based on the updated center position of the target area (S305). The size update process is performed based on a geometric relationship based on the configuration of the fisheye camera 10, the camera viewpoint, the camera parameters, and the position of the character model whose height and width are determined. Alternatively, it can be a known method of using an image pyramid for scale estimation.

[0126] By the above processing, the tracking process is completed and the target area in the current frame image is obtained. The center of the target area after the tracking process is completed is the center position of the tracking object person, and the foreground area in the target area is the existence area (boundary box) of the tracking object person.

[0127] It should be noted that a specific example of tracking processing is described here, but other algorithms other than the above may be used. For example, correction processing may be performed on the response map based on the correlation filter to improve the tracking accuracy.

[0128] <Recovery Process>

[0129] Figure 7 This is a flowchart showing the details of the restoration process in step S113. Figure 8 is the data flow diagram for recovery processing. It should be noted that Figure 8 In FIG. 1 , information pre-stored based on the initial frame is also shown. Figure 7 , Figure 8 The recovery process is described below.

[0130] In step S401 , the restoration unit 26 obtains the visual saliency map 807 corresponding to the current frame image 806 obtained in step S105 from the storage unit 24 , and obtains the position of the maximum value in the visual saliency map 807 .

[0131] The loop process L1 includes steps S402 to S406 and is executed for each local maximum value position found in step S401. The restoration unit 26 calculates a similarity score for each local maximum value position of the visual saliency map 807 corresponding to the frame image where tracking loss has occurred through the loop process L1.

[0132] In step S402, the region estimation unit 112 sets the maximum value position of the visual saliency map 807 as the position of the salient part 805 of the current frame image. Based on the visual saliency map 804 corresponding to the second frame image (initial frame image) 801 in step S109, the salient part 805 is determined in advance and stored in the storage unit 24.

[0133] In step S403, the region estimation unit 112 estimates the region (position and size) of each human body part based on the position of the significant part 805. The storage unit 24 stores a part information table 811 (300), which stores the position / size / visible ratio of each human body part. The region estimation unit 112 can obtain the estimated region 808 of each part based on the position of the significant part 805 and the part information table 811.

[0134] In this way, through the processing of steps S402 to S403 , the region estimating unit 112 can estimate the region where each part of the tracking target exists when it is assumed that a salient part exists at the maximum value position.

[0135] The loop processing L2 includes steps S404 to S405, which are performed for the estimated regions of each part obtained in step S403. In step S404, the similarity score calculation unit 113 calculates the color histogram 809 based on the estimated region 808 of the part. In step S405, the similarity score calculation unit 113 calculates the similarity between the color histogram 809 of the current frame and the color histogram 803 of the second frame image (initial frame image) 801 for the target part. The color histogram 803 of the second frame image is obtained in advance in step S110 and stored in the storage unit 24. The similarity calculated here represents the similarity of each part between the initial frame image and the current frame image.

[0136] When the similarities are calculated for all parts in the loop processing L2, in step S406, the similarity score calculation unit 113 integrates the similarities 809 of each part to calculate a similarity score 810. In this embodiment, the similarity score calculation unit 113 determines the weighted average of the similarities of each part using a weight factor determined according to the visible ratio of each part and whether it is a significant part as the final similarity score.

[0137] More specifically, the similarity score calculation unit 113 calculates the similarity score based on the following formula (1).

[0138]

[0139] Here, SCORE is the similarity score, N is the number of parts, i is the subscript representing the part, and s i is the similarity of part i, WV iis the weight factor based on the visible ratio of part i, WS i is the weight factor based on whether the part i is a significant part, WV imax For WV i The maximum possible value, WS imax For WS i The maximum possible value, s imax For i The maximum possible value.

[0140] Weight factor WV i In this embodiment, the weight factor WV is the same as the value of the visible ratio of the part i. The visible ratio of each part can be known by referring to the part information table 811 (300). i It does not need to be the same value as the visibility ratio of the part i, and may be a value determined according to the visibility ratio of the part i.

[0141] Weight factor WS i When the part i is a prominent part, the fixed value A is used. When the part i is a non-prominent part, the weight factor WS for the prominent part is (1-A) / (N-1). i It can be the weight factor WS relative to the non-significant part i Therefore, the fixed value A is preferably a value larger than 1 / N. For example, when the number of parts is three, and A=0.7, the weight factor WS for the significant part is i The weight factor WS for non-significant parts is 0.7. i is 0.15.

[0142] When the above-described processing of steps S402 to S406 is completed for each maximum value of the visual saliency map, the process proceeds to step S407 .

[0143] In step S407, the resetting unit 114 determines whether the maximum value among the similarity scores obtained for each maximum value is greater than the threshold value TH2. If the maximum similarity score is greater than the threshold value TH2 (S407-YES), the process proceeds to step S408. If it is not greater than the threshold value TH2 (S407-NO), the process proceeds to step S409.

[0144] In step S408, the resetting unit 114 determines that among the maximum value positions of the visual saliency map, the position assigned the largest similarity score has a salient part of the person to be tracked. The resetting unit 114 notifies the tracking unit 22 of the successful recovery of the tracking loss and the position of the person to be tracked, and ends the processing. It should be noted that the position of the tracking object notified by the resetting unit 114 to the tracking unit can be the position of the salient part, or it can be another position obtained based on the position of the salient part. As an example of other positions, the center position of the person to be tracked can be cited. The center position is obtained as the center of an area including salient parts and non-salient parts, and an area including all parts.

[0145] In step S409, the resetting unit 114 notifies the tracking unit 22 that the recovery of the tracking loss has failed, and ends the processing.

[0146] <Working example>

[0147] Reference Fig. 9 , Fig.10 , the work of recovery processing is explained based on a specific example. Fig. 9 This is a diagram for explaining information obtained in the early stage (the second frame in this example) after the start of tracking. Fig.10 This is a diagram for explaining the processing performed when tracking loss occurs.

[0148] Fig. 9 A represents an image 901 in which the part of the person of the tracking object in the second frame (initial frame) is enlarged. Image 901 shows a human body 902 of the tracking object. The learning unit 23 divides the human body 902 into three parts, namely, the head 903, the upper body 904, and the lower body 905, through region segmentation processing. In the figure, the region of each part is represented by a rectangle, but each region can also be represented by a shape other than a rectangle. The learning unit 23 calculates color histograms 906a, 906b, and 906c corresponding to the head 903a, the upper body 903b, and the lower body 903c, respectively, and stores them in the storage unit 24.

[0149] Fig. 9 B represents a visual saliency map 910 obtained based on the second frame image. The visual saliency map 910 has a size corresponding to the entire second frame image, but only the portion corresponding to the image 901 is shown here. The learning unit 23 determines the salient part of the human body 902 based on whether the maximum value position 911 of the visual saliency map 910 is located in the area of ​​any part of the human body 902, and stores it in the storage unit 24. In this example, since the maximum value position 911 is located in the upper body 903b, the salient part is determined to be the upper body.

[0150] The above information is obtained at the beginning of tracking (the second frame in this example) and stored in the storage unit 24. Next, the recovery process when tracking loss occurs will be described.

[0151] Fig.10 A represents a visual saliency map 1001 obtained from the Nth frame image. The Nth frame is a frame image where tracking loss occurs. In this example, the visual saliency map 1001 has two local maximum values ​​1002 and 1003.

[0152] The restoration unit 26 (region estimation unit 112) estimates the region of each part when it is assumed that the significant part (in this example, the upper body) is located at the local maximum values ​​1002 and 1003. The restoration unit 26 can obtain the position and size of each part by referring to the part information table 300. In this example, corresponding to the local maximum value 1002, the region 1004a of the head, the region 1004b of the upper body, and the region 1004c of the lower body can be obtained. Similarly, corresponding to the local maximum value 1003, the region 1005a of the head, the region 1005b of the upper body, and the region 1005c of the lower body can be obtained.

[0153] The similarity score calculation unit 113 first obtains a color histogram based on the area of ​​each part. For example, the color histograms 1010a to 1010c corresponding to the areas 1004a to 1004c of each part are obtained. Then, the similarity score calculation unit 113 compares the color histograms 1010a to 1010c of the area of ​​each part with the color histograms 906a to 906c obtained based on the second frame image, and calculates the similarity s. i .

[0154] The similarity score calculation unit 113 integrates the similarities s and the weight factors determined according to the visible ratio of each part and whether it is a significant part. i , and calculate the final similarity score. For each maximum value 1002, 1003 of the visual saliency map, calculate the similarity score.

[0155] The resetting unit 114 determines that the tracking object exists at the maximum position assigned with the maximum similarity score. However, when the maximum similarity score is smaller than the threshold value TH2, it is determined that the recovery has failed. The resetting unit 114 uses the center position of the entire region of each part of the human body as the center position of the tracking object and resetting the target region. In addition, the resetting unit 114 resetting the size of the target region based on the entire region of each part of the human body.

[0156] <Advantageous Effects of the Present Embodiment>

[0157] In this embodiment, in a human body tracking device, it is possible to perform recovery with good accuracy when tracking fails. In this embodiment, information on which part a salient part is and a color histogram of each part area are stored based on an initial frame. When the person of the tracking object moves and changes his position in the image, the relative position to the fisheye camera 10 changes, and the position displayed in the image changes, or the shape is distorted, but it can be assumed that where the salient part is located does not change. Therefore, in a frame where tracking is lost, the maximum value position of the visual saliency map can be regarded as a candidate position of the salient part of the tracking object. In addition, when a certain designated part is located at a designated position on the image, the positions of other parts and the size / visible ratio of each part can be calculated in advance. Therefore, it is estimated that the area of ​​each part when assuming that there is a salient part at the maximum value position of the visual saliency map, the similarity of the color histogram of the current frame and the initial frame is calculated, and the above similarity is integrated to calculate the similarity score. By considering the maximum value of the visual saliency map as the position of the salient part and calculating the similarity based on the color histogram, it is possible to achieve a robust recovery with respect to shape changes. In addition, when integrating the similarity, the weight factor for the part is determined based on whether the part is a significant part or a non-significant part, and based on the visible ratio of the part. This is based on the fact that in the calculation of the similarity score, a significant part should be given more weight than a non-significant part, and a part with a larger visible ratio should be given more weight. By weighting based on the visible ratio, it is also robust against occlusion. Since the restoration process is implemented through the above process, in this embodiment, it is possible to perform restoration with good accuracy.

[0158] <Other>

[0159] The above-mentioned embodiments are merely exemplary configuration examples of the present invention. The present invention is not limited to the above-mentioned specific embodiments, and various modifications are possible within the scope of the technical concept.

[0160] In addition, in the above-described embodiment, the tracking unit 22 performs tracking processing using a correlation filter, but other algorithms may be used for tracking. For example, deep learning models such as CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory), and pattern recognition models such as SVM (Support Vector Machine) may be used for tracking.

[0161] In addition, in the above embodiment, the fisheye image plane is not expanded and processed, but the image after the fisheye image plane is expanded can be used as the processing object, and the image captured by a normal camera can also be used as the processing object. In addition, in the above embodiment, the optical axis of the camera is set vertically downward at the viewpoint of the camera, but the optical axis of the fisheye camera can also be tilted relative to the vertical direction.

[0162] In addition, the color histogram of the salient parts and each part is obtained according to the second frame ( Figure 4 S108 to S110 of FIG. 10A , but it is not particularly limited which frame image is used to obtain the above information. However, since the earlier the tracking is, the less noise is mixed, it is preferred to use a frame close to the start of tracking. Therefore, the above information can also be obtained based on the frame after the start of tracking (the first frame) or the frame after the third frame.

[0163] In addition, when calculating the similarity of the restoration process, the color histogram is used, but the similarity can also be determined based on information other than the color histogram. However, in an image with a large distortion such as an image taken by a fisheye camera, it is preferable to use a feature quantity that is robust to deformation, such as a color histogram or a brightness histogram.

[0164] In addition, in the above-described embodiment, when the similarities of the respective parts are integrated and the similarity scores are calculated ( Figure 7 In S406), the weight factor WSi corresponding to whether it is a significant part is multiplied by the weight factor WVi corresponding to the visible ratio as the weight factor. However, other methods such as addition can also be used based on the above two weight factors WS i , WV i , determine the applicable weight factor. Alternatively, it is also possible to i With WV i In addition, WS can also be considered. i With WV i In addition, the similarity score can be calculated by simple averaging rather than weighted averaging.

[0165] In addition, in this embodiment, the positions of other parts and the size / visibility ratio of each part when the designated part is located at a designated position in the image are obtained by referring to the pre-stored part information table, but they can also be obtained by calculation when necessary. The calculation method is the same as that in the case of pre-finding in the above embodiment, so the description is omitted.

[0166] In the present embodiment, the tracking unit 22 tracks the person detected by the human body detection unit 21 . However, a user (person) may specify a tracking target and the tracking unit 22 may track the specified target.

[0167] <Note>

[0168] 1. An object tracking device (1), comprising:

[0169] Storage unit (24);

[0170] A tracking unit (22) for tracking an object in a moving image;

[0171] a recovery unit (26) for recovering the object when the object is lost;

[0172] The storage unit (24) stores, for each part, information (803) representing a region image of the object in a first frame image (801) of a plurality of frame images constituting the moving image, and stores a part (805) of the object where a visual saliency map (804) corresponding to the first frame image (801) takes a maximum value,

[0173] The recovery unit (26),

[0174] When the object is lost, for each maximum value position of a visual saliency map (807) corresponding to a second frame image (806) which is a frame image in which the object is lost, (1) an area (808) is estimated where each part of the object exists when the part of the object that has a maximum value in the first frame image exists at the maximum value position, (2) a similarity score (810) is calculated based on the similarity of the regional images of each part of the first frame image and the second frame image,

[0175] It is determined that the object exists at a maximum value position where the similarity score is the largest in the second frame image.

[0176] 2. A tracking loss recovery method (S113), performed by the object tracking device (1) when the tracked object is lost, the tracking loss recovery method performs:

[0177] A storing step of storing (S110) information (803) representing a region image of the object in a first frame image (801) of a plurality of frame images constituting the motion image for each region, and storing (S108-S109) a region (805) of the object where a visual saliency map (804) corresponding to the first frame image (801) takes a maximum value;

[0178] A calculation step, wherein when the object is lost, for each maximum value position of a visual saliency map (807) corresponding to a second frame image (806) as a frame image in which the object is lost, (1) infers (S402-S403) an area (808) where each part of the object exists when it is assumed that a part of the object having a maximum value in the first frame image exists at the maximum value position, and (2) calculates (S404-S406) a similarity score based on the similarity of the regional images of each part of the first frame image and the second frame image;

[0179] A determination step is performed to determine (S408) that the object exists at a maximum position in the second frame image where the similarity score is the largest.

[0180] Description of Reference Numerals

[0181] 1 Human tracking device; 2 Surveillance system; 10 Fisheye camera; 11 Tracking object area; 12 Ceiling; 13 People

Claims

1. An object tracking device, comprising: Storage unit; A tracking unit that tracks an object in a moving image; a recovery unit for recovering the object when the object is lost; The object tracking device is characterized in that The storage unit stores, for each part, information of a region image representing the object in a first frame image of one of a plurality of frame images constituting the motion image, and stores a part of the object where a visual saliency map corresponding to the first frame image takes a maximum value, When the object is lost, the restoration unit (1) estimates, for each maximum position of a visual saliency map corresponding to a second frame image as a frame image in which the object is lost, an area where parts of the object exist when it is assumed that a part of the object that takes a maximum value in the first frame image exists at the maximum position, and (2) calculates a similarity score based on the similarity of the regional images of each part of the first frame image and the second frame image. The restoration unit determines that the object exists at a local maximum position where the similarity score is the largest in the second frame image.

2. The object tracking device according to claim 1, characterized in that: The restoration unit calculates the similarity of the image feature quantities of the regional images of each part of the first frame image and the second frame image, integrates the similarities, and calculates the similarity score.

3. The object tracking device according to claim 2, characterized in that: The restoration unit integrates the similarities using weights corresponding to the visible proportions of the parts in the second frame image to calculate the similarity score.

4. The object tracking device according to claim 2 or 3, characterized in that: The restoration unit integrates the similarities between a portion of the object having a maximum value in the visual saliency map corresponding to the first frame image and other portions thereof using different weights to calculate the similarity score.

5. The object tracking device according to claim 2 or 3, characterized in that: The restoration unit calculates the similarity score according to the following formula (1): Here, SCORE is the similarity score, N is the number of parts, i is the subscript representing the part, and s i is the similarity of part i, WV i is the weight factor based on the visible ratio of part i, WS i is the weight factor based on whether the part i is a significant part, WV imax For WV i The maximum possible value, WS imax For WS i The maximum possible value, s imax For i The maximum possible value.

6. The object tracking device according to any one of claims 1 to 3, characterized in that: The storage unit further associates and stores the position on the image with the position where the other part of the object exists when the other part of the object exists at the position. The restoration unit estimates the area where each part of the object in the second frame image exists based on the maximum value position in the second frame image, the part of the object where the visual saliency map corresponding to the first frame image takes a maximum value, and the storage unit.

7. The object tracking device according to claim 6, characterized in that: The storage unit also stores the visible ratio of each part of the object for each position on the image.

8. The object tracking device according to any one of claims 1 to 3, characterized in that: The first frame image is a frame image after the tracking of the object is started or a frame image next to it.

9. The object tracking device according to any one of claims 1 to 3, characterized in that: The method further comprises a learning unit that performs learning of a correlation filter based on an image of a region where the object has been determined to exist, The tracking unit determines that an object exists at a position where the index calculated using the correlation filter is the highest.

10. The object tracking device according to claim 9, characterized in that: The tracking unit determines that the object is lost when the maximum value of the index calculated using the correlation filter is equal to or smaller than a threshold value.

11. The object tracking device according to any one of claims 1 to 3, characterized in that: A shooting unit for shooting the moving image is also provided.

12. The object tracking device according to claim 11, characterized in that: The shooting unit is a fisheye camera.

13. A tracking loss recovery method, performed by an object tracking device when the object being tracked is lost, the tracking loss recovery method being characterized by executing: a storing step of storing, for each part, information of a region image representing the object in a first frame image, which is one of a plurality of frame images constituting a moving image, and storing a part of the object where a visual saliency map corresponding to the first frame image takes a maximum value; A calculation step, when the object is lost, for each maximum position of a visual saliency map corresponding to a second frame image as a frame image in which the object is lost, (1) inferring an area where each part of the object exists when it is assumed that a part of the object that takes a maximum value in the first frame image exists at the maximum position, and (2) calculating a similarity score based on the similarity of regional images of each part of the first frame image and the second frame image; A determination step is performed to determine that the object exists at a maximum position in the second frame image where the similarity score is the largest.

14. A computer-readable storage medium, characterized in that: A program for causing each step of the method described in claim 13 to be executed on a computer is stored.

Images