Image processing device and image processing method

Abstract

To provide a technology for improving tracking accuracy for a tracking target.SOLUTION: Tracking processing for tracking a tracking target in a captured image is performed using a tracking model. When it is detected that a first object to be tracked has been covered by a second object during tracking, the tracking model is switched to a first model that tracks the second object. When it is detected that the first object has been uncovered, the tracking model is switched to a second model that tracks the first object.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to tracking technology. [Background technology]

[0002] There are several techniques for tracking objects in images, including those that use brightness and color information, template matching, and deep neural networks (DNNs). However, regardless of the method, it is important to deal with situations when the tracking target is occluded by another object. Conventional methods for dealing with this situation include setting multiple feature points for the tracking target, which allows tracking even when the target is partially occluded, or continuing tracking by predicting the target's moving position using the target's motion vector.

[0003] Furthermore, Patent Document 1 discloses a technology that, when a tracking target is blocked by an obstruction, recognizes the obstruction as a temporary tracking target and continues tracking, and when the tracking target reappears, returns the tracking target to the tracking target, thereby enabling continuation of tracking of the tracking target. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 4769943 Summary of the Invention [Problem to be solved by the invention]

[0005] In the method disclosed in Patent Document 1, the threshold for the tracking model to detect the tracking target is lowered in order to track an obstructing object as a temporary tracking target. However, in the method of Patent Document 1, if another object that looks similar to the obstructing object appears in the image while the obstructing object is being tracked as a temporary tracking target, the temporary tracking target may be shifted to the other object, and there is a possibility that the tracking of the tracking target may not function properly thereafter. The present invention provides a technology for improving the tracking accuracy of the tracking target compared to conventional technology. [Means for solving the problem]

[0006] One aspect of the present invention is a tracking device that performs a tracking process to track a tracking target in a captured image using a tracking model; While the tracking means is tracking a first object as a tracking target, 、 a switching means for switching the tracking model to a first model that tracks the second object as a tracking target when it is detected that the first object is occluded by a second object, and for switching the tracking model to a second model that tracks the first object as a tracking target when it is detected that the occlusion has been released; 、 Equipped with 、 The switching means detecting occlusion of the first object by the second object based on a first map representing an area of ​​the first object in the captured image; It is characterized by the following. [Effects of the Invention]

[0007] According to the configuration of the present invention, it is possible to improve the tracking accuracy of the tracking target compared to the conventional technology. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an example of the hardware configuration of an image processing apparatus. [Figure 2] FIG. 2 is a block diagram showing an example of the functional configuration of the system. [Figure 3] 10 is a flowchart of a process for setting a tracking target. [Figure 4] 10 is a flowchart of a tracking process. [Figure 5] FIG. 2 is a block diagram showing an example of the functional configuration of the system. [Figure 6] 10 is a flowchart of a tracking process. [Figure 7] FIG. 1A is a diagram showing an example of an obstructing object area map, and FIG. 1B is a diagram showing an example of an obstructed object area map. [Figure 8] FIG. 10 is a diagram showing an example of detecting the occurrence of occlusion. [Figure 9] FIG. 10 is a diagram showing an example of detecting removal of occlusion. [Figure 10] FIG. 10 is a diagram for explaining the processing in step S303. [Figure 11] FIG. 2 is a block diagram showing an example of the functional configuration of the system. [Figure 12] 10 is a flowchart of a tracking process. [Figure 13] 10A and 10B are diagrams illustrating an example of a method for determining whether to switch a tracking target. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] [First embodiment] In this embodiment, an image processing device will be described that performs tracking processing to track a tracking target in a captured image using a tracking model. When the image processing device according to this embodiment detects that a first object is occluded by a second object while tracking the first object as the tracking target, the image processing device according to this embodiment switches the tracking model to a first model that tracks the second object as the tracking target. Furthermore, when the image processing device according to this embodiment detects that the occlusion has been released, the image processing device according to this embodiment switches the tracking model to a second model that tracks the first object as the tracking target.

[0011] Here, the "tracking target" refers to an object that is determined as the tracking target by the processing described below, either an object (tracking target) that is pre-designated as the target object to be tracked, or an object (obstructing object) that is determined to be blocking the tracking target.

[0012] First, an example of the hardware configuration of an image processing device according to this embodiment will be described using the block diagram in Fig. 1. Note that the configuration shown in Fig. 1 is just an example and can be modified / altered as appropriate.

[0013] The CPU 101 executes various processes using computer programs and data stored in the memory 102. As a result, the CPU 101 controls the operation of the entire image processing device, and also executes or controls various processes that will be described as being performed by the image processing device.

[0014] The memory 102 has an area for storing computer programs and data loaded from the storage unit 104, and an area for storing data received from the outside via the communication unit 108. The memory 102 also has a work area used when the CPU 101 executes various processes. In this way, the memory 102 can provide various areas as needed.

[0015] The input unit 103 is a user interface such as a keyboard, a mouse, or a touch panel, and allows the user to input various instructions to the CPU 101 by operating it.

[0016] The storage unit 104 is a non-volatile memory device such as a hard disk drive. The storage unit 104 stores an OS (operating system), computer programs and data for causing the CPU 101 to execute or control various processes that will be described as being performed by the image processing device, etc. The computer programs and data stored in the storage unit 104 are loaded into the memory 102 as appropriate under the control of the CPU 101, and become the subject of processing by the CPU 101.

[0017] Display unit 105 is a display device having a liquid crystal screen or a touch panel screen, and can display the processing results by CPU 101 as images, characters, etc. Note that display unit 105 may also be a projection device such as a projector that projects images and characters.

[0018] The communication unit 106 is a communication interface for performing data communication with an external device via a wired and / or wireless network such as a LAN or the Internet. The CPU 101, the memory 102, the input unit 103, the storage unit 104, the display unit 105, and the communication unit 106 are all connected to a system bus 107.

[0019] Next, an example of the functional configuration of a system according to this embodiment including such an image processing device will be described using the block diagram of Fig. 2. As shown in Fig. 2, in the system according to this embodiment, an image processing device 100 is connected to an image capturing device 200 and an information storage unit 170, and the image processing device 100 performs data communication with the image capturing device 200 and the information storage unit 170 via a communication unit 106.

[0020] First, the imaging device 200 will be described. The imaging device 200 is an imaging device such as a digital camera or a surveillance camera. The imaging device 200 may be a device that captures moving images and acquires images of each frame in the moving images as captured images, or may be a device that captures still images periodically or irregularly and acquires the still images as captured images. The imaging device 200 outputs the acquired captured images to the image processing device 100.

[0021] Next, the information storage unit 170 will be described. The information storage unit 170 is a "storage device capable of communicating with the image processing device 100 via a wired and / or wireless network such as a LAN or the Internet," such as a nonvolatile memory device such as a hard disk drive device or a server device. The information storage unit 170 may also be an external memory device such as a USB memory device. The image processing device 100 stores information necessary for tracking a tracking target in a captured image in the information storage unit 170 as appropriate. Note that the information storage unit 170 is not essential, and the memory unit 104 may be used instead of the information storage unit 170.

[0022] Next, the image processing device 100 will be described. The image processing device 100 acquires a captured image output from the imaging device 200, and tracks a tracking target in the acquired captured image. In the following, each functional unit in the image processing device 100 may be described as the subject of processing. However, in reality, the functions of the functional units are realized by the CPU 101 executing a computer program that causes the CPU 101 to execute the functions of the functional units.

[0023] In order to perform such tracking processing in the image processing device 100, it is necessary to set in advance an object to be tracked (tracking target). Such processing for setting the tracking target will be described with reference to the flowchart of FIG.

[0024] In step S101, the acquisition unit 110 acquires one captured image captured by the imaging device 200. This one captured image is, for example, a captured image of the first frame in a group of captured images (a group of still images or a group of images in a moving image) to be subjected to tracking processing. Note that the acquisition unit 110 may acquire an image in a partial region of the captured image, such as an object region of an object in the captured image, or a partial object region of the object (an object region of a part such as a face), as a new captured image.

[0025] In step S102, the setting unit 120 performs a setting process to set one of the objects included in the captured image acquired by the acquisition unit 110 in step S101 as a tracking target. There are various setting processes for setting a tracking target, and the setting process is not limited to a specific setting process.

[0026] For example, the setting unit 120 displays the captured image acquired by the acquisition unit 110 in step S101 on the display unit 105 and accepts a designation operation of the object region of the tracking target from the user. The user checks the captured image displayed on the display unit 105 and performs a designation operation of designating the object region of the tracking target in order to set one of the objects included in the captured image as the tracking target. There are various designation methods for the user to designate the object region, and the present embodiment is not limited to a specific designation method. For example, if the display unit 105 has a touch panel screen, the user may designate the object region of the tracking target on the touch panel screen. Alternatively, the user may designate the object region of the tracking target by operating the input unit 103. The setting unit 120 then sets the object region designated in response to the user's operation as the object region of the tracking target.

[0027] The setting unit 120 may detect an object region of a subject to be tracked from the captured image and set the detected object region of the subject as the object region of the tracking target. As a method for automatically detecting a main subject in a captured image, for example, the method described in Japanese Patent No. 6557033 can be applied.

[0028] Furthermore, the setting unit 120 may set the object region of the tracking target in the captured image by using a technique for detecting an object region of an object from a captured image in combination with a user operation. Examples of techniques for detecting an object region of an object from a captured image include "Liu, SSD: SingleShotMultiboxDetector. In: ECCV2016."

[0029] In step S103, the setting unit 130 constructs a "tracking model (tracking target model) used by the tracking processing unit 140 to track the tracking target" using an image (tracking target image) within the object region set as the object region of the tracking target in step S102. Various models are applicable to the tracking target model, and therefore there are various construction methods for the tracking target model.

[0030] For example, when the tracking target model is a neural network such as a DNN, the setting unit 130 performs a learning process on the neural network using the tracking target image, and acquires the learned neural network obtained by the learning process as the tracking target model.

[0031] Furthermore, for example, if the tracking target model is a tracking model that performs template matching, the setting unit 130 acquires, as the tracking target model, the tracking model that performs template matching with the tracking target image.

[0032] In this way, the setting unit 130 performs processing to construct a tracking target model to be used by the tracking processing unit 140 to track the tracking target, as setting processing of the tracking processing unit 140. The setting unit 130 stores the tracking target model (parameters of the tracking target model, etc.) constructed in this way and the tracking target image in the information storage unit 170.

[0033] Next, the process performed by the image processing device 100 to track the tracking target in each captured image output from the imaging device 200 after the above setting process will be described with reference to the flowchart of FIG.

[0034] In step S200, setting unit 130 sets a tracking target model as a tracking model used by tracking processing unit 140 to track the tracking target. More specifically, setting unit 130 reads out the tracking target model stored in information storage unit 170 in the above setting process, and sets the read tracking target model as the tracking model used by tracking processing unit 140 to track the tracking target. Then, the processes of steps S201 to S205 are performed for each captured image output from imaging device 200.

[0035] In step S201, the acquisition unit 110 acquires a captured image output from the imaging device 200. Note that, similar to step S101 above, an image within a partial region of the captured image may be acquired as a new captured image.

[0036] In step S202, the tracking processing unit 140 performs tracking processing to track the tracking target in the captured image acquired by the acquisition unit 110 in step S201, using the tracking model set by the setting unit .

[0037] In step S202, when the process progresses from step S200 to step S201 to step S202, the tracking processing unit 140 performs tracking processing of the tracking target in the captured image using the tracking target model set by the setting unit 130 in step S200. In this way, tracking processing of the tracking target in the captured image is performed.

[0038] In step S202 when the process progresses from step S205 to step S202, the tracking processing unit 140 performs tracking processing of the tracking target in the captured image using the tracking model set in step S205 by the setting unit 130. As a result, tracking processing of the tracking target switched in step S204 is performed in the captured image.

[0039] When the process proceeds from step S203 to step S202, in step S202, the tracking processing unit 140 performs a tracking process of the tracking target in the captured image using the currently set tracking model.

[0040] The tracking processing unit 140 uses a tracking model based on the object region of the tracking target in the captured image of the previous frame to output a plurality of object candidate regions that are candidates for the object region of the tracking target in the captured image of the current frame, and the likelihood of the object candidate regions (likelihood indicating the likelihood of being a tracking target).The tracking processing unit 140 then determines the object candidate region with the highest likelihood from the plurality of object candidate regions as the object region of the tracking target.When a tracking target model is set as the tracking model to be used, the tracking processing unit 140 outputs a plurality of object candidate regions that are candidates for the object region of the tracking target in the captured image of the current frame, and the likelihood of the object candidate regions.The tracking processing unit 140 then determines the object candidate region with the highest likelihood from the plurality of object candidate regions as the object region of the tracking target.An example of a technology for performing this processing is "Real-Time MDNet, ECCV2018". However, other techniques may be adopted as long as it is possible to calculate multiple object candidate regions that are candidates for the object region to be tracked in the captured image of the current frame and the likelihood of the object candidate regions (likelihood indicating the likelihood of being a tracking target).

[0041] In the above example, tracking processing unit 140 determines the object candidate region with the highest likelihood from among multiple object candidate regions as the object region of the tracking target, but other criteria may be used to determine the object region of the tracking target. For example, the distance between the position of the object region of the tracking target in the captured image of the previous frame and the position of each object candidate region in the captured image of the current frame is calculated. Then, the object candidate region with the shortest distance among the object candidate regions in the captured image of the current frame is determined as the object region of the tracking target.

[0042] If the tracking model is a neural network that performs online learning, tracking processing unit 140 may re-learn the tracking model using an image within the object region of the tracked target in the captured image of the current frame, and store the re-learned tracking model and the image in information storage unit 170. Furthermore, if the tracking model is a tracking model that performs template matching, tracking processing unit 140 may store an image within the object region of the tracked target in the captured image of the current frame in information storage unit 170 as an image that the tracking model uses in template matching.

[0043] When a tracking target model is set as the tracking model, the tracking processing unit 140 periodically or irregularly performs a storage process to store in the information storage unit 170 an image within an object candidate area determined as the object area of ​​the tracking target in the captured image of the current frame.

[0044] In step S203, the occlusion detection unit 150 determines whether the tracking target is occluded by another object (an obstructing object) in the captured image acquired by the acquisition unit 110 in step S201, and whether the occlusion has been released. There are various methods for determining whether the tracking target is occluded by an obstructing object in the captured image, and whether the occlusion has been released, and the present embodiment is not limited to a specific method.

[0045] Determining whether the tracking target is occluded by an obstruction in the captured image can be performed, for example, by the following determination process. That is, if the following condition is satisfied: "the likelihood of an object candidate region determined by tracking processing unit 140 to be the object region of the tracking target in the tracking process of the tracking target is less than a threshold, and another object candidate region overlapping with the object candidate region exists," the occlusion detection unit 150 determines that the tracking target is occluded by an obstruction in the captured image. On the other hand, if this condition is not satisfied, the occlusion detection unit 150 determines that the tracking target is not occluded by an obstruction in the captured image.

[0046] Furthermore, the determination of whether the occlusion has been released can be performed, for example, by the following determination process. When there is another object candidate area that overlaps with an object candidate area that tracking processing unit 140 has determined to be the object area of ​​the occluding object in the tracking process of the occluding object, occlusion detection unit 150 calculates the similarity between the image in the other object candidate area and the "image in the object candidate area determined to be the object area of ​​the tracking target" stored in information storage unit 170. Then, if the similarity is equal to or greater than a threshold, occlusion detection unit 150 determines that the occlusion has been released (the tracking target has reappeared in the captured image).

[0047] As a result of such determination, if it is determined that the tracking target is obscured by an obstruction or that the obstruction has been removed in the captured image acquired by the acquisition unit 110 in step S201 (the occurrence of obstruction is detected or the removal of obstruction is detected), the process proceeds to step S204. On the other hand, if the result of such determination is that neither the occurrence nor the removal of obstruction is detected, the process proceeds to step S201, and the processes in the subsequent steps are performed for the next frame.

[0048] In step S204, the switching unit 160 switches (selects) the tracking target to be tracked by the tracking processing unit 140. Here, the operation of the switching unit 160 in step S204 differs depending on whether the occlusion detection unit 150 detects the occurrence of occlusion or the release of the occlusion.

[0049] When the occlusion detection unit 150 detects the occurrence of occlusion, the switching unit 160 determines "another object candidate area that most overlaps with the object candidate area determined to be the object area of ​​the tracking target" as the object area of ​​the occluding object, and selects the occluding object as the tracking target.

[0050] On the other hand, when the occlusion detection unit 150 detects that the occlusion has been removed, the switching unit 160 determines that "another object candidate area that most overlaps with the object candidate area determined to be the object area of ​​the occluding object" is the object area of ​​the tracking target, and selects the tracking target as the object to be tracked.

[0051] In step S205, switching unit 160 instructs setting unit 130 to set "the tracking model for tracking the tracking target selected in step S204" as the tracking model used by tracking processing unit 140 to track the tracking target. Here, the operation of switching unit 160 in step S205 differs depending on whether occlusion detection unit 150 detects the occurrence of occlusion or the release of the occlusion.

[0052] When occlusion detection unit 150 detects the occurrence of occlusion, switching unit 160 instructs setting unit 130 to set a "tracking model that tracks an obstructing object (obstructing object model)" as a tracking model used by tracking processing unit 140 to track the tracking target. In response to the instruction, setting unit 130 sets the obstructing object model as the tracking model used by tracking processing unit 140 to track the tracking target, and stores the obstructing object model in information storage unit 170. When using a tracking model that performs template matching as the obstructing object model, setting unit 130 constructs a tracking model (obstructing object model) that performs template matching using an image within an object candidate region determined to be an object region of an obstructing object. On the other hand, when a neural network such as a DNN is used as the occluding object model, the setting unit 130 performs a learning process (online learning) of the neural network using images (positive examples) within object candidate areas determined to be object areas of occluders and images (negative examples) within object candidate areas not determined to be object areas of occluders, thereby constructing the trained neural network as the occluding object model.

[0053] In other words, when the occlusion detection unit 150 detects the occurrence of occlusion, it constructs an occlusion model using at least an image within the object candidate area determined as the object area of ​​the occlusion, and the tracking processing unit 140 switches the tracking model used to track the tracking target to the constructed occlusion model.

[0054] On the other hand, when occlusion detection unit 150 detects that occlusion has been removed, switching unit 160 instructs setting unit 130 to set a tracking target model as a tracking model to be used by tracking processing unit 140 to track the tracking target. In response to the instruction, setting unit 130 sets the tracking target model stored in information storage unit 170 as the tracking model to be used by tracking processing unit 140 to track the tracking target.

[0055] In other words, in this embodiment, when the occlusion detection unit 150 detects that the occlusion has been removed, the tracking model used by the tracking processing unit 140 to track the tracking target is switched to the tracking target model stored in the information storage unit 170.

[0056] Unless a new switching occurs, tracking processing unit 140 performs tracking processing of the tracking target using the initially set tracking model (tracking target model) or the tracking model switched to by the most recent switching.

[0057] When tracking an obstructing object as a temporary tracking target, if the obstructing object is not accurately tracked, and another object that looks similar to the obstructing object appears in the image, the temporary tracking target may be shifted to the other object, which may result in the tracking of the target not functioning properly.

[0058] In contrast, in this embodiment, when the tracking target is not obscured by an obstruction, a tracking target model is used to perform more accurate tracking of the tracking target, and when the tracking target is obscured by an obstruction, a obstruction model is used to perform more accurate tracking of the obstruction.As a result, even if another object that looks similar to the obstruction appears in the image while the tracking target is obscured by the obstruction, it is possible to continue tracking the obstruction accurately, and even if the tracking target reappears in the image, it is possible to track the tracking target more accurately.

[0059] [Second embodiment] In each of the following embodiments, including this embodiment, differences from the first embodiment will be described, and unless otherwise specified below, it is assumed that each embodiment is the same as the first embodiment. In this embodiment, a tracking target is determined based on the area of ​​an obstruction (obstruction area) in a captured image and the area of ​​an object (obstructed object) obstructed by the obstruction (obstructed object area).

[0060] An example of the functional configuration of the system according to this embodiment will be described using the block diagram of Fig. 5. As shown in Fig. 5, the system according to this embodiment differs from the first embodiment in that the occlusion detection unit 150 includes a detection unit 151 that detects an occlusion area and an occluded object area, and an occlusion determination unit 152 that detects the occurrence or release of occlusion based on the occlusion area and the occluded object area.

[0061] Next, the processing performed by the image processing device 100 to track the tracking target in each captured image output from the imaging device 200 after the above setting processing will be described with reference to the flowchart in Fig. 6. Note that the same processing steps in Fig. 6 as those shown in Fig. 4 are assigned the same step numbers, and descriptions of these processing steps will be omitted.

[0062] In step S301, the detection unit 151 outputs a map (occluded object area map) that holds a value (likelihood) representing the likelihood of an object being obstructed for each pixel of the captured image, and a map (occluded object area map) that holds a value (likelihood) representing the likelihood of an object being obstructed for each pixel of the captured image.

[0063] An example of an obstructing object region map is shown in FIG. 7(a). An example of an obstructed object region map is shown in FIG. 7(b). FIG. 7(a) shows a state in which two people are included, with one person partially obstructed by the other person. Region 701 is a region of the person as an obstructing object, and is, for example, a region corresponding to pixels where the value (likelihood) representing the likelihood of an obstruction is equal to or greater than a threshold. FIG. 7(b) shows a state in which two people are included, with one person partially obstructed by the other person. Region 702 is a region of the person as an obstructed object, and is, for example, a region corresponding to pixels where the value (likelihood) representing the likelihood of an obstructing object is equal to or greater than a threshold.

[0064] A possible technique for acquiring such an obstructing object region map or an obstructed object region map from a captured image is to apply "Olaf Ronneberger, Philipp Fischer, Thomas Brox, U-Net: Convolutional Networks for Biomedical Image Segmentation: MICCAI 2015." This technique can output the region of a specific object that appears in another image by learning images containing the object in advance. Therefore, by learning obstructing objects and obstructed objects using this technique, it is possible to output an obstructing object region map or an obstructed object region map such as that shown in FIG. 7. However, the output region may be other than the example shown in this figure, and any region that can identify the position of the obstructing object region or the obstructed object may be used.

[0065] In step S302, the occlusion determination unit 152 determines whether the tracking target is occluded by an obstruction or whether the obstruction has been removed in the captured image acquired by the acquisition unit 110 in step S201, based on the obstructed object area map or the obstructed object area map output by the detection unit 151. This determination method is divided into a case where the tracking target is the tracking target and a case where the tracking target is an obstructing object, and each case will be described using Fig. 8 and Fig. 9.

[0066] First, the processing of step S302 when the tracking subject is a tracking target will be described. First, the occlusion determination unit 152 sets an arbitrary partial region (for example, a region near the center position of the object candidate region) in the object candidate region of the tracking subject in the captured image as the occlusion determination region. Then, the occlusion determination unit 152 determines whether a corresponding region in the occlusion determination region map corresponding to the occlusion determination region includes an occlusion object region. Then, if the result of such determination shows that the corresponding region includes an occlusion object region, the occlusion determination unit 152 determines that occlusion has occurred, and if the corresponding region does not include an occlusion object region, the occlusion determination unit 152 may be configured to determine that occlusion has occurred if the proportion of the occlusion object region in the corresponding region is equal to or greater than a threshold, and to determine that occlusion has not occurred if the proportion is less than the threshold.

[0067] 8(a), a corresponding region 802 corresponding to the occlusion determination region is located in a corresponding region 801 corresponding to the object candidate region to be tracked in the occlusion object region map, and the corresponding region 802 includes an occlusion object region 803. Therefore, in such a case, the occlusion determination unit 152 determines that occlusion has occurred.

[0068] 8(b), a corresponding region 805 corresponding to the occlusion determination region is located in a corresponding region 804 corresponding to the object candidate region of the tracking target in the occlusion object region map, and the corresponding region 805 does not include the occlusion object region 806. Therefore, in such a case, the occlusion determination unit 152 determines that occlusion has not occurred.

[0069] Next, the processing of step S302 when the tracking target is an obstructing object will be described. First, in the same manner as described above, the occlusion determination unit 152 sets an arbitrary partial region (for example, a region near the center position of the object candidate region) in the object candidate region of the tracking target in the captured image as the occlusion determination region. Then, the occlusion determination unit 152 determines whether or not the following condition is satisfied: "A corresponding region in the obstructing object region map that corresponds to the occlusion determination region includes an obstructing object region, and, in the obstructed object region map, another object candidate region that overlaps with the object candidate region of the tracking target includes an occluded determination region." Then, if the result of such determination shows that the condition is satisfied, the occlusion determination unit 152 determines that the occlusion has been released, and if the condition is not satisfied, the occlusion determination unit 152 determines that the occlusion has not been released.

[0070] In the example of Fig. 9(a), a corresponding region 902 corresponding to the occlusion determination region is located in a corresponding region 901 corresponding to the object candidate region of the tracking target in the occlusion object region map. The following conditions are satisfied: the corresponding region 902 includes an occlusion object region 903, and another object candidate region 904 overlapping with the corresponding region 901 in the occlusion object region map includes an occlusion determination region 905, as shown in Fig. 9(b). Therefore, in such a case, the occlusion determination unit 152 determines that occlusion has been removed.

[0071] If there are multiple object candidate regions in the occluding object region map that overlap with a corresponding region corresponding to the object candidate region of the tracking target, it is sufficient to select one of the tracking candidate regions and determine whether the selected object candidate region includes an occluded determination region. As a method for selecting one object candidate region from multiple object candidate regions that overlap with the corresponding region, for example, a method of selecting an object candidate region that is the shortest distance from the corresponding region can be applied, but the selection method is not limited to a specific one.

[0072] If it is determined as a result of such determination that the tracking target is occluded by an obstruction or that the obstruction has been removed in the captured image acquired by the acquisition unit 110 in step S201, the process proceeds to step S303. On the other hand, if it is determined as a result of such determination that neither the obstruction nor the obstruction has been removed is present, the process proceeds to step S201, and the processes in the subsequent steps are performed for the next frame.

[0073] In step S303, the switching unit 160 switches (selects) the tracking target to be tracked by the tracking processing unit 140. Here, the operation of the switching unit 160 in step S303 differs depending on whether the occlusion detection unit 150 detects the occurrence of occlusion or the release of the occlusion.

[0074] First, for each object candidate region output by tracking processing unit 140, switching unit 160 sets an arbitrary partial region in the object candidate region (for example, a region near the center position of the object candidate region) as a switching determination region.

[0075] When the occlusion detection unit 150 detects the occurrence of occlusion, the switching unit 160 identifies the corresponding area in the shielding object area map that corresponds to the switching determination area and has the highest proportion of shielding object areas. The switching unit 160 then determines (selects) the object in the object candidate area that corresponds to the identified corresponding area as the tracking target. In the example of FIG. 10(a), of the corresponding areas 1002 and 1003 that correspond to the switching determination area in the shielding object area map, the corresponding area that has the highest proportion of shielding object areas 1004 is the corresponding area 1002. Therefore, the object in the object candidate area 1001 that corresponds to the corresponding area 1002 (the object that corresponds to the shielding object area 1004) is determined (selected) as the tracking target.

[0076] Furthermore, when the occlusion detection unit 150 detects that the occlusion has been removed, the switching unit 160 identifies the corresponding area in the masked object area map that corresponds to the switching determination area and has the highest proportion of masked object areas. Then, the switching unit 160 determines (selects) the object in the object candidate area that corresponds to the identified corresponding area as the tracking target. In the example of FIG. 10(b), of the corresponding areas 1006 and 1007 that correspond to the switching determination area in the masked object area map, the corresponding area that has the highest proportion of masked object areas 1008 is the corresponding area 1007. Therefore, the object in the object candidate area 1005 that corresponds to the corresponding area 1007 (the object that corresponds to the masked object area 1008) is determined (selected) as the tracking target.

[0077] As described above, according to this embodiment, the tracking target is switched using a method that utilizes the obstructing object area and the obstructed object area, so that the tracking target can be tracked more accurately, as in the first embodiment.

[0078] [Third embodiment] In this embodiment, even if the occlusion detection unit 150 detects the occurrence of occlusion, the tracking target is not immediately switched, but rather a decision is made as to whether or not to switch depending on the degree to which the tracking target is occluded.

[0079] An example of the functional configuration of the system according to this embodiment will be described with reference to the block diagram of Fig. 11. As shown in Fig. 11, the system according to this embodiment has a configuration in which a switching determination unit 180 is added to the image processing device 100 in Fig. 5.

[0080] Next, the processing performed by the image processing device 100 to track the tracking target in each captured image output from the imaging device 200 after the above setting processing will be described with reference to the flowchart in Fig. 12. Note that the same processing steps in Fig. 12 as those shown in Fig. 6 are assigned the same step numbers, and descriptions of these processing steps will be omitted.

[0081] If it is determined in step S302 that the tracking target is occluded by an obstruction or that the obstruction has been removed in the captured image acquired by the acquisition unit 110 in step S201, the process proceeds to step S401. On the other hand, if it is determined that no obstruction has occurred or that the obstruction has been removed, the process proceeds to step S201, and the processes in the subsequent steps are performed for the next frame.

[0082] In step S401, switching determination unit 180 determines whether or not to switch the tracking target. If the result of this determination is that the tracking target is to be switched, the process proceeds to step S303, and if the tracking target is not to be switched, the process proceeds to step S201, and the processes in the subsequent steps are performed for the next frame.

[0083] As an example of a method for determining whether to switch the tracking target, a method for determining whether to switch the tracking target when the object area of ​​the tracking target disappears is conceivable. In the following, the processing when the occurrence of occlusion is detected will be described, and the processing when the release of occlusion is detected will be the same as in the above embodiment.

[0084] For example, if the condition that "occlusion is detected by the occlusion detection unit 150 and there is no object candidate area that includes θ% or more of the area (number of pixels, etc.) of the occluded object area output by the detection unit 151" is satisfied, it is determined that the object area of ​​the tracking target has disappeared from the captured image.

[0085] The masked object region map in Fig. 13 is a masked object region map in a state where masking is detected by masking detection unit 150. Here, θ = 30%. In this case, in the masked object region map in Fig. 13(a), corresponding region 1302 corresponding to the object candidate region of the masked object includes 30% or more of masked object region 1301 of the masked object, so it is determined that the object region of the tracking target has not disappeared. On the other hand, in the masked object region map in Fig. 13(b), there is no "corresponding region corresponding to the object candidate region of the masked object" that includes 30% or more of masked object region 1303 of the masked object, so it is determined that the object region of the tracking target has disappeared.

[0086] As a method for determining whether to switch the tracking target, for example, when "removal of the blocking" is not detected even once within a predetermined period after detecting that the blocking has occurred, it may be determined that the tracking target is to be switched to the blocking object. Also, for example, when "removal of the blocking" is not detected even once within a predetermined period after detecting that the blocking has been removed, it may be determined that the tracking target is to be switched to the tracking target.

[0087] In this embodiment, during the period from when occlusion detection unit 150 detects the occurrence of occlusion until switching determination unit 180 determines to switch the tracking target, tracking processing unit 140 generates tracking candidate information for each captured image during that period and stores it in information storage unit 170. The tracking candidate information includes information that specifies the position and size of the object candidate region detected by tracking processing unit 140 from the captured image, an image within the object candidate region, and a flag that indicates whether the object candidate region is an object candidate region of an obstructing object. The flag takes two values, TRUE and FALSE, where TRUE indicates that "the object candidate region is an object candidate region of an obstructing object," and FALSE indicates that "the object candidate region is not an object candidate region of an obstructing object."

[0088] In step S303, the switching unit 160 selects one piece of tracking candidate information whose flag is TRUE from among the pieces of tracking candidate information stored in the information storage unit 170, and selects an object in the object candidate area corresponding to the selected piece of tracking candidate information as a tracking target. At this time, instead of selecting one piece of tracking candidate information, a combination of multiple pieces of tracking candidate information may be used.

[0089] Then, in step S205, the switching unit 160 instructs the setting unit 130 to set "the tracking model for tracking the tracking target selected in step S303" as the tracking model used by the tracking processing unit 140 to track the tracking target. At this time, if the tracking model is an online learning model that performs online learning, the online learning model may be learned using the stored tracking candidate information.

[0090] In this way, in this embodiment, a two-stage occlusion determination is performed by the occlusion detection unit 150 and the switching determination unit 180. As a result, in the case of slight occlusion where the tracking target is not completely hidden, unnecessary calculation processing can be omitted by omitting switching of the tracking target.

[0091] Furthermore, if the tracking model is an online learning model that performs online learning, the tracking accuracy after switching the tracking target can be improved by performing learning using the tracking candidate information stored in the information storage unit 170 when setting the tracking processing unit 140.

[0092] In the above embodiment, the functional units of the image processing device 100 shown in FIGS. 2, 5, and 11 are implemented by software (computer programs), but one or more of these functional units may be implemented by hardware.

[0093] In the above embodiment, the imaging device 200 and the image processing device 100 have been described as separate devices. However, the imaging device 200 and the image processing device 100 may be integrated to form a single device having the functions of the imaging device 200 and the image processing device 100. Furthermore, the information storage unit 170 may also be integrated with the image processing device 100, rather than being an external device of the image processing device 100.

[0094] Furthermore, in the above embodiment, the tracking process for the captured image acquired from the captured image has been described, but other processes may be executed in addition to the tracking process as a process related to the captured image. For example, the CPU 101 may display, on the display unit 105, information for notifying the user of the object area of ​​the tracking target in the captured image, in addition to the captured image acquired from the imaging device 200. The "information for notifying the user of the object area of ​​the tracking target" includes one or more pieces of information such as the frame of the object area, the recognition result of the tracking target within the object area (type (person, dog, car, head, arm, leg, etc.), age, gender, etc.), etc.).

[0095] As described above, the system described in the above embodiment may be implemented by one device, or may be configured by any number of devices, two or more. In such cases, the device that executes each function of the system described in the above embodiment may be appropriately modified, and any form may be used.

[0096] Furthermore, the numerical values, processing timing, processing order, processing subject, destination / source / storage location of data (information) and the like used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to such examples.

[0097] Furthermore, some or all of the above-described embodiments may be used in appropriate combination, and some or all of the above-described embodiments and modified examples may be selectively used.

[0098] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0099] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0100] 100: Image processing device 110: Acquisition unit 120: Setting unit 130: Setting unit 140: Processing unit 150: Occlusion detection unit 160: Switching unit 170: Information storage unit 200: Imaging device

Claims

1. a tracking means for performing a tracking process for tracking a tracking target in a captured image using a tracking model; a switching means for switching the tracking model to a first model that tracks the second object as the tracking target when it is detected that the first object is occluded by a second object while the tracking means is tracking the first object as the tracking target, and for switching the tracking model to a second model that tracks the first object as the tracking target when it is detected that the occlusion has been removed; Equipped with The switching means 10. An image processing device comprising: an image processing device that detects occlusion of a first object by a second object based on a first map representing an area of ​​the first object in the captured image.

2. A tracking means for performing a tracking process for tracking a tracking target in a captured image using a tracking model; a switching means for switching the tracking model to a first model that tracks the second object as the tracking target when it is detected that the first object is occluded by a second object while the tracking means is tracking the first object as the tracking target, and for switching the tracking model to a second model that tracks the first object as the tracking target when it is detected that the occlusion has been removed; Equipped with The image processing device is characterized in that, when the occlusion is detected, the switching means determines whether to perform the switching based on a candidate area of ​​the first object in the captured image and a first map representing the area of ​​the first object in the captured image.

3. The switching means When the tracking means detects that a first object is occluded by a second object while tracking the first object as a tracking target, the tracking means switches the tracking model to a first model constructed using an image of the second object, and when the tracking means detects that the occlusion has been removed, the tracking model switches to a second model constructed using an image of the first object.

3. The image processing device according to claim 1, wherein the image processing device is a computer.

4. The switching means 4. The image processing device according to claim 2, wherein the occlusion of the first object by the second object is detected based on a first map representing an area of ​​the first object in the captured image.

5. The switching means 5. The image processing device according to claim 4, characterized in that, when the occlusion is detected, the second object is determined as the tracking target based on a candidate area of ​​the first object in the captured image and a second map representing the area of ​​the second object in the captured image, and the tracking model is switched to a first model that tracks the determined second object as the tracking target.

6. The switching means 6. The image processing device according to claim 1, wherein the removal of the occlusion is detected based on a second map representing an area of ​​a second object in the captured image.

7. The switching means 7. The image processing device according to claim 6, characterized in that, when the removal of the occlusion is detected, the first object is determined as the tracking target based on a candidate area of ​​the first object in the captured image and a first map representing the area of ​​the first object in the captured image, and the tracking model is switched to a second model that tracks the determined first object as the tracking target.

8. The image processing device according to claim 1, characterized in that, when the occlusion is detected, the switching means determines whether to perform the switching based on a candidate area of ​​the first object in the captured image and a first map representing the area of ​​the first object in the captured image.

9. 9. The image processing apparatus according to claim 1, wherein each of the first model and the second model includes a model that performs a neural network or template matching.

10. 10. The image processing apparatus according to claim 1, further comprising an image capturing unit for capturing the captured image.

11. An image processing method performed by an image processing device, a tracking step in which a tracking means of the image processing device performs a tracking process for tracking a tracking target in a captured image using a tracking model; a switching step in which, when it is detected that a first object is occluded by a second object while the first object is being tracked as a tracking target in the tracking step, a switching means of the image processing device switches the tracking model to a first model that tracks the second object as a tracking target, and when it is detected that the occlusion is released, switches the tracking model to a second model that tracks the first object as a tracking target; Equipped with In the switching step, An image processing method comprising: detecting occlusion of a first object by a second object based on a first map representing an area of ​​the first object in the captured image.

12. An image processing method performed by an image processing device, comprising: a tracking step in which a tracking means of the image processing device performs a tracking process for tracking a tracking target in a captured image using a tracking model; a switching step in which, when it is detected that a first object is occluded by a second object while the first object is being tracked as a tracking target in the tracking step, a switching means of the image processing device switches the tracking model to a first model that tracks the second object as a tracking target, and when it is detected that the occlusion is released, switches the tracking model to a second model that tracks the first object as a tracking target; Equipped with In the switching process, when the occlusion is detected, it is determined whether or not to perform the switching based on a candidate area of ​​the first object in the captured image and a first map representing the area of ​​the first object in the captured image.

13. A computer program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 9.

Images