Target tracking method, target tracking device, electronic device and storage medium
By determining the unoccluded area and performing local tracking during target tracking, the problem of target loss in long-term target tracking is solved and the stability and accuracy of tracking are improved.
Patent Information
- Application Number
- CN202011460436.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-11
- Publication Date
- 2025-01-21
- Estimated Expiration
- 2040-12-11
AI Technical Summary
The problem of missing target tracking is prone to occur during long-term target tracking.
By determining whether the target to be tracked is occluded in the previous frame image, if it is occluded, determine the unoccluded area, and based on the tracking information in the previous frame image, perform partial tracking on the unoccluded area to obtain the target Local tracking information in the next frame of image.
It reduces the loss of the target when the target to be tracked is partially occluded, and improves the stability and accuracy of target tracking.
Smart Images
Figure CN112489086B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a target tracking method, a target tracking device, an electronic device and a storage medium. Background Art
[0002] The target tracking problem is an important research direction in the field of computer vision. The target tracking problem can be divided into single target tracking and multi-target tracking according to the number of targets to be tracked. According to the complexity of the tracking process, it can be divided into short-term target tracking and long-term target tracking. At present, the research on target tracking has made many progress and breakthroughs, but there are still many problems to be solved. For example, in the process of long-term target tracking, it is easy to lose target tracking. Summary of the invention
[0003] In view of this, the embodiments of the present disclosure are intended to provide a target tracking method, a target tracking device, an electronic device, and a storage medium.
[0004] The technical solution of the present disclosure is achieved as follows:
[0005] In one aspect, the present disclosure provides a target tracking method.
[0006] The target tracking method provided by the embodiment of the present disclosure includes:
[0007] Determine whether the target to be tracked is blocked in the n-1th frame image, where n is a positive integer greater than 1;
[0008] If a local area of the target to be tracked is blocked in the n-1th frame image, determine an unblocked area of the target to be tracked in the n-1th frame image;
[0009] Based on the tracking information of the target to be tracked in the n-1th frame image, the unobstructed area is locally tracked in the nth frame image to obtain the local tracking information of the target to be tracked in the nth frame image, wherein the n-1th frame image is a frame image previous to the nth frame image.
[0010] In some embodiments, determining whether the target to be tracked is blocked in the n-1th frame image includes:
[0011] Determining the perspective relationship between different targets in the n-1th frame image; wherein the targets at least include the target to be tracked;
[0012] Determining a foreground target of the target to be tracked according to the perspective relationship;
[0013] Determine whether there is an interference object in the foreground target that blocks the target to be tracked.
[0014] In some embodiments, determining whether there is an interference object in the foreground target that blocks the target to be tracked includes:
[0015] Determining whether the position of the foreground target overlaps with the position of the target to be tracked;
[0016] If so, it is determined that the target to be tracked is blocked by the interference object in the n-1th frame image.
[0017] In some embodiments, determining the perspective relationship between different targets in the n-1th frame image, and determining the foreground target of the target to be tracked according to the perspective relationship, includes:
[0018] Determining the bounding boxes of the different targets;
[0019] Determine a first distance between a lower edge of a bounding box of the target to be tracked and a lower edge of the n-1th frame image, and determine a second distance between a lower edge of a bounding box of a target other than the target to be tracked and a lower edge of the n-1th frame image;
[0020] If the first distance is greater than the second distance, the target corresponding to the second distance is determined to be the foreground target of the target to be tracked.
[0021] In some embodiments, determining the unobstructed area of the target to be tracked in the n-1th frame image includes:
[0022] Obtaining the intersection between the outer bounding box of the interference object and the outer bounding box of the target to be tracked, and obtaining the area blocked by the target to be tracked;
[0023] The blocked area is removed from the target to be tracked to obtain the unblocked area of the target to be tracked in the (n-1)th frame image.
[0024] In some embodiments, the method further comprises:
[0025] Determining a relative position relationship of the unobstructed area in the target to be tracked according to the position information of the unobstructed area and the position information of the bounding box of the target to be tracked;
[0026] According to the relative position relationship, the local tracking information of the n-th frame image is converted into the global tracking information of the target to be tracked in the n-th frame image.
[0027] In some embodiments, the method further comprises:
[0028] If it is detected that the target to be tracked in the m-th frame image is not blocked, global tracking of the target to be tracked is resumed from the m-th frame image, wherein m is a positive integer greater than or equal to n.
[0029] On the other hand, the present disclosure provides a target tracking device. The target tracking device provided by the embodiment of the present disclosure includes:
[0030] A first processing unit, used to determine whether the target to be tracked is blocked in the n-1th frame image, where n is a positive integer greater than 1;
[0031] A second processing unit is used to determine an unobstructed area of the target to be tracked in the n-1th frame image if a local area of the target to be tracked is obstructed in the n-1th frame image;
[0032] The third processing unit is used to perform local tracking of the unobstructed area in the nth frame image based on the tracking information of the target to be tracked in the n-1th frame image, so as to obtain the local tracking information of the target to be tracked in the nth frame image, wherein the n-1th frame image is a frame image before the nth frame image.
[0033] In some embodiments, the first processing unit is used to determine whether the target to be tracked is blocked in the n-1th frame image, including:
[0034] The first processing unit is specifically used to determine the perspective relationship between different targets in the n-1th frame image; wherein the targets at least include the target to be tracked;
[0035] Determining a foreground target of the target to be tracked according to the perspective relationship;
[0036] Determine whether there is an interference object in the foreground target that blocks the target to be tracked.
[0037] In some embodiments, the first processing unit is used to determine whether there is an interference object in the foreground target that blocks the target to be tracked, including:
[0038] The first processing unit is specifically configured to determine whether the position of the foreground target overlaps with the position of the target to be tracked;
[0039] If so, it is determined that the target to be tracked is blocked by the interference object in the n-1th frame image.
[0040] In some embodiments, the first processing unit is used to determine the perspective relationship between different targets in the n-1th frame image, and the foreground target of the target to be tracked is determined according to the perspective relationship, including:
[0041] The first processing unit is specifically configured to determine the external frames of the different targets;
[0042] Determining a first distance between a lower edge of a bounding box of the target to be tracked and a lower edge of the n-1th frame image, and determining a second distance between a lower edge of a bounding box of a target other than the target to be tracked and a lower edge of the n-1th frame image; and
[0043] If the first distance is greater than the second distance, the target corresponding to the second distance is determined to be the foreground target of the target to be tracked.
[0044] In some embodiments, the second processing unit is used to determine the unobstructed area of the target to be tracked in the n-1th frame image, including:
[0045] The second processing unit is specifically configured to obtain an intersection between an outer bounding box of the interference object and an outer bounding box of the target to be tracked, so as to obtain an area blocked by the target to be tracked;
[0046] The blocked area is removed from the target to be tracked to obtain the unblocked area of the target to be tracked in the (n-1)th frame image.
[0047] In some embodiments, a fourth processing unit is further included, which is used to determine the relative position relationship of the unobstructed area in the target to be tracked according to the position information of the unobstructed area and the position information of the external frame of the target to be tracked;
[0048] According to the relative position relationship, the local tracking information of the n-th frame image is converted into the global tracking information of the target to be tracked in the n-th frame image.
[0049] In some embodiments, the third processing unit is further used to resume global tracking of the target to be tracked from the mth frame image if it is detected that the target to be tracked in the mth frame image is not obstructed, wherein m is a positive integer greater than or equal to n.
[0050] On the other hand, the present disclosure also provides an electronic device.
[0051] An electronic device provided by an embodiment of the present disclosure includes: a processor and a memory for storing a computer program that can be run on the processor, wherein the processor, when running the computer program, executes the steps of the target tracking method provided by an embodiment of the present disclosure on one hand.
[0052] In yet another aspect, the present disclosure also provides a computer-readable storage medium.
[0053] The computer-readable storage medium provided by the embodiment of the present disclosure stores a computer program thereon, and when the computer program is executed by a processor, the steps of the target tracking method provided by the embodiment of the present disclosure are implemented on the one hand.
[0054] The disclosed embodiment determines the unobstructed area of the target to be tracked as the tracked object, and locally tracks the unobstructed area through the tracking information of the target tracking in the previous frame image (for example, the n-1 frame image), and obtains the local tracking information of the target in the next frame image (for example, the nth frame image). In this way, when the target to be tracked is partially obstructed, the tracking information of the target is obtained by locally tracking the unobstructed area. In the case where the target to be tracked is partially obstructed, if the target to be tracked continues to be tracked as a whole, the target to be tracked cannot be identified as a whole due to the obstruction of the local area, so that the tracking result is judged as the absence of the target in the next frame image, thereby causing the target to be lost. However, in the disclosed embodiment, by locally tracking the unobstructed local area, the situation of target loss when the target to be tracked is partially obstructed can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Figure 1 It is a schematic diagram of the process of the target to be tracked being lost;
[0056] Figure 2 It is a schematic diagram of possible situations where the target to be tracked is blocked;
[0057] Figure 3 is a flow chart of a target tracking method according to an exemplary embodiment;
[0058] Figure 4 is a schematic diagram of occlusion relationship judgment according to an exemplary embodiment;
[0059] Figure 5 is a schematic diagram of target tracking learning according to an exemplary embodiment;
[0060] Figure 6 is an overall flow chart of a target tracking method according to an exemplary embodiment;
[0061] Figure 7 is a schematic diagram of the structure of a target tracking device according to an exemplary embodiment;
[0062] Figure 8 The figure is a schematic diagram showing the structure of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION
[0063] The technical solution of the present invention is further described in detail below in conjunction with the drawings and specific embodiments of the specification. The exemplary embodiments will be described in detail here, and examples thereof are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are only examples of devices and methods consistent with some aspects of the present invention as detailed in the attached claims.
[0064] In the field of computer vision, when tracking a target, the target may be blocked, resulting in loss of the target. Figure 1 This is a schematic diagram of the process of the target being tracked being lost. Figure 1 As shown, it includes an initial state, an occlusion state, and a tracking failure state. In the initial state, the tracking device can track the target to be tracked, and the target to be tracked is not occluded. In the occlusion state, the tracking device finds that the target to be tracked is occluded, and then the target to be tracked is lost, and the target frame (external frame) of the target to be tracked remains on the foreground target, resulting in tracking failure. Figure 2 It is a schematic diagram of the possible situation where the target to be tracked is blocked. Figure 2 As shown, the target to be tracked may be blocked by a human body, a fixed obstacle, or a vehicle.
[0065] The present disclosure provides a target tracking method. Figure 3 FIG. 1 is a flow chart of a target tracking method according to an exemplary embodiment. Figure 3 As shown, the target tracking method includes:
[0066] Step 30: determine whether the target to be tracked is blocked in the n-1th frame image, where n is a positive integer greater than 1;
[0067] Step 31: If a local area of the target to be tracked is blocked in the n-1th frame image, determine an unblocked area of the target to be tracked in the n-1th frame image;
[0068] Step 32: Based on the tracking information of the target to be tracked in the n-1th frame image, locally track the unobstructed area in the nth frame image to obtain the local tracking information of the target to be tracked in the nth frame image, wherein the n-1th frame image is the previous frame image of the nth frame image.
[0069] In this exemplary embodiment, the target to be tracked may be a movable target such as a human body, an animal or a vehicle. The specific target to be tracked is not limited to any one type. Among them, determining whether the target to be tracked is blocked in the n-1th frame image includes: determining whether the imaging area of the target to be tracked in the n-1th frame image is blocked by the imaging of other targets. Imaging refers to image information that reflects the morphological features, outer contours, etc. of the target to be tracked, which embodies individual features for identification. The nth frame image is the frame image after the n-1th frame image. The nth frame image and the n-1th frame image may be two adjacent images in the video.
[0070] The nth frame image and the n-1th frame image may be from the same video, and the video may be from a road monitoring device and / or a security monitoring device. Of course, this is only an example of the device from which the video comes, and the specific implementation is not limited to these example road monitoring devices and / or security monitoring devices. The security monitoring equipment includes but is not limited to: surveillance cameras in residential areas, surveillance cameras in office buildings, surveillance equipment in shopping malls, and / or surveillance equipment in non-road public places such as parks.
[0071] In this exemplary embodiment, global tracking means that the tracking device identifies and locates the target based on the overall features of the target (i.e., the target to be tracked is regarded as a whole). For example, when the target to be tracked is a person, all the features of the person (including the head, limbs, torso, etc.) are regarded as a whole for identification and location. Local tracking means that the tracking device identifies and locates the target based on some features of the target to be tracked. For example, when the target to be tracked is a person, only the head of the person can be identified and located to determine the position of the target.
[0072] In this exemplary embodiment, the local tracking information may be the position information (including coordinate information) of the tracked local features in the image obtained based on the tracking of the target local features and / or the area information of the local features (including the morphological contour, area, etc. constituted by the local features).
[0073] The global tracking information may be the position information (including coordinate information) of the entire target in the image and / or the area information of the entire target (including the morphological contour and area formed by the overall features) obtained based on the overall feature tracking of the target.
[0074] In this exemplary embodiment, the unobstructed area used for tracking the target in the n-1th frame image is not obstructed by any obstacle in the nth frame image; if the unobstructed area used for tracking the target in the n-1th frame image is obstructed by any obstacle in the nth frame image, an unobstructed area can be replaced to continue tracking the target.
[0075] The disclosed embodiment determines the unobstructed area of the target to be tracked as the tracked object, and locally tracks the unobstructed area through the tracking information of the target tracking in the previous frame image (for example, the n-1 frame image), so as to obtain the local tracking information of the target in the next frame image (for example, the nth frame image). In this way, when the target is partially obstructed, the tracking information of the target is obtained by locally tracking the unobstructed area. In the case where the above-mentioned target is partially obstructed, if the target continues to be tracked as a whole, the target cannot be identified as a whole due to the obstruction of the local area, so that the tracking result is judged as the absence of the target in the next frame image, thereby causing the target to be lost. However, in the disclosed embodiment, by locally tracking the unobstructed local area, the situation of target loss when the target is partially obstructed can be reduced.
[0076] In one embodiment, the method further comprises:
[0077] It is determined whether the unobstructed area meets a preset condition, and if the unobstructed area meets the preset condition, local tracking is performed based on the unobstructed area.
[0078] In this exemplary embodiment, the preset condition may be having obvious features or specific features, such as the unobstructed local area meeting preset color requirements, preset contour requirements, or meeting preset limb feature tracking requirements (e.g., arms, head, etc.).
[0079] In one embodiment, the method further comprises:
[0080] If the unobstructed area does not meet the preset conditions, the n+kth frame image is checked according to the image features of the target to be tracked, the position of the target to be tracked in the n+kth frame image is located, and the tracking result of the target to be tracked in the n+kth frame image is obtained, where k is 0 or a positive integer.
[0081] In this exemplary embodiment, when the unobstructed area does not meet the preset conditions, the n+k frame image is checked according to the image features of the target to be tracked, the position of the target to be tracked in the n+k frame image is located, and the tracking result of the target to be tracked in the n+k frame image is obtained. This allows for timely identification and tracking in subsequent images when local tracking cannot be performed in the current image for a while, so as to reduce the loss of the target.
[0082] The unobstructed area meets a preset condition, including:
[0083] The area ratio of the unobstructed area to the entire target reaches a preset threshold;
[0084] The unblocked area is a preset local area within the target, and the preset local area includes but is not limited to: the head of the target.
[0085] In this exemplary embodiment, through the judgment and screening of the preset threshold, the local area with a higher tracking success rate can be used as much as possible for tracking, thereby improving the target tracking success rate.
[0086] In some embodiments, determining whether the target to be tracked is blocked in the n-1th frame image includes:
[0087] Determining the perspective relationship between different targets in the n-1th frame image; wherein the targets at least include the target to be tracked;
[0088] Determining a foreground target of the target to be tracked according to the perspective relationship;
[0089] Determine whether there is an interference object in the foreground target that blocks the target to be tracked.
[0090] In this exemplary embodiment, each image of the target tracking can be obtained by means of camera, radar detection, etc. By analyzing the perspective relationship of each target in the image, it is confirmed whether the target to be tracked is blocked by interference. Interference is a target other than the target to be tracked in the image, including: people other than the tracked person, moving vehicles, fixed devices (including fences, etc.), etc.
[0091] In this exemplary embodiment, the perspective relationship refers to the spatial position, outline and projection of each target in the field of view, including relative relationships such as near is big and far is small, near is high and far is low, near is sparse and far is dense, near is wide and far is narrow, etc. In this exemplary embodiment, the spatial position, outline and other information of each target in the field of view can be captured by a tracking device (e.g., a camera device, etc.) by taking a video or photographing.
[0092] In some embodiments, determining whether there is an interference object in the foreground target that blocks the target to be tracked includes:
[0093] Determining whether the position of the foreground target overlaps with the position of the target to be tracked;
[0094] If so, it is determined that the target to be tracked is blocked by the interference object in the n-1th frame image.
[0095] In this exemplary embodiment, the foreground target refers to a target that is closer to the tracking device than the target to be tracked, based on the tracking device. The foreground target is one or more of the interference objects. That is, any interference object may become a foreground target.
[0096] In this exemplary embodiment, when the position of the foreground target overlaps with the position of the target to be tracked, the interference at the foreground target position blocks the field of view of the tracking device for obtaining information of the target to be tracked. At this time, the target to be tracked is blocked by the interference in the n-1th frame image.
[0097] In some embodiments, determining the perspective relationship between different targets in the n-1th frame image, and determining the foreground target of the target to be tracked according to the perspective relationship, includes:
[0098] Determining the bounding boxes of the different targets;
[0099] Determine a first distance between a lower edge of a bounding box of the target to be tracked and a lower edge of the n-1th frame image, and determine a second distance between a lower edge of a bounding box of a target other than the target to be tracked and a lower edge of the n-1th frame image;
[0100] If the first distance is greater than the second distance, the target corresponding to the second distance is determined to be the foreground target of the target to be tracked.
[0101] In this exemplary embodiment, Figure 4 FIG. 1 is a schematic diagram of determining an occlusion relationship according to an exemplary embodiment. Figure 4 As shown, in the acquired frame image, when the first distance is greater than the second distance, it means that the distance between the interference object and the tracking device is less than the distance between the target to be tracked (the obscured target in the figure) and the tracking device. At this time, the interference object is the foreground target of the target to be tracked. At the same time, if the position of the foreground target overlaps with the position of the target to be tracked, the interference object obscures the target to be tracked. In this exemplary embodiment, the circumscribed frame is a feature capture frame circled according to the tracking feature (global feature or local feature) when the tracking device captures the target. The circumscribed frame can be any shape such as a rectangle or a circle.
[0102] In some embodiments, determining the unobstructed area of the target to be tracked in the n-1th frame image includes:
[0103] Obtaining the intersection between the outer bounding box of the interference object and the outer bounding box of the target to be tracked, and obtaining the area blocked by the target to be tracked;
[0104] The blocked area is removed from the target to be tracked to obtain the unblocked area of the target to be tracked in the (n-1)th frame image.
[0105] In this exemplary embodiment, when tracking a target, an external frame for tracking can be formed to enclose the target features. The external frame can be a rectangle, a circle, or other set shapes. The external frame can enclose the entire tracking target and the overall features of each interference object as needed. At the same time, the external frame can also enclose the local features of the tracking target or interference object.
[0106] In some embodiments, the method further comprises:
[0107] Determining a relative position relationship of the unobstructed area in the target to be tracked according to the position information of the unobstructed area and the position information of the bounding box of the target to be tracked;
[0108] According to the relative position relationship, the local tracking information of the n-th frame image is converted into the global tracking information of the target to be tracked in the n-th frame image.
[0109] In this exemplary embodiment, after obtaining the local tracking information of the target to be tracked in the nth frame image, the global tracking information of the target to be tracked in the nth frame image is determined according to the relative position relationship of the unobstructed local part in the target to be tracked in the n-1th frame image. The relative position relationship may include relative position coordinates.
[0110] For example, the local tracking information of the target to be tracked in the nth frame image includes the position coordinates of the local features of the target to be tracked in the nth frame image (for example, the position coordinates of the upper left corner of the external frame enclosing the local features in the nth frame image: X' 局 , Y' 局 ; where X' 局 is the coordinate of the upper left corner of the bounding box that encloses the local feature in the X-axis direction in the nth frame image, where Y' 局 is the Y-axis coordinate of the upper left corner of the bounding box that encloses the local feature in the nth frame image);
[0111] Determine the position coordinates of the upper left corner of the external frame that encloses the overall features of the target to be tracked in the n-1th frame image: X 整 , Y 整 ; Among them, X 整 is the X-axis coordinate of the upper left corner of the external frame that encloses the overall features of the target to be tracked in the n-1th frame image, and Y 整 The Y-axis coordinate of the upper left corner of the external frame that encloses the overall features of the target to be tracked in the n-1th frame image;
[0112] Determine the position coordinates of the upper left corner of the bounding box that encloses the local features in the n-1th frame image: X 局 , Y 局 ; Among them, X 局is the X-axis coordinate of the upper left corner of the bounding box that delimits the local area feature in the n-1th frame image, Y 局 The Y-axis coordinate of the upper left corner of the bounding box that encloses the local area feature in the n-1th frame image;
[0113] Determine the relative position relationship of the unobstructed local area in the target in the n-1th frame image: X 差 =X 整 -X 局 , Y 差 =Y 整 -Y 局 ; Among them, X 差 Y is the coordinate difference in the X-axis direction between the upper left corner of the bounding box that encloses the overall features of the target to be tracked and the upper left corner of the bounding box that encloses the local area features in the n-1th frame image. 差 is the coordinate difference in the Y-axis direction between the upper left corner of the external frame that encloses the overall features of the target to be tracked and the upper left corner of the external frame that encloses the local area features in the n-1th frame image;
[0114] Determine the global tracking information X' of the target to be tracked in the nth frame image 整 =X' 局 +X 差 , Y' 整 =Y' 局 +Y 差 ; where X' 整 Y' is the coordinate of the upper left corner of the external frame that encloses the overall features of the target to be tracked in the X-axis direction in the nth frame image. 整 The Y-axis coordinate of the upper left corner of the external frame that encloses the overall features of the target to be tracked in the nth frame image.
[0115] Determine the width W of the bounding box that encloses the overall features of the target to be tracked in the n-1th frame image 整 and high H 整 At the same time, the width W' and height H' of the external frame encircling the overall features of the target to be tracked in the nth frame image are the same as the width W and height H in the n-1th frame image.
[0116] In this exemplary embodiment, radar, binocular vision and other methods can also be used to determine the occlusion relationship between targets or between targets and the environment. Combined with background modeling, optical flow and other algorithms, the unoccluded target area can be determined more accurately. The tracking algorithm is changed to use other feature models, deep learning, etc.
[0117] In some embodiments,
[0118] The method further includes: if it is detected that the target to be tracked in the m-th frame image is not blocked, resuming global tracking of the target to be tracked from the m-th frame image, wherein m is a positive integer greater than or equal to n.
[0119] In this exemplary embodiment, when the local tracking of the target to be tracked is completed and the target is not blocked, it can be converted to global tracking.
[0120] In this exemplary embodiment, when the unobstructed local area of the target to be tracked is obstructed, the tracking can be switched to other unobstructed local areas. When all unobstructed local areas are obstructed, the tracking is switched to the global tracking of the target.
[0121] Figure 5 FIG. 1 is a schematic diagram of target tracking learning according to an exemplary embodiment. Figure 5 As shown in the figure, the tracking learning area weights for tracking the target include the whole body and the upper body. The original tracking is based on the overall characteristics of the tracking learning target. The core learning area is at the center of the target, that is, the feature learning weight obtained at the center is the highest, and the learning weight at the edge is the weakest. When the tracking learning area weight is the whole body of the target, the tracking core area is blocked and the tracking target is easily lost. When the tracking learning area weight is the upper body of the target, since the probability of the lower body of the tracking target being blocked is generally high, the tracking core area is not easy to be blocked, and the tracking target will not be easily lost. If the whole body is tracked, the core area of the learning weight is in the waist and abdomen. The lower body is easily blocked due to the alternating movement of the legs and the occlusion of low targets, and the large changes are not conducive to the tracking situation. Therefore, using the upper body for tracking and moving the core area of the learning weight to the chest part makes it easier to track stably. That is, it can improve the rapid changes of the legs that are not conducive to tracking, and can also solve the occlusion of some common targets, such as fences, non-motor vehicles, low cars, etc.
[0122] Figure 6 FIG. 1 is an overall flow chart of a target tracking method according to an exemplary embodiment. Figure 6 As shown, the target tracking method includes:
[0123] Video frame analysis: Get the n-1th frame image (detection frame image) and the nth frame image (tracking frame image).
[0124] Detection frame image analysis: In the n-1th frame image, obtain the position of the bounding box of the target to be tracked (for example, the coordinates of the upper left corner of the bounding box), and determine the occlusion relationship; select a local area for local tracking (obtain the coordinates of the upper left corner of the bounding box of the local area); the target tracking model learns to track the local area.
[0125] Tracking frame image analysis: Track the new position of the local area according to the local area model; obtain the global position according to the new position of the local area and the position and scaling relationship between the local area and the global area in the detection frame image.
[0126] If the frame image processing is completed, the task ends, otherwise the video frame parsing continues.
[0127] Among them, the models used in the above method can be various conventional models commonly used or already available in the field. For example, the target tracking model and the local area model can respectively be various target tracking models and local area models existing in the field. The update of related models is also an existing update method, which will not be described in detail here.
[0128] The present invention uses the image perspective relationship to determine the occlusion relationship of the target and confirms that the target to be tracked is occluded. The local area of the target to be tracked that is not occluded is selected for tracking, and then the motion tracking result of the overall target is converted according to the position ratio relationship between the local area and the overall target. It ensures that the tracking result of the target to be tracked is not affected by partial occlusion.
[0129] In another aspect, the present disclosure provides a target tracking device. Figure 7 FIG. 1 is a schematic diagram showing the structure of a target tracking device according to an exemplary embodiment. Figure 7 As shown, the target tracking device provided by the embodiment of the present disclosure includes:
[0130] A first processing unit 71 is used to determine whether the target to be tracked is blocked in the n-1th frame image, where n is a positive integer greater than 1;
[0131] A second processing unit 72 is configured to determine an unobstructed area of the target to be tracked in the n-1th frame image if a local area of the target to be tracked is obstructed in the n-1th frame image;
[0132] The third processing unit 73 is used to perform local tracking of the unobstructed area in the nth frame image based on the tracking information of the target to be tracked in the n-1th frame image, so as to obtain the local tracking information of the target to be tracked in the nth frame image, wherein the n-1th frame image is the previous frame image of the nth frame image.
[0133] In this exemplary embodiment, the target to be tracked may be a movable target such as a human body, an animal, or a vehicle. The specific target to be tracked is not limited to any one.
[0134] The nth frame image is a frame image subsequent to the n-1th frame image. The nth frame image and the n-1th frame image may be two adjacent images in the video.
[0135] The nth frame image and the n-1th frame image may be from the same video, and the video may be from a road monitoring device and / or a security monitoring device, etc. In this exemplary embodiment, only examples of the devices from which the video comes are given here, and the specific implementation is not limited to these example road monitoring devices and / or security monitoring devices. The security monitoring device includes but is not limited to: surveillance cameras in residential areas, surveillance cameras in office buildings, surveillance equipment in shopping malls, and / or surveillance equipment in non-road public places such as parks.
[0136] In this exemplary embodiment, global tracking means that the tracking device identifies and locates the target based on the overall features of the target (i.e., the target to be tracked is regarded as a whole). For example, when the target to be tracked is a person, all the features of the person (including the head, limbs, torso, etc.) are regarded as a whole for identification and location. Local tracking means that the tracking device identifies and locates the target based on partial features of the target. For example, when the target to be tracked is a person, only the head of the person can be identified and located to determine the position of the target.
[0137] In this exemplary embodiment, the local tracking information may be the position information (including coordinate information) of the tracked local features in the image obtained based on the tracking of the target local features and / or the area information of the local features (including the morphological contour, area, etc. constituted by the local features).
[0138] The global tracking information may be the position information (including coordinate information) of the entire target in the image and / or the area information of the entire target (including the morphological contour and area formed by the overall features) obtained based on the overall feature tracking of the target.
[0139] The disclosed embodiment determines the unobstructed part of the target to be tracked as the tracked object, and locally tracks the unobstructed area through the tracking information of the target tracking in the previous frame image (for example, the n-1 frame image), and obtains the local tracking information of the target to be tracked in the next frame image (for example, the nth frame image). In this way, when the target is partially obstructed, the tracking information of the target is obtained by locally tracking the unobstructed area. In the case where the above-mentioned target is partially obstructed, if the target is continued to be tracked as a whole, it will be impossible to identify the target as a whole due to the partial occlusion, so that the tracking result is judged as the absence of the target in the next frame image, and the target is lost. However, in the disclosed embodiment, by locally tracking the unobstructed area, the situation of target loss when the target is partially occluded can be reduced.
[0140] In some embodiments, the first processing unit is used to determine whether the target to be tracked is blocked in the n-1th frame image, including:
[0141] The first processing unit is specifically used to determine the perspective relationship between different targets in the n-1th frame image; wherein the targets at least include the target to be tracked;
[0142] Determining a foreground target of the target to be tracked according to the perspective relationship;
[0143] Determine whether there is an interference object in the foreground target that blocks the target to be tracked.
[0144] In this exemplary embodiment, each image of the target tracking can be obtained by means of camera, radar detection, etc. By analyzing the perspective relationship of each target in the image, it is confirmed whether the tracking target is blocked by interference. Interference is a target other than the tracking target in the image, including: people other than the tracked person, moving vehicles, fixed devices (including fences, etc.), etc.
[0145] In this exemplary embodiment, the perspective relationship refers to the spatial position, outline and projection of each target in the field of view, including relative relationships such as near is big and far is small, near is high and far is low, near is sparse and far is dense, near is wide and far is narrow, etc. In this exemplary embodiment, the spatial position, outline and other information of each target in the field of view can be captured by a tracking device (e.g., a camera device, etc.) by taking a video or photographing.
[0146] In some embodiments, the first processing unit is used to determine whether there is an interference object in the foreground target that blocks the target to be tracked, including:
[0147] The first processing unit is specifically configured to determine whether the position of the foreground target overlaps with the position of the target to be tracked;
[0148] If so, it is determined that the target to be tracked is blocked by the interference object in the n-1th frame image.
[0149] In this exemplary embodiment, the foreground target refers to a target that is closer to the tracking device than the target to be tracked, based on the tracking device. The foreground target is one or more of the interference objects. That is, any interference object may become a foreground target.
[0150] In this exemplary embodiment, when the position of the foreground target overlaps with the position of the target to be tracked, the interference at the foreground target position blocks the field of view of the tracking device for obtaining information of the target to be tracked. At this time, the target to be tracked is blocked by the interference in the n-1th frame image.
[0151] In some embodiments, the first processing unit is used to determine the perspective relationship between different targets in the n-1th frame image, and the foreground target of the target to be tracked is determined according to the perspective relationship, including:
[0152] The first processing unit is specifically configured to determine the external frames of the different targets;
[0153] Determining a first distance between a lower edge of a bounding box of the target to be tracked and a lower edge of the n-1th frame image, and determining a second distance between a lower edge of a bounding box of a target other than the target to be tracked and a lower edge of the n-1th frame image; and
[0154] If the first distance is greater than the second distance, the target corresponding to the second distance is determined to be the foreground target of the target to be tracked.
[0155] In this exemplary embodiment, Figure 4 FIG. 1 is a schematic diagram of determining an occlusion relationship according to an exemplary embodiment. Figure 4 As shown, in the acquired frame image, when the first distance is greater than the second distance, it means that the distance between the interference object and the tracking device is less than the distance between the target to be tracked (the obscured target in the figure) and the tracking device. At this time, the interference object is the foreground target of the target to be tracked. At the same time, if the position of the foreground target overlaps with the position of the target to be tracked, the interference object obscures the target to be tracked. In this exemplary embodiment, the circumscribed frame is a feature capture frame circled according to the tracking feature (global feature or local feature) when the tracking device captures the target. The circumscribed frame can be any shape such as a rectangle or a circle.
[0156] In this exemplary embodiment, the outer frame is a feature capture frame defined according to the tracking feature (global feature or local feature) when the tracking device captures the target. The outer frame can be any shape such as a rectangle or a circle.
[0157] In some embodiments, the second processing unit is used to determine the unobstructed area of the target to be tracked in the n-1th frame image, including:
[0158] The second processing unit is specifically configured to obtain an intersection between an outer bounding box of the interference object and an outer bounding box of the target to be tracked, so as to obtain an area blocked by the target to be tracked;
[0159] The blocked area is removed from the target to be tracked to obtain the unblocked area of the target to be tracked in the (n-1)th frame image.
[0160] In this exemplary embodiment, when tracking a target to be tracked, an external frame for tracking can be enclosed according to the features of the target to be tracked. The external frame can be a rectangle, a circle, or other set shapes. The external frame can enclose the entire tracking target and the overall features of each interference object as needed. At the same time, the external frame can also enclose the local features of the target to be tracked or the interference object.
[0161] In some embodiments, a fourth processing unit is further included, which is used to determine the relative position relationship of the unobstructed area in the target to be tracked according to the position information of the unobstructed area and the position information of the external frame of the target to be tracked;
[0162] According to the relative position relationship, the local tracking information of the n-th frame image is converted into the global tracking information of the target to be tracked in the n-th frame image.
[0163] In this exemplary embodiment, after obtaining the local tracking information of the target to be tracked in the nth frame image, the global tracking information of the target to be tracked in the nth frame image is determined according to the relative position relationship of the unobstructed local part in the target to be tracked in the n-1th frame image. The relative position relationship may include relative position coordinates.
[0164] For example, the local tracking information of the target to be tracked in the nth frame image includes the position coordinates of the local features of the target to be tracked in the nth frame image (for example, the position coordinates of the upper left corner of the external frame enclosing the local features in the nth frame image: X' 局 , Y' 局 ; where X' 局 is the coordinate of the upper left corner of the bounding box that encloses the local feature in the X-axis direction in the nth frame image, where Y' 局 is the Y-axis coordinate of the upper left corner of the bounding box that encloses the local feature in the nth frame image);
[0165] Determine the position coordinates of the upper left corner of the external frame that encloses the overall features of the target to be tracked in the n-1th frame image: X 整 , Y 整 ; Among them, X 整 is the X-axis coordinate of the upper left corner of the external frame that encloses the overall features of the target to be tracked in the n-1th frame image, and Y 整 The Y-axis coordinate of the upper left corner of the external frame that encloses the overall features of the target to be tracked in the n-1th frame image;
[0166] Determine the position coordinates of the upper left corner of the bounding box that encloses the local features in the n-1th frame image: X 局 , Y 局 ; Among them, X 局 is the X-axis coordinate of the upper left corner of the bounding box that delimits the local area feature in the n-1th frame image, Y 局 The Y-axis coordinate of the upper left corner of the bounding box that encloses the local area feature in the n-1th frame image;
[0167] Determine the relative position relationship of the unobstructed local area in the target in the n-1th frame image: X 差 =X 整-X 局 , Y 差 =Y 整 -Y 局 ; Among them, X 差 Y is the coordinate difference in the X-axis direction between the upper left corner of the bounding box that encloses the overall features of the tracking target and the upper left corner of the bounding box that encloses the local area features in the n-1th frame image, 差 is the coordinate difference in the Y-axis direction between the upper left corner of the bounding box that encloses the overall features of the tracking target and the upper left corner of the bounding box that encloses the local area features in the n-1th frame image;
[0168] Determine the global tracking information X' of the tracking target in the nth frame image 整 =X' 局 +X 差 , Y' 整 =Y' 局 +Y 差 ; where X' 整 The coordinate of the upper left corner of the external frame that encloses the overall features of the tracking target in the X-axis direction in the nth frame image, Y' 整 It is the Y-axis coordinate of the upper left corner of the bounding box that encloses the overall features of the tracking target in the nth frame image.
[0169] Determine the width W of the bounding box that encircles the overall features of the tracking target in the n-1th frame image 整 and high H 整 At the same time, the width W' and height H' of the bounding box encircling the overall features of the tracking target in the n-th frame image are the same as the width W and height H in the n-1-th frame image.
[0170] In this exemplary embodiment, radar, binocular vision and other methods can also be used to determine the occlusion relationship between targets or between targets and the environment. Combined with background modeling, optical flow and other algorithms, the unoccluded target area can be determined more accurately. The tracking algorithm is changed to use other feature models, deep learning, etc.
[0171] In some embodiments, the third processing unit is further used to resume global tracking of the target to be tracked from the mth frame image if it is detected that the target to be tracked in the mth frame image is not obstructed, wherein m is a positive integer greater than or equal to n.
[0172] In this exemplary embodiment, when the local area tracking of the target to be tracked is completed and the target is not blocked, it can be converted to global area tracking.
[0173] In this exemplary embodiment, when the unobstructed local area of the target to be tracked is obstructed, the tracking can be switched to other unobstructed local areas. When all unobstructed local areas are obstructed, the tracking is switched to the global area of the target.
[0174] The present disclosure also provides an electronic device. Figure 8 FIG. 1 is a schematic diagram showing the structure of an electronic device according to an exemplary embodiment. Figure 8 As shown, the electronic device provided by the embodiment of the present disclosure includes: a processor 830 and a memory 820 for storing a computer program that can be run on the processor, wherein the processor 830 executes the steps of the method provided in the above embodiments when running the computer program.
[0175] The present disclosure also provides a computer-readable storage medium. The computer-readable storage medium provided in the embodiments of the present disclosure stores a computer program, which, when executed by a processor, implements the steps of the method provided in the above embodiments.
[0176] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0177] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0178] In addition, all functional units in the embodiments of the present invention may be integrated into one processing module, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0179] In some cases, if any two of the above technical features do not conflict with each other, they can be combined into a new method technical solution.
[0180] In some cases, if any two of the above technical features do not conflict with each other, they can be combined into a new equipment technical solution.
[0181] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks or optical disks, etc., various media that can store program codes.
[0182] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. A target tracking method, characterized in that: The method comprises: Determine whether the target to be tracked is blocked in the n-1th frame image, where n is a positive integer greater than 1; If a local area of the target to be tracked is blocked in the n-1th frame image, determine an unblocked area of the target to be tracked in the n-1th frame image; Based on the tracking information of the target to be tracked in the n-1th frame image, locally track the unblocked area in the nth frame image to obtain the local tracking information of the target to be tracked in the nth frame image, wherein the n-1th frame image is a frame image before the nth frame image; The step of determining whether the target to be tracked is blocked in the n-1th frame image includes: Determining the perspective relationship between different targets in the n-1th frame image; wherein the targets at least include the target to be tracked; Determining a foreground target of the target to be tracked according to the perspective relationship; Determining whether there is an interference object in the foreground target that blocks the target to be tracked; The determining whether there is an interference object in the foreground target that blocks the target to be tracked includes: Determining whether the position of the foreground target overlaps with the position of the target to be tracked; If yes, it is determined that the target to be tracked is blocked by the interference object in the n-1th frame image; The determining of the perspective relationship between different targets in the n-1th frame image, and the determining of the foreground target of the target to be tracked according to the perspective relationship, include: Determining the bounding boxes of the different targets; Determine a first distance between a lower edge of a bounding box of the target to be tracked and a lower edge of the n-1th frame image, and determine a second distance between a lower edge of a bounding box of a target other than the target to be tracked and a lower edge of the n-1th frame image; If the first distance is greater than the second distance, determining that the target corresponding to the second distance is the foreground target of the target to be tracked; The determining of an unobstructed area of the target to be tracked in the n-1th frame image includes: Obtaining the intersection between the outer bounding box of the interference object and the outer bounding box of the target to be tracked, and obtaining the area blocked by the target to be tracked; The blocked area is removed from the target to be tracked to obtain the unblocked area of the target to be tracked in the (n-1)th frame image.
2. The target tracking method according to claim 1, characterized in that: The method further comprises: Determining a relative position relationship of the unobstructed area in the target to be tracked according to the position information of the unobstructed area and the position information of the bounding box of the target to be tracked; According to the relative position relationship, converting the local tracking information of the n-th frame image into the global tracking information of the target to be tracked in the n-th frame image; The method further comprises: If it is detected that the target to be tracked in the m-th frame image is not blocked, global tracking of the target to be tracked is resumed from the m-th frame image, wherein m is a positive integer greater than or equal to n.
3. A target tracking device, characterized in that: The device comprises: A first processing unit, used to determine whether the target to be tracked is blocked in the n-1th frame image, where n is a positive integer greater than 1; A second processing unit is used to determine an unobstructed area of the target to be tracked in the n-1th frame image if a local area of the target to be tracked is obstructed in the n-1th frame image; a third processing unit, configured to perform local tracking of the unobstructed area in the nth frame image based on the tracking information of the target to be tracked in the n-1th frame image, so as to obtain the local tracking information of the target to be tracked in the nth frame image, wherein the n-1th frame image is a frame image before the nth frame image; The first processing unit is used to determine whether the target to be tracked is blocked in the n-1th frame image, including: the first processing unit is specifically used to determine the perspective relationship between different targets in the n-1th frame image; wherein the targets at least include the target to be tracked; Determining a foreground target of the target to be tracked according to the perspective relationship; Determining whether there is an interference object in the foreground target that blocks the target to be tracked; The first processing unit is used to determine whether there is an interference object in the foreground target that blocks the target to be tracked, including: The first processing unit is specifically used to determine whether the position of the foreground target overlaps with the position of the target to be tracked; If yes, it is determined that the target to be tracked is blocked by the interference object in the n-1th frame image; The first processing unit is used to determine the perspective relationship between different targets in the n-1th frame image, and to determine the foreground target of the target to be tracked according to the perspective relationship, including: The first processing unit is specifically configured to determine the external frames of the different targets; Determine a first distance between a lower edge of a bounding box of the target to be tracked and a lower edge of the n-1th frame image, and determine a second distance between a lower edge of a bounding box of a target other than the target to be tracked and a lower edge of the n-1th frame image; and if the first distance is greater than the second distance, determine that the target corresponding to the second distance is a foreground target of the target to be tracked; The second processing unit is used to determine the unobstructed area of the target to be tracked in the n-1th frame image, including: The second processing unit is specifically configured to obtain an intersection between an outer bounding box of the interference object and an outer bounding box of the target to be tracked, so as to obtain an area blocked by the target to be tracked; Removing the blocked area from the target to be tracked, to obtain the unblocked area of the target to be tracked in the n-1th frame image; The system further includes a fourth processing unit, configured to determine a relative position relationship of the unobstructed area in the target to be tracked according to the position information of the unobstructed area and the position information of the bounding box of the target to be tracked; According to the relative position relationship, converting the local tracking information of the n-th frame image into the global tracking information of the target to be tracked in the n-th frame image; The third processing unit is further configured to resume global tracking of the target to be tracked from the mth frame image if it is detected that the target to be tracked in the mth frame image is not blocked, wherein m is a positive integer greater than or equal to n.
4. An electronic device, characterized in that: The method comprises: a processor and a memory for storing a computer program that can be run on the processor, wherein the processor is configured to execute the steps of the method according to any one of claims 1 to 2 when running the computer program.
5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 2 are implemented.