Target tracking method and device, electronic device, computer readable storage medium

Abstract

The application provides a target tracking method and device, electronic equipment and computer readable storage medium, and relates to the field of computer vision and target tracking. The target tracking method comprises the following steps: determining a tracking box of a target in a previous frame and a detection box of the target in a current frame; determining that the target is possibly in an occlusion state according to an intersection-over-union of the tracking box and the detection box; comparing an intersection of the tracking box and the detection box with the detection box to obtain a comparison result; and determining that the target is in the occlusion state according to the comparison result. The application can solve the problem of target loss caused by the occlusion of the target, improve the robustness and accuracy of target tracking, and provide a good user experience.

Description

Technical Field

[0001] The present application relates to the fields of computer vision and target tracking, and specifically to a target tracking method and device, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the continuous improvement of modern living standards, computer technology and image recognition technology have been gradually applied to various fields. By obtaining the target position in each frame of the image, it is possible to track the target. The tracked target can specifically include human bodies, objects, etc.

[0003] However, during the tracking process, complex scenes, especially occluded environments, will affect the robustness and accuracy of target tracking. Summary of the Invention

[0004] In view of this, embodiments of the present application provide a target tracking method and device, an electronic device, and a computer-readable storage medium, which significantly improve the problem of target loss due to target obstruction.

[0005] In a first aspect, the present application provides a target tracking method, including: determining a tracking frame of a target in a previous frame and a detection frame in a current frame; determining that the target may be in an occluded state based on an intersection-over-union ratio (IOR) of the tracking frame and the detection frame; comparing the intersection of the tracking frame and the detection frame with the detection frame to obtain a comparison result; and determining that the target is in an occluded state based on the comparison result.

[0006] In one embodiment, the comparison result is the ratio of the area of ​​the intersection of the tracking frame and the detection frame to the area of ​​the detection frame. Based on the comparison result, it is determined that the target is in an occlusion state, including: when the ratio is greater than or equal to a first preset threshold, it is determined that the target is in an occlusion state.

[0007] In one embodiment, the first preset threshold is 0.60-0.80.

[0008] In one embodiment, determining that the target may be in an occlusion state based on the tracking frame and the detection frame includes: determining that the target may be in an occlusion state when an intersection-over-union ratio of the detection frame and the tracking frame is less than a second preset threshold.

[0009] In one embodiment, the second preset threshold is 0.30-0.45.

[0010] In one embodiment, the target tracking method of the first aspect further includes: when it is determined that the target is in an occlusion state, retaining the target in a tracking list and not activating the state of the target.

[0011] In one embodiment, keeping the target in the tracking list and not activating the target state includes: keeping the target in the tracking list and not updating the tracking frame and target information of the target.

[0012] In one embodiment, the target tracking method of the first aspect further includes: when the target is in a non-occluded state again, triggering the update of the state of the target in the tracking list, fusing the tracking frame with the detection frame to obtain a fusion result, and updating the tracking frame according to the fusion result to activate the state of the target in the tracking list.

[0013] In one embodiment, the target tracking method of the first aspect further includes: tracking the target when it is determined that the target is in a non-blocked state.

[0014] In one embodiment, the target tracking method of the first aspect further includes: when it is determined based on the intersection-over-union ratio that the target may be in an occluded state, and when it is determined based on the comparison result that the target is in a non-occluded state, deleting the target from the tracking list or retaining the target in the tracking list for a certain period of time, and when the target appears again, activating the state of the target and tracking the target.

[0015] In one embodiment, the tracking target is one or more.

[0016] In one embodiment, the aforementioned target tracking method may be used in an offline embedded target tracking system.

[0017] A second aspect of the present application provides a target tracking device, including: a first determination module, which determines the tracking frame of the target in the previous frame and the detection frame in the current frame; a second determination module, which determines that the target may be in an occlusion state based on the intersection-over-union ratio of the tracking frame and the detection frame; a comparison module, which compares the intersection of the tracking frame and the detection frame with the detection frame to obtain a comparison result; and a third determination module, which determines that the target is in an occlusion state based on the comparison result.

[0018] The third aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executed by the processor. When the processor executes the computer program, the steps of the target tracking method provided in any of the above embodiments are implemented.

[0019] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the target tracking method provided in any of the above embodiments are implemented.

[0020] According to the technical solution of the present application, based on the intersection-and-union ratio of the target tracking frame of the previous frame and the target detection frame of the current frame, it is determined that the target may be in an occluded state. Furthermore, for the target determined to be in an occluded state, the intersection of the target tracking frame of the previous frame and the target detection frame of the current frame is compared with the target detection frame of the current frame to obtain a comparison result, and based on the comparison result, it is determined that the target is in an occluded state. The above-mentioned tracking process performs secondary judgment confirmation and screening on the basis of the preliminary judgment of the target state through the intersection-and-union ratio, accurately judges the target state, and enables the subsequent execution of the corresponding tracking strategy based on whether the target is actually in an occluded state, which can solve the problem of target loss due to target occlusion, improve the robustness and accuracy of target tracking, and provide a good user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] To more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be derived from these drawings without inventive effort.

[0022] Figure 1 The figure is a flow chart of a target tracking method provided in one embodiment of the present application.

[0023] Figure 2 The figure is a flowchart of an execution process of determining that a target may be in an obstructed state provided by an embodiment of the present application.

[0024] Figure 3 Shown is a flowchart of a tracking method for different target states provided by an embodiment of the present application.

[0025] Figure 4 Shown is a flow chart of a target tracking method provided in another embodiment of the present application.

[0026] Figure 5 Shown is a schematic diagram of a target tracking frame and a target detection frame when the target is in different states in one embodiment of the present application.

[0027] Figure 6 FIG. 1 is a schematic diagram of the structure of a target tracking device provided in one embodiment of the present application.

[0028] Figure 7 Shown is a schematic structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0030] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are illustrative only, and different aspects of the apparatus and method can use different modules.

[0031] The embodiments of the present application relate to applications in image processing and target tracking. To facilitate understanding, the following briefly introduces the relevant terms and concepts related to target tracking that may be involved in the embodiments of the present application.

[0032] (1) Target detection box

[0033] The target detection box is a real bounding box containing target information extracted from each frame of the image through the target detection algorithm, which is used to identify the location of the target.

[0034] (2) Target tracking frame

[0035] The target tracking frame is used to represent the size and position of the target in the current frame, which is predicted by the model based on the size and position of the target in the previous frame. It should be noted that tracking in this application is sometimes also called following.

[0036] (3) IOU (Intersection over Union)

[0037] IOU is the ratio of the area of ​​the intersection of the target tracking frame of the previous frame and the target detection frame of the current frame to the area of ​​the union of the target tracking frame of the previous frame and the target detection frame of the current frame.

[0038] (4) Target ID

[0039] The target ID is a unique identifier assigned to each tracked target.

[0040] Summary of the application

[0041] With the continuous improvement of modern living standards and the development of science and technology, as well as the diversification of people's entertainment needs, computer vision and image recognition and tracking technologies have gradually been applied to various fields.

[0042] For example, based on the needs of contemporary live broadcasting scenarios, offline embedded intelligent tracking pan-tilt systems are increasingly being used in live broadcasts, enabling real-time tracking of specific targets. For example, by analyzing the positional information of a target in a continuous video sequence, the IOU of the target frame in the upper and lower frames is used to determine the match result with a preset IOU threshold. When the match result meets the preset conditions, the corresponding target is assigned a corresponding ID, and the target is then tracked based on the positional information of the target ID.

[0043] Taking an embedded tracking system as an example, an embedded camera first captures images. Human body and head-and-shoulder detection technologies are then used to locate key human positions, obtain target person information and a detection frame, and assign the target person an ID. During the subsequent tracking process, the IOU of the human frame in the upper and lower frames is calculated and matched against a preset IOU threshold to obtain a matching result. Based on the matching result, the ID of each target person in the current frame is determined. Finally, the target person is tracked based on their ID, and their position information is transmitted to the controller. The controller then controls the motor to drive the pan / tilt system (PTZ) and the camera to follow the target person, thereby achieving automatic tracking of the target person. This IOU matching can generally achieve target tracking in simple scenarios.

[0044] However, in an obstructed environment, the target (e.g., a person) may be temporarily obscured, causing the target ID to switch, leading to the tracking device tracking another target. Consequently, in this case, the tracking device suffers from low robustness and accuracy, resulting in a poor user experience. The above technical solution suffers from the problem of losing the tracked target due to occlusion, which in turn causes the tracking device to switch targets.

[0045] Exemplary method

[0046] Figure 1 This is a flow chart of a target tracking method provided by an embodiment of the present application. The target tracking method can be applied to a target tracking device or a target tracking system and can be executed by a computing device (eg, a server or a user terminal). Figure 1 As shown, the target tracking method 100 includes steps S110 to S140.

[0047] Step S110 , determining the tracking frame of the target in the previous frame and the detection frame in the current frame.

[0048] Specifically, the detection box is a ground-truth bounding box of the target generated by image preprocessing for each frame captured from the video. It is used to identify the target's position in that frame. Additionally, the source of the initial frame image can be a pre-input image containing the tracked target, such as a photo of a live broadcast, mobile chat, or photo, or an image acquired through facial recognition. It can also be a frame captured in real time by an image acquisition device (e.g., a camera) during the tracking process, such as a frame in which the tracked target appears.

[0049] In one embodiment, the above-mentioned image preprocessing mainly includes performing target detection on each frame of the input image through a target detection algorithm, thereby determining the position information of the target and obtaining the target detection frame of each frame. For example, the target detection algorithm can be a single shot multi box detector algorithm (SSD), a region algorithm based on convolutional neural network features (Faster RCNN, Faster Regions with CNN features), a glance target detection algorithm (YOLO, You Only Look Once), etc. The embodiments of the present application are not limited to this, and other target detection algorithms can also be used.

[0050] In one embodiment, one of the aforementioned algorithms can be selected to perform object detection on the current frame and the previous frame, using the current frame and the previous frame as input images to obtain object detection frames for each object in the current frame and the previous frame. The image within each object detection frame represents an object and carries the coordinate position information of the object detection frame.

[0051] In one embodiment, the target tracking frame is a prediction frame of the target position generated based on the target detection frames of the previous frame and the current frame containing the target, as well as the position relationship between the previous frame and the current frame, after the target information of the previous frame and the current frame are fused and updated.

[0052] In another embodiment, the target tracking frame is a predicted frame of the target position generated based on the target detection frame of the first frame.

[0053] Taking human body detection as an example, its main purpose is to find the main position of the human body in the picture, and execute the subsequent target tracking algorithm based on the human body position information. Specifically, the main target detection algorithm is based on deep learning algorithms, such as the Generic Object Tracking Using Regression Networks (GOTURN) and Fast Compressive Tracking (FCT). The above algorithms collect a large amount of human body data and annotate the human body, and then train deep learning models of human and non-human bodies based on deep learning training. It can also realize real-time human body detection function on the acquired images, providing basic information for the subsequent fusion of related target tracking frames and target detection frames and determining the target ID.

[0054] For example, by fusing the target detection frame of each frame in which the target is confirmed to be in a non-occluded state through the following target state judgment with the target information of the tracking frame of the previous frame, and combining it with an adaptive algorithm, a window-adaptive target tracking frame can be obtained, and the state of the target in the tracking list can be set to inactive.

[0055] Step S120 : Based on the intersection-over-union (IoU) of the tracking frame and the detection frame, it is determined that the target may be in an occluded state.

[0056] Commonly used algorithms for judging target occlusion status include IOU matching algorithm, response graph numerical comparison algorithm, etc. In the embodiment of the present application, IOU matching is selected to judge that the tracking target of the current frame may be in an occlusion state, that is, to preliminarily judge that the tracking target of the current frame is in an occlusion state.

[0057] Specifically, IOU matching is to calculate the IOU value between the target tracking frame of the previous frame and the target detection frame of the current frame, and compare it with a second preset threshold (preset IOU threshold). When the calculated IOU value is less than the set threshold, it means that the target tracking frame of the previous frame does not match the target detection frame of the current frame, thereby determining that the tracked target of the current frame may be in an occluded state.

[0058] However, using IOU cannot accurately determine whether the tracked target is in an occluded state, because when the target leaves, the IOU value is also less than the set threshold. Therefore, based on the preliminary determination of which targets may be occluded, it is necessary to further confirm whether the tracked target is actually occluded or has left the video frame.

[0059] It should be understood that the IoU calculation formula of the embodiments of the present application is not limited thereto, and other IoU formulas that reflect the occlusion state or occlusion probability to a certain extent may also be used.

[0060] Step S130 : Compare the intersection of the tracking frame and the detection frame with the detection frame to obtain a comparison result.

[0061] Specifically, for the case where the target is only partially occluded and the IOU value is less than a preset threshold, the inventors of this application have found after long-term research that it is possible to determine whether the target is actually in an occluded state based on whether the ratio of the area of ​​the intersection of the target tracking frame of the previous frame and the target detection frame of the current frame to the area of ​​the target detection frame of the current frame meets the preset relationship. For example, for a tracking target that may be in an occluded state in the current frame, the area of ​​the intersection of the target tracking frame of the previous frame and the target detection frame of the current frame is compared with the area of ​​the target detection frame of the current frame, that is, the IOA (intersection and target detection frame area ratio) value is calculated. If the IOA value meets the preset threshold, it is determined that the target is actually in an occluded state.

[0062] For example, an IOA can be defined as:

[0063]

[0064] Among them, A is the area of ​​the target detection frame of the current frame, B is the area of ​​the target tracking frame of the previous frame, and A∩B is the area of ​​the intersection of the target tracking frame of the current frame and the target detection frame of the previous frame.

[0065] Figure 5 Shown is a schematic diagram of a target tracking frame and a target detection frame when the target is in different states in one embodiment of the present application.

[0066] like Figure 5 As shown in (b), in the n+1th frame, the target 3 is in an occluded state. At this time, IOA is the ratio of the area of ​​the intersection of the target detection frame 22 of the n+1th frame (i.e., A in the definition of IOA above) and the target tracking frame 12 of the nth frame (i.e., B in the definition of IOA above) to the area of ​​the target detection frame 22 of the n+1th frame.

[0067] Step S140: Determine whether the target is in an obstructed state based on the comparison result.

[0068] After the calculated value of IOA is obtained, the calculated value of IOA is compared with a first preset threshold (preset IOA threshold).

[0069] In the embodiment of the present application, the preset IOA threshold is a value between 0.60 and 0.80. Setting the IOA threshold within this range helps to significantly improve the accuracy of determining the occlusion state.

[0070] Based on the comparison results, it is determined whether the tracked target is in an occlusion state or has left the video frame. For example, when the IOA value is 0.85, which is greater than the set IOA threshold, the target is determined to be in an occlusion state.

[0071] According to the technical solution of the present application, based on the intersection over union of the target tracking box of the previous frame and the target detection box of the current frame, it is determined that the target may be in an occluded state. For the target determined to be in the occluded state, the intersection of the target tracking box of the previous frame and the target detection box of the current frame is compared with the target detection box of the current frame to obtain a comparison result, and according to the comparison result, it is determined that the target is in an occluded state. The above tracking process is based on the preliminary judgment of the target state by the intersection over union, and the IOA is used for secondary judgment confirmation and screening, so that the target state can be accurately judged, and the subsequent tracking strategy can be executed according to whether the target is actually in an occluded state. The target loss problem caused by the occlusion of the target can be solved, the robustness and accuracy of target tracking are improved, and the user experience is good.

[0072] Figure 2 The flowchart of the execution process of determining that the target may be in an occluded state (step S120) based on the intersection over union of the tracking box and the detection box provided by an embodiment of the present application is shown. As shown in Figure 2 The step S120 includes:

[0073] Step S201, calculate the IOU value of the target detection box of the current frame and the target tracking box of the previous frame.

[0074] Step S202, compare the calculated IOU value with the second preset threshold (preset IOU threshold).

[0075] In an embodiment of the present application, the preset IOU threshold is a value in 0.30-0.45.

[0076] For example, the calculated IOU value is 0.20, which is less than the preset IOU threshold, and it is determined that the target may be in an occluded state.

[0077] Step S203, according to the comparison result of the calculated IOU value and the second set threshold, execute the subsequent program.

[0078] In an embodiment of the present application, when the calculated IOU value is greater than or equal to the preset IOU threshold, step S2041 is executed to track the target; when the calculated IOU value is less than the set IOU threshold, step S2042 is executed, that is, step S130 is executed to perform IOA calculation to further confirm the target state.

[0079] In another embodiment of the present application, when the calculated IOU value is less than the set IOU threshold, for example, the calculated IOU value is 0, and it is a target not determined before tracking (not in the tracking list), the current frame can be regarded as the first frame, added to the tracking list, and assigned with a target ID for tracking.

[0080] It should be noted that, in the embodiment of the present application, before determining the tracking target state in the current frame, the target state of the tracking target is marked as inactive.

[0081] In one embodiment of the present application, if the calculated IOA value is greater than or equal to the set IOA threshold, the target whose target status is inactive in the current tracking list is determined to be an obscured target; if the calculated IOA value is less than the set IOA value, it is determined that the tracked target has left the video screen.

[0082] Figure 3 The figure shows a schematic diagram of the tracking process of different target states provided by an embodiment of the present application. Figure 3 As shown in the figure, if the matching result obtained by IOU matching (comparing the calculated IOU value with the preset IOU value) is greater than or equal to the set threshold, that is, the target is in a normal tracking state, then tracking strategy 1 is executed. For the target that is initially confirmed to be in an occluded state based on the IOU matching, IOA matching is further performed (comparing the calculated IOA value with the preset IOA value). If the target is determined to be in an occluded state based on the IOA matching result, tracking strategy 2 is executed. If the target is determined to have left the video screen based on the IOA matching, tracking strategy 3 is executed.

[0083] In one embodiment of the present application, tracking strategy 1 is: tracking the target.

[0084] Specifically, tracking strategy 1 is to fuse the target detection frame of the current frame (e.g., the n+1th frame) with the target tracking frame of the previous frame (e.g., the nth frame) before tracking the target to form the target tracking frame of the current frame, and assign the target ID to the tracking target corresponding to the current frame, and activate the target state. Then, track the target. For example, when the IOU calculation value of the target detection frame of the n+1th frame and the target tracking frame of the nth frame is greater than or equal to the preset IOU threshold, the target tracking frame of the nth frame is fused with the target detection frame of the n+1th frame to form the target tracking frame of the n+1th frame, and assign the target ID to the tracking target corresponding to the n+1th frame, activate the target state, and track the target.

[0085] In one embodiment of the present application, tracking strategy 2 is: keeping the target in the tracking list and not activating the state of the target.

[0086] Specifically, tracking strategy 2 is to keep the target in the tracking list and not update the tracking frame and target information of the target; and when the target is in a non-occluded state again, it triggers the update of the state of the target in the tracking list, fuses the tracking frame with the detection frame, and updates the tracking frame to activate the state of the target in the tracking list.

[0087] For example, the target ID is saved in the tracking list (or the target ID is saved in the tracking list for a certain period of time), the target tracking frame and target information are not updated, that is, the target tracking frame and target information are still the same as the previous frame, and then the image is re-acquired, and the subsequent steps are performed in sequence to confirm whether the tracked target is in an occluded state according to the IOA matching, and the process is repeated. Until the target is in an unoccluded state again, the target status update in the tracking list information is triggered, and the target tracking frame of the previous frame is merged with the target detection frame information of the current frame, thereby activating the state of the target in the tracking list.

[0088] It should be noted that the target occlusion state involved in the embodiments of this application refers to the tracking target being blocked by other targets or background. In addition, the determination of the target being in an occlusion state based on the intersection-over-union ratio involved in this application, and the determination of the target being in an unocclusion state based on the comparison result, refer to the target having left the video frame.

[0089] In some embodiments of the present application, after an obscured target is identified, the obscured target is saved in a tracking list (or can be saved in the tracking list for a certain period of time). The tracking target information from the previous frame is not updated, and the tracking target ID from the previous frame is retained, thereby preventing the tracking target from being lost. This prevents the tracking system from switching targets when the target is obscured, thereby improving the problem of system tracking errors.

[0090] Furthermore, tracking strategy 3 is: when it is determined that the target may be in an occluded state according to the intersection-over-union ratio, and when it is determined that the target is in a non-occluded state according to the comparison result, the target is deleted from the tracking list, or the target is retained in the tracking list for a certain period of time. When the target appears again, the target state is activated and the target is tracked.

[0091] For example, when it is determined based on the intersection-over-union ratio that the target in the n+1th frame may be in an occluded state, and based on the comparison result of the subsequent IOA calculation value with the preset IOA threshold, it is determined that the target in the n+1th frame is in a non-occluded state, in some embodiments, it can be deleted from the tracking list; in some embodiments, the target information can also be saved in the tracking list, and a search strategy can be set. When the target appears again, the target state is activated and tracking strategy 1 is executed.

[0092] In one embodiment of the present application, the target may be a person or a moving object, and may be one or more.

[0093] In some embodiments, the target tracking method of the present application can be used in an offline embedded target tracking system.

[0094] The target tracking method provided by the embodiments of the present application can significantly improve the target tracking stability and user experience of the tracking system in multi-person scenarios, especially when the tracking target is obscured, without affecting the tracking effect in single-person scenarios, and can be used in embedded tracking systems.

[0095] The target tracking method and device provided in the embodiment of the present application can be applied to entertainment fields such as live broadcast of urban events that require real-time tracking, and can also be applied to fields in computer vision fields such as safe cities and smart terminals that require video surveillance. Specifically, the technical solution of the present application can be applied to live broadcast scenes and security video surveillance scenes. Figure 4 A brief introduction to live broadcast scenarios and security video surveillance scenarios.

[0096] The following combination Figure 4 of Figure 5 Describe the target tracking process of this application.

[0097] Figure 4 This is a flow chart of a target tracking method provided by another embodiment of the present application. Figure 4 The tracking methods include:

[0098] Step S410: The image acquisition module of the tracking device acquires an image.

[0099] For example, during tracking, the tracking device's image acquisition module captures images via an embedded camera. The initial frame captured can be a pre-entered video frame or photo containing the target, or a frame where the target first appears during tracking. Additional frames can be captured in real time during tracking, for example, during a live broadcast, the tracking system's embedded camera captures images of the live scene in real time.

[0100] Step S420: The information acquisition module of the tracking device pre-processes the image to generate a target detection frame.

[0101] For example, the tracking device's information acquisition module uses the target-containing images captured by the image acquisition module as its data foundation. Using target detection algorithms, combined with current mainstream deep learning algorithms, this involves collecting and annotating a large amount of human data. Deep learning models for both human and non-human subjects are then trained based on deep learning. Furthermore, techniques such as model distillation and int8 quantization are incorporated to optimize model effectiveness and efficiency. The optimized deep learning model can perform real-time human detection on incoming images, capturing target detection frames for each frame.

[0102] Step S430 : generating a target tracking frame based on the target frame of the initial frame containing the target or based on the target frames of the upper and lower frames containing the target.

[0103] For example, based on the target detection frame of the initial frame containing the target, or based on the target tracking frame of the previous frame and the target detection frame of the current frame in which the target is confirmed to be in a non-occluded state after target state judgment, as well as the position relationship between the previous frame and the current frame, the target information of the previous frame containing the target and the current frame are fused and updated to generate the current target tracking frame. At the same time, the target tracking result of the current frame is set to an unupdated state, such as 0, indicating that the target is in an inactive state. Figure 5 (a) is a schematic diagram showing the target 3 in the nth frame in the tracking process when it is in a non-occluded state, wherein 11 is the target tracking frame of the n-1th frame, and 21 is the target detection frame of the nth frame.

[0104] Step S440: Calculate the IOU value between the target tracking frame of the previous frame and the target detection frame of the current frame, and compare it with a preset IOU threshold.

[0105] For example, the IOU value between the target tracking frame of the nth frame and the target detection frame of the n+1th frame is calculated and compared with the preset IOU threshold.

[0106] In one embodiment of the present application, the preset IOU threshold is 0.30-0.45.

[0107] Furthermore, before performing IOU matching, a preliminary determination can be made of the front-to-back positional relationship of the tracked target between the previous and current frames. Object detection algorithms typically show a phenomenon in the target detection frame: the closer the target is to the camera, the larger the target detection frame. In multi-target scenarios, the front-to-back positional relationship of the targets can usually be determined by the size of the target detection frames. Based on this concept, the target detection frames of the current and previous frames are usually sorted by area size. This allows a preliminary determination of the front-to-back positional relationship of the targets in the previous and next frames, providing a reference for subsequent target status determination.

[0108] It should be noted that the image containing the target mentioned in the embodiments of the present application refers to an image in which the target is in a non-occluded state.

[0109] Step S450: According to the IOU matching result, the target state is judged to determine whether the tracked target may be in an occlusion state.

[0110] For example, if the calculated IOU value of the upper and lower frames is 0.25, which is less than the set IOU threshold, the target may be in an occluded state in the current frame.

[0111] Step S460: Perform IOA matching on the target that may be in an occluded state determined by IOU matching.

[0112] For example, for a target that is determined to be in an occluded state, the IOA value of the target tracking frame of the previous frame and the target detection frame of the current frame is calculated, and the IOA value is compared with a preset IOA threshold.

[0113] The preset IOA threshold is a value between 0.60 and 0.80.

[0114] Step S470: Determine whether the tracking target is in an occlusion state based on the IOA matching result.

[0115] Furthermore, according to the IOA matching result, if it is confirmed that the target is in an occluded state, the above tracking strategy 2 is executed. Figure 5 (b) is a schematic diagram of the target 3 in the n+1 frame during the tracking process when it is in an occluded state (occluded by the background 4), where 12 is the target tracking frame of the n frame and 22 is the target detection frame of the n+1 frame.

[0116] For example, in a live broadcast scenario, the calculated IOA value of the target tracking frame of the nth frame and the target detection frame of the n+1th frame is 0.85, confirming that the target is in an occluded state. The target ID is then saved in the tracking list for a certain period of time, and the target tracking frame and target information are not updated. That is, the target tracking frame and target information remain the same as those of the nth frame. Figure 5 (c) shows the tracking process of target 3, which is still occluded in frame n+2. The target tracking frame is still the target tracking frame 12 of frame n, and 23 is the target detection frame of frame n+2. Then, tracking strategy 2 is executed. When the target is unoccluded again in frame n+k, the target status in the tracking list is updated. The target tracking frame of frame n is fused with the target detection frame of frame n+k, activating the target status in the tracking list and tracking the target. Figure 5 (d) is a schematic diagram of the tracking process when the target 3 is in a non-occluded state again at the n+kth frame, where 12 is the target tracking frame of the nth frame before the target 3 is occluded, and 24 is the target detection frame of the n+kth frame. Exemplary apparatus (e) is a schematic diagram of the n+k+1th frame during the tracking process, where 13 is the target tracking frame of the n+kth frame generated by fusing the target tracking frame of the nth frame with the target detection frame of the n+kth frame when the target 3 is unoccluded again in the n+kth frame.

[0117] In step S480, if the target is likely to be in an occlusion state according to the IOU matching of each upper and lower frame, and is in a non-occlusion state according to the IOA matching of each upper and lower frame, tracking strategy 3 is executed.

[0118] Specifically, step S480 determines that the target may be in an occluded state based on the IOU matching of each upper and lower frame, and determines that the target is in an unoccluded state based on the IOA matching of the upper and lower frames, then deletes the target from the tracking list or retains the target in the tracking list for a certain period of time. When the target appears again in the n+k frame, the target state is activated and the target is tracked.

[0119] Step S490: When it is determined through IOU matching that the target is in a non-occluded state, the target state is activated.

[0120] For example, when the target is determined to be in a non-occluded state through IOU matching, the state of the target in the tracking list is activated, the target tracking frame of the previous frame is fused with the target detection frame information of the current frame to form the target tracking frame of the current frame, and the target ID is assigned to the tracking target of the current frame.

[0121] Step S4110 , tracking the target in the activated state, that is, executing tracking strategy 1.

[0122] For example, during a live broadcast, a target person who is determined to be in a non-blocked state and has activated a state is tracked.

[0123] In some embodiments, the target tracking method of the present application may be used in an embedded target tracking system.

[0124] In an embodiment of the present application, there are one or more tracking targets, and the tracking targets are people or objects.

[0125] In summary, the embodiments of the present application can use the image acquired by the image acquisition module as the data basis, obtain the target detection frame of each frame through the target detection algorithm, and generate the target tracking frame of the previous frame. At the same time, the target tracking result of the previous frame is set to an unupdated state, such as 0, indicating that the target is in an inactive state. Then, the IOU calculated value of the target tracking frame of the previous frame and the target detection frame of the current frame is matched and compared with the preset IOU threshold. For targets that are determined to be in an occluded state, IOA matching is further performed to confirm whether the target is in an occluded state or has left the video screen. For targets that are confirmed to be in an occluded state, the target is retained in the tracking list, the target state is not activated, and the tracking frame and target information of the target are not updated. When the target is in a non-occluded state again, the update of the target state in the tracking list is triggered, the tracking frame and the detection frame are fused, and the tracking frame is updated to activate the state of the target in the tracking list for tracking. For targets that have left the video frame, you can delete them from the tracking list or keep them in the tracking list for a certain period of time. When the target reappears, the target status is activated and tracked. By performing a secondary screening and confirmation of IOA matching based on IOU matching, the target status is more accurately determined. Then, different tracking strategies are assigned to the target based on different target states, improving the robustness and accuracy of target tracking and providing a good user experience.

[0126] It should be noted that in some embodiments, when the current frame's target detection frame has multiple candidate frames, an IOU calculation is performed on each detection frame in the current frame and the tracking frame in the previous frame, obtaining N calculation results. These IOU calculation results are then sorted from largest to smallest. If the maximum value after sorting is greater than a preset IOU threshold, the target detection frame in the current frame and the target tracking frame in the previous frame are the same. In this case, the target table ID is assigned to the same target in the current frame, thereby updating the target information in the target frame corresponding to the target ID. Simultaneously, the corresponding target in the tracking list is set to an updated state, such as 1, indicating that the target tracking frame and target information have been updated and are in an active state, thus tracking the target. If the maximum value after sorting is less than the preset IOU threshold, the trigger condition for updating the target data state in the tracking list cannot be generated. Therefore, based on the tracking state information (inactive state), it can be preliminarily determined that the target is occluded. Of course, this is only a preliminary determination of the target, as it is possible that the target is not occluded but has moved away. In order to further confirm that the target is occluded, an IOA secondary judgment mechanism is introduced. It mainly selects the tracking frame of the target whose target tracking frame has not been updated in the tracking list and all the detection results of the current frame (each candidate frame in the current frame) for IOA calculation, and sorts the IOA scores from large to small. Then, the largest IOA calculation value is selected. If the largest IOA calculation value is greater than the preset IOA threshold (the system preset IOA threshold in the embodiment of the present application is 0.60-0.80), the target in the current tracking list that is inactive is considered to be an occluded target.

[0127] Figure 6

[0128] This application also provides some target tracking devices. Figure 6 FIG. 1 is a schematic diagram of the structure of a target tracking device according to an embodiment of the present application. Figure 7 As shown, the target tracking device 600 of this embodiment includes:

[0129] A first determination module 610, a second determination module 620, a comparison module 630, and a third determination module 640. The first determination module 610 is used to determine the target's tracking frame in the previous frame and its detection frame in the current frame. The second determination module 620 is used to determine whether the target is likely to be occluded based on the intersection-over-union ratio (IoU) of the tracking frame and the detection frame. The comparison module 630 is used to compare the intersection of the tracking frame and the detection frame with the detection frame to obtain a comparison result. The third determination module is used to determine whether the target is occluded based on the comparison result.

[0130] In one embodiment, the target tracking device further includes an image acquisition module for acquiring image information containing the target, such as photos of live broadcast scenes, mobile chat scenes, and photo scenes.

[0131] In an embodiment, the image acquisition module is configured to receive image information containing the target, such as a video frame or a photo containing the target.

[0132] In an embodiment, the first determination module 610 (which can also be an information acquisition module) is configured to perform target detection on each frame acquired by the image acquisition module, to obtain target position information and a target detection frame of the previous frame, and to generate a target tracking frame of the previous frame; at the same time, to obtain a target detection frame of the current frame.

[0133] In an embodiment, the second determination module 620 is configured to determine that the target is likely to be in an occluded state according to a comparison result of an IOU value of the target detection frame of the current frame and the target tracking frame of the previous frame with a preset IOU threshold value.

[0134] In an embodiment, the comparison module 630 is configured to compare an area of an intersection of the target detection frame of the current frame and the target tracking frame of the previous frame with an area of the target detection frame of the current frame, to obtain an IOA value after the comparison.

[0135] In an embodiment, the third determination module 640 is configured to perform secondary determination of the target state according to a comparison result of the IOA value of the target detection frame of the current frame and the target tracking frame of the previous frame with a preset IOA threshold value.

[0136] In an embodiment, the target tracking device further includes a tracking execution module configured to execute different tracking strategies according to the target state.

[0137] The target tracking apparatus provided in the embodiment belongs to the same application concept as the target tracking method provided in the embodiments of the present application, can execute the target tracking method provided in any embodiment of the present application, and has the corresponding function modules and beneficial effects of executing the target tracking method. Technical details not described in detail in the embodiment can be referred to the target tracking method provided in the embodiments of the present application, which will not be described here again.

[0138] Exemplary electronic device

[0139] Figure 7 is a structural block diagram of an electronic device provided in an embodiment of the present application. The electronic device can be a server or a mobile terminal. The mobile terminal can be a mobile phone, a computer, etc. As shown in the figure, the electronic device 700 includes a memory 730 and a processor 710 coupled to the memory 730, and the processor 710 is configured to execute the target tracking method in any of the foregoing embodiments based on instructions stored in the memory 730. Figure 7

[0140] ​The memory 730 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, application programs, a boot loader, and other programs.

[0141] In one embodiment, the target tracking system may further include an input device 720 and an output device 740 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0142] For example, the input device 720 may be one or more cameras or a face recognition device for acquiring images.

[0143] The output device 740 can output information to the outside, including tracking results, tracking videos, etc. The output device can be a display, a mobile phone, a connected remote output device, etc.

[0144] Of course, to simplify, Exemplary computer program product and computer readable storage medium Only some of the components related to the present application in the target tracking system 700 are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, the target tracking system 700 may also include any other appropriate components according to specific application scenarios.

[0145] In some embodiments of the present application, the target tracking system is an embedded tracking system. For example, the target tracking system is a consumer-grade embedded tracking system that has image acquisition, motor control, and a deep learning acceleration chip (low-end embedded AI chip).

[0146] In some embodiments, the target tracking apparatus may also be referred to as a target tracking device or a target tracking system.

[0147] ​

[0148] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions. When the computer program instructions are executed by a processor, the processor executes the steps of the method for obtaining training samples according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0149] An embodiment of the present application also includes a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the target tracking method in any of the aforementioned embodiments.

[0150] It will be clear to those skilled in the art that the embodiments of the present application can be provided as methods, devices, electronic devices or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0151] The basic principles of this application have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this application are merely illustrative and not restrictive, and it should not be assumed that these advantages, strengths, and effects are required of each embodiment of this application. In addition, the specific details disclosed above are merely illustrative and comprehensible, and are not intended to be limiting. These details do not limit this application to necessarily being implemented using these specific details.

[0152] The block diagrams of the devices, equipment, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0153] It should also be noted that in the apparatus and method of the present application, each component or each step can be decomposed and / or recombined, and such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0154] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0155] It should be understood that the qualifiers "first", "second", "third", etc. used in the description of the embodiments of this application are only used to more clearly illustrate the technical solutions and cannot be used to limit the scope of protection of this application.

[0156] While various example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, permutations, additions, and sub-combinations thereof.

Claims

1. A target tracking method, characterized in that: include: Determining a tracking frame of the target in the previous frame and a detection frame in the current frame; wherein the tracking frame in the previous frame is the size and position of the target in the current frame predicted based on the size and position of the target in the previous frame; Based on an intersection-over-union (IoU) of the tracking frame and the detection frame, when the IoU of the detection frame and the tracking frame is less than a second preset threshold, determining that the target may be in a partially occluded state or the target has left the video screen; Comparing the intersection of the tracking frame and the detection frame with the detection frame to obtain a comparison result; wherein the comparison result is a ratio of the area of ​​the intersection of the tracking frame and the detection frame to the area of ​​the detection frame; When the ratio is greater than or equal to a first preset threshold, it is determined that the target is in a partially blocked state, excluding the possibility that the target has left the video screen. 2 . The target tracking method according to claim 1 , wherein the first preset threshold is 0.60 to 0.

80. The target tracking method according to claim 1 , wherein the second preset threshold is 0.30 to 0.

45.

4. The target tracking method according to any one of claims 1 to 3, further comprising: When it is determined that the target is in a partially blocked state, the target is retained in a tracking list, and the state of the target is not activated.

5. The target tracking method according to claim 4, wherein the step of retaining the target in a tracking list and not activating the target state comprises: The target is retained in the tracking list, and the tracking frame and target information of the target are not updated.

6. The target tracking method according to claim 5, further comprising: When the target is in a non-occluded state again, the update of the state of the target in the tracking list is triggered, the tracking frame is fused with the detection frame to obtain a fusion result, and the tracking frame is updated according to the fusion result to activate the state of the target in the tracking list.

7. The target tracking method according to any one of claims 1 to 3, further comprising: When it is determined that the target is in a non-blocked state, the target is tracked.

8. The target tracking method according to any one of claims 1 to 3, further comprising: When the target leaves the video screen, the target is deleted from the tracking list or retained in the tracking list for a certain period of time. When the target appears again, the state of the target is activated and the target is tracked.

9. A target tracking device comprising: A first determination module determines a tracking frame of the target in the previous frame and a detection frame in the current frame; wherein the tracking frame in the previous frame is the size and position of the target in the current frame predicted based on the size and position of the target in the previous frame; a second determining module, based on an intersection-over-union (IoU) of the tracking frame and the detection frame, determining that the target may be in a partially occluded state or that the target has left the video screen when the IoU of the detection frame and the tracking frame is less than a second preset threshold; a comparison module, configured to compare an intersection of the tracking frame and the detection frame with the detection frame to obtain a comparison result; wherein the comparison result is a ratio of an area of ​​the intersection of the tracking frame and the detection frame to an area of ​​the detection frame; The third determining module determines that the target is in a partially blocked state, excluding the fact that the target has left the video screen when the ratio is greater than or equal to a first preset threshold.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executed by the processor, wherein: When the processor executes the computer program, the target tracking method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the target tracking method according to any one of claims 1 to 8 is implemented.

Images