Target tracking method, system and computer readable storage medium
By setting different tracking states for the target and selecting the appropriate tracker, the problem of poor tracking in the prior art is solved, and more flexible and accurate target tracking is achieved.
Patent Information
- Application Number
- CN202011641980.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-31
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2040-12-31
AI Technical Summary
The existing target tracking technology cannot be flexibly adjusted according to actual conditions, resulting in poor tracking results.
By setting different tracking states for the target during the target tracking process and selecting different trackers for tracking according to different states, for example, using a deep learning model-based tracker when the target is detected, and using kernel correlation when the target cannot be detected Filter tracker, timely converts tracking status to adapt to the image quality and position changes of the target.
Improve the flexibility and effect of target tracking, avoid the problem of missing images when targets are blocked due to poor image quality or targets, and enhance the accuracy and adaptability of tracking.
Smart Images

Figure CN112581507B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision, and more specifically, to a target tracking method, system and computer-readable storage medium. Background Art
[0002] Object tracking is an important research direction in the field of computer vision. Object tracking has a wide range of applications in video surveillance, human-computer interaction, unmanned driving and other fields.
[0003] The existing technology always uses one type of tracker to track the same target, but cannot be flexibly adjusted according to actual conditions, resulting in poor tracking effect. Summary of the invention
[0004] The present application provides a target tracking method, system, computer-readable storage medium and computer program product to improve the tracking effect of the target.
[0005] In a first aspect, a target tracking method is provided, comprising: in a process of tracking a target in a first state using a first tracker, performing target detection in a first detection frame; if the target is detected in the first detection frame, maintaining the tracking state of the target in the first state; if the target is not detected in the first detection frame, converting the tracking state of the target from the first state to a second state; in a next tracking frame of the first detection frame, selecting a tracker corresponding to the tracking state of the target to continue tracking the target, wherein the first state corresponds to the first tracker and the second state corresponds to the second tracker.
[0006] In one embodiment, if the target is detected at the edge of the first detection frame and moves toward the outside of the edge of the first detection frame, the tracking state of the target is converted from the first state to a third state, and the third state is used to indicate that the target is about to leave the scene corresponding to the image frame containing the target.
[0007] In one embodiment, if the tracking state of the target is the third state and the target is not detected in the next detection frame of the first detection frame, tracking of the target is stopped and the tracking result of the target is output.
[0008] In one embodiment, if the tracking state of the target is converted to the second state, the duration of the target tracking state being the second state is recorded; when the duration is greater than a first threshold, tracking of the target is stopped, and the tracking result of the target is output.
[0009] In one embodiment, before performing target detection in the first detection frame, performing target detection in the second detection frame, wherein the target is a newly detected target in the second detection frame; setting the tracking state of the target to a fourth state, the fourth state being used to indicate that the target is a newly detected target; detecting the image quality of the target in the fourth state; not tracking the target when the image quality of the target is less than a second threshold; and converting the tracking state of the target from the fourth state to the first state when the image quality of the target is greater than the second threshold, and using the first tracker to track the target.
[0010] In one embodiment, during the process of tracking a target, tracking information of the target is maintained, and the tracking information includes: the tracking state of the target, the position of the target in the image frame, and image information of the target image with the highest quality during the tracking process of the target; the image information of the target image includes at least one of the following information: the quality of the target image, the spatial position of the target image in the image frame to which the target image belongs, and the temporal position of the target image in the video sequence.
[0011] In one embodiment, the target is a target of a specific type, the first tracker is a tracker for targets of the specific type, and the second tracker is a tracker for targets of a non-specific type.
[0012] In one embodiment, the first tracker is a deep learning model based tracker, and the second tracker is a kernel correlation filter tracker.
[0013] In a second aspect, a target tracking system is provided, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the first aspect.
[0014] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method according to the first aspect are performed.
[0015] According to a fourth aspect, a computer program product is provided, comprising a computer program, which implements the steps of the method according to the first aspect when executed by a processor.
[0016] The present application sets different tracking states for the tracked target, and selects different trackers to track the target according to the different tracking states, which can improve the flexibility of tracking and the tracking effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 A schematic flowchart of a target tracking method provided in an embodiment of the present application.
[0018] Figure 2 It is a schematic diagram of a video sequence provided in an embodiment of the present application.
[0019] Figure 3 It is a schematic diagram of a tracking frame provided in an embodiment of the present application.
[0020] Figure 4 This is a schematic diagram of an example of the first detection frame provided in an embodiment of the present application.
[0021] Figure 5 It is a schematic diagram of another tracking frame provided in an embodiment of the present application.
[0022] Figure 6 It is a neural network heat map generated during the tracking process provided in the embodiment of the present application.
[0023] Figure 7 This is another example schematic diagram of the first detection frame provided in an embodiment of the present application.
[0024] Figure 8 It is a schematic diagram of a second detection frame provided in an embodiment of the present application.
[0025] Fig. 9 It is a schematic block diagram of a target tracking system provided in an embodiment of the present application. DETAILED DESCRIPTION
[0026] Object tracking is an important research direction in the field of computer vision. Object tracking obtains the position of the object in each frame by processing and analyzing the image frames. Object tracking has a wide range of applications in video surveillance, human-computer interaction, unmanned driving and other fields.
[0027] For example, during target tracking, the target in the image frame can be framed to obtain the target's tracked bounding box. The position of the tracking frame, the image in the tracking frame and other information can reflect the tracking status of the target in the current image frame. The tracking frame may change continuously during the tracking process, so it is necessary to maintain the target's tracking information (or tracking sequence). The tracking process can maintain the tracking information of the tracked target, which may include the target's image, position, quality and other information in each image frame. After the tracking of a target is completed, an image with better quality is generally selected from the target's tracking information for output.
[0028] The present application does not specifically limit the type of target being tracked, for example, it may be a face, a body, a vehicle, or an animal.
[0029] The target is generally tracked by some kind of tracker (or tracking algorithm). For example, the tracker can be a tracker for a specific type of target (generic object tracker), or a tracker for a non-specific type of target (non-generic object tracker), etc. Alternatively, the tracker can be a tracker based on a deep learning model, or a kernel correlation filter (KCF) tracker. Different trackers have different tracking principles, which makes different trackers have different tracking effects in different situations.
[0030] For example, a tracker based on a specific type of target usually has high requirements for the image quality of the target being tracked, otherwise it is easy to lose the target. A tracker based on a non-specific type of target generally tracks based on image content, and can continue to track the target even if the target's image quality is low. Therefore, when the target's image quality is poor, the target's posture is strange, or the target is blocked, the target cannot be detected, or the detected target quality does not meet the requirements, a non-specific type of target tracker can be used to track it; when the target's image quality is restored, a specific type of target tracker can be used to track it.
[0031] For example, the tracker based on the deep learning model has high requirements for the features of the target image. If the image has few features, it is easy to lose the target. The KCF tracker has low requirements for the features of the target. In this case, the KCF tracker can be used to track it.
[0032] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.
[0033] Figure 1 A schematic flow chart of a target tracking method provided in an embodiment of the present application. The method includes steps S110 to S130, which are described below respectively.
[0034] Step S110 , in the process of using the first tracker to track the target in the first state, target detection is performed in the first detection frame.
[0035] For example, the first state may be represented by "TRACK".
[0036] For example, in a tracking frame, a target is tracked using a first tracker, and in a first detection frame after the tracking frame, the target is detected using a detector (or a detection algorithm). The detector may be a detector based on a deep learning model or other types of detectors.
[0037] The interval between the tracking frame and the detection frame can be set according to actual needs, and the present application does not limit this. It is understandable that, because the algorithm for performing target detection is more complex and time-consuming than the algorithm for target tracking, the number of tracking frames can be set to be much larger than the number of detection frames, for example, the number of tracking frames is several times (5-20, etc.) of the detection frames, for example, 10 times. On the other hand, in order to ensure the accuracy of tracking by the tracking frame, one or more detection frames can be set after each continuous number of tracking frames. Optionally, several continuous tracking frames (which can be called tracking frame sequences) can be set after each detection frame, that is, there are tracking frames between two adjacent detection frames, and the interval can be uniform.
[0038] For example, a detection frame can be set every 10 image frames, and the remaining image frames can be set as tracking frames. Figure 2 The shown video sequence includes image frames, wherein the gray image frames are detection frames, for example, the 30th frame and the 40th frame may be detection frames, and the remaining image frames may be tracking frames, for example, the 29th frame, the 31st-39th frames, and the 41st-44th frames may all be tracking frames, wherein a plurality of consecutive tracking frames may be referred to as a tracking frame sequence, for example, the 31st-39th frames may be a tracking frame sequence.
[0039] Step S110 may include: tracking the target using the first tracker in the first tracking frame sequence, and detecting the target in the first detection frame after the first tracking frame sequence, and the first detection frame is adjacent to the first tracking frame sequence, that is, it is the first detection frame after the first tracking frame sequence. Figure 2 , the first tracking frame sequence may be the 21st to 29th frames, and the first detection frame may be the 30th frame.
[0040] Next, according to Figure 2 The video sequence shown, combined with Figure 3 and Figure 4 , the tracking and detection process of the first tracker is illustrated by taking the first tracker tracking a target in a first state as an example.
[0041] Figure 3 The tracking frame is shown, which can be Figure 2 In frame 29, the first tracker is used to track the face with ID face0, and face0 is in the TRACK state. Figure 3 It is marked with a light grey box 0.
[0042] Figure 4 The first detection frame is shown, which may be Figure 2 In the 30th frame, face0 is not detected (for a more intuitive representation, face0 is Figure 4 marked with a dark grey box 0).
[0043] S121: If the target is detected in the first detection frame, keep the tracking state of the target in the first state.
[0044] S122: If the target is not detected in the first detection frame, convert the tracking state of the target from the first state to the second state.
[0045] S130, in a next tracking frame of the first detection frame, selecting a tracker corresponding to the tracking state of the target to continue tracking the target, wherein the first state corresponds to the first tracker, and the second state corresponds to the second tracker. That is, in the next tracking frame, if the tracking state of the target is the first state, the first tracker is used for tracking, and if the tracking state of the target is the second state, the second tracker is used for tracking.
[0046] It can be understood that the next tracking frame can be a tracking frame immediately adjacent to the first detection frame, such as Figure 2 The 31st tracking frame is the next tracking frame of the 30th detection frame. The next tracking frame can also be any frame in the tracking sequence immediately adjacent to the first detection frame, such as Figure 2 The next tracking frame of the 30th frame may be any frame from the 31st frame to the 39th frame, for example, the 32nd frame, the 36th frame or the 39th frame.
[0047] Specifically, in any tracking frame (current frame), the tracker to be used for tracking can be determined according to the current tracking state: if the current tracking state is the first state, the first tracker is used for tracking; if the current tracking state is the second state, the second tracker is used for tracking. On the other hand, the current tracking state can also be used to indicate the detection result of the nearest detection frame before the current frame, wherein the current tracking state is the first state, indicating that the target is detected in the nearest detection frame before the current frame, and conversely, the current tracking state is the second state, indicating that the target is not detected in the nearest detection frame before the current frame.
[0048] Step S121 may include: if the target is detected in the first detection frame, keeping the tracking state of the target in the first state.
[0049] Optionally, if the target is detected in the first detection frame and does not satisfy the condition that "the target is located at the edge of the first detection frame and the moving speed of the target is toward the edge", the tracking state is kept in the first state. On the contrary, if the target is detected in the first detection frame and is located at the edge of the first detection frame and the moving speed of the target is toward the edge, the state is updated to the third state, as shown in the following combination Figure 7 Part of the description.
[0050] It can be understood that when the tracking state of the target is the first state, the process of using the first tracker in the second tracking frame sequence after the first detection frame is similar to the process of using the first tracker in the first tracking frame sequence in S110 above, and will not be repeated here.
[0051] Similarly, step S122 may include: if no target is detected in the first detection frame or the target is not included in the first detection frame, converting the tracking state of the target from the first state to the second state.
[0052] It can be understood that when the tracking state of the target is the second state, the second tracking frame sequence after the first detection frame uses the second tracker to track the target. The second tracking frame sequence can be a tracking frame after and adjacent to the first detection frame. Figure 2 , the first detection frame may be the 30th frame, and the second tracking frame sequence may be the 31st to 39th frames.
[0053] The target is detected in the first detection frame, which may be detected within the frame of the first detection frame. For example, the detected target may be in a portion close to the center of the detection frame, or in a portion close to the edge of the detection frame. Alternatively, the detected target may be completely or partially in the area covered by the first detection frame.
[0054] For example, the second state may be represented by "PENDING".
[0055] Optionally, since the target is not detected, the second tracker can track based on the tracking frame of the previous tracking frame of the first detection frame. Figure 2-Figure 5 , if the target face0 is not detected in the 30th frame, the 31st to 39th frames can be based on the tracking frame of the 29th frame ( Figure 3 Light grey box (0) continues tracking using the second tracker.
[0056] Next, continue to combine Figure 2-Figure 5 The embodiment shown is described in detail.
[0057] From the above, we can see that Figure 4 In the 30th frame shown, face0 is not detected, so it can be Figure 1In step S122, the tracking state of face0 is converted from TRACK to PENDING.
[0058] Figure 5 The next tracking frame after the 30th frame may be, for example, any frame from the 31st frame to the 39th frame, for example, the 39th frame. Since the tracking state of face0 is PENDING, the second tracker may be used to track face0 in the 39th frame.
[0059] It can be understood that if face0 is detected in the 30th frame, Figure 1 In step S121, the tracking state of face0 is kept as TRACK. In the 39th frame, if the tracking state of face0 is TRACK, the first tracker can be used to track the target.
[0060] Among them, the first tracker can be a tracker that is greatly affected by the target detection result, such as a tracker for a specific type. Trackers for specific types of targets usually obtain targets of the specific type in image frames by detection, and then track these specific types of targets. For example, a tracker for faces can track faces in image frames, and the tracker usually does not track objects that are not detected as faces. Trackers for specific types of targets can be, for example, trackers based on deep learning models, or other types of trackers.
[0061] The second tracker can be a tracker that is less affected by the target detection result or is not affected by the target detection result, such as a tracker for a non-specific type. The tracker for a non-specific type of target is not limited to tracking a specific type of target, but can track multiple types of targets. For example, a tracker for a non-specific type of target can track both faces and vehicles. In some embodiments, the tracker for a non-specific type of target can search for content with a high degree of similarity to the content in the tracking frame based on the content to be tracked, without having to understand what the content is specifically. The tracker for a non-specific type of target can be, for example, a KCF tracker, or other types of trackers.
[0062] For example, the first tracker may also be a tracking algorithm on a CPU, or a tracking algorithm on a neural network processing chip. When the first tracker is a tracking algorithm on a neural network processing chip, the resource restriction of the target tracking on the CPU may be reduced.
[0063] Alternatively, the second tracker may also be a tracking algorithm on a CPU. An embodiment of the present application uses a fast and universal KCF tracker as the second tracker, which can be deployed on the CPU of the terminal device. This specialized KCF tracker only takes 0.4ms to track a target on the HISI-CPU.
[0064] It can be understood that the first tracker and the second tracker are different trackers, and the difference here may be the type of tracker, parameters, model structure, hardware running the tracker, etc.
[0065] When the target cannot be detected, since the first tracker is greatly affected by the detection results, if the first tracker is continued to be used for tracking, the tracking frame may have abnormal phenomena such as arbitrary jumps, which makes it difficult to continue tracking. In response to this undetectable situation, the present application does not immediately end the tracking and output the tracking result (the image of the target can be output, for example, the best quality image in the tracking information of the target), but uses the second tracker that is less affected by the detection results to continue tracking. By using the second tracker for auxiliary tracking, a connection with the target that may be re-detected later is established. On the one hand, the present application sets the tracking state and switches the tracking state according to whether the target is detected, so that different trackers can be selected for tracking according to the state of the target, making the tracking process more flexible. On the other hand, when the target cannot be detected, the present application adaptively and purposefully changes the tracker to continue tracking the target, thereby achieving a better tracking effect.
[0066] Next, taking the first tracker as a tracker based on a deep learning model as an example, a possible implementation method of the first tracker is given.
[0067] For each tracking target, a Kalman model can be maintained. Before applying the deep learning model tracker to track, the tracking box of the next tracking frame can be predicted based on the Kalman model. With the Kalman-predicted box as the center, an enlarged search image is created, and the size of the enlarged search image can be adjusted to 32x32 pixels. According to the Pnet topology, an output feature map of 11x11 pixels can be obtained. After applying softmax to the classification branch, a Figure 6 The neural network heatmap with a size of 11x11 pixels shown in FIG. 1 may be, for example, a Pnet-heatmap. The position with the highest score on the Pnet-heatmap is the result of deep learning model tracking. Figure 6 In the embodiment of , the target moves to the lower right. Optionally, for more accurate tracking, a bounding box regression can also be applied to the position with the highest score.
[0068] The present application also improves the method for determining whether a target leaves a scene corresponding to an image frame, so as to avoid the target being incorrectly tracked when the target leaves the scene, thereby improving the tracking accuracy.
[0069] One implementation method is that when it is detected that the target is located at the edge of the first detection frame and the target moves outside the edge of the first detection frame, it can be determined that the target is about to leave the scene corresponding to the image frame, and the tracking state of the target is converted from the first state to the third state. For example: if the tracking frame of the target touches the edge of the first detection frame and the Kalman speed of the target is consistent with the edge of the first detection frame, it can be determined that the target is about to leave the scene corresponding to the image frame.
[0070] As an implementation method, the first detection frame is, for example, Figure 7 In the detection frame shown, target 0 arrives Figure 7 The left border of Figure 7 If the x component of the Kalman velocity of target 0 points to the left, that is, the value of the x component is negative, it can be determined that target 0 is about to leave the scene corresponding to the image frame. Optionally, the tracking information of target 0 is updated, and the tracking state of the target is converted to a third state, which can be used to indicate that the target is about to leave (or is leaving) the scene corresponding to the image frame containing the target.
[0071] The third state can be represented by "OUTSIDE" or "PENDING_OUTSIDE", for example. For example: when the target tracked by the first tracker is about to leave the scene corresponding to the image frame containing the target, the tracking state of the target can be converted from TRACK to OUTSIDE. In addition, when the target tracked by the second tracker is about to leave the scene corresponding to the image frame containing the target, the tracking state of the target can be converted from PENDING to PENDING_OUTSIDE. It can be understood that the tracker corresponding to the third state OUTSIDE is the first tracker, and the tracker corresponding to the third state PENDING_OUTSIDE is the second tracker. That is to say, if the tracking state is converted to the third state in the first detection frame, then the tracking frame after the first detection frame can be tracked using the tracker corresponding to the third state.
[0072] Another implementation method is that after S110, when the target is detected to be at the edge of the first detection frame and the target moves toward the outside of the edge of the first detection frame, it can be determined that the target is about to leave the scene corresponding to the image frame, and the tracking state of the target can be converted from the first state to the first state and the third state. Similarly, after S122, when the detection frame after the first detection frame detects that the target is at the edge of the image and the moving speed of the target is toward the edge, it can be determined that the target is about to leave the scene corresponding to the image frame, and the tracking state of the target can be converted from the second state to the second state and the third state.
[0073] That is, the tracking state of the target may include the first state and the third state, or include the second state and the third state. The third state may be represented by "OUTSIDE", for example.
[0074] For example, for a first target in the first state TRACK, if the first target is detected at the edge of the image and moves toward the edge in the first detection frame, the tracking state of the first target can be updated to "TRACK+OUTSIDE" in the first detection frame. In addition, in the tracking frame after the first detection frame, the first tracker corresponding to the first state TRACK can be used for tracking.
[0075] For another example, for a second target in the second state PENDING, if the second target is detected at the edge of the image and moves toward the edge in the first detection frame, the tracking state of the second target can be updated to "PENDING+OUTSIDE" in the first detection frame. In addition, in the tracking frame after the first detection frame, the second tracker corresponding to the second state PENDING can be used for tracking.
[0076] Optionally, when the target is in the third state, in the next detection frame, it can be determined whether the target leaves the image frame, and if the target leaves the image frame, the tracking task is terminated. This can simplify the implementation and save computing resources.
[0077] Optionally, if a target in the previous detection frame is about to leave the scene corresponding to the image frame, and the target is not detected in the next detection frame, tracking of the target is stopped and the tracking result of the target is output.
[0078] Optionally, if the detection frame does not detect the target that is about to leave, the tracking result of the target can be output immediately without delay. This can increase the output speed of the tracked target and help enhance the user experience.
[0079] The embodiments of the present application can continuously track missed targets, but for some targets that cannot be detected for a long time, continuous tracking is not appropriate. For example, if the target has actually left the scene where the image frame is located, the subsequent detection frame will never be able to detect the target. For another example, if the image quality of the target is continuously poor, it is difficult for the second tracker to accurately track the target.
[0080] In view of the above situation, the present application records the tracking time of the second tracker when tracking the target using the second tracker. When the tracking time is greater than the first threshold, the target is stopped from being tracked, and the tracking result of the target is output. The tracking result can be obtained from the tracking information, such as the position coordinate sequence of the target, the highest quality image of the target, etc.
[0081] Optionally, the tracking duration of the second tracker can be calculated by the number of consecutive frames, and the first threshold can be a set number of consecutive frames. For example, the first threshold can be 50 frames. When 50 consecutive image frames (or for Figure 2 In the video sequence shown, no target is detected for 5 consecutive detection frames, that is, the second tracker is used for tracking during this period, then the tracking task of the target is ended and the tracking result of the target is output.
[0082] Optionally, the tracking time of the second tracker can be calculated by the number of consecutive detection frames in which the target is not detected. Exemplarily, the tracking time can also be called the tracking age of the second tracker. For example, the first threshold can be 4. When the target is not detected in 4 consecutive detection frames during the use of the second tracker, Figure 2 In the example, if the target is not detected in the 40th frame, the 50th frame, the 60th frame, and the 70th frame, it means that the tracking time is greater than the first threshold, the tracking of the target is stopped, and the tracking result of the target is output.
[0083] It should be noted that the present application does not limit the specific value of the first threshold, and it can be flexibly selected according to the type of target, the tracking scenario, the type of terminal running the second tracker, etc.
[0084] When a target enters the scene, the detection frame detects the target for the first time, and the subsequent tracking frame will track the target. This type of target can be called a newly detected target (new object).
[0085] It is difficult to avoid unexpected errors in the process of detecting targets, such as the presence of similar targets, poor shooting angles, etc. This can easily lead to the situation where the detected target is not a specific type of target, i.e. a false alarm. Figure 8The face detection result in the detection frame is displayed. The detection frame detects that face3 is a face, but in fact face3 is a handbag, not a face. At this time, face3 is a false alarm.
[0086] In view of the above problems, the embodiment provided by the present application effectively filters out false alarms through quality detection. Before performing target detection in the first detection frame, target detection is performed in the second detection frame. If there is a newly detected target in the second detection frame, the tracking state of the target is set to the fourth state. The fourth state is used to indicate that the target is a newly detected target, and the fourth state can be represented by "NEWDETECT", for example.
[0087] Optionally, when a detected new target (new detection) cannot be assigned to any existing tracking information, the target can be determined to be a newly detected target and marked as NEWDETECT. For a target in the NEWDETECT state, tracking information of the target can be generated, and target tracking can be started for the target in the NEWDETECT state.
[0088] In addition to the above four states, other types of tracking states can be set according to actual needs. For example, when the tracked target overlaps with other targets, the tracking state of the target can be marked as the fifth state. For example, the fifth state can be represented by "OCCLUDED".
[0089] The target quality is evaluated to obtain a quality score. If the quality score is lower than the second threshold, the tracking information of the target can be deleted immediately. That is, if the quality score of the newly detected target is lower than the second threshold, the target is considered to be a false alarm and is not tracked. Alternatively, if the quality score of the newly detected target is higher than or equal to the second threshold, the tracking state in the tracking information of the target is converted from the fourth state to the first state, and the first tracker can be used to track the target in subsequent tracking frames.
[0090] It should be noted that the present application does not impose any restrictions on the specific value of the second threshold, which can be flexibly selected according to factors such as the tracking scenario, tracking type or actual tracking requirements.
[0091] against Figure 8 The face tracking shown in the figure is described in detail as an example. Figure 8 The second detection frame shown has a newly detected target. Figure 8 Take face0 as an example, face0 is detected in the second detection frame ( Figure 8 In the example, the white box 0 is used to mark the tracking state of face0 as the fourth state, for example, NEWDETECT. For face0, new tracking information can be established. Figure 8In the second detection frame shown, the image quality of face0 can be detected to obtain a quality score. If the face0 score is greater than or equal to the second threshold, the tracking state of face0 is converted from the fourth state to the first state, for example, from NEWDETECT to TRACK, and in the next tracking frame, face0 can be tracked using the first tracker.
[0092] Or, Figure 8 Taking face3 as an example, the process of determining whether face3 is a newly detected target is basically similar to that of face0. However, since the image quality score of face3 is less than the second threshold, face3 can be determined as a false alarm, and the tracking information of face3 can be deleted. In the next tracking frame, face3 will not be tracked.
[0093] In the process of tracking the target, tracking information of the target may be maintained, which may include the tracking state of the target, the position of the target in the image frame, and image information of the target image with the highest quality during the tracking process of the target.
[0094] The tracking state in the tracking information may include at least one of a first state, a second state, a third state, or a fourth state. These four states have been described above and will not be described again here.
[0095] The position of the target in the image frame in the tracking information is used to indicate the position of the target in the current image, and can also be used to estimate the position of the target in the next frame. In the tracking frame, the position of the target in the image frame can be the position of the tracking frame of the target in the image frame. In the detection frame, the position of the target in the image frame can be the position of the detection frame of the target in the image frame.
[0096] In one implementation, when the target tracking task is completed, the image information of the highest quality image of the target is provided to the user, and the image information of the image is output as the tracking result, so as to provide the user with the clearest image of the target at the best angle. For example, the image information of the target image with the highest quality may be at least one of the following information: the quality of the target image, the spatial position of the target image in the image frame to which the target image belongs, and the temporal position of the target image in the video sequence.
[0097] For example, the quality of the target image can be obtained through the quality score during the tracking process. For example, in face tracking, the quality assessment model can be used to obtain information about the image quality of the target, such as yaw, pitch, occlusion, blur, etc., and a total score is calculated based on this information. This total score is the image quality score, which can be used as a criterion for judging image quality. For example, the image with the highest target quality score can be considered to have a higher image quality. Among them, the quality assessment model can be, for example, a multi-task face quality assessment model based on a deep learning model.
[0098] For example, the spatial position of the target image with the highest quality may include the position coordinates of the target image in the image frame, and may also include the size of the target image.
[0099] For example, the time position of the target image with the highest quality may be the frame number of the image in the entire video sequence.
[0100] It should be noted that the target in the present application may also be referred to as an object, such as a newly detected target may also be referred to as a newly detected object.
[0101] In addition, it is understandable that the target tracking method in the present application can be used for target capture, such as face capture. The specific tracking result obtained by the above-mentioned target tracking method is the result of target capture. If the target tracking is possible, the highest quality image in the video will be used as the output of the capture, ensuring that the capture result is more accurate and providing higher quality images for subsequent further applications.
[0102] In addition, the present application also provides a target tracking system, such as Fig. 9 As shown, it includes a memory, a processor and a computer program stored in the memory. And when the processor executes the computer program, the above combination can be achieved Figure 1 The steps of the method.
[0103] The memory may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM), etc.
[0104] The processor can adopt a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC) or one or more integrated circuits to execute relevant programs to implement the method of the embodiment of the present application.
[0105] The processor may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the method of the present application may be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above-mentioned processor may also be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and combines its hardware to complete the functions required to be performed by the units included in the system of the embodiment of the present application, or executes the method of the method embodiment of the present application.
[0106] Optionally, the system may further include a communication interface and a bus. The communication interface uses a transceiver such as, but not limited to, a transceiver to achieve communication with other devices or networks. For example, multiple images can be acquired from an image acquisition device through a communication interface, image processing results can be transmitted to other external devices through a communication interface, and so on. The bus may include a path for transmitting information between various components of the device (e.g., memory, processor, communication interface).
[0107] Understandable, Fig. 9 The target tracking system in the embodiment may be a computer system, a computer device, etc. In some embodiments, the target tracking system may be a mobile terminal, such as a handheld mobile terminal, for example, a mobile phone.
[0108] In addition, an embodiment of the present invention further provides a computer storage medium on which a computer program is stored. When the computer program is executed by a computer or a processor, the aforementioned combination Figure 1 The steps of the method described. For example, the computer storage medium is a computer readable storage medium.
[0109] In one embodiment, the computer program instructions, when executed by a computer or processor, cause the computer or processor to perform the following steps: in the process of tracking a target in a first state using a first tracker, performing target detection in a first detection frame; if the target is detected in the first detection frame, maintaining the tracking state of the target in the first state; if the target is not detected in the first detection frame, converting the tracking state of the target from the first state to the second state; in the next tracking frame of the first detection frame, selecting a tracker corresponding to the tracking state of the target to continue tracking the target, wherein the first state corresponds to the first tracker and the second state corresponds to the second tracker.
[0110] Computer storage media may include, for example, a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. A computer-readable storage medium may be any combination of one or more computer-readable storage media.
[0111] In addition, an embodiment of the present invention further provides a computer program product, which includes a computer program or an instruction. When the computer program or the instruction is executed by a computer or a processor, the above-mentioned combination is executed. Figure 1 The steps of the method are described.
[0112] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as required by the appended claims.
[0113] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
Claims
1. A target tracking method, characterized in that: include: In the process of tracking the target in the first state using the first tracker, performing target detection in the first detection frame; If the target is detected in the first detection frame, maintaining the tracking state of the target in the first state; If the target is not detected in the first detection frame, converting the tracking state of the target from the first state to a second state; In a next tracking frame of the first detection frame, a tracker corresponding to the tracking state of the target is selected to continue tracking the target, wherein the first state corresponds to a first tracker and the second state corresponds to a second tracker.
2. The method according to claim 1, characterized in that The method further comprises: If the target is detected to be located at the edge of the first detection frame and the target moves toward the outside of the edge of the first detection frame, the tracking state of the target is converted from the first state to a third state, and the third state is used to indicate that the target is about to leave the scene corresponding to the image frame containing the target.
3. The method according to claim 2, characterized in that The method further comprises: If the tracking state of the target is the third state and the target is not detected in the next detection frame of the first detection frame, tracking of the target is stopped and a tracking result of the target is output.
4. The method according to claim 1, characterized in that: Also includes: If the tracking state of the target is converted to the second state, record the duration of the tracking state of the target being the second state; When the duration is greater than a first threshold, tracking of the target is stopped, and a tracking result of the target is output.
5. The method according to claim 1, characterized in that Before performing target detection in the first detection frame, the method further includes: Performing target detection in a second detection frame, wherein the target is a target newly detected in the second detection frame; Setting the tracking state of the target to a fourth state, wherein the fourth state is used to indicate that the target is a newly detected target; detecting the image quality of the target in the fourth state; When the image quality of the target is less than a second threshold, not tracking the target; When the image quality of the target is greater than the second threshold, the tracking state of the target is converted from the fourth state to the first state, and the first tracker is used to track the target.
6. The method according to claim 1, characterized in that The method further comprises: In the process of tracking the target, maintaining tracking information of the target, the tracking information including: the tracking state of the target, the position of the target in the image frame, and image information of the target image with the highest quality in the process of tracking the target; The image information of the target image includes at least one of the following information: the quality of the target image, the spatial position of the target image in the image frame to which the target image belongs, and the temporal position of the target image in the video sequence.
7. The method according to any one of claims 1 to 6, characterized in that The target is a target of a specific type, the first tracker is a tracker for the target of the specific type, and the second tracker is a tracker for a target of a non-specific type.
8. The method according to any one of claims 1 to 6, characterized in that The first tracker is a tracker based on a deep learning model, and the second tracker is a kernel correlation filter tracker.
9. A target tracking system, comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of any one of the methods of claims 1 to 8.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.
