Target tracking method and device, medium, electronic equipment and product
Through the synchronous processing of visual object detection and tracking data, combined with Kalman filtering algorithm and motion consistency distance index, the problem of time-consuming and affecting frame rate of the detection module is solved, and the efficiency and stability of visual multi-object tracking is improved, which is suitable for practical purposes Multithreaded scenarios in the application.
Patent Information
- Application Number
- CN202310627500.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-30
- Publication Date
- 2024-12-03
AI Technical Summary
In the existing visual multi-objective tracking algorithm, the time-consuming detection module will affect the frame rate of the system processing, which cannot be ignored in actual applications, resulting in poor target tracking results.
By synchronizing the input data of visual object detection and visual object tracking, multi-threaded parallelization is achieved using decoded frame queues to avoid the impact of time-consuming detection module on frame rate, Kalman filtering algorithm or single-object tracking algorithm is used for prediction and association. Increase the distance indicator of motion consistency to improve the matching success rate.
It improves the target tracking efficiency, reduces the impact on the frame rate during system processing, makes visual multi-object tracking more suitable for visual detection systems in practical applications, and improves the accuracy of target trajectory prediction and the stability of tracking algorithms.
Smart Images

Figure CN119068377A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of visual multi-target tracking, and in particular to a target tracking method, device, medium, electronic equipment and product. Background Art
[0002] This section is intended to provide a background or context to the embodiments recited in the claims. No admission is made that the description herein is prior art by inclusion in this section.
[0003] Visual multi-target tracking is an important task in computer vision, and it has a wide range of applications in autonomous driving, intelligent video surveillance, and behavior analysis. Video multi-target tracking algorithms detect and estimate different individuals of multiple types of targets such as vehicles, pedestrians, and animals in video streams, and match targets between adjacent frames through the correlation results of detection values and estimation values to form information such as target IDs and trajectories for application. According to the target initialization method, visual multi-target tracking algorithms can be divided into two categories: a) DBT (Detection-Based Tracking) and b) DFT (Detection-Free Tracking). In the former, target initialization is completed by trained model detection, while in the latter, it needs to be completed manually in the first frame. DBT is more suitable for practical applications, so this type of method is the research focus in the field of visual multi-target tracking. In the study of DBT methods, it is often assumed that the video frame rate is high enough so that the target motion can be approximated as linear motion, and the detection module time consumption is ignored. In practical applications, the detection module time consumption that cannot be ignored will also affect the frame rate during system processing. Summary of the invention
[0004] In view of the situation where the time consumption of the detection module will also affect the frame rate during system processing, the present invention provides a target tracking method, device, medium, electronic device and product, which can improve the target tracking efficiency and effectively reduce the impact on the frame rate during system processing by synchronizing the input data of visual target detection and visual target tracking.
[0005] In a first aspect, an embodiment of the present invention provides a target tracking method, comprising:
[0006] Obtain the video stream in real time and decode the video stream, and store the decoded video frame data in the decoded frame queue;
[0007] Perform synchronization processing based on the video frame data in the decoded frame queue to synchronize input data of visual object detection and visual object tracking;
[0008] Perform visual target detection and obtain visual target detection results;
[0009] Visual target tracking is performed based on the visual target tracking input data and the visual target detection result.
[0010] In some implementations, the synchronous processing based on the video frame data in the decoded frame queue to synchronize input data for visual target detection and visual target tracking includes:
[0011] In the current visual target detection cycle, if the number of video frames covered in the previous visual target detection cycle is greater than 0, read the head image in the decoded frame queue and output it as input data for visual target tracking, and reduce the number of video frames covered in the previous visual target detection cycle by 1;
[0012] Until the current visual target detection cycle is completed, the first number of images at the head of the decoded frame queue are output as input data for visual target tracking, wherein the first number is the number of video frames covered in the previous visual target detection cycle, and the number of video frames covered in the current visual target detection cycle is used as the number of video frames covered in the previous visual target detection cycle to enter the next current visual target detection cycle.
[0013] In some implementations, the synchronous processing based on the video frame data in the decoded frame queue to synchronize input data for visual target detection and visual target tracking further includes:
[0014] Read the last image from the decoded frame queue;
[0015] Determine whether it is in the visual target detection cycle:
[0016] If so, then executing the step of reading the head image in the decoded frame queue and outputting it as input data for visual target tracking when the number of video frames covered in the previous visual target detection cycle is greater than 0, and subtracting 1 from the number of video frames covered in the previous visual target detection cycle;
[0017] Otherwise, after the current visual target detection cycle starts, the tail image is output as the input data for visual target detection, and the number of video frames covered in the current visual target detection cycle is increased by 1, and the process returns to execute the steps of reading the tail image from the decoded frame queue and determining whether it is in the visual target detection cycle.
[0018] In some implementations, the synchronous processing based on the video frame data in the decoded frame queue to synchronize input data for visual target detection and visual target tracking further includes:
[0019] Determine whether the current visual target detection cycle is completed:
[0020] If completed, the first number of images at the head of the decoded frame queue is output as input data for visual target tracking, wherein the first number is the number of video frames covered in the previous visual target detection cycle, and the number of video frames covered in the current visual target detection cycle is used as the number of video frames covered in the new previous visual target detection cycle;
[0021] If not completed, the number of video frames covered in the current visual target detection cycle is increased by 1.
[0022] In some implementations, performing visual target tracking based on the visual target tracking input data and the visual target detection result includes:
[0023] Determine whether the visual target detection result is currently obtained;
[0024] If the visual target detection result is not obtained, a prediction algorithm is used to predict the motion feature state of the visual target, and the step of returning to execute synchronization processing based on the video frame data in the decoded frame queue to synchronize the input data of the visual target detection and the visual target tracking is performed.
[0025] In some implementations, the performing visual target tracking based on the visual target tracking input data and the visual target detection result further includes:
[0026] If the visual target detection result is obtained, a preset appearance feature extraction model is used to extract the target appearance feature information of each visual target, a prediction algorithm is used to predict the state of the visual target, a cost is calculated based on the target appearance feature information and the state prediction information, and data association is performed between the existing visual target and the visual target in the current visual target detection result based on the cost; for the existing visual target and the visual target in the current visual target detection result that have not achieved data association, a secondary association is performed using the intersection and union ratio to obtain an association result; based on the association result, the state of the existing visual target is updated.
[0027] In some implementations, the prediction algorithm includes a Kalman filter algorithm or a single target tracking algorithm.
[0028] In some implementations, the cost calculation based on the target appearance feature information and the state prediction information includes:
[0029] The following calculation formula is used to calculate the cost based on the target appearance feature information and state prediction information:
[0030] cost (i,j) =λ1·dist1(i,j)+λ2·dist2(i,j)+λ3·dist3(i,j)
[0031] Among them, dist1 represents the cosine distance or Euclidean distance of the appearance features between the visual target in the current video frame and the existing visual target in the visual target detection result, dist2 represents the Mahalanobis distance between the visual target in the current video frame and the existing visual target in the visual target detection result, dist3 represents the distance of the consistency of the visual target motion vector between the visual target in the current video frame and the existing visual target in the visual target detection result, i and j represent the existing visual target and the visual target in the current video frame in the visual target detection result, respectively, λ1 represents the weight of dist1, λ2 represents the weight of dist2, and λ3 represents the weight of dist3.
[0032] In a second aspect, an embodiment of the present invention provides a target tracking device, comprising:
[0033] The video acquisition and decoding module is used to obtain and decode the video stream in real time, and store the decoded video frame data in the decoded frame queue;
[0034] A synchronization module, for performing synchronization processing based on the video frame data in the decoded frame queue to synchronize input data of visual target detection and visual target tracking;
[0035] A detection module is used to perform visual target detection and obtain visual target detection results;
[0036] The tracking module is used to perform visual target tracking based on the input data of the visual target tracking and the visual target detection result.
[0037] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by at least one processor, the method described in the first aspect is implemented.
[0038] In a fourth aspect, an embodiment of the present invention provides an electronic device, comprising a memory and at least one processor, wherein the memory stores a computer program, and when the computer program is executed by the at least one processor, the method described in the first aspect is implemented.
[0039] In some implementations, the processor includes an embedded processor.
[0040] In a fifth aspect, an embodiment of the present invention provides a computer program product, which executes the method according to the first aspect when running on a processor.
[0041] Compared with the prior art, the present invention can at least bring the following beneficial effects:
[0042] The present invention takes into account the time consumption of the detection module in actual application development and the association between various functional modules in the system, completes the synchronization of input video frame data for detection and tracking through data synchronization processing, and uses different threads to complete the detection and tracking functions in parallel, so that visual multi-target tracking is adapted to a multi-threaded visual multi-target tracking framework, thereby making visual multi-target tracking more suitable for visual detection systems in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and should not be regarded as limiting the scope.
[0044] Figure 1 is a flow chart of a target tracking method provided by an embodiment of the present invention;
[0045] Figure 2 is a schematic diagram of a visual target detection cycle provided by an embodiment of the present invention;
[0046] Figure 3 The present invention provides a block diagram of a target tracking device. DETAILED DESCRIPTION
[0047] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.
[0048] In the application development of embedded processors (such as GPU), the device processing performance is relatively limited, which limits the system processing frame rate. At the same time, the time consumption of the detection module, which cannot be ignored in actual applications, will also affect the frame rate of the system processing. In addition, there are often situations that do not meet the linear motion requirements, such as occlusion, detection loss, and non-linear motion of the target. All of the above factors will lead to poor target tracking results.
[0049] In related technologies, the target tracking algorithm execution process is all executed in a serial manner (such as DeepSORT). However, in practical applications such as visual inspection systems, functional modules such as video acquisition and decoding, visual target detection, visual multi-target tracking, and result output often use parallel processing. In addition, in the target matching stage, in addition to using appearance features, the intersection-and-union ratio is often used for nearest neighbor matching on spatial features, without considering that in application scenarios with relatively limited processing performance such as embedded devices, the target displacement between adjacent frames may be large, and the nearest neighbor matching cannot be fully applied, which easily leads to poor tracking effects.
[0050] Embodiments of the present invention provide a target tracking method, device, medium, electronic device and product, which improve the target tracking efficiency and effectively reduce the impact on the frame rate during system processing by synchronizing the input data of visual target detection and visual target tracking.
[0051] Embodiment 1
[0052] Figure 1 Provide a target tracking method flow chart, such as Figure 1 As shown, the target tracking method provided in this embodiment at least includes steps S101 to S104:
[0053] Step S101: acquiring a video stream in real time and decoding the video stream, and storing the decoded video frame data in a decoded frame queue.
[0054] Step S102: Perform synchronization processing based on the video frame data in the decoded frame queue to synchronize input data for visual target detection and visual target tracking.
[0055] In some implementations, performing synchronization processing based on video frame data in a decoded frame queue to synchronize input data for visual object detection and visual object tracking includes:
[0056] In the current visual target detection cycle, if the number of video frames covered in the previous visual target detection cycle is greater than 0, read the head image in the decoded frame queue and output it as input data for visual target tracking, and reduce the number of video frames covered in the previous visual target detection cycle by 1;
[0057] Until the current visual target detection cycle is completed, the first number of images at the head of the decoded frame queue are output as input data for visual target tracking, so as to complete target tracking in combination with the visual target detection results. The first number is the number of video frames covered in the previous visual target detection cycle at this time, and the number of video frames covered in the current visual target detection cycle is used as the number of video frames covered in the previous visual target detection cycle to enter the next current visual target detection cycle.
[0058] The visual target detection cycle refers to the time required to complete a visual target detection. The number of video frames covered in the visual target detection cycle refers to the number of video frames corresponding to the video stream obtained in the visual target detection cycle. The image at the end of the team is the latest video frame, and the image at the head of the team is the historical video frame.
[0059] In this embodiment, real-time tracking is achieved by continuously inputting the head image in the decoded frame queue into the tracking module during the visual target detection cycle until the current visual target detection cycle is completed, and then the historical video frame of the first data is input into the tracking module. This fully considers the time consumption of visual target detection under multi-threaded parallel conditions such as video acquisition decoding, target detection, and target tracking, and realizes the synchronization of input data of visual target detection and visual target tracking, thus avoiding the time consumption problem of the target detection module that affects the frame rate during system processing and is ignored by target detection and tracking in a serial mode.
[0060] In some cases, visual target detection may not have started yet, that is, the thread of visual target detection does not exist yet, such as the period between two visual detection cycles. Therefore, before performing synchronization within the current visual target detection cycle, it is necessary to determine whether it is within the visual target detection cycle to determine whether the thread of visual target detection exists. At this time, synchronization processing is performed based on the video frame data in the decoded frame queue to synchronize the input data of visual target detection and visual target tracking, and may also include:
[0061] Read the last image from the decoded frame queue;
[0062] Determine whether it is in the visual target detection cycle:
[0063] If so, then executing the step of reading the head image in the decoded frame queue and outputting it as input data for visual target tracking when the number of video frames covered in the previous visual target detection cycle is greater than 0, and subtracting 1 from the number of video frames covered in the previous visual target detection cycle;
[0064] Otherwise, after the current visual target detection cycle starts, the tail image is output as the input data for visual target detection, and the number of video frames covered in the current visual target detection cycle is increased by 1, and the process returns to execute the aforementioned steps of reading the tail image from the decoded frame queue and determining whether it is in the visual target detection cycle.
[0065] In some implementations, performing synchronization processing based on video frame data in a decoded frame queue to synchronize input data for visual object detection and visual object tracking further includes:
[0066] Determine whether the current visual target detection cycle is completed:
[0067] If completed, the first number of images at the head of the decoded frame queue is output as input data for visual target tracking, wherein the first number is the number of video frames covered in the previous visual target detection cycle, and the number of video frames covered in the current visual target detection cycle is used as the number of video frames covered in the new previous visual target detection cycle;
[0068] If not completed, the number of video frames covered in the current visual target detection cycle is increased by 1.
[0069] The following is a specific example of performing synchronization processing based on the video frame data in the decoded frame queue to synchronize the input data of visual target detection and visual target tracking, including the following process:
[0070] Let preTrackFrames be the number of video frames covered in the previous visual target detection cycle, and curTrackFrames be the number of video frames covered in the current visual target detection cycle. Both are initialized to 0. The schematic diagram of the visual target detection cycle is shown in Figure 2 shown.
[0071] Step a, reading the tail image from the decoded frame queue;
[0072] Step b: Determine whether the thread for visual target detection exists:
[0073] If it does not exist, after the detection module creates a visual target detection thread, the tail image is output to the detection module and the process goes to step f.
[0074] If it exists (visual target detection is in progress), go to step c;
[0075] Step c: If preTrackFrames>0, read the head image in the decoded frame queue and input it into the tracking module, dequeue the head image, and decrement preTrackFrames by 1;
[0076] Step d, determining whether the thread of visual target detection is completed: if completed, executing step e; if the thread of visual target detection is not completed, executing step f;
[0077] Step e: output the preTrackFrames images at the head of the decoded frame queue to the tracking module, complete the visual target tracking in combination with the visual target detection result, and record the number of video frames covered in the current visual target detection cycle as curTrackFrames, and set preTrackFrames = curTrackFrames;
[0078] Step f, curTrackFrames increases by 1, and then goes to step a.
[0079] In the multi-threaded processing framework, detection and tracking processing are completed using different threads. Through the synchronization processing algorithm of this embodiment, the synchronization of video frame data input to the detection module and the tracking module is completed, so that visual multi-target tracking can be adapted to multi-threaded visual detection and tracking scenarios.
[0080] Step S103: perform visual target detection to obtain a visual target detection result.
[0081] In this embodiment, visual target detection in the current video frame can be completed through machine learning or deep learning algorithms, and the visual target detection results can be output to the tracking module.
[0082] Step S104: performing visual target tracking based on the visual target tracking input data and the visual target detection result.
[0083] In some implementations, performing visual target tracking based on the visual target tracking input data and the visual target detection result may further include:
[0084] Step S104a, determining whether a visual target detection result is currently obtained;
[0085] Step S104b: If the visual target detection result is not obtained, a prediction algorithm is used to predict the motion feature state of the visual target, and the process returns to step S102 to perform synchronization processing based on the video frame data in the decoded frame queue to synchronize the input data of the visual target detection and visual target tracking.
[0086] The prediction algorithm can be expressed as follows:
[0087]
[0088] in, represents the motion characteristic state of the visual target at time t, f pred (image) represents the prediction function, and image represents the current video frame.
[0089] In some implementations, the prediction algorithm includes a Kalman filter algorithm or a single target tracking algorithm.
[0090] When the computing power of the device is limited, the time between two adjacent detections is relatively long, the target displacement may be large, and the visual target detection result does not exist for a long time between two visual target detection cycles. In this embodiment, a multi-step Kalman filter algorithm or a multi-step single target tracking algorithm is used to predict the video frames between the two visual target detection cycles, which can improve the accuracy of the target trajectory prediction value and thereby improve the stability of the system tracking algorithm.
[0091] Step S104c: If the visual target detection result is obtained, the preset appearance feature extraction model is used to extract the target appearance feature information of each visual target, the prediction algorithm is used to predict the state of the visual target, the cost is calculated based on the target appearance feature information and the state prediction information, and the data between the existing visual target and the visual target in the current visual target detection result is associated based on the cost; for the existing visual target that has not achieved data association and the visual target in the current visual target detection result, the intersection and union ratio is used to perform secondary association to obtain the association result; according to the association result, the state of the existing visual target is updated. In practical applications, the state of the visual target may include the target ID, the appearance feature set of each target, etc.
[0092] In some implementations, the following calculation formula may be used to calculate the cost based on the target appearance feature information and the state prediction information:
[0093] cost (i,j) =λ1·dist1(i,j)+λ2·dist2(i,j)+λ3·dist3(i,j)
[0094] dist3(i,j)=α·f(m Dir (i,j))+(1-α)·f(m Dist (i,j) / s)
[0095] m Dir (i,j)=1-mv j ·mv i / ||mv j |·||mv i ||
[0096] m Dist (i,j)=||mv j -mv i ||
[0097] Wherein, dist1 represents the cosine distance or Euclidean distance of the appearance features between the visual target in the current video frame and the existing visual target in the visual target detection result, dist2 represents the Mahalanobis distance between the visual target in the current video frame and the existing visual target in the visual target detection result, dist3 represents the distance of consistency of the visual target motion vector between the visual target in the current video frame and the existing visual target in the visual target detection result, the motion vector includes angle and / or spatial displacement, i, j represent the existing visual target and the visual target in the current video frame in the visual target detection result, respectively, λ1 represents the weight of dist1, λ2 represents the weight of dist2, and λ3 represents the weight of dist3. f is a sigmoid function, mv is a motion vector, s is a motion vector distance term scaling factor, α is a weight factor, s and α can be set according to actual conditions, and no specific numerical value is limited in this embodiment.
[0098] In conventional DBT algorithms, the nearest neighbor concept is used in the matching stage. However, when the computing power of the device is limited, the time between two adjacent detections is relatively long, and the target displacement may be large. At this time, the target matching method that only uses the nearest neighbor concept cannot completely solve the association between targets with different displacement sizes. This embodiment adds a consistency distance index item of target motion to constrain the target matching stage, which can improve the matching success rate of targets with large displacement between adjacent frames.
[0099] In some cases, the method of this embodiment may further include:
[0100] Step S105: The tracked visual target is regarded as an obstacle, and the obstacle information is output in the form of an obstacle information list.
[0101] In practice, steps S101 to S105 are not serial but different parallel threads, which can be applied to multi-threaded parallel processing framework scenarios. Through the solution of this embodiment, data synchronization of the two threads of visual target detection and visual target tracking can be achieved, the target tracking efficiency can be improved, and the impact on the frame rate during system processing can be effectively reduced.
[0102] Embodiment 2
[0103] like Figure 3 As shown, this embodiment provides a target tracking device, including:
[0104] The video acquisition and decoding module is used to obtain and decode the video stream in real time, and store the decoded video frame data in the decoded frame queue;
[0105] A synchronization module, for performing synchronization processing based on the video frame data in the decoded frame queue to synchronize input data of visual target detection and visual target tracking;
[0106] A detection module is used to perform visual target detection and obtain visual target detection results;
[0107] The tracking module is used to perform visual target tracking based on the input data of the visual target tracking and the visual target detection result.
[0108] In some implementations, the synchronization module, when performing synchronization processing based on the video frame data in the decoded frame queue to synchronize input data for visual target detection and visual target tracking, includes:
[0109] In the current visual target detection cycle, if the number of video frames covered in the previous visual target detection cycle is greater than 0, read the head image in the decoded frame queue and output it as input data for visual target tracking, and reduce the number of video frames covered in the previous visual target detection cycle by 1;
[0110] Until the current visual target detection cycle is completed, the first number of images at the head of the decoded frame queue are output as input data for visual target tracking, so as to complete target tracking in combination with the visual target detection results. The first number is the number of video frames covered in the previous visual target detection cycle at this time, and the number of video frames covered in the current visual target detection cycle is used as the number of video frames covered in the previous visual target detection cycle to enter the next current visual target detection cycle.
[0111] The visual target detection cycle refers to the time required to complete a visual target detection. The number of video frames covered in the visual target detection cycle refers to the number of video frames corresponding to the video stream obtained in the visual target detection cycle. The image at the end of the team is the latest video frame, and the image at the head of the team is the historical video frame.
[0112] In this embodiment, real-time tracking is achieved by continuously inputting the head image in the decoded frame queue into the tracking module during the visual target detection cycle until the current visual target detection cycle is completed, and then the historical video frame of the first data is input into the tracking module. This fully considers the time consumption of visual target detection under multi-threaded parallel conditions such as video acquisition decoding, target detection, and target tracking, and realizes the synchronization of input data of visual target detection and visual target tracking, thus avoiding the time consumption problem of the target detection module that affects the frame rate during system processing and is ignored by target detection and tracking in a serial mode.
[0113] In some cases, the visual target detection thread does not exist yet, such as the period between two visual detection cycles. Therefore, before performing synchronization within the current visual target detection cycle, it is necessary to determine whether it is within the visual target detection cycle to determine whether the visual target detection thread exists. At this time, synchronization processing is performed based on the video frame data in the decoded frame queue to synchronize the input data of the visual target detection and visual target tracking, and may also include:
[0114] Read the last image from the decoded frame queue;
[0115] Determine whether it is in the visual target detection cycle:
[0116] If so, then executing the step of reading the head image in the decoded frame queue and outputting it as input data for visual target tracking when the number of video frames covered in the previous visual target detection cycle is greater than 0, and subtracting 1 from the number of video frames covered in the previous visual target detection cycle;
[0117] Otherwise, after the current visual target detection cycle starts, the tail image is output as the input data for visual target detection, and the number of video frames covered in the current visual target detection cycle is increased by 1, and the process returns to execute the aforementioned steps of reading the tail image from the decoded frame queue and determining whether it is in the visual target detection cycle.
[0118] In some implementations, performing synchronization processing based on video frame data in a decoded frame queue to synchronize input data for visual object detection and visual object tracking further includes:
[0119] Determine whether the current visual target detection cycle is completed:
[0120] If completed, the first number of images at the head of the decoded frame queue is output as input data for visual target tracking, wherein the first number is the number of video frames covered in the previous visual target detection cycle, and the number of video frames covered in the current visual target detection cycle is used as the number of video frames covered in the new previous visual target detection cycle;
[0121] If not completed, the number of video frames covered in the current visual target detection cycle is increased by 1.
[0122] In the multi-threaded processing framework, detection and tracking processing are completed using different threads. Through the synchronization processing algorithm of this embodiment, the synchronization of video frame data input to the detection module and the tracking module is completed, so that visual multi-target tracking can be adapted to multi-threaded visual detection and tracking scenarios.
[0123] In the detection module, the visual target detection in the current video frame can be completed through machine learning or deep learning algorithms, and the visual target detection results can be output to the tracking module.
[0124] In some implementations, performing visual target tracking based on the visual target tracking input data and the visual target detection result may further include:
[0125] Determine whether the visual target detection result is currently obtained;
[0126] If the visual target detection result is not obtained, a prediction algorithm is used to predict the motion feature state of the visual target, and the synchronization processing based on the video frame data in the decoded frame queue is returned to synchronize the input data of the visual target detection and visual target tracking.
[0127] In some implementations, the prediction algorithm includes a Kalman filter algorithm or a single target tracking algorithm.
[0128] When the computing power of the device is limited, the time between two adjacent detections is relatively long, the target displacement may be large, and the visual target detection result does not exist for a long time between two visual target detection cycles. In this embodiment, a multi-step Kalman filter algorithm or a multi-step single target tracking algorithm is used to predict the video frames between the two visual target detection cycles, which can improve the accuracy of the target trajectory prediction value and thereby improve the stability of the system tracking algorithm.
[0129] If the visual target detection result is obtained, the preset appearance feature extraction model is used to extract the target appearance feature information of each visual target, the prediction algorithm is used to predict the state of the visual target, the cost is calculated based on the target appearance feature information and the state prediction information, and the data association between the existing visual target and the visual target in the current visual target detection result is performed based on the cost; for the existing visual target that has not achieved data association and the visual target in the current visual target detection result, the intersection and union ratio is used to perform secondary association to obtain the association result; according to the association result, the state of the existing visual target is updated. In practical applications, the state of the visual target may include the target ID, the appearance feature set of each target, etc.
[0130] In some implementations, the following calculation formula may be used to calculate the cost based on the target appearance feature information and the state prediction information:
[0131] cost (i,j) =λ1·dist1(i,j)+λ2·dist2(i,j)+λ3·dist3(i,j)
[0132] Among them, dist1 represents the cosine distance or Euclidean distance of the appearance features between the visual target in the current video frame and the existing visual target in the visual target detection result, dist2 represents the Mahalanobis distance between the visual target in the current video frame and the existing visual target in the visual target detection result, dist3 represents the distance of the consistency of the visual target motion vector between the visual target in the current video frame and the existing visual target in the visual target detection result, and the motion vector includes angle and / or spatial displacement, i and j represent the existing visual target and the visual target in the current video frame in the visual target detection result, respectively, λ1 represents the weight of dist1, λ2 represents the weight of dist2, and λ3 represents the weight of dist3.
[0133] When the computing power of the device is limited, the time between two adjacent detections is relatively long, and the target displacement may be large. At this time, the target matching method that only uses the nearest neighbor idea cannot completely solve the association between targets with different displacement sizes. This embodiment adds a consistency distance index item of target motion to constrain the target matching stage, which can improve the matching success rate of targets with large displacement between adjacent frames.
[0134] In some cases, this embodiment may further include an output control module that regards the tracked visual target as an obstacle and outputs the obstacle information in the form of an obstacle information list.
[0135] In practice, steps S101 to S105 are not directly executed sequentially, but different parallel threads, which can be applied to multi-threaded parallel processing framework scenarios. Through the solution of this embodiment, data synchronization of the two threads of visual target detection and visual target tracking can be achieved, the target tracking efficiency can be improved, and the impact on the frame rate during system processing can be effectively reduced.
[0136] It should be understood that when the tracking module is not available, the visual target detection results can be directly output to the output control module.
[0137] Embodiment 3
[0138] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by at least one processor, the method of the first embodiment is implemented.
[0139] Among them, the computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, disk or optical disk.
[0140] The processor can be an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller unit (MCU), a microprocessor or other electronic components.
[0141] Embodiment 4
[0142] This embodiment provides an electronic device, including a memory and at least one processor. The memory stores a computer program, and when the computer program is executed by the at least one processor, the method of the first embodiment is implemented.
[0143] In some implementations, the processor includes an embedded processor, such as a graphics processing unit (GPU).
[0144] Embodiment 5
[0145] This embodiment provides a computer program product. When the computer program product is run on a processor, the method of the first embodiment is executed.
[0146] In practical applications, the computer program product can be implemented and run in electronic devices.
[0147] In the several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed apparatus and method can also be implemented in other ways. The apparatus and method embodiments described above are merely illustrative.
[0148] It should be noted that, in this article, the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. The terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a..." does not exclude the presence of other identical elements in the process, method, article or device including the element.
[0149] Although the embodiments disclosed in the present invention are as above, the above contents are only embodiments adopted for facilitating the understanding of the present invention and are not intended to limit the present invention. Any technician in the technical field to which the present invention belongs can make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in the present invention, but the patent protection scope of the present invention shall still be subject to the scope defined in the attached claims.
Claims
1. A target tracking method, characterized in that: include: Obtain the video stream in real time and decode the video stream, and store the decoded video frame data in the decoded frame queue; Perform synchronization processing based on the video frame data in the decoded frame queue to synchronize input data of visual object detection and visual object tracking; Perform visual target detection and obtain visual target detection results; Visual target tracking is performed based on the visual target tracking input data and the visual target detection result.
2. The target tracking method according to claim 1, characterized in that: The synchronous processing based on the video frame data in the decoded frame queue to synchronize the input data of the visual target detection and the visual target tracking includes: In the current visual target detection cycle, if the number of video frames covered in the previous visual target detection cycle is greater than 0, read the head image in the decoded frame queue and output it as input data for visual target tracking, and reduce the number of video frames covered in the previous visual target detection cycle by 1; Until the current visual target detection cycle is completed, the first number of images at the head of the decoded frame queue are output as input data for visual target tracking, wherein the first number is the number of video frames covered in the previous visual target detection cycle, and the number of video frames covered in the current visual target detection cycle is used as the number of video frames covered in the previous visual target detection cycle to enter the next current visual target detection cycle.
3. The target tracking method according to claim 2, characterized in that: The synchronous processing based on the video frame data in the decoded frame queue to synchronize the input data of the visual target detection and the visual target tracking also includes: Read the last image from the decoded frame queue; Determine whether it is in the visual target detection cycle: If so, then executing the step of reading the head image in the decoded frame queue and outputting it as input data for visual target tracking when the number of video frames covered in the previous visual target detection cycle is greater than 0, and subtracting 1 from the number of video frames covered in the previous visual target detection cycle; Otherwise, after the current visual target detection cycle starts, the tail image is output as the input data for visual target detection, and the number of video frames covered in the current visual target detection cycle is increased by 1, and the process returns to execute the steps of reading the tail image from the decoded frame queue and determining whether it is in the visual target detection cycle.
4. The target tracking method according to claim 2, characterized in that: The synchronous processing based on the video frame data in the decoded frame queue to synchronize the input data of the visual target detection and the visual target tracking also includes: Determine whether the current visual target detection cycle is completed: If completed, the first number of images at the head of the decoded frame queue is output as input data for visual target tracking, wherein the first number is the number of video frames covered in the previous visual target detection cycle, and the number of video frames covered in the current visual target detection cycle is used as the number of video frames covered in the new previous visual target detection cycle; If not completed, the number of video frames covered in the current visual target detection cycle is increased by 1.
5. The target tracking method according to claim 1, characterized in that: The visual target tracking based on the input data of the visual target tracking and the visual target detection result comprises: Determine whether the visual target detection result is currently obtained; If the visual target detection result is not obtained, a prediction algorithm is used to predict the motion feature state of the visual target, and the step of returning to execute synchronization processing based on the video frame data in the decoded frame queue to synchronize the input data of the visual target detection and the visual target tracking is performed.
6. The target tracking method according to claim 5, characterized in that: The performing visual target tracking based on the input data of the visual target tracking and the visual target detection result also includes: If the visual target detection result is obtained, a preset appearance feature extraction model is used to extract the target appearance feature information of each visual target, a prediction algorithm is used to predict the state of the visual target, a cost is calculated based on the target appearance feature information and the state prediction information, and data association is performed between the existing visual target and the visual target in the current visual target detection result based on the cost; for the existing visual target and the visual target in the current visual target detection result that have not achieved data association, a secondary association is performed using the intersection and union ratio to obtain an association result; based on the association result, the state of the existing visual target is updated.
7. The target tracking method according to claim 5 or 6, characterized in that: The prediction algorithm includes a Kalman filter algorithm or a single target tracking algorithm.
8. The target tracking method according to claim 6, characterized in that: The cost calculation based on the target appearance feature information and the state prediction information includes: The following calculation formula is used to calculate the cost based on the target appearance feature information and state prediction information: cost (i,j) =λ1·dist1(i,j)+λ2·dist2(i,j)+λ3·dist3(i,j) Among them, dist1 represents the cosine distance or Euclidean distance of the appearance features between the visual target in the current video frame and the existing visual target in the visual target detection result, dist2 represents the Mahalanobis distance between the visual target in the current video frame and the existing visual target in the visual target detection result, dist3 represents the distance of the consistency of the visual target motion vector between the visual target in the current video frame and the existing visual target in the visual target detection result, i and j represent the existing visual target and the visual target in the current video frame in the visual target detection result, respectively, λ1 represents the weight of dist1, λ2 represents the weight of dist2, and λ3 represents the weight of dist3.
9. A target tracking device, characterized in that: include: The video acquisition and decoding module is used to obtain and decode the video stream in real time, and store the decoded video frame data in the decoded frame queue; A synchronization module, for performing synchronization processing based on the video frame data in the decoded frame queue to synchronize input data of visual target detection and visual target tracking; A detection module is used to perform visual target detection and obtain visual target detection results; The tracking module is used to perform visual target tracking based on the input data of the visual target tracking and the visual target detection result.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, the method according to any one of claims 1 to 8 is implemented.
11. An electronic device, characterized in that: The method comprises a memory and at least one processor, wherein a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the method according to any one of claims 1 to 8 is implemented.
12. The electronic device according to claim 11, characterized in that: The processor comprises an embedded processor.
13. A computer program product, characterized in that The computer program product performs the method according to any one of claims 1 to 8 when executed on a processor.
