Target detection and tracking method based on low computing power platform
By combining the YOLO-V5 series object detection algorithm and SORT object tracking algorithm, the problem of tight computing resources of object detection and tracking algorithms on low-computing power platforms is solved, efficient object detection and tracking is achieved, meeting real-time requirements and reducing accuracy losses.
Patent Information
- Application Number
- CN202411801595.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-09
- Publication Date
- 2025-03-28
AI Technical Summary
The existing high-precision object detection and tracking algorithms face the problem of tight computing resources when applied on low-computing platforms, which is difficult to meet the real-time requirements, and there is also a balance between accuracy loss and real-time.
Combining the YOLO-V5 series object detection algorithm and SORT object tracking algorithm, the object detection and tracking performance in low-computing environments is optimized. The YOLO-V5 series performs object detection in each frame, extracts detection frame information, and passes this information to the SORT algorithm for multi-frame tracking.
It effectively improves the efficiency of object detection and tracking, reduces the demand for computing resources, meets the real-time requirements in low-computing environments, and minimizes accuracy losses as much as possible.
Smart Images
Figure CN119723416A_ABST
Abstract
Description
Technical Field
[0001] The present invention application relates to the field of computer vision and intelligent monitoring technology, specifically a target detection and tracking method based on a low computing power platform. Background Art
[0002] Image target detection and tracking is an important part of computer vision technology and is widely used in video surveillance, autonomous driving, drones and other fields. It mainly realizes identification, matching or positioning by extracting the color, contour, texture, motion parameters and other features of the target object. The auxiliary light source is used to illuminate the characteristic area of the target with visible light or infrared light to obtain a clearer target image; the camera realizes the acquisition of the target image; the image acquisition and transmission circuit is used to read and transmit the image data collected by the camera; the image processing equipment includes a variety of optional forms such as computers and embedded systems, which are mainly used to deploy and run algorithms such as image enhancement and target tracking.
[0003] At present, most image target detection and tracking systems are mainly built on graphics workstations or high-performance computers, using Microsoft Visual Studio and OpenCV for algorithm software design, and using 50ms / frame as the rate threshold to determine whether the system has real-time performance. As general-purpose processors, the multiplication capacity of graphics workstations or high-performance computer CPUs is generally limited. Researchers usually use a general-purpose computer system framework of CPU-PCIe-GPU, using GPU to assist the heterogeneous operation mode of CPU to accelerate the operation of image target detection and tracking algorithms. Traditional target detection algorithms usually rely on deep neural network models. Although these models have high detection accuracy, they also have relatively high computing resource requirements.
[0004] Existing high-precision target detection and tracking algorithms face the problem of limited computing resources when applied on low-computing platforms, which makes it difficult for these algorithms to meet real-time requirements. Although some methods try to reduce computational complexity by optimizing network structure or algorithm strategy, there is still a balance between accuracy loss and real-time performance.
[0005] Therefore, there is an urgent need for a method that can achieve efficient target detection and tracking on a low-computing power platform to meet the needs of real-time processing while minimizing accuracy loss. Summary of the invention
[0006] The purpose of this application is to provide an efficient solution suitable for low-computing power platforms, especially for environments with high real-time requirements but limited computing resources, such as embedded systems, mobile devices and other low-computing power computing platforms.
[0007] To achieve the above objectives, the technical solutions adopted in the embodiments of the present application are as follows.
[0008] Specifically, the present invention application provides a target detection and tracking method based on a low computing power platform, combining the YOLO series target detection algorithm with the SORT target tracking algorithm to optimize the target detection and tracking performance in a low computing power environment.
[0009] Specifically, a target detection and tracking method suitable for low computing power platforms uses the YOLO-V5 series target detection algorithm and the SORT algorithm to track the detected target, which is characterized by:
[0010] The target detection module uses the YOLO-V5 series to identify and locate the original image target in each frame of the video, extract the detection frame information of the target object, and pass this information to the target tracking module;
[0011] The target tracking module uses the SORT algorithm to track the detected target in multiple frames. The SORT algorithm uses a Kalman filter to predict the target position of the current frame, and compares and matches the target information of the current frame with the trajectory information of the previous frame, and finally obtains the target detection and tracking results.
[0012] Furthermore, the YOLO-V5 series target detection algorithm adopts a single-stage target detection algorithm to preprocess and scale the input image to make it consistent with the input size requirement, and extracts abstract features of the image through convolution, pooling, activation and other operations on the input image as the basis for target detection.
[0013] Furthermore, the abstract feature fusion result is used as input, and the normalized data is restored to the original width and height and decoded by non-maximum suppression operations to obtain the final target detection result, and the target's bounding box coordinates and category information are output.
[0014] Furthermore, after the target detection is completed, the SORT algorithm is used to track the target, the motion trajectory of the target is predicted by the Kalman filter, and the target data and detection data are associated with the Hungarian algorithm. The SORT tracking algorithm uses a target detector to detect the target image in each frame, and the detection result of the current frame is associated with the tracking target of the previous frame through the Hungarian algorithm, and its position in the next frame is predicted according to the current position, speed and other states of the target.
[0015] Furthermore, the similarity or distance between the detection box and the tracked target box is calculated, and the detection result is associated with the tracked target according to the target box IOU threshold to update the position and state of the target including the target box IOU threshold matching:
[0016] For the detection frame that is not matched to the target frame, a new tracker is created and the tracker is initialized with the detector;
[0017] For trackers that are not matched to the target frame, the life cycle is reduced by 1, the tracker is not updated, and the tracker is directly deleted if it is not in the life cycle;
[0018] Match the detection box of the tracker and use the detection box to update the tracker.
[0019] Furthermore, predicting the position of the next frame of the moving image where the current target may appear in the detection frame is called target position tracking and capturing. Based on the position where the target may appear, the Kalman filter prediction method is used to first predict the position of the next frame where the target may appear, and the predicted position is retrieved in the next frame image to see if the target exists; according to the detection frame result, the target captured in the next frame image is matched with the target determined in the previous frame. If it is completely matched, it is called a high-score detection frame. The target captured in the next frame image is matched with the target determined in the previous frame. If the target is not matched, it is called a low-score detection frame, and the target position tracking and capturing is repeated in the subsequent three frames. If no matching target appears in the repeated target position tracking and capturing of these three frames, the target is deleted from the tracker and the tracker is updated in the detection frame; the target frame data of the tracker set is used to interact with the target detection data, and the Kalman filter is used to predict the position of the next frame of the moving image where the current target may appear in the detection frame, so that the target can be tracked in real time at a fast speed in the video stream.
[0020] Furthermore, the strategy of YOLO-V5 series one-frame detection and SORT multi-frame tracking is preferably YOLO-V5 series one-frame detection and SORT three-frame tracking.
[0021] The present invention has the following beneficial effects:
[0022] The one-frame target detection and multi-frame tracking method based on a low-computing platform detects the target in each frame, and tracks the target through the SORT algorithm in subsequent multiple frames. The frame rate of the one-frame and three-frame tracking is higher than that of the single-frame tracking, which can effectively improve the tracking performance. The single-frame detection and multi-frame tracking method reduces the demand for computing resources. In the multi-frame tracking stage, there is no need to repeat the target detection, which greatly improves the overall processing efficiency and effectively meets the real-time requirements in the actual deployment of low-computing environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0024] Figure 1 A flow chart of the target detection and tracking method applied for by the present invention;
[0025] Figure 2 A flow chart of target detection deployment for the present invention;
[0026] Figure 3 A target tracking flow chart for the present invention;
[0027] Figure 4 This is the RTMP streaming result diagram of the YOLO-V5 single-frame detection and SORT multi-frame tracking deployed on the RK platform applied for by the present invention. DETAILED DESCRIPTION
[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. It should be noted that, in the absence of conflict, the various features in the embodiments of the present application can be combined with each other, and the combined embodiments are still within the scope of protection of the present application.
[0029] The present invention proposes a target detection and tracking method suitable for low computing power platforms, and its flow chart is shown in the attached figure. Figure 1 As shown in the figure: the original image input is preprocessed and scaled, and the abstract features of the image are extracted by operating the input image as the basis for target detection. The detected target is tracked, and the processed data is sent and received through the serial port and pushed through the network port to achieve communication with the client.
[0030] Correspondingly, the system architecture of the low computing power platform applied for by the present invention also includes two main modules: target detection module and target tracking module. The target detection module uses the YOLO series algorithm to identify and locate the target in each frame of the image; the target tracking module uses the SORT algorithm to track the detected target in multiple frames. The entire system aims to adopt the efficient detection capability of the YOLO series and the real-time tracking capability of SORT to achieve efficient and real-time target detection and tracking, and combine the serial port to send and receive data with the network port to achieve communication with the client.
[0031] Object Detection Module
[0032] The mainstream method of deep learning target detection and recognition is mainly based on convolutional neural network (CNN). The algorithm branches include one-state target detection and two-stage target detection. The characteristics of the one-state target detection algorithm are end-to-end detection and recognition, fast speed and high processing accuracy. A typical one-state target detection network structure mainly consists of input, network backbone, network output connection (neck) and network detection head (prediction).
[0033] The two-stage target detection algorithm is carried out in two steps: 1. Get the input image and extract the candidate regions; 2. Scale the image of each candidate region to a fixed size of 224x224, input it into the CNN network, and then input the result into the classifier for category determination.
[0034] The present invention application adopts a single-stage target detection algorithm to preprocess and scale the input image to keep it consistent with the input size requirement. The network backbone is the main part of the entire convolutional neural network. It extracts abstract features of the image through operations such as convolution, pooling, and activation of the input image as the basis for target detection. The network output connection is a "bridge" connecting the network backbone and the network detection head. Model reasoning performs feature fusion on the extracted image features to make the final prediction result more accurate. The network detection head part uses the feature fusion result output by the network connection component as input, and undergoes data decoding processing, such as normalizing the data to restore the original width and height and non-maximum suppression (NMS) and other operations to obtain the final target detection result. The process of target detection deployment on the RK platform is shown in the attached figure. Figure 2 shown.
[0035] In the present invention application, referring to Table 1-Comparison of the performance of the YOLO series models deployed on the Rockchip platform, after comprehensively considering the target detection algorithm deployment performance, algorithm accuracy and usage scenarios, it is preferred to use the YOLO-V5 series algorithm for image target detection on the Rockchip platform. The YOLO-V5 series uses a lightweight but powerful convolutional neural network structure CSP Darknet53 as its backbone network, which combines the CSP and Darknet53 architectures to improve the efficiency and performance of feature extraction. The present invention application adopts the YOLO-V5 series with high comprehensive performance such as convergence speed and performance accuracy and low computing power requirements, allowing the model to extract image features from artificial neural network layers of different scales, and is designed to help predict targets at various scales.
[0036] Table 1 Performance comparison of YOLO series models deployed on Rockchip platform
[0037]
[0038] The loss functions of the YOLO-V5 system include positioning loss and classification loss, which are customized and optimized to adapt to target detection tasks. The present invention adopts the YOLO-V5 system as a target detection algorithm to extract targets from each frame of the image, and processes them at different scales to adapt to targets of different sizes. It is further optimized through pooling and batch normalization, and achieves excellent accuracy and speed on multiple benchmark data sets. It can quickly and accurately identify targets in images and output the bounding box coordinates and category information of the target. The YOLO-V5 system can perform target detection at different scales, maintaining a high processing speed while providing high-precision detection results.
[0039] Target tracking module
[0040] After the target detection is completed, the SORT tracking algorithm is used to track the target. The SORT tracking algorithm predicts the target's motion trajectory through the Kalman filter, and combines the Hungarian algorithm to associate the target data with the detection data. It can effectively track the detected target in continuous frames with fast processing speed.
[0041] The SORT tracking algorithm can realize online real-time tracking of targets in video streams. The target tracking process is shown in the attached figure. Figure 3 As shown in the figure, after receiving the image of the target detection result, the SORT tracking algorithm uses the target detector to detect the target image in each frame, associates the detection result of the current frame with the tracked target of the previous frame through the Hungarian algorithm, and predicts its position in the next frame based on the current state of the target (position, speed, etc.). Such a prediction model can effectively track the motion trajectory of the target and provide a certain robustness when the target moves in the video. After detection and matching, the SORT tracking algorithm uses the data association method to update the state of the target.
[0042] Calculate the similarity or distance between the detection box and the tracked target box, and associate the detection result with the tracked target according to the target box IOU threshold to update the position and status of the target.
[0043] The results of the target box IOU threshold matching are as follows:
[0044] For the detection frame that is not matched to the target frame, a new tracker is created and the tracker is initialized with the detector;
[0045] For trackers that are not matched to the target frame, the life cycle is reduced by 1, the tracker is not updated, and the tracker is directly deleted if it is not in the life cycle;
[0046] Match the detection frame of the tracker and use the detection frame to update the tracker;
[0047] The target frame data of the tracker set is used to interact with the target detection data, and the Kalman filter is used to predict the position of the next frame of the motion image where the current target may appear in the detection frame, so that the target can be tracked in real time at a fast speed in the video stream to meet the requirements of real-time online target tracking.
[0048] Among them, predicting the position of the next frame of motion image where the current target may appear in the detection frame is called target position tracking and capture. Based on the position where the target may appear, the Kalman filter prediction method is used to first predict the position of the next frame where the target may appear, and then predict the position in the next frame to retrieve whether the target exists; according to the detection frame result, the target captured in the next frame is matched with the target determined in the previous frame. If it is a complete match, it is called a high-score detection frame. The target captured in the next frame is matched with the target determined in the previous frame. If the target is not matched, it is called a low-score detection frame, and the target position tracking and capture are performed in the subsequent 3 frames. If no matching target appears in the repeated target position tracking and capture in these 3 frames, the target is deleted from the tracker and the tracker is updated in the detection frame.
[0049] Adopting the strategy of one-frame detection and SORT multi-frame tracking of the YOLO-V5 series, target detection is performed in each frame, and the target is tracked by the SORT algorithm in subsequent multi-frames. The performance comparison of single-frame detection and single-frame, two-frame, and three-frame tracking is compared. See the performance comparison table 2 deployed on the RK3562 platform. It can be clearly known that under the same time consumption, the frame rate of two-frame and three-frame tracking is improved compared with that of single-frame tracking, which can effectively improve the tracking efficiency. In the embodiment of the present invention, the performance of YOLO-V5 single-frame detection and SORT three-frame tracking is the highest. The method of single-frame detection and multi-frame tracking reduces the demand for computing resources. In the multi-frame tracking stage, there is no need to repeat target detection, thereby greatly improving the overall processing efficiency.
[0050] Table 2 Performance comparison of different tracking methods deployed on the RK3562 platform
[0051]
[0052] Attached Figure 4 This is the RTMP streaming result diagram of the YOLO-V5 single-frame detection and SORT three-frame tracking deployed on the low-computing RK3562 platform.
[0053] The parts not involved in the present application are the same as the prior art or can be implemented by using the prior art.
[0054] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A target detection and tracking method suitable for low computing power platforms, using the YOLO-V5 series target detection algorithm and the SORT algorithm to track the detected target, characterized in that: The target detection module uses the YOLO-V5 series to identify and locate the original image target in each frame of the video, extract the detection frame information of the target object, and pass this information to the target tracking module; The target tracking module uses the SORT algorithm to track the detected target in multiple frames. The SORT algorithm uses a Kalman filter to predict the target position of the current frame, and compares and matches the target information of the current frame with the trajectory information of the previous frame, and finally obtains the target detection and tracking results.
2. The method according to claim 1, characterized in that The YOLO-V5 series target detection algorithm adopts a single-stage target detection algorithm to preprocess and scale the input image to keep it consistent with the input size requirements. It extracts abstract features of the image through convolution, pooling, activation and other operations on the input image as the basis for target detection.
3. The method according to claim 2, characterized in that The abstract feature fusion result is used as input, and the normalized data is restored to the original width and height and non-maximum suppression and other operations are decoded to obtain the final target detection result, and the target's bounding box coordinates and category information are output.
4. The method according to claim 1, characterized in that After the target detection is completed, the SORT algorithm is used to track the target. The Kalman filter is used to predict the target's motion trajectory, and the Hungarian algorithm is combined to associate the target data and the detection data. The SORT tracking algorithm uses a target detector to detect the target image in each frame, and the Hungarian algorithm is used to associate the detection result of the current frame with the tracking target of the previous frame, and predicts the position of the target in the next frame based on its current position, speed and other states.
5. The method according to claim 4, characterized in that Calculate the similarity or distance between the detection box and the tracked target box, and associate the detection result with the tracked target according to the target box IOU threshold to update the position and state of the target including the target box IOU threshold match: For the detection frame that is not matched to the target frame, a new tracker is created and the tracker is initialized with the detector; For trackers that are not matched to the target frame, the life cycle is reduced by 1, the tracker is not updated, and the tracker is directly deleted if it is not in the life cycle; Match the detection box of the tracker and use the detection box to update the tracker.
6. The method according to claim 4, characterized in that Predicting the position of the next frame of the motion image where the current target may appear in the detection frame is called target position tracking and capture. Based on the position where the target may appear, the Kalman filter prediction method is used to predict the position of the next frame where the target may appear, and then predict the position in the next frame to retrieve whether the target exists; according to the detection frame result, the target captured in the next frame is matched with the target determined in the previous frame. If it is completely matched, it is called a high-score detection frame. The target captured in the next frame is matched with the target determined in the previous frame. If the target is not matched, it is called a low-score detection frame, and the target position tracking and capture is repeated in the subsequent 3 frames. If no matching target appears in the repeated target position tracking and capture in these 3 frames, the target is deleted from the tracker and the tracker is updated in the detection frame; The target frame data of the tracker set is used to interact with the target detection data, and the Kalman filter is used to predict the position of the next frame of the motion image where the current target may appear in the detection frame, so that the target can be tracked in real time at a fast speed in the video stream.
7. The method according to claim 1, characterized in that The strategy of one-frame detection of YOLO-V5 series and multi-frame tracking of SORT is preferably one-frame detection of YOLO-V5 series and three-frame tracking of SORT.