Multi-target tracking implementation method based on YOLO and improved DeepSORT

Abstract

According to the multi-target tracking implementation method based on the YOLO and the improved DeepSORT provided by the invention, high-precision real-time detection of small high-speed targets is realized through the YOLO, and continuous recognition and trajectory prediction of a plurality of targets are realized in combination with the improved DeepSORT; meanwhile, through a model format conversion and edge deployment mechanism, the trained deep learning model is efficiently migrated to a lightweight terminal platform from a PC terminal; in addition, Raspberry Pi 5 is adopted as a core control unit, local reasoning control and equipment scheduling are realized through a lightweight edge computing platform, and a Hailo-8 edge AI accelerator is adopted to perform accelerated reasoning on a YOLO target detection algorithm, so that the detection frame rate and the response speed of the system on edge equipment are greatly improved, and the system performance is improved. And it is ensured that target identification and decision can be completed within millisecond-level response time.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence and computer vision, and particularly relates to a multi-target tracking implementation method based on YOLO and improved DeepSORT. BACKGROUND

[0002] In recent years, the rapid development of artificial intelligence and computer vision technology has promoted the wide application of intelligent target recognition and tracking systems in the fields of military, security and competition. However, when dealing with small high-speed targets (such as enemy projectiles: hand grenades, smoke bombs, and burning bottles), the traditional static monitoring and passive defense means are slow in response and lack of precision, which cannot meet the high dynamic defense needs in complex tactical scenarios.

[0003] Therefore, there is an urgent need for a method capable of realizing multi-target tracking to solve the above problems. SUMMARY

[0004] In order to solve the shortcomings and deficiencies of the prior art, a multi-target tracking implementation method based on YOLO and improved DeepSORT is provided, so as to solve the problems of slow response and insufficient precision of traditional static monitoring and passive defense means.

[0005] A multi-target tracking implementation method based on YOLO and improved DeepSORT is provided to achieve the purpose of the present application, comprising the following steps:

[0006] Step S1, running environment building: building an adaptive Python virtual environment and Hailo SDK configuration environment on PC and Raspberry Pi device respectively;

[0007] Step S2, model training and deployment: collecting data sets, training YOLO model on PC end using PyTorch and exporting as ONNX format, and then generating HEF file through Hailo compiler to deploy to Hailo-8 accelerator;

[0008] Step S3, realizing accelerated reasoning: realizing image pre-processing through building optimized GStreamer pipeline, and combining Hailo hardware accelerated reasoning and multi-frame batch processing technology;

[0009] Step S4, improving DeepSORT algorithm: replacing traditional DeepSORT, designing and implementing lightweight SimpleTracker tracking module, completing efficient target association based on IOU greedy matching strategy, and being suitable for resource limited platform;

[0010] Step S5, MQTT publishes the prediction point: use paho.mqtt.client to build an MQTT client, and publish the target prediction point in JSON format to the EMQX message service in real time.

[0011] The beneficial effects of the present application are:

[0012] Compared with the prior art, the multi-target tracking implementation method based on YOLO and improved DeepSORT provided by the present application realizes high-precision real-time detection of small high-speed targets through YOLO, realizes continuous identification and trajectory prediction of multiple targets in combination with improved DeepSORT, simultaneously realizes efficient migration of the trained deep learning model from the PC end to the lightweight terminal platform through model format conversion and edge deployment mechanism, and adopts Raspberry Pi 5 as the core control unit to realize local inference control and device scheduling through the lightweight edge computing platform, and adopts Hailo-8 edge AI accelerator to accelerate the inference of the YOLO target detection algorithm, which greatly improves the detection frame rate and response speed of the system on the edge device, and ensures that the target identification and decision can be completed within a millisecond-level response time.

[0013] In summary, the system of the present application is flexible in deployment, compact in structure, low in power consumption, and widely applicable, and has realized the core functions from target detection to tracking and prediction, and can be applied to various complex environments such as military defense, security patrol, competitive confrontation, etc. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a flowchart of the present application;

[0015] Figure 2 is an effect diagram of the present application for identifying trajectory and prediction point;

[0016] Figure 3 is a schematic diagram of trajectory recording and drawing using SimpleTracker target tracking in the present application;

[0017] Figure 4 is a schematic diagram of successful MQTT prediction point publishing;

[0018] Figure 5 is a schematic diagram of the control end successfully subscribing to node information and responding. DETAILED DESCRIPTION

[0019] The specific embodiments of the present application will be further described in detail below in combination with the accompanying drawings:

[0020] Example 1

[0021] According to Figures 1-5As shown, the present application provides a multi-target tracking implementation method based on YOLOv8 and improved DeepSORT, comprising the following steps:

[0022] Step S1, running environment building: build the adapted Python virtual environment and Hailo SDK configuration environment on PC and Raspberry Pi 5 devices respectively.

[0023] The operation process of the above step S1 is as follows:

[0024] Step S1.1, PC environment building:

[0025] The operating system of PC is Ubuntu 22.04. Download and install Anaconda on the official website, complete the installation and configure the environment variable using the terminal command, so that the system can recognize the conda command. Then create a Python virtual environment in Anaconda, use Python 3.10 version to ensure the compatibility of YOLOv8 and PyTorch, and adapt the python version of Ubuntu 22.04. After creating and activating the environment, install the Ultralytics YOLOv8 model library, PyTorch framework and related dependencies using pip; download the compatible x86 software package from the Hailo official website, including HailoRT runtime library, Model Zoo model management toolkit and TAPPAS (Tool Application for Preprocessing and Postprocessing At Scale) inference example suite compatible with Python 3.10 environment, to ensure the integrity and stability of model compilation, deployment and running.

[0026] Step S1.2, Hailo-8 installation and configuration on Raspberry Pi side:

[0027] Raspberry Pi installs 64-bit version of Raspberry Pi OS system to ensure compatibility with Hailo driver and runtime library, and Raspberry Pi has M.2 interface for connecting Hailo-8 accelerator.

[0028] Download the ARM64-compatible software package compatible with the Python 3.10 environment from the Hailo official website, including HailoRT SDK, Hailo Model Zoo, and TAPPAS; to avoid environment conflicts, create an independent environment on the Raspberry Pi local Python virtual environment management tool venv, fix the python version to 3.10, transfer the downloaded installation package to the Raspberry Pi, decompress and install one by one, to ensure that the installed Hailo components are compatible with Python 3.10 version.

[0029] Step S2, model training and deployment: collect the data set, use PyTorch to train the YOLOv8 model on the PC and export it to the ONNX format, and then generate the HEF file through the Hailo compiler to deploy it to the Hailo-8 accelerator.

[0030] The operation process of the above step S2 is as follows:

[0031] Train the YOLOv8 target detection model on the PC using the PyTorch framework; due to special reasons, collect data by throwing scenes from movies and TV series.

[0032] Open the recorded video file using the cv2.VideoCapture() function of OpenCV; read one frame of image and visualize it through matplotlib; use the cv2.cvtColor() function to convert the BGR color format used by default by OpenCV to RGB used by matplotlib.

[0033] The program re-reads the video and starts to traverse frame by frame, and increments the frame counter by one for each processed frame. When the frame number reaches the specified interval value (every 5 frames), save the current frame as a JPG image, and save it in the specified folder with the frame number as the file name and.jpg as the suffix, such as 5.jpg, 10.jpg, etc.

[0034] Use labelimg and label it, divide the labeled data set into training set (train) and validation set (val) according to the ratio of 4:1, and store the images and labels in the images and labels folders respectively, finally put these files into the datasets directory, configure the data set description file.yaml, use the model.train() function to train in the model training process, configure the following parameters: iteration rounds (epochs) is set to 50, data loading thread number (workers) is set to 1, batch size (batch) is set to 16, train to generate the optimal weight file.pt;

[0035] Convert the.pt weight to ONNX format using the export function provided by Ultralytics, and after obtaining the ONNX model, use the Hailo Dataflow Compiler (Hailo Dataflow Compiler, DFC) to compile the ONNX model into Hailo executable format (HEF). When converting, specify the target hardware architecture as Hailo-8 to avoid performance waste or compatibility problems.

[0036] Step S3, realize accelerated inference: realize image preprocessing by building an optimized GStreamer pipeline, and combine Hailo hardware accelerated inference and multi-frame batch processing technology.

[0037] The operation process of the above step S3 is as follows:

[0038] Step S3.1, configure Hailo AI Kit on Raspberry Pi (with Hailo-8 accelerator, peak computing power up to 26TOPS), send the Hailo executable format (HEF) generated in step S2 to Raspberry Pi end, and load the HEF model through Hailo (HailoRT).

[0039] Step S3.2, video stream acquisition: the system acquires real-time video stream through USB camera. The advantage of USB camera is that it is universal, plug and play, good scalability and low cost, and has better compatibility than CSI camera.

[0040] Step S3.3, preprocessing: before sending to Hailo DPU, use multi-level queue (queue) and multi-thread operator in GStreamer pipeline to complete preprocessing in parallel, that is, through videoscalen-threads=2, scale each frame in two CPU threads at the same time, adjust the resolution to the size required by the model input; Then use videoconvertn-threads=2 to do color space conversion in parallel, ensure that the pixel format (BGR ↔ RGB) is consistent with the HEF compilation; Insert queue leaky=no max-size-buffers=3 before and after each operator to decouple the speed difference between upstream and downstream, avoid short blocking, and use multi-thread cache to process concurrently, so that scaling and format conversion are completely parallelized on the CPU side.

[0041] Step S3.4, inference: hailonet executes the quantized convolution and full connection related operators in parallel on the Hailo hardware side. hailofilter also downgrades the post-processing NMS to the device, reducing the burden on the host CPU; batch_size=2 allows two frames to be processed in parallel, reducing context switching overhead.

[0042] Inference time formula:

[0043] ;

[0044] Where: is single frame inference time; B is batch size; is context switch and data transfer overhead;

[0045] NMS post-processing generates ROI:

[0046] ;

[0047] Sort bounding boxes by confidence, keep boxes with confidence higher than nms-score-threshold=0.3, for the highest confidence box , calculate IoU with other boxes , if greater than nms-iou-threshold=0.45, suppress ;

[0048] The following is the IoU calculation method used by NMS:

[0049] ;

[0050] Where: is the intersection over union of boxes and ; is the i-th candidate box; w·h is the intersection area, ;

[0051] nms-score-threshold: confidence threshold (0.3); nms-iou-threshold: IoU threshold (0.45);

[0052] GStreamer pipeline optimization, including multi-threaded processing videoscale and videoconvert use multi-threading (n-threads=2);

[0053] ;

[0054] Where: is single-threaded processing time (~10ms, 640x480 frame); is the number of threads; is multi-threaded processing time;

[0055] Disable QoS (qos=false) to avoid frame loss;

[0056] Queue management: limit buffer size with QUEUE;

[0057] ;

[0058] wherein: is the queue delay; is the maximum buffer number; is the single frame processing time;

[0059] Dynamic resolution and format adaptation: input frame adjustment to network_width = 640, network_height = 640, network_format = RGB;

[0060] ;

[0061] wherein: x, y are the original coordinates of the input frame, input_width, input_height are the input resolution, network_width, network_height are the resolution required by the model;

[0062] Final accelerated inference effect: through GStreamer multi-threaded scaling and format conversion, the pre-processing delay is reduced to 3~5ms, QoS is disabled to prevent frame loss due to load, combined with a 3-frame buffer queue, the total delay is controlled within about 100ms, ensuring 30 FPS real-time inference effect.

[0063] Step S4, improve the DeepSORT algorithm: replace the traditional DeepSORT, design and implement a lightweight SimpleTracker tracking module, complete efficient target association based on the IOU greedy matching strategy, suitable for resource-limited platforms.

[0064] The operation process of the above step S4 is as follows:

[0065] Step S4.1, target detection:

[0066] Use the app_callback() function to obtain the detection result roi from the GStreamer buffer() through hailo.get_roi_from_buffer().

[0067] Extract objects of type HAILO_DETECTION from the roi, only keep the detection results with label OBJ_LABEL; convert the relative coordinates of the detection box (xmin, ymin, xmax, ymax) to pixel coordinates, x1=bb.xmin()*width, y1=bb.ymin()*height, x2=bb.xmax()*width, y2=bb.ymax()*height, these bounding boxes (bboxes) are passed as input to the tracker;

[0068] where the coordinate conversion formula of the detection box is:

[0069] ;

[0070] where: 、 is the normalized coordinate (range [0, 1]) output by the detection model; width, height are the width and height (pixels) of the video frame; 、 is the converted pixel coordinate;

[0071] Step S4.2, SimpleTracker target tracking:

[0072] The DeepSORT tracking module assigns a unique identifier to each detection target and tracks its movement trajectory in consecutive frames. This tracker combines Kalman filtering and feature matching algorithms, which can maintain identity consistency in cases of target occlusion or intersection. However, DeepSort relies on the Re-ID model, Kalman filter, and Hungarian algorithm, which are computationally intensive and can significantly slow down the frame rate (less than 5 FPS) on Raspberry Pi CPU, making it difficult to meet real-time requirements. It needs to rely on PyTorch / TensorFlow, pre-trained Re-ID model, which occupies a lot of resources and is difficult to run stably on lightweight edge devices.

[0073] To adapt to the deployment needs of resource-constrained platforms such as Raspberry Pi, the lightweight target tracking algorithm of SimpleTracker is encapsulated in the simple_sort.py file, and the core idea of this algorithm is to use the IOU greedy matching strategy to associate targets between frames, thereby replacing the computationally intensive method of DeepSort that relies on Kalman filter, Hungarian algorithm, and appearance feature Re-ID. This simplified processing method not only greatly reduces the CPU computation burden, but also improves the response speed and stability of the overall system, especially suitable for fixed-angle, low-occlusion, and low-target-density edge computing scenarios.

[0074] Target tracking: SimpleTracker class implemented in Simple SORT algorithm combines IoU for target association; calculate the IoU between the current frame detection box (dets) and the last frame tracking box (trks) to form an IoU matrix; match the detection box and the tracking box through the maximum IoU; update the matched tracking box, increase the age of the unmatched tracking box, create a new tracking for the unmatched detection box, and remove the tracking over max_age;

[0075] IoU algorithm formula:

[0076]

[0077] Among them: , Two boundary boxes, format , represents the upper left corner and the right lower corner ;

[0078] The intersection area is calculated as:

[0079] ;

[0080] The area of the boundary box , the calculation formula is:

[0081] ;

[0082] is a constant to prevent division by zero;

[0083] Tracking update: the matched tracking box is updated to the coordinates of the current detection box, age is set to 0, and hit count (hit_streak) is incremented by 1; the unmatched tracking box age is incremented by 1, and the hit count is set to 0; the unmatched detection box creates a new tracking, assigns a new track_id, the initial age is 0, and the hit count is 1; delete the tracking whose age exceeds max_age;

[0084] Output format: the tracking result is an array of M*5, each row , only return the tracking whose hit_streak >= min_hits;

[0085] max_age: the maximum number of frames for which the tracking box is unmatched, which is deleted when it exceeds 30;

[0086] min_hits: the tracking box needs to be continuously matched for 1 frame to be output;

[0087] iou_thresh: IoU threshold, below 0.3 this value is not matched;

[0088] track_id: unique track ID, generated by_next_id increment;

[0089] Step S4.3, track record and draw:

[0090] Use track_history to record the history of the center point of each track ID:

[0091] c x =int((x1+x2) / 2), c y =int((y1+y2) / 2);

[0092] hist=track_history[tid], hist.append((c x , c y ));

[0093] Limit the history length to 50 to prevent memory overflow, use OpenCV's cv2.polylines to draw the track, and the center point calculation formula:

[0094] ;

[0095] Hist: a history list that stores up to 50 points;

[0096] Predictive point calculation (linear extrapolation): calculate the average displacement by the center points of the last 4 frames to predict the position of the future steps frames;

[0097] The calculation formula of the average displacement in the x direction and the average displacement in the y direction is:

[0098] ;

[0099] Where: n = 4, i.e. take the last 4 frames; 、 is the center point coordinate of the i-th frame;

[0100] The calculation formula of the predicted point coordinate ( , ) is:

[0101] ;

[0102] Where: x last , y lastThe center point coordinates of the last frame; steps is the number of predicted future frames (5).

[0103] In step S5, the MQTT publishes the predicted point: an MQTT client is built by using paho.mqtt.client, and the target predicted point is published in real time to the EMQX message service in JSON format.

[0104] The operation process of step S5 is as follows:

[0105] An MQTT client is built by using paho.mqtt.client, connected to the EMQX message broker deployed on the Ubuntu system, and the asynchronous communication loop is started in the start_mqtt_client() function. The target predicted point is published in real time to the topic detection / predictions in JSON format. MQTT is used as a lightweight and efficient data communication mechanism in the system, and the future position prediction results of the tracking target are decoupled and published to the network, so as to facilitate subsequent remote visualization, extended control or data acquisition, and support low-latency information interaction in the distributed edge computing scenario.

[0106] The above embodiments are not limited to the technical solutions of the embodiments themselves, and the embodiments can be combined with each other to form new embodiments. The above embodiments are only used to illustrate the technical solutions of the present application and not to limit them. Any modification or equivalent replacement without departing from the spirit and scope of the present application should be covered within the scope of the technical solutions of the present application.

Claims

1. A multi-target tracking method based on YOLO and improved DeepSORT, characterized by: The steps include: Step S1: Setting up the operating environment: Building an appropriate Python virtual environment and Hailo SDK configuration environment on the PC and Raspberry Pi device respectively; Step S2, model training and deployment: Collect the dataset, use PyTorch to train the YOLO model on the PC and export it to ONNX format, then use the Hailo compiler to generate the HEF file and deploy it to the Hailo-8 accelerator; Step S3: Implement accelerated inference: Implement image pre-processing by building an optimized GStreamer pipeline and combining Hailo hardware-accelerated inference with multi-frame batch processing technology. Step S4, improve the DeepSORT algorithm: replace the traditional DeepSORT, design and implement a lightweight SimpleTracker tracking module, and achieve efficient target association based on the IOU greedy matching strategy, which is suitable for resource-constrained platforms; Step S5, MQTT publishes the prediction point: Use paho.mqtt.client to build an MQTT client and publish the target prediction point in JSON format to the EMQX message service in real time.

2. The multi-target tracking method based on YOLO and improved DeepSORT according to claim 1, characterized in that: The operation process of step S1 is as follows: Step S1.1, PC environment setup: The PC's operating system is Ubuntu 22.04; Download and install Anaconda from the official website. Use terminal commands to complete the installation and configure environment variables so that the system can recognize the conda command. Then, create a Python virtual environment in Anaconda and use Python 3.10 to ensure compatibility with YOLO and PyTorch, and adapt the Python version to Ubuntu 22.

04. After creating and activating the environment, use pip to install the Ultralytics YOLO model library, the PyTorch framework, and related dependencies. Download the x86-compatible packages compatible with Python 3.10 from the Hailo official website to ensure the integrity and stability of model compilation, deployment, and operation. These x86-compatible packages include the HailoRT runtime library, the Model Zoo model management toolkit, and the TAPPAS (Tool Application for Preprocessing and Postprocessing AtScale) inference example suite. Step S1.2, Hailo-8 installation and configuration on Raspberry Pi: The Raspberry Pi is installed with the 64-bit version of the Raspberry Pi OS to ensure compatibility with the Hailo driver and runtime library. The Raspberry Pi also has an M.2 interface for connecting to the Hailo-8 accelerator. Download the ARM64-compatible software package for Python 3.10 from the official Hailo website. To avoid environment conflicts, create an independent environment on the Raspberry Pi using the Python virtual environment management tool venv, set Python version to 3.10, transfer the downloaded installation package to the Raspberry Pi, and decompress and install each Hailo component one by one to ensure compatibility with Python 3.

10. The ARM64-compatible software packages for Python 3.10 include the HailoRT SDK, Hailo Model Zoo, and TAPPAS.

3. The multi-target tracking method based on YOLO and improved DeepSORT according to claim 1, characterized in that: The operation process of step S2 is as follows: The YOLO target detection model was trained using the PyTorch framework on the PC. Due to special restrictions, data was collected by analyzing clips from film and television dramas. Use OpenCV's cv2.VideoCapture() function to open the recorded video file; read a frame of the image and visualize it using matplotlib; Use the cv2.cvtColor() function to convert the BGR color format used by OpenCV by default to RGB used by matplotlib; The program rereads the video and starts traversing frame by frame, incrementing the frame counter by one each time it processes a frame. When the frame count reaches the specified interval (every 5 frames), it saves the current frame as a JPG image in the specified folder and names it with the frame number and the suffix .jpg. Use labelimg to label the dataset, divide the labeled dataset into a training set (train) and a validation set (val) in a ratio of 4:1, and store the images and labels in the images and labels folders respectively. Finally, put these files in the datasets directory, configure the dataset description file .yaml, and use the model.train() function to train the model during training. Configure the following parameters: set the number of iterations (epochs) to 50, the number of data loading threads (workers) to 1, and the batch size (batch) to 16. Perform training and generate the optimal weight file .pt; Use the export function provided by Ultralytics to convert the .pt weights to the ONNX format. After obtaining the ONNX model, use the Hailo Dataflow Compiler (DFC) to compile the ONNX model into the Hailo Executable Format (HEF). During the conversion, specify the target hardware architecture as Hailo-8 to avoid performance loss or compatibility issues.

4. The multi-target tracking method based on YOLO and improved DeepSORT according to claim 1, characterized in that: The operation process of step S3 is as follows: In step S3.1, configure the Hailo AI Kit (with a Hailo-8 accelerator, capable of 26 TOPS of peak computing power) on the Raspberry Pi. Send the Hailo Executable Format (HEF) generated in step S2 to the Raspberry Pi, and load the HEF model through Hailo (HailoRT). Step S3.2, video stream acquisition: the system collects real-time video stream through the USB camera; Step S3.3, pre-processing: Before the data is officially sent to the Hailo DPU, pre-processing is completed in parallel in the GStreamer pipeline using multi-level queues and multi-threaded operators. Specifically, videoscalen-threads=2 is used to scale each frame simultaneously in two CPU threads, adjusting any resolution to the required size for the model input. This is followed by parallel color space conversion using videoconvertn-threads=2 to ensure that the pixel format (BGR↔RGB) is consistent with that compiled at HEF. Queue leaky=no max-size-buffers=3 is inserted before and after each operator to decouple upstream and downstream speed differences and avoid short-term blocking. Multi-threaded caching is used for concurrent processing, thus fully parallelizing the scaling and format conversion steps on the CPU side. Step S3.4, inference: Hailonet executes the quantized convolution and fully connected operators in parallel on the Hailo hardware side; Hailofilter also offloads the post-processing NMS to the device, reducing the host CPU burden; batch_size=2 allows two frames to be processed in parallel, reducing context switching overhead; Inference time formula: ； in: is the single frame inference time; B is the batch size; For context switching and data transfer overhead; NMS post-processing generates ROI: ； Sort the detection boxes by confidence, retain the boxes with confidence higher than nms-score-threshold=0.3, and for the boxes with the highest confidence , calculation and other boxes If the IoU is greater than nms-iou-threshold=0.45, suppress ; The following is the IoU calculation method used by NMS: ； in: For the frame and The intersection and union ratio of is the i-th candidate box; w·h is the intersection area, ; nms-score-threshold: confidence threshold (0.3); nms-iou-threshold: IoU threshold (0.45); Optimize GStreamer pipeline, including multithreading videoscale and videoconvert (n-threads=2). ； in: Single-thread processing time (~10ms, 640x480 frame); is the number of threads; Processing time for multiple threads; Disable QoS (qos=false) to avoid frame loss; Queue management: QUEUE limits the buffer size; ； in: is the queue delay; is the maximum number of buffers; is the single frame processing time; Dynamic resolution and format adaptation: input frame is adjusted to network_width=640, network_height=640, network_format=RGB; ； Among them: x, y are the original coordinates of the input frame, input_width, input_height are the input resolution, network_width, network_height are the required resolutions of the model; Ultimately, the inference effect is accelerated: GStreamer multi-threaded scaling and format conversion are used to reduce the pre-processing latency to 3-5ms. QoS is disabled to prevent frame drops due to load. Combined with three frame buffer queues, the total latency is controlled within approximately 100ms, ensuring 30 FPS real-time inference.

5. The multi-target tracking method based on YOLO and improved DeepSORT according to claim 1, characterized in that: The operation process of step S4 is as follows: Step S4.1, target detection: Use the app_callback() function to get the detection result roi from the GStreamer buffer() through hailo.get_roi_from_buffer(); Extract objects of type HAILO_DETECTION from roi and keep only detection results with label OBJ_LABEL; convert the relative coordinates (xmin, ymin, xmax, ymax) of the detection box to pixel coordinates, x1=bb.xmin()*width, y1=bb.ymin()*height, x2=bb.xmax()*width, y2=bb.ymax()*height, and pass these bounding boxes (bboxes) as input to the tracker; The coordinate transformation formula of the detection frame is: ； in: 、 The normalized coordinates output by the detection model (range [0, 1]); width and height are the width and height of the video frame (pixels); 、 is the pixel coordinate after transformation; Step S4.2, SimpleTracker target tracking: The SimpleTracker lightweight target tracking algorithm and its overall logic are encapsulated in the simple_sort.py file. The core concept of this algorithm is to use the IOU greedy matching strategy to associate targets between frames. Target tracking: The SimpleTracker class implemented in the Simple SORT algorithm combines IoU with target association. The IoU matrix is ​​calculated between the detection box (dets) of the current frame and the tracking box (trks) of the previous frame. The detection box and the tracking box are matched using the maximum IoU. The matching tracking box is updated, the age (age) of the unmatched tracking box is increased, new tracks are created for the unmatched detection boxes, and tracks with an age exceeding max_age are removed. IoU algorithm formula:

6. Among them: 、 For two bounding boxes, the format is , indicating the upper left corner and lower right corner ; is the intersection area, and the calculation formula is: ； Bounding box The area is calculated as: ； is a constant Prevent division by zero; Tracking update: The matching tracking frame is updated to the coordinates of the current detection frame, the age is set to 0, and the hit count (hit_streak) is increased by 1; the age of the unmatched tracking frame is increased by 1, and the hit count is set to 0; a new track is created for the unmatched detection frame, and a new track_id is assigned, with an initial age of 0 and a hit count of 1; tracks with an age exceeding max_age are deleted; Output format: The tracking result is an array of M*5, each line is , where only traces with hit_streak >= min_hits are returned; max_age: The maximum number of frames that the tracking frame does not match. If it exceeds 30, it will be deleted. min_hits: The tracking frame needs to match the number of consecutive frames 1 before it is output; iou_thresh: IoU threshold, no matching is performed if the value is lower than 0.3; track_id: unique tracking ID, generated incrementally by _next_id; Step S4.3, trajectory recording and drawing: Use track_history to record the center point history of each tracking ID: c x =int((x1+x2) / 2),c y =int((y1+y2) / 2); hist=track_history[tid],hist.append((c x , c y )); Limit the history length to 50 to prevent memory overflow. Use OpenCV's cv2.polylines to draw the trajectory. The center point calculation formula is: ； Hist: Storage History list, up to 50 points; Prediction point calculation (linear extrapolation): Calculate the average displacement of the center points of the last four frames and predict the position of the next steps frame; Average displacement in the x-direction and the average displacement in the y direction The calculation formula is: ； Where: n = 4, that is, take the last 4 frames; 、 is the center point coordinate of the i-th frame; Prediction point coordinates ( , ) is calculated as: ； Where: x last 、y last is the center point coordinate of the last frame; steps is the predicted number of future frames (5).

7. The multi-target tracking method based on YOLO and improved DeepSORT according to claim 1, characterized in that: The operation process of step S5 is as follows: Use paho.mqtt.client to establish an MQTT client and connect to the EMQX message broker deployed on the Ubuntu system. In the start_mqtt_client() function, start the asynchronous communication loop and publish the target prediction points in JSON format to the topic detection / predictions in real time. MQTT is used as a lightweight and efficient data communication mechanism in the system to decouple and publish the future position prediction results of the tracking target to the network, facilitating the subsequent remote visualization, extended control or data collection, and supporting low-latency information interaction in distributed edge computing scenarios.

Images