Multi-target tracking deployment method for low-computing-power edge device

Abstract

The invention discloses a multi-target tracking deployment method for a low-computing-power edge device, which belongs to the technical field of computer vision, and comprises the following steps: constructing an edge computing system of an IoT (Internet of Things) system architecture, starting a v4l2 framework to carry out video acquisition, constructing an FIFO (First In First Out) group cache pool, carrying out frame number complementation and reasoning judgment, carrying out sub-thread reasoning processing, carrying out video coding, and carrying out flow control and data transmission. According to the method, the detection precision and the tracking stability are ensured by using a yolov5m model and a deport model, the real-time performance of a picture is not influenced by adopting a strategy of concurrency of reasoning and video sequence acquisition, the tracking accuracy and stability are remarkably improved, the tracking ID is effectively prevented from being frequently switched in the tracking process, the tracking accuracy is improved, and the tracking efficiency is improved. The tracking stability is obviously improved, and the continuity and consistency of target tracking are ensured.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a multi-target tracking deployment method for a low-computing power edge device. Background Art

[0002] With the rapid development of science and technology, Internet of Things (IoT) technology has been widely used in smart homes, smart patrols, smart cities and other fields, profoundly affecting our daily lives. IoT technology connects objects in the physical world to the network through various information sensing devices, uses protocols to achieve information exchange and communication, and combines artificial intelligence and big data technology to achieve intelligent identification, positioning and tracking of objects. However, with the rapid increase in the number of IoT devices and the sharp increase in data processing needs, the traditional centralized cloud computing architecture has gradually exposed a series of problems, such as rising costs, increased latency, excessive bandwidth requirements, and insufficient data privacy and security. In order to solve these problems, the market demand for sinking computing power is becoming increasingly urgent, and edge computing has come into being.

[0003] Edge computing significantly reduces latency, increases data processing speed, and improves user experience by pushing computing and data storage capabilities to the edge of the network, close to data sources and end users. As artificial intelligence technology extends from data centers to the edge, edge AI has gradually become an important driving force for edge computing. By performing AI reasoning locally, edge AI not only reduces data transmission costs, but also avoids the risk of leakage caused by transmitting sensitive data over public networks, supports offline operation, and thus enhances the robustness and reliability of the system.

[0004] However, the demand for computing power in current edge device application scenarios varies greatly, and the computing power and storage space of edge devices themselves are limited, and cannot support complex AI algorithm models for real-time reasoning. Existing AI algorithms usually need to adapt to lightweight, low-power, and low-real-time requirements by reducing model parameters and sacrificing accuracy. Although this approach can ensure the real-time performance of edge devices to a certain extent, it also significantly reduces the detection accuracy and tracking stability of the algorithm, especially in multi-target tracking.

[0005] The multi-target tracking algorithm aims to accurately detect and track multiple moving targets in a video or image sequence and predict the movement trajectories of these targets. The accuracy of the algorithm depends not only on the accuracy of target detection, but also requires complex matching strategies. However, in low-computing edge devices, existing technologies often have to sacrifice detection accuracy and tracking stability in exchange for real-time reasoning. This trade-off has resulted in a certain impact on the accuracy and stability of tracking in practical applications, making it difficult to meet high-demand application scenarios. Summary of the invention

[0006] The purpose of the present invention is to provide a multi-target tracking deployment method for low-computing power edge devices, which solves the technical problem that high-precision algorithms and real-time images cannot be achieved at the same time.

[0007] To achieve the above object, the present invention adopts the following technical solution:

[0008] A multi-target tracking deployment method for a low-computing edge device includes the following steps:

[0009] Step 1: Build the edge computing system of the IoT system architecture, including the video acquisition module, reasoning and parallel processing module, video encoding module, and flow control module;

[0010] The video acquisition module uses the v4l2 framework to continuously obtain real-time video frames from the camera and stores the real-time video frames in the FIFO frame buffer pool to obtain continuous frame data;

[0011] Step 2: Build the YOLOv5m model and DeepSORT model in the inference and parallel processing module, use a FIFO group cache pool of N frames, use the first frame in the frame data for inference alone, and directly transfer the remaining frames;

[0012] Determine whether the current frame is the first frame in the group by finding the modulus of the frame sequence of the frame data, and use the child thread to perform reasoning;

[0013] Step 3: For each group of data in the FIFO group buffer pool, the video encoding module performs H254 encoding in sequence, generates ts packets, and stores them in the ts buffer pool in sequence;

[0014] Step 4: The flow control module cyclically obtains ts packets from the ts buffer pool and transmits them to the host computer using the UDP protocol.

[0015] Preferably, when executing step 1, the specific steps are as follows:

[0016] Step 1-1: The video acquisition module starts the v4l2 framework to obtain real-time video frames from the camera. The v4l2 framework is used to process the data stream of the video frames output by the camera and convert it into a standard format;

[0017] Step 1-2: The v4l2 framework continuously captures real-time video frames from the camera, stores all video frames in sequence into the FIFO frame buffer pool, and obtains frame data;

[0018] The FIFO frame buffer pool is used to temporarily store frame data so that the frame data is not lost and is stably and continuously output to the inference stage and encoding stage.

[0019] Preferably, when executing step 2, the following steps are specifically included:

[0020] Step 2-1: Storing frame data in the FIFO frame buffer pool in a manner of N frames as a group of data, and constructing a FIFO group buffer pool, where N is a positive integer and the value of N is a preset value;

[0021] The first frame in each set of data will be used for inference processing, and the remaining frames will be stored in the corresponding positions in order to ensure the order and continuity of the frame data;

[0022] Step 2-2: extract frame data from the FIFO group buffer pool, and perform a modulo operation based on the frame sequence of the frame data and the value of N: if the result is 0, it means that the current frame is the first frame in the group and needs to be inferred; otherwise, no inference processing is required;

[0023] Step 2-3: For frame data with a remainder of 0, use a child thread to perform non-waiting inference processing.

[0024] Preferably, when executing step 2-3, the following steps are specifically included:

[0025] Step 2-3-1: Target detection, specifically inputting the frame data into the YOLOv5m target detection model, the YOLOv5m target detection model detects all targets in the current frame and outputs the category and coordinate information of each target;

[0026] Step 2-3-2: Appearance feature extraction, specifically, intercepting the target image of each target according to the coordinates of the target detected in step 2-3-1;

[0027] Each target image is input into the feature extraction network separately to extract the high-dimensional appearance features of each target;

[0028] Step 2-3-3: Kalman prediction, specifically, for each tracked object that has been tracked, use the Kalman filter to predict its position in the current frame and generate a prediction box;

[0029] Step 2-3-4: Appearance feature matching, specifically matching the high-dimensional appearance features of the target extracted in step 2-3-2 with the appearance features of each tracked object in the DeepSORT tracking queue;

[0030] Use the minimum cosine distance and Mahalanobis distance to make judgments: if the match is successful, it is considered that the current target and the corresponding tracked object are the same object;

[0031] Step 2-3-5: IOU matching, specifically, for targets whose appearance feature matching fails, IOU is used for matching, and the detection boxes of these targets that failed to match are compared with the predicted boxes of the unmatched tracking objects to determine the matching situation: if the IOU value exceeds the set threshold, the match is considered successful, and the status of the tracking object is updated;

[0032] Step 2-3-6: Kalman update, specifically, for the tracked object that has been successfully matched through appearance feature matching or IOU matching, use its corresponding target detection box to update the Kalman filter and correct the motion trajectory of the tracked object;

[0033] For unmatched targets, they are initialized as new tracking objects and assigned a new tracking ID;

[0034] Step 2-3-7: Tracking result output, specifically drawing the information of the successfully matched tracking object onto the current frame data, and storing the updated frame data into the corresponding position in the FIFO group buffer pool.

[0035] Preferably, when executing step 3, the following steps are specifically included:

[0036] Step 3-1: extract each group of frame data from the FIFO group buffer pool in sequence;

[0037] Step 3-2: Encode the extracted frame data in sequence using H254;

[0038] Step 3-3: Construct TS packets for the encoded data so that each set of frame data is encapsulated in sequence;

[0039] Step 3-4: Store the constructed TS packets in the ts buffer pool in order for subsequent data transmission.

[0040] Preferably, when executing step 4, the following steps are specifically included:

[0041] Step 4-1: Loop to get TS packets from the TS cache pool;

[0042] Step 4-2: Use UDP protocol to transmit TS packets;

[0043] Step 4-3: The host computer decodes the received TS packets and displays the video data in real time to complete the multi-target tracking process.

[0044] The multi-target tracking deployment method of a low-computing power edge device described in the present invention solves the technical problem that high-precision algorithms and real-time images cannot be achieved at the same time. The present invention uses the YOLOv5m model for target detection. Compared with lightweight models such as MobileNetV3, it provides higher detection accuracy. This high-precision detection directly improves the accuracy of tracking, because the accuracy of tracking depends to a large extent on the accuracy of initial detection. The present invention also uses the DeepSORT algorithm that retains the appearance feature extraction and matching functions, which significantly improves the stability of tracking and ensures the continuity and consistency of target tracking. The present invention successfully achieves a balance between accuracy and real-time performance on a low-computing power edge device by adopting a parallel strategy of reasoning and acquisition of a group of N frames, avoiding the dilemma of weighing between accuracy and real-time performance due to insufficient computing power of the device, and ensuring the smoothness of video output. The parallel processing of reasoning and acquisition not only improves the efficiency of reasoning, but also ensures the continuity of real-time video images, and will not affect the smoothness of the image due to the delay of the reasoning process, so that low-computing power devices can still achieve real-time and stable tracking of multiple targets while maintaining high detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is the main flow chart of the present invention;

[0046] Figure 2 It is the overall flow chart of the present invention;

[0047] Figure 3 Flow chart of steps 2-3 of the present invention. DETAILED DESCRIPTION

[0048] like Figure 1-Figure 3 A multi-target tracking deployment method for a low-computing edge device is shown, comprising the following steps:

[0049] Step 1: Build the edge computing system of the IoT system architecture, including the video acquisition module, reasoning and parallel processing module, video encoding module, and flow control module;

[0050] In this embodiment, the video acquisition module, inference and parallel processing module, video encoding module and flow control module are all deployed on the edge device. The edge device and the host computer communicate with each other through the Internet, and the edge device and the camera communicate through a video data cable or USB bus.

[0051] The video acquisition module uses the v4l2 framework to continuously obtain real-time video frames from the camera and stores the real-time video frames in the FIFO frame buffer pool to obtain continuous frame data;

[0052] The specific steps are as follows:

[0053] Step 1-1: The video acquisition module starts the v4l2 framework to obtain real-time video frames from the camera. The v4l2 framework is used to process the data stream of the video frames output by the camera and convert it into a standard format;

[0054] Step 1-2: The v4l2 framework continuously captures real-time video frames from the camera, stores all video frames in sequence into the FIFO frame buffer pool, and obtains frame data;

[0055] The FIFO frame buffer pool is used to temporarily store frame data so that the frame data is not lost and is stably and continuously output to the inference stage and encoding stage.

[0056] In this embodiment, the edge device uses the v4l2 framework to continuously obtain real-time frame data from the camera and stores it in the FIFO frame buffer pool, saving the time of each frame image from acquisition to inference and then to encoding output, and ensuring the continuity of the video frame.

[0057] Step 2: Build the YOLOv5m model and DeepSORT model in the inference and parallel processing module, use a FIFO group cache pool of N frames, use the first frame in the frame data for inference alone, and directly transfer the remaining frames;

[0058] Determine whether the current frame is the first frame in the group by finding the modulus of the frame sequence of the frame data, and use the child thread to perform reasoning;

[0059] The FPS30 camera takes a fixed time of 33ms to collect each frame. In this embodiment, the FIFO group buffer pool is constructed based on the idea of ​​grouping N frames.

[0060] The first frame of each group in the FIFO group cache pool is the frame data that is separated out using a sub-thread for independent inference. The remaining frames in the group are subsequent frame data that are not processed and are directly transferred to the corresponding positions in the group in sequence, ensuring that the total time required to collect N-1 frames is slightly greater than the time required for inference, which solves the problem of screen freezes and leaves sufficient time for inference.

[0061] The specific steps include:

[0062] Step 2-1: Storing frame data in the FIFO frame buffer pool in a manner of N frames as a group of data, and constructing a FIFO group buffer pool, where N is a positive integer and the value of N is a preset value;

[0063] The first frame in each set of data will be used for inference processing, and the remaining frames will be stored in the corresponding positions in order to ensure the order and continuity of the frame data;

[0064] Step 2-2: extract frame data from the FIFO group buffer pool, and perform a modulo operation based on the frame sequence of the frame data and the value of N: if the result is 0, it means that the current frame is the first frame in the group and needs to be inferred; otherwise, no inference processing is required;

[0065] Step 2-3: For frame data with a remainder of 0, use a child thread to perform non-waiting inference processing.

[0066] When executing step 2-3, the specific steps include:

[0067] Step 2-3-1: Target detection, specifically inputting the frame data into the YOLOv5m target detection model, the YOLOv5m target detection model detects all targets in the current frame and outputs the category and coordinate information of each target;

[0068] In this embodiment, before calculating the frame data whose remainder is 0, format conversion and preprocessing are performed first, and then sent to the yolov8m target detection algorithm to obtain the category, coordinates and other information boxes of the target existing in the current frame image.

[0069] Step 2-3-2: Appearance feature extraction, specifically, capturing the target image of each target according to the coordinates of the target detected in step 2-3-1.

[0070] Each target image is input into the feature extraction network separately to extract the high-dimensional appearance features of each target.

[0071] Step 2-3-3: Kalman prediction: specifically, for each tracked object that is already being tracked, use the Kalman filter to predict its position in the current frame and generate a prediction box.

[0072] Step 2-3-4: Appearance feature matching, specifically matching the high-dimensional appearance features of the target extracted in step 2-3-2 with the appearance features of each tracked object in the DeepSORT tracking queue.

[0073] The minimum cosine distance and Mahalanobis distance are used for judgment: if the match is successful, it is considered that the current target and the corresponding tracked object are the same object.

[0074] In this embodiment, the appearance features corresponding to the detected target are matched with the appearance features stored in the ReID corresponding to the confirmed tracking object in the current tracking queue of deepsort using the minimum cosine distance and Mahalanobis distance. If the minimum cosine distance and Mahalanobis distance are less than a certain threshold, it is considered that the current target matches the tracking object successfully.

[0075] Step 2-3-5: IOU matching. Specifically, for targets whose appearance features fail to match, IOU is used for matching. The detection boxes of these targets that failed to match are compared with the predicted boxes of the unmatched tracked objects to determine the matching status. If the IOU value exceeds the set threshold, the match is considered successful and the status of the tracked object is updated.

[0076] In this embodiment, the detection frame coordinates of the redundant detection targets that failed to match the appearance features are matched with the prediction frames of the confirmed tracking objects in the current tracking queue of the redundant deepsort that failed to match and the unconfirmed tracking objects in the current tracking queue of the deepsort, and the match is determined by calculating whether the combat ratio of the two exceeds the threshold. The state of the unconfirmed tracking object that successfully matches is converted to the confirmed state; the tracking object that failed to match is determined to have temporarily disappeared in the picture, and if it still fails to match after retaining 10 frames, it is lost by default.

[0077] Step 2-3-6: Kalman update, specifically, for the tracked object that has successfully passed appearance feature matching or IOU matching, use its corresponding target detection box to update the Kalman filter and correct the motion trajectory of the tracked object.

[0078] For unmatched targets, they are initialized as new tracking objects and assigned a new tracking ID.

[0079] In this embodiment, the tracking object with successful appearance feature matching and iou matching uses the detection frame coordinates to perform Kalman filtering update to correct the motion trajectory. The redundant detection target that fails iou matching is initialized as a new tracking object, and the initial state is unconfirmed.

[0080] Step 2-3-7: Tracking result output, specifically drawing the information of the successfully matched tracking object onto the current frame data, and storing the updated frame data into the corresponding position in the FIFO group buffer pool.

[0081] In this embodiment, if the remainder is not 0, the current frame is placed at a position corresponding to the remainder in the group according to the remainder result;

[0082] When the remainder is n-1, the inference ends by default, the inference result is obtained for reuse, and the n-1 frames of data except the first frame in the FIFO group buffer pool group are framed to ensure that the tracking frame in the output image is visible to the naked eye.

[0083] Step 3: For each group of data in the FIFO group buffer pool, the video encoding module performs H254 encoding in sequence, generates ts packets, and stores them in the ts buffer pool in sequence;

[0084] The specific steps include:

[0085] Step 3-1: extract each group of frame data from the FIFO group buffer pool in sequence;

[0086] Step 3-2: Encode the extracted frame data in sequence using H254;

[0087] Step 3-3: Construct TS packets for the encoded data so that each set of frame data is encapsulated in sequence;

[0088] Step 3-4: Store the constructed TS packets in the ts buffer pool in order for subsequent data transmission.

[0089] Step 4: The flow control module cyclically obtains ts packets from the ts buffer pool and transmits them to the host computer using the UDP protocol;

[0090] The specific steps include:

[0091] Step 4-1: Loop to get TS packets from the TS cache pool;

[0092] Step 4-2: Use UDP protocol to transmit TS packets;

[0093] Step 4-3: The host computer decodes the received TS packets and displays the video data in real time to complete the multi-target tracking process.

[0094] The multi-target tracking deployment method of a low-computing power edge device described in the present invention solves the technical problem that high-precision algorithms and real-time images cannot be achieved at the same time. The present invention uses the YOLOv5m model for target detection. Compared with lightweight models such as MobileNetV3, it provides higher detection accuracy. This high-precision detection directly improves the accuracy of tracking, because the accuracy of tracking depends to a large extent on the accuracy of initial detection. The present invention also uses the DeepSORT algorithm that retains the appearance feature extraction and matching functions, which significantly improves the stability of tracking and ensures the continuity and consistency of target tracking. The present invention successfully achieves a balance between accuracy and real-time performance on a low-computing power edge device by adopting a parallel strategy of reasoning and acquisition of a group of N frames, avoiding the dilemma of weighing between accuracy and real-time performance due to insufficient computing power of the device, and ensuring the smoothness of video output. The parallel processing of reasoning and acquisition not only improves the efficiency of reasoning, but also ensures the continuity of real-time video images, and will not affect the smoothness of the image due to the delay of the reasoning process, so that low-computing power devices can still achieve real-time and stable tracking of multiple targets while maintaining high detection accuracy.

Claims

1. A multi-target tracking deployment method for low-computing edge devices, characterized by: The steps include: Step 1: Build the edge computing system of the IoT system architecture, including the video acquisition module, reasoning and parallel processing module, video encoding module, and flow control module; The video acquisition module uses the v4l2 framework to continuously obtain real-time video frames from the camera and stores the real-time video frames in the FIFO frame buffer pool to obtain continuous frame data; Step 2: Build the YOLOv5m model and DeepSORT model in the inference and parallel processing module, use a FIFO group cache pool of N frames, use the first frame in the frame data for inference alone, and directly transfer the remaining frames; Determine whether the current frame is the first frame in the group by finding the modulus of the frame sequence of the frame data, and use the child thread to perform reasoning; Step 3: For each group of data in the FIFO group buffer pool, the video encoding module performs H254 encoding in sequence, generates ts packets, and stores them in the ts buffer pool in sequence; Step 4: The flow control module cyclically obtains ts packets from the ts buffer pool and transmits them to the host computer using the UDP protocol.

2. A multi-target tracking deployment method for low-computing edge devices as described in claim 1, characterized in that: When executing step 1, the specific steps are as follows: Step 1-1: The video acquisition module starts the v4l2 framework to obtain real-time video frames from the camera. The v4l2 framework is used to process the data stream of the video frames output by the camera and convert it into a standard format; Step 1-2: The v4l2 framework continuously captures real-time video frames from the camera, stores all video frames in sequence into the FIFO frame buffer pool, and obtains frame data; The FIFO frame buffer pool is used to temporarily store frame data so that the frame data is not lost and is stably and continuously output to the inference stage and encoding stage.

3. A multi-target tracking deployment method for low-computing edge devices as claimed in claim 1, characterized in that: When executing step 2, the specific steps include: Step 2-1: Storing frame data in the FIFO frame buffer pool in a manner of N frames as a group of data, and constructing a FIFO group buffer pool, where N is a positive integer and the value of N is a preset value; The first frame in each set of data will be used for inference processing, and the remaining frames will be stored in the corresponding positions in order to ensure the order and continuity of the frame data; Step 2-2: extract frame data from the FIFO group buffer pool, and perform a modulo operation based on the frame sequence of the frame data and the value of N: if the result is 0, it means that the current frame is the first frame in the group and needs to be inferred; otherwise, no inference processing is required; Step 2-3: For frame data with a remainder of 0, use a child thread to perform non-waiting inference processing.

4. A multi-target tracking deployment method for low computing power edge devices as described in claim 3, characterized in that: When executing step 2-3, the specific steps include: Step 2-3-1: Target detection, specifically inputting the frame data into the YOLOv5m target detection model, the YOLOv5m target detection model detects all targets in the current frame and outputs the category and coordinate information of each target; Step 2-3-2: Appearance feature extraction, specifically, intercepting the target image of each target according to the coordinates of the target detected in step 2-3-1; Each target image is input into the feature extraction network separately to extract the high-dimensional appearance features of each target; Step 2-3-3: Kalman prediction, specifically, for each tracked object that has been tracked, use the Kalman filter to predict its position in the current frame and generate a prediction box; Step 2-3-4: Appearance feature matching, specifically matching the high-dimensional appearance features of the target extracted in step 2-3-2 with the appearance features of each tracked object in the DeepSORT tracking queue; Use the minimum cosine distance and Mahalanobis distance to make judgments: if the match is successful, it is considered that the current target and the corresponding tracked object are the same object; Step 2-3-5: IOU matching, specifically, for targets whose appearance feature matching fails, IOU is used for matching, and the detection boxes of these targets that failed to match are compared with the predicted boxes of the unmatched tracking objects to determine the matching situation: if the IOU value exceeds the set threshold, the match is considered successful, and the status of the tracking object is updated; Step 2-3-6: Kalman update, specifically, for the tracked object that has been successfully matched through appearance feature matching or IOU matching, use its corresponding target detection box to update the Kalman filter and correct the motion trajectory of the tracked object; For unmatched targets, they are initialized as new tracking objects and assigned a new tracking ID; Step 2-3-7: Tracking result output, specifically drawing the information of the successfully matched tracking object onto the current frame data, and storing the updated frame data into the corresponding position in the FIFO group buffer pool.

5. The multi-target tracking deployment method of a low-computing power edge device according to claim 1, characterized in that: When executing step 3, the specific steps include: Step 3-1: extract each group of frame data from the FIFO group buffer pool in sequence; Step 3-2: Encode the extracted frame data in sequence using H254; Step 3-3: Construct TS packets for the encoded data so that each set of frame data is encapsulated in sequence; Step 3-4: Store the constructed TS packets in the ts buffer pool in order for subsequent data transmission.

6. A multi-target tracking deployment method for low computing power edge devices as claimed in claim 1, characterized in that: When executing step 4, the specific steps include: Step 4-1: Loop to get TS packets from the TS cache pool; Step 4-2: Use UDP protocol to transmit TS packets; Step 4-3: The host computer decodes the received TS packets and displays the video data in real time to complete the multi-target tracking process.

Images