Target tracking method and system based on OpenCL acceleration
By using OpenCL to accelerate MNN framework on domestic GPUs, the problem that domestic GPUs cannot accelerate DeepSORT target tracking tasks is solved, high-performance and real-time target tracking are achieved, and the computing ecosystem of domestic GPUs is enhanced.
Patent Information
- Application Number
- CN202411356221.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-27
- Publication Date
- 2025-01-14
AI Technical Summary
Most domestic GPUs are unable to accelerate DeepSORT target tracking tasks on the PyTorch platform, resulting in performance degradation and real-time problems.
The MNN framework adopts OpenCL as the backend inference, converts the object detection model and feature extraction model into the MNN framework format, and deploys and optimizes it on the MNN framework to realize preprocessing of video frames, object detection and feature extraction, and accelerates the entire process through OpenCL. The reasoning process.
It improves the target tracking performance and real-time performance on domestic GPUs, enhances the autonomous and controllable capabilities of the general computing ecosystem of domestic GPUs, and ensures high-performance and real-time target tracking.
Smart Images
Figure CN119313704A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of target detection and tracking, and more specifically, to a target tracking method and system based on OpenCL acceleration. Background Art
[0002] Target tracking technology is of great significance in the field of computer vision. Target tracking algorithms play a key role in video surveillance, autonomous driving, enhanced realization, motion analysis, and intelligent transportation by combining observation models, motion models, and data association methods. In video surveillance, they can track targets in the area in real time to assist in safety monitoring and anomaly detection; in autonomous driving systems, they can identify and track other vehicles and pedestrians to provide support for intelligent driving decisions; in augmented reality, they can track virtual objects in the real world to achieve a more immersive augmented reality experience; in motion analysis, they can help track and evaluate athletes’ performance; in intelligent transportation systems, they can be used for traffic flow monitoring and traffic management to improve road safety and traffic efficiency. Target tracking technology is of great significance in the intelligent, efficient, and safe development of society.
[0003] SORT is a common target tracking method, which consists of a target detector and a tracker. However, when the tracked target is occluded or not detected for other reasons, the motion model prediction results and the observation model prediction results cannot be matched, and the tracking performance is significantly reduced. DeepSORT introduces the re-identification algorithm in deep learning to extract the appearance features of the detected target. After each frame of detection and tracking, the target detection ROI area is extracted for feature similarity matching of appearance features for each subsequent observation model prediction step, avoiding ID jumps caused by occlusion and missed detection.
[0004] Currently, the DeepSORT target tracking method is mainly implemented on the PyTorch framework, but the PyTorch framework only supports the CUDA computing platform when using GPU acceleration. CUDA is NVIDIA's proprietary technology and is not open source. As a result, most domestic GPUs cannot accelerate the DeepSORT target tracking task on the PyTorch platform. Summary of the invention
[0005] In view of the technical problems existing in the prior art, the present invention provides a target tracking method and system based on OpenCL acceleration, which solves the problem that most domestic GPUs cannot accelerate the DeepSORT target tracking task on the PyTorch platform.
[0006] According to a first aspect of the present invention, a target tracking method based on OpenCL acceleration is provided, comprising:
[0007] Build an MNN framework with OpenCL as the backend inference;
[0008] Converting the target detection model and the feature extraction model into a format recognizable by the MNN framework, and deploying the converted target detection model and the feature extraction model on the MNN framework;
[0009] Creating an interpreter and a session of the target detection model and the feature extraction model on the MNN framework, setting the inference parameters of the MNN framework based on the interpreter, and setting the model parameters of the target detection model and the feature extraction model;
[0010] The video frames are preprocessed and then input into the MNN framework, the target objects in the video frames are detected and identified based on the target detection model, the features of the identified target objects in the video frames are extracted based on the feature extraction model, and multi-target tracking is achieved based on the feature information of each target object in different video frames.
[0011] According to a second aspect of the present invention, there is provided a target tracking system based on OpenCL acceleration, comprising:
[0012] Building a module for building an MNN framework with OpenCL as the backend inference;
[0013] A deployment module, used to convert the target detection model and the feature extraction model into a format recognizable by the MNN framework, and deploy the converted target detection model and the feature extraction model on the MNN framework;
[0014] A setting module, used to create an interpreter and a session of the target detection model and the feature extraction model on the MNN framework, set the inference parameters of the MNN framework based on the interpreter, and set the model parameters of the target detection model and the feature extraction model;
[0015] The detection and tracking module is used to input the video frames into the MNN framework after preprocessing, detect the target objects in the video frames based on the target detection model, extract features of the target objects identified in the video frames based on the feature extraction model, and realize multi-target tracking based on the feature information of each target in different video frames.
[0016] The present invention provides a target tracking method and system based on OpenCL acceleration, which realizes the entire reasoning process under the domestic framework MNN framework and optimizes the overall performance. It aims to solve the application problem that the current domestic GPU cannot accelerate the target tracking task, increase the autonomous and controllable capability of the domestic GPU general computing ecology, and ensure high-performance and real-time target tracking. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 A flow chart of a target tracking method based on OpenCL acceleration provided by the present invention;
[0018] Figure 2 A schematic diagram of the logical process of target tracking based on OpenCL acceleration;
[0019] Figure 3 A schematic diagram of the structure of an OpenCL-accelerated target tracking system provided by the present invention. DETAILED DESCRIPTION
[0020] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention can be arbitrarily combined with each other to form a feasible technical solution. This combination is not subject to the constraints of the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in this field to achieve. When the combination of technical solutions is contradictory or cannot be achieved, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.
[0021] The traditional DeepSORT target tracking method is mainly implemented on the PyTorch framework, and mainly supports the CUDA computing platform for parallel acceleration. However, most domestic GPUs currently do not support CUDA, and CUDA is a closed-source technology. Therefore, improving the efficiency of target tracking through parallel acceleration of domestic GPUs is a thorny issue.
[0022] OpenCL is an open computing standard that supports multiple hardware architectures, not just NVIDIA's GPUs. Code written once can run on a variety of devices that support OpenCL, helping to reduce dependence on a single vendor and improve code flexibility and portability.
[0023] Based on this, the present invention proposes a DeepSORT target tracking method based on OpenCL, implements the entire reasoning process under the domestic framework MNN, improves computing performance, and optimizes real-time performance. It aims to solve the current application problem that domestic GPUs cannot accelerate target tracking tasks, increase the autonomous and controllable capabilities of the domestic GPU general computing ecosystem, and ensure high performance and real-time target tracking.
[0024] The present invention proposes a target tracking method based on OpenCL, see Figure 1and Figure 2 , the method mainly comprises the following steps:
[0025] Step 1: Build an MNN framework with OpenCL as the backend inference.
[0026] It can be understood that the MNN framework is initialized and compiled to generate the MNN framework including OpenCL as the backend reasoning for subsequent model reasoning.
[0027] Step 2: convert the target detection model and the feature extraction model into a format recognizable by the MNN framework, and deploy the converted target detection model and the feature extraction model on the MNN framework.
[0028] It is understandable that the target tracking task involves target detection and feature extraction, which correspond to the target detection model and feature extraction model respectively. The target detection model is used to detect and identify the target objects in the video frame, and the feature extraction model is used to extract the feature vectors of the target objects identified in the video frame.
[0029] Since the present invention builds a domestic MNN framework that supports OpenCL, it is necessary to convert the target detection model and the feature extraction model into a format that can be recognized by the MNN framework so that the model can be inferred based on MNN.
[0030] After the target detection model and feature extraction model are transformed, the target detection model and feature extraction model are optimized. The target detection model and feature extraction model both include multiple neural network operators. The target detection model and feature extraction model are optimized by fusing multiple neural network operators to reduce the computational and memory access overhead. Then, the target detection model and feature extraction are quantized while maintaining the model classification performance to reduce the computational complexity of the target detection model and feature extraction.
[0031] Step 3: Create an interpreter and a session of the target detection model and the feature extraction model on the MNN framework, set the inference parameters of the MNN framework based on the interpreter, and set the model parameters of the target detection model and the feature extraction model.
[0032] It is understandable that the interpreter (Interpreter) for target detection and feature extraction is created, the reasoning parameters are set and the session is created. The reasoning parameters mainly include the GPU and CPU operating environment parameters of the MNN framework, as well as the model parameters of the target detection model and feature extraction model, so that the speed and accuracy of target detection tracking are maintained in a relatively good state. Set the parameters such as searching for the OpenCL backend for reasoning and saving the reasoning mode. Since multiple reasoning can share the same model and the reasoning initialization phase is relatively long, saving parameters in advance can improve the real-time recognition.
[0033] Step 4: Preprocess the video frame and input it into the MNN framework, detect the target object in the video frame based on the target detection model, extract features of the identified target object in the video frame based on the feature extraction model, and realize multi-target tracking based on the feature information of each target in different video frames.
[0034] It is understandable that before performing target tracking inference, the captured video frames are first preprocessed, mainly including loading and modifying the size, RGB channels and normalization of the input video frame images, and preprocessing them into the format required by the model.
[0035] The preprocessed video frames are input into the MNN framework, and reasoning operations are performed based on the target detection model, that is, the target objects in the video frames are detected and identified to obtain the target detection results in the video frames. The target detection results include the type information and position coordinate information of the identified target objects.
[0036] The video frames marked with the category information and position coordinate information of the target object are input into the feature extraction model. The feature extraction model verifies the boundary conditions of each target based on the target position coordinates to ensure that the identified target coordinates are in the video frame; the feature vector of each target identified by target detection is extracted based on the feature extraction model.
[0037] Based on the target detection results of multiple frames and the feature vectors of each target object in each frame, the corresponding trajectory is created according to the target detection result of the first frame. The trajectory state of the first frame is initialized to an undetermined state. The target detection results and feature vectors of each subsequent frame are matched with the target detection results and feature vectors of the previous multiple frames to establish the trajectory of each target and achieve tracking of each target.
[0038] When establishing the trajectory of each target object, the Hungarian algorithm is used to match the predicted trajectory with the target detection result in the current frame in combination with the observation model and the motion model. When the matching result is a mismatch, the target detection result and the trajectory in the current frame cannot be matched. When the target detection results of multiple frames continuously mismatch the predicted trajectory, the target ID is deleted. When the matching result is not detected before, a new trajectory is assigned to the target. When the matching result is a successful match, the target detection result of the current frame is added to the matched trajectory, the Kalman filter state is updated, and the trajectory library is updated. Finally, the unique identifier, bounding box coordinates, category label and confidence of each tracked target are output to complete the tracking of each target in the video.
[0039] See also Figure 3 , provides a target tracking system based on OpenCL, the system comprising:
[0040] Building module 301, used to build an MNN framework including OpenCL as backend reasoning;
[0041] A deployment module 302, used to convert the target detection model and the feature extraction model into a format recognizable by the MNN framework, and deploy the converted target detection model and the feature extraction model on the MNN framework;
[0042] A setting module 303, used to create an interpreter and a session of the target detection model and the feature extraction model on the MNN framework, set the inference parameters of the MNN framework based on the interpreter, and set the model parameters of the target detection model and the feature extraction model;
[0043] The detection and tracking module 304 is used to pre-process the video frame and input it into the MNN framework, detect each target object in the video frame based on the target detection model, extract features of the identified target object in the video frame based on the feature extraction model, and realize multi-target tracking based on the feature information of each target object in different video frames.
[0044] It can be understood that the target tracking system based on OpenCL acceleration provided by the present invention corresponds to the target tracking method based on OpenCL acceleration provided by the aforementioned embodiments. The relevant technical features of the target tracking system based on OpenCL acceleration can refer to the relevant technical features of the target tracking method based on OpenCL acceleration, which will not be repeated here.
[0045] The embodiment of the present invention provides a target tracking method and system based on OpenCL acceleration. The environment is built based on the domestic graphics card Lingjiu GP201 and the domestic platform, and the target recognition and feature extraction model is accelerated and optimized based on OpenCL; then the DeepSORT target tracking method is implemented under the domestic reasoning framework MNN to perform the target tracking task, the process of the target tracking method is optimized, and the operation efficiency of the entire target tracking method is improved; finally, through model optimization, the model operation reasoning efficiency is improved, so that the entire method has better real-time performance when implemented, and the intelligent ecological library of domestic GPUs is expanded. These characteristics reflect the advancement and application value of this patent.
[0046] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0047] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0048] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0049] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0050] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0051] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0052] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A target tracking method based on OpenCL acceleration, characterized in that: include: Build an MNN framework with OpenCL as the backend inference; Converting the target detection model and the feature extraction model into a format recognizable by the MNN framework, and deploying the converted target detection model and the feature extraction model on the MNN framework; Creating an interpreter and a session of the target detection model and the feature extraction model on the MNN framework, setting the inference parameters of the MNN framework based on the interpreter, and setting the model parameters of the target detection model and the feature extraction model; The video frames are preprocessed and then input into the MNN framework, the target objects in the video frames are detected and identified based on the target detection model, the features of the target objects identified in the video frames are extracted based on the feature extraction model, and multi-target tracking is achieved based on the feature information of each target object in different video frames.
2. The target tracking method based on OpenCL acceleration according to claim 1, characterized in that: The converting of the target detection model and the feature extraction model into a format recognizable by the MNN framework also includes optimizing the target detection model and the feature extraction model respectively, wherein the model optimization includes: Multiple operators in the model are fused to form the target detection model and the feature extraction model, and the target detection model and the feature extraction model are lightweighted respectively while maintaining the model performance.
3. The target tracking method based on OpenCL acceleration according to claim 1, characterized in that: The reasoning parameters of the MNN framework are set based on the interpreter, including the GPU where the MNN framework is located, the CPU operating environment and the reasoning efficiency.
4. The target tracking method based on OpenCL acceleration according to claim 1, characterized in that: The preprocessing of the video frame and inputting it into the MNN framework includes: The input video frame is loaded and modified in terms of size, RGB channels and normalization, and the video frame is preprocessed into the target detection model and the ROI region in the video frame is processed into the format required by the feature extraction model.
5. The target tracking method based on OpenCL acceleration according to claim 4, characterized in that: The detection result after the target detection model detects the target object in the video frame includes the type information and position coordinate information of each target object in the video frame.
6. The target tracking method based on OpenCL acceleration according to claim 5, characterized in that: The extracting features of each target in the video frame based on the feature extraction model includes: Inputting a video frame marked with the target category and target position coordinates of each target into the feature extraction model, the feature extraction model verifies the boundary conditions of each target based on the target position coordinates so that the identified target coordinates are in the video frame; A feature vector of each identified target object in the video frame is extracted based on the feature extraction model.
7. The target tracking method based on OpenCL acceleration according to claim 6, characterized in that: The tracking of multiple targets based on the feature information of each target in different video frames includes: Create corresponding trajectories based on the target detection results of the first frame. Initialize the trajectory state of the first frame to be undetermined. Match the target detection results and feature vectors of each subsequent frame with the target detection results and feature vectors of the previous multiple frames to establish the trajectory of each target and track each target.
8. The target tracking method based on OpenCL acceleration according to claim 7, characterized in that: The target matching is performed based on the target detection results and feature vectors of each subsequent frame and the target detection results and feature vectors of the previous multiple frames to establish the trajectory of each target, including: Combined with the observation model and the motion model, the predicted trajectory is matched with the target detection result in the current frame using the Hungarian algorithm. When the matching result is a mismatch, the target detection result in the current frame and the trajectory cannot be matched. When the target detection results of multiple frames continuously mismatch the predicted trajectory, the target ID is deleted; when the matching result is not detected before, a new trajectory is assigned to the target; when the matching result is a successful match, the target detection result of the current frame is added to the matched trajectory, the Kalman filter state is updated, and the trajectory library is updated; Outputs a unique identifier, bounding box coordinates, class label, and confidence for each tracked object.
9. A target tracking system based on OpenCL acceleration, characterized in that: include: Building a module for building an MNN framework with OpenCL as the backend inference; A deployment module, used to convert the target detection model and the feature extraction model into a format recognizable by the MNN framework, and deploy the converted target detection model and the feature extraction model on the MNN framework; A setting module, used to create an interpreter and a session of the target detection model and the feature extraction model on the MNN framework, set the inference parameters of the MNN framework based on the interpreter, and set the model parameters of the target detection model and the feature extraction model; The detection and tracking module is used to input the video frames into the MNN framework after preprocessing, detect the target objects in the video frames based on the target detection model, extract features of the target objects identified in the video frames based on the feature extraction model, and realize multi-target tracking based on the feature information of each target in different video frames.