A multi-target tracking method, device and processing equipment

Abstract

The present application provides a multi-target tracking method, apparatus and processing equipment. By designing a novel image processing model, the model includes a detection module obtained on the basis of the YOLOv8 algorithm and a tracking module obtained on the basis of a simple online real-time tracking algorithm centered on detection. In addition to achieving better multi-target detection effects in the early stage and multi-target tracking effects in the later stage, a good foundation is also laid for subsequent detail optimization that can be easily carried out, so that high-quality multi-target tracking needs can be met in practical applications.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a multi-target tracking method, apparatus and processing device. Background Art

[0002] Multiple Object Tracking (MOT) is the detection and tracking of multiple targets in a video when the number of targets is unknown. Multi-target tracking is a key technology in the field of computer vision. It is not only widely used in typical artificial intelligence fields such as autonomous driving, intelligent monitoring, and behavior recognition, but also extended to scientific research scenarios such as cell tracking, cluster target tracking, and blasting flying stone tracking, providing an important basis for subsequent timely analysis and decision-making.

[0003] However, compared with applications in artificial intelligence, occlusions are more frequent, there are more weak targets, and the motion patterns are more complex in scientific research scenes, which poses a huge challenge to fast, comprehensive and accurate multi-target tracking.

[0004] That is to say, when faced with the demand for high-quality multi-target tracking, existing multi-target tracking solutions have limitations in processing accuracy. Summary of the invention

[0005] The present application provides a multi-target tracking method, apparatus and processing equipment. By designing a novel image processing model, the model includes a detection module obtained on the basis of the YOLOv8 algorithm and a tracking module obtained on the basis of a simple online real-time tracking algorithm centered on detection. In addition to achieving better multi-target detection effects in the early stage and multi-target tracking effects in the later stage, a good foundation is also laid for subsequent detail optimization that can be easily carried out, so that high-quality multi-target tracking needs can be met in practical applications.

[0006] In a first aspect, the present application provides a multi-target tracking method, the method comprising:

[0007] Acquire continuous frame images;

[0008] Importing continuous frame images into a pre-configured image processing model, wherein the image processing model includes a detection module obtained based on the eighth version of the You Only Look Once version 8 (YOLOv8) algorithm and a tracking module obtained based on the Observation- Centric Simple Online and Realtime Tracking (OC-SORT) algorithm. The detection module performs multi-target detection on continuous frame images. The detection module includes three parts: a backbone network, a feature fusion, and a detection head. The tracking module performs multi-target tracking based on the tracking trajectory corresponding to the multi-target detection result output by the detection module. The multi-target tracking includes two steps: activating trajectory association and inactivating trajectory recovery. The activated trajectory indicates that the trajectory is continuously tracked in the current state, and the inactivated trajectory indicates that the tracking is interrupted.

[0009] Extract the multi-target tracking results output by the image processing model.

[0010] In a second aspect, the present application provides a multi-target tracking device, the device comprising:

[0011] An acquisition unit, used for acquiring continuous frame images;

[0012] A processing unit, used for importing continuous frame images into a preconfigured image processing model, wherein the image processing model includes a detection module obtained on the basis of a YOLOv8 algorithm and a tracking module obtained on the basis of a simple online real-time tracking algorithm centered on detection, the detection module performs multi-target detection on the continuous frame images, the detection module includes three parts: a backbone network, a feature fusion and a detection head, the tracking module performs multi-target tracking based on the tracking trajectory corresponding to the multi-target detection result output by the detection module, and the multi-target tracking includes two steps of activating trajectory association and inactivating trajectory recovery, wherein the activating trajectory indicates that the trajectory is continuously tracked in the current state, and the inactivating trajectory indicates that the tracking is interrupted;

[0013] The extraction unit is used to extract the multi-target tracking results output by the image processing model.

[0014] In a third aspect, the present application provides a processing device, including a processor and a memory, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the method provided in the first aspect of the present application or any possible implementation method of the first aspect of the present application is executed.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium, which stores multiple instructions, and the instructions are suitable for a processor to load to execute the method provided by the first aspect of the present application or any possible implementation of the first aspect of the present application.

[0016] From the above content, it can be concluded that the present application has the following beneficial effects:

[0017] For multi-target tracking, the present application designs a novel image processing model, which includes a detection module obtained based on the YOLOv8 algorithm and a tracking module obtained based on a simple online real-time tracking algorithm centered on detection. The detection module performs multi-target detection on continuous frame images of the input model, and the tracking module performs multi-target tracking based on the tracking trajectory corresponding to the multi-target detection results output by the detection module. When the early multi-target detection effect and the later multi-target tracking effect are good, a good foundation is also laid for the subsequent detail optimization that can be easily carried out, so that high-quality multi-target tracking needs can be met in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A flowchart of a multi-target tracking method of this application;

[0020] Figure 2 A schematic diagram of the structure of a multi-target tracking device of the present application;

[0021] Figure 3 A schematic diagram of the structure of the processing equipment of this application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0023] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described here can be implemented in an order other than that illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The naming or numbering of steps in this application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The process steps that have been named or numbered can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved.

[0024] The division of modules in this application is a logical division. There may be other division methods when it is implemented in actual applications. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection between modules can be electrical or other similar forms, which are not limited in this application. In addition, the modules or submodules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules, and some or all of the modules may be selected according to actual needs to achieve the purpose of the present application.

[0025] Before introducing the multi-target tracking method provided by this application, the background content involved in this application is first introduced.

[0026] The multi-target tracking method, device and computer-readable storage medium provided in the present application can be applied to processing equipment. By designing a novel image processing model, the model includes a detection module obtained on the basis of the YOLOv8 algorithm and a tracking module obtained on the basis of a simple online real-time tracking algorithm centered on detection. In addition to the good multi-target detection effect in the early stage and the multi-target tracking effect in the later stage, a good foundation is also laid for the subsequent detail optimization that can be conveniently carried out, so that high-quality multi-target tracking needs can be met in practical applications.

[0027] The multi-target tracking method mentioned in this application can be executed by a multi-target tracking device, or a server, physical host or user equipment (UE) and other different types of processing devices integrated with the multi-target tracking device. The multi-target tracking device can be implemented in hardware or software, and the UE can be a terminal device such as a smart phone, tablet computer, laptop computer, desktop computer or personal digital assistant (PDA), and the processing device can be set in the form of a device cluster.

[0028] It can be understood that in actual applications, the processing equipment that executes the multi-target tracking method of the present application, or the processing equipment equipped with the application service corresponding to the multi-target tracking method of the present application, considering that the present application scheme is mainly data processing based on existing data, its specific equipment type and equipment deployment form are relatively flexible and can be flexibly configured according to actual needs.

[0029] Among them, corresponding to the two aspects of model training and model application that may be involved in this application, the processing device can also be divided into two parts, the first part is dedicated to model training, and the second part is dedicated to model application.

[0030] In addition, if specific image acquisition operations are involved, the processing equipment should also have the software and hardware conditions required to acquire images. Of course, the focus of this application is on the configuration of the model rather than how to acquire the input images used by the model. Therefore, as introduced above, the images involved can use ready-made data.

[0031] Next, we will introduce the multi-target tracking method provided by this application.

[0032] First, see Figure 1 , Figure 1 A schematic diagram of a flow chart of the multi-target tracking method of the present application is shown. The X method provided by the present application may specifically include the following steps S101 to S103:

[0033] Step S101, acquiring continuous frame images;

[0034] It can be understood that multi-target tracking is the tracking of multiple specific targets in a dynamic video. Therefore, the model input image involves the video (composed of continuous frame images). In specific operations, it can be either the video itself containing continuous frame images or the continuous frame images obtained after video parsing. The configuration here can be performed according to actual needs. If it is the former, the model input part can also involve video parsing processing to parse the video into continuous frame images.

[0035] Among them, the continuous frame images obtained here (which can be in picture format or video format) obviously correspond to the multiple targets that can be detected and tracked by the subsequent image processing model, that is, the continuous frame images include multiple targets that can be detected and tracked by the image processing model.

[0036] In addition, the acquisition and processing of continuous frame images here is usually the acquisition and processing of ready-made data, that is, ready-made continuous frame images, which can be obtained by manual input, reception / download from other devices, or extraction from a local specific storage space. It is easy to understand that the specific acquisition method involved here is relatively flexible and can be considered to belong to the scope of the prior art, so the specific description will not be elaborated here.

[0037] In actual situations, the acquisition and processing of continuous frame images, or the scheme of the present application, is usually initiated with the image tracking task. The initiation method of the image tracking task is also relatively flexible. The image tracking task can be a manually initiated task, a task forwarded from other devices, or a task initiated autonomously by the device according to a pre-configured initiation strategy. This can also be flexibly configured, and the present application does not make any specific limitations.

[0038] Step S102, importing the continuous frame images into a pre-configured image processing model, wherein the image processing model includes a detection module obtained on the basis of the YOLOv8 algorithm and a tracking module obtained on the basis of a simple online real-time tracking algorithm centered on detection, the detection module performs multi-target detection on the continuous frame images, the detection module includes three parts: a backbone network, a feature fusion and a detection head, the tracking module performs multi-target tracking based on the tracking trajectory corresponding to the multi-target detection result output by the detection module, and the multi-target tracking includes two steps of activating trajectory association and inactivating trajectory recovery, wherein the activated trajectory indicates that the trajectory is continuously tracked in the current state, and the inactivated trajectory indicates that the tracking is interrupted;

[0039] The image processing model configured in this application includes two major parts: a detection module (also called a target detection network) and a tracking module (also called a target tracking network). This also corresponds to the situation where target tracking technology is implemented based on target detection technology.

[0040] It is easy to see that for the detection module, this application is specifically built on the basis of the YOLOv8 algorithm (also known as the YOLOv8 network architecture), that is, the YOLOv8 algorithm is used as the basic model.

[0041] For the tracking module, the present application is specifically built on the basis of a simple online real-time tracking algorithm centered on detection (also referred to as a simple online real-time tracking network architecture centered on detection), that is, a simple online real-time tracking algorithm centered on detection is used as the basic model.

[0042] Among them, it is necessary to understand that in the present application scheme, on the basis of the existing network architecture, further algorithm / model improvements can be made in details, so as to make it have better target detection / tracking effects. Therefore, it is also necessary to note that the present application does not simply select two algorithms to implement the detection module and tracking module configured in the present application. When the early image detection effect and the later image tracking effect are better, it also lays a good foundation for the subsequent detail optimization that can be easily carried out.

[0043] Below, the advantages of the YOLOv8 algorithm and the simple online real-time tracking algorithm centered on detection are specifically explained.

[0044] The YOLOv8 algorithm adopts the structure of backbone network (backbone) + feature fusion (neck) + detection head (head). The backbone network part mainly extracts features of the target information in the image through the convolutional network. The feature fusion part is responsible for multi-scale feature fusion of the extracted feature map, and finally passes the features to the detection head part for prediction. The detection head part is also called the detect layer, where the final classification confidence calculation and detection box regression prediction are performed. The representative technical features of each part are introduced as follows:

[0045] (1) Cross-stage partial connections

[0046] The cross-stage partial connection structure exists in both the backbone network and the feature fusion part of the YOLOv8 model. The core idea is to improve the information flow of the network through cross-stage partial connections, making the information more sufficient, improving the network's representation ability and efficiency, and reducing computational costs.

[0047] (2) Path aggregation network

[0048] The Path Aggregation Network (PANet) structure mainly exists in the feature fusion part of the YOLOv8 model. Its core idea is to use a top-down and bottom-up bidirectional fusion backbone network to promote the flow of information, while adding a "shortcut" between the bottom and top layers to shorten the path between layers. The Path Aggregation Network also includes two modules: adaptive feature pooling and fully connected fusion. Adaptive feature pooling can be used to aggregate features between different layers to ensure the integrity and diversity of features, while fully connected fusion can obtain more accurate prediction results.

[0049] (3) Based on the complete intersection-over-union loss function

[0050] The complete intersection over union (CIoU) loss function appears in the detection box loss of the detection head part, which is used to measure the difference between the detection box and the actual box. The calculation formula based on the complete intersection over union loss function is as follows:

[0051] ,

[0052] Among them, the subscript and Indicates forecast and actual situation, , and are the center, width and height of the corresponding bounding box respectively, and the parameters is the diagonal of the bounding box, It can be seen that the complete intersection-over-union loss function is a comprehensive indicator that takes into account three properties between bounding boxes: overlap ratio, distance between center points, and aspect ratio.

[0053] For the tracking module, it performs multi-target tracking based on the tracking trajectory corresponding to the multi-target detection results output by the detection module, and the data processing of multi-target tracking specifically includes two steps: activated trajectory association and inactivated trajectory recovery. Among them, the activated trajectory indicates that the trajectory is continuously tracked in the current state, and the inactivated trajectory indicates that the tracking is interrupted.

[0054] For the target tracking algorithm, in the specific operation, this application selects the best tracking algorithm from the three algorithms of Strong SORT, ByteTrack, and OC-SORT, which are finally put into use. These three tracking algorithms all adopt the Tracking by Detection (TBD) paradigm in multi-target tracking, and obtain the tracking trajectory by associating the target detection box through similarity, which is flexible, accurate, robust, and real-time.

[0055] By configuring a series of complex and diverse test samples, as well as a series of complex and diverse evaluation indicators, this application ultimately concludes that a simple online real-time tracking algorithm centered on detection has the best overall performance in terms of robustness in handling occlusion, comprehensiveness and accuracy in tracking, efficiency and effectiveness of real-time applications, etc.

[0056] In this way, based on the specific configuration and selection of the YOLOv8 algorithm and the simple online real-time tracking algorithm centered on detection, the subsequent model optimization design can achieve better processing results compared to other selectable algorithms. To a certain extent, the processing efficiency and processing accuracy have been effectively improved.

[0057] After obtaining the continuous frame images that need to be processed this time, they can be input into the image processing model pre-configured in this application, which includes two major modules, to carry out corresponding target detection processing and target tracking processing, and obtain corresponding multi-target tracking results.

[0058] Step S103, extracting the multi-target tracking result output by the image processing model.

[0059] It can be understood that after the image processing model completes the image processing work of the continuous frame images that need to be processed this time, the corresponding multi-target tracking results output by the model can be extracted.

[0060] At this point, the multi-target tracking results can be further processed by data application, such as storage, forwarding, displaying content, outputting prompts of completed image processing, or performing further data analysis.

[0061] It is easy to understand that the data application processing that continues to be carried out based on the multi-target tracking results obtained by the model can be carried out according to the pre-configured data application strategy and the real-time configured data application strategy. It is more flexible in actual application, and this application does not make any specific limitations on this.

[0062] from Figure 1 It can be seen from the illustrated embodiments that, for image detection and target tracking, the present application designs a novel image processing model, which includes a detection module obtained on the basis of the YOLOv8 algorithm and a tracking module obtained on the basis of a simple online real-time tracking algorithm centered on detection. The detection module performs multi-target detection on continuous frame images of the input model, and the tracking module performs multi-target tracking based on the tracking trajectory corresponding to the multi-target detection results output by the detection module. When the multi-target detection effect in the early stage and the multi-target tracking effect in the later stage are better, a good foundation is also laid for the subsequent detail optimization that can be conveniently carried out, so that high-quality multi-target tracking requirements can be met in practical applications.

[0063] Next, corresponding to the above-mentioned situation that the present application can further optimize the design of the image processing model in details, possible embodiments of the model configuration work involved in the present application in practical applications are elaborated in detail.

[0064] As mentioned above, the detection module mainly includes the backbone network part, the feature fusion part and the detection head part. In this regard, the work content of the detection module, in brief, includes:

[0065] Extract features through the backbone network part;

[0066] Multi-scale feature fusion is performed through the feature fusion part;

[0067] The target classification and location are obtained through the detection head part.

[0068] (1) For the backbone network, it is mainly composed of a stack of Convolution BatchNorm SiLU (CBS) modules and Cross Stage Partial (CSP) modules. Finally, a Spatial Pyramid Pooling-Fast (SPPF) module is used to extract features of different scales, thereby improving the detection accuracy of the model. The cross-stage partial connection module is mainly reflected in the C2F module, which is a combination of the C3 module and the Efficient Layer Aggregation Networks (ELAN) idea. The C2F module removes a two-dimensional convolution module from the C3 module, and also changes another convolution module to a normal convolution operation, modifies the number of channels, adds slicing operations, and jumps the original feature maps before and after to the final channel splicing module for connection operations. Through the parallelization of multiple gradient flow branches, richer gradient information, higher accuracy, and more reasonable latency are obtained.

[0069] From the above description, we can see that the core idea of ​​using the C2F module is to improve the information flow of the network through cross-stage partial connections, so that the information is more sufficient, the representation ability and efficiency of the network are improved, and the computational cost is reduced. The fast spatial pyramid pooling module realizes the fusion of local and global features at the featherMap level, which is used in the final step of feature extraction. It has the advantages of increasing the receptive field and extracting features of different scales, thereby improving the detection accuracy of the model.

[0070] (2) For the feature fusion part, as an exemplary implementation method, the present application introduces a bidirectional feature pyramid network in the detection module to replace the path aggregation network of the original feature fusion part. Compared with the path aggregation network, the bidirectional feature pyramid network takes the following improvement measures. Specifically, the bidirectional feature pyramid network optimizes cross-scale connections by deleting nodes with only one input edge, adding additional edges between input and output nodes at the same level, and treating each bidirectional path as a feature network layer and repeating it multiple times. At the same time, the bidirectional feature pyramid network introduces learnable weights to determine the importance of different input features. The above-mentioned improvement measures improve the effect of feature fusion while also increasing the complexity of the calculation.

[0071] On the one hand, in order to further enhance the flow of information between the same levels of the bidirectional feature pyramid network and improve the effect of feature fusion, for the feature fusion part in the detection module, on the basis of replacing the original path aggregation network with the bidirectional feature pyramid network, this application uses a dense connection method for the bidirectional feature pyramid network to connect nodes in the same feature layer across scales.

[0072] On the other hand, in order to compensate for the increase in computational complexity caused by dense connections, the output calculations of all intermediate nodes use the same simple superposition method as the original path aggregation network, and only introduce learnable weights for the last output layer nodes.

[0073] Specifically, for the bidirectional feature pyramid network, an additional weight is added to each input of the output node, and the corresponding feature fusion formula is as follows:

[0074] ,

[0075] in, represents the output of the output node, Represents the output node Inputs, express The corresponding weights, ≥0, represents the stability factor, =0.0001.

[0076] A series of experimental results show that the improved bidirectional feature pyramid network not only improves the comprehensive detection performance including precision, recall and average precision, but also reduces the number of parameters.

[0077] (3) For the detection head, a decoupled head structure is used to separate classification and detection, making the network training and reasoning more efficient. Its design idea is to decompose the object detection task into multiple independent subtasks, each of which is responsible for predicting the bounding box of a scale. This design enables the network to focus more on the feature information of each scale, improving the accuracy of object detection. At the same time, since each scale has an independent detector, the detection model can perform object detection at different scales in parallel, further improving the detection speed.

[0078] From the above analysis, we can see that the structure of the decoupling head has the advantages of flexibility, high efficiency, generalization and interpretability, which allows the model to flexibly learn the category and location information of the target and improve the generalization performance. At the same time, performing classification and regression separately is conducive to speeding up the training and reasoning speed of the model, so as to learn the target category and location information more quickly.

[0079] The general ideas for detection model training are roughly:

[0080] Configure training samples and annotate the required model processing results (the annotated results are the results that the model should output in theory);

[0081] In each round of model training, the training samples are input into the model so that the model can carry out corresponding prediction processing and realize forward propagation. Then, the loss function is calculated based on the prediction results and the annotation results. The model parameters are optimized according to the loss function calculation results to realize back propagation. In this way, when the preset model training requirements such as training time, training times or prediction accuracy are met, the model training can be satisfied and the model can be put into practical use.

[0082] In this process, it can be understood that the specific model training architecture and loss function involved can be adjusted according to actual needs.

[0083] In the specific operation, in order to achieve better model training effect, this application also makes further optimization configuration for the loss function.

[0084] As an exemplary embodiment here, for the detection module, during the model training process, the third version of the bounding box loss function based on the dynamic non-monotonic focusing mechanism can be used to replace the original loss function based on the complete intersection-over-union ratio.

[0085] It is understandable that the target detection performance depends on the design of the loss function, and the bounding box regression (BBR) loss, as an important part of the loss function, is of self-evident importance. The original YOLOv8 model uses complete intersection-over-union and distribution focal loss (DFL) as its bounding box regression loss function. Among them, the complete intersection-over-union loss function has better performance and higher accuracy than the traditional loss function, but the complexity of the implementation also increases.

[0086] In addition, most current work focuses on high-quality anchor boxes and is committed to enhancing the fitting ability of bounding box regression loss. However, the actual training dataset inevitably contains low-quality anchor boxes. If the bounding box regression of low-quality anchor boxes is simply strengthened, it will inevitably affect the model performance.

[0087] Based on the above problems, this application specifically selects the third version of the bounding box loss function based on the dynamic non-monotonic focusing mechanism (Wise Intersection over Union version 3, WIoUv3) algorithm with better effect as the bounding box regression loss function. The third version of the bounding box loss function based on the dynamic non-monotonic focusing mechanism combines the dynamic non-monotonic focusing mechanism with the loss function based on the intersection over Union (IoU), defines the outlier degree to evaluate the quality of the anchor frame, and gives a gradient gain allocation strategy. The outlier degree is negatively correlated with the quality of the anchor frame. If the outlier degree is too large or too small, a small gradient gain will be assigned to it to prevent harmful gradients generated by low-quality anchor frames, reduce the competitiveness of high-quality anchor frames, and make anchor frames of ordinary quality get more focus.

[0088] Compared with the original loss function based on complete intersection-over-union (IoU) ratio, the third version of the bounding box loss function based on the dynamic non-monotonic focusing mechanism reduces the computational time. In addition, it also takes into account the annotation quality of the anchor box, so that the anchor boxes of ordinary quality receive more attention in the bounding box regression. It can also reduce the adverse effects of high-quality and low-quality anchor boxes, thereby improving the regression positioning accuracy of the prediction box and achieving better regression effect.

[0089] In specific operations, experiments have proved that through the above optimization scheme, the optimization performance of target detection can promote the tracking ability of the target tracking algorithm, so that the accuracy, robustness and other performance of the tracking module are enhanced. However, in some occlusion problems, the tracking target will still be lost. Therefore, the tracking module can be further improved to further improve the ability to track occluded targets.

[0090] In this regard, we also focus on the loss function. For the tracking module, the distance intersection-over-union loss function is used to replace the original generalized intersection-over-union loss function in the matching of the trajectory prediction box and the target detection box.

[0091] It can be understood that the intersection over union loss function can measure the distance between the prediction box and the detection box and reflect the detection effect, so it is often used as an indicator of the detection algorithm. When the intersection over union is equal to 0, it cannot reflect the distance between the two, which will lead to a loss value of 0, the gradient disappears, and the model cannot be trained. It also cannot accurately reflect the overlap problem when the intersection over union is equal in different situations, so the generalized intersection over union (GIoU) ​​loss function is proposed.

[0092] The generalized intersection-over-union loss function is a distance metric like the intersection-over-union loss function. Unlike the intersection-over-union loss function, which only focuses on the overlapping area, the generalized intersection-over-union loss function focuses on the overlapping area and other non-overlapping areas. It can well reflect the overlap between the prediction box and the detection box. In particular, when the two target boxes overlap infinitely, the intersection-over-union ratio and the generalized intersection-over-union ratio are equal, both of which are 1.

[0093] The distance intersection over union (DIoU) loss function not only takes into account the overlap rate between the prediction box and the detection box, but also the distance between the center points, which is more in line with the detection box regression mechanism than the generalized intersection over union loss function. Compared with the generalized intersection over union loss function, the distance intersection over union loss function can still provide the moving direction for the target box while keeping the target box non-overlapping, and also improves the convergence speed and regression speed in special cases, because the distance intersection over union loss function can directly minimize the distance between the two target boxes.

[0094] The generalized intersection-over-union loss function is used in the simple online real-time tracking algorithm centered on detection. This application replaces it with the distance intersection-over-union loss function, which helps to further improve the tracking algorithm capability.

[0095] Additionally, for the tracking module, define the set of active tracks Continuous tracking of the trajectory in the current state, without activating the trajectory set The trajectory tracking is interrupted in the current state. Its work content is briefly described as follows:

[0096] (1) Initialize trajectory: Use the high-resolution frame detected in the first frame as the initial trajectory;

[0097] (2) Activation trajectory association: The activation trajectory set of the current frame predicted by the Kalman filter algorithm , and uses the Hungarian algorithm and the target detection frame of the current frame Associating, we get three sets, which are matched to the detection box The set of activated trajectories , No matching detection frame The set of activated trajectories And the detection box that does not match the activation track For the active trajectory set , set it as an inactive track and set the number of lost frames to 1;

[0098] (3) Inactive trajectory recovery: For the inactive trajectory set , search for the detection box directly near the last frame detection box before its target is lost , and get three sets, which are searched detection boxes Inactive tracks , no detection box was found The set of inactive tracks And the detection box of the inactive track is not searched For inactive tracks , performs detection-centered online smoothing by generating a virtual track between the detection box of the last frame before the target is lost and the detection box of the current frame searched, assisting the Kalman filter in making predictions and online smoothing parameters; for the inactive track set , add 1 to the number of lost frames and compare it with the lost threshold. If it does not exceed the threshold, it will be retained. If it exceeds the threshold, it means that the tracking is interrupted for a long time, so the track deletion operation is performed; for the detection frame , as the initial trajectory of the new trajectory;

[0099] (4) Repeat the above steps until all frames are traversed in sequence, and finally return the trajectory tracking result.

[0100] It can be seen from the above tracking steps that both the search for inactivated trajectories and the online smoothing of the corresponding Kalman filter parameters are centered on the detection value rather than the prediction value, which reduces the error accumulation from the moment the target is lost to the moment it reappears, thereby reducing the number of track losses caused by occlusion or nonlinear motion. The algorithm can take into account simplicity, online, real-time, and robustness to target occlusion, and adapt to nonlinear motion.

[0101] In addition, for the tracking module, this application believes that the byte method can also be introduced to associate data, effectively reducing the number of track identification switches caused by missed detection and maintaining track continuity. The byte method believes that the appearance of low-resolution frames is because the target is temporarily blocked and cannot be simply deleted, so almost all detection frames are retained for matching.

[0102] Specifically, in the early configuration conditions (since no track appears in the first frame, the algorithm will create tracking track objects for all high-scoring target detection frames. At this time, the created tracks are marked as tracked tracks, and the tracking tracks are gradually marked from the second frame), as an exemplary embodiment here, for the tracking module, the byte method association data is introduced, and the corresponding processing content may include byte-based activation track association and byte-based inactivated track recovery, wherein, according to the score, the target detection frame Divided into high score boxes and low frame ; Activate track Indicates that the track is continuously tracked in the current state and the track is not activated Indicates that tracking is interrupted.

[0103] For the step of activating trajectory association, the byte method is used to convert the detection box According to the score, it is divided into two parts: high score frame and low frame , and then activate the trajectory association in turn. The corresponding processing content includes:

[0104] (1) First, the current frame activation trajectory set predicted by the Kalman filter , using the Hungarian algorithm and high-resolution frame Matching is performed to obtain three sets, which are matched to high-score frames The set of activated trajectories , No high score frame was matched The set of activated trajectories And high-score boxes that are not matched to the activation track ;

[0105] (2) For the activation trajectory set , further adopting the Hungarian algorithm and low-resolution frame Matching is performed to obtain three sets, which are matched to low-score frames The set of activated trajectories , No low score frame matched The set of activated trajectories And low-scoring boxes that do not match the activation track .

[0106] For the steps of inactive trajectory recovery, the byte method is also used, and the corresponding processing process includes:

[0107] (1) First, for the inactive trajectory set , directly search for a high-score frame near the last frame detection frame before the target is lost , and get three sets, which are respectively searched for high-scoring boxes The set of inactive tracks , no high score frame was found The set of inactive tracks And the high-score boxes of inactive tracks are not searched ;

[0108] (2) For the inactive trajectory set , further searching for low-scoring boxes near the last frame detection box before its target is lost , and get three sets, which are searched for low score frames The set of inactive tracks , no low-scoring frames were found The set of inactive tracks And low-scoring boxes where inactive tracks are not searched ;

[0109] (3) For the inactive trajectory set and inactive track sets , performs detection-centered online smoothing;

[0110] (4) For the inactive trajectory set , add 1 to the number of lost frames and compare it with the lost threshold. If it does not exceed the threshold, it will be retained. If it exceeds the threshold, it means that the tracking is interrupted for a long time and the track deletion operation is performed;

[0111] (5) For high-resolution frames , as the initial trajectory of the new trajectory;

[0112] (6) For low-resolution frames , it is identified as the background that does not contain the target, and the detection frame is deleted.

[0113] In this way, the difference between high-resolution frames and low-resolution frames is used for data association, which effectively handles the occlusion problem in the target tracking process, reduces the number of track identification switches caused by missed detection, and maintains the continuity of track tracking. At the same time, this method only uses the motion model and does not use appearance similarity, and does not require training, so it is simple and efficient.

[0114] The above is an introduction to the multi-target tracking method provided by the present application. In order to facilitate better implementation of the target tracking method provided by the present application, the present application also provides a multi-target tracking device from the perspective of functional modules.

[0115] See also Figure 2 , Figure 2 This is a schematic diagram of the structure of a multi-target tracking device of the present application. In the present application, the multi-target tracking device 200 may specifically include the following structure:

[0116] An acquisition unit 201 is used to acquire continuous frame images;

[0117] The processing unit 202 is used to import the continuous frame images into a preconfigured image processing model, wherein the image processing model includes a detection module obtained on the basis of the YOLOv8 algorithm and a tracking module obtained on the basis of a simple online real-time tracking algorithm centered on detection, the detection module performs multi-target detection on the continuous frame images, the detection module includes three parts: a backbone network, a feature fusion and a detection head, the tracking module performs multi-target tracking based on the tracking trajectory corresponding to the multi-target detection result output by the detection module, and the multi-target tracking includes two steps of activating trajectory association and inactivating trajectory recovery, wherein the activating trajectory indicates that the trajectory is continuously tracked in the current state, and the inactivating trajectory indicates that the tracking is interrupted;

[0118] The extraction unit 203 is used to extract the multi-target tracking result output by the image processing model.

[0119] In an exemplary embodiment, for the feature fusion part in the detection module, based on replacing the original path aggregation network with a bidirectional feature pyramid network, a dense connection method is used to connect nodes in the same feature layer across scales.

[0120] In another exemplary embodiment, for a bidirectional feature pyramid network, an additional weight is added to each input of an output node, and the corresponding feature fusion formula is as follows:

[0121] ,

[0122] in, represents the output of the output node, Represents the output node Inputs, express The corresponding weights, ≥0, represents the stability factor, = 0.0001.

[0123] In yet another exemplary embodiment, for the detection module, a third version of the bounding box loss function based on a dynamic non-monotonic focusing mechanism is used to replace the original loss function based on complete intersection-over-union ratio during the model training process.

[0124] In yet another exemplary embodiment, for the tracking module, a distance intersection-over-union loss function is used to replace the original generalized intersection-over-union loss function in matching the trajectory prediction box with the target detection box.

[0125] In another exemplary embodiment, for the step of activating trajectory association, a byte method is used to convert the detection box According to the score, it is divided into two parts: high score frame and low frame , and then activate the trajectory association in turn. The corresponding processing content includes:

[0126] (1) First, the current frame activation trajectory set predicted by the Kalman filter , using the Hungarian algorithm and high-resolution frame Matching is performed to obtain three sets, which are matched to high-score frames The set of activated trajectories , No high score frame was matched The set of activated trajectories And high-score boxes that are not matched to the activation track ;

[0127] (2) For the activation trajectory set , further adopting the Hungarian algorithm and low-resolution frame Matching is performed to obtain three sets, which are matched to low-score frames The set of activated trajectories , No low score frame matched The set of activated trajectories And low-scoring boxes that do not match the activation track .

[0128] In another exemplary embodiment, for the step of not activating trajectory recovery, the corresponding processing includes:

[0129] (1) First, for the inactive trajectory set , directly search for a high-score frame near the last frame detection frame before the target is lost , and get three sets, which are respectively searched for high-scoring boxes The set of inactive tracks , no high score frame was found The set of inactive tracks And the high-score boxes of inactive tracks are not searched ;

[0130] (2) For the inactive trajectory set , further searching for low-scoring boxes near the last frame detection box before its target is lost , and get three sets, which are searched for low score frames The set of inactive tracks , no low-scoring frames were found The set of inactive tracks And low-scoring boxes where inactive tracks are not searched ;

[0131] (3) For the inactive trajectory set and inactive track sets , performs detection-centered online smoothing;

[0132] (4) For the inactive trajectory set , add 1 to the number of lost frames and compare it with the lost threshold. If it does not exceed the threshold, it will be retained. If it exceeds the threshold, it means that the tracking is interrupted for a long time and the track deletion operation is performed;

[0133] (5) For high-resolution frames , as the initial trajectory of the new trajectory;

[0134] (6) For low-resolution frames , it is identified as the background that does not contain the target, and the detection frame is deleted.

[0135] This application also provides a processing device from the perspective of hardware structure, see Figure 3 , Figure 3 301, a memory 302, and an input / output device 303. The processor 301 is used to execute the computer program stored in the memory 302 to implement the following Figure 1 The steps of the multi-target tracking method in the corresponding embodiment; or, the processor 301 is used to execute the computer program stored in the memory 302 to implement the following Figure 2 Corresponding to the functions of each unit in the embodiment, the memory 302 is used to store the processor 301 executing the above Figure 1 The computer program required by the multi-target tracking method in the corresponding embodiment.

[0136] Exemplarily, the computer program may be divided into one or more modules / units, one or more modules / units are stored in the memory 302, and executed by the processor 301 to complete the present application. One or more modules / units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the computer device.

[0137] The processing device may include, but is not limited to, a processor 301, a memory 302, and an input / output device 303. Those skilled in the art will appreciate that the illustration is merely an example of a processing device and does not constitute a limitation on the processing device, and may include more or fewer components than shown in the illustration, or a combination of certain components, or different components. For example, the processing device may also include a network access device, a bus, etc., and the processor 301, the memory 302, the input / output device 303, etc. are connected via a bus.

[0138] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the processing device, and uses various interfaces and lines to connect various parts of the entire device.

[0139] The memory 302 can be used to store computer programs and / or modules. The processor 301 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 302 and calling the data stored in the memory 302. The memory 302 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the processing device, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0140] When the processor 301 is used to execute the computer program stored in the memory 302, the following functions can be implemented:

[0141] Acquire continuous frame images;

[0142] Importing continuous frame images into a preconfigured image processing model, wherein the image processing model includes a detection module obtained on the basis of the YOLOv8 algorithm and a tracking module obtained on the basis of a simple online real-time tracking algorithm centered on detection. The detection module performs multi-target detection on continuous frame images. The detection module includes three parts: a backbone network, a feature fusion, and a detection head. The tracking module performs multi-target tracking based on the tracking trajectory corresponding to the multi-target detection result output by the detection module. The multi-target tracking includes two steps: activating trajectory association and inactivating trajectory recovery. The activated trajectory indicates that the trajectory is continuously tracked in the current state, and the inactivated trajectory indicates that the tracking is interrupted.

[0143] Extract the multi-target tracking results output by the image processing model.

[0144] Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the target tracking device, processing equipment and its corresponding units described above can refer to the following. Figure 1 The description of the target tracking method in the corresponding embodiment will not be repeated here.

[0145] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0146] To this end, the present application provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the present application as follows: Figure 1 The steps of the multi-target tracking method in the corresponding embodiment, the specific operation can refer to the following Figure 1 The description of the multi-target tracking method in the corresponding embodiment will not be repeated here.

[0147] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0148] Due to the instructions stored in the computer-readable storage medium, the present application can be executed. Figure 1 The steps of the multi-target tracking method in the corresponding embodiment, therefore, the present application can be implemented as follows Figure 1 The beneficial effects that can be achieved by the multi-target tracking method in the corresponding embodiment are detailed in the previous description and will not be repeated here.

[0149] The multi-target tracking method, device, processing equipment and computer-readable storage medium provided by the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A multi-target tracking method, characterized in that: The method comprises: Acquire continuous frame images; Importing the continuous frame images into a preconfigured image processing model, wherein the image processing model includes a detection module obtained on the basis of the YOLOv8 algorithm and a tracking module obtained on the basis of a simple online real-time tracking algorithm centered on detection, the detection module performs multi-target detection on the continuous frame images, the detection module includes three parts: a backbone network, a feature fusion, and a detection head, the tracking module performs multi-target tracking based on the tracking trajectory corresponding to the multi-target detection result output by the detection module, and the multi-target tracking includes two steps of activating trajectory association and inactivating trajectory recovery, wherein the activated trajectory indicates that the trajectory is continuously tracked in the current state, and the inactivated trajectory indicates that the tracking is interrupted; Extracting the multi-target tracking result output by the image processing model; For the step of activating trajectory association, a byte method is used to convert the detection box According to the score, it is divided into two parts: high score frame and low frame , and then perform the activation trajectory association in sequence, and the corresponding processing content includes: (1) First, the current frame activation trajectory set predicted by the Kalman filter , using the Hungarian algorithm and the high-resolution frame Matching is performed to obtain three sets, which are matched to the high-score frames The set of activated trajectories , the high score frame is not matched The set of activated trajectories and high-score boxes that do not match the activation trajectory ; (2) For the activation trajectory set , further using the Hungarian algorithm and the low-resolution frame Matching is performed to obtain three sets, which are matched to the low score frame The set of activated trajectories , the low score frame is not matched The set of activated trajectories and low-scoring boxes that do not match the activation track ; For the step of restoring the inactivated track, the corresponding processing process includes: (1) First, for the inactive trajectory set , directly search for the high-score frame near the last frame detection frame before the target is lost , and obtain three sets, which are respectively the high-scoring frames The set of inactive tracks , the high-scoring frame was not found The set of inactive tracks And the high-score boxes of inactive tracks are not searched ; (2) For the inactivated trajectory set , further search for the low score frame near the last frame detection frame before the target is lost , and obtain three sets, which are respectively searched for the low score frame The set of inactive tracks , the low-scoring frame was not found The set of inactive tracks And the low-scoring frame of the inactive track is not searched ; (3) For the inactivated trajectory set and the inactive track set , performs detection-centered online smoothing; (4) For the inactivated trajectory set , add 1 to the number of lost frames and compare it with the lost threshold. If it does not exceed the threshold, it will be retained. If it exceeds the threshold, it means that the tracking is interrupted for a long time and the track deletion operation is performed; (5) For the high-resolution frame , as the initial trajectory of the new trajectory; (6) For the low score frame , it is identified as the background that does not contain the target, and the detection frame is deleted.

2. The method according to claim 1, characterized in that: For the feature fusion part in the detection module, based on replacing the original path aggregation network with a bidirectional feature pyramid network, a dense connection method is used to connect nodes in the same feature layer across scales.

3. The method according to claim 2, characterized in that For the bidirectional feature pyramid network, an additional weight is added to each input of the output node, and the corresponding feature fusion formula is as follows: ， in, represents the output of the output node, Represents the output node Inputs, express The corresponding weight, ≥0, represents the stability factor, = 0.0001.

4. The method according to claim 1, characterized in that For the detection module, during the model training process, a third version of the bounding box loss function based on a dynamic non-monotonic focusing mechanism is used to replace the original loss function based on complete intersection-over-union ratio.

5. The method according to claim 1, characterized in that For the tracking module, a distance intersection-over-union loss function is used to replace the original generalized intersection-over-union loss function in matching the trajectory prediction frame with the target detection frame.

6. A multi-target tracking device, characterized in that: The device comprises: An acquisition unit, used for acquiring continuous frame images; A processing unit, used for importing the continuous frame images into a preconfigured image processing model, wherein the image processing model includes a detection module obtained on the basis of a YOLOv8 algorithm and a tracking module obtained on the basis of a simple online real-time tracking algorithm centered on detection, wherein the detection module performs multi-target detection on the continuous frame images, wherein the detection module includes three parts: a backbone network, a feature fusion, and a detection head, wherein the tracking module performs multi-target tracking based on a tracking trajectory corresponding to the multi-target detection result output by the detection module, wherein the multi-target tracking includes two steps of activating trajectory association and inactivating trajectory recovery, wherein the activated trajectory indicates that the trajectory is continuously tracked in the current state, and the inactivated trajectory indicates that the tracking is interrupted; An extraction unit, used for extracting the multi-target tracking result output by the image processing model; For the step of activating trajectory association, a byte method is used to convert the detection box According to the score, it is divided into two parts: high score frame and low frame , and then perform the activation trajectory association in sequence, and the corresponding processing content includes: (1) First, the current frame activation trajectory set predicted by the Kalman filter , using the Hungarian algorithm and the high-resolution frame Matching is performed to obtain three sets, which are matched to the high-score frames The set of activated trajectories , the high score frame is not matched The set of activated trajectories and high-score boxes that do not match the activation trajectory ; (2) For the activation trajectory set , further using the Hungarian algorithm and the low-resolution frame Matching is performed to obtain three sets, which are matched to the low score frame The set of activated trajectories , the low score frame is not matched The set of activated trajectories and low-scoring boxes that do not match the activation track ; For the step of restoring the inactivated track, the corresponding processing process includes: (1) First, for the inactive trajectory set , directly search for the high-score frame near the last frame detection frame before the target is lost , and obtain three sets, which are respectively the high-scoring frames The set of inactive tracks , the high-scoring frame was not found The set of inactive tracks And the high-score boxes of inactive tracks are not searched ; (2) For the inactivated trajectory set , further search for the low score frame near the last frame detection frame before the target is lost , and obtain three sets, which are respectively searched for the low score frame The set of inactive tracks , the low-scoring frame was not found The set of inactive tracks And the low-scoring frame of the inactive track is not searched ; (3) For the inactivated trajectory set and the inactive track set , performs detection-centered online smoothing; (4) For the inactivated trajectory set , add 1 to the number of lost frames and compare it with the lost threshold. If it does not exceed the threshold, it will be retained. If it exceeds the threshold, it means that the tracking is interrupted for a long time and the track deletion operation is performed; (5) For the high-resolution frame , as the initial trajectory of the new trajectory; (6) For the low score frame , it is identified as the background that does not contain the target, and the detection frame is deleted.

7. A processing device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and the processor executes the method according to any one of claims 1 to 5 when calling the computer program in the memory.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the method according to any one of claims 1 to 5.

Images