Video multi-target tracking method, system, device and medium in complex scenes

Abstract

The disclosed embodiment provides a method, system, device and medium for video multi-target tracking in complex scenes, which belongs to the field of computer vision technology, and specifically includes: step 1, constructing a multi-target tracking model based on the YOLOX detection model; step 2, adding a re-identification network with key point detection and alignment operations to the tracking network; step 3, in the data association stage, redesigning the trade-off between the predicted value and the observed value of the Kalman filter in the tracking network, so that the Kalman filter is centered on the observation; step 4, designing the IOU measurement method TIOU measurement containing time series and adding it to the multi-target tracking model; step 5, inputting the data sets of each category into the redesigned multi-target tracking model to obtain the multi-target tracking result. Through the solution disclosed in the present invention, the tracking accuracy MOTA and HOTA are effectively improved, and the ID switching can be greatly reduced, thereby improving the recognition efficiency and tracking speed.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and more particularly to a method, system, device, and medium for video multi-target tracking in complex scenarios. Background Art

[0002] Currently, with the continuous advancements in intelligent robotics, autonomous driving, smart security, and intelligent transportation, the reliance on computer vision technology is increasing. Video multi-object tracking has become a key technology in these fields. Deep learning, currently the most popular branch of machine learning, has seen rapid developments in its most representative models, convolutional neural networks and recurrent neural networks, leading to remarkable progress in computer vision and natural language processing. Within computer vision, object detection is a fundamental task and a focus of considerable attention. The iterative advancements in deep convolutional networks, coupled with powerful backbone networks, advanced detection models, and large-scale datasets, have pushed object detection to new heights. However, when applying still-image-based object detection to frame-by-frame video sequences, performance is often unsatisfactory due to the degradation of appearance features caused by occlusion, motion blur, and object deformation. Object detection involves training a large number of images of various objects to obtain an excellent detection model, then detecting, identifying, and classifying different types of objects. As a multi-task learning problem, object detection requires the joint optimization of object classification and localization. Classification focuses on learning salient features that distinguish different categories, while localization aims to accurately locate the entire object and its boundaries. These two independent tasks often require different spatial features. The initial method used was a two-stage detector, which designed two independent parallel branches to predict positioning and classification. Due to the lack of interactivity between the two tasks, the predictions are inconsistent. After calculating a large number of potential bounding boxes, the true bounding boxes must be screened out using IOU. This makes the calculation very complex and difficult to achieve end-to-end training. In addition, the existing IOU method lacks temporal information when migrating to the field of multi-target tracking.

[0003] It can be seen that there is an urgent need for an efficient, simple, and highly accurate video multi-target tracking method in complex scenes. Summary of the Invention

[0004] In view of this, the embodiments of the present disclosure provide a method, system, device and medium for video multi-target tracking in complex scenarios, which at least partially solve the problems of poor recognition efficiency and accuracy in the prior art.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for tracking multiple objects in a video in a complex scene, including:

[0006] Step 1: Build a multi-target tracking model based on the YOLOX detection model, wherein the multi-target tracking model includes a feature extraction network, a target classification network, and a tracking network;

[0007] Step 2: Add a re-identification network with key point detection and alignment operations to the tracking network;

[0008] Step 3: In the data association phase, the trade-off between the predicted value and the observed value of the Kalman filter in the tracking network is redesigned so that the Kalman filter is observation-centric.

[0009] Step 4: Design the time-series IOU metric and add it to the multi-target tracking model.

[0010] Step 5: Input the data sets of each category into the redesigned multi-target tracking model to obtain the multi-target tracking results.

[0011] According to a specific implementation of the embodiment of the present disclosure, the feature extraction network includes the Transformer model and a convolutional neural network;

[0012] The classification network includes a feature decoder and a classifier.

[0013] According to a specific implementation of the embodiment of the present disclosure, the steps of step 4 specifically include:

[0014] Based on CIOU, the speed of the tracking target is introduced, that is, the size and direction of the displacement of the target in the unit continuous frame, and the TIOU measurement is obtained. The representation of the target includes the horizontal coordinate of the center point, the vertical coordinate of the center point, the height of the target box, the width of the target box, the speed of the target in the previous frame, and the speed of the target in the next frame.

[0015] According to a specific implementation of the embodiment of the present disclosure, the expression of the Kalman filter is:

[0016]

[0017]

[0018]

[0019]

[0020] P t|t =(IK t H t )P t|t-1

[0021] in, is the fused tracking result at time t-1, is the predicted value of the Kalman filter at time t based on the tracking result at time t-1, F t is the transition coefficient, H t is the observation value detected by the target detector at time t, Q t 、R t is the process noise and observation noise, P is the covariance matrix, since The predicted value contains the error accumulation over time, taking random loss To reduce the accumulation of errors over time, or to discard them directly when the confidence level of the observations is high

[0022] According to a specific implementation of the embodiment of the present disclosure, step 5 specifically includes:

[0023] According to the data set of each category, the ReID features of the target's bounding box and the TIOU calculation cost matrix are determined, and then the Kalman filter and Hungarian algorithm are used for target tracking. At the same time, secondary data association is used for low-scoring detection boxes to obtain multi-target tracking results.

[0024] According to a specific implementation of an embodiment of the present disclosure, the selection method includes randomly losing or directly selecting the current frame target positioning frame based on the target frame with a larger confidence level.

[0025] In a second aspect, the embodiments of the present disclosure provide a video multi-target tracking system in complex scenarios, including:

[0026] A construction module is used to build a multi-target tracking model based on the YOLOX detection model, wherein the multi-target tracking model includes a feature extraction network, a target classification network and a tracking network;

[0027] The first design module is used to add a re-ID network with keypoint detection and alignment operations to the tracking network;

[0028] The second design module is used to redesign the trade-off between the predicted value and the observed value of the Kalman filter in the tracking network during the data association phase, so that the Kalman filter is observation-centric;

[0029] The third design module is used to design the time-series IOU measurement method TIOU measurement and add it to the multi-target tracking model;

[0030] The target tracking module is used to input data sets of various categories into the redesigned multi-target tracking model to obtain multi-target tracking results.

[0031] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0032] At least one processor; and a memory communicatively connected to the at least one processor; wherein,

[0033] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video multi-target tracking method in complex scenes in the aforementioned first aspect or any implementation of the first aspect.

[0034] In a fourth aspect, an embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the video multi-target tracking method in complex scenarios in the aforementioned first aspect or any implementation of the first aspect.

[0035] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the method for video multi-target tracking in complex scenes in the aforementioned first aspect or any implementation of the first aspect.

[0036] The video multi-target tracking solution in complex scenarios in the embodiment of the present disclosure includes: step 1, constructing a multi-target tracking model based on the YOLOX detection model, wherein the multi-target tracking model includes a feature extraction network, a target classification network and a tracking network; step 2, adding a re-identification network with key point detection and alignment operations to the tracking network; step 3, in the data association stage, redesigning the trade-off between the predicted value and the observed value of the Kalman filter in the tracking network, so that the Kalman filter is centered on the observation; step 4, designing the IOU measurement method TIOU measurement containing time series and adding it to the multi-target tracking model; step 5, inputting the data sets of each category into the redesigned multi-target tracking model to obtain the multi-target tracking results.

[0037] The beneficial effects of the embodiments of the present disclosure are as follows: through the scheme of the present disclosure, a new IOU calculation formula TIOU is designed from the perspective of kinematics to realize a measurement method with temporal information that can characterize neighborhood relationships, and better utilize the temporal context information of the video to perform multi-target tracking; by adding key point detection and feature alignment operations, the situation where the ReID feature recognition in the shared feature is not obvious is improved, so that the addition of the ReID feature has a significant improvement on the final tracking result; at the same time, the observation-centered Kalman filter is realized by randomly losing the Kalman filter prediction value or ignoring the Kalman filter prediction value under the target frame with higher detection confidence, which can effectively avoid the tracking drift caused by the accumulation of Kalman filter prediction error over time. Through the scheme of this example, the tracking accuracy MOTA and HOTA can be effectively improved in multi-target tracking, and the ID switching can be reduced, thereby improving the tracking speed and tracking accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0039] Figure 1 A flowchart of a method for video multi-target tracking in complex scenarios provided by an embodiment of the present disclosure;

[0040] Figure 2 A schematic diagram of a result of obtaining CIOU through IOU position relationship provided by an embodiment of the present disclosure;

[0041] Figure 3 A schematic diagram of a TIOU provided in an embodiment of the present disclosure;

[0042] Figure 4 A schematic diagram of the structure of a video multi-target tracking system in complex scenarios provided by an embodiment of the present disclosure;

[0043] Figure 5 A schematic diagram of an electronic device provided in an embodiment of the present disclosure;

[0044] Figure 6 A schematic diagram of the data processing flow of a multi-target tracking system provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0045] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0046] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0047] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement an apparatus and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this apparatus and / or practice this method.

[0048] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0049] Additionally, in the following description, specific details are provided to provide a thorough understanding of the examples. However, one skilled in the art will appreciate that the aspects described can be practiced without these specific details.

[0050] The embodiments of the present disclosure provide a method for video multi-target tracking in complex scenarios. The method can be applied to multi-target recognition and tracking processes in scenarios such as unmanned driving, intelligent security, mobile robots, intelligent elderly care, human-computer interaction, and sports video analysis.

[0051] See also Figure 1 , is a flow chart of a method for video multi-target tracking in complex scenes provided by an embodiment of the present disclosure. Figure 1 As shown, the method mainly includes the following steps:

[0052] Step 1: Build a multi-target tracking model based on the YOLOX detection model, wherein the multi-target tracking model includes a feature extraction network, a target classification network, and a tracking network;

[0053] Furthermore, the feature extraction network includes the Transformer model and a convolutional neural network;

[0054] The classification network includes a feature decoder and a classifier.

[0055] During specific implementation, the detection network of the multi-target tracking model can select the anchor-free YOLOX and CenterNet detection methods. The feature extraction network includes the Transformer model and the convolutional neural network. The tracking classification network includes a feature decoder and a classifier, which divides the targets into categories such as pedestrians, bicycles, motorcycles, tricycles, trucks, cars, and automobiles. The tracking network is mainly completed by the Kalman filter and the Hungarian algorithm.

[0056] The algorithm is compared with the one-stage detection algorithm YOLOv5. Anchor-based object detection algorithms predefine anchor boxes based on the characteristics of the scene and dataset, and then generate object bounding boxes from the predefined anchor boxes through methods such as classification and regression. While this algorithm has the advantage of relatively mature technology, it also has the disadvantages of poor generalization and computational redundancy. Anchor-free object detection algorithms directly predict the probability that each pixel in the image belongs to the object to be detected and the object's bounding box information, and then generate a bounding box based on this information. This algorithm has the advantages of strong generalization, a simpler framework, and high accuracy in detecting objects of unusual scales, making it suitable for multi-scale and small object detection. However, it has the disadvantage of lower accuracy than anchor-based algorithms.

[0057] For example, the backbone network can use ResNet-34, with an enhanced version of Deep Layer Aggregation (DLA) applied to the backbone to fuse multiple layers of features. It has more skip connections between low-level and high-level features, similar to the Feature Pyramid Network (FPN). The convolutional layers in the upsampling are replaced by deformable convolutions, allowing the receptive field to be dynamically adjusted based on the scale and pose of the target, and also utilizing the alignment of target keypoints.

[0058] At the same time, in data processing, the official MOT16, MOT17, and MOT20 were selected as pedestrian tracking datasets, and a variety of other data were selected as pre-training in the training set, such as Caltech, CUHK-SYSU, PRW, DETRAC, ETHZ, Crowdhuman, Cityscapes, visDrone2019, dancetrack, etc.

[0059] Step 2: Add a re-identification network with key point detection and alignment operations to the tracking network;

[0060] In practice, a re-identification network with keypoint detection and alignment can be added to the tracking network to extract unique, invariant features of the target, distinguishing similar targets and associating them with targets that are more similar than their previous frames. Since the object detection network uses an anchor-free detection network to detect targets, it is essentially a keypoint detection network. Besides being used for detection tasks, CenterNet can be expanded to keypoint detection by adding two additional branches to its original branch.

[0061] For the COCO dataset, each person has 17 key points. The offsets of the 17 key points can be directly predicted through the center point, and then a heatmap is predicted for matching, because directly regressing the key points is not as accurate as using a heatmap.

[0062] The alignment operation mainly detects the pedestrian's hands, feet, eyes, mouth, head, etc. and marks them with feature points. It uses hierarchical alignment from coarse to fine deformation to gradually update adjacent frames so that they are aligned with the target of the current frame at the feature level.

[0063] Step 3: In the data association phase, the trade-off between the predicted value and the observed value of the Kalman filter in the tracking network is redesigned so that the Kalman filter is observation-centric.

[0064] Furthermore, the expression of the Kalman filter is

[0065]

[0066]

[0067]

[0068]

[0069] P t|t =(IK t H t )P t|t-1

[0070] in, is the fused tracking result at time t-1, is the predicted value of the Kalman filter at time t based on the tracking result at time t-1, F t is the transition coefficient, H t is the observation value detected by the target detector at time t, Q t 、R t is the process noise and observation noise, P is the covariance matrix, since The predicted value contains the error accumulation over time, taking random loss To reduce the accumulation of errors over time, or to discard them directly when the confidence level of the observations is high

[0071] Furthermore, the selection method includes randomly losing or directly selecting the current frame target positioning frame based on the target frame with greater confidence.

[0072] In the specific implementation, considering that the essence of the error accumulation caused by the Kalman filter is prediction-centered, and due to the advancement of the current target detection algorithm, the reliability of its detection results can exceed the reliability of the prediction results of the Kalman filter prediction model, a Kalman filter centered on observation is designed.

[0073] In the embodiment, the multi-target tracking adopts Kalman filtering in the data association stage, and its formula is:

[0074]

[0075]

[0076]

[0077]

[0078] P t|t =(IK t H t )P t|t -1

[0079] In the formula is the fused tracking result at time t-1, is the predicted value of the Kalman filter at time t based on the tracking result at time t-1, F t is the transition coefficient, H t is the observation value detected by the target detector at time t, Q t 、R t is the process noise and observation noise, and P is the covariance matrix. The predicted value contains the error accumulation over time, taking random loss To reduce the accumulation of errors over time, or to discard them directly when the confidence level of the observations is high (u, v) represents the horizontal and vertical coordinates of the target center point, s represents the area of ​​the target box, and r is the aspect ratio of the bounding box. is the relevant time derivative, z = [u, v, w, h, c, p, q], w, h, c represent the target box width, height and target box confidence respectively, p is the slope of the target in the previous frame, and q is the target slope in the current frame.

[0080] In this embodiment, pedestrian motion can be considered uniform linear motion, so the p and q values ​​can be approximately equal. This provides a criterion for distinguishing a target from surrounding objects. The p and q values ​​of a target are roughly equal. If adjacent surrounding objects have different motion trajectories, the p and q values ​​of the surrounding objects will be completely different from those of the target. If the surrounding objects have the same motion trajectory, they can be easily distinguished based on the different center positions of the targets.

[0081] In the detection branch, we design five detection heads: heatmap, object center coordinates, bounding box size, previous frame object center slope, and current frame object center slope. Each detection head is implemented by applying a 3×3 convolution to the output features of the DLA-34, followed by a 1×1 convolution layer for the final object. The loss function for the heatmap is a pixel-by-pixel logistic regression:

[0082]

[0083] Where M is the estimated heat map, α, β are the predetermined parameters of corner loss, and N represents the number of objects in the current frame.

[0084] The correlation between different targets should be smaller than the correlation between the same targets. The loss of ReID features is defined as:

[0085]

[0086] Where K is the number of categories.

[0087]

[0088]

[0089] L detection =L heat +L box +L ratio

[0090]

[0091] Where O i , S i are the estimated sizes and offsets of the corresponding positions,

[0092] p i ,q i is the target slope of the previous frame and the current frame; w1, w2 are learnable parameters to balance the two tasks, L ratio The L1 norm used.

[0093] Step 4: Design the time-series IOU metric and add it to the multi-target tracking model.

[0094] Based on the above embodiment, the steps of step 4 specifically include:

[0095] Based on CIOU, the speed of the tracking target is introduced, that is, the size and direction of the displacement of the target in the unit continuous frame, and the TIOU measurement is obtained. The representation of the target includes the horizontal coordinate of the center point, the vertical coordinate of the center point, the height of the target box, the width of the target box, the speed of the target in the previous frame, and the speed of the target in the next frame.

[0096] When implementing it specifically, Figure 2 and Figure 3 As shown, considering that the displacement of the same target in two consecutive frames is very small, when using IOU measurement, the IOU ratio of the current frame to the previous frame and the IOU ratio of the adjacent target to the previous frame are very likely to be the same, and even the IOU ratio of the adjacent target to the previous frame is greater than the IOU ratio of the previous frame itself. This will cause the identity ID of the previous frame to be passed to the adjacent target when it is transferred to the next frame according to the IOU ratio, resulting in the switching of the target identity ID, which leads to tracking failure. The measurement module can add the slope and size of the current target center point and the target center point of the previous frame based on the idea of ​​IOU, that is, the target speed of the previous frame; and calculate the slope and size between the current frame and the next frame target, that is, the target speed of the current frame. The formula is:

[0097]

[0098]

[0099] CIOU=DIOU-αv

[0100]

[0101] The purpose of using CIOU is to add a penalty term for the aspect ratio to make the evaluation more accurate.

[0102] Step 5: Input the data sets of each category into the redesigned multi-target tracking model to obtain the multi-target tracking results.

[0103] Based on the above embodiment, step 5 specifically includes:

[0104] According to the data set of each category, the ReID features of the target's bounding box and the TIOU calculation cost matrix are determined, and then the Kalman filter and Hungarian algorithm are used for target tracking. At the same time, secondary data association is used for low-scoring detection boxes to obtain multi-target tracking results.

[0105] In specific implementation, after designing the multi-target tracking model, the datasets for each category can be input into the redesigned multi-target tracking model for identification. Data association is performed using the ReID features of the bounding boxes and the TIOU cost matrix, then implemented using the Kalman filter and Hungarian algorithm. Furthermore, during data association, low-scoring detection frames are not discarded directly; instead, a secondary data association process is employed. This means that after the high-scoring detection frames are initially matched with the object trajectories, the remaining low-scoring detection frames are then matched with the object trajectories of the high-scoring detection frames.

[0106] The video multi-target tracking method in complex scenarios provided by this embodiment designs a new IOU calculation formula TIOU from the perspective of kinematics to realize a measurement method with temporal information that can characterize neighborhood relationships, and better utilizes the temporal context information of the video to perform multi-target tracking; by adding key point detection and feature alignment operations, the situation where the ReID feature recognition is not obvious in the shared features is improved, so that the addition of ReID features can greatly improve the final tracking results; at the same time, the observation-centered Kalman filter is realized by randomly losing the Kalman filter prediction value or ignoring the Kalman filter prediction value under the target frame with higher detection confidence, which can effectively avoid the tracking drift caused by the accumulation of errors over time. The entire multi-target tracking system is as follows Figure 6 Through the solution of this example, the tracking accuracy MOTA and HOTA can be effectively improved in multi-target tracking, and ID switching can be reduced, thereby improving recognition efficiency and accuracy.

[0107] Corresponding to the above method embodiment, see Figure 4 The embodiment of the present disclosure further provides a video multi-target tracking system 40 in complex scenarios, including:

[0108] A construction module 401 is used to construct a multi-target tracking model based on the YOLOX detection model, wherein the multi-target tracking model includes a feature extraction network, a target classification network, and a tracking network;

[0109] A first design module 402 is configured to add a re-identification network with keypoint detection and alignment operations to the tracking network;

[0110] The second design module 403 is used to redesign the trade-off between the predicted value and the observed value of the Kalman filter in the tracking network during the data association phase, so that the Kalman filter is observation-centric;

[0111] The third design module 404 is used to design an IOU metric with time series and add it to the multi-target tracking model;

[0112] The target tracking module 405 is used to input the data sets of each category into the redesigned multi-target tracking model to obtain the multi-target tracking results.

[0113] Figure 4 The system shown can correspondingly execute the contents of the above method embodiment. For the parts not described in detail in this embodiment, refer to the contents recorded in the above method embodiment and will not be repeated here.

[0114] See also Figure 5 The present disclosure also provides an electronic device 50, comprising at least one processor and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the method for multi-target tracking in complex video scenes described in the aforementioned method embodiment.

[0115] The embodiment of the present disclosure also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the video multi-target tracking method in complex scenes in the aforementioned method embodiment.

[0116] An embodiment of the present disclosure also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the video multi-target tracking method in complex scenes in the aforementioned method embodiment.

[0117] Reference below Figure 5 , which shows a schematic structural diagram of an electronic device 50 suitable for implementing the embodiments of the present disclosure. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0118] like Figure 5As shown, the electronic device 50 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the electronic device 50 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0119] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 50 to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows the electronic device 50 with various devices, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0120] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0121] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0122] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0123] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device can perform the relevant steps of the method embodiment.

[0124] Alternatively, the computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device can perform the relevant steps of the above method embodiment.

[0125] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0126] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0127] The units involved in the embodiments described in this disclosure may be implemented by software or hardware.

[0128] It should be understood that various parts of the present disclosure can be implemented in hardware, software, firmware, or a combination thereof.

[0129] The above description is merely a specific embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A video multi-target tracking method in complex scenes, characterized by: include: Step 1: Build a multi-target tracking model based on the YOLOX detection model, wherein the multi-target tracking model includes a feature extraction network, a target classification network, and a tracking network; Step 2: Add a re-identification network with key point detection and alignment operations to the tracking network; Step 3: In the data association stage, the trade-off between the predicted value and the observed value of the Kalman filter in the tracking network is redesigned so that the Kalman filter is centered on the observation. The expression of the Kalman filter is: ； ； ； ； ； in, for The tracking results after fusion at each moment, Kalman filter is based on The tracking results of the moment The predicted value at time, is the transition coefficient, for The observation value detected by the target detector at time , 、 are process noise and observation noise, is the covariance matrix, since The predicted value contains the error accumulation over time, taking random loss To reduce the accumulation of errors over time, or to discard them directly when the confidence level of the observations is high ; Step 4: Design the time-series IOU metric and add it to the multi-target tracking model. The steps of step 4 specifically include: Based on CIOU, the speed of the tracking target is introduced, that is, the size and direction of the displacement of the target in the unit continuous frame, and the TIOU metric is obtained. The representation of the target includes the horizontal coordinate of the center point, the vertical coordinate of the center point, the height of the target frame, the width of the target frame, the speed of the target in the previous frame, and the speed of the target in the next frame. Step 5: Input the data sets of each category into the redesigned multi-target tracking model to obtain the multi-target tracking results; The step 5 specifically includes: According to the data set of each category, the ReID features of the target's bounding box and the TIOU calculation cost matrix are determined, and then the Kalman filter and Hungarian algorithm are used for target tracking. At the same time, secondary data association is used for low-scoring detection boxes to obtain multi-target tracking results.

2. The method according to claim 1, characterized in that ,The feature extraction network includes a Transformer model and a convolutional neural network; The classification network includes a feature decoder and a classifier.

3. The method according to claim 1, characterized in that ,The selection method includes random loss or directly selecting the current frame target ,positioning frame based on the target frame with greater confidence.

4. A video multi-target tracking system in a complex scene, wherein the video multi-target tracking system in a complex scene is used to execute the video multi-target tracking method in a complex scene according to claim 1, characterized in that: include: A construction module is used to build a multi-target tracking model based on the YOLOX detection model, wherein the multi-target tracking model includes a feature extraction network, a target classification network and a tracking network; The first design module is used to add a re-ID network with keypoint detection and alignment operations to the tracking network; The second design module is used to redesign the trade-off between the predicted value and the observed value of the Kalman filter in the tracking network during the data association phase, so that the Kalman filter is observation-centric; The third design module is used to design the time-series IOU measurement method TIOU measurement and add it to the multi-target tracking model; The target tracking module is used to input data sets of various categories into the redesigned multi-target tracking model to obtain multi-target tracking results.

5. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video multi-target tracking method in complex scenes as described in any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the video multi-target tracking method in complex scenes according to any one of claims 1 to 3.

Images