Multi-target tracking method and device, electronic equipment and readable storage medium

Abstract

The application provides a multi-target tracking method and device, electronic equipment and readable storage medium. After obtaining high-level semantic features and fine-grained features corresponding to each video frame of a to-be-processed video sequence, the method detects the high-level semantic features of any video frame based on a preset point target detection algorithm, obtains a target detection result of the video frame, and encodes the corresponding fine-grained features to obtain an appearance feature vector graph; the target detection result of the video frame and the appearance feature vector graph are input into a trained feature matching model to obtain a current target detection result of each detection target; based on the current target detection result, each detection target in the current video frame is matched with any trajectory in a trajectory set to determine a target trajectory of each detection target in the current video frame; and the method enhances the feature expression ability of the detection target and improves the accuracy of subsequent target tracking.

Description

Technical Field

[0001] This application relates to the field of target tracking technology, and more specifically, to a multi-target tracking method. Background Technology

[0002] Autonomous driving is a complex engineering project involving multiple fields. It is an intelligent control system integrating perception, localization, trajectory prediction, path planning, and control. Among these, environmental perception, as the information input link of this automated system, provides the input basis for subsequent trajectory prediction and path planning, and is a crucial part. Current environmental perception algorithms, based on sufficient computing and data resources, mainly employ deep learning technology to achieve various perception tasks. Among these, 2D multi-object tracking is a comprehensive task combining object detection and data association; the accuracy, precision, continuity, and efficiency of its algorithm all determine the effectiveness of autonomous driving implementation.

[0003] Currently, the main technologies for handling 2D multi-target tracking tasks include the existing TBD paradigm and the existing JDE paradigm. However, the existing TBD paradigm has high performance but low operating efficiency, while the existing TBD paradigm has high operating efficiency but suffers from certain performance losses, namely, low target tracking accuracy. Summary of the Invention

[0004] The purpose of this application is to provide a multi-target tracking method, apparatus, electronic device, and readable storage medium. By improving the existing JDE paradigm, the above-mentioned problems existing in the prior art are solved, and the accuracy of multi-target tracking is improved, that is, the target tracking effect is improved.

[0005] Firstly, a multi-target tracking method is provided, which may include:

[0006] Obtain high-level semantic features and fine-grained features corresponding to each video frame of the video sequence to be processed; each video frame includes multiple detection targets;

[0007] Based on a preset point-based target detection algorithm, the high-level semantic features of any video frame are detected to obtain the target detection result of the video frame. The corresponding fine-grained features are then encoded to obtain an appearance feature vector map. The appearance feature vector map includes the appearance feature vectors of each position in the video frame. The target detection result includes the position, category information, and confidence level of each detected target.

[0008] The target detection results of the video frame and the appearance feature vector map are input into the trained feature matching model to obtain the appearance feature vector of each detected target;

[0009] Based on the motion state and appearance state of each detected target corresponding to the current target detection result, each detected target in the current video frame is matched with any trajectory in the trajectory set to determine the target trajectory of each detected target in the current video frame; the trajectory set is determined based on the position of the same detected target in all video frames from the first video frame to the current video frame, and the appearance state of the detected target is the appearance feature vector of the corresponding detected target.

[0010] In an optional implementation, high-level semantic features and fine-grained features corresponding to each video frame of the video sequence to be processed are obtained, including:

[0011] A preset feature extraction network is used to extract features from each video frame of the video sequence to be processed, thereby obtaining the image features corresponding to each video frame.

[0012] A preset spatial attention mechanism is used to decouple the image features, thereby obtaining the high-level semantic features and fine-grained features of each video frame.

[0013] In an optional implementation, the default target detection algorithm for points is a point-based mechanism algorithm;

[0014] In an optional implementation, the feature matching model includes an appearance feature sampling point prediction network and a weighting unit;

[0015] The appearance feature sampling point prediction network is used to receive the target detection results and appearance feature vector map of the video frame; for the position of any detected target in the target detection results, position sampling is performed in the appearance feature vector map to obtain a preset number of sampling positions and the position weight of the corresponding sampling positions.

[0016] The weighting unit is used to calculate the weighted sum of the apparent feature vectors and corresponding position weights corresponding to the preset number of sampling positions; and to determine the weighted sum as the current target detection result of the detection target.

[0017] In an optional implementation, after obtaining the target detection results for each of the detected targets, the method further includes:

[0018] The target detection results are filtered using a preset filtering method to obtain filtered target detection results, and the filtered target detection results are determined as new target detection results.

[0019] The preset screening methods include a confidence threshold screening method and an NMS screening method.

[0020] In an optional implementation, based on the motion state and apparent state of each detected target corresponding to the current target detection result, each detected target in the current video frame is matched with any trajectory in the trajectory set to determine the target trajectory of each detected target in the current video frame, including:

[0021] Kalman filtering is performed on the trajectory feature vector of any trajectory in the previous video frame adjacent to the current video frame within the trajectory set to predict the motion state of each detected target in the current video frame.

[0022] Based on the motion and appearance of each detected target in the current video frame, the successfully matched detected targets and their trajectories are determined.

[0023] Secondly, a multi-target tracking device is provided, which may include:

[0024] The acquisition unit is used to acquire the high-level semantic features and fine-grained features corresponding to each video frame of the video sequence to be processed; each video frame includes multiple detection targets;

[0025] The detection unit is used to detect the high-level semantic features of any video frame based on a preset point target detection algorithm, obtain the target detection result of the video frame, and encode the corresponding fine-grained features to obtain an appearance feature vector map; the appearance feature vector map includes the appearance feature vectors of each position in the video frame, and the target detection result includes the position, category information and confidence of each detected target.

[0026] The acquisition unit is further configured to input the target detection results of the video frame and the appearance feature vector map into the trained feature matching model to obtain the appearance feature vector of each detected target;

[0027] The matching unit is used to match each detected target in the current video frame with any trajectory in the trajectory set based on the motion state and appearance state of each detected target corresponding to the current target detection result, and to determine the target trajectory of each detected target in the current video frame; the trajectory set is determined based on the same detected target in all video frames from the first video frame to the current video frame, and the appearance state of the detected target is the appearance feature vector of the corresponding detected target.

[0028] Thirdly, an electronic device is provided, which includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0029] Memory, used to store computer programs;

[0030] When a processor executes a program stored in memory, it implements any of the steps described in the first aspect above.

[0031] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements the steps of any of the methods described in the first aspect above.

[0032] The multi-target tracking method provided in this application, after acquiring the high-level semantic features and fine-grained features corresponding to each video frame of the video sequence to be processed, detects the high-level semantic features of any video frame based on a preset point target detection algorithm, obtaining the target detection result of the video frame, and encodes the corresponding fine-grained features to obtain an appearance feature vector map; the appearance feature vector map includes the appearance feature vectors at each position in the video frame, and the target detection result includes the position, category information, and confidence score of each detected target; the target detection result and the appearance feature vector map of the video frame are input into a trained feature matching model. The apparent feature vectors of each detected target are obtained. Based on the motion state and apparent state of each detected target corresponding to the current target detection result, the contour feature point information of each detected target in the current video frame is matched with any trajectory in the trajectory set to determine the target trajectory of each detected target in the current video frame. The trajectory set is determined based on the position of the same detected target in all video frames from the first video frame to the current video frame. The apparent state of the detected target is the apparent feature vector of the corresponding detected target. The apparent feature vector with stronger expressive power is extracted from the apparent feature vector, which improves the accuracy of subsequent target tracking. Attached Figure Description

[0033] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 A flowchart illustrating a multi-target tracking method provided in an embodiment of this application;

[0035] Figure 2 This is a schematic diagram illustrating the detection of a target bounding box in an object detection task, as provided in an embodiment of this application.

[0036] Figure 3 A schematic diagram illustrating the receptive field of a target rectangle and an apparent feature vector provided in an embodiment of this application;

[0037] Figure 4This is a schematic diagram of the structure of a multi-target tracking device provided in an embodiment of this application;

[0038] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0039] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0040] The multi-target tracking method provided in this application can be applied to servers or terminals with high computing power. The server can be a physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal can be a user equipment (UE) such as a mobile phone, smartphone, laptop, digital radio receiver, personal digital assistant (PDA), or tablet computer (PAD), a handheld device, an in-vehicle device, a wearable device, a computing device, or other processing devices connected to a wireless modem, a mobile station (MS), or a mobile terminal.

[0041] The multi-target tracking method provided in this application is based on the efficient algorithm structure of the existing JDE paradigm, and the algorithm is improved to solve the performance degradation problem in the algorithm, namely the problem of inaccurate target tracking, so as to realize the application of 2D target tracking algorithm in autonomous driving traffic scenarios.

[0042] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0043] Figure 1 This is a flowchart illustrating a multi-target tracking method provided in an embodiment of this application. Figure 2 As shown, the method may include:

[0044] Step S110: Obtain the high-level semantic features and fine-grained features corresponding to each video frame of the video sequence to be processed.

[0045] Each video frame in the video sequence to be processed may include multiple detection targets.

[0046] In the specific implementation, a preset feature extraction network is used to extract features from each video frame of the video sequence to be processed, thereby obtaining the image features corresponding to each video frame. Specifically, since the DLA-34 model is a high-performance backbone in the field of key point detection, the DLA-34 model can be used to extract features from each video frame of the video sequence to be processed, thereby obtaining the image features corresponding to each video frame, that is, extracting the basic features of each video frame.

[0047] A pre-defined spatial attention mechanism is employed to decouple image features, yielding high-level semantic features and fine-grained features for each video frame. Specifically, the spatial attention mechanism can be the CCN module in CSTrackv1, which decouples image features to obtain high-level semantic features for object detection and fine-grained features for appearance feature encoding in each video frame.

[0048] like Figure 2 As shown, the output of existing object detection tasks uses rectangular boxes to locate the detected target. The representation of rectangular boxes is a relatively coarse representation, which is mixed with a lot of background or other target information. Therefore, the appearance features extracted from the rectangular boxes contain a lot of appearance features that are not of the detected target (i.e., background features and other target features).

[0049] The target detection algorithm in this application changes the representation of the detected target to a form that uses several feature points to describe the contour of the detected target. This can be combined with the output matching mechanism mentioned above to extract a more expressive appearance feature vector.

[0050] Step S120: Based on the high-level semantic features and fine-grained features corresponding to each video frame, obtain the target detection results and appearance feature vector map of the corresponding video frame.

[0051] In practice, based on a pre-defined point target detection algorithm, the high-level semantic features of any video frame are detected to obtain the target detection result of the corresponding video frame, and the appearance feature encoding of the fine-grained features of any video frame is performed to obtain the appearance feature vector map of the corresponding video frame.

[0052] Specifically, a point-base mechanism algorithm is used to detect the high-level semantic features of any video frame and obtain the target detection result of the corresponding video frame. The target detection result can be represented as (C, H, W), where (H, W) includes the set of (h, w) positions of each detected target in the corresponding video frame, and each (h, w) position represents a detected target.

[0053] The C of the detected target at each location can include the contour feature point information, the confidence score, and the category. The contour feature point information can be the information of k feature points of the detected target contour, the confidence score can be the probability that the target of interest exists at that location, and the category can be a probability vector of N categories of interest. The k feature points are random points on the target contour, not fixed points. The k feature points correspond to 2k values, and the 2k values ​​represent the offset of the k feature points relative to the anchor point. Then, the representation of the k feature points can be converted into the representation of bounding boxes, i.e., "virtual boxes." At this point, each detected target in the corresponding video frame is represented in the form of a "virtual box." The specific conversion method for the above feature points can use the method of taking the extrema of the k feature points, which will not be elaborated here.

[0054] The corresponding fine-grained features are encoded to obtain an appearance feature vector map, which can be represented as (L, H, W), where L represents the dimension of the appearance feature vector. This allows the extraction of appearance feature vectors at each location (or, encoded as vectors in a low-dimensional space, which represent the appearance features of the corresponding region in a single video frame and serve as tracking cues for subsequent similarity calculations).

[0055] Step S130: Input the target detection results and appearance feature vector map of the video frame into the trained feature matching model to obtain the appearance state of each detected target.

[0056] The feature matching model provided in this application may include an appearance feature sampling point prediction network and a weighting unit. The appearance feature sampling point prediction network is used to receive the target detection result and appearance feature vector map of any video frame. For the position of any detected target in the target detection result, position sampling is performed in the corresponding appearance feature vector map to obtain a preset number of sampling positions and corresponding position weights. The weighting unit is used to calculate the weighted sum of the appearance feature vectors corresponding to the preset number of sampling positions and the corresponding position weights. The weighted sum is determined as the appearance state of the corresponding detected target. The appearance feature sampling point prediction network includes position sampling parameters and position weight adjustment parameters. The appearance state of the detected target is the appearance feature vector of the corresponding detected target.

[0057] In practice, after the target detection result and appearance feature vector map of the video frame are input into the trained feature matching model, the appearance feature sampling point prediction network performs position sampling in the corresponding appearance feature vector map based on the position of any detected target in the target detection result and the sampling algorithm of the feature matching model, to obtain a preset number of sampling positions and corresponding position weights. The weighting unit calculates the weighted sum of the appearance feature vectors corresponding to the preset number of sampling positions and the corresponding position weights, and determines the weighted sum as the appearance feature vector of the corresponding detected target output by the feature matching model.

[0058] like Figure 3 As shown, current object detection and appearance feature encoding tasks typically output maps based on spatial location matching, which are then used for subsequent trajectory tracking. However, the spatial locations of the matched "detected target" and the "receptive field corresponding to the appearance feature vector" are not aligned on the video frame (both center position and size are misaligned). In other words, the appearance feature vector selected for the detected target cannot represent the appearance features of that detected target, which leads to problems in subsequent trajectory tracking and inaccurate target tracking.

[0059] The above-described implementation method of this application obtains the current target detection result of each detection target by searching for the corresponding apparent feature vector in the apparent feature vector map, thereby solving the above-mentioned problem.

[0060] Step S140: Based on the apparent feature vectors of each detected target and the motion state of each detected target, match each detected target in the current video frame with any trajectory in the trajectory set to determine the target trajectory of each detected target in the current video frame.

[0061] In practice, the trajectory set is determined based on the positions of different detected targets in adjacent video frames. Specifically, based on the positions of each detected target within each video frame, the detected targets between video frames are matched to obtain all possible trajectories, thus obtaining the trajectory set. That is, the trajectory set is determined based on the positions of the same detected target in all video frames from the first video frame to the current video frame, where each trajectory in the trajectory set corresponds to a trajectory feature vector.

[0062] Kalman filtering is performed on the trajectory feature vector of a trajectory in the previous video frame adjacent to the current video frame within the trajectory set to predict the motion state of each detected target in the current video frame. Specifically, the motion state of the detected target may include: the center coordinates (x and y coordinates) of the virtual box, area, aspect ratio, direction of motion, rate of change of x and y coordinates, rate of change of area, etc., which include the motion state variables necessary for the Kalman filtering algorithm to calculate.

[0063] Based on the motion state of each detected target in the current video frame, the successfully matched detected targets and their trajectories are determined, i.e., the target trajectories of the detected targets are determined. Specifically, the cosine distance matrix between the contour feature point information of the corresponding detected target and the trajectory feature vector of any trajectory is calculated; based on the motion state of each detected target in the current video frame, the contour feature point information of the corresponding detected target and the trajectory feature vector of the target trajectory are selected to match them; then, the contour feature point information of the corresponding detected target and the trajectory feature vector of the trajectory are matched using the Hungarian algorithm and the cosine distance matrix to obtain the successfully matched detected targets and target trajectory data in the current video frame.

[0064] Furthermore, for the detected targets and target trajectories that failed to match, a secondary matching can be performed. Specifically: obtain the detected targets and specified trajectories that failed to match; calculate the IOU distance matrix between the detected targets and specified trajectories that failed to match; and perform a secondary matching on the detected targets and specified trajectories that failed to match using the Hungarian algorithm and the IOU distance matrix to obtain the detected targets and target trajectories that successfully matched in the secondary matching.

[0065] By performing two separate matching operations with the target bounding box and trajectory using different calculation methods, the utilization rate of the detected target bounding box is greatly improved, as is the efficiency of target detection and tracking.

[0066] In the above matching process, the matching degree between the detected target and the target trajectory is improved by calculating the IOU distance matrix between the detected target and the target trajectory and combining it with the Hungarian algorithm for secondary matching. The Hungarian algorithm can match all valid detected targets with valid trajectories again as much as possible, thereby reducing the number of times the target and trajectory are lost and improving the target tracking effect.

[0067] In some embodiments, after obtaining the target detection results of the video frame, a preset filtering method can be used to filter the obtained target detection results to obtain filtered target detection results, and the filtered target detection results can be determined as new target detection results; wherein, the preset filtering method may include a confidence threshold filtering method and an NMS filtering method.

[0068] Specifically, based on the confidence level corresponding to the location of each detection target and the preset confidence level threshold, the target confidence level that is not greater than the confidence level threshold is found, and the relevant detection results of the detection target corresponding to the target confidence level are deleted from the target detection results to obtain new target detection results;

[0069] Based on the category corresponding to the location of each detected target, the NMS filtering method is used to filter the new target detection results to obtain the filtered previous target detection results, and then step S140 is executed.

[0070] In some embodiments, the training process for the feature matching model includes:

[0071] Obtain the training dataset, which may include a large number of target detection results training data and appearance feature vector map training data from different training video frames, as well as the label data of the corresponding training video frames.

[0072] The target detection results training data and appearance feature vector map training data of different training video frames are sequentially input into the appearance feature sampling point prediction network of the feature matching model to be trained. The preset number of sampling positions and the corresponding position weights of the sampling positions are obtained from the output of the appearance feature sampling point prediction network. The weighting unit calculates the weighted sum of the appearance feature vectors and corresponding position weights corresponding to the preset number of sampling positions, and determines the weighted sum as the current target detection result of the corresponding detected target output by the feature matching model to be trained.

[0073] Based on the current target detection result of the corresponding detected target and the label data of the corresponding training video frame, the loss value is calculated to return the adjusted position sampling parameters and position weight adjustment parameters to obtain the trained feature matching model.

[0074] Corresponding to the above method, embodiments of this application also provide a multi-target tracking device, such as... Figure 4 As shown, the device includes:

[0075] The acquisition unit 410 is used to acquire the high-level semantic features and fine-grained features corresponding to each video frame of the video sequence to be processed; each video frame includes multiple detection targets;

[0076] The detection unit 420 is used to detect the high-level semantic features of any video frame based on a preset point target detection algorithm, obtain the target detection result of the video frame, and encode the corresponding fine-grained features to obtain an appearance feature vector map; the appearance feature vector map includes the appearance feature vectors of each position in the video frame, and the target detection result includes the position, category information and confidence of each detected target.

[0077] The acquisition unit 410 is also used to input the target detection result of the video frame and the appearance feature vector map into the trained feature matching model to obtain the appearance feature vector of each detected target;

[0078] The matching unit 430 is used to match each detected target in the current video frame with any target trajectory in the trajectory set based on the apparent feature vector of each detected target and the motion state of each detected target, so as to determine the target trajectory of each detected target in the current video frame; the trajectory set is determined based on the position of the same detected target in all video frames from the first video frame to the current video frame.

[0079] The functions of each functional unit of the multi-target tracking device provided in the above embodiments of this application can be implemented through the above methods and steps. Therefore, the specific working process and beneficial effects of each unit in the multi-target tracking device provided in the embodiments of this application will not be repeated here.

[0080] This application also provides an electronic device, such as... Figure 5 As shown, it includes a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540.

[0081] Memory 530 is used to store computer programs;

[0082] When the processor 510 executes the program stored in the memory 530, it performs the following steps:

[0083] Obtain high-level semantic features and fine-grained features corresponding to each video frame of the video sequence to be processed; each video frame includes multiple detection targets;

[0084] Based on a preset point-based target detection algorithm, the high-level semantic features of any video frame are detected to obtain the target detection result of the video frame. The corresponding fine-grained features are then encoded to obtain an appearance feature vector map. The appearance feature vector map includes the appearance feature vectors of each position in the video frame. The target detection result includes the position, category information, and confidence level of each detected target.

[0085] The target detection results of the video frame and the appearance feature vector map are input into the trained feature matching model to obtain the appearance feature vector of each detected target;

[0086] The target trajectory of each detected target in the current video frame is determined by matching each detected target in the trajectory set with the trajectory of any target in the trajectory set. The trajectory set is determined based on the position of the same detected target in all video frames from the first video frame to the current video frame.

[0087] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0088] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0089] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0090] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0091] The implementation methods and beneficial effects of the various components of the electronic device in the above embodiments for solving the problem can be found in [reference needed]. Figure 1 The steps in the illustrated embodiments are used to implement the electronic device. Therefore, the specific working process and beneficial effects of the electronic device provided in this application will not be repeated here.

[0092] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform any of the multi-target tracking methods described in the above embodiments.

[0093] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the multi-target tracking methods described in the above embodiments.

[0094] Those skilled in the art will understand that the embodiments in this application can be provided as methods, systems, or computer program products. Therefore, the embodiments in this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, the embodiments in this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0095] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0096] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0097] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0098] Although preferred embodiments have been described in this application, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of this application.

[0099] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of the embodiments of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims in this application and their equivalents, then this application also intends to include these modifications and variations.

Claims

1. A multi-target tracking method, characterized in that, The method includes: Obtain high-level semantic features and fine-grained features corresponding to each video frame of the video sequence to be processed; each video frame includes multiple detection targets; Based on a preset point-based target detection algorithm, the high-level semantic features of any video frame are detected to obtain the target detection result of the video frame. The corresponding fine-grained features are then encoded to obtain an appearance feature vector map. The appearance feature vector map includes the appearance feature vectors of each position in the video frame. The target detection result includes the position, category information, and confidence level of each detected target. The target detection results of the video frame and the appearance feature vector map are input into the trained feature matching model to obtain the appearance feature vector of each detected target; Based on the apparent feature vectors and motion states of each detected target, each detected target in the current video frame is matched with any trajectory in the trajectory set to determine the target trajectory of each detected target in the current video frame; the trajectory set is determined based on the position of the same detected target in all video frames from the first video frame to the current video frame. The feature matching model includes an appearance feature sampling point prediction network and a weighting unit; The appearance feature sampling point prediction network is used to receive the target detection results and appearance feature vector map of the video frame; for the position of any detected target in the target detection results, position sampling is performed in the appearance feature vector map to obtain a preset number of sampling positions and the position weight of the corresponding sampling positions. The weighting unit is used to calculate the weighted sum of the apparent feature vectors corresponding to the preset number of sampling positions and the corresponding position weights; and to determine the weighted sum as the apparent state of the detected target.

2. The method as described in claim 1, characterized in that, Obtain the high-level semantic features and fine-grained features corresponding to each video frame of the video sequence to be processed, including: A preset feature extraction network is used to extract features from each video frame of the video sequence to be processed, thereby obtaining the image features corresponding to each video frame. A preset spatial attention mechanism is used to decouple the image features, thereby obtaining the high-level semantic features and fine-grained features of each video frame.

3. The method as described in claim 1, characterized in that, The preset target detection algorithm is a point-based mechanism algorithm.

4. The method as described in claim 1, characterized in that, After obtaining the target detection result of the video frame, the method further includes: The target detection results are filtered using a preset filtering method to obtain filtered target detection results, and the filtered target detection results are determined as new target detection results. The preset screening methods include a confidence threshold screening method and an NMS screening method.

5. The method as described in claim 1, characterized in that, Based on the apparent feature vectors and motion states of each detected target, each detected target in the current video frame is matched with any trajectory in the trajectory set to determine the target trajectory of each detected target in the current video frame, including: Kalman filtering is performed on the trajectory feature vector of any trajectory in the previous video frame adjacent to the current video frame within the trajectory set to predict the motion state of each detected target in the current video frame. Based on the motion and appearance of each detected target in the current video frame, the successfully matched detected targets and their trajectories are determined.

6. A multi-target tracking device, characterized in that, The device includes: The acquisition unit is used to acquire the high-level semantic features and fine-grained features corresponding to each video frame of the video sequence to be processed; each video frame includes multiple detection targets; The detection unit is used to detect the high-level semantic features of any video frame based on a preset point target detection algorithm, obtain the target detection result of the video frame, and encode the corresponding fine-grained features to obtain an appearance feature vector map; the appearance feature vector map includes the appearance feature vectors of each position in the video frame, and the target detection result includes the position, category information and confidence of each detected target. The acquisition unit is further configured to input the target detection results of the video frame and the appearance feature vector map into the trained feature matching model to obtain the appearance feature vector of each detected target; The matching unit is used to match each detected target in the current video frame with any trajectory in the trajectory set based on the apparent feature vector of each detected target and the motion state of each detected target, so as to determine the target trajectory of each detected target in the current video frame; the trajectory set is determined based on the position of the same detected target in all video frames from the first video frame to the current video frame; The feature matching model includes an appearance feature sampling point prediction network and a weighting unit; The appearance feature sampling point prediction network is used to receive the target detection results and appearance feature vector map of the video frame; for the position of any detected target in the target detection results, position sampling is performed in the appearance feature vector map to obtain a preset number of sampling positions and the position weight of the corresponding sampling positions. The weighting unit is used to calculate the weighted sum of the apparent feature vectors corresponding to the preset number of sampling positions and the corresponding position weights; and to determine the weighted sum as the apparent state of the detected target.

7. The apparatus as claimed in claim 6, characterized in that, The acquisition unit is specifically used for: A preset feature extraction network is used to extract features from each video frame of the video sequence to be processed, thereby obtaining the image features corresponding to each video frame. A preset spatial attention mechanism is used to decouple the image features, thereby obtaining the high-level semantic features and fine-grained features of each video frame.

8. An electronic device, characterized in that, The electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-5.

Images