Bird's-eye view-based multi-target tracking methods, devices, storage media, and equipment

Abstract

This application provides a multi-target tracking method, apparatus, storage medium, and device based on a bird's-eye view, relating to autonomous driving technology and computer vision. The method may include: acquiring view images captured by each camera at the current moment; determining feature maps under the bird's-eye view based on the view images captured by each camera at the current moment; and performing multi-target tracking processing based on the feature maps under the bird's-eye view. The multi-target tracking method, apparatus, storage medium, and device based on a bird's-eye view provided in this application can create conditions for accurate tracking and can reduce the computational power consumption during the multi-target tracking process.

Description

Technical Field

[0001] This application relates to autonomous driving technology and computer vision, and in particular to a multi-target tracking method, device, storage medium and equipment based on a bird's-eye view. Background Technology

[0002] Multi-object tracking (MOT) is a key technology for many applications, including autonomous driving, human-computer interaction, video surveillance, and military aerospace. Specifically, MOT is the process of describing the positional information of an uncertain number of targets over a time series.

[0003] In practical applications, video-based multi-object tracking has received widespread attention due to advancements in computer vision and video processing hardware. However, many challenges remain when applying it to real-world scenarios. Due to limitations in video acquisition equipment, common problems include image blurring, variable backgrounds, and lighting variations. Furthermore, the variety and number of targets, interactions between targets and between targets and the background, changes in scale and shape due to target rotation leading to alterations in appearance, and changes in the observer's position and perspective all negatively impact the performance of multi-object tracking algorithms. Summary of the Invention

[0004] The purpose of this application is to provide a multi-target tracking method, apparatus, storage medium, and device based on a bird's-eye view, which can be used to solve the above-mentioned technical problems.

[0005] In a first aspect, embodiments of this application provide a multi-target tracking method based on a bird's-eye view, the method comprising:

[0006] Acquire the viewpoint image captured by each camera at the current moment;

[0007] The feature map under the bird's-eye view is determined based on the view images captured by each camera at the current moment;

[0008] Multi-target tracking is performed based on the feature map from the bird's-eye view.

[0009] In one possible implementation, determining the feature map from the bird's-eye view based on the view images captured by each camera at the current moment includes:

[0010] For each viewpoint image, feature extraction is performed to obtain a corresponding feature map, which includes at least one of the following: multi-scale feature map, handmade feature map, and depth feature map;

[0011] The corresponding feature map is mapped to the feature space under the bird's-eye view to obtain the feature map under the bird's-eye view.

[0012] In another possible implementation, mapping the corresponding feature map to a feature space under a bird's-eye view to obtain the feature map under the bird's-eye view includes:

[0013] The corresponding feature map is mapped to the feature space under the bird's-eye view using a transformer to obtain the feature map under the bird's-eye view:

[0014] Generate keys and values ​​in the transformer based on the corresponding feature map;

[0015] The corresponding feature maps are pooled to obtain a global vector;

[0016] The feature space under the bird's-eye view is rasterized and positionally encoded to obtain the positionally encoded feature space.

[0017] The feature space encoded by the location is concatenated with the global variable;

[0018] Generate queries in the transformer from the concatenated features;

[0019] Based on the query and the key, determine the relationship between the feature map in the bird's-eye view and the corresponding feature map;

[0020] Based on the relationship and the value, the feature map under the bird's-eye view is determined.

[0021] In another possible implementation, the multi-target tracking processing based on the feature map from the bird's-eye view includes:

[0022] Obtain the relative position codes between targets from a bird's-eye view;

[0023] Based on the relative position encoding and the feature map from the bird's-eye view, and through feature encoding, the encoded features are obtained.

[0024] Obtain trajectory queries for each target, including new trajectory queries for new targets at the current moment and historical trajectory queries for targets that appeared before the current moment;

[0025] Based on the encoded features, the relative position encoding, and the trajectory query, and through a feature decoder, decoded features are obtained;

[0026] The decoded features are regressed and classified to obtain the tracking results of each target. The tracking results of any target include: the position information of the target in the bird's-eye view and the target identity or category corresponding to the target.

[0027] In another possible implementation, obtaining the relative position encoding between targets from a bird's-eye view includes:

[0028] Obtain the relative distance between any two targets from a bird's-eye view;

[0029] The relative distances are converted into relative position codes based on a learnable parameter layer to obtain the relative position codes between the targets.

[0030] In another possible implementation, the step of obtaining encoded features based on the relative position encoding and the feature map from the bird's-eye view, and through feature encoding, includes:

[0031] Reduce the dimension of the feature channels corresponding to the feature map under the bird's-eye view;

[0032] The spatial dimension of the feature map from a bird's-eye view after reducing the feature dimension is collapsed to a preset dimension.

[0033] Based on the relative position encoding and the feature map under the collapsed bird's-eye view, the coded features are obtained by performing feature encoding through a feature encoder.

[0034] In another possible implementation, the feature encoder includes a multi-head self-attention layer and a forward propagation layer;

[0035] The process involves encoding features based on the relative position encoding and the feature map from a bird's-eye view, and then encoding these features using a feature encoder to obtain the encoded features, including:

[0036] The relative position encoding and the feature map under the bird's-eye view are input into the query channel and key channel of the multi-head self-attention layer, and the feature map under the bird's-eye view is input into the value channel of the multi-head self-attention layer to obtain the multi-head attention feature;

[0037] The multi-head attention features are subjected to residual and normalization processing to obtain the features after residual and normalization processing;

[0038] The residuals and normalized features are encoded through the forward propagation layer to obtain the encoded features.

[0039] The encoded features are subjected to residual and normalization processing to obtain the encoded features.

[0040] In another possible implementation, the trajectory query includes: trajectory query corresponding to a first original target, wherein the first original target is a target that has appeared before the current frame;

[0041] The process of obtaining the trajectory query corresponding to the original target includes:

[0042] Obtain the new trajectory query corresponding to the previous frame, where the previous frame is the frame preceding the current frame, and the new trajectory query corresponding to the previous frame includes the trajectory query corresponding to the new target appearing in the previous frame; and / or

[0043] For each second original target, historical features containing each second original target are obtained, and trajectory queries are performed based on the historical features containing each second original target to obtain the historical trajectory query corresponding to each second original target. The second original target is the target that has appeared before the previous frame.

[0044] The newly generated trajectory query corresponding to the previous frame, and / or the historical trajectory query corresponding to each second original target, are determined as the trajectory query corresponding to the first original target.

[0045] In another possible implementation, the trajectory query further includes: querying the trajectory of a newly emerging target, wherein the newly emerging target is a target that appears in the current frame;

[0046] The process of obtaining the trajectory query corresponding to the new target includes:

[0047] The new trajectory query corresponding to the new target is generated based on the set number of queries.

[0048] In another possible implementation, the feature decoder includes a multi-head cross-attention layer and a forward propagation layer;

[0049] The process of obtaining decoded features based on the encoded features, the relative position encoding, and the trajectory query, and through a feature decoder, includes:

[0050] The trajectory query is input into the spatial attention model to obtain self-attention features;

[0051] The trajectory query and the self-attention feature are input into the query channel of the multi-head cross-attention layer, the encoding feature and the relative position encoding are input into the key channel of the multi-head cross-attention layer, and the encoding feature is input into the value channel of the multi-head cross-attention layer to obtain the multi-head cross-attention feature;

[0052] The multi-head cross-attention features are subjected to residual and normalization processing to obtain the processed multi-head cross-attention features.

[0053] The processed multi-head cross-attention features are input into the forward propagation layer to obtain the decoded features;

[0054] The decoded features are obtained by performing residual and normalization calculations on the decoded features.

[0055] In another possible implementation, if the trajectory query includes a trajectory query corresponding to the first original target and a trajectory query corresponding to the newly generated target,

[0056] The step of inputting the trajectory query into the spatial attention model to obtain self-attention features includes:

[0057] The trajectory query corresponding to the first original target and the trajectory query corresponding to the new target are merged to obtain the merged trajectory query;

[0058] The fused trajectory query is input into the spatial attention model to obtain self-attention features.

[0059] In another possible implementation, after regressing and classifying the decoded features to obtain the tracking results for each target, the method further includes:

[0060] Create existing trajectory set variables and / or terminate trajectory set variables;

[0061] Based on the existing trajectory set variables and / or the terminated trajectory set variables, the tracking results of each target are managed.

[0062] In another possible implementation, the tracking results of each target are managed based on the existing trajectory set variables and / or the terminated trajectory set variables, including:

[0063] Assign identity information to the new target and record the tracking results corresponding to the new target, wherein the tracking results carry the corresponding time unit tag; and / or,

[0064] Remove trajectories that meet preset conditions from the existing trajectory set variable, and / or move the trajectories that meet the preset conditions into the termination trajectory set variable; and / or,

[0065] Within the time delay threshold range, remove existing trajectories that meet preset conditions from the time delay range.

[0066] In another possible implementation, the method further includes:

[0067] Create a set of trajectory variables for key objectives;

[0068] Store all key targets output in the current frame in the trajectory set variable of the key targets; and / or,

[0069] Store the historical key targets output within the time delay threshold range in the trajectory set variable of the key targets; and / or,

[0070] If the current frame output contains non-critical targets, then non-critical targets that meet specific conditions are identified from the output non-critical targets, and these non-critical targets that meet specific conditions are stored in the trajectory set variable of the critical targets. The non-critical targets that meet the specific conditions include: the non-critical target is transformed from a critical target, and the time of transformation from a critical target to a non-critical target falls within the time delay threshold range; and / or,

[0071] The newly generated targets output in the current frame are stored in the trajectory set variable of the key targets.

[0072] In another possible implementation, the method further includes:

[0073] The tracking results of each target in the termination trajectory set variable are output in text and / or visual form according to time sequence and / or identity information.

[0074] In another possible implementation, the method further includes:

[0075] Collect a dataset, which includes samples of at least one category and attribute information of each sample;

[0076] The model is trained based on the dataset to obtain a trained model. The model includes the feature encoder, the feature decoder, the regression module, and the classification module. The regression module is used to regress the decoded features, and the classification module is used to classify the decoded features.

[0077] In another possible implementation, training the model based on the dataset to obtain the trained model includes:

[0078] Determine the sum of losses for each query;

[0079] Determine the loss in recall rate for key targets;

[0080] Calculate the target loss function based on the loss of each query and the loss of the key target recall rate;

[0081] The model is trained based on the dataset and the target loss function to obtain the trained model.

[0082] Secondly, embodiments of this application provide a multi-target tracking device based on a bird's-eye view, the device comprising:

[0083] The acquisition module is used to acquire the viewpoint image captured by each camera at the current moment;

[0084] The determination module is used to determine the feature map under the bird's-eye view based on the view images captured by each camera at the current moment;

[0085] A multi-target tracking processing module is used to perform multi-target tracking processing based on the feature map under the bird's-eye view.

[0086] In one possible implementation, when the determining module determines the feature map under the bird's-eye view based on the view images acquired by each camera at the current moment, it is specifically used for:

[0087] For each viewpoint image, feature extraction is performed to obtain a corresponding feature map, which includes at least one of the following: multi-scale feature map, handmade feature map, and depth feature map;

[0088] The corresponding feature map is mapped to the feature space under the bird's-eye view to obtain the feature map under the bird's-eye view.

[0089] In one possible implementation, when the determining module maps the corresponding feature map to the feature space under the bird's-eye view to obtain the feature map under the bird's-eye view, it is specifically used for:

[0090] The corresponding feature map is mapped to the feature space under the bird's-eye view using a transformer to obtain the feature map under the bird's-eye view:

[0091] Generate keys and values ​​in the transformer based on the corresponding feature map;

[0092] The corresponding feature maps are pooled to obtain a global vector;

[0093] The feature space under the bird's-eye view is rasterized and positionally encoded to obtain the positionally encoded feature space.

[0094] The feature space encoded by the location is concatenated with the global variable;

[0095] Generate queries in the transformer from the concatenated features;

[0096] Based on the query and the key, determine the relationship between the feature map in the bird's-eye view and the corresponding feature map;

[0097] Based on the relationship and the value, the feature map under the bird's-eye view is determined.

[0098] In one possible implementation, when the multi-target tracking processing module performs multi-target tracking processing based on the feature map under the bird's-eye view, it is specifically used for:

[0099] Obtain the relative position codes between targets from a bird's-eye view;

[0100] Based on the relative position encoding and the feature map from the bird's-eye view, and through feature encoding, the encoded features are obtained.

[0101] Obtain trajectory queries for each target, including new trajectory queries for new targets at the current moment and historical trajectory queries for targets that appeared before the current moment;

[0102] Based on the encoded features, the relative position encoding, and the trajectory query, and through a feature decoder, decoded features are obtained;

[0103] The decoded features are regressed and classified to obtain the tracking results of each target. The tracking result of any target includes: the position information of any target in the bird's-eye view and the target type corresponding to any target.

[0104] In one possible implementation, when the multi-target tracking processing module acquires the relative position codes between targets from a bird's-eye view, it is specifically used for:

[0105] Obtain the relative distance between any two targets from a bird's-eye view;

[0106] The relative distances are converted into relative position codes based on a learnable parameter layer to obtain the relative position codes between the targets.

[0107] In one possible implementation, when the multi-target tracking processing module obtains encoded features based on the relative position encoding and the feature map from the bird's-eye view, and through a feature encoder, it specifically performs the following:

[0108] Reduce the dimension of the feature channels corresponding to the feature map under the bird's-eye view;

[0109] The spatial dimension of the feature map from a bird's-eye view after reducing the feature dimension is collapsed to a preset dimension.

[0110] Based on the relative position encoding and the feature map under the collapsed bird's-eye view, the coded features are obtained by performing feature encoding through a feature encoder.

[0111] In one possible implementation, the feature encoder includes a multi-head self-attention layer and a forward propagation layer;

[0112] The multi-target tracking processing module, when encoding features based on the relative position encoding and the feature map from the bird's-eye view, and obtaining encoded features through a feature encoder, is specifically used for:

[0113] The relative position encoding and the feature map under the bird's-eye view are input into the query channel and key channel of the multi-head self-attention layer, and the feature map under the bird's-eye view is input into the value channel of the multi-head self-attention layer to obtain the multi-head attention feature;

[0114] The multi-head attention features are subjected to residual and normalization processing to obtain the features after residual and normalization processing;

[0115] The residuals and normalized features are encoded through the forward propagation layer to obtain the encoded features.

[0116] The encoded features are subjected to residual and normalization processing to obtain the encoded features.

[0117] In one possible implementation, the trajectory query includes: a trajectory query corresponding to a first original target, wherein the first original target is a target that has appeared before the current frame;

[0118] Specifically, when the multi-target tracking processing module obtains the trajectory query corresponding to the original target, it is used for:

[0119] Obtain the new trajectory query corresponding to the previous frame, where the previous frame is the frame preceding the current frame, and the new trajectory query corresponding to the previous frame includes the trajectory query corresponding to the new target appearing in the previous frame; and / or,

[0120] For each second original target, historical features containing each second original target are obtained, and trajectory queries are performed based on the historical features containing each second original target to obtain the historical trajectory query corresponding to each second original target. The second original target is the target that has appeared before the previous frame.

[0121] The newly generated trajectory query corresponding to the previous frame, and / or the historical trajectory query corresponding to each second original target, are determined as the trajectory query corresponding to the first original target.

[0122] In one possible implementation, the trajectory query further includes: querying the trajectory of a newly emerging target, wherein the newly emerging target is a target that appears in the current frame;

[0123] Specifically, when the multi-target tracking processing module obtains the trajectory query corresponding to the newly generated target, it is used for:

[0124] The new trajectory query corresponding to the new target is generated based on the set number of queries.

[0125] In one possible implementation, the feature decoder includes a multi-head cross-attention layer and a forward propagation layer;

[0126] When the multi-target tracking processing module obtains decoded features based on the encoded features, the relative position encoding, and the trajectory query, and through a feature decoder, it is specifically used for:

[0127] The trajectory query is input into the spatial attention model to obtain self-attention features;

[0128] The trajectory query and the self-attention feature are input into the query channel of the multi-head cross-attention layer, the encoding feature and the relative position encoding are input into the key channel of the multi-head cross-attention layer, and the encoding feature is input into the value channel of the multi-head cross-attention layer to obtain the multi-head cross-attention feature;

[0129] The multi-head cross-attention features are subjected to residual and normalization processing to obtain the processed multi-head cross-attention features.

[0130] The processed multi-head cross-attention features are input into the forward propagation layer to obtain the decoded features;

[0131] The decoded features are obtained by performing residual and normalization calculations on the decoded features.

[0132] In one possible implementation, when the trajectory query includes the trajectory query corresponding to the first original target and the trajectory query corresponding to the newly generated target...

[0133] When the multi-target tracking processing module inputs the trajectory query into the spatial attention model to obtain self-attention features, it is specifically used for:

[0134] The trajectory query corresponding to the first original target and the trajectory query corresponding to the new target are merged to obtain the merged trajectory query;

[0135] The fused trajectory query is input into the spatial attention model to obtain self-attention features.

[0136] In one possible implementation, the apparatus further includes: a first creation module and a tracking management module, wherein,

[0137] The first creation module is used to create existing trajectory set variables and / or terminate trajectory set variables;

[0138] The tracking management module is used to manage the tracking results of each target based on the existing trajectory set variables and / or the terminated trajectory set variables.

[0139] In one possible implementation, when the tracking management module manages the tracking results of each target based on the existing trajectory set variables and / or the terminated trajectory set variables, it is specifically used for:

[0140] Assign identity information to the new target and record the tracking results corresponding to the new target, wherein the tracking results carry the corresponding time unit tag; and / or,

[0141] Remove trajectories that meet preset conditions from the existing trajectory set variable, and / or move the trajectories that meet the preset conditions into the termination trajectory set variable; and / or,

[0142] Within the time delay threshold range, remove existing trajectories that meet preset conditions from the time delay range.

[0143] In one possible implementation, the apparatus further includes at least one of a first storage module, a second storage module, a third storage module, and a fourth storage module, and a second creation module, wherein,

[0144] The second creation module is used to create trajectory set variables for key targets;

[0145] The first storage module is used to store all key targets output in the current frame in the trajectory set variable of the key targets;

[0146] The second storage module is used to store the historical key targets output within the time delay threshold range in the trajectory set variable of the key targets;

[0147] The third storage module is used to determine non-critical targets that meet specific conditions from the output non-critical targets when there are non-critical targets in the current frame output, and store the non-critical targets that meet specific conditions in the trajectory set variable of the critical targets. The non-critical targets that meet specific conditions include: the non-critical targets are transformed from the critical targets, and the time of transformation from the critical targets to the non-critical targets is within the time delay threshold range.

[0148] The fourth storage module is used to store the newly generated target output in the current frame into the trajectory set variable of the key target.

[0149] In one possible implementation, the device further includes: an output module, wherein,

[0150] The output module is used to output the tracking results of each target in the termination trajectory set variable in text and / or visualization manner, according to time sequence and / or identity information.

[0151] In one possible implementation, the device further includes: a data acquisition module and a training module, wherein,

[0152] The acquisition module is used to acquire a dataset, which includes samples of at least one category and attribute information of each sample.

[0153] The training module is used to train the model based on the dataset to obtain a trained model. The model includes the feature encoder, the feature decoder, the regression module, and the classification module. The regression module is used to regress the decoded features, and the classification module is used to classify the decoded features.

[0154] In one possible implementation, when the training module trains the model based on the dataset to obtain the trained model, it is specifically used for:

[0155] Determine the sum of losses for each query;

[0156] Determine the loss in recall rate for key targets;

[0157] Calculate the target loss function based on the loss of each query and the loss of the key target recall rate;

[0158] The model is trained based on the dataset and the target loss function to obtain the trained model.

[0159] Thirdly, an electronic device is provided, the electronic device comprising:

[0160] One or more processors;

[0161] Memory;

[0162] One or more applications, wherein the applications are stored in memory and configured to be executed by one or more processors, the applications being configured to: perform operations corresponding to the multi-target tracking method with a bird's-eye view as shown in any possible implementation of the first aspect.

[0163] Fourthly, a computer-readable storage medium is provided, the storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement a bird's-eye view multi-target tracking method as shown in any possible implementation of the first aspect.

[0164] The beneficial effects of the technical solutions provided in some embodiments of this application include at least the following:

[0165] This application provides a multi-target tracking method, apparatus, storage medium, and device based on a bird's-eye view. In this embodiment, the view images captured by each camera on the vehicle-mounted device at the current moment are obtained to determine the feature map under the bird's-eye view, and multi-target tracking processing is performed based on the feature map. In other words, by projecting the scene onto a bird's-eye view, the unreliability and instability of appearance features caused by target occlusion due to viewpoint issues are minimized, creating conditions for accurate tracking. The algorithm can adaptively utilize global information to distinguish different targets under the bird's-eye view, while reducing redundant detection. Furthermore, through encoding and decoding, target detection and tracking are further integrated, allowing the model to focus on fewer specific key targets, thus reducing the computational cost of the multi-target tracking process. Attached Figure Description

[0166] Figure 1 This is a flowchart illustrating a multi-target tracking method based on a bird's-eye view, as provided in an embodiment of this application.

[0167] Figure 2 This is a schematic diagram of a feature encoding process provided in an embodiment of this application;

[0168] Figure 3 This is a schematic diagram of a feature decoding process provided in an embodiment of this application;

[0169] Figure 4 This is a schematic diagram of the structure of a multi-target tracking device based on a bird's-eye view, provided in an embodiment of this application;

[0170] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0171] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0172] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this application, it should be noted that, unless otherwise expressly specified and limited, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0173] For multi-object tracking, besides the impact of the algorithm's input dimension, there are also many unavoidable problems from the algorithm itself. For example, the computational scale of data association algorithms is a significant factor. Common multi-object tracking algorithms often need to strike a trade-off between computational complexity and accuracy; considering more cues increases computational cost, and vice versa. In video multi-object tracking, especially in the field of autonomous driving, there is another insurmountable problem: using 2D video to solve 3D problems. In fact, this is the biggest problem in video multi-object tracking; the root cause of occlusion is essentially the incomplete cue information due to viewpoint limitations. Therefore, designing effective multi-object tracking algorithms has significant theoretical and practical value.

[0174] This application aims to design a video multi-object tracking method that balances accuracy and robustness, achieving efficient tracking of various targets in videos. Addressing the sensitivity of video multi-object tracking algorithms to detection quality and the sensitivity of target detection algorithms to depth of field, this application proposes a method for joint multi-object detection and tracking from a bird's-eye view. By unifying the detection task into the tracking task, features can be provided based on a shared backbone network, simultaneously optimizing both detection and tracking tasks end-to-end. Furthermore, the entire model can perform multi-object tracking frame-by-frame, achieving online tracking. To address the occlusion problem in complex scenes, this application proposes an attention-based bird's-eye view multi-object tracking method. This method collects surrounding environmental information through a camera and projects it onto the bird's-eye view, minimizing target occlusion caused by perspective issues and thus reducing the unreliability of appearance features. To address the diverse types of targets in real-world scenarios, a multi-class loss is employed in the category term of the comprehensive loss function, ensuring the model is not limited to a specific target group. To address the issue of insufficient feature representation caused by mutual interference between targets and between targets and the background in multi-target tracking, this application proposes a spatial attention mechanism based on self-attention and cross-attention. This mechanism adaptively distinguishes different targets by learning and combining global information, reducing redundant detection and thus improving tracking accuracy. To address the identity swapping problem in multi-target tracking, this application proposes a temporal attention mechanism based on temporal memory units. This mechanism extracts time-related features through learning, thereby reducing identity swapping. To address the sensitivity of multi-target tracking algorithms to the number of targets, this application proposes a method for extracting key targets based on distance ranking, thereby reducing the computational complexity of multi-target tracking algorithms.

[0175] This application provides a bird's-eye view multi-target tracking method based on an attention mechanism. By performing video multi-target tracking from a bird's-eye view, and combining an attention mechanism and a deep learning framework, a video multi-target tracking method that balances accuracy, robustness, and speed is finally formed, as detailed below:

[0176] 1. This application proposes a joint framework for multi-target detection and tracking frame-by-frame from a bird's-eye view. This framework allows the network to be jointly optimized and trained on both tasks in an end-to-end manner based on a single backbone network. Furthermore, the entire model can perform multi-target tracking frame-by-frame, achieving online tracking. Moreover, by unifying the detection task into the tracking task, it eliminates the dependence of classic multi-target tracking methods on target detection algorithms and results.

[0177] 2. In this embodiment of the application, by projecting the scene onto a bird's-eye view, the unreliability and instability of information caused by the perspective problem are minimized, thus creating conditions for accurate tracking;

[0178] 3. This application proposes a target feature representation method based on self-attention and cross-attention mechanisms from a bird's-eye view. Through learning, the algorithm can adaptively use global information to distinguish different targets from a bird's-eye view, while reducing redundant detection. Furthermore, through joint optimization, the target features are made more in line with the needs of tracking, thereby improving the accuracy of multi-target tracking.

[0179] 4. This application proposes an attention mechanism based on time memory units to optimize target features from a bird's-eye view. Through learning, the algorithm can use information about the target in different time units to enhance the robustness of the network for target feature representation, thereby reducing the identity swapping problem in multi-target tracking.

[0180] 5. The embodiments of this application involve a new comprehensive loss function that integrates target detection, tracking, and key target recall. By integrating spatial and temporal attention mechanisms, it can better learn social relationships and further identify key targets in the spatial domain. This allows the model to reduce the computational cost of multi-target tracking by focusing on fewer specific key targets.

[0181] 6. The embodiments of this application also relate to a multi-class, multi-target tracking model, which is not limited to a certain target group and has better generalization and robustness in practical problems.

[0182] For a detailed description of the embodiments of this application, please refer to the following embodiments:

[0183] First, let's introduce the concepts and definitions involved in the embodiments of this application, as shown below:

[0184] Observation subject: The observation subject is the vehicle in this example, which is actually the sensor itself. The sensor data needs to be converted by the parameters of the sensor relative to the center of the vehicle before it is transmitted to the system. In this embodiment, it is assumed that the data received by the system has already been converted.

[0185] Frame: The basic unit of data transmission by a sensor. Each frame represents a certain unit of time (e.g., one-thirtieth of a second). The frame transmission and reception frequency is determined by the system used by the vehicle.

[0186] Target (or object): In video multi-target tracking, it refers to an object in an image that is distinct from the background and has a certain physical meaning. It is usually marked in the form of a bounding box.

[0187] Detection: In video multi-object tracking, it refers to the candidate units in each frame of the image that are suspected to be targets, output by the detection algorithm.

[0188] Feature extraction is the process of filtering and abstracting information from image regions, typically performed on the region to which the target or detection result belongs.

[0189] Data association: In video multi-object tracking, it is the process of separating detectable objects or targets with different identities from those in different frames, and matching those with the same identity together.

[0190] Tracklet: A short segment of location information obtained by integrating the corresponding data results of a target within a short time interval (multiple time units);

[0191] Trajectory: The location information of the target in the entire video sequence, which can be further generated from trajectory segments through data association;

[0192] Tracking: The process of generating a target trajectory in a video sequence;

[0193] The present application will now be described in detail with reference to specific embodiments.

[0194] This application provides a multi-target tracking method based on a bird's-eye view. This method can be executed by an electronic device, which can be a server or a terminal device. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smartphone, tablet, laptop, desktop computer, etc., but is not limited to these. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, and this application does not impose any limitations on this.

[0195] It should be noted that electronic devices used to perform multi-target tracking methods based on bird's-eye view can also include: in-vehicle devices installed on various intelligent vehicles and intelligent robots, etc.

[0196] Furthermore, such as Figure 1 As shown, the method may include:

[0197] Step S101: Obtain the viewpoint image captured by each camera at the current moment.

[0198] In this embodiment of the application, before acquiring the viewpoint image captured by each camera at the current moment, the method may further include: acquiring environmental information and motion information of the observed subject captured by multiple cameras within a certain range around the current frame from different perspectives, and determining the viewpoint image captured by each camera at the current moment based on the environmental information and motion information of the observed subject captured by multiple cameras within a certain range around the current frame from different perspectives. In this embodiment of the application, the motion information of the observed subject can be measured by an inertial measurement unit.

[0199] The camera can be a monocular vision camera or a multi-view vision camera, or a camera group consisting of a monocular camera and / or a multi-view vision camera. For example, the number of cameras can be six. Furthermore, the viewpoint image obtained in this embodiment can be an image from a single viewpoint or an image from multiple viewpoints; no limitation is made in this embodiment.

[0200] Step S102: Determine the feature map under the bird's-eye view based on the view images captured by each camera at the current moment.

[0201] In the embodiments of this application, if the acquired image is a single-view image, then the feature map under the bird's-eye view is determined based on the single-view image; if the acquired image is an image from multiple views, then the feature map under the bird's-eye view is determined based on the multiple-view images.

[0202] Step S103: Perform multi-target tracking processing based on feature maps from a bird's-eye view.

[0203] This application provides a multi-target tracking method from a bird's-eye view. In this embodiment, the method acquires viewpoint images from each camera on the vehicle-mounted device at the current moment to determine feature maps under the bird's-eye view, and then performs multi-target tracking processing based on these feature maps. In other words, by projecting the scene onto a bird's-eye view, the method minimizes the unreliability and instability of appearance features caused by target occlusion due to viewpoint issues, creating conditions for accurate tracking. This allows the algorithm to adaptively utilize global information to distinguish different targets from a bird's-eye view, while reducing redundant detection. Furthermore, through encoding and decoding, it further integrates target detection and tracking, enabling the model to focus on fewer specific key targets, thus reducing the computational cost of the multi-target tracking process.

[0204] Furthermore, the acquisition of the viewpoint image captured by each camera at the current moment in step S101 can be found in the above embodiments. In addition, any other method of acquiring the viewpoint image captured by the camera at the current moment is within the protection scope of the embodiments of this application.

[0205] Specifically, the step S102, which involves determining the feature map under the bird's-eye view based on the viewpoint images acquired by each camera at the current moment, may include: step S1021 (not shown in the figure) and step S1022 (not shown in the figure), wherein,

[0206] Step S1021: Extract features from each viewpoint image to obtain the corresponding feature map.

[0207] Specifically, feature extraction is the process of filtering and abstracting information from image regions, typically performed on the region containing the target or detection result. In this embodiment, feature extraction is performed on the region containing the target in the viewpoint image. In this embodiment, the feature map may include at least one of: multi-scale feature maps, handcrafted feature maps, and depth feature maps.

[0208] Specifically, this embodiment of the application uses multi-scale feature extraction of viewpoint images to obtain multi-scale feature maps as an example. In this embodiment, multi-scale feature maps are extracted from the viewpoint images acquired by each camera at the current time using a backbone network and a Feature Pyramid Network (FPN). In this embodiment, a combination of RegNet and BiFPN framework can be used, or other fully connected layer neural networks can be used. BiFPN can have four scales. Assuming the height and width of the viewpoint image are H and W respectively, four scales can be obtained. , , , , , , , The feature maps of the four scales are then upsampled and stitched together to obtain multi-scale feature maps of the images from each viewpoint.

[0209] Step S1022: Map the corresponding feature map to the feature space under the bird's-eye view to obtain the feature map under the bird's-eye view.

[0210] Specifically, in this embodiment, the feature map of each viewpoint image is mapped to the feature space under the bird's-eye view. Alternatively, the features of each viewpoint image can be mapped to other feature spaces. In this embodiment, the dimensions of both the feature space under the bird's-eye view and other feature spaces are pre-defined.

[0211] Specifically, the corresponding feature map is mapped to the feature space under the bird's-eye view to obtain the feature map under the bird's-eye view. This can be further divided into: mapping the corresponding feature map to the feature space under the bird's-eye view through a transformer to obtain the feature map under the bird's-eye view.

[0212] Furthermore, the corresponding feature map is mapped to the feature space under the bird's-eye view using a transformer to obtain the feature map under the bird's-eye view. Specifically, this may include: generating keys and values ​​in the transformer based on the corresponding feature map; performing pooling processing on the corresponding feature map to obtain a global vector; rasterizing and positionally encoding the feature space under the bird's-eye view to obtain a positionally encoded feature space; concatenating the positionally encoded feature space with global variables; generating a query in the transformer from the concatenated features; determining the relationship between the feature map under the bird's-eye view and the corresponding feature map based on the query and key; and determining the feature map under the bird's-eye view based on the relationship and the value.

[0213] Taking multi-scale feature maps as an example, the process involves two steps: firstly, each multi-scale feature map is processed through a multilayer perceptron to generate keys and values ​​in the transformer; secondly, global pooling is performed on the multi-scale feature maps to obtain a global vector, and the resulting bird's-eye view space is rasterized and positionally encoded. These positional codes are then concatenated with the global vector, and subsequently processed through a multilayer perceptron to obtain the query in the transformer. The query dimension is the same as the dimension of the feature map in the bird's-eye view. Furthermore, since both keys and values ​​reside in 2D image coordinate space, the final result requires attention between the query and the keys to determine the weights of the 2D image pixels received by each raster in the bird's-eye view. This determines the relationship between the feature map in the bird's-eye view and the input. Finally, these weights are used to weighted sum the values ​​in the image plane to obtain the final feature map in the bird's-eye view.

[0214] For example, the feature map from a bird's-eye view is Its dimensions are C. Where C is the number of feature channels. Indicates the size of the first dimension. Indicates the size of the second dimension. , This represents a time unit, in this example, a frame. Indicates the length of the video sequence.

[0215] Furthermore, after obtaining the feature map from the bird's-eye view, multi-target tracking processing is performed based on the feature map from the bird's-eye view. Specifically, this may include: step S1031 (not shown in the figure), step S1032 (not shown in the figure), step S1033 (not shown in the figure), step S1034 (not shown in the figure), and step S1035 (not shown in the figure), wherein,

[0216] Step S1031: Obtain the relative position codes between each target from a bird's-eye view.

[0217] Specifically, in order to better express the relative relationships between targets, the position information input into the feature encoder adopts relative position encoding. In this embodiment, obtaining the relative position encoding between targets from a bird's-eye view can specifically include: obtaining the relative distance between any two targets from a bird's-eye view; and converting each relative distance into a relative position encoding based on a learnable parameter layer to obtain the relative position encoding between targets.

[0218] Assuming the current frame The index of any two features is First, calculate the relative distance between the two pairs. Then, the relative distance is converted into a relative position encoding through a learnable parameter layer (embedding), specifically represented by formula (1):

[0219] Formula (1);

[0220] in, and These are the position encoding parameters corresponding to the query and key in the transformer framework. This limits the maximum value of the relative distance; when the relative distance between two features exceeds [a certain value], [the maximum value is limited]. After that, the distance values ​​are all 0. This setting is based on the assumption that relative positions lose their meaning when they exceed a certain range, and it can make the model robust to situations that do not occur during training.

[0221] Step S1032: Based on relative position encoding and feature maps from a bird's-eye view, feature encoding is performed through a feature encoder to obtain encoded features.

[0222] Specifically, the main structure of the feature encoder consists of a multi-head self-attention layer that loops N times and a feed forward network (FFN). Residual and normalization calculations are performed after each iteration through the self-attention layer and the feed forward network. In other words, the encoding comprises two stages: the first stage performs spatial self-attention calculation on the input features, and the second stage encodes the output features.

[0223] Specifically, based on relative position encoding and feature maps from a bird's-eye view, and through feature encoding, coded features are obtained. This can be further categorized as follows: inputting the relative position encoding and feature maps from a bird's-eye view into the query and key channels of a multi-head self-attention layer, and inputting the feature maps from the bird's-eye view into the value channel of the multi-head self-attention layer to obtain multi-head attention features; performing residual and normalization processing on the multi-head attention features to obtain residual and normalized features; performing feature encoding on the residual and normalized features through a forward propagation layer to obtain encoded features; and performing residual and normalization processing on the encoded features to obtain coded features, such as... Figure 2 As shown.

[0224] Specifically, the first stage of encoding is to perform spatial self-attention calculation on the input features. The attention in the model can be represented by formula (2):

[0225] ; Formula (2);

[0226] In formula (2), Q, K, and V correspond to query, key, and value in the model. In specific applications, they can be given specific physical meanings. The self-attention mechanism is the case where Q=K. This represents the dimension of the feature channels. The inputs to Q and K incorporate the relative position encoding from step S1031 to ensure the model's invariance to relative positions.

[0227] For a multi-head self-attention layer, multiple attention features are calculated using formula (2), and the features from multiple heads are concatenated to obtain the multi-head attention features. Assume that the feature channel dimensions of Q, K, and V are unified as follows: For a multi-head self-attention system with M heads, the feature channel dimension of each head's output is... Therefore, the output feature dimension after attention calculation is still 1. .

[0228] The features after the attention layer need to be residuald and normalized, as shown in formula (3):

[0229] Z=LayerNorm Sublayer ; Formula (3);

[0230] In formula (3), Sublayer represents the sub-network before the residual layer, such as the multi-head attention layer or the forward propagation layer; X represents the input feature of the sub-network; LayerNorm represents the intra-layer normalization operation, such as batch normalization; and Z represents the output of the residual normalization layer.

[0231] Furthermore, the second stage of encoding is output feature encoding, which is achieved through a fully connected forward propagation layer, as shown in formula (4):

[0232] ; Formula (4);

[0233] In formula (4), Z represents the attention output after residual and normalization. and These represent the parameters of the two fully connected layers in the forward propagation layer. and This is the bias for the corresponding layer.

[0234] Then, according to formula (4), the obtained output is calculated through a residual and normalization layer to obtain the final feature code.

[0235] Furthermore, to make the output encoding more reasonable and robust, based on relative position encoding and feature maps from a bird's-eye view, feature encoding is performed through a feature encoder to obtain encoded features, which may specifically include:

[0236] Step 1: Reduce the dimension of the feature channels corresponding to the feature map from the bird's-eye view;

[0237] In this embodiment of the application, the feature channel dimension is reduced from C to a smaller dimension d by using 1x1 convolution, as shown in formula (5):

[0238] Formula (5);

[0239] in, d represents the new feature dimension. This represents the convolution operation.

[0240] Step 2: Collapse the spatial dimension of the feature map from the bird's-eye view after reducing the feature dimension to a preset dimension; that is, reduce the spatial dimension of the feature map from the bird's-eye view after reducing the feature dimension. The spatial dimension collapses into one dimension, that is... Transforming into dimensions is The characteristic matrix of .

[0241] Step 3: Based on the relative position encoding and the feature map from the collapsed bird's-eye view, feature encoding is performed through a feature encoder to obtain the encoded features. That is, the feature matrix from step 2, along with the relative position encoding, is input into the encoder to obtain the final encoding, which serves as the input to the decoder in subsequent steps. In this embodiment, the specific method of feature encoding based on the relative position encoding and the feature map from the collapsed bird's-eye view in step 3 can be found in the above embodiments and will not be repeated here.

[0242] It should be noted that in this example, the number of loops N is set to 6. Furthermore, the relative position encoding in the third step input is... Figure 2 In this way, it is added to the query and key input of the corresponding attention layer to ensure the model's invariance to relative position information.

[0243] Furthermore, after obtaining the encoded features through the above embodiments, a trajectory query concept is introduced for joint detection and tracking. From a model perspective, this trajectory query design is consistent with the target query during decoding in the target detection framework, except that it corresponds to the identity and historical trajectory information of a certain target. In the framework of this application embodiment, trajectory query is divided into new trajectory query and historical trajectory query. By using new trajectory query to represent target query, target query is unified in trajectory query, thereby unifying multi-target detection into multi-target tracking (the specific scheme is introduced in subsequent embodiments). In fact, in multi-target tracking, historical trajectory query also includes two cases. Assuming the current frame is t, one is the feature corresponding to the target that just appeared in frame t-1 (i.e., the new trajectory query result of the previous frame), which only has information from one frame and no historical information from other time units. The other is the target that already has a trajectory, which has historical information from more than one frame. For the former, the trajectory query is directly performed based on the features of frame t-1. For the latter, which has trajectory information in other time units, historical trajectory query is optimized based on time attention in this application embodiment. The specifics are as follows:

[0244] Step S1033: Obtain the trajectory query for each target.

[0245] The trajectory query includes the query of the new trajectory of the new target at the current moment (hereinafter referred to as the new trajectory query) and the query of the historical trajectory of the target that appeared before the current moment (hereinafter referred to as the historical trajectory query).

[0246] Specifically, trajectory query includes: trajectory query corresponding to the first original target, where the first original target is a target that has appeared before the current frame. For example, if the current frame is t, the first original target is frame t-1 and targets that appeared before frame t-1. Based on this, obtaining the trajectory query corresponding to the original target can specifically include: at least one of steps S1033a (not shown in the figure) and S1033b (not shown in the figure) and S1033c (not shown in the figure), wherein,

[0247] Step S1033a: For each second original target, obtain the historical features of each second original target, and perform trajectory query based on the historical features of each second original target to obtain the historical trajectory query corresponding to each second original target.

[0248] In this context, the second original target is a target that appeared before the previous frame. Continuing with the previous example, if the current frame is t, the second original target is a target that appeared before frame t-1.

[0249] Furthermore, suppose a target already has a certain trajectory. Historical characteristics , The dimension is ,in This is the feature dimension output by the decoder. Before each trajectory query for this target, the target... Track query in the current frame Update according to the following formula (6):

[0250] ; Formula (6);

[0251] in, These represent the parameters, bias term, and memory unit in a Long Short Term Memory Network (LSTM), respectively. This computation, based on temporal and current information, can be viewed as a temporal attention mechanism in multi-target tracking. Through input, forget, and output gates within the time unit, effective historical information about the target trajectory is filtered, thus focusing more on the necessary information. In this embodiment, since the system input can be vehicle sensor information used for autonomous driving, the focus on long-term memory can be reduced, with more emphasis on medium and short-term memory. To reduce computational cost, the input time unit range is assumed to be... Then one option could be ,in It's the camera's frame rate.

[0252] It should be noted that, in addition to using the LSTM network, other recurrent neural networks that express time can also be used in the above formula (6), which is not limited in the embodiments of this application.

[0253] Step S1033b: Obtain the new trajectory query corresponding to the previous frame.

[0254] Among them, the previous frame is the frame before the current frame, and the new trajectory query corresponding to the previous frame includes the trajectory query corresponding to the new target that appeared in the previous frame.

[0255] Specifically, target query is generally a concept in target detection tasks. In this embodiment, to jointly perform multi-target detection and tracking, target query is unified under the concept of trajectory query, i.e., new trajectory query. In fact, for a newly appearing target in a frame, since it lacks existing identity information, its physical meaning is a newly generated trajectory. Therefore, the new trajectory query in this example, like the target detection task, requires manually setting a query quantity to obtain the query vector. That is, obtaining the trajectory query corresponding to the new target can specifically include: generating a new trajectory query corresponding to the new target based on the set query quantity. Assume the current frame needs to perform multi-target detection and tracking... Querying the trajectory of new students and Decoding each trajectory query then inputs to the spatial attention layer (i.e.) Figure 3 The spatial attention mechanism network shown has the same feature dimension as the feature encoding. A query vector.

[0256] It should be noted that the number of historical trajectory queries... The number of key targets is determined by the number of key targets in the following embodiments, and its feature vector is obtained based on step S1033a, while the new trajectory query initializes the query vector according to the general target detection method.

[0257] It should be noted that, for trajectory queries, without considering... Figure 3 Given the key objective shown, there are two cases. In the first case, if the input is the first frame of the video sequence, the historical trajectory query is empty, and the input only contains the new trajectory query, then the output will only contain the corresponding features. In the second case, for any frame t>0 other than the first frame, the decoder outputs the following number of features: Each feature vector represents a new trajectory and a historical trajectory, respectively. Here, since not all targets appear in the first frame, the number of trajectory queries is not fixed; assuming... If the set of identities corresponds to the current frame, then it only represents the set of identities across the entire video sequence. A subset of [the set]. Furthermore, the number of trajectory queries also varies due to the impact of trajectory creation and removal. Trajectory removal is detailed in the following embodiments and will not be repeated here.

[0258] Step S1033c: Determine the new trajectory query corresponding to the previous frame and / or the historical trajectory query corresponding to each second original target as the trajectory query corresponding to the first original target. That is, the trajectory query corresponding to the first original target may include: the trajectory query corresponding to the new target that appeared in the previous frame of the current frame (e.g., the trajectory query result corresponding to the new target that appeared in frame t-1), and may also include: the trajectory query corresponding to the target that appeared before the previous frame (e.g., the trajectory query result corresponding to the target that appeared before frame t-1).

[0259] Furthermore, after obtaining the trajectory query corresponding to the first original target, the trajectory query corresponding to the first original target is input into the spatial attention model for further decoding processing. For details, please refer to the following embodiments, which will not be repeated here.

[0260] Step S1034: Based on the encoding features, relative position encoding, and trajectory query, and decoded by the feature decoder, the decoded features are obtained.

[0261] In the embodiments of this application, decoding via a feature decoder can specifically include two stages: first, feature decoding of the trajectory query in the current frame based on spatial self-attention and cross-attention; second, outputting the tracking result, extracting key targets, and optimizing the trajectory query for the next frame. In the embodiments of this application, the first stage of decoding is similar to encoding, both involving the calculation of self-attention. The difference is that the input to the self-attention layer is not a fusion of feature maps and relative position encoding, but rather a trajectory query. That is, in the network model, there are two types of position-related encoding: relative position encoding and output encoding (trajectory query). The former is the spatial position information input to the encoder, while the latter is essentially the feature representation that the model needs to learn for multi-object tracking. Furthermore, as... Figure 3 As shown, this output encoding (trajectory query) also participates in the computation of each attention layer, which includes self-attention and cross-attention, through residual connections.

[0262] Specifically, the feature decoder includes a multi-head cross-attention layer and a forward propagation layer; in step S1034, based on the encoded features, relative position encoding, and trajectory query, and decoded by the feature decoder, the decoded features are obtained. This can specifically include steps S1034a (not shown in the figure), S1034b (not shown in the figure), S1034c (not shown in the figure), S1034d (not shown in the figure), and S1034e (not shown in the figure), wherein...

[0263] Step S1034a: Input the trajectory query into the spatial attention model to obtain self-attention features.

[0264] Specifically, in this embodiment of the application, the new trajectory query and the historical trajectory query are input into the spatial attention model to obtain self-attention features.

[0265] Specifically, if the trajectory query includes the trajectory query corresponding to the first original target and the trajectory query corresponding to the newly generated target, then in step S1034a, the trajectory query is input into the spatial attention model to obtain self-attention features. This can specifically include: fusing the trajectory query corresponding to the first original target and the trajectory query corresponding to the newly generated target to obtain a fused trajectory query; and inputting the fused trajectory query into the spatial attention model to obtain self-attention features. That is, if... Figure 3 As shown, the new trajectory query and the historical trajectory query are fused to obtain the trajectory query, which is then input into the spatial attention model.

[0266] Step S1034b: Input the trajectory query and self-attention features into the query channel of the multi-head cross-attention layer, input the encoding features and relative position encoding into the key channel of the multi-head cross-attention layer, and input the encoding features into the value channel of the multi-head cross-attention layer to obtain the multi-head cross-attention features.

[0267] like Figure 3 As shown, the trajectory query, relative position encoding, and feature encoding after passing through the self-attention layer are input into the decoder together for multi-head cross-attention calculation. Cross-attention is also based on the attention mechanism formula (2), but it differs from self-attention in that its inputs Q and K are no longer the same. Q is the trajectory query, which is the fusion of its output after self-attention and its own through residual connection. K is the fusion of the feature encoding output by the encoder and the relative position encoding. In formula (2), V directly uses the feature encoding output by the encoder. In fact, the difference between Q and K is the essence of cross-attention and the core of the encoder-decoder framework.

[0268] Step S1034c: Perform residual and normalization processing on the multi-head cross-attention features to obtain the processed multi-head cross-attention features.

[0269] Specifically, the features after passing through the multi-head cross-attention layer are also residuald and normalized according to formula (3) and enter the third stage of decoding. Formula (3) will not be repeated here.

[0270] Step S1034d: Input the processed multi-head cross-attention features into the forward propagation layer to obtain the decoded features.

[0271] Furthermore, the third stage is the forward propagation layer. Similar to the encoding process, the decoder calculates the output of the fully connected layer according to formula (4) to obtain the decoded features.

[0272] Step S1034e: Perform residual and normalization calculations on the decoded features to obtain the decoded features.

[0273] Furthermore, after the decoder calculates the output of the fully connected layer according to formula (4), it performs residual and normalization on it according to formula (3) to obtain the final decoding features.

[0274] Specifically, in order to output a more reasonable and robust feature representation, the specific steps are as follows:

[0275] The first step is to merge historical trajectory queries and new trajectory queries to obtain the dimension as follows: Track query for d;

[0276] The second step is to perform self-attention calculation on the trajectory query;

[0277] The third step is to calculate the cross-attention features based on the output of self-attention, feature encoding, and relative position encoding.

[0278] The fourth step involves forward propagation, residual and normalization calculations of the output features of the cross-attention layer to obtain the final decoded features.

[0279] It should be noted that, for the target query, the number of... In this example, a fixed value of 100 is used, which represents the 100 newly appearing targets in the query scene. Typical models would choose a setting significantly larger than the maximum number of targets in the scene. Since this example involves multi-target tracking in a vehicle video, the number will generally not exceed 100. The specific setting can be adjusted according to the specific application scenario.

[0280] Furthermore, the decoder mentioned in this example has its loop structure stacked 6 times. Through this stacking method, target features accumulate information related to target location and category across multiple consecutive loop structures, making the final output features more reasonable. Physically speaking, the self-attention mechanism in decoding is mainly used to obtain the autocorrelation of targets, avoiding repeated tracking and detection of already tracked targets. The cross-attention mechanism, on the other hand, tends to update the target's feature representation by reasoning about the spatial relationships between targets using the global context information of the entire image. This implicitly includes its identity and location information, and due to the optimization method corresponding to step S1033a, it also implicitly includes motion information for targets with a historical trajectory.

[0281] Step S1035: Regress and classify the decoded features to obtain the tracking results of each target.

[0282] The tracking results for any target include: the target's location information from a bird's-eye view and the target's corresponding identity (or category) information.

[0283] Specifically, the decoded features obtained in the above embodiments are regressed and classified using a feedforward network (FFN). The basic structure of the feedforward network is two fully connected layers, and the calculation principle is consistent with formula (4). Among them, the regression task is used to output the position information of the target from a bird's-eye view, and the classification task is used to distinguish the identity (or category) of the target. It should be noted that the feedforward network (FFN) used for each output shares parameters.

[0284] like Figure 3 As shown, embodiments of this application can directly output or search for key targets from a bird's-eye view based on the position and size of the regression output, and optimize historical trajectory queries accordingly, thereby optimizing the calculation process of multi-target tracking in the next frame and reducing the computational overhead of the model when the number of targets is large. In practical applications, if the number of targets being tracked is large, the model can reduce the number of trajectory queries by focusing only on key targets that meet specific conditions, thereby reducing the computational complexity of the algorithm. The search strategy in embodiments of this application may include: searching for key targets through a threshold-based spatial attention mechanism, such as... Figure 3The attention mechanism network shown enables the model to extract key targets through learning. In this case, key targets in the real samples need to be labeled. Alternatively, a rectangular region can be defined around the vehicle based on the target's regression position as the key region, and targets falling within it are considered key targets. Another approach is to cover a specific area around the vehicle in a fan shape, strictly defining the radius, and heuristically select the first target to appear as the key target. Of course, a simple nearest neighbor search based on distance ranking statistics can also be used, such as identifying the top 10 targets closest to the observed subject. Furthermore, in autonomous driving, second-order relationships may exist, and a second-order search centered on first-order key targets can be attempted to obtain second-order key targets. Of course, adaptive or heuristic algorithms can be designed for specific thresholds, or these methods can be combined.

[0285] Furthermore, the above embodiments introduced methods for obtaining multi-target tracking results through feature encoders, feature decoders, regression modules, and classification modules. The following embodiments describe the training process of feature encoders, feature decoders, regression modules, and classification modules. In this application embodiment, the training process is mainly divided into four steps: collection and annotation of datasets, loading and augmentation, training of backbone networks, and training and testing of joint frameworks. For details, please refer to the following embodiments.

[0286] Furthermore, the method may also include: step Sh (not shown in the figure) and step Si (not shown in the figure), wherein,

[0287] Step Sh: Collect the dataset.

[0288] The dataset includes samples of at least one category and attribute information for each sample. It should be noted that the dataset collected in step Sh represents the dataset after collection, annotation, loading, and augmentation.

[0289] Specifically, data collection and labeling: mainly collect sensor data related to the application scenario, and divide the data into training set, validation set and test set according to the scenario and dataset size, with a ratio of 10:1:1;

[0290] Regarding dataset annotation, relevant technologies focus on multi-class, multi-target tracking datasets, primarily those related to autonomous driving and drones, such as KITTI and VisDrone. These datasets define target categories mainly based on functional classifications within traffic scenarios. Since this application uses vehicle sensors as an example, this approach is also adopted. A specific category classification could be: car, truck, bus, pedestrian, two-wheeled non-motorized vehicle (bicycle), three-wheeled non-motorized vehicle (tricycle), van, two-wheeled motorized vehicle (motor-bicycle), and three-wheeled motorized vehicle (motor-tricycle). During annotation, each category corresponds to a label. The basic attributes of each sample in the dataset include its frame number, identity information, category information, and its position and size within the corresponding frame. Furthermore, since this application includes a loss term for key targets, key targets are additionally labeled in the sample attributes. This annotation can be done manually or heuristically, and is therefore not difficult to obtain.

[0291] Meanwhile, to better distinguish between targets and non-targets, this embodiment also labels some static targets with strong physical meaning as background, and adds an ignore class to differentiate between meaningful and meaningless backgrounds. It should be noted that if the sensor uses a camera array, the camera number must also be labeled in the attributes of the target sample. Furthermore, this embodiment can also perform reinforcement training for target occlusion, which requires labeling the degree of occlusion for each sample during the labeling process; the specific labeling method will not be described in detail in this embodiment.

[0292] Furthermore, due to the difficulty in acquiring datasets and the large data requirements, data augmentation is generally necessary before loading. In this embodiment, the labeled dataset is a continuous video sequence. Because the time units are small, the differences between adjacent frames are very small. Loading frames sequentially for training would not be conducive to enhancing the robustness of the network. Therefore, in actual training, video frames are randomly sampled with small jumps; that is, the continuously input frames are not adjacent frames at fixed time unit intervals, but the temporal order is preserved. Simultaneously, at the sample level, a portion of targets are randomly removed, and negative samples from other frames are added, thereby increasing the perturbation of the original data. In addition, other methods beneficial for data augmentation are included in this embodiment, and the above methods are not intended to limit the scope of this embodiment.

[0293] Step Si: Train the model based on the dataset to obtain the trained model.

[0294] The model includes a feature encoder, a feature decoder, a regression module, and a classification module. The regression module performs regression on the decoded features, and the classification module classifies the decoded features. It should be noted that the model may also include a backbone network, which is used for feature extraction from each viewpoint image.

[0295] Specifically, training the model based on the dataset to obtain the trained model can include training the backbone network based on the dataset.

[0296] Specifically, in the training of the backbone network, the weights of the original pre-trained network are first loaded, and then the data-enhanced dataset is input into the network. According to the original scheme, the backbone network is fine-tuned on the dataset labeled in this embodiment to adapt it to the recognition of multiple targets and multiple classes.

[0297] Furthermore, since the model also includes a feature encoder, a feature decoder, a regression module, and a classification module, training the model based on the dataset to obtain the trained model can specifically include: steps Si1 (not shown in the figure), Si2 (not shown in the figure), Si3 (not shown in the figure), and Si4 (not shown in the figure). Step Si1 can be executed before, after, or simultaneously with step Si2; this is not limited in the embodiments of this application, as detailed below:

[0298] Step Si1: Determine the loss sum for each query.

[0299] Furthermore, to optimize the model and calculate the loss function, the matching results are needed. Let's assume we use... This represents the ground truth set of the target. This represents the set of N predictions output by the network. (Due to manual settings...) If the number of samples is greater than the actual number of samples, then y is filled with multiple background class targets to make its size N. The data association optimization objective between these two sets is to find the solution that minimizes the matching cost (as shown in formula (7)).

[0300] ; Formula (7);

[0301] in, Indicates according to the allocation scheme Predict the target Compared with real samples The matching cost, which follows the standard bipartite graph matching formula, can be solved using the Hungarian algorithm. Specifically, in this example, the matching cost needs to consider the accuracy of both identity (or category) prediction and location prediction. For each element in the real sample set... ,like ),at this time This indicates the identity (or category) label of the corresponding sample. This indicates the position of the corresponding sample in the image, and thus we can further define based on... For the target The probability of classification is The location prediction result is ,but The calculation method is shown in formula (8):

[0302] ;Formula (8);

[0303] in, Indicates a non-target class. For indicating constants.

[0304] Next, based on the matching results, the model loss can be calculated. Based on the cost function of Hungarian matching, this example uses the aggregate loss function of negative log-likelihood and coordinate loss, as shown in formula (9):

[0305] ; Formula (9);

[0306] ;Formula (10);

[0307] In formula (10), π represents the value based on... The matching result, if a target If it is assigned to a non-target class, then it does not belong to π. This represents the loss for each query. In practical applications, this is typically adjusted by setting weight parameters. The class size is used to address class imbalance issues; the specific parameters are determined by the number of targets in the scene and the preset value of N. In this example, a manually set constant is used directly. .

[0308] For the location prediction cost or loss in formula (10) This application embodiment uses a simple calculation method based on IOU and distance loss, as shown in formula (11), where,

[0309] ; Formula (11);

[0310] Because the L1 norm is used, a hyperparameter is added to formula (11). and This is used to balance the relative magnitudes of the two losses, and these two losses are normalized according to the number of samples involved in the calculation in the batch.

[0311] Furthermore, in this embodiment of the application, a loss term for calculating the recall rate of key targets is designed into the loss function of the network model, which is used to measure the network's attention to key targets and its expressive ability, as shown below.

[0312] Step Si2: Determine the loss in recall rate for key targets.

[0313] The loss in recall rate for key targets is detailed in formula (12), where,

[0314] Formula (12);

[0315] In formula (12), This indicates the number of key targets in the real sample. Represents a local minimum value. Based on distance The sorting operation here The goal The relative distances to the observed subject are used to sort the tracked targets in ascending order of proximity. This indicates a statistical operation based on ranking consistency, that is, the consistency between the ranking of the target identity in the real sample and the ranking in the predicted result. A consistent ranking is counted as 1, otherwise it is counted as 0.

[0316] Step Si3: Calculate the target loss function based on the loss of each query and the loss of the key target recall rate.

[0317] The target loss function is shown in formula (13).

[0318] ;Formula (13);

[0319] in, Used to balance the order of magnitude between two losses Used to characterize the sum of losses for each query. Loss is used to characterize the recall rate of key targets.

[0320] Step Si4: Train the model based on the dataset and using the target loss function to obtain the trained model.

[0321] Specifically, the model is trained iteratively multiple times based on the dataset and using the target loss function to obtain the trained model.

[0322] For a single iteration, the network input is a feature map output by the backbone network, and the tracking result is based on... Figure 2 as well as Figure 3 The network structure shown is output, and then the loss is calculated according to formula (13). For the entire network optimization, the accumulated loss value in a complete iteration or a batch is backpropagated, and the network parameters are updated according to gradient descent until the convergence condition is met or the preset number of iterations is reached.

[0323] After the model training is completed, testing is performed. First, the training weights are loaded, and then multi-class multi-target tracking is performed on the test set. The tracking results of each frame are written to a text document, where each data point contains the position and identity information of a target in a certain frame. Based on this, the tracking algorithm can be objectively evaluated using evaluation metrics in multi-target tracking.

[0324] Furthermore, after regressing and classifying the decoded features to obtain the tracking results for each target, the process may further include: creating existing trajectory set variables and / or terminating trajectory set variables; and managing the tracking results for each target based on the existing trajectory set variables and / or terminating trajectory set variables. In this embodiment, existing trajectory set variable O and / or terminating trajectory set variable G may be created first. In this embodiment, the pre-creation of existing trajectory set variable O and terminating trajectory set variable G will be used as an example.

[0325] Specifically, based on existing trajectory set variables and / or terminated trajectory set variables, the tracking results of each target are managed, which may include at least one of steps Sb1 (not shown in the figure), Sb2 (not shown in the figure), and Sb3 (not shown in the figure), wherein,

[0326] Step Sb1: Assign identity information to the new target and record the tracking results corresponding to the new target. In this embodiment, the tracking results carry the corresponding time unit tag.

[0327] Specifically, regarding the goals for new students, based on The system already has identities, so different identity IDs are assigned to them, and based on the corresponding network output results, their location size information and feature information with time unit labels are recorded.

[0328] Step Sb2: Remove the existing trajectories that meet the preset conditions from the existing trajectory set variable, and / or move the trajectories that meet the preset conditions into the termination trajectory set variable.

[0329] In the embodiments of this application, the removal of a trajectory refers to the removal of an existing trajectory set variable. Existing trajectories that meet certain preset conditions are removed from the existing trajectory set variable O. In this embodiment, trajectory removal may include: removing existing trajectories that meet preset conditions from the existing trajectory set variable O, or moving existing trajectories removed from the existing trajectory set variable O into the termination trajectory set variable G.

[0330] Specifically, in this application embodiment, two management strategies are set. The first is to terminate the process if the network output of a trajectory query is a non-target class (i.e., no matching target is found in the frame). The second is to calculate the intersection over union (IOU) of all trajectories obtained in each frame and perform non-maximum suppression (NMS) based on a manual threshold, retaining only the trajectories that meet the conditions, thereby further eliminating duplicate detections missed in the attention mechanism.

[0331] Furthermore, the termination trajectory set variable The trajectories in the dataset include not only the terminating trajectories generated during the removal process, but also the existing set of trajectories as of the end of the tracking process. All trajectories in the system. When no more video frames are input to the system, the existing trajectory set variable will be used. Move all trajectories into This yields the final set of output trajectories.

[0332] Furthermore, the method may also include: outputting the tracking results of each target in the termination trajectory set variable in a textual and / or visual manner, according to temporal sequence and / or identity information. In other words, the termination trajectory set variable... The trajectory information is output in text and / or visual format according to time sequence and identity, providing corresponding location and size information. (Set) The trajectory in the data includes not only the terminating trajectory generated during the removal process, but also the trajectory from the set at the end of the tracking process. All tracks in the system. When there are no more video frames input to the system, this technology will... Move all trajectories into The final set of output trajectories is obtained and written to a text document as the output trajectory. If visualization is required, a video output trajectory result can be generated based on the data in the document.

[0333] Step Sb3: Within the time delay threshold range, remove the delays of existing trajectories that meet the preset conditions.

[0334] Specifically, before step Sb3, the method may further include: obtaining a time delay threshold range. In this embodiment, the time delay threshold can be input by the user or preset by the system. In this embodiment, within the threshold time, trajectories that meet the removal conditions are delayed in removal, and for frames that are not matched, no trajectory information is recorded temporarily. During the delay period, the trajectory is still queried. If a tracking result is obtained again, the corresponding information is recorded in the existing trajectory set variable. In the corresponding trajectory, the position and size information during the delay period is recovered based on motion models such as Kalman filtering.

[0335] Since this application embodiment involves a method for extracting key targets in the current frame and using it as a historical trajectory query for the next frame, a trajectory query management strategy based on key targets needs to be designed. Therefore, the method may further include at least one of steps Sd (not shown in the figure), Se (not shown in the figure), Sf (not shown in the figure), and Sg (not shown in the figure), and step Sc (not shown in the figure), wherein...

[0336] Step Sc: Create trajectory set variables for key targets. In this embodiment, additional trajectory set variables for key targets are created. For each frame of output, this application embodiment designs four strategies to manage the set of key target trajectories. The four strategies designed in this application embodiment can be detailed in the implementation methods corresponding to steps Sd, Se, Sf, and Sg below. Furthermore, this application embodiment uses four strategies to manage the key target trajectory set L, but this is not a limitation of this application embodiment. For example, this application embodiment can also use one strategy, two strategies, three strategies, or even more than four strategies to manage the key target trajectory set L.

[0337] Step Sd: Store all key targets output in the current frame in the key target trajectory set variable. In other words, the key target trajectory set L contains all key targets output in the current frame.

[0338] Step Se: Store the historical key targets output within the time delay threshold range in the trajectory set variable of the key targets.

[0339] Specifically, for historical critical targets that are within the delayed removal period, they are still included in the critical target trajectory set even if no results are output in the current frame. middle;

[0340] Step Sf: If there are non-critical targets in the current frame output, then determine the non-critical targets that meet specific conditions from the output non-critical targets, and store the non-critical targets that meet specific conditions in the trajectory set variable of critical targets.

[0341] Among them, non-critical targets under specific conditions include: the non-critical target is transformed from a critical target, and the time of transformation from a critical target to a non-critical target falls within the time delay threshold range;

[0342] Step Sg: Store the newly generated target output in the current frame into the trajectory set variable of the key target.

[0343] In this embodiment of the application, all newly generated targets in the current frame are included in the key target trajectory set. middle.

[0344] It should be noted that sets Only as a variable related to trajectory query, and with sets The management of the trajectory is irrelevant; the latter is a variable used to manage whether the trajectory continues to exist. Both are used to receive the results of the model output and manage the input and output of the trajectory according to their respective management methods.

[0345] The above embodiments describe a multi-target tracking method based on a bird's-eye view from the perspective of the method flow. The following embodiments describe a multi-target tracking device based on a bird's-eye view from the perspective of a virtual module. For details, please refer to the following embodiments.

[0346] This application provides a multi-target tracking device based on a bird's-eye view, such as... Figure 4 As shown, the multi-target tracking device 40 based on a bird's-eye view may include: an acquisition module 41, a determination module 42, and a multi-target tracking processing module 43, wherein,

[0347] The acquisition module 41 is used to acquire the viewpoint image captured by each camera at the current moment;

[0348] The determination module 42 is used to determine the feature map under the bird's-eye view based on the view images acquired by each camera at the current moment;

[0349] The multi-target tracking processing module 43 is used for multi-target tracking processing based on feature maps from a bird's-eye view.

[0350] In one possible implementation of this application embodiment, when determining the feature map under the bird's-eye view based on the view images acquired by each camera at the current moment, the determining module 42 is specifically used to: extract features from each view image to obtain a corresponding feature map, the feature map including at least one of a multi-scale feature map, a handmade feature map, and a depth feature map; and map the corresponding feature map to the feature space under the bird's-eye view to obtain the feature map under the bird's-eye view.

[0351] In another possible implementation of this application embodiment, when determining the feature map by mapping the corresponding feature map to the feature space under the bird's-eye view, the module 42 is specifically used to: map the corresponding feature map to the feature space under the bird's-eye view through a transformer to obtain the feature map under the bird's-eye view; generate keys and values ​​in the transformer based on the corresponding feature map; perform pooling processing on the corresponding feature map to obtain a global vector; perform rasterization and position encoding on the feature space under the bird's-eye view to obtain a position-encoded feature space; concatenate the position-encoded feature space with global variables; generate a query in the transformer from the concatenated features; determine the relationship between the feature map under the bird's-eye view and the corresponding feature map based on the query and the key; and determine the feature map under the bird's-eye view based on the relationship and the value.

[0352] In another possible implementation of this application embodiment, when the multi-target tracking processing module 43 performs multi-target tracking processing based on the feature map under the bird's-eye view, it is specifically used to: obtain the relative position encoding between each target under the bird's-eye view; based on the relative position encoding and the feature map under the bird's-eye view, and perform feature encoding through a feature encoder to obtain coded features; obtain the trajectory query of each target, the trajectory query including the new trajectory query of the new target at the current time and the historical trajectory query of the target that appeared before the current time; based on the coded features, the relative position encoding, and the trajectory query, and perform decoding through a feature decoder to obtain decoded features; perform regression and classification on the decoded features to obtain the tracking results of each target, the tracking result of any target including: the position information of any target under the bird's-eye view and the target identity or category corresponding to any target.

[0353] In another possible implementation of this application embodiment, when the multi-target tracking processing module 43 obtains the relative position codes between targets under the bird's-eye view, it is specifically used to: obtain the relative distance between any two targets under the bird's-eye view; and convert each relative distance into a relative position code based on a learnable parameter layer to obtain the relative position codes between targets.

[0354] In another possible implementation of this application embodiment, when the multi-target tracking processing module 43 obtains encoded features based on relative position encoding and a feature map under a bird's-eye view by performing feature encoding through a feature encoder, it is specifically used to: reduce the feature channel dimension corresponding to the feature map under the bird's-eye view; collapse the spatial dimension of the feature map under the bird's-eye view after the feature dimension is reduced to a preset dimension; and obtain encoded features based on relative position encoding and the collapsed feature map under the bird's-eye view by performing feature encoding through a feature encoder.

[0355] In another possible implementation of this application embodiment, the feature encoder includes a multi-head self-attention layer and a forward propagation layer; when the multi-target tracking processing module 43 obtains encoded features based on the feature map under relative position encoding and bird's-eye view through the feature encoder, it specifically performs the following steps: inputting the feature map under relative position encoding and bird's-eye view into the query channel and key channel of the multi-head self-attention layer, and inputting the feature map under bird's-eye view into the value channel of the multi-head self-attention layer to obtain multi-head attention features; performing residual and normalization processing on the multi-head attention features to obtain residual and normalized features; performing feature encoding on the residual and normalized features through the forward propagation layer to obtain encoded features; and performing residual and normalization processing on the encoded features to obtain encoded features.

[0356] One possible implementation of this application embodiment is that trajectory query includes: trajectory query corresponding to a first original target, where the first original target is a target that has appeared before the current frame;

[0357] Specifically, when the multi-target tracking processing module 43 obtains the trajectory query corresponding to the original target, it is used to: obtain the new trajectory query corresponding to the previous frame, where the previous frame is the frame preceding the current frame, and the new trajectory query corresponding to the previous frame contains the trajectory query corresponding to the new target that appeared in the previous frame; and / or, for each second original target, obtain the historical features containing each second original target, and perform trajectory query based on the historical features containing each second original target to obtain the historical trajectory query corresponding to each second original target, where the second original target is the target that appeared before the previous frame; and determine the new trajectory query corresponding to the previous frame and / or the historical trajectory query corresponding to each second original target as the trajectory query corresponding to the first original target.

[0358] Another possible implementation of this application embodiment includes: querying the trajectory of a newly emerging target, wherein the newly emerging target is a newly appearing target in the current frame;

[0359] Specifically, when the multi-target tracking processing module 43 obtains the trajectory query corresponding to the new target, it is used to generate the new trajectory query corresponding to the new target based on the set number of queries.

[0360] In another possible implementation of this application embodiment, the feature decoder includes a multi-head cross-attention layer and a forward propagation layer; when the multi-target tracking processing module 43 obtains decoded features based on encoded features, relative position encoding, and trajectory query, and decodes them through the feature decoder, it specifically performs the following steps: inputting the trajectory query into the spatial attention model to obtain self-attention features; inputting the trajectory query and self-attention features into the query channel of the multi-head cross-attention layer, inputting the encoded features and relative position encoding into the key channel of the multi-head cross-attention layer, and inputting the encoded features into the value channel of the multi-head cross-attention layer to obtain multi-head cross-attention features; performing residual and normalization processing on the multi-head cross-attention features to obtain processed multi-head cross-attention features; inputting the processed multi-head cross-attention features into the forward propagation layer to obtain decoded features; and performing residual and normalization calculations on the decoded features to obtain decoded features.

[0361] In another possible implementation of this application embodiment, when the trajectory query includes the trajectory query corresponding to the first original target and the trajectory query corresponding to the new target, the multi-target tracking processing module 43, when inputting the trajectory query into the spatial attention model to obtain self-attention features, specifically performs the following: fusing the trajectory query corresponding to the first original target and the trajectory query corresponding to the new target to obtain the fused trajectory query; and inputting the fused trajectory query into the spatial attention model to obtain self-attention features.

[0362] In another possible implementation of this application embodiment, the apparatus 40 further includes: a first creation module and a tracking management module, wherein...

[0363] The first creation module is used to create existing trajectory set variables and / or terminate trajectory set variables;

[0364] The tracking management module is used to manage the tracking results of each target based on existing trajectory set variables and / or terminated trajectory set variables.

[0365] In another possible implementation of this application embodiment, when the tracking management module manages the tracking results of each target based on the existing trajectory set variables and / or the termination trajectory set variables, it is specifically used to: assign identity information to new targets and record the tracking results corresponding to the new targets, with the tracking results carrying the corresponding time unit tags; and / or, remove trajectories that meet preset conditions from the existing trajectory set variables, and / or, move trajectories that meet preset conditions into the termination trajectory set variables; and / or, within the time delay threshold range, delay the removal of trajectories that meet preset conditions from the existing trajectories.

[0366] In another possible implementation of this application embodiment, the apparatus 40 further includes at least one of a first storage module, a second storage module, a third storage module, and a fourth storage module, and a second creation module, wherein...

[0367] The second creation module is used to create trajectory set variables for key targets;

[0368] The first storage module is used to store all key targets output in the current frame in the trajectory set variable of the key targets;

[0369] The second storage module is used to store the historical key targets output within the time delay threshold range in the trajectory set variable of the key targets;

[0370] The third storage module is used to determine non-critical targets that meet specific conditions from the output non-critical targets when there are non-critical targets in the current frame output, and store the non-critical targets that meet specific conditions in the trajectory set variable of critical targets. The non-critical targets that meet specific conditions include: the non-critical targets are transformed from critical targets, and the time of transformation from critical targets to non-critical targets is within the time delay threshold range.

[0371] The fourth storage module is used to store the newly generated targets output in the current frame into the trajectory set variable of the key targets.

[0372] In another possible implementation of this application embodiment, the device 40 further includes: an output module, wherein...

[0373] The output module is used to output the tracking results of each target in the termination trajectory set variable in text and / or visual form, according to the time sequence and / or identity information.

[0374] In another possible implementation of this application embodiment, the device 40 further includes: a data acquisition module and a training module, wherein...

[0375] The data acquisition module is used to acquire a dataset, which includes samples of at least one category and attribute information of each sample.

[0376] The training module is used to train the model based on the dataset to obtain the trained model. The model includes a feature encoder, a feature decoder, a regression module, and a classification module. The regression module is used to regress the decoded features, and the classification module is used to classify the decoded features.

[0377] In another possible implementation of this application, when the training module trains the model based on the dataset to obtain the trained model, it is specifically used to: determine the sum of losses for each query; determine the loss for the recall rate of the key target; calculate the target loss function based on the sum of losses for each query and the loss for the recall rate of the key target; and train the model based on the dataset and through the target loss function to obtain the trained model.

[0378] This application provides a multi-target tracking device with a bird's-eye view. In this embodiment, the device acquires viewpoint images from each camera on the vehicle-mounted device at the current moment to determine feature maps under the bird's-eye view, and then performs multi-target tracking processing based on these feature maps. In other words, by projecting the scene onto a bird's-eye view, the device minimizes the unreliability and instability of appearance features caused by target occlusion due to viewpoint issues, creating conditions for accurate tracking. This allows the algorithm to adaptively utilize global information to distinguish different targets under the bird's-eye view, while reducing redundant detection. Furthermore, through encoding and decoding, target detection and tracking are further integrated, enabling the model to focus on fewer specific key targets, thus reducing the computational cost of the multi-target tracking process.

[0379] It should be noted that the first creation module and the second creation module can be the same creation module or different creation modules; the first storage module, the second storage module, the third storage module and the fourth storage module can be the same storage module, different storage modules, or partially the same storage module, and are not limited in the embodiments of this application.

[0380] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0381] This application provides an electronic device, such as... Figure 5 As shown, Figure 5 The illustrated electronic device 500 includes a processor 501 and a memory 503. The processor 501 and the memory 503 are connected, for example, via a bus 502. Optionally, the electronic device 500 may also include a transceiver 504. It should be noted that in practical applications, the transceiver 504 is not limited to one type, and the structure of this electronic device 500 does not constitute a limitation on the embodiments of this application.

[0382] Processor 501 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 501 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0383] Bus 502 may include a pathway for transmitting information between the aforementioned components. Bus 502 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 502 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0384] The memory 503 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0385] The memory 503 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 501. The processor 501 is used to execute the application code stored in the memory 503 to implement the content shown in the foregoing method embodiments.

[0386] Electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Servers can also be included. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0387] This application provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments. Compared with related technologies, this application embodiment acquires the viewpoint images captured by each camera on the vehicle-mounted device at the current moment to determine the feature map under the bird's-eye view, and performs multi-target tracking processing based on the feature map under the bird's-eye view. That is, by projecting the scene under the bird's-eye view, the unreliable and unstable appearance features caused by target occlusion due to viewpoint issues are minimized, creating conditions for accurate tracking. The algorithm can adaptively use global information to distinguish different targets under the bird's-eye view, while reducing redundant detection. Through encoding and decoding, target detection and tracking are further integrated, allowing the model to reduce the computational consumption of the multi-target tracking process by focusing on fewer specific key targets.

[0388] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0389] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0390] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0391] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0392] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0393] The above description of the embodiments is only used to provide a detailed introduction to the technical solutions of this application. However, the description of the above embodiments is only for the purpose of helping to understand the methods and core ideas of this application, and should not be construed as a limitation of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. A multi-target tracking method based on an aerial view perspective, characterized in that, The method comprises: acquiring a perspective image collected by each camera at a current time; determining a feature map in the bird's-eye perspective based on the perspective image collected by each camera at the current time; acquiring relative position encodings between targets in the bird's-eye perspective; based on the relative position encodings and the feature map in the bird's-eye perspective, and through a feature encoder, obtaining encoded features; acquiring trajectory queries of the targets, the trajectory queries including a new trajectory query of a new target at the current time and a historical trajectory query of a target that has appeared before the current time; based on the encoded features, the relative position encodings and the trajectory queries, and through a feature decoder, obtaining decoded features; regressing and classifying the decoded features to obtain tracking results of the targets, the tracking result of any target including position information of the target in the bird's-eye perspective and a target identity or category corresponding to the target.

2. The method of claim 1, wherein, The method comprises: extracting features from each perspective image to obtain a corresponding feature map, the feature map including at least one of a multi-scale feature map, a handcrafted feature map and a deep feature map; mapping the corresponding feature map to a feature space in the bird's-eye perspective to obtain the feature map in the bird's-eye perspective.

3. The method of claim 2, wherein, The method comprises: mapping the corresponding feature map to a feature space in the bird's-eye perspective through a transformer to obtain the feature map in the bird's-eye perspective; generating keys and values in the transformer based on the corresponding feature map; performing pooling processing on the corresponding feature map to obtain a global vector; performing rasterization and position encoding on the feature space in the bird's-eye perspective to obtain a position-encoded feature space; performing splicing processing on the position-encoded feature space and the global vector; generating a query in the transformer based on the spliced feature; determining a relationship between the feature map in the bird's-eye perspective and the corresponding feature map based on the query and the keys; determining the feature map in the bird's-eye perspective based on the relationship and the values.

4. The method of claim 1, wherein, The method comprises: acquiring relative distances between any two targets in the bird's-eye perspective; transforming the relative distances into relative position encodings based on a learnable parameter layer to obtain relative position encodings between the targets.

5. The method of claim 1, wherein, The method comprises: reducing a feature channel dimension corresponding to the feature map in the bird's-eye perspective; collapsing a spatial dimension of the feature map in the bird's-eye perspective after the feature dimension is reduced to a preset dimension; based on the relative position encodings and the feature map in the bird's-eye perspective after the collapse, and through a feature encoder, obtaining encoded features.

6. The method according to claim 4 or 5, characterized in that, The feature encoder comprises a multi-head self-attention layer and a forward propagation layer; the feature encoding is performed based on the relative position encoding and the feature map under the bird's eye view, and the encoded feature is obtained through the feature encoder, comprising: The relative position encoding and the feature map under the bird's eye view are input into the query channel and the key channel of the multi-head self-attention layer, and the feature map under the bird's eye view is input into the value channel of the multi-head self-attention layer, to obtain multi-head attention features; The multi-head attention features are subjected to residual and normalization processing to obtain features after residual and normalization processing; The features after residual and normalization processing are subjected to feature encoding through the forward propagation layer to obtain encoded features; The encoded features are subjected to residual and normalization processing to obtain the encoded features.

7. The method of claim 1, wherein, The trajectory query comprises: The trajectory query corresponding to the first original target, wherein the first original target is a target that has appeared before the current frame; wherein the trajectory query corresponding to the original target is obtained, comprising: Obtaining a new trajectory query corresponding to the previous frame, wherein the new trajectory query corresponding to the previous frame contains a trajectory query corresponding to a new target appearing in the previous frame; and / or for each second original target, obtaining historical features containing each second original target, and performing trajectory query based on the historical features containing each second original target to obtain a historical trajectory query corresponding to each second original target, wherein the second original target is a target that has appeared before the previous frame; The new trajectory query corresponding to the previous frame and / or the historical trajectory query corresponding to each second original target are determined as the trajectory query corresponding to the first original target.

8. The method of claim 1, wherein, The trajectory query further comprises: The new trajectory query corresponding to the new target, wherein the new target is a target newly appearing in the current frame; wherein the trajectory query corresponding to the new target is obtained, comprising: The new trajectory query corresponding to the new target is generated based on the set number of queries.

9. The method of claim 1, wherein, The feature decoder comprises a multi-head cross-attention layer and a forward propagation layer; the decoding is performed based on the encoded feature, the relative position encoding and the trajectory query, and the decoded feature is obtained through the feature decoder, comprising: The trajectory query is input into the spatial attention model to obtain self-attention features; The trajectory query and the self-attention features are input into the query channel of the multi-head cross-attention layer, the encoded feature and the relative position encoding are input into the key channel of the multi-head cross-attention layer, and the encoded feature is input into the value channel of the multi-head cross-attention layer to obtain multi-head cross-attention features; The multi-head cross-attention features are subjected to residual and normalization processing to obtain processed multi-head cross-attention features; The processed multi-head cross-attention features are input into the forward propagation layer to obtain decoded features; The decoded features are subjected to residual and normalization calculation to obtain decoded features.

10. The method of claim 9, wherein, If the trajectory query includes a trajectory query corresponding to a first original target and a trajectory query corresponding to a new target, the inputting the trajectory query into the spatial attention model to obtain self-attention features includes: fusing the trajectory query corresponding to the first original target and the trajectory query corresponding to the new target to obtain a fused trajectory query; inputting the fused trajectory query into the spatial attention model to obtain self-attention features.

11. The method of claim 1, wherein, After the decoding features are regressed and classified to obtain tracking results of each target, the method further includes: creating an existing trajectory set variable and / or a terminated trajectory set variable; managing the tracking results of each target based on the existing trajectory set variable and / or the terminated trajectory set variable.

12. The method of claim 11, wherein, The management of the tracking results of each target based on the existing trajectory set variable and / or the terminated trajectory set variable includes: allocating identity information to the new target and recording tracking results corresponding to the new target, the tracking results carrying corresponding time unit labels; and / or removing trajectories in the existing trajectory set variable that meet a preset condition, and / or moving the trajectories that meet the preset condition into the terminated trajectory set variable; and / or delaying the removal of trajectories in the existing trajectory set variable that meet a preset condition within a time delay threshold range.

13. The method of claim 12, wherein, The method further includes: creating a trajectory set variable of key targets; storing all key targets output in a current frame in the trajectory set variable of key targets; and / or storing historical key targets output within the time delay threshold range in the trajectory set variable of key targets; and / or if there are non-key targets output in the current frame, determining non-key targets that meet a specific condition from the output non-key targets and storing the non-key targets that meet the specific condition in the trajectory set variable of key targets, the non-key targets that meet the specific condition including: the non-key targets are converted from key targets, and the time of conversion from key targets to non-key targets belongs to the time delay threshold range; and / or storing a new target output in the current frame in the trajectory set variable of key targets.

14. The method of claim 12, wherein, The method further includes: outputting tracking results of each target in the terminated trajectory set variable in a chronological order and / or according to identity information in a text manner and / or a visual manner.

15. The method of claim 1, wherein, The method further includes: collecting a data set, the data set including samples of at least one category and attribute information of each sample; training a model based on the data set to obtain a trained model, the model including the feature encoder, the feature decoder, a regression module, and a classification module, the regression module being configured to regress the decoding features, and the classification module being configured to classify the decoding features.

16. The method of claim 15, wherein, The training of the model based on the data set to obtain the trained model includes: determining a loss sum of each query; determining a loss of a key target recall rate; calculating a target loss function based on the loss sum of each query and the loss of the key target recall rate; Training a model based on the data set and through the target loss function to obtain a trained model.

17. An aerial view based multi-target tracking apparatus, characterized by, The device comprises: An acquisition module configured to acquire a perspective image captured by each camera at a current time; A determination module configured to determine a feature map in the bird's-eye perspective based on the perspective image captured by each camera at the current time; A multi-target tracking processing module configured to acquire relative position encodings between targets in the bird's-eye perspective; encode features based on the relative position encodings and the feature map in the bird's-eye perspective through a feature encoder to obtain encoded features; acquire a trajectory query of each target, the trajectory query including a new trajectory query of a new target at the current time and a historical trajectory query of a target that has appeared before the current time; decode based on the encoded features, the relative position encodings and the trajectory query through a feature decoder to obtain decoded features; and regress and classify the decoded features to obtain a tracking result of each target, the tracking result of any target including position information of the any target in the bird's-eye perspective and a target identity or category corresponding to the any target.

18. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions, which are suitable for being loaded and executed by the processor, and perform the method steps of any one of claims 1-16.

19. An electronic device, comprising: Comprise: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded and executed by the processor, and performs the method steps of any one of claims 1-16.

Images