A method and apparatus for multi-target tracking from a bird's-eye view

Abstract

This invention relates to a multi-target tracking method and apparatus, electronic device, and computer-readable storage medium from a bird's-eye view. The method addresses the occlusion and spatiotemporal feature modeling problems in existing video multi-target tracking technologies. It acquires image information of the current vehicle's environment and its motion information, projects this information onto a bird's-eye view, and performs lightweight image matching frame-by-frame on a second-by-second basis for targets within a given time interval to obtain trajectory segments and spatiotemporal features. For targets in adjacent time intervals, online multi-target tracking is performed second-by-second using their respective spatiotemporal features based on an end-to-end encoding / decoding model. This method significantly mitigates the problems of target occlusion and unreliable appearance features caused by the viewing angle, thereby greatly reducing the impact of viewing angle issues on multi-target tracking performance and improving tracking speed and accuracy.

Description

Technical Field

[0001] This invention relates to the fields of intelligent driving and computer vision technology, specifically to a multi-target tracking method and device, electronic device, and computer-readable storage medium from a bird's-eye view. Background Technology

[0002] Multiple object tracking (MOT) is a technique for studying the changes in the positional information of an uncertain number of targets over time, and it is an important component in many practical applications. Video-based multi-object tracking plays a crucial role in autonomous driving, human-computer interaction, video surveillance, and military and aerospace fields.

[0003] With the development of computer vision-related hardware and software technologies, video-based perception has become the preferred method for machine perception, leading to increasing attention on vision-based multi-object tracking. However, in practical applications, many challenges remain to be addressed due to various factors and limitations. For example, image blurring, background and lighting changes generated during video acquisition pose significant difficulties for feature extraction. Furthermore, the classic target interaction problem in multi-object tracking, including the interaction between the target and the background, as well as the mutual influence between targets, coupled with variations in the target's shape, scale, and angle, presents numerous challenges in video multi-object tracking, especially in 2D video-based tracking. Under these physical implications, the design of video multi-object tracking algorithms requires comprehensive consideration of many issues, such as target occlusion, the description and modeling of target relationships, high computational overhead, and the difficulty in balancing accuracy and precision. Moreover, vision-based multi-object tracking, such as in autonomous driving technology, primarily relies on low-cost video acquisition, which is generally based on 2D video. Therefore, how to solve real-world 3D problems using 2D video has become an insurmountable challenge for many practical applications. In fact, the target occlusion problem also stems from this, and occlusion is essentially a problem of incomplete information caused by the viewpoint. Therefore, designing a multi-target tracking algorithm that can deal with the above problems has important theoretical significance and practical application value.

[0004] To understand the current state of technological development, the following technical solutions have been proposed:

[0005] The related technology discloses a multi-view multi-target detection and tracking method. This technology, on the one hand, uses a deep learning network to perform frame-by-frame online tracking of targets monitored by a single camera, generating tracking trajectory segments. On the other hand, it performs unsupervised clustering of the same target from different viewpoints, and obtains the tracking trajectory based on the clustering results and the tracking trajectory segments from each camera, enabling simultaneous detection and tracking of multiple targets in a public area by multiple cameras. However, its overall model design is relatively simple, mainly designed for monitoring videos in public areas, lacking generalization and difficult to transfer to other multi-target tracking application scenarios. Furthermore, in the process of merging multi-view trajectories, this scheme adopts a strategy of tracking separately from each viewpoint before clustering the trajectories. It lacks specific measures for viewpoint alignment problems in complex or unpredictable scenarios, leading to highly unstable tracking performance in practical problems. Simultaneously, this method is essentially a hierarchical clustering method, which is highly sensitive to the number of targets. When the number of targets is large, the computational cost of the algorithm is difficult to keep within the acceptable range of the system.

[0006] This paper discloses a deep learning-based method for multi-target vehicle tracking in drones. The method improves the algorithm's adaptability to targets of different sizes by introducing a variable convolutional deep model, and uses Oriented R-CNN to detect vehicle orientation, thereby tracking targets in drone-captured videos. Although this approach is a top-down view method for multi-target video tracking, its design focuses primarily on target feature extraction, especially for small targets. This is because the method is designed specifically for drone videos, and its overemphasis on specific target groups results in poor generalization and difficulty in transferring to other practical applications. Furthermore, the multi-target tracking model is based on the Re-ID concept, focusing excessively on target feature extraction while neglecting the modeling of relationships between targets. This leads to poor performance when there are many targets and frequent interactions. Additionally, the overly detailed target features result in computational overhead that is unacceptable for many practical applications when the number of targets is large. Moreover, the simple design based on intersection-union ratio (IoU) in the matching process makes it difficult to guarantee the model's robustness.

[0007] This paper discloses a multi-target tracking method based on multi-camera fusion. This technique improves the ID swapping problem caused by occlusion in multi-target detection by using multi-camera fusion technology and auxiliary viewpoints to correct the matching. The scheme provides a tracking framework combined with deep learning, specifically designed for occlusion in a single viewpoint. However, its overall model design is relatively simple and idealistic, lacking generalization ability for practical problems and difficult to transfer to other multi-target tracking applications. Furthermore, the scheme lacks targeted improvements to the target detection algorithm, Kalman filter, and Hungarian algorithm used in the tracking process, and lacks strategies to cope with complex scenes, resulting in highly unstable tracking performance in real-world problems. Additionally, the cascaded matching used in this method is highly sensitive to the number of targets; when the number of targets is large, the computational cost of the algorithm is difficult to estimate stably in practical problems.

[0008] This paper proposes an attention-based tracking paradigm. This technique models multi-object tracking as a set prediction problem, unifying detection and tracking in multi-object tracking based on encoding and decoding, enabling end-to-end training of the model through joint optimization. While this work is prominent in pedestrian multi-object tracking, it is difficult to transfer to other real-world multi-object tracking scenarios. Furthermore, the model addresses the alignment between the encoding / decoding framework and the physical meaning of the multi-object tracking process, lacking solutions and strategies for many real-world problems, such as occlusion and a large number of targets.

[0009] This paper proposes a spatiotemporal graph-based encoding-decoding model. This model effectively models the spatiotemporal relationships between targets in multi-target tracking and proposes a cascaded data association framework to address low-confidence detection and long-term occlusion, thereby further improving the model's tracking performance. While this technique achieves good results in pedestrian multi-target tracking, it lacks generalization ability to real-world scenarios. Furthermore, the cascaded framework increases the correlation between tracking performance and human experience. Simultaneously, the spatiotemporal graph requires the construction of a complex graph structure and high-dimensional tensor calculations, making the model sensitive to the number of targets and its computational cost unsatisfactory for practical applications.

[0010] In summary, although the relevant technologies are all based on computer vision algorithms and solve the problem of multi-object tracking in video sequences by extracting features, detecting objects, and associating data from video frames, and have achieved results to varying degrees, they have not yet effectively addressed the performance degradation caused by viewpoint issues. Summary of the Invention

[0011] In view of this, the purpose of the present invention is to provide devices, electronic devices, and computer-readable storage media to solve the problem of target occlusion caused by the viewpoint problem in the prior art, which affects the performance of multi-target tracking.

[0012] According to a first aspect of the present invention, a multi-target tracking method from a bird's-eye view is provided, comprising:

[0013] Acquire image information of the current vehicle's environment and the vehicle's motion information within the current unit of time. The image information includes at least multiple video frames captured within a unit of time from the perspective of a camera.

[0014] Multi-scale feature maps for each video frame are extracted using a pre-trained feature extraction model.

[0015] Based on the motion information, the multi-scale feature maps at the same moment from different camera perspectives are projected onto the feature space from the bird's-eye view to obtain the feature map from the bird's-eye view at that moment.

[0016] Target detection is performed on the feature map to obtain the target from the bird's-eye view at that moment;

[0017] For multiple video frames captured per unit time from each camera's perspective, perform the following steps:

[0018] Based on the lightweight graph matching method, the trajectory segments of each target per unit time are obtained;

[0019] Based on the trajectory segments and the relative positions of each target, spatiotemporal characteristics describing each target itself and the relationships between targets within a unit of time are obtained.

[0020] Based on the spatiotemporal characteristics of each target in adjacent time units, multi-target tracking is performed in adjacent time units to obtain the multi-target tracking results from the current bird's-eye view.

[0021] According to a second aspect of the present invention, a multi-target tracking device with a bird's-eye view is provided, comprising:

[0022] The acquisition unit is used to acquire image information of the current vehicle's environment and the current vehicle's motion information within a current unit of time; the image information includes at least multiple video frames acquired within a unit of time from a camera's perspective;

[0023] The extraction unit is used to extract multi-scale feature maps for each video frame using a pre-trained feature extraction model.

[0024] The projection unit is used to project multi-scale feature maps at the same moment from different camera perspectives onto the feature space at the bird's-eye view based on the motion information, so as to obtain the feature map at the bird's-eye view at that moment.

[0025] The detection unit is used to perform target detection on the feature map to obtain the target from the bird's-eye view at that moment;

[0026] The execution unit is used to perform the following steps on multiple video frames acquired per unit time from each camera's viewpoint:

[0027] Based on the lightweight graph matching method, the trajectory segments of each target per unit time are obtained;

[0028] Based on the trajectory segments and the relative positions of each target, spatiotemporal characteristics describing each target itself and the relationships between targets within a unit of time are obtained.

[0029] Based on the spatiotemporal characteristics of each target in adjacent time units, multi-target tracking is performed in adjacent time units to obtain the multi-target tracking results from the current camera perspective.

[0030] According to a third aspect of the present invention, an electronic device is provided, characterized in that it comprises:

[0031] A communication module, a processor, and a memory, wherein the memory stores program instructions;

[0032] The processor is used to execute program instructions stored in the memory and perform the above-described method.

[0033] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having an erasable and rewritable computer program stored thereon.

[0034] When the computer program is run on a computer device, the computer device performs the above-described method.

[0035] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0036] To address the occlusion and spatiotemporal feature modeling issues in existing video multi-target tracking technologies, this paper collects image information of the current vehicle's environment and its motion information, projects it onto a bird's-eye view, and performs lightweight graph matching frame by frame for targets within a unit of time, obtaining trajectory segments and spatiotemporal features. For targets in adjacent units of time, online multi-target tracking is performed second by second using their respective spatiotemporal features based on an end-to-end encoding and decoding model. This approach minimizes the impact of viewpoint-related target occlusion and unreliable appearance features, thereby significantly reducing the impact of viewpoint issues on multi-target tracking performance and improving tracking speed and accuracy.

[0037] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0038] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0039] Figure 1 This is a flowchart illustrating a multi-target tracking method from a bird's-eye view, according to an exemplary embodiment;

[0040] Figure 2 This is a schematic diagram illustrating the spatiotemporal characteristics of each target and the targets in relation to each other within a unit of time, according to an exemplary embodiment.

[0041] Figure 3 This is a schematic diagram of the structure of a feature encoder according to an exemplary embodiment;

[0042] Figure 4 This is a schematic diagram of the structure of a feature decoder according to an exemplary embodiment;

[0043] Figure 5 This is a schematic block diagram of a multi-target tracking device from a bird's-eye view, according to an exemplary embodiment. Detailed Implementation

[0044] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0045] As described in the background section, existing technologies suffer from target occlusion due to viewpoint issues, which affects the performance of multi-target tracking. To address these shortcomings, this invention provides the following exemplary embodiments. It should be noted that:

[0046] 1. In the embodiments of the present invention, "current vehicle" refers to "the vehicle where the vehicle-mounted camera is located".

[0047] 2. The “target” mentioned in the embodiments of the present invention refers to various static targets (e.g., green trees, signs, mud pits, stones, etc.) and / or dynamic targets (e.g., moving vehicles, pedestrians, animals, etc.) on the lane observed by the “current vehicle” as the observation subject.

[0048] 3. The "unit time" mentioned in the embodiments of the present invention can be set according to user needs, for example, set to 1 second. Each vehicle-mounted camera acquires multiple video frames within a unit time. The number of frames is determined by the acquisition frame rate of the vehicle-mounted camera. That is, the image information acquired by each vehicle-mounted camera is a video sequence composed of multiple video frames.

[0049] Example 1

[0050] Figure 1 This is a flowchart illustrating a multi-target tracking method from a bird's-eye view, according to an exemplary embodiment, such as... Figure 1 As shown, the method includes:

[0051] Step S11: Obtain image information of the current vehicle's environment and motion information of the current vehicle within the current unit time. The image information includes at least multiple video frames captured within a unit time from the perspective of a camera.

[0052] Step S12: Extract multi-scale feature maps for each video frame using a pre-trained feature extraction model;

[0053] Step S13: Based on the motion information, project the multi-scale feature maps at the same moment from different camera perspectives onto the feature space of the bird's-eye view to obtain the feature map of the bird's-eye view at that moment.

[0054] Step S14: Perform target detection on the feature map to obtain the target from the bird's-eye view at that moment;

[0055] Step S15: For multiple video frames captured per unit time from each camera's viewpoint, perform the following steps:

[0056] Step S151: Based on the lightweight graph matching method, obtain the trajectory segments of each target per unit time.

[0057] Step S152: Based on the trajectory segments and the relative positional relationships between each target, obtain the spatiotemporal characteristics of each target itself and the relationships between each target within a unit of time.

[0058] Step S153: Based on the spatiotemporal characteristics of each target in adjacent time units, perform multi-target tracking in adjacent time units to obtain the multi-target tracking results from the current camera perspective.

[0059] It should be noted that the application scenarios applicable to the technical solution provided in this embodiment include, but are not limited to, autonomous driving and assisted driving of vehicles. In actual use, the technical solution provided in this embodiment can be loaded into the current vehicle's central control system or into electronic devices that can control the current vehicle; these electronic devices include, but are not limited to, onboard computers and external computer devices.

[0060] The technical solution provided in this embodiment is applicable to various vehicles equipped with vehicle-mounted cameras, which can be one or multiple. When there is one vehicle-mounted camera, the technical solution provided in this embodiment achieves target tracking from the perspective of a single camera; when there are multiple vehicle-mounted cameras, the technical solution provided in this embodiment achieves target tracking from the perspectives of multiple cameras.

[0061] It is understandable that in the field of intelligent driving, target tracking in the vehicle's surrounding environment involves both temporal and spatial target tracking. The technical solution provided in this embodiment considers image information from other camera perspectives in the spatial domain when determining the target from each camera's viewpoint. By projecting image information collected from different camera perspectives onto a bird's-eye view before determining the target in each video frame, the consistency of the determined target in the spatial domain can be guaranteed. Furthermore, through the cooperation of multiple camera perspectives, the projection onto the bird's-eye view effectively solves the problem of target occlusion caused by perspective issues in existing technologies, which affects multi-target tracking performance.

[0062] To facilitate understanding of the technical solution provided in this embodiment, the implementation method of each step of the technical solution provided in this embodiment is explained below:

[0063] Step S11, "acquiring image information of the current vehicle's environment and motion information within the current unit of time," specifically refers to:

[0064] The vehicle-mounted camera acquires image information of the current vehicle's environment within a given time period. If there is only one vehicle-mounted camera, it acquires image information from the perspective of one camera; if there are multiple vehicle-mounted cameras, it acquires image information from multiple camera perspectives. The image information from different camera perspectives within a given time period is grouped and processed, and each group of image information is a video sequence composed of multiple video frames.

[0065] The vehicle's motion information is obtained through onboard inertial measurement sensors.

[0066] Step S12, "Extracting multi-scale feature maps for each video frame using a pre-trained feature extraction model," includes:

[0067] Multi-scale feature maps are extracted using a backbone network and a Feature Pyramid Network (FPN). Preferably, the technical solution provided in this embodiment employs a combined framework of the RegNet model and the BiFPN model to extract multi-scale feature maps for each video frame.

[0068] The BiFPN model can have four scales. Assuming the height and width of the current video frame are H and W respectively, then four scales can be obtained. The feature maps of the four scales output by the BiFPN model are then upsampled and stitched together to obtain the multi-scale feature maps of the video frame.

[0069] Step S13, "based on the motion information, projecting the multi-scale feature maps at the same moment from different camera viewpoints onto the feature space of the bird's-eye view to obtain the feature map of the bird's-eye view at that moment," includes:

[0070] A transformer-based feature transformation method is employed. The multi-scale feature maps described above are processed through a multilayer perceptron to generate keys and values ​​in the transformer. Simultaneously, global pooling is performed on the multi-scale feature maps to obtain a global vector. The output feature space is then rasterized and positionally encoded. These positional codes are concatenated with the global vector, and subsequently processed through a multilayer perceptron to obtain the query in the transformer. In the computation, the query dimension is consistent with the feature space dimension in the bird's-eye view. Since both keys and values ​​reside in 2D image coordinate space, the final result requires attention between the query and keys to determine the weights of the 2D image pixels received by each bird's-eye view raster. This determines the relationship between the feature map in the bird's-eye view and the input. Finally, these weights are used to weighted sum the values ​​in the image plane to obtain the feature map in the bird's-eye view.

[0071] In this embodiment, the feature map from the bird's-eye view is denoted as... Its dimension is H BEV *W BEV *C. Where C is the number of feature channels, t∈1,2,...,T, t represents the frame number, and T represents the length of the video sequence per unit time, i.e., the frame rate.

[0072] Step S14, "Perform target detection on the feature map to obtain the target from the bird's-eye view at that moment," includes:

[0073] By using a pre-set multi-target detection algorithm (e.g., the DETR target detection algorithm), the targets contained in each video frame from a bird's-eye view are obtained.

[0074] Step S15: For multiple video frames captured per unit time from each camera's viewpoint, perform the following steps:

[0075] Step S151: Based on the lightweight graph matching method, obtain the trajectory segments of each target per unit time, including:

[0076] 1. Starting from the first frame, assign identity identifiers and sorting numbers to targets in each frame in sequence;

[0077] 2. Based on the identity identifier and sorting number, obtain the graph representation corresponding to each frame. The graph representation is used to describe the positional relationship between each target and to determine the similarity of the graph structure between two frames.

[0078] 3. Starting from the second frame, according to the diagram, targets with the same identity in each frame are associated together to obtain the trajectory segments of each target per unit time.

[0079] Step S152: Based on the trajectory segments and the relative positional relationships between the targets, obtain the spatiotemporal characteristics describing each target itself and the relationships between targets per unit time (for ease of understanding of the spatiotemporal characteristics mentioned in this embodiment, the abstract spatiotemporal characteristics of each target itself are now referred to as...). Figure 2 The solid elliptical lines in the diagram illustrate the spatiotemporal characteristics of the abstract targets, using, for example... Figure 2 The dashed lines in the diagram illustrate this; it should be noted that during the actual execution of the algorithm, Figure 2 The diagram shown is not actually generated; it is merely a schematic representation of an abstract concept.

[0080] Step S153: Based on the spatiotemporal characteristics of each target in adjacent unit time, perform multi-target tracking in adjacent unit time to obtain the multi-target tracking result under the current camera view.

[0081] See Figure 2 Starting from the first frame, assign identity identifiers to targets in each frame sequentially (e.g., ...). Figure 2 Circles, triangles, squares, and pentagons in the text. Figure 2 The five-pointed star in the image represents the current vehicle. It should be noted that... Figure 2 These identifiers are merely for ease of understanding, to concretize abstract concepts, and are presented as a schematic diagram (this diagram will not be generated during the actual execution of the algorithm), and are assigned sorting numbers (such as...). Figure 2 The numbers 1, 2, and 3 on each target are as follows:

[0082] For the first frame, prioritizing the X-axis and traversing from the top left corner in coordinate order, each target is assigned an identity identifier in turn.

[0083] For other frames, an identity identifier is assigned to the target based on the matching results with all previous frames;

[0084] For each frame, calculate the relative distance between each target and the current vehicle, and assign a sorting number to each target in ascending order of the relative distance.

[0085] Starting from the second frame, according to the diagram representation (e.g.) Figure 2The graph representation corresponding to frame number t=1, the graph representation corresponding to t=2, ..., the graph representation corresponding to t=30, sequentially associates targets with the same identity in each frame, including:

[0086] If the number of targets in two adjacent frames is equal and the graph structure similarity is 1, then targets with the same identity in the two adjacent frames are associated together.

[0087] If the number of targets in two adjacent frames is not equal or the graph structure similarity is not 1, a fast graph matching method based on top-1 greedy matching is used to associate the nodes with the highest similarity between the two graph representations.

[0088] Specifically, for cases where the number of targets is the same in two adjacent frames (such as...) Figure 2 (As shown in the first three frames), based on the identity identifier and sorting number of each target, an accumulation operator N is designed to count the number of nodes in two frames whose identity identifier and sorting number are completely consistent, and to calculate the graph structure similarity between the two frames accordingly:

[0089]

[0090] In formula (1), S represents the graph structure similarity between two frames, i represents the target's identity identifier, n represents the number of targets within the frame, N is the accumulation operator, R is the distance ranking operator, and d is the relative distance between the target and the observing subject. The ranking operator sorts the targets by power of their relative distances to the observing subject. The result of the accumulation operator is the number of nodes in the two graph representations whose identity identifiers and ranking numbers are consistent, i.e., the number of targets in the two graph representations that have the same identity identifier and the same distance ranking relative to the observing subject. The result obtained by formula (1) is the graph structure similarity between two frames. When the similarity is 1, the targets in the two frames are directly associated based on their identity identifiers (e.g., ...). Figure 2 (The cases of frames t=1 and t=2).

[0091] For two consecutive frames with different target counts or similarity scores not equal to 1, a top-1 greedy fast graph matching method is used for data association. Here, top-1 greediness associates the nodes with the highest similarity between the two graphs. Specifically, for targets belonging to different frames in two consecutive frames, the similarity is calculated sequentially to obtain a similarity matrix A. Then, the association matrix is ​​obtained by solving the top-1 greedy strategy, thus revealing the matching relationship between each target in the two frames. Each element A in matrix A... ij The similarity between target i and target j, which belong to different frames, is calculated using the following formula:

[0092] A ij =αSim L +βSimB +γSim F (2)

[0093] In the formula, Sim L Sim represents the spatial similarity of targets. B Sim represents the scale similarity of targets. F This represents the extracted deep feature similarity, where α, β, and γ control the effect of these three similarities on the overall similarity A. ij The influence weights. Alternative solutions include the Hungarian algorithm, attention-based neural network algorithms, or other graph-based solution algorithms. In this example, a top-1 algorithm based on a greedy strategy is used because, when targets are dense, the complexity of the greedy algorithm is stably positively correlated with the number of targets and the dimensionality of the target features, and can be manually constrained, such as in Sim... L or Sim B A threshold is added to the calculation to control the sparsity of the similarity matrix, thereby making the matching process lightweight. However, methods such as the Hungarian algorithm have a cubic computational complexity, which fluctuates greatly with the density of the targets and is difficult to estimate. Even with the addition of a threshold, it may still consume a lot of computational power, which is inconsistent with the lightweight design idea.

[0094] If the number of targets in two adjacent frames is not equal or the graph structure similarity is not 1, the disappearance and appearance of targets during the association process are managed using the following strategies:

[0095] Calculate the similarity matrix between two adjacent frames and solve the similarity matrix according to the top-1 greedy strategy. Design a dummy target in the current frame and match the elements in the similarity matrix with similarity less than the threshold to the dummy. If the matching result is a dummy, it is determined that the target corresponding to the element existed in the previous frame and disappeared in the current frame.

[0096] In the current frame, manually design a meta-target. Match elements in the similarity matrix that have a similarity less than the threshold and have not been matched to dummy elements to the meta-target. If the matching result is a meta-target, determine that the element is a newly appearing target in the current frame and assign it a new identity.

[0097] For a vanished target, its identity is retained within a unit of time and it is not removed to deal with the situation of reappearing after occlusion. However, for frames during the target's vanishing process, a motion model is not used to recover its trajectory.

[0098] To facilitate understanding of the management strategy provided in this embodiment, it is now described in the following example. Figure 2 For example, the management strategy for the disappearance and appearance of targets during the association process is explained as follows:

[0099] 1) Situations where the target disappears, such as Figure 2 The circular target is shown in frame t=3. This target disappears in frame t=15. In this case, this embodiment designs a dummy target and, during the top-1 greedy association process, sets a similarity threshold. Element pairs in the similarity matrix with similarity less than the threshold are not matched. Specifically, in the graph matching process at frame t=15, assuming the frame numbers are a and b, and the corresponding number of targets are n and m, the similarity matrix A has a dimension of n*(m+1), where 1 corresponds to a dummy column. This column can match multiple targets out of the n targets, and the matching result corresponds to the situation where the target disappears in the current frame.

[0100] 2) During graph matching in frames t>15, if the number of targets in two adjacent frames is different or the graph structure similarity is not 1, the targets that have disappeared need to be included in the calculation when calculating the corresponding similarity matrix A. Specifically, assuming the sequence numbers of the two frames are a and b, and the corresponding number of targets are n and m, respectively, and the number of targets that have disappeared before the matching of frame b is k, then the dimension of the similarity matrix A is (n+k)*(m+1).

[0101] 3) The circumstances under which the target appears, such as Figure 2 The pentagonal target at frame t=29 is shown, and this target appears for the first time. During graph matching, since a target that was not present in the previous frame appears, the similarity matrix A needs to be calculated. At this point, this embodiment designs a meta-target, similar to a dummy target. During the matching process, the newly appearing target is associated with the meta-target. Specifically, assuming the frame numbers are a and b, and the corresponding number of targets are n and m, respectively, and the number of targets that have disappeared before the matching of frame b is k, then the dimension of the similarity matrix A is (n+k+1)*(m+1).

[0102] Under the above management strategy, trajectory segments within a unit of time can be obtained frame by frame. It should be noted that for a disappearing target, its identity is retained within the unit of time and it is not removed. If it meets the above management strategy, an attempt is made to match it to deal with the situation of reappearing after occlusion. However, for frames during the disappearance process of the target, the motion model is not used to recover its trajectory.

[0103] By performing lightweight graph matching frame by frame within a unit of time, this embodiment can obtain the trajectory segments of each target within a unit of time. This method can be regarded as a kind of clustering of targets on the time axis, such as... Figure 2 The result is framed by an ellipse.

[0104] The generation of spacetime boundaries. For example... Figure 2As shown in the second row, for trajectory segments within a unit of time, this embodiment designs two types of boundary relationships: first-order boundary and second-order boundary, which are used to characterize the spatiotemporal characteristics of the target itself and the targets to each other, respectively.

[0105] First-order boundaries are primarily used to characterize the spatiotemporal features of the target itself, such as... Figure 2 The solid lines in the second row illustrate how the information of a target individual changes along the time axis. Specifically, assuming the target's identity is i, and the set of frames containing its information per unit time is T, then the first-order boundary of target i can be represented in the following form:

[0106]

[0107] In the formula, L i To describe the variable B, which describes the position of target i from a bird's-eye view. i To describe the variables of target i at the bird's-eye view scale, These correspond to the variables describing the changes in the position and scale of target i over time t.

[0108] Second-order boundaries are mainly used to characterize the spatiotemporal characteristics between targets, such as Figure 2 The dashed lines in the second row primarily describe how the relationships between the target and other targets change along the time axis. Specifically, assuming the target's identity is i, the set of all frames per unit time is... The second-order boundary of objective i can then be represented in the following form:

[0109]

[0110] In the formula, N represents the total number of targets appearing per unit time. This indicates the relative positional relationship between target i and target j. Let be the corresponding variable representing the change in their relative positional relationship with time t. It should be noted that the relative positional relationship here, such as... Figure 2 As shown, this refers to frames where target i and target j appear simultaneously. For frames where they do not appear simultaneously, the relationship is marked as empty, indexed by t. Furthermore, to improve the efficiency of subsequent calculations, this embodiment designs an optimization strategy based on nearest neighbor search. For targets that are relatively far apart, second-order boundaries can be disregarded. Specifically, in this example, a two-stage nearest neighbor search can be used. Centered on the observation subject, targets participating in the generation of second-order boundary relationships are considered at most only the nearest neighbors of the nearest neighbor target of the observation subject.

[0111] Step S153, "Based on the spatiotemporal characteristics of each target in adjacent time units, perform multi-target tracking in adjacent time units to obtain the multi-target tracking results from the current camera's perspective," includes:

[0112] Based on the aforementioned spatiotemporal characteristics, determine the spatiotemporal self-attention encoding and spatiotemporal interactive attention encoding of the target for the current unit of time;

[0113] The spatiotemporal self-attention encoding and spatiotemporal interactive attention encoding of the target in the current unit of time, as well as the spatiotemporal features of the target in the previous unit of time, are input into the pre-trained feature encoder for feature encoding to obtain the encoded features of the target in the previous unit of time.

[0114] The spatiotemporal self-attention encoding, spatiotemporal interactive attention encoding, spatiotemporal features of the current unit time target, and the encoded features of the previous unit time target are input into a pre-trained feature decoder for feature decoding to obtain the correlation matrix between the current unit time target and the previous unit time target.

[0115] Spatiotemporal self-attention coding (SDI) encodes the target's state changes over time through embedded representation, obtaining an encoding consistent with the target's feature dimensions. This encoding is then fused with the target features sequentially, serving as input to the subsequent encoder and decoder. Essentially, it provides an implicit temporal representation of the target's state changes, enhancing its discriminative power. Specifically, assuming the current time is the k-th second, and the target set within the current unit of time is... Where N k Let represent the number of targets within the current unit of time. For the i-th target, its first-order boundary is upgraded and its features are embedded through a parameter-shared multilayer perceptron to obtain its spatiotemporal self-attention encoding:

[0116]

[0117]

[0118] In the formula, t represents a frame within the k-th second, and ω p Represents the learnable parameters. This represents the first-order boundary of target i in frame t, after passing through a multilayer perceptron. The dimensions are consistent with the feature dimensions of the target from a bird's-eye view.

[0119] Furthermore, assuming the target characteristics of the i-th target within the k-th second are: Then, the spatiotemporal self-attention encoding is fused based on the following formula principle:

[0120]

[0121] Fusion feature notation Used as input for subsequent encoders and decoders.

[0122] Spatiotemporal interaction attention encoding encodes the state information of targets changing over time through embedded representation, obtaining an encoding consistent with the target feature dimensions. This encoding then uses spatial attention as the bias term for attention. Essentially, it is an implicit representation of the mutual influence between targets in the time domain. Specifically, assuming the current time is the k-th second, and the target set within the current unit of time is... Where N k Let represent the number of targets within the current unit of time. For the i-th target, its second-order boundary is upgraded and its features are embedded through a parameter-shared multilayer perceptron, and its spatiotemporal interaction attention bias representation is obtained through spatial attention:

[0123] α ij =ω GAT (ω α *e ij j∈N k ,j≠i (8)

[0124] In the formula, j represents the target other than target i within the k-th second, and ω α ω represents the learnable parameters used for dimensionality increase and feature embedding representation. GAT Denotes the subnet parameters based on spatial attention, e ij Let represent the second-order boundary of target i in the k-th second. In this example, the spatial attention-based subnet uses a graph attention model (GAT). Because spatiotemporal interaction attention will serve as the bias term input for subsequent attention, it is sensitive to the input-output dimension. Therefore, a graph attention model that is insensitive to the input dimension is adopted. Assume the original attention is A. ij Then, based on the following formula principle, it fuses spatiotemporal interactive attention:

[0125]

[0126] In the formula The attention, after being corrected by spatiotemporal interaction attention bias, is used as the input for the subsequent encoder and decoder.

[0127] like Figure 3 and Figure 4 As shown, the multi-target tracking method provided in this embodiment uses an attention-based feature encoding and decoding framework to calculate the correlation matrix. In this example, based on the transformer framework, feature encoding and decoding are performed using a query, key, and value approach.

[0128] See Figure 3 The spatiotemporal self-attention encoding and spatiotemporal interactive attention encoding of the target in the current unit time (second k) and the spatiotemporal features of the target in the previous unit time (second k-1) are combined. Figure 3 The target features (displayed as those at the (k-1)th second) are input into a pre-trained feature encoder for feature encoding, yielding the encoded features of the target in the previous unit of time (the (k-1)th second), specifically:

[0129] The spatiotemporal features of the target in the previous unit of time are used as the value input V of the feature encoder. The fusion value of the spatiotemporal features of the target in the previous unit of time and the spatiotemporal self-attention encoding is used as the key K and query Q input of the feature encoder. The spatiotemporal interactive attention encoding is used as the bias α of the spatiotemporal interactive attention and input to the feature encoder for feature encoding to obtain the encoded features of the target in the previous unit of time.

[0130] Calculate the encoded features at the (k-1)th second based on the attention mechanism:

[0131] Attention(Q,K,V)=Softmax(A)V (l0)

[0132]

[0133] In the formula, α represents the spatiotemporal interactive attention bias, which is the encoder used in this embodiment. Its self-attention calculation is corrected by the spatiotemporal interactive attention bias, implicitly representing the relationships between targets within the current unit of time. Specifically, the target feature dimension used for fusion is N. k-1 *ΔT*d, where N k-1 Let ΔT represent the number of targets in the (k-1)th second, ΔT represent the number of frames spanned by the target features used for fusion per unit time, and in this example, ΔT = 10 (generally one-third of the frame rate), indicating that the features of the target in the last 10 frames within a unit time are taken. d is the dimension of the target features within a given frame. Afterwards, through the feature encoder, the output encoded features have a dimension of N. k-1 *ΔT*d.

[0134] See Figure 4 The spatiotemporal self-attention encoding, spatiotemporal interactive attention encoding, and target features of the target in the current unit time (second k) are then used to encode the target. Figure 4 The coded features of the target at the k-th second and the target at the previous unit time (k-1 seconds) are input into a pre-trained feature decoder for feature decoding, resulting in the correlation matrix between the target at the current unit time (k-th second) and the target at the previous unit time (k-1 seconds), specifically:

[0135] The encoded features of the target in the previous unit time are used as the key K and value V input of the feature decoder. The fusion value of the spatiotemporal features of the target in the current unit time and the spatiotemporal self-attention encoding is used as the query Q input of the feature decoder. The spatiotemporal interactive attention encoding is used as the bias α of the spatiotemporal interactive attention and input to the pre-trained feature decoder for feature decoding to obtain the correlation matrix of the target in the current unit time and the target in the previous unit time.

[0136] An attention-based correlation matrix. Specifically, to output an N-dimensional... k-1 *N k The association matrix, in this example, the dimension of query Q should be N. k-1 *N k *d, and the dimensions of the target features used for fusion need to remain consistent. Therefore, before generating query Q, the target features of the k-th second-previous ΔT frame are averaged in the time dimension to make its dimension N. k *1*d. Next, using the temporal self-attention encoding of the first frame at the k-th second, the target features and the temporal self-attention encoding are fused according to formula (7), and then copied to make its dimension N. k-1 *N k *d, obtain the query Q at second k. Then, based on the encoded feature key K and value V, and the spatiotemporal interaction attention bias, the feature decoder outputs a dimension of N. k-1 *N k The correlation matrix.

[0137] It should be noted that although this embodiment uses the classic decoder model, like the encoding process, a spatiotemporal interactive attention bias is added to its attention module. Unlike the encoding process, the decoding process generally includes two modules: self-attention and cross-attention. The bias of the self-attention module is consistent with that of the encoding process, while the cross-attention module needs to generate a second-order margin for the target set spanning two time units according to (4), and generate a spatiotemporal interactive attention bias according to formula (8).

[0138] Furthermore, since the classic decoding model possesses permutation invariance, to avoid its impact on solving the association matrix, this embodiment additionally designs a cross-bias to optimize the features of query Q. Specifically, assuming query q ij To query the correlation between target j within the (k-1)th second and target i within the kth second, the cross-bias of its features is obtained by fusing the average of the i-th target in the ΔT frames before the kth second with the time self-attention encoding of the j-th target in the last frame of the (k-1)th second, according to the principle of formula (7):

[0139]

[0140]

[0141] In the formula, p j f represents the temporal self-attention encoding of the j-th target in the last frame at second k-1. ij Indicates query q ij Features This represents the corresponding feature after cross-bias optimization.

[0142] based on Figure 4 The output correlation matrix can be directly solved using an optimization algorithm to obtain the matching results at second k and second k-1, and the disappearance and appearance of targets are managed. The final output is the tracking result up to second k. Specifically, this embodiment uses an attention-based encoding / decoding model to learn the mapping between two multi-target sets. This mapping models the spatiotemporal correlation of targets, and the correlation matrix is ​​obtained based on the input correlation matrix. The correlation matrix can be solved using the aforementioned top-1 greedy strategy or the Hungarian algorithm. For target management, consistent with the aforementioned strategy for managing target disappearance and appearance, corresponding dummy elements and elements are designed to handle target disappearance and the generation of new trajectories.

[0143] It should be noted that in this embodiment, when processing multi-target association between units of time, there is a temporal overlap between the two units of time. The overlap range is 3 frames, that is, the last 3 frames of the (k-1)th second of the previous unit of time are the same as the first 3 frames of the kth second of the current unit of time. This setting mainly considers the temporal continuity and correlation of target motion. At the same time, the tracking results are represented frame by frame when outputting, and the overlapping frames also play a role in connecting the preceding and following frames. In addition, when extracting target features, for cases where the target disappears in a certain frame within a unit of time, resulting in discontinuity in features, the features are filled in by interpolation of the frames before and after it.

[0144] Model training and testing include: dataset collection and labeling, loading and augmentation, backbone network training, and joint framework training and testing.

[0145] 1) Data collection and annotation. This mainly involves collecting sensor data relevant to the application scenario, and dividing the dataset into training, validation, and test sets according to the scenario and dataset size, with a ratio of approximately 10:1:1.

[0146] Regarding dataset annotation, currently available multi-class, multi-object tracking datasets mainly focus on autonomous driving and drone-related datasets, such as KITTI and VisDrone. These datasets primarily define target categories based on their function within traffic scenarios. Since this approach uses vehicle sensors as an example, it also adopts this format. A specific category can be categorized as: car, truck, bus, pedestrian, bicycle, tricycle, van, motor-bicycle, and motor-tricycle. During annotation, each category corresponds to a label. The basic attributes of each sample in the dataset include its frame number, identity information, category information, and its position and size within the corresponding frame. To better distinguish between targets and non-targets, this approach also labels some static targets with strong physical meaning as the background class, and adds an "ignore" class to differentiate between meaningful and meaningless backgrounds. It should be noted that if the sensor uses a camera array, the target sample's attributes also need to be labeled with its camera number. Furthermore, this embodiment can also perform reinforcement training for target occlusion, which requires labeling the degree of occlusion for each sample during the labeling process. However, since this labeling is costly and this label-based training is not innovative in this solution, it will not be elaborated upon in this embodiment.

[0147] 2) Dataset Loading and Augmentation. Due to the difficulty in obtaining datasets and the large volume of data required, data augmentation is generally necessary before loading. The dataset labeled in this embodiment is a continuous video sequence. Since the model is based on temporal cross-sampling, loading frames sequentially for training would hinder the robustness of the network. Therefore, in actual training, video frames within a unit of time are randomly downsampled to create artificial occlusion; that is, consecutively input frames are not adjacent frames at fixed intervals, but the temporal order is preserved. Simultaneously, at the sample level, a portion of targets are randomly removed, and negative samples from other frames are added, thereby increasing the perturbation of the original data. Furthermore, other methods beneficial for data augmentation can be added to this scheme.

[0148] 3) Training the backbone network. In training the backbone network, the weights of the original pre-trained network are first loaded, and then the data-augmented dataset is input into the network. According to the original scheme, the backbone network is fine-tuned on the labeled dataset to adapt it to the recognition of multiple targets and multiple classes.

[0149] 4) Training and Testing of the Joint Framework. This embodiment employs an end-to-end training approach, using the association matrix corresponding to the real samples to supervise the model training. Specifically, the prediction of the association matrix is ​​formulated as a binary classification problem, and cross-entropy loss is used to optimize the network:

[0150]

[0151] in the formula and A ij These are the (i, j)th elements in the correlation matrix, where the former corresponds to the ground truth correlation matrix and the latter to the predicted correlation matrix output by the network. During network optimization, the accumulated loss value from a complete iteration or batch is backpropagated, and the network parameters are updated using gradient descent until the convergence condition is met or the preset number of iterations is reached.

[0152] After model training is complete, testing is performed. First, the trained weights are loaded, and then multi-class, multi-object tracking is conducted on the test set. The tracking results per second are written to a text document, where each data point contains the position and identity information of a specific object within a given frame. Based on this, the tracking algorithm can be objectively evaluated using evaluation metrics in multi-object tracking.

[0153] It is understood that the technical solution provided in this embodiment addresses the occlusion problem and spatiotemporal feature modeling problem in existing video multi-target tracking. By collecting image information of the current vehicle's environment and the vehicle's motion information, projecting it onto a bird's-eye view, and performing lightweight graph matching frame by frame for targets within a unit of time (second by second), trajectory segments and spatiotemporal features are obtained. For targets in adjacent units of time, online multi-target tracking is performed second by second based on an end-to-end encoding and decoding model using their respective spatiotemporal features. This minimizes the target occlusion and unreliable appearance feature problems caused by the viewpoint, thereby significantly reducing the impact of viewpoint problems on multi-target tracking performance and improving tracking speed and accuracy.

[0154] This embodiment proposes a lightweight graph matching method based on distance sorting, which enables the model to efficiently and accurately associate targets frame by frame within a unit of time to obtain the corresponding trajectory segments, and further obtain the first-order and second-order spatiotemporal margins within a unit of time, which are used to extract the spatiotemporal relationship between the target itself and the targets.

[0155] For second-by-second online multi-target tracking within adjacent time units, an end-to-end multi-target tracking framework based on encoding and decoding is proposed. For the spatiotemporal feature representation of the target itself, a spatiotemporal self-attention encoding is proposed, enabling the model to embed the target's own state information changing over time, and fusing it with the target's appearance features during the encoding and decoding process. For the spatiotemporal feature representation of the targets to each other, a spatiotemporal interactive attention encoding is proposed, enabling the model to embed the state information of the targets to each other changing over time, and further using spatial attention as the bias term input for attention during the encoding and decoding process. Taking advantage of the permutation invariance of the correlation matrix of the decoding model, a cross-bias is proposed to optimize the corresponding features of the correlation matrix, further improving multi-target tracking performance, tracking speed, and accuracy.

[0156] In addition, since the multi-target tracking model proposed in this embodiment is not limited to a certain target category or group, it has better generalization and robustness in practical problems.

[0157] Example 2

[0158] Figure 5 This is a schematic block diagram of a multi-target tracking device 100 from a bird's-eye view, according to an exemplary embodiment. Figure 5 As shown, the device 100 includes:

[0159] The acquisition unit 101 is used to acquire image information of the current vehicle's environment and the current vehicle's motion information within a current unit of time; the image information includes at least multiple video frames acquired within a unit of time from a camera's perspective.

[0160] Extraction unit 102 is used to extract multi-scale feature maps of each video frame through a pre-trained feature extraction model;

[0161] The projection unit 103 is used to project multi-scale feature maps at the same moment from different camera perspectives onto the feature space at the bird's-eye view based on the motion information, so as to obtain the feature map at the bird's-eye view at that moment.

[0162] Detection unit 104 is used to perform target detection on the feature map to obtain the target from the bird's-eye view at that moment;

[0163] Execution unit 105 is used to perform the following steps on multiple video frames acquired per unit time from each camera viewpoint:

[0164] Based on the lightweight graph matching method, the trajectory segments of each target per unit time are obtained;

[0165] Based on the trajectory segments and the relative positions of each target, spatiotemporal characteristics describing each target itself and the relationships between targets within a unit of time are obtained.

[0166] Based on the spatiotemporal characteristics of each target in adjacent time units, multi-target tracking is performed in adjacent time units to obtain the multi-target tracking results from the current camera perspective.

[0167] It should be noted that the application scenarios applicable to the technical solution provided in this embodiment include, but are not limited to, autonomous driving and assisted driving of vehicles. In actual use, the technical solution provided in this embodiment can be loaded into the current vehicle's central control system or into electronic devices; the electronic devices include, but are not limited to, on-board computers and external computer devices.

[0168] It should be noted that since the implementation methods of each module in this embodiment can be found in the relevant description in Embodiment 1, this embodiment will not repeat them.

[0169] It is understood that the technical solution provided in this embodiment addresses the occlusion problem and spatiotemporal feature modeling problem in existing video multi-target tracking. By collecting image information of the current vehicle's environment and the vehicle's motion information, projecting it onto a bird's-eye view, and performing lightweight graph matching frame by frame for targets within a unit of time (second by second), trajectory segments and spatiotemporal features are obtained. For targets in adjacent units of time, online multi-target tracking is performed second by second based on an end-to-end encoding and decoding model using their respective spatiotemporal features. This minimizes the target occlusion and unreliable appearance feature problems caused by the viewpoint, thereby significantly reducing the impact of viewpoint problems on multi-target tracking performance and improving tracking speed and accuracy.

[0170] Example 3

[0171] An electronic device according to an exemplary embodiment includes:

[0172] A communication module, a processor, and a memory, wherein the memory stores program instructions;

[0173] The processor is used to execute program instructions stored in the memory to perform the method as described in Embodiment 1; and / or, to perform the method as described in Embodiment 2; and / or, to perform the method as described in Embodiment 3.

[0174] It should be noted that the electronic devices include, but are not limited to, in-vehicle computers and external computer devices. Communication modules include, but are not limited to, wired communication modules and wireless communication modules, such as WCDMA, GSM, CDMA and / or LTE communication modules, ZigBee modules, Bluetooth modules, Wi-Fi modules, etc.

[0175] Processors include, but are not limited to: CPU, microcontroller, PLC controller, FPGA controller, etc.

[0176] The memory may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory; it may also include other removable / non-removable, volatile / non-volatile computer system storage media. The memory may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0177] It is understood that the technical solution provided in this embodiment addresses the occlusion problem and spatiotemporal feature modeling problem in existing video multi-target tracking. By collecting image information of the current vehicle's environment and the vehicle's motion information, projecting it onto a bird's-eye view, and performing lightweight graph matching frame by frame for targets within a unit of time (second by second), trajectory segments and spatiotemporal features are obtained. For targets in adjacent units of time, online multi-target tracking is performed second by second based on an end-to-end encoding and decoding model using their respective spatiotemporal features. This minimizes the target occlusion and unreliable appearance feature problems caused by the viewpoint, thereby significantly reducing the impact of viewpoint problems on multi-target tracking performance and improving tracking speed and accuracy.

[0178] Example 4

[0179] A computer-readable storage medium is shown according to an exemplary embodiment, on which an erasable and rewritable computer program is stored;

[0180] When the computer program is run on a computer device, the computer device performs the method as described in Embodiment 1; and / or performs the method as described in Embodiment 2; and / or performs the method as described in Embodiment 3.

[0181] The computer-readable storage media disclosed in this embodiment include, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0182] It is understood that the technical solution provided in this embodiment addresses the occlusion problem and spatiotemporal feature modeling problem in existing video multi-target tracking. By collecting image information of the current vehicle's environment and the vehicle's motion information, projecting it onto a bird's-eye view, and performing lightweight graph matching frame by frame for targets within a unit of time (second by second), trajectory segments and spatiotemporal features are obtained. For targets in adjacent units of time, online multi-target tracking is performed second by second based on an end-to-end encoding and decoding model using their respective spatiotemporal features. This minimizes the target occlusion and unreliable appearance feature problems caused by the viewpoint, thereby significantly reducing the impact of viewpoint problems on multi-target tracking performance and improving tracking speed and accuracy.

[0183] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0184] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0185] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0186] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0187] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0188] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0189] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0190] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0191] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A multi-target tracking method in an aerial view perspective, characterized in that, The method comprises the following steps: acquiring image information of an environment where a current vehicle is located and motion information of the current vehicle in a current unit time, the image information comprising at least a plurality of video frames collected in the unit time from a camera perspective; extracting multi-scale feature maps of each video frame through a pre-trained feature extraction model; projecting the multi-scale feature maps of the same moment from different camera perspectives into a feature space in a bird's-eye perspective according to the motion information, to obtain a feature map in the bird's-eye perspective at the moment; performing target detection on the feature map to obtain targets in the bird's-eye perspective at the moment; for the plurality of video frames collected in the unit time from each camera perspective, the following steps are performed: obtaining trajectory segments of each target in the unit time according to a lightweight graph matching method; obtaining spatiotemporal features of each target itself and mutual targets in the unit time according to the trajectory segments and relative positional relationships among the targets; performing multi-target tracking on adjacent unit times according to the spatiotemporal features of the targets in the adjacent unit times, to obtain multi-target tracking results in the current bird's-eye perspective.

2. The method of claim 1, wherein, The method of obtaining the trajectory segments of each target in the unit time according to the lightweight graph matching method comprises the following steps: starting from a first frame, assigning an identity to each target in each frame and assigning a sorting sequence number to each target; obtaining a graph representation corresponding to each frame according to the identity and the sorting sequence number, the graph representation being used to describe positional relationships among the targets and to determine similarity of graph structures between two frames; starting from a second frame, associating targets with the same identity in each frame to obtain the trajectory segments of each target in the unit time according to the graph representation.

3. The method of claim 2, wherein, The method of assigning an identity to each target in each frame and assigning a sorting sequence number to each target starting from the first frame comprises the following steps: for the first frame, assigning an identity to each target in the order of X-axis priority and coordinate sequence from the top left corner; for other frames, assigning an identity to each target according to matching results with all previous frames; for each frame, calculating a relative distance of each target from the current vehicle and assigning a sorting sequence number to each target in ascending order of the relative distance.

4. The method of claim 2, wherein, The method of associating targets with the same identity in each frame starting from the second frame according to the graph representation comprises the following steps: if the number of targets in adjacent two frames is equal and the similarity of graph structures is 1, associating targets with the same identity in the adjacent two frames; if the number of targets in adjacent two frames is not equal or the similarity of graph structures is not 1, using a fast graph matching method based on top-1 greediness to associate nodes with the greatest similarity between two graph representations.

5. The method of claim 4, wherein, The method of associating nodes with the greatest similarity between two graph representations using the fast graph matching method based on top-1 greediness comprises the following steps: for targets belonging to different frames in consecutive two frames, calculating similarities in sequence to obtain a similarity matrix; solving the similarity matrix according to a top-1 greediness strategy to obtain an association matrix, the association matrix representing matching relationships between targets in adjacent two frames.

6. The method of claim 5, wherein, If the number of targets in adjacent two frames is not equal or the similarity of graph structures is not 1, the following management strategy is used to manage disappearance and appearance of targets in the association process. The similarity matrix of two adjacent frames before and after calculation is obtained, and the similarity matrix is solved according to a top-1 greedy strategy; a dummy target is designed in the current frame, and elements with similarity less than a threshold in the similarity matrix are matched to the dummy; if the matching result is the dummy, it is determined that the target corresponding to the element exists in the previous frame and disappears in the current frame; A meta target is manually designed in the current frame, and elements with similarity less than the threshold and not matched to the dummy in the similarity matrix are matched to the meta target; if the matching result is the meta target, it is determined that the element is a new target appearing in the current frame, and a new identity is assigned to the element; For the disappeared target, its identity is retained in the unit time, and is not removed to cope with the situation of appearing again after occlusion, but the motion model is not used to restore the trajectory of the target in the frame during the disappearance process.

7. The method of claim 1, wherein, The method comprises the following steps: According to the spatio-temporal characteristics of the targets in the adjacent unit time, multi-target tracking is performed on the adjacent unit time, which comprises the following steps: According to the spatio-temporal characteristics, the spatio-temporal self-attention encoding and the spatio-temporal interaction attention encoding of the target in the current unit time are determined; The spatio-temporal self-attention encoding and the spatio-temporal interaction attention encoding of the target in the current unit time, the spatio-temporal characteristics of the target in the current unit time, and the encoding characteristics of the target in the previous unit time are input into a pre-trained feature encoder for feature encoding to obtain the encoding characteristics of the target in the previous unit time; 8. The method of claim 7, wherein, The spatio-temporal self-attention encoding and the spatio-temporal interaction attention encoding of the target in the current unit time, the spatio-temporal characteristics of the target in the current unit time, and the encoding characteristics of the target in the previous unit time are input into a pre-trained feature decoder for feature decoding to obtain the correlation matrix of the target in the current unit time and the target in the previous unit time; The correlation matrix is optimized and solved to obtain the target matching result of the current unit time and the previous unit time. The method comprises the following steps:

9. The method of claim 7, wherein, The spatio-temporal self-attention encoding of the target is determined by embedding the spatio-temporal characteristics of the target changing over time to obtain an encoding consistent with the target feature dimension; The spatio-temporal interaction attention encoding of the target is determined by embedding the spatio-temporal characteristics between targets changing over time to obtain an encoding consistent with the target feature dimension, and further inputting the spatial attention as a bias term of the attention. The method comprises the following steps: The spatio-temporal characteristics of the target in the previous unit time are input as the value of the feature encoder, the fusion of the spatio-temporal characteristics of the target in the previous unit time and the spatio-temporal self-attention encoding are input as the key and the query of the feature encoder, and the spatio-temporal interaction attention encoding is input as the bias of the spatio-temporal interaction attention, and then the feature encoder is used for feature encoding to obtain the encoding characteristics of the target in the previous unit time; The method comprises the following steps: The encoding features of the previous unit time target are input as keys and values of the feature decoder, the spatio-temporal features of the current unit time target and the fusion value of the spatio-temporal self-attention encoding are input as queries of the feature decoder, and the spatio-temporal interaction attention encoding is input as a bias of the spatio-temporal interaction attention to the pre-trained feature decoder for feature decoding to obtain the association matrix of the current unit time target and the previous unit time target.

10. A multi-target tracking apparatus in an aerial view perspective, characterized by, Comprise: An acquisition unit is configured to acquire image information of an environment in which a current vehicle is located and motion information of the current vehicle in a current unit time; The image information comprises at least a plurality of video frames collected in a unit time from a camera perspective; An extraction unit is configured to extract a multi-scale feature map of each video frame by using a pre-trained feature extraction model; A projection unit is configured to project the multi-scale feature maps of the same moment from different camera perspectives into a feature space in a bird's eye perspective according to the motion information to obtain a feature map in the bird's eye perspective at the moment; A detection unit is configured to perform target detection on the feature map to obtain a target in the bird's eye perspective at the moment; An execution unit is configured to perform the following steps on the plurality of video frames collected in a unit time from each camera perspective: Obtain trajectory segments of each target in a unit time according to a lightweight graph matching method; Obtain spatio-temporal features of each target itself and each target relative to each other in a unit time according to the trajectory segments and the relative position relationship between the targets; Perform multi-target tracking on adjacent unit times according to the spatio-temporal features of the targets in the adjacent unit times to obtain a multi-target tracking result in the current camera perspective.

11. An electronic device, comprising: Comprise: A communication module, a processor and a memory, wherein the memory stores program instructions; The processor is configured to execute the program instructions stored in the memory to execute the method of any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, A computer program is stored thereon; When the computer program runs on the computer device, the computer device executes the method of any one of claims 1-9.

Images