Data fusion method and device, tracking method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data fusion method and device, a tracking method and device, electronic equipment and a storage medium, relates to the technical field of computer vision, and can effectively improve the accuracy of a fusion feature formed by a fisheye image and four-dimensional millimeter wave radar point cloud data so as to improve the accuracy and reliability of target tracking. The data fusion method comprises the steps that a bird's-eye view BEV feature pair is acquired, the BEV feature pair comprises a first BEV feature determined based on four-dimensional millimeter wave radar point cloud data and a second BEV feature determined based on a fisheye image, and the first BEV feature and the second BEV feature are located in the same coordinate system and are in time alignment; generating a first space attention weight map corresponding to the first BEV feature, and generating a second space attention weight map corresponding to the second BEV feature; and based on the first space attention weight map and the second space attention weight map, fusing the first BEV feature and the second BEV feature to obtain a fused BEV feature.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a data fusion method, tracking method, device, electronic device and storage medium. Background Technology

[0002] In assisted driving scenarios, multi-sensor fusion solutions can be used when vehicles track surrounding targets. For example, fisheye cameras offer advantages in panoramic monitoring and wide coverage, while four-dimensional millimeter-wave radar possesses characteristics such as four-dimensional perception, strong dynamic target tracking capabilities, and strong penetration. Fusion of these two technologies can effectively complement each other, improving the accuracy and reliability of target tracking. However, how to effectively fuse heterogeneous fisheye images and four-dimensional millimeter-wave radar point cloud data is a technical problem that urgently needs to be solved. Summary of the Invention

[0003] This application provides a data fusion method, tracking method, apparatus, electronic device, and storage medium that can effectively solve the heterogeneous fusion problem between fisheye images and four-dimensional millimeter-wave radar point cloud data, thereby improving the accuracy of fusion features and thus enhancing the accuracy and reliability of target tracking.

[0004] In a first aspect, embodiments of this application provide a data fusion method, the method comprising: Obtain BEV feature pairs from bird's-eye view. Each BEV feature pair includes a first BEV feature determined based on four-dimensional millimeter-wave radar point cloud data and a second BEV feature determined based on fisheye image. The first BEV feature and the second BEV feature are located in the same coordinate system and are time-aligned. Generate a first spatial attention weight map corresponding to the first BEV feature, and generate a second spatial attention weight map corresponding to the second BEV feature; Based on the first spatial attention weight map and the second spatial attention weight map, the first BEV feature and the second BEV feature are fused to obtain the fused BEV feature.

[0005] In this embodiment, firstly, the first BEV feature corresponding to the four-dimensional millimeter-wave radar point cloud data and the second BEV feature corresponding to the fisheye image are obtained. Then, a first spatial attention weight map corresponding to the first BEV feature and a second spatial attention weight map corresponding to the second BEV feature are generated. The first spatial attention weight map indicates which positions in the first BEV feature need to be emphasized and which can be relatively ignored; similarly, the second spatial attention weight map indicates which positions in the second BEV feature need to be emphasized and which can be relatively ignored. Based on this, the first and second spatial attention weight maps are used to guide the fusion of the time-aligned first and second BEV features, making the fused BEV features more accurate.

[0006] Optionally, based on the first spatial attention weight map and the second spatial attention weight map, the first BEV feature and the second BEV feature are fused to obtain fused BEV features, including: The fused BEV feature is obtained by fusing the product of the first BEV feature and the second spatial attention weight map, and the product of the second BEV feature and the first spatial attention weight map.

[0007] In this embodiment, calculating the product of the first BEV feature and the second spatial attention weight map is equivalent to re-examining, from the perspective of the second BEV feature, which positions in the first BEV feature need to be focused on and which positions can be relatively ignored. Similarly, calculating the product of the second BEV feature and the first spatial attention weight map is equivalent to re-examining, from the perspective of the first BEV feature, which positions in the second BEV feature need to be focused on and which positions can be relatively ignored. Therefore, the resulting fused BEV feature can contain richer information and has stronger out-of-distribution adaptability and robustness.

[0008] Optionally, based on the first spatial attention weight map and the second spatial attention weight map, the first BEV feature and the second BEV feature are fused to obtain fused BEV features, including: The fused BEV feature is obtained by fusing the product of the first BEV feature and the first spatial attention weight map, and the product of the second BEV feature and the second spatial attention weight map.

[0009] In this embodiment, calculating the product of the first BEV feature and the first spatial attention weight map ensures the most accurate identification and enhancement of important regions in the first BEV feature without introducing semantic bias due to the use of external weights. Similarly, calculating the product of the second BEV feature and the second spatial attention weight map ensures the most accurate identification and enhancement of important regions in the second BEV feature without introducing semantic bias due to the use of external weights. Thus, the fused features can maintain the independent semantic structure and important information of each original feature to the greatest extent, forming a "superimposed essence" feature representation.

[0010] Optionally, generating a first spatial attention weight map corresponding to the first BEV feature includes: Perform various pooling operations on the first BEV feature respectively; The pooling features of the various pooling operations are fused to obtain the first spatial attention weight map.

[0011] In this embodiment, different pooling operations can be considered as extracting different types of features from the first BEV features from different angles, so that the fused first spatial attention weight map can more comprehensively and accurately reflect which positions in the first BEV features need to be focused on and which positions can be relatively ignored.

[0012] Optionally, the multiple pooling operations include at least max pooling and average pooling.

[0013] In this embodiment, max pooling is used to capture the most significant and discriminative local features in the first BEV feature. It focuses on the location point with the largest response in a certain region. Average pooling is used to capture the overall contextual information and statistical features of the first BEV feature. It calculates the average response of a region, reflecting the overall situation of the region. By performing max pooling and average pooling operations on the first BEV feature respectively, the fused first spatial attention weight map can more comprehensively and accurately reflect which positions in the first BEV feature need to be focused on and which positions can be relatively ignored.

[0014] Optionally, the four-dimensional millimeter-wave radar point cloud data is obtained after preprocessing with a diffusion model.

[0015] In this embodiment of the application, the four-dimensional millimeter-wave radar point cloud data used to determine the first BEV feature is preprocessed by a diffusion model, which can effectively improve the data quality of the four-dimensional millimeter-wave radar point cloud data, thereby improving the quality of the generated first BEV feature.

[0016] Secondly, embodiments of this application provide a target tracking method, the method comprising: Obtain the current fused BEV features, wherein the current fused BEV features are obtained through the data fusion method described in any one of the first aspects; Based on the current fused BEV features, the current detection result is determined; Calculate the similarity score between the current detection result and the historical detection results to form an affinity matrix. The historical detection results are the detection results corresponding to the historical fused BEV features. The affinity matrix is ​​input into the Sinkhorn algorithm to obtain a matching probability matrix, which is used to indicate the probability that the current detection result matches the historical detection result; Target tracking is performed based on the matching probability matrix.

[0017] In this embodiment, the current fused BEV features can be obtained using the data fusion method described in the first aspect above. Similarly, the historical fused BEV features can also be obtained using the data fusion method described in the first aspect above. Then, the current detection result corresponding to the current fused BEV features can be determined. Similarly, historical fused BEV features also have corresponding historical detection results. Based on this, the similarity score between the current detection result and the historical detection result can be calculated to form an affinity matrix. Using the affinity matrix as input to the Sinkhorn algorithm, the probability of matching between the current detection result and the historical detection result can be quickly obtained, thereby enabling rapid target tracking. Furthermore, due to the differentiability of the Sinkhorn algorithm's calculation process, end-to-end joint optimization of the detection and tracking tasks is achieved. It allows the network to not only learn how to detect objects but also how to generate object representations that are conducive to stable tracking.

[0018] Thirdly, embodiments of this application provide a data fusion apparatus, the apparatus comprising: The acquisition unit is used to acquire a pair of BEV features from a bird's-eye view. The pair of BEV features includes a first BEV feature determined based on four-dimensional millimeter-wave radar point cloud data and a second BEV feature determined based on a fisheye image. The first BEV feature and the second BEV feature are located in the same coordinate system and are time-aligned. The attention weight map generation unit is used to generate a first spatial attention weight map corresponding to the first BEV feature and to generate a second spatial attention weight map corresponding to the second BEV feature. The fusion unit is used to fuse the first BEV feature and the second BEV feature based on the first spatial attention weight map and the second spatial attention weight map to obtain fused BEV features.

[0019] Optionally, the fusion unit is specifically used for: The fused BEV feature is obtained by fusing the product of the first BEV feature and the second spatial attention weight map, and the product of the second BEV feature and the first spatial attention weight map.

[0020] Optionally, the fusion unit is specifically used for: The fused BEV feature is obtained by fusing the product of the first BEV feature and the first spatial attention weight map, and the product of the second BEV feature and the second spatial attention weight map.

[0021] Optionally, the attention weight map generation unit is specifically used for: Perform various pooling operations on the first BEV feature respectively; The pooling features of the various pooling operations are fused to obtain the first spatial attention weight map.

[0022] Optionally, the multiple pooling operations include at least max pooling and average pooling.

[0023] Optionally, the four-dimensional millimeter-wave radar point cloud data is obtained after preprocessing with a diffusion model.

[0024] Fourthly, embodiments of this application provide a target tracking device, the device comprising: An acquisition unit is configured to acquire current fused BEV features, wherein the current fused BEV features are obtained through the data fusion method described in any one of the first aspects; The detection unit is used to determine the current detection result based on the currently fused BEV features; The calculation unit is used to calculate the similarity score between the current detection result and the historical detection results to form an affinity matrix, wherein the historical detection results are the detection results corresponding to the historical fused BEV features; An optimization unit is used to input the affinity matrix into the Sinkhorn algorithm to obtain a matching probability matrix, wherein the matching probability matrix is ​​used to indicate the probability that the current detection result matches the historical detection result; A tracking unit is used to track a target based on the matching probability matrix.

[0025] Fifthly, embodiments of this application provide an electronic device including a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to perform steps of the method as described in any of the embodiments of the first or second aspect.

[0026] In a sixth aspect, embodiments of this application provide a computer-readable storage medium for storing computer instructions that, when the computer is run, cause the computer to perform steps of the method as described in any of the embodiments of the first or second aspect.

[0027] It should be understood that aspects three to five of the embodiments of this application are consistent with the technical solutions of the first or second aspects of the embodiments of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A flowchart illustrating a data fusion method provided in an embodiment of this application; Figure 2 A flowchart illustrating a method for generating a first spatial attention weight map, provided in an embodiment of this application; Figure 3 A flowchart illustrating the first method for generating a fused BEV as provided in this application embodiment; Figure 4 A flowchart illustrating the second method for generating a fused BEV provided in this application embodiment; Figure 5 A flowchart illustrating a target tracking method provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a data fusion device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of a target tracking device provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] To better understand the technical solutions in this specification, the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0031] It should be understood that the described embodiments are merely some, not all, of the embodiments in this specification. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without inventive effort are within the scope of protection of this specification.

[0032] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0033] The following is an explanation of the technical terms used in this application.

[0034] Four-dimensional millimeter-wave radar is an upgraded sensor based on traditional millimeter-wave radar (detecting three-dimensional parameters of distance, azimuth, and velocity). The core addition is the ability to perceive the height dimension, enabling the detection of four-dimensional parameters: distance, azimuth, velocity, and altitude.

[0035] Fisheye camera: An optical imaging device equipped with a fisheye lens (a short focal length, ultra-wide-angle lens with a field of view of ≥180°, and some up to 220°). Essentially, it restores the visual information of the environment around the vehicle through "optical imaging + image algorithms" and is the core component of "360° panoramic imaging".

[0036] Bird's Eye View (BEV): This is a core environmental perception and modeling technology. Essentially, it uses algorithms to fuse and transform information collected by multiple sensors to construct a 360° global bird's-eye view three-dimensional environment model centered on the vehicle itself. This simulates the perspective of "looking down on the vehicle and its surroundings from the air," providing a unified and intuitive environmental input for the decision-making and planning of the driver assistance system.

[0037] A query can be understood as a vector used by the model to "ask questions." Each query aims to focus on and represent potential target objects (such as a car or a pedestrian) in the scene. Queries are dynamic, learnable, and are ultimately decoded into specific bounding boxes or tracking trajectories.

[0038] In assisted driving scenarios, multi-sensor fusion solutions can be used when vehicles track surrounding targets. For example, in parking scenarios, fisheye cameras offer advantages such as panoramic monitoring and wide coverage, while four-dimensional millimeter-wave radar possesses characteristics such as four-dimensional perception, strong dynamic target tracking capabilities, and strong penetration. Fusion of these two technologies can effectively complement each other, improving the accuracy and reliability of target tracking in parking scenarios. However, how to effectively fuse heterogeneous fisheye images and four-dimensional millimeter-wave radar point cloud data is a technical problem that urgently needs to be solved.

[0039] In view of this, embodiments of this application provide a data fusion method, tracking method, apparatus, electronic device, and storage medium, which can improve the accuracy of the fusion features formed by fisheye images and four-dimensional millimeter-wave radar point cloud data, thereby improving the accuracy and reliability of target tracking.

[0040] The technical solution protected by this application will now be described in detail with reference to the accompanying drawings.

[0041] Please see Figure 1This is a flowchart illustrating a data fusion method provided in an embodiment of this application. The method is applied to electronic devices, such as vehicles, including but not limited to pure electric vehicles, plug-in hybrid vehicles, and range-extended vehicles. Of course, the electronic device can also be other devices, such as servers; this application does not impose any particular limitations on this. The flowchart of the method is described as follows: Step 101: Obtain the BEV feature pairs from the bird's-eye view.

[0042] In this embodiment, taking a vehicle as an example, the electronic device is equipped with a four-dimensional millimeter-wave radar and a fisheye camera. The number of each can be determined according to actual needs, and this application does not impose any special restrictions on this. The four-dimensional millimeter-wave radar can acquire point cloud data in real time, and the fisheye camera can acquire fisheye images in real time.

[0043] The BEV feature pair includes a first BEV feature determined based on acquired four-dimensional millimeter-wave radar point cloud data, and a second BEV feature determined based on acquired fisheye images. It should be understood that the first and second BEV features are temporally aligned and reside in the same coordinate system.

[0044] The following section provides a detailed explanation of how to generate the first BEV feature.

[0045] Phase 1: Acquiring 4D millimeter-wave radar point cloud data.

[0046] 1. Signal transmission and reception: The radar antenna array transmits high-frequency electromagnetic wave signals, and the receiving antenna captures the signals scattered back after encountering the target object.

[0047] 2. Signal Processing: Mixing and filtering: The received signal is mixed with the transmitted local oscillator signal to generate an intermediate frequency signal, which is then filtered by bandpass to remove noise.

[0048] Information computing: Distance: Calculated based on the round-trip time delay of the signal.

[0049] Radial velocity: calculated based on the Doppler frequency shift of the signal.

[0050] Angle: The azimuth and elevation angles are calculated using beamforming technology of the antenna array.

[0051] 3. Point cloud generation: Integrate the distance, velocity, and angle information of each target to generate a 4D point cloud containing three-dimensional spatial coordinates (X,Y,Z) and velocity dimension (Doppler velocity).

[0052] Phase 2: Point cloud data preprocessing.

[0053] Objective: To process raw point clouds into high-quality, coherent point cloud data.

[0054] 1. Point cloud segmentation: Input: 4D point cloud data (which already includes velocity information for each point).

[0055] Method: Dynamic and static discrimination method based on single-frame Doppler velocity.

[0056] Operation: Calculate the absolute value of the velocity at each point and compare it with a set threshold (e.g., 0.2 m / s). Points with velocities less than the threshold are marked as static objects, and points with velocities greater than or equal to the threshold are marked as dynamic objects.

[0057] 2. Motion estimation and compensation: Motion estimation for dynamic objects: For objects labeled as dynamic, algorithms such as Kalman filtering are used to estimate their precise motion state (such as velocity vector and pose change).

[0058] Vehicle motion compensation: All point clouds (dynamic and static) are uniformly transformed to the vehicle coordinate system at the current time (frame T) to eliminate the influence of vehicle motion. Specifically, for static objects: rigid body transformation is directly applied using the vehicle's pose changes (rotation matrix R and translation vector t) to stabilize its position. For dynamic objects: vehicle motion compensation is performed first, followed by inverse translation based on the motion estimation to counteract the motion, thus presenting a consistent spatial position across multiple point clouds.

[0059] 3. Framing: Integrate historical 4D millimeter-wave radar point cloud data with current 4D millimeter-wave radar point cloud data in chronological or spatial order to form denser and more continuous "framing point cloud data".

[0060] 4. Noise addition and noise reduction: Input: Point cloud data after framing.

[0061] Method: A diffusion model was used.

[0062] Process: Noise is added to the point cloud data during the diffusion process; during the reverse diffusion process, the model learns to identify and remove outliers and noise points, retaining valid points that reflect the true shape and position of objects, thereby significantly improving the quality of the point cloud.

[0063] Phase 3: BEV Feature Generation Objective: To convert high-quality 3D point clouds into 2D bird's-eye view features.

[0064] 1. Voxelization: Dividing the three-dimensional space into regular grids (voxels) and assigning point clouds to the corresponding voxels, thereby reducing the computational complexity of the data.

[0065] 2. Local feature extraction: Within each voxel, the PointNet network is used to aggregate the features of the internal point cloud and extract the local geometric and semantic features of each voxel.

[0066] 3. Feature mapping to the BEV plane: A mapping matrix is ​​constructed based on the geometric correspondence between the center coordinates of the voxelized mesh and the pixels of the BEV image.

[0067] The feature values ​​of each voxel are mapped to the corresponding positions in the BEV plane, ultimately generating a first BEV feature map.

[0068] Final output: The first BEV feature corresponding to the 4D millimeter-wave radar point cloud data. This first BEV feature structurally contains information such as the position, shape, and velocity of target objects in the scene, providing a strong foundation for subsequent scene understanding tasks such as object detection and tracking.

[0069] The following section details how to generate the second BEV feature.

[0070] Phase 1: Fisheye Image Acquisition and Surround View Stitching 1. Camera Installation and Image Acquisition: Fisheye cameras are precisely installed in the front, rear, left, and right directions of the vehicle to ensure full 360-degree coverage. Each camera acquires fisheye images within its field of view in real time.

[0071] 2. Image preprocessing: The acquired fisheye images are preliminarily processed, including noise reduction and contrast enhancement, to improve image quality.

[0072] 3. Distortion Correction: Based on the fisheye lens distortion model, a correction algorithm is used to process the fisheye image to eliminate distortion.

[0073] 4. Image registration and stitching: Image registration technology is used to determine the overlapping areas and transformation relationships of adjacent images, feature point matching is performed to establish the correspondence between images, and finally the matched images are stitched together to form a complete panoramic fisheye image.

[0074] Phase Two: Image Feature Extraction and Generation of Multi-Scale Feature Maps 1. Feature Extraction Backbone Network: The distortion-reduced panoramic fisheye image is input into an efficient convolutional neural network (such as ResNet), which uses its convolutional layers to automatically extract local and global features of the image.

[0075] 2. Multi-scale feature generation: Feature maps of different resolutions are generated progressively through downsampling operations of the network. Simultaneously, a Feature Pyramid Network (FPN) structure is introduced, which integrates spatial detail information from high-resolution feature maps with semantic information from low-resolution feature maps via lateral or skip connections, enhancing the expressive power of multi-scale features.

[0076] 3. Output: The network outputs a series of feature maps at different scales to prepare for subsequent transformations.

[0077] Phase 3: Spatial feature transformation to obtain the second BEV feature 1. Spatial alignment preprocessing: Spatial alignment processing is performed on multi-scale feature maps. Geometric differences between feature maps of different scales are corrected by coordinate transformation or perspective projection to ensure the consistency of feature maps in spatial dimensions.

[0078] 2. Viewpoint Mapping: A Spatial Transformer Network is used to construct an accurate projection matrix by combining camera intrinsic and extrinsic parameters, which maps the pixels in the multi-scale feature map from the original image viewpoint to the corresponding positions under the BEV viewpoint.

[0079] 3. Feature Fusion: The projection results of feature maps at different scales under the BEV perspective are integrated through feature fusion mechanisms (such as weighted averaging or attention mechanisms) to preserve multi-level semantic information and spatial details.

[0080] 4. Output: Finally, a second BEV feature with uniform spatial resolution and semantic consistency is generated.

[0081] It should be noted that the electronic device is equipped with a cache pool, which stores the first BEV features corresponding to historical 4D millimeter-wave radar point cloud data and the second BEV features corresponding to historical fisheye images. When the first BEV features generated based on the current 4D millimeter-wave radar point cloud data and the second BEV features generated based on the current fisheye image are generated, both the first BEV features corresponding to the current 4D millimeter-wave radar point cloud data and the second BEV features corresponding to the current fisheye image are added to the aforementioned cache pool. The capacity of this cache pool can be considered fixed. When the actual capacity of the cache pool reaches a preset upper limit, the BEV features that entered the cache pool first can be removed according to the "first-in, first-out" principle, or the BEV features with the lowest weight can be removed according to a preset weight priority, thereby achieving dynamic updates of the two types of BEV features in the cache pool.

[0082] Since the aforementioned cache pool stores multiple first BEV features generated at different times, as well as multiple second BEV features generated at different times, on the one hand, the multiple first BEV features and multiple second BEV features can be time-synchronized, thereby forming time-aligned first BEV features and second BEV features into BEV feature pairs. On the other hand, the multiple first BEV features and multiple second BEV features can be spatially aligned separately, that is, coordinate transformation or interpolation methods can be used to eliminate the displacement differences between BEV features of the same type generated at different times, and then the spatially aligned BEV features of the same type can be concatenated in chronological order to form a first BEV feature sequence and a second BEV feature sequence containing temporal information, thereby ensuring the positional continuity and semantic consistency of the target at different times.

[0083] Step 102: Generate the first spatial attention weight map corresponding to the first BEV feature, and generate the second spatial attention weight map corresponding to the second BEV feature.

[0084] In this embodiment of the application, the first spatial attention weight map is used to indicate the importance distribution of different spatial locations in the first BEV feature, that is, which locations need to be focused on and which locations can be relatively ignored.

[0085] The following section provides a detailed explanation of how to generate the first-space attention weight map.

[0086] Please see Figure 2 This is a flowchart illustrating a method for generating a first spatial attention weight map according to an embodiment of this application. The method flow is described as follows: Step 1021: Perform various pooling operations on the first BEV feature respectively.

[0087] In this embodiment, the various pooling operations can be considered as extracting different types of features from the first BEV feature from different perspectives. For example, the various pooling operations include max pooling and average pooling. Max pooling is used to capture the most significant and discriminative local features in the first BEV feature, focusing on the location point with the largest response in a certain region. Average pooling is used to capture the overall contextual information and statistical features of the first BEV feature, calculating the average response of a region, reflecting the overall situation of that region.

[0088] Step 1022: Fuse the pooling features of various pooling operations to obtain the first spatial attention weight map.

[0089] In this embodiment, the pooling features obtained from various pooling operations are sequentially concatenated, convolved, and activated to obtain the first spatial attention weight map. The calculation of the first spatial attention weight map is shown in formula (1): (2) in, This represents the first spatial attention weight map. Indicates the first BEV characteristic, This represents an activation function, such as the sigmoid function. This represents the convolution operation. This indicates that a channel-dimensional max pooling operation is performed on the first BEV feature. This indicates that the channel-dimensional average pooling operation is performed on the first BEV feature. This indicates that the results of the max pooling operation and the average pooling operation are concatenated along the channel dimension.

[0090] Similarly, the second spatial attention weight map is used to indicate the importance distribution of different spatial locations in the second BEV feature, i.e., which locations need to be given priority attention and which locations can be relatively ignored. The generation process of the second spatial attention weight map is similar to that of the first spatial attention weight map, and will not be repeated here. The calculation of the second spatial attention weight map is shown in formula (2): (1) in, This represents the second-space attention weight map. Indicates the second BEV characteristic, This represents an activation function, such as the sigmoid function. This represents the convolution operation. This indicates that a channel-dimensional max pooling operation is performed on the second BEV feature. This indicates that channel-dimensional average pooling is performed on the second BEV feature. This indicates that the results of the max pooling operation and the average pooling operation are concatenated along the channel dimension.

[0091] The following section uses the first attention weight map as an example to explain its generation process in detail.

[0092] 1. Input: First BEV feature Assume its shape is [Batch,C,H,W], where Barch represents the batch size (i.e. the number of BEVs processed simultaneously), C represents the feature vector dimension of each spatial location in the feature map, and H and W are the grid height and width of the BEV space.

[0093] 2. Step 1: Dual Pooling and Stitching Max pooling: Performed along the channel dimension C, resulting in a feature map of shape [Batch, 1, H, W]. It represents the most salient feature response across all channels at each spatial location (H, W).

[0094] Average pooling: This is also performed along the channel dimension C, resulting in a feature map of [Batch, 1, H, W]. It represents the average response (i.e., contextual information) of all feature channels at each spatial location (H, W).

[0095] Concatenation: The two feature maps [Batch, 1, H, W] are concatenated along the channel dimension to obtain a feature map [Batch, 2, H, W]. Now, each spatial location has two highly representative statistics: local significance and global context.

[0096] 3. Second step: Convolution + activation function Convolutional layers (usually 1x1 convolutions, but other sizes are also possible; this application does not impose any particular restrictions on them): Function: To learn how to fuse the two pieces of information obtained in the first step (saliency and context). The 1x1 convolution acts as a mini fully connected layer, learning to assign appropriate weights to the max-pooling and average-pooling features, and then combining them into a single feature map.

[0097] Output: Reduce the number of channels from 2 to 1 to generate an intermediate feature map [Batch, 1, H, W]. This map already contains the fused spatial importance information.

[0098] Activation function (usually Sigmoid): Function: Normalizes the value to the [0,1] interval.

[0099] Output: A weighted map of [Batch, 1, H, W], where the value of each pixel is between 0 and 1.

[0100] The [Batch,1,H,W] tensor is the first-space attention weight graph.

[0101] A value close to 1 indicates that the spatial location is very important (e.g., there may be key objects such as vehicles or pedestrians).

[0102] A value close to 0 indicates that the spatial location is relatively unimportant (e.g., it could be an open sky or ground).

[0103] Step 103: Based on the first spatial attention weight map and the second spatial attention weight map, fuse the first BEV feature and the second BEV feature to obtain the fused BEV feature.

[0104] In this embodiment of the application, since the first spatial attention weight map and the second spatial attention weight map respectively indicate the importance distribution of different spatial positions in their respective BEV features, the first spatial attention weight map and the second spatial attention weight map can be used to guide the fusion of the first BEV features and the second BEV features, so that the fused BEV features are more accurate.

[0105] The following section details the process of generating fused BEV features.

[0106] Please see Figure 3 This is a flowchart illustrating the first method for generating a fused BEV according to an embodiment of this application. Step 103 can be specifically implemented by executing sub-step 1031: Step 1031: Fuse the product of the first BEV feature and the second spatial attention weight map, and the product of the second BEV feature and the first spatial attention weight map, to obtain the fused BEV feature.

[0107] In this embodiment, calculating the product of the first BEV feature and the second spatial attention weight map is equivalent to re-examining, from the perspective of the second BEV feature, which positions in the first BEV feature need to be given priority attention and which positions can be relatively ignored. Even if some positions are considered relatively ignore from the perspective of the first BEV feature itself, they may be considered to need priority attention from the perspective of the second BEV feature.

[0108] Similarly, calculating the product of the second BEV feature and the first spatial attention weight map is equivalent to re-examining, from the perspective of the first BEV feature, which positions in the second BEV feature need to be emphasized and which positions can be relatively ignored. Even if some positions are considered relatively negligible from the perspective of the second BEV feature itself, they may be considered to require emphasis from the perspective of the first BEV feature.

[0109] Based on this, the resulting fused BEV features can contain richer information and have stronger out-of-distribution adaptability and robustness.

[0110] The first calculation method for integrating BEV features is shown in formula (3): (3) in, Indicates the integration of BEV features, This represents convolution, batch normalization, and activation operations. This represents the second-space attention weight map. Indicates the first BEV characteristic, This represents the first spatial attention weight map. This indicates the second BEV characteristic.

[0111] Please see Figure 4 This is a flowchart illustrating the second method for generating a fused BEV provided in this application embodiment. Step 103 can be specifically implemented by executing sub-step 1032: Step 1032: Fuse the product of the first BEV feature and the first spatial attention weight map, and the product of the second BEV feature and the second spatial attention weight map to obtain the fused BEV feature.

[0112] In this embodiment, calculating the product of the first BEV feature and the first spatial attention weight map ensures the most accurate identification and enhancement of important regions in the first BEV feature without introducing semantic bias due to the use of external weights. Similarly, calculating the product of the second BEV feature and the second spatial attention weight map ensures the most accurate identification and enhancement of important regions in the second BEV feature without introducing semantic bias due to the use of external weights. Thus, the fused features can maintain the independent semantic structure and important information of each original feature to the greatest extent, forming a "superimposed essence" feature representation.

[0113] The second calculation method for integrating BEV features is shown in formula (4): (4) in, Indicates the integration of BEV features, This represents convolution, batch normalization, and activation operations. Indicates the first BEV characteristic, This represents the first spatial attention weight map. Indicates the second BEV characteristic, This represents the attention weight map of the second space.

[0114] Please see Figure 5 This is a flowchart illustrating a target tracking method provided in an embodiment of this application. The method is applied to electronic devices, such as vehicles, and the flowchart is described below: Step 201: Obtain the current fused BEV features, wherein the current fused BEV features are obtained through... Figures 1 to 4 The data was obtained through a data fusion method.

[0115] In this embodiment of the application, the current fused BEV features can be obtained. For details, please refer to the above description, which will not be repeated here.

[0116] Step 202: Determine the current detection result based on the current fused BEV features.

[0117] In this embodiment, the current fused BEV features are configured with a corresponding initial query, which is a set of learnable vectors (N in number and D in dimension). Its task is to find and describe a potential target object (such as a vehicle or pedestrian) within the current fused BEV features. Initially, the initial query is general and does not contain specific scene information. The number of initial queries is typically greater than the number of typical targets in the current fused BEV features to ensure coverage of all potential targets. Then, each initial query "extracts" the most relevant information from the current fused BEV features, which are rich in global information. The specific process is as follows: 1. Mapping to the same space: The current fused BEV features and the initial query are mapped to a space with the same feature dimension through a linear transformation layer in order to perform similarity calculation.

[0118] 2. Calculate Attention Weights: Treat each query as the "Query" and all positions in the currently fused BEV features as the "Response" (Key and Value). A set of attention weights is obtained by calculating the feature similarity (dot product) between the query and each position in the currently fused BEV features. This set of weights essentially defines the region in the BEV space that the query focuses on. For example, a query representing "vehicles to the left front" will have a high attention weight in the left front region of the BEV feature map.

[0119] 3. Weighted Summation and Integration: Based on attention weights, the features at all locations in the currently fused BEV features are weighted and summed. This summation result is the extracted contextual information most relevant to the query. This information is then integrated (usually added) into the initial query to form the updated query. The updated query can be considered to contain rich target-specific information.

[0120] Based on this, the updated query can be input into a multilayer perceptron (MLP), which performs non-linear transformations on each updated query, ultimately outputting the current detection result: bounding box parameters: defining the size and position of the target in 3D space; class probability: representing the likelihood of belonging to each class; confidence score: typically the value of the highest class probability, indicating the reliability of the detection. Of course, the multilayer perceptron can also be replaced by kernel functions, graph neural networks, etc., and this application does not impose any particular restrictions on this.

[0121] It should be noted that not all queries will output valid targets. The model filters queries based on the confidence level of the MLP output, and only queries with a confidence level higher than a preset threshold are retained, forming the valid query set corresponding to the current fused BEV features.

[0122] Step 203: Calculate the similarity score between the current detection result and the historical detection results to form an affinity matrix. The historical detection results are the detection results corresponding to the historical fused BEV features.

[0123] In this embodiment, the cache pool can be considered to store historical detection results corresponding to historically fused BEV features. These historical detection results can be multiple valid queries corresponding to historically fused BEV features. Similarly, the current detection result can be multiple valid queries corresponding to the current fused BEV features. Therefore, multiple valid queries corresponding to the current fused BEV features and multiple valid queries corresponding to historically fused BEV features can be input into a multilayer perceptron to obtain similarity scores between different queries in different BEV features, thereby forming an affinity matrix. Of course, the aforementioned multilayer perceptron can also be replaced with kernel functions, graph neural networks, etc., and this application does not impose any particular restrictions on this.

[0124] It should be noted that the affinity matrix is ​​a... The matrix M represents the number of valid queries corresponding to the historical fused BEV features, and N represents the number of valid queries corresponding to the current fused BEV features. The value in the i-th row and j-th column represents the similarity between the i-th query (target) in history and the j-th query (target) in the present.

[0125] Step 204: Input the affinity matrix into the Sinkhorn algorithm to obtain the matching probability matrix, which is used to indicate the probability that the current detection result matches the historical detection results.

[0126] In this embodiment, the similarity scores in the affinity matrix may be unbalanced or there may be conflicting matches (e.g., a historical query may have a high similarity score with multiple current queries). The Sinkhorn algorithm, an iterative algorithm for solving the optimal transmission problem, transforms the affinity matrix into a doubly random matrix (matching probability matrix) where all elements are between 0 and 1, and the sum of each row and column is 1. This ensures that each historical query matches at most one current query, and the current query matches at most one historical query, conforming to the one-to-one correspondence principle in tracking.

[0127] The matching probability matrix above represents the matching probability between different queries corresponding to different BEV features. For example, the value in the i-th row and j-th column represents the probability that the i-th query (target) in history matches the j-th query (target) in the current query (target) (i.e., they are the same target).

[0128] Step 205: Target tracking based on the matching probability matrix.

[0129] In this embodiment of the application, if the probability value indicated by the i-th row and j-th column in the above-mentioned matching probability matrix is ​​the highest in the i-th row and exceeds a preset threshold (e.g., 0.5), it can be considered that the i-th query (target) in history matches the current j-th query (target). Then the ID of the i-th query (target) is assigned to the current j-th query (target), thereby realizing target tracking.

[0130] If all probability values ​​in the i-th row of the above matching probability matrix are lower than the preset threshold (e.g., 0.5), it indicates that the i-th query (target) in history has been lost, and the ID value of the i-th query (target) in history can be temporarily retained.

[0131] If all probability values ​​in the j-th column of the above matching probability matrix are lower than the preset threshold (e.g., 0.5), it indicates that the current j-th query (target) is newly added. Therefore, a new ID value can be assigned to the j-th query (target).

[0132] It should be noted that, due to the differentiability of the Sinkhorn algorithm's computation process, end-to-end joint optimization of the detection and tracking tasks can be achieved. This allows the network to not only learn how to detect objects, but also how to generate object representations that facilitate stable tracking.

[0133] The following is a specific example to illustrate the solution.

[0134] Assuming that adjacent BEV features are processed: The fused BEV features in frame T-1 correspond to three targets: {X1: red sedan, X2: blue truck, X3: motorcycle}.

[0135] The T-frame fused with BEV features corresponds to three targets: {Y1: red sedan, Y2: yellow taxi, Y3: motorcycle} Calculate the similarity score: Similarity between X1 and Y1: 0.95 (high similarity, both are red cars) Similarity between X1 and Y2: 0.15 (low similarity, the car models and colors are different) Similarity between X1 and Y3: 0.25 (low similarity, different car models) Similarity between X2 and Y1: 0.30 (moderate similarity, both are four-wheeled vehicles but different colors) Similarity between X2 and Y2: 0.35 (moderate similarity, both are four-wheeled vehicles) Similarity between X2 and Y3: 0.10 (low similarity, large differences in vehicle models) Similarity between X3 and Y1: 0.20 (low similarity, different car models) Similarity between X3 and Y2: 0.15 (low similarity, different car models) Similarity between X3 and Y3: 0.90 (high similarity, both are motorcycles) The resulting affinity matrix is ​​shown in Table 1 below: Table 1 The affinity matrix above is input into the Sinkhorn algorithm. After iterative normalization, a double random matrix (matching probability matrix) is obtained, as shown in Table 2: Table 2 If the preset threshold is 0.6 (i.e., a probability higher than 0.6 is considered a valid match): Successful tracking scenario: Since the matching probability of X1 and Y1 is 0.8, which exceeds the preset threshold (0.6), the ID of X1 (e.g., 111) is assigned to Y1, meaning that Y1 and X1 have the same ID (111). Similarly, since the matching probability of X3 and Y3 is 0.9, which exceeds the preset threshold (0.6), the ID of X3 (e.g., 222) is assigned to Y3, meaning that X3 and Y3 have the same ID (222).

[0136] Target loss scenario: Since the matching probability of X2 with Y1, Y2 and Y3 is lower than the preset threshold (0.6), it can be considered that X2 is lost in the Tth frame. X2's ID can be temporarily retained (e.g., 333). However, if it is determined that X2 is lost in multiple subsequent frames, X2's ID will be deleted.

[0137] New target scene: Since the matching probability of Y2 with X1, X2 and X3 is lower than the preset threshold (0.6), Y2 can be considered as a new target in frame T. Therefore, Y2 can be assigned a brand new ID (e.g., 555).

[0138] Please see Figure 6 This is a schematic diagram of a data fusion device provided in an embodiment of this application. The device includes: The acquisition unit 301 is used to acquire a bird's-eye view BEV feature pair. The BEV feature pair includes a first BEV feature determined based on four-dimensional millimeter-wave radar point cloud data and a second BEV feature determined based on fisheye image. The first BEV feature and the second BEV feature are located in the same coordinate system and are time-aligned. Attention weight map generation unit 302 is used to generate a first spatial attention weight map corresponding to the first BEV feature and to generate a second spatial attention weight map corresponding to the second BEV feature. The fusion unit 303 is used to fuse the first BEV feature and the second BEV feature based on the first spatial attention weight map and the second spatial attention weight map to obtain the fused BEV feature.

[0139] Optionally, the fusion unit 303 is specifically used for: The product of the first BEV feature and the second spatial attention weight map, and the product of the second BEV feature and the first spatial attention weight map are fused to obtain the fused BEV feature.

[0140] Optionally, the fusion unit 303 is specifically used for: The product of the first BEV feature and the first spatial attention weight map, and the product of the second BEV feature and the second spatial attention weight map are fused to obtain the fused BEV feature.

[0141] Optionally, the attention weight map generation unit 302 is specifically used for: Multiple pooling operations are performed on the first BEV feature; The pooling features of various pooling operations are fused to obtain the first spatial attention weight map.

[0142] Optionally, the pooling operations may include at least max pooling and average pooling.

[0143] Optionally, the four-dimensional millimeter-wave radar point cloud data is obtained after preprocessing with a diffusion model.

[0144] Please see Figure 7 This is a schematic diagram of a target tracking device provided in an embodiment of this application. The device includes: Acquisition unit 401 is used to acquire the current fused BEV features, wherein the current fused BEV features are obtained through... Figures 1 to 4 The data is obtained using the aforementioned data fusion method; The detection unit 402 is used to determine the current detection result based on the currently fused BEV features; The calculation unit 403 is used to calculate the similarity score between the current detection result and the historical detection results to form an affinity matrix. The historical detection results are the detection results corresponding to the historical fused BEV features. The optimization unit 404 is used to input the affinity matrix into the Sinkhorn algorithm to obtain the matching probability matrix, which is used to indicate the probability that the current detection result matches the historical detection results. Tracking unit 405 is used for target tracking based on the matching probability matrix.

[0145] Regarding the modules / units included in the various devices described in the above embodiments, they can be software modules / units, hardware modules / units, or a combination of both. For example, for devices applied to or integrated into a chip, all modules / units can be implemented using hardware methods such as circuits, or at least some modules / units can be implemented using software programs running on a processor integrated within the chip, while the remaining modules / units can be implemented using hardware methods such as circuits. For devices applied to or integrated into a chip module, all modules / units can be implemented using hardware methods such as circuits. Different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components of the chip module, or at least some modules / units can be implemented using software programs. The software program runs on the processor integrated inside the chip module, and the remaining modules / units can be implemented using hardware methods such as circuits. For each device applied to or integrated into an electronic terminal device, each of its modules / units can be implemented using hardware methods such as circuits. Different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components within the electronic terminal device. Alternatively, at least some modules / units can be implemented using software programs that run on the processor integrated inside the electronic terminal device, and the remaining (if any) modules / units can be implemented using hardware methods such as circuits.

[0146] Please see Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device includes at least one processor 501, which is used to execute computer program instructions stored in a memory to implement the implementation provided in the embodiment of this application. Figures 1 to 4 The data fusion method shown or such Figure 5 The flowchart shown illustrates the steps of the target tracking method.

[0147] Optionally, the processor 501 may be a central processing unit, a specific ASIC, or one or more integrated circuits used to control program execution.

[0148] Optionally, the electronic device may further include a memory 502 connected to at least one processor 501. The memory 502 may include ROM, RAM, and disk storage. The memory 502 stores data required for the processor 501 to run, i.e., it stores instructions executable by at least one processor 501. The at least one processor 501 executes instructions stored in the memory 502 to perform tasks such as... Figures 1 to 5 The method is shown. The number of memories 502 is one or more.

[0149] This application embodiment also provides a computer storage medium, wherein the computer storage medium stores computer instructions, which, when executed on a computer, cause the computer to perform actions such as... Figures 1 to 5 The method described.

[0150] The aforementioned computer-readable storage medium may be any combination of one or more computer-readable media. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0151] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device.

[0152] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, radio frequency (RF), etc., or any suitable combination thereof.

[0153] Computer program code for performing the operations described herein can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0154] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0155] In the several embodiments provided in this specification, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0156] Furthermore, the functional units in the various embodiments of this specification can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0157] This application provides a computer program product, which includes computer instructions that, when executed by a computer, cause the computer to perform actions such as... Figures 1 to 5 The method described.

[0158] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. A data fusion method, characterized in that, The method includes: Obtain a pair of BEV features from a bird's-eye view. The pair of BEV features includes a first BEV feature determined based on four-dimensional millimeter-wave radar point cloud data and a second BEV feature determined based on a fisheye image. The first BEV feature and the second BEV feature are located in the same coordinate system and are time-aligned. Generate a first spatial attention weight map corresponding to the first BEV feature, and generate a second spatial attention weight map corresponding to the second BEV feature; Based on the first spatial attention weight map and the second spatial attention weight map, the first BEV feature and the second BEV feature are fused to obtain the fused BEV feature.

2. The method according to claim 1, characterized in that, Based on the first spatial attention weight map and the second spatial attention weight map, the first BEV features and the second BEV features are fused to obtain fused BEV features, including: The fused BEV feature is obtained by fusing the product of the first BEV feature and the second spatial attention weight map, and the product of the second BEV feature and the first spatial attention weight map.

3. The method according to claim 1, characterized in that, Based on the first spatial attention weight map and the second spatial attention weight map, the first BEV features and the second BEV features are fused to obtain fused BEV features, including: The fused BEV feature is obtained by fusing the product of the first BEV feature and the first spatial attention weight map, and the product of the second BEV feature and the second spatial attention weight map.

4. The method according to claim 1, characterized in that, Generate a first spatial attention weight map corresponding to the first BEV feature, including: Perform various pooling operations on the first BEV feature respectively; The pooling features of the various pooling operations are fused to obtain the first spatial attention weight map.

5. The method according to claim 4, characterized in that, The various pooling operations include at least max pooling and average pooling.

6. The method according to claim 1, characterized in that, The four-dimensional millimeter-wave radar point cloud data was obtained after preprocessing with a diffusion model.

7. A target tracking method, characterized in that, The method includes: Obtain the current fused BEV features, wherein the current fused BEV features are obtained by the data fusion method according to any one of claims 1-6; Based on the current fused BEV features, the current detection result is determined; Calculate the similarity score between the current detection result and the historical detection results to form an affinity matrix. The historical detection results are the detection results corresponding to the historical fused BEV features. The affinity matrix is ​​input into the Sinkhorn algorithm to obtain a matching probability matrix, which is used to indicate the probability that the current detection result matches the historical detection result; Target tracking is performed based on the matching probability matrix.

8. A data fusion device, characterized in that, The device includes: The acquisition unit is used to acquire a pair of BEV features from a bird's-eye view. The pair of BEV features includes a first BEV feature determined based on four-dimensional millimeter-wave radar point cloud data and a second BEV feature determined based on a fisheye image. The first BEV feature and the second BEV feature are located in the same coordinate system and are time-aligned. The attention weight map generation unit is used to generate a first spatial attention weight map corresponding to the first BEV feature and to generate a second spatial attention weight map corresponding to the second BEV feature. The fusion unit is used to fuse the first BEV feature and the second BEV feature based on the first spatial attention weight map and the second spatial attention weight map to obtain fused BEV features.

9. A target tracking device, characterized in that, The device includes: An acquisition unit is configured to acquire current fused BEV features, wherein the current fused BEV features are obtained by the data fusion method according to any one of claims 1-6; The detection unit is used to determine the current detection result based on the currently fused BEV features; The calculation unit is used to calculate the similarity score between the current detection result and the historical detection results to form an affinity matrix, wherein the historical detection results are the detection results corresponding to the historical fused BEV features; An optimization unit is used to input the affinity matrix into the Sinkhorn algorithm to obtain a matching probability matrix, wherein the matching probability matrix is ​​used to indicate the probability that the current detection result matches the historical detection result; A tracking unit is used to track a target based on the matching probability matrix.

10. An electronic device, characterized in that, The electronic device includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to perform the steps of the method as described in any one of claims 1-6 or 7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store computer instructions that, when executed in a computer, cause the computer to perform the steps of the method as described in any one of claims 1-6 or 7.

Images