Multi-sensor category fusion method and device, electronic equipment and storage medium

Abstract

The invention discloses a multi-sensor category fusion method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a detection tracking target frame of a target continuous frame based on multiple sensors, and enabling the detection tracking target frame to serve as training sample data after processing, the detection tracking target frame at least comprises a point cloud detection result, an image detection result, a millimeter wave detection result and an infrared detection result; and training according to the training sample data to obtain a GraphTransform model so as to identify the category of the target. According to the invention, efficient and robust fusion of target or scene category information from various heterogeneous sensors is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, and in particular to a multi-sensor category fusion method and device, electronic equipment and a storage medium. BACKGROUND

[0002] Multi-sensor systems are popular in the fields of automatic driving, robot perception, intelligent monitoring and industrial detection, and obtain complementary environmental information through heterogeneous sensors (such as cameras, lidar, millimeter wave radar, ultrasonic sensors, infrared sensors, IMU, etc.). In these applications, accurately identifying target semantic categories (such as vehicles, pedestrians, traffic signs, obstacle types) or scene categories (such as indoor / outdoor, road types, weather conditions) is the core foundation for implementing advanced decision-making and intelligent behavior. However, the effective fusion of multi-source information faces the following severe challenges:

[0003] (1) Heterogeneity: The data modalities (images, point clouds, signal strengths, time series) and characteristics (resolution, viewing angle, noise pattern, semantic abstraction level) output by different sensors differ significantly, and traditional methods are difficult to handle uniformly.

[0004] (2) Temporal and spatial asynchrony and calibration: Sensor data collection times are not synchronized, and accurate spatial coordinate system alignment (external parameter calibration) is required, and the fusion process must address the problem of temporal and spatial inconsistency.

[0005] (3) Confidence difference: Due to environmental conditions such as light, weather, and occlusion, the confidence of each sensor in identifying the category of the same target varies significantly.

[0006] (4) Relationship: The complex spatial, semantic and temporal relationships between targets, targets and the environment, and sensor observations are crucial to the final category determination, but existing technologies lack effective modeling methods. SUMMARY

[0007] Embodiments of the present application provide a multi-sensor category fusion method, device, electronic equipment and storage medium to efficiently and robustly fuse target or scene category information from multiple heterogeneous sensors using a GraphTransformer model.

[0008] Embodiments of the present application employ the following technical solutions:

[0009] In a first aspect, embodiments of the present application provide a multi-sensor category fusion method, wherein the method comprises:

[0010] Based on the multi-sensor, a detection tracking target frame of a target continuous frame is obtained, and after processing, the detection tracking target frame is taken as training sample data, and the detection tracking target frame at least includes a point cloud detection result, an image detection result, a millimeter wave detection result, and an infrared detection result.

[0011] According to the training sample data, a GraphTransformer model is trained to identify the category of the target.

[0012] In some embodiments, the detection tracking target frame of the target continuous frame based on the multi-sensor includes:

[0013] Respectively, the detection results of each sensor are obtained, including the type Type of the target, the detection frame Box of the target, and the speed Velocity of the target.

[0014] The detection result of the camera: the type Type of the target.

[0015] The detection result of the millimeter wave radar: the type Type of the target and the speed Velocity of the target.

[0016] The detection result of the infrared camera: the type Type of the target and the detection frame Box of the target.

[0017] In some embodiments, the detection tracking target frame of the target continuous frame based on the multi-sensor further includes:

[0018] According to the detection results of the respective sensors, time and space synchronization processing is performed.

[0019] According to the time and space synchronization processing result, a dynamic confidence weight mechanism is adopted to establish a dynamic confidence model for different output dimensions of each sensor.

[0020] According to the dynamic confidence model, the detection results of different sensors on the same target are associated.

[0021] For the associated successful target, the final attribute of the associated successful target is obtained by weighted average of the attributes of each sensor, and the weight of the associated successful target is the dynamic confidence of the sensor on the attribute.

[0022] In some embodiments, the detection tracking target frame of the target continuous frame based on the multi-sensor further includes the steps of screening and quality inspection: error correction, elimination of invalid data, or confirmation of difficult cases.

[0023] In some embodiments, the GraphTransformer model is trained according to the training sample data to identify the category of the target, including:

[0024] According to the coding layer, target tracking information feature extraction and sensor target information feature extraction are respectively performed;

[0025] The graph self-attention coding is adopted, and the result of the target tracking information feature extraction includes encoded target tracking features;

[0026] The graph self-attention coding is adopted, and the result of the sensor target information feature extraction includes multi-source sensor target depth features.

[0027] In some embodiments, the GraphTransformer model is trained according to the training sample data to identify the category of the target, and further includes:

[0028] The detection results of the same target at different sensors and different times are associated with the same tracker target to obtain association information of the tracking target and the sensor target.

[0029] In some embodiments, the GraphTransformer model is trained according to the training sample data to identify the category of the target, and further includes:

[0030] According to the association information of the tracking target and the sensor target, the graph cross-attention is adopted to determine which sensor node Key or Value can be associated with the corresponding tracking target node Query, with the tracking target node as Query, the sensor observation node as Key, and the sensor observation node as Value.

[0031] After the association, the tracking features and the sensor features are fused.

[0032] In a second aspect, the embodiments of the present application further provide a multi-sensor category fusion device, wherein the device includes:

[0033] A sample processing module is configured to obtain a detection tracking target frame of a target based on multiple sensors, and process the detection tracking target frame as training sample data, wherein the detection tracking target frame at least includes point cloud detection results, image detection results, millimeter wave detection results, and infrared detection results.

[0034] A training module is configured to train a GraphTransformer model according to the training sample data to identify the category of the target.

[0035] In a third aspect, the embodiments of the present application further provide an electronic device, which includes a processor and a memory arranged to store computer executable instructions, the executable instructions, when executed, causing the processor to perform the above method.

[0036] In a fourth aspect, the embodiments of the present application further provide a computer-readable storage medium storing one or more programs, which, when executed by an electronic device comprising a plurality of application programs, cause the electronic device to perform the above method.

[0037] The above at least one technical solution adopted by the embodiments of the present application can achieve the following beneficial effects: based on a plurality of sensors, a detection tracking target frame of a target is obtained, the detection tracking target frame is processed to obtain training sample data, and then a GraphTransformer model is trained according to the training sample data to identify the category of the target. By providing a multi-sensor category fusion method based on Graph Transformer, the sensor nodes, target nodes and their complex spatial, semantic and temporal relationships in the multi-sensor system are explicitly and uniformly modeled. At the same time, by using the powerful global attention mechanism of Graph Transformer, the importance weight of different sensor observations on target category judgment is adaptively learned, and robust fusion is realized. In addition, the multi-modal heterogeneous sensor data and its inherent spatio-temporal asynchrony can also be effectively processed, and finally more accurate and robust target or scene semantic category information is output. BRIEF DESCRIPTION OF DRAWINGS

[0038] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application and illustrate the illustrative embodiments of the present application and their description serve to explain the present application, and do not constitute improper limitations on the present application. In the drawings:

[0039] Figure 1 A flowchart of a multi-sensor category fusion method in the embodiments of the present application is shown;

[0040] Figure 2 A schematic diagram of the implementation principle of the multi-sensor category fusion method in the embodiments of the present application is shown;

[0041] Figure 3 A schematic diagram of a Graph Transformer classification network in the multi-sensor category fusion method in the embodiments of the present application is shown;

[0042] Figure 4 A structural schematic diagram of a multi-sensor category fusion device in the embodiments of the present application is shown;

[0043] Figure 5 A structural schematic diagram of an electronic device in the embodiments of the present application is shown. DETAILED DESCRIPTION

[0044] In order to make the purposes, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in connection with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0045] The inventors have found that the current mainstream fusion methods have obvious limitations:

[0046] (a) When feature-level fusion (such as simple concatenation or weighted average feature vector) is used, the problem of heterogeneous data compatibility cannot be overcome, and it is difficult to model complex relationships and is susceptible to noise or failed sensor interference.

[0047] (b) When decision-level fusion (such as Bayesian theory, D-S evidence theory) is used, although the independent classification results are probabilistically fused, the potential correlation between observations is ignored, and the performance of the single-sensor classifier is excessively relied on.

[0048] (c) When early graph neural network fusion (such as GCN / GAT) is used, although the graph structure is introduced to model the relationship, the message passing mechanism is inefficient in capturing long-distance dependencies and global context, and is sensitive to graph structure perturbation.

[0049] (d) Sequence fusion based on Transformer treats sensor data as flat sequence input, which destroys the inherent spatial / topological structure information, has high computational complexity, and cannot explicitly express the relationship between sensors.

[0050] In view of the above deficiencies, the multi-sensor category fusion method in the embodiments of the present application provides a new fusion framework that uniformly models heterogeneous data, adaptively fuses spatial and asynchronous information, dynamically weights multi-source confidence, and explicitly captures complex relationships, to improve the robustness and accuracy of category perception.

[0051] The multi-sensor category fusion method in the embodiments of the present application provides a global adaptive fusion mechanism. The multi-head attention calculation layer of the Graph Transformer is used to dynamically learn the interaction weight between the sensor node and the target node. By fusing node features, edge attributes and position encoding, context-aware aggregation of cross-modal features is achieved, which significantly improves the robustness to noise, occlusion and sensor failure.

[0052] The multi-sensor category fusion method in the embodiments of the present application performs confidence-driven decision optimization. Based on the attention weight, a sensor contribution distribution map is generated, and combined with the hierarchical feature evolution of the target node, a category label with uncertainty quantization and a confidence score are output, providing a reliable basis for downstream decision-making.

[0053] The technical solutions provided by the embodiments of the present application are described in detail below with reference to the drawings.

[0054] The embodiments of the present application provide a multi-sensor category fusion method, as shown in the following figure, a multi-sensor category fusion method flowchart in the embodiments of the present application is provided, and the method at least includes the following steps S110 to S120: Figure 1

[0055] Step S110, based on the multi-sensor, the detection and tracking target frame of the target is obtained, and after processing, the detection and tracking target frame is used as training sample data, and the detection and tracking target frame at least includes point cloud detection result, image detection result, millimeter wave detection result, and infrared detection result.

[0056] Multi-sensor data acquisition and initial processing, through laser radar sensor, camera sensor, millimeter wave sensor, infrared sensor, the point cloud detection result of the target is obtained respectively, including: Type (target type), Box (BEV target frame length, width and azimuth representation), Velocity (velocity representation).

[0057] Image detection result: Type, millimeter wave detection result: Type, Velocity, infrared detection result: Type, Box. Complete the data acquisition of the multi-sensor fusion tracking target frame. Use multi-sensor fusion technology to manage the whole life cycle tracking process of the target, and provide multi-dimensional original information for subsequent processing.

[0058] Through multi-sensor fusion, integrate multi-dimensional data such as point cloud, image, millimeter wave, and infrared, and make up for the deficiency of single sensor in target detection and classification.

[0059] After processing, the detection and tracking target frame is used as training sample data, which also includes time and space synchronization, that is, time synchronization and space calibration. Wherein, time synchronization: using hardware PPS signal synchronization pulse or network synchronization based on precise time protocol (PTP), unifying the time stamp of all sensors to the master clock (such as industrial computer or GNSS receiver), ensuring that the data time consistency error is less than millisecond level. Wherein, space calibration: pre-acquire the external parameter matrix of each sensor relative to the vehicle coordinate system, that is, the center of the vehicle body, including rotation matrix R and translation vector T, so as to convert all sensor observation data to a unified coordinate system.

[0060] ​The processed detection and tracking target frame is further processed as training sample data, which further includes a pre-annotation process. The type fusion result generated after synchronization is automatically generated as a standard format label file, including but not limited to a COCO format JSON file or a KITTI format TXT file.

[0061] The processed detection and tracking target frame is further processed as training sample data, which further includes manual screening and quality inspection. Specifically, an annotator can view the entire tracking sequence of a target, thereby more easily judging the correctness of the pre-annotation result. Mainly error correction, such as incorrect ID switching (ID-Switch), incorrect classification, and inaccurate frame; eliminate invalid data, such as transient ghost objects; confirm difficult cases, such as severely occluded or truncated targets.

[0062] Step S120, training a GraphTransformer model according to the training sample data to identify the category of the target.

[0063] First, build the GraphTransformer network. Design the network structure and loss function.

[0064] Second, when training the model, load the GraphTransformer model, set appropriate learning parameters, input the preprocessed data into the model for training, use the self-attention mechanism of the Transformer to mine the correlation of target features in multi-sensor fusion data, and learn the patterns and rules of target classification.

[0065] Preferably, model evaluation can also be performed. During and after training, reasonable evaluation indicators including but not limited to classification accuracy, recall rate, F1 value, etc. are used to evaluate the performance of the model, and it is determined whether the model is the best model for the current performance. When the model evaluation is the best performance, save the weight parameters of the current model for subsequent actual target classification applications.

[0066] Preferably, model prediction can also be performed. The preprocessed test data features are input into the trained model, forward inference calculation is performed, the prediction result is generated and decoded into the final classification label.

[0067] Through the above method, the detection and tracking target frame of the target continuous frame is obtained based on the multi-sensor, and the processed detection and tracking target frame is used as training sample data. The detection and tracking target frame at least includes point cloud detection result, image detection result, millimeter wave detection result, and infrared detection result. Through multi-sensor fusion, multi-dimensional data such as point cloud data, image data, millimeter wave data, and infrared data are integrated, which makes up for the shortcomings of single sensor in target detection and classification.

[0068] Through the above method, the GraphTransformer model is trained according to the training sample data to identify the category of the target, overcome the heterogeneous data compatibility problem, construct a unified heterogeneous interaction graph, and provide a structured knowledge representation basis for fusion.

[0069] Through the above method, the GraphTransformer model is used to mine data feature association, improve target classification accuracy and robustness, and is suitable for target recognition and classification tasks in intelligent transportation, security monitoring and other scenarios, and provides an effective technical means for target perception in complex environments.

[0070] In addition, through the above method, before the GraphTransformer model is trained according to the training sample data, a pre-annotation industrialization pipeline is used to improve annotation efficiency and automatically complete data closed loop.

[0071] In an embodiment of the present application, the detection and tracking target frame of the target based on multiple sensors includes: obtaining the detection results of each sensor respectively, including the detection results of the laser radar, the type Type of the target, the detection frame Box of the target, and the speed Velocity of the target; the detection results of the camera: the type Type of the target; the detection results of the millimeter wave radar: the type Type of the target and the speed Velocity of the target; the detection results of the infrared camera: the type Type of the target and the detection frame Box of the target.

[0072] Multi-sensor data acquisition and initial processing, through laser radar sensor, camera sensor, millimeter wave sensor, infrared sensor, respectively obtaining the point cloud detection results of the target including: Type (target type), Box (BEV target frame length, width and azimuth representation), Velocity (speed representation).

[0073] Image detection results: Type, millimeter wave detection results: Type, Velocity, infrared detection results: Type, Box. Complete the data acquisition of the multi-sensor fusion tracking target frame. Use multi-sensor fusion technology to manage the whole life cycle tracking process of the target, and provide multi-dimensional original information for subsequent processing.

[0074] In an embodiment of the present application, the detection and tracking target frame based on the multi-sensor further comprises: time and space synchronization processing according to the detection results of the respective sensors; a dynamic confidence weight mechanism is used to establish a dynamic confidence model for different output dimensions of each sensor according to the time and space synchronization processing results; the detection results of different sensors on the same target are associated according to the dynamic confidence model; for the successfully associated target, the final attribute of the successfully associated target is obtained by weighted average of the attributes of each sensor, and the weight of the successfully associated target is the dynamic confidence of the sensor on the attribute.

[0075] Through multi-sensor fusion, multi-dimensional data such as point cloud, image, millimeter wave, and infrared are integrated to make up for the shortcomings of single sensors in target detection and classification.

[0076] The detection and tracking target frame after processing is used as training sample data, which further comprises time and space synchronization, i.e. time synchronization and space calibration. The time synchronization uses a hardware PPS signal synchronization pulse or a network synchronization based on a precise time protocol (PTP) to unify the time stamps of all sensors to a master clock (such as an industrial computer or a GNSS receiver), so as to ensure that the data time consistency error is less than milliseconds. The space calibration: the external parameter matrix containing the rotation matrix R and the translation vector T of each sensor relative to the vehicle coordinate system (i.e. the vehicle body center) is accurately obtained in advance by a calibration algorithm, so as to convert all sensor observation data to a unified coordinate system.

[0077] The detection and tracking target frame after processing is used as training sample data, which further comprises a rule algorithm fusion based on dynamic confidence, which specifically introduces a dynamic confidence weight mechanism. A dynamic confidence model is established for different output dimensions of each sensor.

[0078] Firstly, the Box and speed confidence of the laser radar are high, and the type confidence is low when the point cloud boundary and target are truncated. Secondly, the type confidence of the camera is high when the light is good, and the type confidence is low at night or in strong backlight. Further, the speed confidence of the millimeter wave radar is always high, but the type resolution ability is poor and the confidence is low. Finally, the Box of the infrared sensor has high confidence at night and in bad weather.

[0079] Based on the above, the detection results of different sensors on the same physical target are associated based on IOU (intersection over union), Mahalanobis distance, feature similarity, etc. using the Hungarian algorithm. For the successfully associated target, the final attribute is obtained by weighted average of the attributes of each sensor, and the weight is the dynamic confidence of the sensor on the attribute, which is expressed by the formula:

[0080]

[0081] in, and These represent the type and weight of LiDAR detection,

[0082] in, and These represent the type and weight of camera detection,

[0083] in, and These represent the type and weight detected by the infrared camera, respectively.

[0084] In one embodiment of this application, the detection and tracking target bounding boxes obtained from continuous target frames based on multiple sensors further include screening and quality inspection steps: error correction, removal of invalid data, or confirmation of difficult cases.

[0085] Specifically, annotators can view the entire tracking sequence of the target, making it easier to judge the correctness of the pre-annotation results. They mainly perform error correction, such as incorrect ID switching, incorrect classification, and inaccurate bounding boxes; remove invalid data, such as ghost objects that appear only briefly; and confirm difficult cases, such as severely occluded or truncated targets.

[0086] In one embodiment of this application, the step of training a GraphTransformer model based on the training sample data to identify the category of the target includes: extracting tracking target information features and sensor target information features according to the encoding layer; using graph self-attention encoding to obtain the result of the tracking target information feature extraction including the encoded tracking target features; and using graph self-attention encoding to obtain the result of the sensor target information feature extraction including the depth features of multi-source sensor targets.

[0087] S1, Build a Graph Transformer classification network, encoding layer - tracking target information feature extraction.

[0088] The input is the tracking target features from all n frames in a training sample, viewed as a fully connected graph with n nodes. Graph cross-attention is used to aggregate the features. The features of each node are mapped to Query, Key, and Value using a learnable weight matrix.

[0089] Self-attention calculation: .in, The correlation between all pairs of nodes in the graph is calculated, thereby learning the temporal dependencies of the target itself. Encoded tracking target features. It contains rich temporal context information.

[0090] The encoding layer extracts target information features based on graph self-attention, using initial features of the target. (Trajectory points, velocity, type, and size, etc.) are input to the encoding layer. Target features are mapped to queries (Q), keys (K), and values ​​(V), mining internal correlations within the target (such as historical trajectory dependencies and motion patterns). After attention-weighted summation, the encoded tracking target features are output. .pass Layered self-attention computation allows for full interaction between target features.

[0091] S2, Build a Graph Transformer classification network, encoding layer - sensor target information feature extraction.

[0092] Feature extraction of target information from the encoding layer of the sensor. For the target perceived by the sensor, graph self-attention encoding is also used. Since different sensors output feature dimensions and semantics differ, normalization is first performed, and then projected onto the same feature space to obtain the features. By learning the correlations between sensor targets through self-attention (such as the cross-sensor observation consistency of the same physical target), and after one layer of encoding, the deep features of multi-source sensor targets are output. This supplements the environmental perception information of the tracked target.

[0093] Within the same n frames, all the features of the detected targets generated by all sensors (LiDAR, Camera, Radar, IR) constitute a single node. Since the feature dimensions and physical meanings differ between different sensors, a sensor-specific multilayer perceptron (MLP) is first used to project each observed feature onto a unified feature space (with dimensions of 1). Then, these unified sensor observation nodes are input into another graph self-attention layer. This layer allows all sensor observation nodes to interact with each other, learn the spatial and semantic relationships between them (e.g., fusing information about the "car" seen by the camera and the "moving object" detected by the radar), and outputs encoded sensor target features. .

[0094] Correlation information between tracking target and sensor target The core objective of association is to ensure that the detection results of the same physical target on different sensors at different times are associated with the same tracker target. The association relationship is represented in the form of an association matrix, with associated targets represented as 1, and the association score (a weighted sum of distance, feature similarity, motion consistency, etc.) is stored, while unassociated targets are represented as 0.

[0095] The decoding layer extracts side information features from both the tracking target and the sensor target. Graph Cross-Attention is used to fuse the encoded tracking features with the sensor target features. (The sensor features are then used as the basis for further processing.) Track features for keys (K) and values ​​(V). For query (Q), track target-sensor target correlation information. For the edge. (Through) Layer cross-attention iteration outputs features of fused information. It accurately correlates tracking logic with sensor observations, improving the estimation accuracy of target categories.

[0096] The model also includes a classification layer design, which inputs the decoded features into the classification head. It uses a simple network structure, a single linear layer and activation function to avoid overfitting, and focuses on semantic discrimination of fused features to directly serve tracking task decisions (target attribute recognition, tracking quality assessment).

[0097] The model also includes a loss function design, with the classification loss using cross-entropy loss to measure the difference between the predicted class distribution and the true label.

[0098] S3. Build a Graph Transformer classification network to construct a correlation graph between the tracking target and the sensor target.

[0099] The association relationships are represented by a sparse, weighted bipartite graph, serving as edge information for the cross-attention in the decoding layer. The association matrix is ​​represented as follows: If the i-th tracked target node (in frame t) is associated with the j-th sensor observation node (in frame t) (representing the same physical target), then = 1, otherwise 0. This matrix comes from the fusion result or truth labels of step 5.

[0100] Correlation score S As an attention prior, it is a weighted sum of the metrics used for association in the steps described above. This score matrix can be used to modulate attention weights in cross-attention.

[0101] S4, build a Graph Transformer classification network, decoding layer - feature fusion based on graph cross attention.

[0102] The goal is to encode the tracking features and sensor features Deep fusion is performed. Graph cross-attention is used, with the target node being tracked as the query and the sensor observation nodes as the key and value. The association matrix A defines which sensor nodes (key / value) are eligible to interact with which target node (query). Unassociated attention weights are masked. The cross-attention calculation formula is:

[0103] .

[0104] Each tracked target node aggregates rich features from all associated sensor observation nodes to obtain a fused target representation. .

[0105] A Graph Transformer classification network is constructed. The classification head consists of a linear layer and a Dropout layer, followed by a Softmax activation function, which outputs the probability distribution of all candidate classes. This simple design avoids overfitting and forces the model to encode the most discriminative information into the features.

[0106] Build a Graph Transformer classification network and design the loss function; the classification loss uses labeled smooth cross-entropy loss. This enhances the model's generalization ability. It also encourages the model to learn similar features embedded in associated targets and dissimilar features in unassociated targets. A triplet loss function is employed. .

[0107] Total loss: , where α is the equilibrium hyperparameter.

[0108] In one embodiment of this application, the step of training a GraphTransformer model based on the training sample data to identify the category of the target further includes: associating the detection results of the same target with the same tracker target at different sensors and at different times, thereby obtaining the association information between the tracked target and the sensor target.

[0109] Specifically, the correlation information between the tracking target and the sensor target. The core objective of association is to ensure that the detection results of the same physical target on different sensors at different times are associated with the same tracker target. The association relationship is represented in the form of an association matrix, with associated targets represented as 1, and the association score (a weighted sum of distance, feature similarity, motion consistency, etc.) is stored, while unassociated targets are represented as 0.

[0110] In one embodiment of this application, the step of training a GraphTransformer model based on the training sample data to identify the category of the target further includes: based on the association information between the tracked target and the sensor target, using graph cross-attention with the tracked target node as the query and the sensor observation node as the key and value, determining which sensor node's key or value can be associated with the corresponding tracked target node query; and fusing the tracking features and sensor features after association.

[0111] The decoding layer extracts side information features from both the tracking target and the sensor target. Graph Cross-Attention is used to fuse the encoded tracking features with the sensor target features. (The sensor features are then used as the basis for further processing.) Track features for keys (K) and values ​​(V). For query (Q), track target-sensor target correlation information. For the edge. (Through) Layer cross-attention iteration outputs features of fused information. It accurately correlates tracking logic with sensor observations, improving the estimation accuracy of target categories.

[0112] Model training uses the AdamW optimizer, whose weight decay strategy helps prevent overfitting. Cosine annealing with warm restarts is employed to help the model escape local optima.

[0113] Model evaluation and deployment, in addition to accuracy, recall, and F1 score, focus on the classification confusion matrix to analyze specific error patterns, and temporal smoothness (whether the class frequently jumps in the sequence). Model preservation not only saves the best-performing model on the validation set but also uses the exponential moving average (EMA) technique to save a smoothed version of the model, which typically has better generalization performance. The trained model is converted into a high-performance inference engine using TensorRT, the computation graph is optimized, and quantization (INT8 precision) is performed to meet the real-time requirements of automotive embedded platforms.

[0114] During online inference, the process for model prediction is as follows: data acquisition - rule fusion - constructing the tracking graph and sensor observation graph for the current time window - GraphTransformer model forward inference - outputting classification probabilities - decoding into final category labels using threshold or maximum probability methods and updating target attributes.

[0115] This application embodiment also provides a multi-sensor category fusion device 400, such as... Figure 4As shown, a structural schematic diagram of an embodiment of this application is provided. The multi-sensor category fusion device 400 includes at least: a sample processing module 410 and a training module 420, wherein:

[0116] In one embodiment of this application, the sample processing module 410 is specifically used to: obtain detection and tracking target boxes of continuous target frames based on multiple sensors, and use the detection and tracking target boxes as training sample data after processing. The detection and tracking target boxes include at least point cloud detection results, image detection results, millimeter wave detection results, and infrared detection results.

[0117] Multi-sensor data acquisition and initial processing: Point cloud detection results of the target are obtained through lidar sensors, camera sensors, millimeter-wave sensors, and infrared sensors, including: Type (target type), Box (the length, width, and azimuth of the target bounding box in BEV format). (in terms of speed) express).

[0118] Image detection results: Type; Millimeter wave detection results: Type, Velocity; Infrared detection results: Type, Box. Data acquisition for target bounding box tracking was completed using multi-sensor fusion tracking. Multi-sensor fusion technology was used to manage the entire lifecycle tracking process of the target, providing multi-dimensional raw information for subsequent processing.

[0119] By integrating multi-sensor fusion, data from multiple dimensions such as point cloud, image, millimeter wave, and infrared are combined to make up for the shortcomings of a single sensor in target detection and classification.

[0120] The processed target bounding boxes used as training sample data also include: spatiotemporal synchronization, namely time synchronization and spatial calibration. Time synchronization: Hardware PPS signal synchronization pulses or network synchronization based on Precise Time Protocol (PTP) are used to unify the timestamps of all sensors to the master clock (such as an industrial control computer or GNSS receiver), ensuring data time consistency errors are below milliseconds. Spatial calibration: An external parameter matrix containing the rotation matrix R and translation vector T of each sensor relative to the vehicle coordinate system (i.e., the vehicle center) is accurately obtained in advance using a calibration algorithm, thereby transforming all sensor observation data to a unified coordinate system.

[0121] The process of using the detected and tracked target bounding boxes as training sample data also includes a pre-annotation process. The type fusion results generated after synchronization are automatically converted into standard format label files, including but not limited to JSON files in COCO format or TXT files in KITTI format.

[0122] The processing of the detected and tracked target bounding boxes as training sample data also includes manual screening and quality control. Specifically, annotators can view the entire tracking sequence of the target, making it easier to judge the correctness of the pre-annotation results. This mainly involves error correction, such as incorrect ID-switch, incorrect classification, and inaccurate bounding boxes; removing invalid data, such as ghost objects that appear only briefly; and confirming difficult cases, such as severely occluded or truncated targets.

[0123] In one embodiment of this application, the training module 420 is specifically used to: train a GraphTransformer model based on the training sample data to identify the category of the target.

[0124] First, build the GraphTransformer network. Design the network structure and loss function.

[0125] Secondly, during model training, the GraphTransformer model is loaded, appropriate learning parameters are set, and preprocessed data is input into the model for training. The self-attention mechanism of Transformer is used to explore the correlation between target features in multi-sensor fusion data and learn the patterns and rules of target classification.

[0126] Preferably, model evaluation can also be performed. During and after training, reasonable evaluation metrics, including but not limited to classification accuracy, recall, and F1 score, are used to evaluate the model performance and determine whether the model is the best performing model at present. When the model is evaluated as the best performing model, the weight parameters of the current model are saved for subsequent practical target classification applications.

[0127] Preferably, model prediction can also be performed. The preprocessed test data features are input into the fully trained model, forward inference calculation is performed, prediction results are generated and decoded into final classification labels.

[0128] In one embodiment of this application, the training module 420 is specifically used for:

[0129] Acquire the detection results of each sensor, including the target type (Type), target detection box (Box), and target velocity (Velocity) from the LiDAR.

[0130] Camera detection results: Target type (Type);

[0131] Millimeter-wave radar detection results: target type and target velocity;

[0132] Infrared camera detection results: Target type (Type) and target detection box (Box).

[0133] In one embodiment of this application, the training module 420 is specifically used for:

[0134] Time and space synchronization processing is performed based on the detection results of each sensor;

[0135] Based on the time and space synchronization processing results, a dynamic confidence weighting mechanism is adopted to establish a dynamic confidence model for different output dimensions of each sensor;

[0136] Based on the dynamic confidence model, the detection results of different sensors for the same target are correlated;

[0137] For a successfully associated target, the final attribute of the successfully associated target is obtained by a weighted average of the attributes of each sensor, and the weight of the successfully associated target is the dynamic confidence level of the sensor on that attribute.

[0138] In one embodiment of this application, a screening and quality control module is also included for: error correction, removal of invalid data, or confirmation of difficult cases.

[0139] In one embodiment of this application, the training module 420 is specifically used for:

[0140] Based on the coding layer, feature extraction is performed on tracking target information and sensor target information respectively;

[0141] Using graph self-attention encoding, the result of the tracking target information feature extraction includes the encoded tracking target features;

[0142] Using graph self-attention coding, the result of extracting the sensor target information features includes the depth features of multi-source sensor targets.

[0143] In one embodiment of this application, the training module 420 is specifically used for:

[0144] By associating the detection results of the same target with different sensors at different times, we can obtain the association information between the tracked target and the sensor target.

[0145] In one embodiment of this application, the training module 420 is specifically used for:

[0146] Based on the association information between the tracking target and the sensor target, graph cross attention is used to determine which sensor node's key or value can be associated with the corresponding tracking target node's query, with the tracking target node as the query and the sensor observation node as the key and value.

[0147] After correlation, the tracking features and sensor features are fused.

[0148] It is understood that the above-described multi-sensor category fusion device can implement all the steps of the multi-sensor category fusion method provided in the foregoing embodiments. The relevant explanations of the multi-sensor category fusion method are applicable to the multi-sensor category fusion device, and will not be repeated here.

[0149] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 5 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0150] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0151] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0152] The processor reads the corresponding computer program from non-volatile memory into main memory and then runs it, forming a multi-sensor category fusion device at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0153] The detection and tracking target boxes are obtained from continuous frames of the target obtained by multiple sensors. The detection and tracking target boxes are processed and used as training sample data. The detection and tracking target boxes include at least point cloud detection results, image detection results, millimeter wave detection results, and infrared detection results.

[0154] The GraphTransformer model is trained based on the training sample data to identify the category of the target.

[0155] The above is as stated in this application. Figure 1 The method executed by the multi-sensor fusion device disclosed in the illustrated embodiments can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0156] The electronic device can also perform Figure 1 A method for implementing a multi-sensor category fusion device, and the realization of the multi-sensor category fusion device in... Figure 1 The functions of the embodiments shown are not described in detail here.

[0157] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by an electronic device including multiple applications, enable the electronic device to perform... Figure 1 The method executed by the multi-sensor category fusion device in the illustrated embodiment is specifically used to perform:

[0158] The detection and tracking target boxes are obtained from continuous frames of the target obtained by multiple sensors. The detection and tracking target boxes are processed and used as training sample data. The detection and tracking target boxes include at least point cloud detection results, image detection results, millimeter wave detection results, and infrared detection results.

[0159] The GraphTransformer model is trained based on the training sample data to identify the category of the target.

[0160] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0161] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0162] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0163] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes ​ The steps of the function specified in one or more boxes.

[0164] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0165] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0166] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0167] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0168] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0169] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A multi-sensor category fusion method, wherein, The method includes: The detection and tracking target boxes are obtained from continuous frames of the target obtained by multiple sensors. The detection and tracking target boxes are processed and used as training sample data. The detection and tracking target boxes include at least point cloud detection results, image detection results, millimeter wave detection results, and infrared detection results. The GraphTransformer model is trained based on the training sample data to identify the category of the target.

2. The method as described in claim 1, wherein, The target detection and tracking bounding box obtained based on multiple sensors to obtain continuous target frames includes: Acquire the detection results of each sensor, including the target type (Type), target detection box (Box), and target velocity (Velocity) from the LiDAR detection results; Camera detection results: Target type (Type); Millimeter-wave radar detection results: target type and target velocity; Infrared camera detection results: Target type (Type) and target detection box (Box).

3. The method as described in claim 2, wherein, The target detection and tracking bounding box obtained based on multiple sensors to obtain continuous target frames also includes: Time and space synchronization processing is performed based on the detection results of each sensor; Based on the time and space synchronization processing results, a dynamic confidence weighting mechanism is adopted to establish a dynamic confidence model for different output dimensions of each sensor; Based on the dynamic confidence model, the detection results of different sensors for the same target are correlated; For a successfully associated target, the final attribute of the successfully associated target is obtained by a weighted average of the attributes of each sensor, and the weight of the successfully associated target is the dynamic confidence level of the sensor on that attribute.

4. The method of claim 3, wherein, The detection and tracking target bounding box based on multiple sensors to obtain continuous target frames also includes screening and quality inspection steps: error correction, removal of invalid data or confirmation of difficult cases.

5. The method of claim 1, wherein, The step of training a GraphTransformer model based on the training sample data to identify the category of the target includes: Based on the coding layer, feature extraction is performed on tracking target information and sensor target information respectively; Using graph self-attention encoding, the result of the tracking target information feature extraction includes the encoded tracking target features; Using graph self-attention coding, the result of extracting the sensor target information features includes the depth features of multi-source sensor targets.

6. The method of claim 5, wherein, The step of training a GraphTransformer model based on the training sample data to identify the category of the target also includes: By associating the detection results of the same target with different sensors at different times, we can obtain the association information between the tracked target and the sensor target.

7. The method of claim 6, wherein, The step of training a GraphTransformer model based on the training sample data to identify the category of the target also includes: Based on the association information between the tracking target and the sensor target, graph cross attention is used, with the tracking target node as the query and the sensor observation node as the key and value, to determine which sensor node's key or value can be associated with the corresponding tracking target node query. After correlation, the tracking features and sensor features are fused.

8. A multi-sensor category fusion device, wherein, The device includes: The sample processing module is used to obtain the detection and tracking target bounding boxes of the target in continuous frames based on multiple sensors, and to process the detection and tracking target bounding boxes as training sample data. The detection and tracking target bounding boxes include at least point cloud detection results, image detection results, millimeter wave detection results, and infrared detection results. The training module is used to train a GraphTransformer model based on the training sample data to identify the category of the target.

9. An electronic device, comprising: processor; as well as A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of applications, cause the electronic device to perform the method of any one of claims 1 to 7.

Images