High-performance loosely-coupled multi-modal data fusion system for smart driving environmental perception system and vehicle-mounted device

Abstract

Disclosed are a high-performance loosely-coupled multi-modal data fusion system for a smart driving environmental perception system and a vehicle-mounted device, comprising: a fusion detection model based on a modality-independent feature interaction strategy, which is configured for converting a LiDAR point cloud, a camera image, and a millimeter-wave radar point cloud into a unified bird's-eye view representation, and performing multi-modal fusion; and a fusion tracking model based on a motion-appearance feature cascaded coupling data association strategy, which is configured for performing subsequent trajectory tracking and matching according to multi-modal fusion feature information. A VoD data set and a K-Radar data set are selected for training, verifying, and testing the comprehensive performance of the models, and a TensorRT accelerated inference model is applied, then quantized, and deployed to a vehicle-mounted computational testing platform. The present invention is compatible with mainstream sensor deployment solutions, and achieves the efficient complementary fusion of multi-source heterogeneous sensor information, significantly improving the reliability, accuracy, and adaptability of vehicle-mounted perception systems, thereby effectively responding to extreme operating conditions such as complex traffic scenarios and inclement weather.

Description

High-performance loosely coupled multi-modal data fusion system for intelligent driving environment perception system and vehicle-mounted device TECHNICAL FIELD

[0001] The present application belongs to the field of intelligent networked vehicle environment perception, and particularly relates to a high-performance loosely coupled multi-modal data fusion system for three-dimensional target detection and tracking tasks and a vehicle-mounted device. BACKGROUND

[0002] With the gradual landing of intelligent driving technology, the safe driving problem of intelligent networked vehicles has attracted more and more attention. Environment perception, as the primary task of intelligent driving systems, is the basis and premise for subsequent decision planning and control execution tasks. Currently, the environment perception method based on multi-modal data fusion has gradually become mainstream, that is, through multiple types of vehicle-mounted sensors such as cameras, laser radars, and millimeter wave radars, the surrounding environment is comprehensively perceived. The fusion perception method can overcome the shortcomings of a single sensor to a certain extent, realize the coordinated optimization and comprehensive processing of multi-sensor data, and improve the adaptability of the environment perception system in complex traffic scenes and bad weather.

[0003] Different types of vehicle-mounted sensors have different working principles, advantages and disadvantages. Camera images contain dense color and texture information, but are easily affected by exposure, resulting in semantic distortion of the image; laser point clouds can accurately depict the three-dimensional structure information of the surrounding environment, but also have sparsity and disorder; millimeter wave radars have strong penetration ability and are almost not affected by bad weather, but also have low angular resolution, false alarm and clutter interference problems. Relying solely on single modal information will inevitably lead to inaccurate perception in some scenarios. On the contrary, the perception method based on multi-modal data fusion can utilize effective information in camera images, laser point clouds and millimeter wave point clouds, suppress environmental noise interference, and achieve accurate and reliable perception.

[0004] However, it is not easy to maximize the advantages of multi-modal data fusion. Different sensors depict the environment in different ways, with significant differences. Simply fusing different modal data not only destroys the original data structure, causing mutual interference, but also increases the data dimension, making it difficult for the network to understand. Therefore, scholars have made a lot of exploration in the field of fusion perception, trying to reveal the interaction mechanism between multi-modal data, so as to build a more reasonable and effective fusion architecture. So far, the mainstream feature-level and target-level fusion strategies have the problem of inefficient utilization of multi-modal complementary information, which seriously restricts the accuracy and reliability of detection and tracking tasks.

[0005] Feature-level data fusion strategy is widely used in fusion detection tasks. This strategy first uses the backbone to extract features from multi-modal data, and then fuses the obtained multi-modal feature maps. Its essence is multi-sensor data hard association fusion, that is, the splicing fusion of multi-modal features. However, this method also has limitations. Due to the heterogeneous characteristics of multi-modal data, the semantic information expressed by heterogeneous features at the same spatial position is not always consistent. Simple hard association fusion of multi-modal features not only introduces a lot of environmental noise, but also to some extent suppresses the potential information contained in a single modality.

[0006] While the target-level fusion strategy is widely used in tracking tasks. This strategy first associates the perception results of different sensors, and then uses Kalman filtering for state prediction. Although this fusion method is simple and intuitive, it can reduce the uncertainty of the tracked target through multi-path observation, but it inevitably causes a lot of information loss. On the one hand, this method ignores a lot of appearance features (color, texture, and shape features) provided by multi-modal features, significantly increasing the difficulty of data association; on the other hand, this method also does not establish an accurate kinematic model for the tracked target, also leading to inefficient use of spatial motion information.

[0007] SUMMARY

[0008] The present application aims to overcome the defects of the prior art, and proposes a high-performance loosely coupled multi-modal data fusion system and a vehicle-mounted device, which significantly improves the accuracy, robustness and adaptability of the environmental perception system in complex traffic scenarios and severe weather, and ensures the safe operation of intelligent networked vehicles in all working conditions.

[0009] In order to achieve the above purpose, the present application specifically proposes a modal independent feature interaction strategy and a motion-appearance feature cascade coupling data association strategy. Under the premise of fully considering the heterogeneous characteristics of multi-source information, the representation ability and utilization rate of the fused features are significantly improved, thereby effectively taking advantage of the complementary fusion of multi-modal data. The system construction mainly includes the following steps:

[0010] Step 1, select the VoD (View of Delft) dataset containing complex traffic scenarios and the K-Radar dataset containing various severe weather, to train, verify and test the comprehensive performance of the proposed method;

[0011] Step 2, construct a fusion detection model based on the modal independent feature interaction strategy;

[0012] Step 3, construct a fusion tracking model based on the motion-appearance feature cascade coupling data association strategy;

[0013] Step 4, apply TensorRT to accelerate the inference model and quantize it for deployment on a vehicle-mounted computing test platform.

[0014] Specifically, the VoD dataset needs to be divided into a training set, a validation set and a test set according to traffic scenes (campus, suburb, urban area and viaduct, etc.); the K-Radar dataset needs to be divided into a training set, a validation set and a test set according to weather conditions (cloudy, rainy, foggy and snowstorm, etc.); the dataset reference system is uniformly set as a vehicle body coordinate system, and the coordinate origin is set as the installation position of the millimeter wave radar; a global scaling data enhancement method is adopted.

[0015] The fusion detection model based on the modal independent feature interaction strategy is composed of a multi-modal feature extraction and bird's eye view feature generation module, a modal independent target query initialization module and a multi-modal feature fusion module based on a deformable Transformer; the model is constructed based on a DETR architecture, and is supervised and trained by adopting a bipartite graph optimal matching loss.

[0016] The fusion tracking model based on the motion-appearance feature cascade coupling data association strategy is composed of a continuous frame multi-modal appearance feature generation module, a first-level data association module based on multi-class multi-model state prediction, a second-level data association module based on multi-modal time memory appearance features and a track management module; the model adopts a TBD architecture, and does not need to set an additional appearance feature extractor.

[0017] The model quantization link includes weight pruning and model distillation; the model deployment is realized based on the nodes and communication functions provided by a ROS system.

[0018] The beneficial effects of the present application are as follows:

[0019] The high-performance loosely coupled multi-modal data fusion architecture proposed in the present application can effectively improve the accuracy and reliability of the intelligent driving environment perception system, and is compatible with almost all mainstream sensor deployment schemes;

[0020] The modal independent feature interaction strategy proposed in the present application can give full play to the multi-observation advantage on the premise of retaining the potential information of a single mode, realize efficient complementary fusion of multi-source heterogeneous information, and significantly improve the performance of the fusion detection algorithm;

[0021] The motion-appearance feature cascade coupling data association strategy proposed in the present application can comprehensively consider spatial motion information and multi-modal appearance features, improve the integrity and success rate of data association, and significantly improve the performance of the fusion tracking algorithm;

[0022] 4、The multi-sensor fusion perception system proposed in the present application can effectively cope with complex traffic scenes and extreme working conditions such as severe weather, and guarantee the safe operation of intelligent networked vehicles. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is a high-performance loosely coupled multi-modal data fusion architecture diagram for an intelligent driving environment perception system.

[0024] Figure 2 is a fusion detection algorithm flow chart based on a modal independent feature interaction strategy.

[0025] Figure 3 is a modal independent target query initialization schematic diagram.

[0026] Figure 4 is an encoder and decoder schematic diagram based on multi-modal deformable attention.

[0027] Figure 5 is a fusion tracking algorithm flow chart based on a motion-appearance feature cascade coupling data association strategy.

[0028] Figure 6 is a secondary nonlinear kinematic model schematic diagram.

[0029] Figure 7 is a secondary data association module schematic diagram based on multi-modal time memory appearance features. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution of the present application more clear and understandable, the specific embodiments of the present application are further described in detail in combination with the drawings of the specification.

[0031] The high-performance loosely coupled multi-modal data fusion architecture of the present application for an intelligent driving environment perception system is shown in Figure 1, which is composed of a fusion detection model and a fusion tracking model in cascade. The present application can be compatible with multi-source heterogeneous information provided by mainstream vehicle-mounted sensors such as laser radar, visible light camera and millimeter wave radar, and output accurate and reliable three-dimensional target detection and tracking results. The implementation of the high-performance loosely coupled multi-modal data fusion system of the present application for an intelligent driving environment perception system mainly includes the following steps:

[0032] Step 1, constructing a fusion detection model based on a modal independent feature interaction strategy

[0033] The overall architecture of the fusion detection model based on the modal independent feature interaction strategy proposed by the present application is shown in Figure 2, which is composed of a multi-modal feature extraction and bird's eye view feature generation network, a modal independent target query initialization model and a multi-modal feature fusion model based on deformable Transformer.

[0034] First, a multi-modal feature extraction and bird's eye view feature generation network is constructed. Different sensors represent environmental information in different forms, laser radar outputs high-resolution three-dimensional point cloud information, camera describes the environment through the foreground image, and millimeter wave radar usually uses continuous wave frequency signals. In order to avoid the semantic difference caused by heterogeneous sensor information and unify the representation form of environmental information, the present application unifies the multi-modal features to the shared bird's eye view space. In addition, in order to ensure the universality and compatibility of the overall architecture, the multi-modal information remains independent in the feature extraction process.

[0035] Specifically, the voxelization method is applied to process the lidar point cloud. Firstly, the original point cloud is dynamically voxelized, and then three-dimensional sparse convolution is used for step-by-step feature extraction to obtain three-dimensional voxel features. Subsequently, the non-empty voxel features are dimensionally compressed in the height direction. Finally, two-dimensional convolution is further used for feature extraction, and the dense point cloud bird's eye view feature F lid is obtained. The above process can be represented as:

[0036] In the formula, represents the cascade operation, PC lid is the original lidar point cloud; Voxel, SPConv, Aggn and Conv represent dynamic voxelization, three-dimensional sparse convolution, feature aggregation and two-dimensional convolution, respectively.

[0037] Due to the lack of long-distance depth information, it is not a simple task to convert the camera foreground image into a bird's eye view. At present, most methods try to use the powerful dynamic coding ability of the Transformer to learn the mapping relationship from the foreground view to the bird's eye view, so as to construct a relatively accurate bird's eye view feature. However, these dense prediction methods often bring heavy burden to the network. However, in the multi-modal method, the network does not only make inferences according to single modal information, which means that a certain amount of information loss is acceptable. Therefore, the present application constructs a lightweight image bird's eye view feature generation network to improve the real-time performance. Firstly, VoVNet is used to extract image features to obtain multi-scale image features. Subsequently, the Lift network is used to predict the discrete depth distribution of the features, and the discrete frustum features are obtained. Finally, the prism pooling is used to compress the height information, and the image bird's eye view feature F cam is obtained. The above process can be represented as:

[0038] In the formula, Img is a multi-view visible light camera image; VoVNet, DepthPred and FrustPool represent VoVNet-57 feature extraction backbone, Lift depth prediction network and prism pooling kernel, respectively.

[0039] Since the discrete depth estimation can only predict rough depth information, the image bird's eye view feature here is only used to implicitly reveal the existence of the target, and the remaining modal information will be further supplemented subsequently.

[0040] With the maturity of 4D imaging radar technology, it is also increasingly applied to the field of environmental perception. Compared with traditional millimeter wave radar, 4D imaging radar can not only provide height information, but also significantly improve the angular resolution. After signal processing, the confidence of the millimeter wave radar point cloud is higher and more dense. Therefore, the feature extraction method of laser radar point cloud can also be used to process 4D radar point cloud. However, the density of 4D radar point cloud is still far lower than that of laser radar point cloud. In order to capture the interaction between sparse millimeter wave radar point clouds as much as possible, the invention introduces a neighborhood Transformer as the backbone network. First, the original millimeter wave radar point cloud is columnized to convert it into a two-dimensional pseudo image. Second, the neighborhood Transformer is used for feature extraction to obtain dense radar features. Finally, the ResNet is used to adjust the feature scale to keep the spatial semantics consistent with the laser radar features, and finally obtain the millimeter wave radar bird's eye view feature F rad . The above process can be represented as:

[0041] In the formula, PC rad is the millimeter wave radar point cloud. Pillarize, NAT and ResNet represent columnization, neighborhood Transformer feature extraction backbone and residual backbone network, respectively.

[0042] Further, a modal independent target query initialization model is established. The original DETR architecture optimizes the randomly initialized target query through multiple Transformer decoder layers. Subsequent studies have shown that an initial target query containing dense prior information can effectively reduce the optimization difficulty and improve the final performance of the detector. Therefore, most of the current mainstream fusion detection methods use a simple and intuitive strategy, that is, the high response area in the multi-modal feature map is used as the initial target query. However, this strategy is not always reasonable. When the semantic information of each modal feature is inconsistent, semantic conflict will occur, which will cause the potential information existing in a certain modal to be submerged in a large amount of environmental noise. Unlike the above strategy, the invention regards all the potential target information provided by the modal features as candidate regions, and the specific implementation method is shown in FIG. 3. First, the multi-modal bird's eye view features obtained previously will be sent into the possession prediction network in parallel to obtain multi-modal heat maps. The multi-modal bird's eye view features and heat maps are only stacked in form, but remain independent of each other in the processing process. Second, by setting a threshold, the high response area with a confidence higher than the threshold in the heat map is screened out as a candidate region. When constructing the target query embedding, it is further divided into position embedding and semantic embedding. The multi-modal candidate region itself contains the spatial position information of the potential target, and the invention designs a simple multi-layer perceptron structure to encode it into position embedding. The above process can be represented as:

[0043] where F lid , F cam and F rad are the laser point cloud aerial view feature, the multi-view camera image aerial view feature and the millimeter wave radar aerial view feature, respectively.

[0044] The construction of semantic embedding is more skillful. Specifically, the modality where the candidate region is located is defined as the main modality, and the other two modalities are defined as the auxiliary modalities. First, according to the coordinates of the candidate region, the corresponding multi-modal aerial view features are indexed. Second, two attention mechanisms are used to process the auxiliary modality features in parallel. Among them, the main modality feature is used as the target query, and the auxiliary modality feature is used as the key and value. Through the global matching of the target query and the key, the potential effective information in the auxiliary modality is filtered out. Finally, after stacking the main feature and the processed auxiliary feature, dimension reduction mapping is performed, and the semantic embedding is obtained.

[0045] The position embedding and the semantic embedding are summed up to obtain the initial target query embedding QE.

[0046] The above process can be represented as: QE = CE + PE (7)

[0047] where Q m is the main modality target query, K s1 , K s2 , V s1 and V s2 are the keys and values of the other two auxiliary modalities; F m is the main modality feature, F s1 and F s2 are the auxiliary modality features obtained; softmax is a multi-classification exponential function; CE, PE and QE are semantic embedding, position embedding and target query embedding, respectively; FFN represents a forward propagation neural network, and d represents the feature dimension of the embedding vector.

[0048] Due to the consistency of multi-path observation results in close-range scenes, the target query will have partial redundancy and overlap. On the one hand, the target query should be more than the maximum number of targets in the scene, and a certain number of non-target negative samples should be retained. On the other hand, the overlapping target query can also alleviate the problem of missed detection in highly occluded scenes to some extent. It cannot be denied that the modality independent method proposed in the present application retains as much potential target information as possible provided by multi-path observation.

[0049] Finally, a multi-modal feature fusion model based on multi-modal deformable Transformer is established. In the decoding process, complete environmental features need to be provided to further optimize the initial target query. For this purpose, most methods try to obtain comprehensive environmental feature expression through multi-modal feature fusion. However, in order to make the most of the potential information in all modal features, the present application does not substantially fuse multi-modal features in the modal independent target query initialization. Therefore, a naive idea is to directly stack and fuse multi-modal feature maps. In fact, this is a simple and rough approach. On the one hand, the high-dimensional heterogeneous features after fusion may interfere with each other, making it difficult for the network to understand; on the other hand, the stacking operation does not perform data screening, and the multi-modal features after fusion are often too redundant, causing a huge burden to the network. Therefore, the present application constructs a lightweight multi-modal Transformer encoder based on deformable attention to realize efficient feature fusion, and the specific structure is shown in FIG. 4. First, the multi-modal bird's eye view features obtained before stacking are stacked. Second, the encoder proposed in the present application is used to further screen and integrate the multi-modal bird's eye view features, so as to obtain lightweight multi-modal fusion features.

[0050] Further, the detailed structure of the encoder proposed in the present application is shown in the Encoder part of FIG. 4. The present application generalizes deformable attention to the multi-modal field and designs multi-modal deformable attention. The present application adds modal dimension on the basis of deformable attention. That is, the sampling features of all modalities are adaptively weighted and summed, so as to aggregate multi-modal features. The multi-modal deformable attention can be expressed as:

[0051] In the formula, p q and F com are the normalized sampling point coordinates and the multi-modal stacked bird's eye view features, respectively; W i and W' represent the multi-branch attention normalization weight and the normalized sampling point weight, respectively; i, m and k represent the multi-branch attention serial number, the modal serial number and the sampling point serial number; N, M and K represent the total number of multi-branch attention branches, the total number of modalities and the total number of sampling points, respectively. Δp imqk and A imqk represent the sampling point offset and the bird's eye view feature weight corresponding to the serial number combination, respectively, and the weight needs to be globally normalized.

[0052] The design of the decoder follows the original Transformer architecture, and the detailed structure is shown in the Decoder part of Figure 4. The core modules are self-attention and cross-attention. The self-attention is only used for the interaction between target queries, and the whole process does not involve the fusion of multi-modal information in essence. The application sets it as the original self-attention mechanism. The cross-attention is used for the interaction between the target query and the multi-modal fusion feature, and its performance directly affects the utilization efficiency of the target query to the multi-modal fusion feature. The application sets it as the original deformable attention, which is convenient for the target query to capture effective information in the fusion feature. The whole process can be represented as:

[0053] In the formula, z q and F fus respectively represent the initialized target query feature and the obtained lightweight bird's eye view feature; A iqk and Δp iqk respectively represent the multi-modal bird's eye view feature weight and the sampling point offset corresponding to the combination of the serial numbers.

[0054] The decoder outputs the optimized target query, and predicts the final confidence and three-dimensional bounding box information through forward propagation.

[0055] Step 2, construct a fusion tracking model based on motion-appearance feature cascade coupling data association strategy

[0056] The fusion tracking model based on the motion-appearance feature cascade coupling data association strategy proposed by the application has the overall structure as shown in Figure 5, which is composed of a continuous frame multi-modal appearance feature generation module, a first-level data association module based on multi-class multi-model state prediction, a second-level data association module based on multi-modal time memory appearance feature, and a track management module. The fusion tracking model receives the continuous frame information provided by the detection model, so as to perform subsequent track tracking. The tracking model and the detection model share the multi-modal feature extraction and bird's eye view feature generation module, that is, the continuous frame multi-modal appearance feature generation module is the same as the multi-modal feature extraction and bird's eye view feature generation module.

[0057] Firstly, a primary data association model based on multi-class multi-model state prediction is established. There are clear physical and rule constraints in the real world, so the motion state of the target in the world coordinate system will not change abruptly and can be observed and predicted. Most 3D target tracking methods use a linear motion model to predict the motion state of the target in the future frame. However, in a real traffic scene, the motion of most traffic participants is highly nonlinear. Obviously, a simple linear model cannot accurately reflect the kinematic characteristics of the tracked target. Even if the multi-path observation model can provide complete kinematic parameters in the current frame, it is difficult to accurately predict the future state according to the incomplete motion model, thereby leading to inefficient use of spatial motion information. Therefore, the present application introduces multiple quadratic nonlinear motion models to establish accurate kinematic models for different categories of traffic participants.

[0058] Specifically, according to the statistical distribution law of the motion characteristics of the traffic participants, the quadratic motion model with the highest fitting degree is selected. As shown in FIG. 6, the present application introduces a constant turn rate and velocity model (CTRV), a constant turn rate and acceleration model (CTRA), and a kinematic bicycle model. Among them, the CTRV model assumes that the target moves along a straight line while also moving at a fixed turn rate and constant speed. The CTRV model considers that the speed direction and the heading angle of the target are always consistent, which can more comprehensively reflect the kinematic characteristics of pedestrians, as shown in FIG. 6(a). The state transition equation of the pedestrian track can be expressed as:

[0059] In the formula, CTRV X represents the state parameters of the pedestrian kinematic model, θ, v and ω respectively represent the vehicle heading angle, vehicle speed and angular velocity, T k is the time interval, and subscript k represents the k time, k+1 represents the k+1 time.

[0060] The CTRA model is further based on the CTRV model, which assumes that the target moves at a fixed turn rate and constant acceleration. Furthermore, the CTRA model considers that the acceleration direction, speed direction and heading angle of the target are always consistent. Since the CTRA model introduces the acceleration variable, its kinematic characteristics are more consistent with vehicle targets such as cars, trucks, etc., as shown in FIG. 6(b). The state variables and state transition equation of the vehicle target track can be expressed as:

[0061] In the formula, CTRA X represents the state parameters of the vehicle kinematic model, g x(x k ,T k ) and g y (x k ,T k ) are intermediate variables, and a represents the target acceleration.

[0062] The kinematic bicycle model has more degrees of freedom, and it is considered that the target speed direction and the heading angle are not always consistent, and the front wheel steering angle and the center of mass side slip angle are additionally introduced to describe the motion state of the target. In order to reduce the complexity of the model, it is assumed that the speed of the target and the front wheel steering angle remain unchanged. Since the bicycle model considers the structural rigidity of the target, there is an obvious coupling relationship between the kinematic parameters, which can better reflect the highly nonlinear kinematic characteristics of the rider, as shown in Fig. 6(c). The state variables and state transition equations of the rider category track can be expressed as:

[0063] In the formula, BIC X represents the state parameters of the kinematic model of the rider, β and δ are the center of mass side slip angle and the front wheel steering angle of the vehicle. l f and l r are the distances from the front wheel and the rear wheel of the vehicle to the center of mass of the vehicle.

[0064] Both the secondary motion models are nonlinear, and the Kalman filter is no longer applicable. Therefore, the present application further introduces the unscented Kalman filter to predict the future state of the track. The prediction process can be expressed as: X k+1 =f(X k ,ν k ), Z k+1 =h(X k+1 )+w k+1 (15) X k+1|k+1 =X k+1|k +K k+1|k (Z k+1 -Z k+1|k ) (16)

[0065] In the formula, f(·) and h(·) represent the state transition equation and the state prediction equation respectively, ν i and w i represent Gaussian noise. X i and Z i are motion state variables and observation state variables respectively, i=k or k+1, representing the time, K k+1|k is the Kalman gain, X k+1|k+1 , X k+1|k and Z k+1|k represent the unscented Kalman prediction value, the motion model prediction value and the observation model observation value of the track respectively.

[0066] Kinematic characteristics of multi-class traffic participants are different, and the adaptability of cost function and associated threshold is also different. Therefore, the application sets up a simple class filter to associate the multi-class targets and tracks in the same class. Further, the application adopts the G-IoU BEV As a cost function, the similarity of the motion state is evaluated uniformly. G-IoU BEV Can be expressed as: G-IoU BEV (B1,B2)=S I / S U -(S C -S U ) / S C (18)

[0067] In the formula, S U , S I , S C respectively represent the area of the union set of the bounding box of the current frame detection result and the track prediction result, the area of the bounding box of the current frame detection result, the area of the bounding box of the track prediction result, the area of the intersection of the bounding box of the current frame detection result and the track prediction result, and the area of the enclosing convex polygon of the union set of the bounding box of the current frame detection result and the track prediction result.

[0068] Further, the application sets different thresholds for multi-class data association to adapt to the kinematic characteristics of traffic participants. After the first-level data association is completed, a small amount of remaining unmatched detection and tracks will be sent to the second-level data association for further matching.

[0069] Secondly, a second-level data association model based on multi-modal time memory appearance features is established. In fact, in a complex traffic scene, the kinematic characteristics of some traffic participants may not be significant. According to the social force model, the target will be affected by the social field such as road environment, traffic rules and mutual interference to change its motion state. In this case, the motion state of the target reflects more sociality rather than kinematic characteristics. In other words, the motion model established before is no longer applicable, and only relying on spatial motion information cannot accurately predict the future state of the target. In addition, the social force model focuses on the description of group motion, and it is difficult to quantitatively describe the motion state of individual micro-objects. Therefore, the application additionally introduces multi-modal appearance features as the basis for second-level data association, further associating the remaining unmatched detection and tracks.

[0070] Specifically, the application constructs a multi-modal time memory appearance feature aggregation network for encoding time sequence features, and then realizes data association through cross-modal attention. The detailed structure of the two-level data association network based on multi-modal time memory appearance features is shown in FIG. 7. First, a time memory buffer is set to store the appearance features of all tracks within a period of time. In order to reduce the data storage pressure, the appearance features in the buffer comply with the first-in-first-out principle. Second, for the tracks that have not been matched in the first-level data association module, their long short-term memory aggregation features are further calculated. Several frames (including the current frame) close to the current frame are defined as short-term memory, and all frames stored in the buffer are defined as long-term memory. Then, the short-term and long-term time sequence appearance features are encoded through short-term cross-modal attention and long-term cross-modal attention modules, respectively. Among them, the current frame feature is taken as the query, and the short-term memory and long-term memory are taken as the key and value. Finally, the mean value of the encoded short-term and long-term time sequence features is taken, that is, the final long short-term memory aggregation feature is obtained. The encoding and aggregation process can be represented as: App agr =Mean(App s +App l ) (21)

[0071] In the formula, App s , App l and App agr are short-term memory appearance features, long-term memory appearance features and long short-term memory aggregation appearance features, respectively; Q t-1 , and represent historical frame query features, short-term memory query features and long-term memory query features, respectively; Attn and Mean represent attention and arithmetic mean operation, respectively.

[0072] The appearance features of the target will change with the real-time changes of the observation angle and the scene, which are difficult to predict. However, their color or structural features often remain consistent as a whole. Therefore, the application sets long short-term attention to capture time sequence change information as much as possible, and then completes aggregation to obtain comprehensive appearance features. Finally, cross-modal attention is applied to match the appearance features of the detection and the track, evaluate the similarity between them, and perform data association according to the threshold.

[0073] After the completion of the secondary data association, the final matching results are sent to the track management module. For the successfully matched tracks, the current frame detection is used as the posteriori update of the track state information according to the lossless Kalman filter. For the detections that have not completed the matching, track initialization is performed after the minimum number of hits is reached. For the unmatched tracks, the prediction results of the secondary motion model are used to update the track state information, and track extinction processing is performed after the maximum life cycle is reached.

[0074] In summary, the motion-appearance feature cascade coupling data association strategy objectively divides the tracks into two categories of significant kinematic characteristics and social significance, and uses spatial state information and multi-modal appearance features as the basis for data association, respectively. The present application sets a higher threshold for the first level data association based on multi-class multi-model state prediction (the threshold range is [-1, 1], and the higher value is generally set at -0.4 or -0.3). While avoiding false matches caused by low-precision motion state prediction, it tries to filter out socially significant targets as much as possible, so as to further complete the matching in the secondary data association based on multi-modal temporal memory appearance features. The two matching processes complement each other, and the efficient use of spatial motion information and multi-modal appearance features is achieved as much as possible.

[0075] The above series of detailed descriptions are only specific descriptions of the feasible embodiments of the present application, and are not intended to limit the protection scope of the present application. Any equivalent means or changes that do not deviate from the technology of the present application shall be included in the protection scope of the present application.

Claims

1. A high-performance loosely coupled multi-modal data fusion system for intelligent driving environment perception system, characterized in that, Comprise: A fusion detection model based on modal independent feature interaction strategy, which converts lidar point cloud, camera image and millimeter wave radar point cloud into unified bird's eye view features and performs multi-modal fusion; A fusion tracking model based on motion-appearance feature cascade coupling data association strategy, which performs subsequent trajectory tracking and matching according to the multi-modal fusion feature information provided by the fusion detection model.

2. The high-performance loosely coupled multi-modal data fusion system for intelligent driving environment perception system according to claim 1, characterized in that, The fusion detection model based on modal independent feature interaction strategy comprises a multi-modal feature extraction and bird's eye view feature generation module, a modal independent target query initialization module and a multi-modal feature fusion module based on deformable Transformer; The model is constructed based on the DETR architecture and supervised training is performed using the bipartite graph optimal matching loss.

3. The high-performance loosely coupled multi-modal data fusion system for intelligent driving environment perception system according to claim 2, wherein, The multi-modal feature extraction and bird's eye view feature generation module converts multi-modal features into shared bird's eye view space, and multi-modal information remains independent during feature extraction; Specifically as follows: The voxelization method is used to process the lidar point cloud: first, the original point cloud is dynamically voxelized, then three-dimensional sparse convolution is used for step-by-step feature extraction to obtain three-dimensional voxel features, then non-empty voxel features are dimensionally compressed in the height direction, and finally two-dimensional convolution is used for feature extraction to obtain dense point cloud bird's eye view features; The above process can be represented as: In the formulae, denotes a concatenation operation, PC lid denotes a concatenation operation, PC lid denotes a concatenation operation, PC lid denotes a concatenation operation, PC lid denotes a concatenation operation, PC lid denotes a concatenation operation, PC lid denotes a concatenation operation, PC lid denotes a concatenation operation, PC lid denotes a concatenation operation, PC The camera foreground image is converted into a bird's eye view: first, VoVNet is used to extract features from the image to obtain multi-scale image features; then, the Lift network is used to predict the discrete depth distribution of the features and convert it into discrete frustum features; finally, the frustum pooling is used to compress the height information to obtain the image bird's eye view features; The above process can be represented as: In the formula, Img is a multi-view visible light camera image; VoVNet, DepthPred and FrustPool represent VoVNet-57 feature extraction backbone, Lift depth prediction network and frustum pooling kernel respectively; 4D radar point cloud is converted into a bird's eye view: first, the original radar point cloud is columnarized to convert it into a two-dimensional pseudo image; second, the neighborhood Transformer is used for feature extraction to obtain dense radar features; finally, ResNet Adjusting the feature scale to keep spatial semantic consistency with the lidar features; the above process can be represented as: where PC rad is the millimeter-wave radar point cloud, Pillarize, NAT, and ResNet represent the pillarization, neighborhood Transformer feature extraction backbone, and residual backbone network, respectively.

4. The high-performance loosely coupled multi-modal data fusion system for intelligent driving environment perception system according to claim 2, wherein, The modal independent target query initialization module: first, the multi-modal bird's eye view feature obtained by the multi-modal feature extraction and bird's eye view feature generation module is sent into the occupancy prediction network in parallel to obtain a multi-modal heat map, and the multi-modal bird's eye view feature and the heat map are only stacked in form and remain independent in the processing process; second, through threshold setting, the high response area with confidence higher than the threshold in the heat map is screened out as a candidate area, and in the construction of the target query embedding, the candidate area is divided into a position embedding and a semantic embedding, and the multi-modal candidate area itself contains the spatial position information of the potential target, which is encoded into the position embedding through a multi-layer perception machine; the above process can be represented as: where F lid , F cam , and F rad are laser point cloud aerial view features, multi-view camera image aerial view features, and millimeter wave radar aerial view features, respectively, and Concat, OccPred, and MLP represent concatenation operation, occupancy grid prediction network, and multi-layer perceptron, respectively. For semantic embedding, the modality in which the candidate region is located is defined as the main modality, and the other two modalities are defined as auxiliary modalities. First, the coordinates of the candidate region are indexed to correspond to the multi-modal bird's eye view features; second, two attention mechanisms are used to process the auxiliary modality features in parallel; among them, the main modality feature is used as the target query, and the auxiliary modality feature is used as the key and value; through the global matching of the target query and the key, the potential effective information in the auxiliary modality is screened out; finally, the main feature and the processed auxiliary feature are stacked and dimensionally mapped to obtain the semantic embedding; Summing the position embedding and the semantic embedding can obtain an initial target query embedding; the above process can be represented as: QE=CE+PE (7) where Q m is the main modality target query, K s1 , K s2 , V s1 and V s2 are the keys and values of the other two subsidiary modalities, respectively; F m is the main modality feature, F s1 and F s2 are the subsidiary modality features obtained; CE, PE and QE are semantic embedding, position embedding and target query embedding, respectively; and FFN represents a forward propagation neural network.

5. The high-performance loosely coupled multi-modal data fusion system for intelligent driving environment perception system according to claim 2, wherein, The multi-modal feature fusion module based on deformable Transformer: First, the multi-modal bird's eye view features obtained by the multi-modal feature extraction and bird's eye view feature generation module are stacked; second, an encoder is used to further screen and integrate the multi-modal bird's eye view features to obtain lightweight multi-modal fusion features; the encoder adopts a multi-modal deformable attention mechanism to adaptively weight and sum the sampling features of all modes to aggregate the multi-modal features; the multi-modal deformable attention mechanism is represented as: where p q and F com are the normalized sample point coordinates and the multi-modal stacked bird's eye view features, respectively; W i and W' represent multi-branch attention normalization weights and normalized sampling point weights, respectively; i, m and k represent multi-branch attention sequence number, modality sequence number and sampling point sequence number; N, M and K represent the total number of multi-branch attention branches, the total number of modalities and the total number of sampling points, respectively, Δp imqk and A imqk represent the sampling point offset and the aerial view feature weight corresponding to the combination of the sequence numbers, respectively, and the weight needs to be globally normalized.

6. The high-performance loosely coupled multi-modal data fusion system for intelligent driving environment perception system according to claim 1, wherein, The fusion tracking model based on the motion-appearance feature cascade coupling data association strategy comprises a continuous frame multi-modal appearance feature generation module, a first-level data association module based on multi-class multi-model state prediction, a second-level data association module based on multi-modal time memory appearance features, and a track management module; the model adopts a TBD architecture and does not need to set an additional appearance feature extractor.

7. The high-performance loosely coupled multi-modal data fusion system for intelligent driving environment perception system according to claim 6, characterized in that The first-level data association module based on multi-class multi-model state prediction introduces a plurality of quadratic nonlinear motion models to establish accurate kinematic models for different categories of traffic participants, and is specifically as follows: According to the statistical distribution of the motion characteristics of traffic participants, the highest fitting degree of the secondary motion model is screened, the constant turning rate and speed model CTRV, the constant turning rate and acceleration model CTRA and the kinematic bicycle model are introduced; wherein, the CTRV assumes that the target moves along a straight line, and can also move at a fixed turning rate and constant speed, the speed direction of the target in the CTRV is always consistent with the heading angle, and the kinematic characteristics of the pedestrian can be more comprehensively reflected, and the state transition equation of the pedestrian track is expressed as: wherein CTRV X represents the state parameters of the pedestrian kinematic model; θ, v and ω are the vehicle heading angle, vehicle speed and angular velocity, respectively, T k is the time interval; The CTRA model is further based on the CTRV model, assuming that the target moves at a fixed turning rate and constant acceleration, and that the acceleration direction, velocity direction and heading angle of the target are always consistent. The kinematic characteristics of the target are more consistent with the automobile target, and the state variables and state transition equation of the automobile target track are represented as: wherein CTRA X represents the state parameters of the car kinematic model, g x (x k , T k ) and g y (x k , T k ) are intermediate variables, a represents the target acceleration; Kinematic bicycle model, the speed direction of the target and the heading angle are not always consistent, the front wheel steering angle and the mass center side slip angle are introduced to describe the motion state of the target, the speed of the target and the front wheel steering angle are assumed to be constant, since the bicycle model considers the structural rigidity of the target, there is an obvious coupling relationship between the kinematic parameters, which can better reflect the highly nonlinear kinematic characteristics of the rider, the state variables and the state transition equation of the rider category track are represented as: wherein BIC X represents the state parameters of the cyclist kinematic model, β and δ are the center of mass side slip angle and the front wheel steering angle of the vehicle, l f and l r are the distances from the front and rear wheels to the center of mass of the vehicle, respectively. The future state of the track is predicted by introducing an unscented Kalman filter, and the prediction process is represented as: X k+1 = f(X k , v k ), Z k+1 = h(X k+1 ) + w k+1 (14) X k+1|k+1 = X k+1|k + K k+1|k (Z k+1 - Z k+1|k ) (15) where f(·) and h(·) represent state transition equation and state prediction equation, respectively, v i and w i are Gaussian noises. X i and Z i are motion state variable and observation state variable, respectively, i = k or k + 1, represents time, K k+1|k is Kalman gain, X k+1|k+1 , X k+1|k and Z k+1|k are lossless Kalman prediction value, motion model prediction value and observation model observation value of the track, respectively; A category filter is set to associate multi-category targets and tracks within the category, and G-IoU with relatively comprehensive performance is adopted BEV As a cost function, the similarity of motion state is evaluated uniformly, G-IoU BEV is expressed as: G-IoU BEV (B1,B2) = S I / S U -(S C -S U ) / S C (17) In the formula, S U 、 S I , S C respectively represent the area of the union of the bounding boxes of the current frame detection result and the trajectory prediction result, the area of the bounding box of the current frame detection result, the area of the bounding box of the trajectory prediction result, the area of the intersection of the bounding boxes of the current frame detection result and the trajectory prediction result, and the area of the enclosing convex polygon of the union of the bounding boxes of the current frame detection result and the trajectory prediction result. Different thresholds are set for multi-class data association to adapt to the kinematic characteristics of traffic participants.

8. The high-performance loosely coupled multi-modal data fusion system for intelligent driving environment perception system according to claim 6, wherein, The secondary data association module based on the multi-modal time memory appearance feature: first, a time memory buffer is set to store appearance features of all tracks within a period of time; second, for the tracks that are not matched in the primary data association module, long short-term memory aggregated features are calculated, and several frames (including the current frame) adjacent to the current frame are defined as short-term memory, and all frames stored in the buffer are defined as long-term memory; then, the short-term and long-term time sequence appearance features are encoded through short-term cross-modal attention and long-term cross-modal attention, respectively, wherein the current frame feature is taken as a query, and the short-term memory and the long-term memory are taken as a key and a value; then, the mean value of the encoded short-term and long-term time sequence appearance features is taken, and the final long short-term memory aggregated features are obtained; the encoding and aggregation process can be represented as: App agr =Mean(App s +App l ) (20) In the formula, App s , App l , and App agr are short-term memory appearance features, long-term memory appearance features, and long-short-term memory aggregated appearance features, respectively; Q t-1 , and respectively represent historical frame query features, short-time memory query features and long-time memory query features; Attn and Mean represent attention and arithmetic mean operations respectively; Finally, cross-modal attention matching detection and track appearance features are applied to evaluate the similarity between them, and data association is performed according to the threshold.

9. The high-performance loosely coupled multi-modal data fusion system for intelligent driving environment perception system according to claim 6, wherein, The track management module: for the successfully matched track, the current frame detection is used as the posterior update track state information according to the unscented Kalman filter; for the detection that has not completed matching, track initialization is performed after the minimum hit number is reached; for the track that has not completed matching, the prediction result of the quadratic motion model is used to update the track state information, and track extinction processing is performed after the maximum life cycle is reached.

10. An intelligent driving vehicle device, characterized by comprising: The vehicle-mounted device internally deploys the fusion detection model based on the modal independent feature interaction strategy and the fusion tracking model based on the motion-appearance feature cascade coupling data association strategy of claim 1.

Images