Road dynamic obstacle millimeter wave-visual fusion perception method and system

Abstract

The invention provides a road dynamic obstacle millimeter wave-visual fusion perception method and system, and belongs to the technical field of intelligent traffic and automatic driving, and the method comprises the steps: obtaining the initial features of an obstacle through the fusion of a millimeter wave radar and a visual sensor, constructing a multi-dimensional time-space representation which comprises a motion mode, a multi-scale time sequence feature and a state transition model, and carrying out the real-time detection. Then constructing a dynamic sparse graph structure, fusing heterogeneous data and carrying out feature propagation, combining phase space mapping and physical constraints to realize nonlinear trajectory prediction, generating a plurality of candidate trajectories for integration, and finally outputting a prediction result and alarm information to improve path planning safety; a topological structure expression of obstacle movement is established, a nonlinear and non-stable complex movement mode can be effectively captured, and the recognition capability of sudden behaviors is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation and autonomous driving technology, and in particular to a millimeter wave-vision fusion perception method and system for dynamic road obstacles, which are applied to dynamic obstacle detection, tracking and trajectory prediction in road traffic environments. Background Art

[0002] With the rapid development of intelligent transportation and autonomous driving technologies, accurate perception of dynamic obstacles in road environments has become a critical technology for ensuring traffic safety. Existing single-sensor perception methods have significant limitations. For example, millimeter-wave radars have limited target classification capabilities in complex environments, while visual sensors are susceptible to lighting and weather conditions. While multi-sensor fusion technology has been applied to some extent, existing fusion methods often rely on simple data-level or feature-level fusion, which fails to fully utilize the complementary properties of different sensors. This is particularly problematic when dealing with nonlinear motion patterns and multi-target interaction scenarios.

[0003] Existing trajectory prediction technologies are mostly based on linear or simple nonlinear models, such as Kalman filters and extended Kalman filters. These methods struggle to accurately predict complex movements, such as sudden pedestrian turns and rapid lane changes. Furthermore, existing methods fail to adequately consider interactions between obstacles, making it difficult to understand and predict group behavior patterns, such as vehicle following and pedestrian avoidance.

[0004] On the other hand, existing graph neural networks, when applied to multi-target trajectory prediction, often use dense graph structures, which are computationally intensive and contain a large amount of redundant information, impacting both the system's real-time performance and prediction accuracy. As the number of targets in a scene increases, the computational complexity increases quadratically, making it difficult to meet real-time processing requirements. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a millimeter wave-vision fusion perception method and system for dynamic road obstacles. This method integrates millimeter wave radar and vision sensor data, combines multi-dimensional spatiotemporal feature representation, dynamic sparse graph convolutional network and nonlinear trajectory prediction technology, and realizes high-precision detection, tracking and trajectory prediction of dynamic road obstacles.

[0006] The present invention discloses a millimeter wave-vision fusion perception method for dynamic road obstacles, comprising:

[0007] Collect road scene data through millimeter-wave radar and visual sensors to obtain initial feature information of obstacles;

[0008] Based on the initial feature information of the obstacle, a multi-dimensional spatiotemporal feature representation is constructed, including:

[0009] Map the obstacle's historical trajectory to the topological structure space to form the obstacle's motion pattern representation;

[0010] Extract obstacle motion features in multiple time scale windows to generate multi-scale temporal features;

[0011] Establish an obstacle state transition probability model to characterize the obstacle state evolution law;

[0012] Based on the multi-dimensional spatiotemporal feature representation, an obstacle relationship graph structure is constructed and a dynamic sparse graph convolutional network is applied for feature propagation, including:

[0013] Perform heterogeneous information fusion of millimeter-wave radar data and visual data to generate a unified feature representation;

[0014] A dynamic sparse graph structure is constructed with obstacles as nodes, and dynamic pruning of the graph structure is achieved through radar signal-to-noise ratio thresholds, visual confidence thresholds, and Top-K edge sampling mechanisms.

[0015] By combining graph convolution operations with attention mechanisms to propagate and aggregate features, key feature representations are enhanced.

[0016] Based on the feature representation output by the graph convolutional network, nonlinear trajectory prediction is performed, including:

[0017] Map the obstacle's historical trajectory into phase space and identify the motion pattern;

[0018] Introducing physical constraints to ensure the rationality of the predicted trajectory;

[0019] Generate multiple candidate trajectories and integrate them to form the final prediction result;

[0020] The prediction results are verified and evaluated to generate obstacle trajectory prediction information and alarm prompts.

[0021] Preferably, collecting road scene data by using millimeter wave radar and visual sensor to obtain initial obstacle feature information specifically includes:

[0022] The millimeter-wave radar collects the reflection intensity, reflection surface angle and emission angle of the obstacle point at a polling frequency of 10kHz;

[0023] The system collects 4K resolution images through visual sensors and uses object detection algorithms to extract the location, size, and category information of obstacles.

[0024] Converting the obstacle position information obtained by the visual sensor from a pixel coordinate system to a millimeter-wave radar coordinate system to achieve coordinate unification;

[0025] The speed information of the obstacle is calculated based on the reflection intensity, reflection surface angle and emission angle.

[0026] Preferably, mapping the obstacle's historical trajectory into a topological structure space to form an obstacle motion pattern representation specifically includes:

[0027] Construct an n-dimensional obstacle state vector containing position, velocity, acceleration, and size;

[0028] Using obstacles as nodes and spatiotemporal relationships as edges, a dynamic topological structure is constructed;

[0029] The original features are mapped to high-dimensional manifold space through nonlinear transformation to enhance the ability to express nonlinear motion patterns.

[0030] Preferably, the step of extracting obstacle motion features in multiple time scale windows to generate multi-scale temporal features specifically includes:

[0031] The obstacle historical trajectory data is divided into three time windows: short-term (0.1-0.5 seconds), medium-term (0.5-2 seconds) and long-term (2-5 seconds);

[0032] Design dedicated feature extractors for different time windows to capture microscopic motion details and macroscopic motion trends respectively;

[0033] Features at different time scales are weighted and combined to form a multi-scale time series representation.

[0034] Preferably, the heterogeneous information fusion of millimeter-wave radar data and visual data to generate a unified feature representation specifically includes:

[0035] Scale normalization and dimension alignment of millimeter-wave radar features and visual features;

[0036] Project heterogeneous features into a unified feature space through nonlinear transformation;

[0037] The weights of millimeter-wave radar data and visual data are dynamically adjusted according to environmental conditions to strengthen complementary features and suppress redundant features.

[0038] Preferably, the dynamic sparse graph structure is constructed with obstacles as nodes, and dynamic pruning of the graph structure is achieved through radar signal-to-noise ratio threshold, visual confidence threshold and Top-K edge sampling mechanism, specifically including:

[0039] Calculate the signal-to-noise ratio (SNR) for each radar point / cluster node in each frame, and retain nodes whose SNR is greater than the preset threshold θ1;

[0040] Open cross-modal edge connections for nodes whose image detection box confidence p is greater than the preset threshold θ2;

[0041] For each node, only the K edges with the highest relationship scores are retained, so that the sparseness rate of the graph structure does not exceed 20%;

[0042] When each frame of data arrives, the node scores and edge weights are recalculated in real time, and the graph structure is dynamically updated.

[0043] Preferably, the graph convolution operation is combined with the attention mechanism to perform feature propagation and aggregation to enhance the key feature representation, specifically including:

[0044] Design multi-level feature aggregation operations to expand the receptive field layer by layer;

[0045] Calculate attention weights based on node relevance and importance;

[0046] Dynamically adjust feature propagation path and strength based on attention weights;

[0047] Capture relational features of different dimensions through a multi-head attention mechanism.

[0048] Preferably, mapping the obstacle's historical trajectory into phase space and identifying the motion pattern specifically includes:

[0049] Construct a high-dimensional phase space containing dimensions such as position, velocity, and acceleration;

[0050] Map the historical trajectory point sequence into the phase space to form a trajectory manifold;

[0051] Retrieve similar historical trajectory patterns based on phase space distance metrics;

[0052] Extract the evolution law of the trajectory in the phase space and identify typical motion patterns.

[0053] Preferably, the physical constraints are introduced to ensure the rationality of the predicted trajectory, specifically including:

[0054] Establish dynamic constraints including physical parameters such as maximum acceleration and steering angular rate;

[0055] Integrate geometric constraints such as road boundaries and lane lines;

[0056] Consider the interaction constraints between traffic rules and obstacles;

[0057] Filter out predicted trajectories that do not conform to physical laws through constrained optimization;

[0058] Verifying and evaluating the prediction results to generate obstacle trajectory prediction information and alarm prompts specifically includes:

[0059] Perform intersection-over-union (IOU) verification on the predicted trajectory and the real-time detection results;

[0060] When the IOU value is less than 0.75, further analyze the cause of trajectory deviation;

[0061] Identify trajectory pairs that may collide and calculate the collision risk index;

[0062] Generate different levels of alarm information according to risk level;

[0063] Risk information is displayed through a visual interface to assist decision-making.

[0064] The millimeter wave-vision fusion perception system for dynamic road obstacles includes:

[0065] The perception data acquisition module is used to collect road scene data through millimeter-wave radar and visual sensors to obtain initial feature information of obstacles;

[0066] A spatiotemporal feature representation module, configured to construct a multi-dimensional spatiotemporal feature representation based on the initial obstacle feature information, wherein the multi-dimensional spatiotemporal feature representation includes: a topological structure representation of the obstacle motion pattern, multi-scale temporal features, and an obstacle state transition probability model;

[0067] A graph convolutional network module, configured to construct an obstacle relationship graph structure and perform feature propagation based on the multi-dimensional spatiotemporal feature representation, wherein the graph convolutional network module comprises: a heterogeneous information fusion unit, a dynamic sparse graph construction unit, and an attention feature propagation unit;

[0068] A trajectory prediction module, configured to perform nonlinear trajectory prediction based on the feature representation output by the graph convolutional network module, the trajectory prediction module comprising: a phase space mapping unit, a physical constraint unit, and a probabilistic trajectory generation unit;

[0069] The verification and alarm module is used to verify and evaluate the prediction results of the trajectory prediction module and generate obstacle trajectory prediction information and alarm prompts.

[0070] The dynamic sparse graph construction unit dynamically prunes the graph structure by evaluating node reliability and edge importance in real time, including:

[0071] Calculate the radar node signal-to-noise ratio (SNR) and visual node detection confidence, and set corresponding thresholds for node filtering;

[0072] For each node, only the K edges with the highest relationship scores are retained to keep the graph structure sparsity rate within 20%;

[0073] The graph structure is updated in real time when each frame of data arrives, so that sparse graph convolution can keep the computation lightweight while capturing key spatiotemporal relationships.

[0074] The present invention has the following beneficial effects:

[0075] 1. Through multi-dimensional spatiotemporal feature representation technology, a topological structure representation of obstacle motion is established, which can effectively capture nonlinear and non-stationary complex motion patterns and improve the ability to identify sudden behaviors;

[0076] 2. Dynamic sparse graph convolutional network technology is used to dynamically prune the graph structure through radar signal-to-noise ratio thresholds, visual confidence thresholds, and a top-K edge sampling mechanism. This reduces the computational workload by over 50% while maintaining prediction accuracy, significantly improving the system's real-time performance.

[0077] 3. Dynamic sparse GNN based on SNR thresholds and confidence thresholds provides a real-time, data-driven pruning scale. This not only filters out false targets such as raindrops, but also ensures that the retained nodes and edges have high information value, making graph convolution both lightweight and accurate.

[0078] 4. The complementary advantages of millimeter wave and visual data enhance the system's perception stability in various environmental conditions, especially maintaining reliable performance in adverse weather and lighting conditions;

[0079] 5. The introduction of nonlinear trajectory prediction technology with physical constraints, combined with phase space mapping and probabilistic trajectory generation, significantly improves prediction accuracy and robustness, and is capable of predicting complex behaviors such as sudden turns by pedestrians and lane changes to overtake vehicles.

[0080] 6. The system has a multi-level anomaly detection and alarm mechanism that can promptly identify potential risks and issue warnings in advance, providing sufficient response time for subsequent decision-making and control;

[0081] 7. The overall system has strong real-time and scalability, and is suitable for various deployment modes such as vehicle-mounted, roadside and vehicle-road collaborative, effectively supporting intelligent transportation and autonomous driving applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 This is the overall architecture diagram of the millimeter wave-vision fusion perception system for dynamic road obstacles of the present invention;

[0083] Figure 2 This is a workflow diagram of the perception data acquisition module of the present invention;

[0084] Figure 3 This is a structural diagram of the multi-dimensional spatiotemporal feature representation module of the present invention;

[0085] Figure 4 This is a working principle diagram of the dynamic sparse graph convolutional network module of the present invention;

[0086] Figure 5 is a flow chart of the nonlinear trajectory prediction module of the present invention;

[0087] Figure 6This is a workflow diagram of the verification and alarm module of the present invention.

[0088] Figure 1: Perception data acquisition module 1, spatiotemporal feature representation module 2, graph convolutional network module 3, trajectory prediction module 4, verification and alarm module 5, millimeter wave radar perception unit 11, visual perception unit 12, coordinate conversion unit 13, topological structure mapping unit 21, multi-scale temporal feature unit 22, state transition model unit 23, heterogeneous information fusion unit 31, dynamic sparse graph construction unit 32, attention feature propagation unit 33, phase space mapping unit 41, physical constraint unit 42, probabilistic trajectory generation unit 43, trajectory verification unit 51, risk assessment unit 52, alarm generation unit 53. DETAILED DESCRIPTION

[0089] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0090] Reference Figure 1 The millimeter wave-vision fusion perception system for dynamic road obstacles provided by the present invention includes a perception data acquisition module 1, a spatiotemporal feature representation module 2, a graph convolutional network module 3, a trajectory prediction module 4, and a verification and alarm module 5. The data flow between these modules follows the standard process of collection-acquisition-processing-output, forming a complete perception closed loop.

[0091] The system can be deployed on an on-board platform, on a roadside unit, or in a vehicle-to-infrastructure (V2I) system. In an on-board deployment scenario, the millimeter-wave radar and camera are installed on the front of the vehicle; in a roadside deployment scenario, the millimeter-wave radar and 4K camera are installed on smart traffic light poles; in a V2I deployment scenario, data from both on-board and roadside sensors is shared through V2X communication.

[0092] The millimeter wave-vision fusion perception method for dynamic road obstacles and the implementation of the system thereof are described in detail below.

[0093] Reference Figure 2 The perception data acquisition module 1 is used to collect road scene data through millimeter wave radar and visual sensors to obtain initial feature information of obstacles. The specific implementation method is as follows:

[0094] The perception data acquisition module 1 includes a millimeter-wave radar perception unit 11, a visual perception unit 12, and a coordinate conversion unit 13. The millimeter-wave radar perception unit 11 uses a 77GHz millimeter-wave radar with a horizontal field of view of 120° and a vertical field of view of 20°. It collects obstacle point reflection intensity P1, reflection surface angle θ, and emission angle φ at a polling frequency of 10kHz. The millimeter-wave radar preferably has a lateral resolution D of 0.5 meters, which effectively distinguishes obstacles in adjacent lanes.

[0095] The visual perception unit 12 uses a 4K resolution camera with a frame rate of 60 fps and a 90° field of view to capture high-definition images. It uses the YOLO-v7 object detection algorithm to extract the location, size, and category of obstacles. The YOLO-v7 algorithm can identify common road obstacle types, including vehicles, pedestrians, and bicycles, and outputs the obstacle's bounding box information, including the center coordinates (m, a), width p, and height k.

[0096] The coordinate conversion unit 13 converts the obstacle position information obtained by the visual sensor from the pixel coordinate system to the millimeter wave radar coordinate system to achieve coordinate unification. The coordinate conversion uses the following formula:

[0097] .

[0098] .

[0099] in: is the horizontal coordinate of the obstacle in the millimeter-wave radar coordinate system, in meters; is the vertical coordinate of the obstacle in the millimeter-wave radar coordinate system, in meters; The horizontal coordinate of the obstacle bounding box center in the image pixel coordinate system, in pixels; The vertical coordinate of the obstacle bounding box center in the image pixel coordinate system, in pixels; is the projection of the camera's optical center onto the imaging plane along the column coordinate axis, in pixels; is the projection of the camera's optical center onto the imaging plane along the row coordinate axis, in pixels; and are the focal lengths of the camera in the x and y directions, respectively, in pixels; is the lateral resolution of the millimeter-wave radar, in meters.

[0100] In addition, based on the reflection intensity P1, reflection surface angle θ, and emission angle φ collected by the millimeter-wave radar, the speed V of the obstacle can be calculated:

[0101] .

[0102] in: is the obstacle speed in m / s; is the distance change in meters; is the time interval in seconds; is the phase change in radians; is the speed of light, which is 299792458 m / s; The operating frequency of the millimeter wave radar is set to 77GHz.

[0103] In practical applications, the time interval t is usually set to 0.1 second, which can provide a speed update frequency of 10 Hz to meet real-time perception requirements.

[0104] Reference Figure 3 The spatiotemporal feature representation module 2 is used to construct a multi-dimensional spatiotemporal feature representation based on the initial feature information of the obstacle. This module includes a topology mapping unit 21, a multi-scale temporal feature unit 22, and a state transition model unit 23.

[0105] The topology mapping unit 21 maps the obstacle's historical trajectory to the topology space to form an obstacle motion pattern representation. First, an n-dimensional obstacle state vector S containing position, velocity, acceleration, and size is constructed:

[0106] .

[0107] in: is the obstacle state vector; 、 is the coordinate of the obstacle center, in meters; 、 are the velocity components in the x and y directions, in m / s; 、 are the acceleration components in the x and y directions, in m / s^2; 、 are the width and height of the obstacle, respectively, in meters; Code for obstacle category; superscript Represents the transpose of a vector.

[0108] Preferably, the obstacle category One-hot encoding is used, for example, vehicles are encoded as [1, 0, 0], pedestrians are encoded as [0, 1, 0], and bicycles are encoded as [0, 0, 1]. This encoding method is beneficial for subsequent graph convolutional network processing.

[0109] Then, with obstacles as nodes and spatiotemporal relationships as edges, a dynamic topological structure G is constructed:

[0110] .

[0111] in: is a graph structure; is a node collection representing all obstacles in the scene; is an edge set, representing the spatiotemporal relationship between obstacles; is the adjacency matrix, which describes the connection relationship between nodes. The elements of the adjacency matrix A are The calculation is as follows:

[0112] .

[0113] in: is an element in the adjacency matrix A, representing the connection strength between nodes i and j, dimensionless; is the Euclidean distance between obstacles i and j, in meters; is the distance impact factor, in meters, usually set to 5 meters; The distance threshold is in meters and is usually set to 20 meters; represents the natural exponential function. This threshold setting ensures that only obstacles that are spatially close and likely to interact are connected.

[0114] The multi-scale temporal feature unit 22 extracts obstacle motion features within multiple time-scale windows to generate multi-scale temporal features. Specifically, it divides the obstacle trajectory data into three time windows: short-term (0.1-0.5 seconds), medium-term (0.5-2 seconds), and long-term (2-5 seconds), capturing microscopic motion details and macroscopic motion trends, respectively.

[0115] For the short-term time window, we mainly focus on the instantaneous behavior of the obstacle, such as sudden braking, sharp turns, etc.; for the medium-term time window, we focus on the movement pattern of the obstacle, such as acceleration, deceleration, lane changing, etc.; for the long-term time window, we focus on the movement intention of the obstacle, such as overtaking, U-turn, etc.

[0116] For different time windows, extract the corresponding feature vectors:

[0117] .

[0118] .

[0119] .

[0120] in: is the short-term time window feature vector; is the mid-term time window feature vector; is the long-term time window feature vector; is the current velocity component, in m / s; is the current acceleration component, in m / s²; is the rate of change of direction, in rad / s; are the average velocity and acceleration, in m / s and m / s² respectively; is the direction change variance, in rad²; is the trajectory curvature, in units of 1 / m; These are the changing trends of velocity, acceleration, and direction, with units of m / s², m / s³, and rad / s² respectively; is the lane change probability, dimensionless, ranging from [0,1].

[0121] Then, the features of different time scales are weighted and combined to form multi-scale time series features :

[0122] .

[0123] in: is the multi-scale time series feature vector; is the weight coefficient of different time scales, dimensionless, satisfying . Dynamically adjusted according to the scene type. In urban road scenes, the typical weight setting is , emphasizing short-term behavior prediction; in the highway scenario, the weight is set to , emphasizing long-term intention prediction.

[0124] The state transition model unit 23 establishes an obstacle state transition probability model to characterize the evolution of obstacle states. First, the continuous state space is divided into finite discrete states. For example, the speed is divided into four states: stationary (0-5 km / h), low speed (5-30 km / h), medium speed (30-60 km / h), and high speed (>60 km / h). Then, the state transition probability matrix P is constructed based on historical observation data:

[0125] .

[0126] in: From the state Transfer to state The conditional probability of is dimensionless and ranges from [0,1]; is the observed state Transfer to state the number of times; To observe the state The total number of times.

[0127] Furthermore, the environmental context information is introduced to adjust the transition probability and form the conditional state transition probability:

[0128] .

[0129] in: Given the environment context C, from the state Transfer to state The conditional probability of is dimensionless and ranges from [0,1]; Environmental context information, including road type, traffic light status, etc. is the conditional probability of observing a specific environmental context under a known state transition, dimensionless, and ranges from [0,1]; is the prior probability of the environmental context, dimensionless, and ranges from [0,1].

[0130] In practical applications, environmental context can significantly influence state transition probabilities. For example, at a red light, the probability of a vehicle transitioning from a moving state to a stopped state increases significantly; on a highway on-ramp, the probability of a vehicle accelerating increases significantly. By incorporating environmental context, the state transition model can more accurately predict obstacle state changes.

[0131] Reference Figure 4 The graph convolutional network module 3 is used to construct an obstacle relationship graph structure based on multi-dimensional spatiotemporal feature representation and apply the graph convolutional network for feature propagation. This module includes a heterogeneous information fusion unit 31, a dynamic sparse graph construction unit 32, and an attention feature propagation unit 33.

[0132] The heterogeneous information fusion unit 31 performs heterogeneous information fusion on the millimeter-wave radar data and the visual data to generate a unified feature representation. First, the millimeter-wave radar features and the visual features are normalized to eliminate the scale differences of the data from different sensors:

[0133] .

[0134] in: is the normalized eigenvector, dimensionless; is the original feature vector, the unit depends on the feature type; is the feature mean, and its unit is the same as the original feature; is the feature standard deviation, and its unit is the same as the original feature. This normalization step ensures that the data from different sensors have the same dimension and value range, which facilitates subsequent fusion processing.

[0135] Then, the normalized heterogeneous features are projected into a unified feature space through nonlinear transformation:

[0136] .

[0137] in: is the unified eigenvector after fusion, dimensionless; is the normalized millimeter-wave radar feature vector, dimensionless; is the normalized visual feature vector, dimensionless; and is the projection weight matrix, dimensionless; is the bias vector, dimensionless; It is a nonlinear activation function, usually the ReLU function is used, which is defined as .

[0138] Preferably, the dimension of the projection weight matrix is designed as follows: if the millimeter wave radar feature dimension is , the visual feature dimension is , the unified feature dimension is ,but The dimension is , The dimension is In practice, Usually 5-10 dimensions, containing information such as position and velocity; Usually 10-20 dimensions, including appearance, size, category, etc. It is usually set to 16 or 32 to balance expressiveness and computational efficiency.

[0139] In addition, the weights of millimeter-wave radar data and visual data are dynamically adjusted according to environmental conditions:

[0140] .

[0141] in: is the final fused feature vector, dimensionless; is the millimeter-wave radar characteristic vector, dimensionless; is the visual feature vector, dimensionless; is the fusion weight coefficient, dimensionless, ranging from [0,1].

[0142] Fusion weight coefficient Dynamic adjustment according to environmental conditions: Under normal lighting conditions, Set to 0.3 to emphasize visual features; at night or in bad weather conditions, Increased to 0.7, emphasizing the millimeter-wave radar characteristics. This dynamic weight adjustment mechanism significantly improves the system's adaptability in various environmental conditions.

[0143] The dynamic sparse graph construction unit 32 constructs a dynamic sparse graph structure with obstacles as nodes, and implements dynamic pruning of the graph structure through radar signal-to-noise ratio thresholds, visual confidence thresholds, and Top-K edge sampling mechanisms. First, the signal-to-noise ratio (SNR) is calculated for each radar point / cluster node in each frame:

[0144] .

[0145] in: is the signal-to-noise ratio, in decibels (dB); is the signal power in watts (W); is the noise power, in watts (W); Represents the base 10 logarithm.

[0146] Only keep the data with SNR greater than the preset threshold Radar nodes, usually Setting it to 8dB can effectively filter out false targets such as raindrops. For visual nodes, only those with detection confidence p greater than the preset threshold are retained. Nodes, usually θ2 is set to 0.6 to ensure the reliability of visual nodes.

[0147] Then, for the retained nodes, calculate the edge weights:

[0148] .

[0149] in: is the edge weight between nodes i and j, dimensionless, ranging from [0,1]; is the spatial distance in meters; and is the node velocity vector, in m / s; The Euclidean norm of the velocity vector difference is expressed in m / s; is the interaction possibility, dimensionless, ranging from [0,1]; is the weight coefficient, dimensionless, satisfying ; and are the influencing factors of distance and speed, in meters and m / s respectively; represents the natural exponential function.

[0150] Interaction possibilities Calculation based on the relative position and movement direction of the nodes:

[0151] .

[0152] in: is the interaction possibility, dimensionless, ranging from [0,1]; is the angle between the direction of motion of node i and the direction of the line connecting i to node i, in radians; is the distance between nodes, in meters; is the cosine value of the angle, dimensionless, ranging from [-1, 1]; sigmoid is the S-type activation function, defined as sigmoid , maps the value to the interval [0,1].

[0153] The key innovation lies in the Top-K edge sampling mechanism: for each node, we sort the edges from largest to smallest by edge weight, retaining only the top K edges and setting the remaining edge weights to 0. Typically, K is set to 20% of the total number of nodes to ensure that the graph structure does not exceed 20% sparsity, significantly reducing computational complexity while preserving key connections. The adjacency matrix A' after edge sampling is calculated as follows:

[0154] .

[0155] in: is the adjacency matrix element after sampling, indicating the node arrive The connection strength is dimensionless and has a range of values. .

[0156] As each frame of data arrives, the system recalculates node scores and edge weights in real time, dynamically updating the graph structure to ensure it adapts to changing scenarios. This dynamic sparse graph structure not only significantly reduces computational complexity but also retains the most informative nodes and connections through a data-driven pruning scale, making graph convolution both lightweight and accurate.

[0157] The attention feature propagation unit 33 performs feature propagation and aggregation through graph convolution combined with the attention mechanism to strengthen the representation of key features. First, a multi-level feature aggregation operation is designed to expand the receptive field layer by layer:

[0158] .

[0159] in: is the node feature matrix of the lth layer, and the matrix dimension is , N is the number of nodes, is the feature dimension of the lth layer; is the node feature matrix of the l+1th layer, and the matrix dimension is ; is the adjacency matrix, and the matrix dimension is ; Is the degree matrix, a diagonal matrix with dimension , whose diagonal elements represents the degree of node i; is the learnable weight matrix of the lth layer, and the matrix dimension is ; It is a nonlinear activation function, and the ReLU function is usually used; The adjacency matrix is normalized to ensure that the difference in node degree during feature propagation does not lead to numerical instability.

[0160] Then, the attention weights are calculated based on node relevance and importance:

[0161] .

[0162] .

[0163] in: is the attention score of node i to node j, dimensionless; is the attention vector, dimensionless; is the weight matrix, dimensionless; and is the node eigenvector, dimensionless; Represents the vector concatenation operation; LeakyReLU is the leaky rectified linear unit activation function, defined as LeakyReLU ; is the normalized attention coefficient, dimensionless, ranging from [0,1]; is the neighborhood of node i, that is, the set of all nodes connected to node i; represents the natural exponential function.

[0164] Preferably, the attention vector The dimension of matches the dimension of the node features after concatenation. For example, if the dimension of a single node feature is 32, the concatenated dimension is 64, and the attention vector It is also 64-dimensional.

[0165] Next, feature aggregation is performed based on the attention weights:

[0166] .

[0167] in: is the updated feature vector of node i, dimensionless; is a nonlinear activation function; Indicates the sum of all neighbor nodes of node i; is the attention coefficient; is the weight matrix; is the feature vector of the neighboring node.

[0168] In addition, a multi-head attention mechanism is used to capture relational features of different dimensions:

[0169] .

[0170] in: is the updated feature vector of node i, dimensionless; is the number of attention heads, usually an integer; The attention coefficient calculated for the kth attention head, dimensionless, range [0,1]; is the weight matrix of the kth attention head, dimensionless; Indicates the average of the results of K attention heads. In practical applications, the number of attention heads K is usually set to 4 or 8 to ensure expressiveness while controlling computational complexity.

[0171] The multi-head attention mechanism can learn the relationship between nodes from different perspectives at the same time. For example, one attention head may focus on spatial position relationship, another on speed similarity, and the third on category relationship, etc., thereby comprehensively capturing the multi-dimensional relationship between nodes.

[0172] Reference Figure 5 The trajectory prediction module 4 is used to perform nonlinear trajectory prediction based on the feature representation output by the graph convolutional network module. This module includes a phase space mapping unit 41, a physical constraint unit 42, and a probabilistic trajectory generation unit 43.

[0173] The phase space mapping unit 41 maps the obstacle's historical trajectory into the phase space to identify the motion pattern. First, a high-dimensional phase space containing dimensions such as position, velocity, and acceleration is constructed:

[0174] .

[0175] in: is the phase space state vector at time t; and is the position coordinate in meters; and is the velocity component, in m / s; and is the acceleration component, in m / s²; Represents the transpose of a vector.

[0176] Then, the historical trajectory point sequence is mapped into the phase space to form a trajectory manifold:

[0177] .

[0178] in: is the trajectory manifold, which represents the set of trajectory points in the phase space; is the number of historical time steps, usually an integer; is the time interval in seconds; Indicates tk The state vector at the moment . Usually, Set to 10-20, Set it to 0.1 seconds, corresponding to 1 to 2 seconds of historical trajectory.

[0179] Retrieve similar historical trajectory patterns based on phase space distance metrics:

[0180] .

[0181] in: Trajectory manifold and The distance between them, the unit depends on the unit of the state vector; is the time weight, dimensionless, and usually the recent time point has a larger weight; is the Euclidean distance of the state vector at the corresponding time point, and its unit depends on the unit of the state vector; represents the summation over all time steps.

[0182] Preferably, the time weight Using exponential decay form: ,in is the attenuation coefficient, dimensionless, and is usually set to 0.2. This weight setting reflects that recent states have a greater impact on future predictions.

[0183] The physical constraint unit 42 introduces physical constraints to ensure the rationality of the predicted trajectory. First, dynamic constraints including physical parameters such as maximum acceleration and steering angular rate are established:

[0184] .

[0185] .

[0186] in: and are the acceleration components in the x and y directions, in m / s²; is the maximum acceleration in m / s²; is the steering angle rate, in rad / s; is the maximum steering angular rate, in rad / s; represents the square of the total acceleration, Indicates the absolute value of the steering angle rate.

[0187] For different types of obstacles, the physical constraint parameters are set differently. For example, for an ordinary car, Set to 4m / s², Set to 0.5rad / s; for pedestrians, Set to 2m / s², Set to 1.5 rad / s. These parameter settings are based on real road traffic data statistics to ensure that the predicted trajectory conforms to the physical characteristics of various obstacles.

[0188] In addition, geometric constraints such as road boundaries and lane lines are integrated:

[0189] .

[0190] in: is the distance from the predicted point P to the road boundary B, in meters; It is the safety distance in meters, usually set to 0.5 to 1 meter.

[0191] It is also necessary to consider the interaction constraints between traffic rules and obstacles:

[0192] .

[0193] in: is the collision probability, dimensionless, ranging from [0,1]; is a safety threshold, dimensionless, typically set to 0.05. The collision probability is calculated based on the predicted trajectory of the obstacle and the uncertainty estimate.

[0194] Filter out predicted trajectories that do not conform to physical laws through constrained optimization:

[0195] .

[0196] in: is the set of valid trajectories; is the i-th candidate trajectory; is the constraint decision function, when the trajectory Returns True if all constraints are met, otherwise returns False; Indicates the judgment of all candidate trajectories i.

[0197] The probability trajectory generation unit 43 generates multiple candidate trajectories and integrates them to form the final prediction result. First, the probability distribution parameters of the future position are estimated:

[0198] .

[0199] in: For a given current state and historical trajectory Conditions, future state The conditional probability distribution of ; Indicates the mean , the covariance matrix is Multivariate Gaussian distribution of ; is the mean vector of the predicted state, and its unit is the same as that of the state vector; is the covariance matrix of the predicted state, with units of the square of the units of the state vector components.

[0200] Then, multiple candidate trajectories are generated by sampling from the probability distribution:

[0201] .

[0202] in: is the i-th candidate trajectory, representing the state vector of a series of consecutive time points; is the state vector at the current moment; is the i-th trajectory at t+k The predicted state vector at time t is sampled from the probability distribution at the corresponding time t. Typically, 5 to 10 candidate trajectories are generated, covering different possible motion modes.

[0203] Finally, the trajectories are scored and integrated based on indicators such as physical plausibility and historical accuracy:

[0204] .

[0205] .

[0206] in: is the score of the i-th trajectory, dimensionless; Score for physical rationality, dimensionless, range is the historical accuracy score, dimensionless, ranging from [0,1]; is the interaction rationality score, dimensionless, ranging from [0,1]; is the weight coefficient, dimensionless, satisfying is the number of candidate trajectories; is the final predicted trajectory after weighted integration; Indicates the sum of N candidate trajectories; represents the normalized weight of the i-th track.

[0207] Preferably, the weight coefficient is set to , balancing physical plausibility and historical accuracy while taking interaction factors into account.

[0208] In practical applications, trajectories are predicted for the next 3 to 5 seconds, with a time interval of 0.1 seconds, corresponding to 30 to 50 prediction points. The prediction results include the trajectory point sequence and its uncertainty estimate, providing comprehensive information for subsequent decision-making.

[0209] Reference Figure 6 The verification and alarm module 5 is used to verify and evaluate the prediction results of the trajectory prediction module, generate obstacle trajectory prediction information and alarm prompts. This module includes a trajectory verification unit 51, a risk assessment unit 52 and an alarm generation unit 53.

[0210] The trajectory verification unit 51 performs an intersection-over-union (IOU) verification on the predicted trajectory and the real-time detection result. The IOU value of the predicted bounding box and the detected bounding box is calculated as:

[0211] .

[0212] in: is the intersection-union ratio, dimensionless, ranging from [0,1]; is the predicted bounding box; To detect the bounding box, is the intersection area in square meters; is the union area in square meters.

[0213] If the IOU value is less than 0.75, the cause of the trajectory deviation is further analyzed. This deviation may occur due to a sudden lane change, a sudden change in speed, or the appearance of a new obstacle. The analysis results serve as feedback to the trajectory prediction module, dynamically adjusting the prediction parameters. The risk assessment unit 52 identifies trajectory pairs that may collide and calculates a collision risk index.

[0214] First, calculate the minimum distance between any two obstacle trajectories:

[0215] .

[0216] in: For the trajectory and The minimum distance between them, in meters; and are the predicted positions of the obstacle and j at time t, respectively, in meters; For the forecast time range, For the current moment, is the prediction time length in seconds; Indicates taking the minimum value within the forecast time range; Represents the Euclidean distance between two points.

[0217] Then, the collision probability is calculated based on the minimum distance and the obstacle size:

[0218] .

[0219] in: For the trajectory and The collision probability between them is dimensionless and ranges from [0,1]; and are the equivalent radii of obstacles i and j, respectively, in meters; is the uncertainty parameter, the unit is meter, usually set to 1 meter; represents the natural exponential function.

[0220] Finally, the risk index is calculated by combining the collision probability and time urgency:

[0221] .

[0222] in: For trajectory pair The risk index is dimensionless and ranges from [0,1]; is the collision time (TimeToCollision), in seconds; is the time decay coefficient, the unit is 1 / second, usually set to 0.5; represents the natural exponential function.

[0223] The alarm generation unit 53 generates different levels of alarm information according to the risk level. The risk level is divided into four levels:

[0224] Safe: , no need to call the police;

[0225] Attention: , prompt alarm;

[0226] Warning: , intermediate alarm;

[0227] Emergency: , advanced alarm.

[0228] For different alarm levels, corresponding prompt information and visual effects are generated. For example, for the attention level, the relevant obstacles are marked with a yellow border on the interface; for the warning level, an orange border and text prompts are used; for the emergency level, a red flashing border is used and an audible alarm is triggered.

[0229] In addition, the warning information also includes details such as risk type (such as forward collision, side collision), time urgency and recommended operations (such as deceleration, avoidance), providing decision-making reference for the driver or autonomous driving system.

[0230] The millimeter-wave and visual fusion perception method and system for dynamic road obstacles have demonstrated excellent performance in practical applications. In standard test scenarios, compared to traditional single-sensor methods, detection accuracy improved by 35%, tracking stability by 40%, and prediction accuracy by 50%. The system maintains stable perception performance, particularly in adverse weather conditions such as heavy rain and dense fog.

[0231] The dynamic sparse graph construction mechanism reduces system computational overhead by over 50%, lowering end-to-end processing latency from 100ms to under 50ms, meeting the demands of real-time applications. Furthermore, SNR threshold filtering effectively removes false targets such as raindrops, visual confidence threshold filtering reduces background false detections, and Top-K edge sampling ensures that the sparse graph structure retains the most critical interactions.

[0232] In complex urban road scenario testing, the system accurately predicted nonlinear motion behaviors such as pedestrians swerving and vehicles changing lanes rapidly, with a prediction timeframe of 3-5 seconds, providing ample reaction time for risk avoidance decisions. Furthermore, the system significantly reduced resource usage, making it suitable for stable operation on embedded platforms with medium computing power.

[0233] Furthermore, the system supports multiple deployment modes, including standalone on-board deployment, roadside deployment, and vehicle-road collaborative deployment, adapting to different application scenarios and needs. The system's core algorithm has been optimized for excellent engineering practicality.

[0234] In summary, the millimeter wave-vision fusion perception method and system for dynamic road obstacles provided by the present invention comprehensively improves the accuracy, reliability, and foresight of dynamic road obstacle perception through innovative multi-dimensional spatiotemporal feature representation, dynamic sparse graph convolutional network, and nonlinear trajectory prediction technology, while significantly reducing computational complexity, providing important support for the development of intelligent transportation and autonomous driving technologies.

[0235] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. The millimeter wave-vision fusion perception method for dynamic road obstacles is characterized by: include: Collect road scene data through millimeter-wave radar and visual sensors to obtain initial feature information of obstacles; Based on the initial feature information of the obstacle, a multi-dimensional spatiotemporal feature representation is constructed, including: Map the obstacle's historical trajectory to the topological structure space to form the obstacle's motion pattern representation; Extract obstacle motion features in multiple time scale windows to generate multi-scale temporal features; Establish an obstacle state transition probability model to characterize the obstacle state evolution law; Based on the multi-dimensional spatiotemporal feature representation, an obstacle relationship graph structure is constructed and a dynamic sparse graph convolutional network is applied for feature propagation, including: Perform heterogeneous information fusion of millimeter-wave radar data and visual data to generate a unified feature representation; A dynamic sparse graph structure is constructed with obstacles as nodes, and dynamic pruning of the graph structure is achieved through radar signal-to-noise ratio thresholds, visual confidence thresholds, and Top-K edge sampling mechanisms. By combining graph convolution operations with attention mechanisms to propagate and aggregate features, key feature representations are enhanced. Based on the feature representation output by the graph convolutional network, nonlinear trajectory prediction is performed, including: Map the obstacle's historical trajectory into phase space and identify the motion pattern; Introducing physical constraints to ensure the rationality of the predicted trajectory; Generate multiple candidate trajectories and integrate them to form the final prediction result; The prediction results are verified and evaluated to generate obstacle trajectory prediction information and alarm prompts.

2. The millimeter wave-vision fusion perception method for dynamic road obstacles according to claim 1 is characterized in that: The method of collecting road scene data through millimeter wave radar and visual sensors to obtain initial obstacle feature information specifically includes: The millimeter-wave radar collects the reflection intensity, reflection surface angle and emission angle of the obstacle point at a polling frequency of 10kHz; The system collects 4K resolution images through visual sensors and uses object detection algorithms to extract the location, size, and category information of obstacles. Converting the obstacle position information obtained by the visual sensor from a pixel coordinate system to a millimeter-wave radar coordinate system to achieve coordinate unification; The speed information of the obstacle is calculated based on the reflection intensity, reflection surface angle and emission angle.

3. The millimeter wave-vision fusion perception method for dynamic road obstacles according to claim 1 is characterized in that: Mapping the obstacle's historical trajectory to the topological structure space to form an obstacle motion pattern representation specifically includes: Construct an n-dimensional obstacle state vector containing position, velocity, acceleration, and size; Using obstacles as nodes and spatiotemporal relationships as edges, a dynamic topological structure is constructed; The original features are mapped to high-dimensional manifold space through nonlinear transformation to enhance the ability to express nonlinear motion patterns.

4. The millimeter wave-vision fusion perception method for dynamic road obstacles according to claim 1 is characterized in that: The step of extracting obstacle motion features within multiple time scale windows to generate multi-scale temporal features specifically includes: The obstacle historical trajectory data is divided into three time windows: short-term 0.1-0.5 seconds, medium-term 0.5-2 seconds, and long-term 2-5 seconds; Design dedicated feature extractors for different time windows to capture microscopic motion details and macroscopic motion trends respectively; Features at different time scales are weighted and combined to form a multi-scale time series representation.

5. The millimeter wave-vision fusion perception method for dynamic road obstacles according to claim 1 is characterized in that: The heterogeneous information fusion of millimeter-wave radar data and visual data to generate a unified feature representation specifically includes: Scale normalization and dimension alignment of millimeter-wave radar features and visual features; Project heterogeneous features into a unified feature space through nonlinear transformation; The weights of millimeter-wave radar data and visual data are dynamically adjusted according to environmental conditions to strengthen complementary features and suppress redundant features.

6. The millimeter wave-vision fusion perception method for dynamic road obstacles according to claim 1 is characterized in that: The dynamic sparse graph structure is constructed with obstacles as nodes, and dynamic pruning of the graph structure is achieved through radar signal-to-noise ratio threshold, visual confidence threshold and Top-K edge sampling mechanism. Specifically, the following steps are involved: Calculate the signal-to-noise ratio (SNR) for each radar point / cluster node in each frame, and retain nodes whose SNR is greater than the preset threshold θ1; Open cross-modal edge connections for nodes whose image detection box confidence p is greater than the preset threshold θ2; For each node, only the K edges with the highest relationship scores are retained, so that the sparseness rate of the graph structure does not exceed 20%; When each frame of data arrives, the node scores and edge weights are recalculated in real time, and the graph structure is dynamically updated.

7. The millimeter wave-vision fusion perception method for dynamic road obstacles according to claim 1 is characterized in that: The graph convolution operation is combined with the attention mechanism to perform feature propagation and aggregation, and strengthen the key feature representation, specifically including: Design multi-level feature aggregation operations to expand the receptive field layer by layer; Calculate attention weights based on node relevance and importance; Dynamically adjust feature propagation path and strength based on attention weights; Capture relational features of different dimensions through a multi-head attention mechanism.

8. The millimeter wave-vision fusion perception method for dynamic road obstacles according to claim 1 is characterized in that: Mapping the obstacle's historical trajectory into the phase space and identifying the motion pattern specifically includes: Construct a high-dimensional phase space containing dimensions such as position, velocity, and acceleration; Map the historical trajectory point sequence into the phase space to form a trajectory manifold; Retrieve similar historical trajectory patterns based on phase space distance metrics; Extract the evolution law of the trajectory in the phase space and identify typical motion patterns.

9. The millimeter wave-vision fusion perception method for dynamic road obstacles according to claim 1, characterized in that: The physical constraints introduced above ensure the rationality of the predicted trajectory, including: Establish dynamic constraints including physical parameters such as maximum acceleration and steering angular rate; Integrate geometric constraints such as road boundaries and lane lines; Consider the interaction constraints between traffic rules and obstacles; Filter out predicted trajectories that do not conform to physical laws through constrained optimization; Verifying and evaluating the prediction results to generate obstacle trajectory prediction information and alarm prompts specifically includes: Verify the predicted trajectory with the real-time detection result by IOU; When the IOU value is less than 0.75, further analyze the cause of trajectory deviation; Identify trajectory pairs that may collide and calculate the collision risk index; Generate different levels of alarm information according to risk level; Risk information is displayed through a visual interface to assist decision-making.

10. The millimeter wave-vision fusion perception system for dynamic road obstacles is characterized by: include: The perception data acquisition module is used to collect road scene data through millimeter-wave radar and visual sensors to obtain initial feature information of obstacles; A spatiotemporal feature representation module, configured to construct a multi-dimensional spatiotemporal feature representation based on the initial obstacle feature information, wherein the multi-dimensional spatiotemporal feature representation includes: a topological structure representation of the obstacle motion pattern, multi-scale temporal features, and an obstacle state transition probability model; A graph convolutional network module, configured to construct an obstacle relationship graph structure and perform feature propagation based on the multi-dimensional spatiotemporal feature representation, wherein the graph convolutional network module comprises: a heterogeneous information fusion unit, a dynamic sparse graph construction unit, and an attention feature propagation unit; A trajectory prediction module, configured to perform nonlinear trajectory prediction based on the feature representation output by the graph convolutional network module, the trajectory prediction module comprising: a phase space mapping unit, a physical constraint unit, and a probabilistic trajectory generation unit; A verification and alarm module, used to verify and evaluate the prediction results of the trajectory prediction module, and generate obstacle trajectory prediction information and alarm prompts; The dynamic sparse graph construction unit dynamically prunes the graph structure by evaluating node reliability and edge importance in real time, including: Calculate the radar node signal-to-noise ratio (SNR) and visual node detection confidence, and set corresponding thresholds for node filtering; For each node, only the K edges with the highest relationship scores are retained to keep the graph structure sparsity rate within 20%; The graph structure is updated in real time when each frame of data arrives, so that sparse graph convolution can keep the computation lightweight while capturing key spatiotemporal relationships.

Images