Multi-mode time-space fusion intelligent vehicle danger pre-judgment method and system

Abstract

The invention relates to the technical field of intelligent traffic and auxiliary travel, and discloses a multi-modal space-time fusion intelligent vehicle danger pre-judgment method and system, and the method comprises the steps: obtaining multi-modal sensing data around an intelligent vehicle, constructing a deep learning-based danger pre-judgment model which comprises a space-time fusion module and a risk evolution module; the space-time fusion module carries out space-time alignment and fusion on multi-modal sensing data to obtain unified space-time voxel features, and the space-time alignment and fusion processing process is focused on a high-uncertainty region through the risk attention gating module. The risk evolution module identifies objects around the intelligent vehicle according to the space-time voxel features, and predicts the motion trails and potential collision risks of the objects; and performing real-time danger judgment on the intelligent vehicle by using the world model distillation danger pre-judgment model and the post-distillation danger pre-judgment model. According to the invention, various sensing data can be effectively fused to carry out danger judgment in the vehicle driving process, and the judgment accuracy is improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation and assisted mobility technology, and in particular to a multimodal spatiotemporal fusion intelligent vehicle hazard prediction method and system. Background Technology

[0002] With the rapid development of autonomous driving technology and intelligent transportation systems, intelligent vehicles are gradually becoming an important part of the modern transportation system. However, in complex urban traffic environments, intelligent vehicles still face many challenges due to potential risk factors such as dynamically changing vehicle flow interactions, sudden pedestrian crossings, and complex road obstacles.

[0003] Currently, most mainstream intelligent vehicle systems adopt a single sensor architecture, which has shortcomings such as large perception blind spots, poor environmental adaptability, and insufficient multi-target tracking accuracy. It is difficult to achieve comprehensive perception and accurate prediction of complex traffic scenarios. Moreover, there is no effective method for judging dangers during vehicle driving that can use multiple sensor data, which seriously restricts the safety performance of intelligent vehicles. Summary of the Invention

[0004] Therefore, the technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a multimodal spatiotemporal fusion intelligent vehicle hazard prediction method and system, which can effectively integrate multiple sensor data to judge the hazard during vehicle driving and improve the accuracy of the judgment.

[0005] To address the aforementioned technical problems, this invention provides a multimodal spatiotemporal fusion intelligent vehicle hazard prediction method, comprising: Acquire multimodal sensing data around the intelligent vehicle, including point cloud data collected by lidar, image data collected by camera, and motion data collected by millimeter-wave radar; A deep learning-based hazard prediction model is constructed, comprising a spatiotemporal fusion module and a risk evolution module. The spatiotemporal fusion module performs spatiotemporal alignment and fusion processing on the multimodal sensor data to obtain unified spatiotemporal voxel features. The spatiotemporal fusion module includes a risk attention gating module, which focuses the spatiotemporal alignment and fusion processing on high uncertainty regions. The risk evolution module identifies objects around the intelligent vehicle, predicts the motion trajectory of the objects, and identifies potential collision risks based on the spatiotemporal voxel features. The hazard prediction model is distilled using a world model, and the distilled hazard prediction model is used to make real-time hazard judgments for intelligent vehicles.

[0006] Furthermore, the spatiotemporal fusion module performs spatiotemporal alignment and fusion processing on the multimodal sensing data to obtain unified spatiotemporal voxel features, specifically: The spatiotemporal fusion module uses the Transformer architecture as the backbone network, including a graph convolutional neural network, a temporal encoder, and a spatial encoder. The temporal encoder and the spatial encoder contain self-attention modules and convolutional pooling layers. The graph convolutional neural network extracts graph trajectory feature sequences based on the multimodal sensing data, divides the graph trajectory feature sequences into batches according to time and space, and then simultaneously inputs them into the temporal encoder and the spatial encoder. The temporal encoder captures feature associations in the time dimension through a self-attention mechanism to obtain a temporal feature sequence. The spatial encoder uses a self-attention mechanism to capture feature associations in the spatial dimension between different nodes to obtain a spatial feature sequence. The temporal feature sequence, spatial feature sequence, and graph trajectory feature sequence are fused to obtain spatiotemporal voxel features.

[0007] Furthermore, the temporal encoder captures feature correlations in the temporal dimension through a self-attention mechanism to obtain a temporal feature sequence, specifically as follows: Let the timing characteristics of the input time-domain encoder be denoted as . , This represents the i-th node in the temporal features of the input temporal encoder. This represents the t-th time step of the i-th node within the sampling period; i = 1, 2, ..., M, where M represents the number of nodes; t = 1, 2, ..., n, where n represents the number of time steps within the sampling period; Will Projecting these vectors onto the query vector, key vector, and value vector respectively yields the query matrix, key matrix, and value matrix; let the query matrix be denoted as . , , for The query vector obtained by projection; the key matrix is ​​denoted as... , , for The key vector obtained by projection; the value matrix is ​​denoted as , , for The value vector obtained by projection; The temporal feature sequence obtained through the self-attention mechanism is as follows: , In the formula, Let represent the i-th node in the time-domain feature sequence, T represent the transpose operation, d represent the feature dimension, and softmax represent the softmax function.

[0008] Furthermore, the spatial encoder uses a self-attention mechanism to capture the spatial feature correlations between different nodes to obtain a spatial feature sequence, specifically: Let the timing characteristics of the input spatial encoder be denoted as . , This indicates selecting all nodes at time t. Let i represent the i-th node at time t, where i = 1, 2, ..., N, and N represents the number of nodes at time t. The message passing between the i-th node and the j-th node at time t is calculated as follows: , In the formula, This represents the message passing between the i-th node and the j-th node at time t. It is the query vector obtained by projecting the i-th node at time t. It is the key vector obtained by projecting the j-th node at time t, where T represents the transpose operation. It is the corresponding element in the adjacency matrix used to represent the interaction strength; The spatial feature sequence obtained through the self-attention mechanism is as follows: , In the formula, This represents the i-th node in the spatial feature sequence. Let represent the value vector obtained by projecting the i-th node at time t. represents the feature dimension, and softmax represents the softmax function.

[0009] Furthermore, the spatiotemporal voxel features are as follows: , In the formula, Represents spatiotemporal voxel characteristics, Represents time-domain feature sequences, Represents spatial feature sequences. Represents the feature sequence of the graph trajectory. This indicates a feature fusion operation. This indicates a feature extraction operation.

[0010] Furthermore, the risk evolution module identifies objects around the intelligent vehicle, predicts the motion trajectory of the objects, and identifies potential collision risks based on the spatiotemporal voxel features, specifically as follows: The risk evolution module identifies the object categories around the intelligent vehicle based on the spatiotemporal voxel features corresponding to the image data, identifies the distance and shape of the objects around the intelligent vehicle based on the spatiotemporal voxel features corresponding to the point cloud data, predicts the motion trajectory of the objects based on the spatiotemporal voxel features corresponding to the motion data, and judges potential collision risks by combining the object categories, object distances and shapes, and object motion trajectories. When predicting the motion trajectory of an object based on the spatiotemporal voxel features corresponding to the motion data, a mask matrix is ​​used to ensure that the information at each position only focuses on the portion at subsequent positions.

[0011] Furthermore, the step of ensuring that the information at each position is only relevant to subsequent positions using a mask matrix specifically involves: The feature sequence for constructing the input decoder is: , In the formula, This represents the feature sequence input to the decoder. The starting marker consists of all zeros. This is a sequence extracted from the feature sequence of the graph trajectory. Indicates feature fusion operation; according to Constructing the query matrix, key matrix, and value matrix yields the feature sequence output by the decoder: , In the formula, Q represents the feature sequence output by the decoder, where Q represents the sequence based on the decoder output. The constructed query matrix, where K represents the query matrix based on... The constructed key matrix, V represents the key matrix constructed according to... The constructed value matrix, The dimension represents the feature dimension, and M represents the mask matrix.

[0012] Furthermore, when the spatiotemporal alignment and fusion processing is focused on the high-uncertainty region through the risk attention gating module, the uncertainty is calculated as follows: , In the formula, (x,y,t) represents the spatiotemporal coordinates, where x and y represent the x-coordinates and y-coordinates in the spatial dimension, and t represents the time dimension. Indicates uncertainty, This represents the probability distribution output by the hazard prediction model. Represents the L2 norm; The method for calculating the learnable gating vector is as follows: , In the formula, This represents a learnable gated vector. This represents the sigmoid activation function. Represents the gate weight matrix. This represents the spatiotemporal voxel features input to the risk attention gating module. This indicates the gating bias term.

[0013] Furthermore, after focusing the spatiotemporal alignment and fusion processing on the high-uncertainty region through the risk attention gating module, the final spatiotemporal voxel features are: , In the formula, ⊙ represents the final spatiotemporal voxel feature, and ⊙ represents the Hadamard product.

[0014] This invention also provides a multimodal spatiotemporal fusion intelligent vehicle hazard prediction system, comprising: A multimodal data acquisition module, including a lidar, a camera, and a millimeter-wave radar, is installed on an intelligent vehicle and collects multimodal sensing data around the intelligent vehicle in real time; the multimodal sensing data includes point cloud data collected by the lidar, image data collected by the camera, and motion data collected by the millimeter-wave radar. A hazard prediction model module is used to construct a deep learning-based hazard prediction model. The hazard prediction model includes a spatiotemporal fusion module and a risk evolution module. The spatiotemporal fusion module performs spatiotemporal alignment and fusion processing on the multimodal sensor data to obtain unified spatiotemporal voxel features. The spatiotemporal fusion module includes a risk attention gating module, which focuses the spatiotemporal alignment and fusion processing on high uncertainty regions. The risk evolution module identifies objects around the intelligent vehicle, predicts the motion trajectory of objects, and identifies potential collision risks based on the spatiotemporal voxel features. The distillation module is used to distill the hazard prediction model using the world model, and to use the distilled hazard prediction model to make real-time hazard judgments on intelligent vehicles. The early warning and decision-making module is used to generate corresponding early warning signals and decision suggestions based on real-time hazard assessments. The communication module communicates with surrounding vehicles and road infrastructure in real time via C-V2X to correct prediction results.

[0015] Compared with the prior art, the above-described technical solution of the present invention has the following advantages: This invention acquires multimodal sensor data around intelligent vehicles and performs spatiotemporal alignment and fusion processing to obtain unified spatiotemporal voxel features. Based on these features, the invention uses them to predict and warn of potential dangers during the intelligent vehicle's operation in real time, thereby improving the accuracy of judgment and significantly enhancing the safety and autonomy of intelligent vehicles. Attached Figure Description

[0016] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein: Figure 1 This is a flowchart of a method in a preferred embodiment of the present invention.

[0017] Figure 2 This is a flowchart illustrating the process of using the world model distillation risk prediction model in a preferred embodiment of the present invention.

[0018] Figure 3 This is a flowchart illustrating the process of using the risk attention gating module in a preferred embodiment of the present invention.

[0019] Figure 4 This diagram illustrates the process of using a distilled hazard prediction model to perform real-time hazard assessment of intelligent vehicles in a preferred embodiment of the present invention. Detailed Implementation

[0020] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0021] Reference Figure 1 As shown, this invention discloses a multimodal spatiotemporal fusion intelligent vehicle hazard prediction method, comprising the following steps: S1: Acquire multimodal sensing data around the intelligent vehicle. The multimodal sensing data includes point cloud data collected by lidar (for detecting the distance and shape of surrounding objects), image data collected by camera (for identifying traffic signs, pedestrians and other vehicles), and motion data collected by millimeter-wave radar (for detecting the speed and direction of speed of surrounding objects).

[0022] S2: Construct a deep learning-based hazard prediction model, which includes a spatiotemporal fusion module (ST-Transformer) and a risk evolution module (World-model Risk Evolution, WRE) based on a Transformer network. The spatiotemporal fusion module performs spatiotemporal alignment and fusion processing on the multimodal sensor data to obtain unified spatiotemporal voxel features (STV). The spatiotemporal fusion module includes a risk-attention gating module (RAG), which focuses the spatiotemporal alignment and fusion process on high-uncertainty regions, as shown in the example. Figure 3As shown, the risk evolution module identifies objects around the intelligent vehicle, predicts their trajectories, and identifies potential collision risks based on the spatiotemporal voxel features.

[0023] S2-1: The spatiotemporal fusion module fuses point cloud data, image data, and motion data through the transformation of timestamps and spatial coordinate systems to generate unified spatiotemporal voxel features, thereby reflecting the dynamic changes of the environment surrounding the intelligent vehicle in real time.

[0024] The spatiotemporal fusion module uses a Transformer architecture as its backbone network, including Graph Convolutional Networks (GCN), a temporal encoder, and a spatial encoder. The temporal and spatial encoders contain self-attention modules and convolutional pooling layers. The spatiotemporal fusion module performs spatiotemporal alignment and fusion processing on the multimodal sensing data to obtain unified spatiotemporal voxel features, specifically: S2-1-1: The graph convolutional neural network extracts graph trajectory feature sequences based on the multimodal sensing data, divides the graph trajectory feature sequences into batches according to time and space, and simultaneously inputs them into the temporal encoder and the spatial encoder to capture features in two dimensions. S2-1-2: The temporal encoder captures the feature associations in the time dimension through a self-attention mechanism to obtain a temporal feature sequence, and the spatial encoder uses a self-attention mechanism to capture the feature associations in the spatial dimension between different nodes to obtain a spatial feature sequence.

[0025] The temporal encoder applies a self-attention mechanism to the temporal features of a single node. Specifically, the temporal encoder captures feature correlations along the temporal dimension through this self-attention mechanism to obtain a temporal feature sequence. Let the timing characteristics of the input time-domain encoder be denoted as . , This represents the i-th node in the temporal features of the input temporal encoder. This represents the t-th time step of the i-th node within the sampling period; i = 1, 2, ..., M, where M represents the number of nodes; t = 1, 2, ..., n, where n represents the number of time steps within the sampling period; Will Projecting these vectors onto the query vector, key vector, and value vector respectively yields the query matrix, key matrix, and value matrix; let the query matrix be denoted as . , , for The query vector obtained by projection; the key matrix is ​​denoted as... , , for The key vector obtained by projection; the value matrix is ​​denoted as , , for The value vector obtained by projection; The temporal feature sequence obtained through the self-attention mechanism is as follows: , In the formula, Let represent the i-th node in the time-domain feature sequence, T represent the transpose operation, d represent the feature dimension, and softmax represent the softmax function.

[0026] The spatial encoder applies a self-attention mechanism to all node features at a single time step. Specifically, the spatial encoder uses the self-attention mechanism to capture the spatial feature correlations between different nodes to obtain a spatial feature sequence. Let the timing characteristics of the input spatial encoder be denoted as . , This indicates selecting all nodes at time t. Let i represent the i-th node at time t, where i = 1, 2, ..., N, and N represents the number of nodes at time t. Unlike the self-attention mechanism design of temporal encoders, the Q and K of the self-attention mechanism in spatial encoders are designed based on the interaction relationship between nodes. Therefore, this invention designs an edge mask self-attention mechanism to embed edge information into the node information transmission. Specifically, the message transmission between the i-th node and the j-th node (i≠j) at time t is calculated as follows: , In the formula, This represents the message passing between the i-th node and the j-th node at time t. It is the i-th node at time t (i.e. The query vector obtained by projection, It is the j-th node at time t (i.e. The key vector obtained by projection, where T represents the transpose operation. It is the corresponding element in the adjacency matrix used to represent the interaction strength, representing the interaction strength between the i-th node and the j-th node; The spatial feature sequence obtained through the self-attention mechanism is as follows: , In the formula, This represents the i-th node in the spatial feature sequence. Let represent the value vector obtained by projecting the i-th node at time t. represents the feature dimension, and softmax represents the softmax function.

[0027] S2-1-3: The spatiotemporal voxel features obtained after fusing the temporal feature sequence, spatial feature sequence, and graph trajectory feature sequence are as follows: , In the formula, Represents spatiotemporal voxel characteristics, Represents time-domain feature sequences, , Represents spatial feature sequences. , Represents the feature sequence of the graph trajectory. This indicates a feature fusion operation, specifically a concatenation operation. This refers to feature extraction operations, specifically multilayer perceptron operations.

[0028] Using the above spatiotemporal voxel feature alignment mechanism, the spatiotemporal misalignment problem between sparse point cloud data and dense RGB features of image data can be solved, making the alignment error less than 2 cm.

[0029] When the spatiotemporal alignment and fusion processing is focused on a high-uncertainty region using the risk attention gating module, the uncertainty is calculated as follows: , In the formula, (x,y,t) represents the spatiotemporal coordinates, where x and y represent the x-coordinates and y-coordinates in the spatial dimension, and t represents the time dimension. It represents uncertainty and reflects the confidence level of the model's prediction. This represents the probability distribution heatmap of the prediction, i.e., the probability distribution output by the hazard prediction model. Represents the L2 norm; The method for calculating the learnable gating vector is as follows: , In the formula, This represents learnable gated vectors and feature modulation for risk perception. This indicates the sigmoid activation function, which compresses the output to the range (0,1). Let represent the gate weight matrix, and be learnable linear transformation parameters. This represents the spatiotemporal voxel features input to the risk attention gating module. This represents the gated bias term, which is a learnable bias parameter; When the uncertainty (i.e., the gating coefficient) is greater than a preset threshold (0.9 in this embodiment), the spatiotemporal voxel features are weighted and enhanced. The spatiotemporal voxel features enhanced by the risk attention gating module are as follows: , In the formula, This represents the spatiotemporal voxel features enhanced by the risk attention gating module, where ⊙ represents the Hadamard product. This represents the final spatiotemporal voxel features obtained by the spatiotemporal fusion module.

[0030] S2-2: The risk evolution module identifies objects around the intelligent vehicle and predicts the motion trajectory and potential collision risks of the objects based on the spatiotemporal voxel features.

[0031] The risk evolution module identifies the object categories around the intelligent vehicle based on the spatiotemporal voxel features corresponding to the image data, identifies the distance and shape of the objects around the intelligent vehicle based on the spatiotemporal voxel features corresponding to the point cloud data, and predicts the motion trajectory of the objects based on the spatiotemporal voxel features corresponding to the motion data. In this embodiment, it specifically outputs a probability distribution heatmap of dynamic obstacles within the next 3 seconds, and judges potential collision risks by combining the object category, distance and shape, and motion trajectory of the objects. When predicting the motion trajectory of objects based on the spatiotemporal voxel features corresponding to the motion data, a mask matrix is ​​used to ensure that the information of each position only focuses on the part of the subsequent position, thereby avoiding the problems of slow speed and cumulative error in long sequence prediction by the autoregressive mechanism in traditional Transformer.

[0032] A mask matrix is ​​used to ensure that the information at each position only concerns the portion at subsequent positions, specifically: The feature sequence for constructing the input decoder is: , In the formula, This represents the feature sequence input to the decoder. The starting marker consists of all zeros. This serves as a placeholder for the target sequence to ensure that the encoder output matches the set predicted trajectory sequence length. This is a sequence extracted from the feature sequence of the graph trajectory. The length is , The value should be set according to the actual settings. This indicates a feature fusion operation, specifically a concatenation operation; unlike the traditional Transformer which uses a fixed, specific flag as the start marker of the sequence, this invention will... and After concatenation, it serves as an earlier segment preceding the predicted output sequence, acting as a starting marker for the input sequence. Using a mask matrix ensures that information at each position is only relevant to subsequent positions, preventing information leakage and autoregression issues.

[0033] according to Constructing the query matrix, key matrix, and value matrix yields the feature sequence output by the decoder: , In the formula, Q represents the feature sequence output by the decoder, where Q represents the sequence based on the decoder output. The constructed query matrix, where K represents the query matrix based on... The constructed key matrix, V represents the key matrix constructed according to... The constructed value matrix, The feature dimension is represented by M, which represents the mask matrix. The choice of the mask matrix depends on the actual settings.

[0034] Will and The hidden states are fed into the risk evolution module, where a fully connected layer maps them to the target predicted trajectory. This allows for parallel prediction of the trajectory across all time steps in a single inference iteration, unlike other time series analysis models that require iterative forward operations step by step.

[0035] S3: Distill the aforementioned hazard prediction model using a world model. In this embodiment, the large cloud model (>10 B parameters) is distilled into a lightweight edge model (<50 M parameters), achieving 30 FPS real-time inference on an NVIDIA Jetson Orin Nano with a model accuracy loss of <3%. The distillation process is as follows: Figure 2 As shown.

[0036] S4: Use the distilled hazard prediction model to perform real-time hazard assessment for intelligent vehicles. Specifically, this can be achieved by using a reinforcement learning policy network to generate optimal strategies for deceleration, detour, or emergency braking based on a probability distribution heatmap.

[0037] This invention also discloses a multimodal spatiotemporal fusion intelligent vehicle hazard prediction system, comprising: The multimodal data acquisition module, including a lidar, a camera, and a millimeter-wave radar, is installed on the intelligent vehicle and collects multimodal sensing data around the intelligent vehicle in real time. The multimodal sensing data includes point cloud data collected by the lidar, image data collected by the camera, and motion data collected by the millimeter-wave radar. In this embodiment, fisheye cameras, inertial measurement units (IMUs), flexible seat pressure arrays, and ultra-wideband (UWB) vehicle-to-everything (V2X) communication modules can also be used to collect more data.

[0038] A hazard prediction model module is used to construct a hazard prediction model based on deep learning. The hazard prediction model includes a spatiotemporal fusion module and a risk evolution module. The spatiotemporal fusion module performs spatiotemporal alignment and fusion processing on the multimodal sensor data to obtain unified spatiotemporal voxel features. The spatiotemporal fusion module includes a risk attention gating module, which focuses the spatiotemporal alignment and fusion processing on high uncertainty regions. The risk evolution module identifies objects around the intelligent vehicle, predicts the motion trajectory of objects, and identifies potential collision risks based on the spatiotemporal voxel features.

[0039] The distillation module is used to distill the hazard prediction model using the world model, and to use the distilled hazard prediction model to make real-time hazard judgments on intelligent vehicles.

[0040] The early warning and decision-making module is used to generate corresponding early warning signals and decision suggestions based on real-time hazard assessment. For example, when a vehicle is detected approaching rapidly in front of the intelligent vehicle, it can issue an audible alarm and automatically slow down or stop; when a pedestrian is detected suddenly crossing the road in front of the intelligent vehicle, it can issue a vibration reminder and adjust its driving direction to avoid the pedestrian.

[0041] The communication module communicates in real time with surrounding vehicles and road infrastructure (such as traffic lights) via C-V2X, corrects prediction results, receives real-time information from surrounding vehicles and road infrastructure and adjusts driving speed in advance, and sends the intelligent vehicle's status information to surrounding vehicles to remind other vehicles to pay attention and avoid them.

[0042] The process of using a distilled hazard prediction model to perform real-time hazard assessment for intelligent vehicles is as follows: Figure 4 As shown.

[0043] Compared with the prior art, the advantages of the present invention are:

[0044] 1. This invention utilizes multi-source sensors, including LiDAR, visual cameras, and millimeter-wave radar, to collaboratively perceive and acquire multimodal sensor data surrounding intelligent vehicles. It then performs spatiotemporal alignment and fusion processing on this multimodal sensor data to obtain unified spatiotemporal voxel features, thus resolving the asynchronous nature of multi-source data. By employing the spatiotemporal voxel feature alignment mechanism in this invention, the spatiotemporal misalignment between sparse point cloud data and dense RGB features in image data can be resolved, reducing the alignment error to less than 2 cm, achieving a 5-fold improvement in accuracy compared to existing technologies.

[0045] 2. This invention uses a deep learning-based hazard prediction model to predict and warn of potential hazards in the driving process of intelligent vehicles in real time using spatiotemporal voxel features. It can accurately identify potential risk factors in complex traffic environments, including dynamic obstacle trajectory prediction, blind spot hazard warning, and multi-vehicle cooperative conflict detection, thereby improving the accuracy of judgment.

[0046] Furthermore, by introducing a learnable risk attention gating module into the ST-Transformer, the network can focus on regions of high uncertainty, reducing the false alarm rate by 42%.

[0047] 3. This invention, through the world model distillation hazard prediction model, can maintain a prediction accuracy of 93.7% while achieving an end-side inference speed of 30 FPS, meeting real-time safety requirements.

[0048] 4. This invention enhances the driving safety and autonomous decision-making capabilities of intelligent vehicles in complex traffic environments through tiered early warning and automatic control strategies, significantly improving the safety and autonomy of intelligent vehicles. This invention is not only applicable to intelligent vehicles. It can be extended to other autonomous driving vehicles and intelligent transportation systems, possessing broad application prospects and significant social value.

[0049] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0050] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0051] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0052] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0053] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A multimodal spatiotemporal fusion intelligent vehicle hazard prediction method, characterized in that, include: Acquire multimodal sensing data around the intelligent vehicle, including point cloud data collected by lidar, image data collected by camera, and motion data collected by millimeter-wave radar; A deep learning-based hazard prediction model is constructed, comprising a spatiotemporal fusion module and a risk evolution module. The spatiotemporal fusion module performs spatiotemporal alignment and fusion processing on the multimodal sensor data to obtain unified spatiotemporal voxel features. The spatiotemporal fusion module includes a risk attention gating module, which focuses the spatiotemporal alignment and fusion processing on high uncertainty regions. The risk evolution module identifies objects around the intelligent vehicle, predicts the motion trajectory of the objects, and identifies potential collision risks based on the spatiotemporal voxel features. The hazard prediction model is distilled using a world model, and the distilled hazard prediction model is used to make real-time hazard judgments for intelligent vehicles.

2. The multimodal spatiotemporal fusion intelligent vehicle hazard prediction method according to claim 1, characterized in that: The spatiotemporal fusion module performs spatiotemporal alignment and fusion processing on the multimodal sensing data to obtain unified spatiotemporal voxel features, specifically: The spatiotemporal fusion module uses the Transformer architecture as the backbone network, including a graph convolutional neural network, a temporal encoder, and a spatial encoder. The temporal encoder and the spatial encoder contain self-attention modules and convolutional pooling layers. The graph convolutional neural network extracts graph trajectory feature sequences based on the multimodal sensing data, divides the graph trajectory feature sequences into batches according to time and space, and then simultaneously inputs them into the temporal encoder and the spatial encoder. The temporal encoder captures feature associations in the time dimension through a self-attention mechanism to obtain a temporal feature sequence. The spatial encoder uses a self-attention mechanism to capture feature associations in the spatial dimension between different nodes to obtain a spatial feature sequence. The temporal feature sequence, spatial feature sequence, and graph trajectory feature sequence are fused to obtain spatiotemporal voxel features.

3. The multimodal spatiotemporal fusion intelligent vehicle hazard prediction method according to claim 2, characterized in that: The temporal encoder obtains a temporal feature sequence by capturing feature correlations in the temporal dimension through a self-attention mechanism, specifically: Let the timing characteristics of the input time-domain encoder be denoted as . , This represents the i-th node in the temporal features of the input temporal encoder. This represents the t-th time step of the i-th node within the sampling period; i = 1, 2, ..., M, where M represents the number of nodes; t = 1, 2, ..., n, where n represents the number of time steps within the sampling period; Will Projecting these vectors onto the query vector, key vector, and value vector respectively yields the query matrix, key matrix, and value matrix; let the query matrix be denoted as . , , for The query vector obtained by projection; the key matrix is ​​denoted as... , , for The key vector obtained by projection; the value matrix is ​​denoted as , , for The value vector obtained by projection; The temporal feature sequence obtained through the self-attention mechanism is as follows: ， In the formula, Let represent the i-th node in the time-domain feature sequence, T represent the transpose operation, d represent the feature dimension, and softmax represent the softmax function.

4. The multimodal spatiotemporal fusion intelligent vehicle hazard prediction method according to claim 2, characterized in that: The spatial encoder uses a self-attention mechanism to capture the spatial feature correlations between different nodes to obtain a spatial feature sequence, specifically: Let the timing characteristics of the input spatial encoder be denoted as . , This indicates selecting all nodes at time t. Let i represent the i-th node at time t, where i = 1, 2, ..., N, and N represents the number of nodes at time t. The message passing between the i-th node and the j-th node at time t is calculated as follows: ， In the formula, This represents the message passing between the i-th node and the j-th node at time t. It is the query vector obtained by projecting the i-th node at time t. It is the key vector obtained by projecting the j-th node at time t, where T represents the transpose operation. It is the corresponding element in the adjacency matrix used to represent the interaction strength; The spatial feature sequence obtained through the self-attention mechanism is as follows: ， In the formula, This represents the i-th node in the spatial feature sequence. Let represent the value vector obtained by projecting the i-th node at time t. represents the feature dimension, and softmax represents the softmax function.

5. The multimodal spatiotemporal fusion intelligent vehicle hazard prediction method according to claim 2, characterized in that: The spatiotemporal voxel features are as follows: ， In the formula, Represents spatiotemporal voxel characteristics, Represents time-domain feature sequences, Represents spatial feature sequences. Represents the feature sequence of the graph trajectory. This indicates a feature fusion operation. This indicates a feature extraction operation.

6. The multimodal spatiotemporal fusion intelligent vehicle hazard prediction method according to claim 2, characterized in that: The risk evolution module identifies objects around the intelligent vehicle, predicts their trajectories and potential collision risks based on the spatiotemporal voxel features, specifically: The risk evolution module identifies the object categories around the intelligent vehicle based on the spatiotemporal voxel features corresponding to the image data, identifies the distance and shape of the objects around the intelligent vehicle based on the spatiotemporal voxel features corresponding to the point cloud data, predicts the motion trajectory of the objects based on the spatiotemporal voxel features corresponding to the motion data, and judges potential collision risks by combining the object categories, object distances and shapes, and object motion trajectories. When predicting the motion trajectory of an object based on the spatiotemporal voxel features corresponding to the motion data, a mask matrix is ​​used to ensure that the information at each position only focuses on the portion at subsequent positions.

7. The multimodal spatiotemporal fusion intelligent vehicle hazard prediction method according to claim 6, characterized in that: The method of using a mask matrix to ensure that information at each position only focuses on subsequent positions is as follows: The feature sequence for constructing the input decoder is: ， In the formula, This represents the feature sequence input to the decoder. The starting marker consists of all zeros. This is a sequence extracted from the feature sequence of the graph trajectory. Indicates feature fusion operation; according to Constructing the query matrix, key matrix, and value matrix yields the feature sequence output by the decoder: ， In the formula, Q represents the feature sequence output by the decoder, where Q represents the sequence based on the decoder output. The constructed query matrix, where K represents the query matrix based on... The constructed key matrix, V represents the key matrix constructed according to... The constructed value matrix, The dimension represents the feature dimension, and M represents the mask matrix.

8. The multimodal spatiotemporal fusion intelligent vehicle hazard prediction method according to any one of claims 2-7, characterized in that: When the spatiotemporal alignment and fusion processing is focused on a high-uncertainty region using the risk attention gating module, the uncertainty is calculated as follows: ， In the formula, (x,y,t) represents the spatiotemporal coordinates, where x and y represent the x-coordinates and y-coordinates in the spatial dimension, and t represents the time dimension. Indicates uncertainty, This represents the probability distribution output by the hazard prediction model. Represents the L2 norm; The method for calculating the learnable gating vector is as follows: ， In the formula, This represents a learnable gated vector. This represents the sigmoid activation function. Represents the gate weight matrix. This represents the spatiotemporal voxel features input to the risk attention gating module. This indicates the gating bias term.

9. The multimodal spatiotemporal fusion intelligent vehicle hazard prediction method according to claim 8, characterized in that: After the risk attention gating module focuses the spatiotemporal alignment and fusion process on the high uncertainty region, the final spatiotemporal voxel features are: ， In the formula, ⊙ represents the final spatiotemporal voxel feature, and ⊙ represents the Hadamard product.

10. A multimodal spatiotemporal fusion intelligent vehicle hazard prediction system, characterized in that, include: A multimodal data acquisition module, including a lidar, a camera, and a millimeter-wave radar, is installed on an intelligent vehicle and collects multimodal sensing data around the intelligent vehicle in real time; the multimodal sensing data includes point cloud data collected by the lidar, image data collected by the camera, and motion data collected by the millimeter-wave radar. A hazard prediction model module is used to construct a deep learning-based hazard prediction model. The hazard prediction model includes a spatiotemporal fusion module and a risk evolution module. The spatiotemporal fusion module performs spatiotemporal alignment and fusion processing on the multimodal sensor data to obtain unified spatiotemporal voxel features. The spatiotemporal fusion module includes a risk attention gating module, which focuses the spatiotemporal alignment and fusion processing on high uncertainty regions. The risk evolution module identifies objects around the intelligent vehicle, predicts the motion trajectory of objects, and identifies potential collision risks based on the spatiotemporal voxel features. The distillation module is used to distill the hazard prediction model using the world model, and to use the distilled hazard prediction model to make real-time hazard judgments on intelligent vehicles. The early warning and decision-making module is used to generate corresponding early warning signals and decision suggestions based on real-time hazard assessments. The communication module communicates with surrounding vehicles and road infrastructure in real time via C-V2X to correct prediction results.

Images