Automatic driving collision risk assessment method and system based on sparse soft supergraph learning

Abstract

The application discloses an automatic driving collision risk assessment method and system based on sparse soft supergraph learning, and the method comprises the following steps: acquiring a continuous traffic scene image sequence in the driving process of an automatic driving vehicle to form a traffic scene node set; constructing a soft supergraph structure, generating a node-superedge soft association matrix, and performing sparse screening to obtain a sparse soft supergraph structure; performing supergraph message propagation to obtain traffic risk features fused with high-order interaction information; acquiring global environment features corresponding to the traffic scene, performing attention fusion, and obtaining spatiotemporal fusion risk features; performing time series modeling on frame-level spatial fusion risk features to predict the risk probability of a collision event occurring in the current traffic scene, and outputting a collision risk assessment result. The application can improve the accuracy of collision risk assessment in complex traffic scenes. The application can be widely applied to the technical field of computer vision as an automatic driving collision risk assessment method and system based on sparse soft supergraph learning.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for assessing collision risks in autonomous driving based on sparse soft hypergraph learning. Background Technology

[0002] With the rapid development of autonomous driving technology, assessing the collision risk between autonomous vehicles and various road users during driving has become a critical issue. Unlike accident detection after a collision has occurred, collision risk assessment requires inferring potential hazardous situations based on limited historical observation frames before an accident has occurred or fully manifested. Therefore, this task not only requires the model to understand traffic objects in the current image, but also to understand the spatial relationships between traffic participants, short-term motion changes, and the overall road environment.

[0003] Existing collision risk assessment methods can be broadly categorized into trajectory-based methods, methods based on overall video visual features, and methods based on traffic scene graphs. However, in real-world autonomous driving scenarios, changes in lighting, rain / snow, and viewing angle can affect the quality of trajectory extraction, and relying solely on trajectory relationships is insufficient to fully represent the visual semantic information in complex scenes. Furthermore, overall video visual feature methods lack explicit representation of the interactions between traffic entities, making it difficult to identify which vehicle, pedestrian, or entity combinations lead the model to perceive a high-risk scenario. In addition, traditional scene graph structures typically employ a pairwise connection approach, only expressing binary relationships between two nodes. Summary of the Invention

[0004] To address the aforementioned technical problems, the present invention aims to provide an autonomous driving collision risk assessment method and system based on sparse soft hypergraph learning, which can improve the accuracy of collision risk assessment in complex traffic scenarios.

[0005] The first technical solution adopted in this invention is: an autonomous driving collision risk assessment method based on sparse soft hypergraph learning, comprising the following steps:

[0006] Acquire a continuous sequence of traffic scene images during the driving process of an autonomous vehicle, detect and extract features of traffic participants in the traffic scene image sequence, construct entity node features corresponding to each traffic participant, and form a traffic scene node set based on several entity node features.

[0007] A soft hypergraph structure is constructed based on the traffic scene node set. A node-hyperedge soft correlation matrix is ​​generated to represent the high-order interaction relationship of multiple traffic participants. The candidate hyperedges are assessed for risk correlation and sparse screening is performed to obtain the sparse soft hypergraph structure.

[0008] Hypergraph message propagation is performed based on sparse soft hypergraph structure to transmit and update the group interaction information of multiple traffic participants to the corresponding entity node features, thereby obtaining traffic risk features that integrate high-order interaction information.

[0009] The global environmental features corresponding to the traffic scene are obtained, and the traffic risk features that integrate high-order interaction information are fused with the global environmental features by attention fusion to obtain frame-level spatial fusion risk features.

[0010] Time series modeling is performed on frame-level spatial fusion risk features to predict the probability of collision events in the current traffic scenario and output collision risk assessment results.

[0011] Furthermore, the step of detecting and extracting features from traffic participants in the traffic scene image sequence specifically includes:

[0012] The target detection network is used to identify targets in each image frame of a traffic scene image sequence to determine the location regions of vehicles, pedestrians, non-motorized vehicles and other road participants;

[0013] Extract the location parameters, category information and visual feature information of the corresponding traffic participants from each location area. The location parameters include at least one or more of the following: target center coordinates, width, height and detection confidence.

[0014] Furthermore, the step of constructing the entity node features corresponding to each traffic participant specifically includes:

[0015] Generate spatial geometric features based on the location parameters of traffic participants;

[0016] Generate semantic category features based on the category information of traffic participants;

[0017] Extracting visual appearance features based on visual feature information of traffic participants;

[0018] By performing feature splicing and mapping on spatial geometric features, semantic category features, and visual appearance features, entity node features of each traffic participant are obtained.

[0019] Furthermore, the step of generating a node-hyperedge soft correlation matrix to characterize the high-order interaction relationships among multiple traffic participants specifically includes:

[0020] Based on the entity node characteristics of each traffic participant in the traffic scene node set, a dynamic superedge representation of the corresponding candidate superedge is generated.

[0021] Calculate the similarity between the features of each entity node and the representation of each dynamic hyperedge to obtain the association weight between the corresponding node and the hyperedge;

[0022] All association weights are normalized to obtain a node-hyperedge soft association matrix. The elements in the node-hyperedge soft association matrix are continuous values, which are used to characterize the degree of participation of the corresponding traffic participant node in the corresponding hyperedge.

[0023] Furthermore, the step of assessing the risk correlation of candidate hyperedges and performing sparse screening to obtain a sparse soft hypergraph structure specifically includes:

[0024] Based on the association weight distribution of each candidate hyperedge in the node-hyperedge soft correlation matrix, calculate the risk activation weight of each candidate hyperedge;

[0025] All candidate superedges are sorted from high to low according to their risk activation weight;

[0026] Candidate hyperedges whose risk activation weights meet the preset threshold conditions are selected as target hyperedges, and low-correlation candidate hyperedges are eliminated to form a sparse soft hypergraph structure.

[0027] Furthermore, the hypergraph message propagation includes the information aggregation process from nodes to hyperedges and the information propagation process from hyperedges to nodes, wherein:

[0028] For the information aggregation process from nodes to hyperedges, the entity node features of each traffic participant within the same target hyperedge are weighted and fused according to the association weights corresponding to the node-hyperedge soft association matrix to generate hyperedge interaction features.

[0029] For the information propagation process from the hyperedge to the node, the hyperedge interaction features are transmitted to the entity node features of each traffic participant according to the participation level of the corresponding node, and the node features of each entity node are updated to obtain traffic risk features that integrate high-order interaction information.

[0030] Furthermore, the step of performing attention fusion between traffic risk features that integrate high-level interaction information and global environmental features specifically includes:

[0031] Traffic risk features that integrate high-level interactive information are used as query vectors, and global environmental features are used as key and value vectors.

[0032] The association weights of the query vector, key vector, and value vector are calculated through a cross-attention mechanism, and the global environmental features are weighted and filtered to obtain the weighted global environmental features.

[0033] The weighted global environmental features are spliced ​​and fused with traffic risk features that integrate high-order interactive information to obtain frame-level spatial fusion risk features. The global environmental features include at least one of road structure information, illumination information, weather information, background environment information, and unidentified potential risk information.

[0034] Furthermore, the step of performing time-series modeling on frame-level spatial fusion risk features to predict the probability of a collision event occurring in the current traffic scenario and outputting a collision risk assessment result specifically includes:

[0035] Temporal convolutional networks are used to extract short-term motion change features of frame-level spatial risk features corresponding to consecutive image frames;

[0036] Based on short-term motion change characteristics, a long-distance temporal encoder is used to capture the temporal dependencies in the long-term temporal dimension to obtain temporal correlation risk characteristics;

[0037] Perform a fully connected mapping on the temporally correlated risk features to output the probability of a collision event occurring in the current traffic scenario.

[0038] Furthermore, it also includes:

[0039] Based on the risk activation weights of each target hyperedge in the sparse soft hypergraph structure and the degree of participation of each entity node feature in the corresponding hyperedge, the key traffic participants and key interaction relationships that affect the collision risk prediction results are located.

[0040] Based on the key traffic participants and key interaction relationships that affect the collision risk prediction results, the risk association structure explanation information of the corresponding traffic scenario is output.

[0041] The second technical solution adopted in this invention is: an autonomous driving collision risk assessment system based on sparse soft hypergraph learning, comprising:

[0042] The first module is used to acquire a continuous sequence of traffic scene images during the driving process of autonomous vehicles, detect and extract features of traffic participants in the traffic scene image sequence, construct entity node features corresponding to each traffic participant, and form a traffic scene node set based on several entity node features.

[0043] The second module is used to construct a soft hypergraph structure based on the traffic scene node set, generate a node-hyperedge soft correlation matrix to represent the high-order interaction relationship of multiple traffic participants, conduct risk correlation assessment on candidate hyperedges and perform sparse screening to obtain a sparse soft hypergraph structure.

[0044] The third module is used for hypergraph message propagation based on sparse soft hypergraph structure, which transmits and updates the group interaction information of multiple traffic participants to the corresponding entity node features, and obtains traffic risk features that integrate high-order interaction information.

[0045] The fourth module is used to obtain the global environmental features corresponding to the traffic scene. It performs attention fusion between the traffic risk features that integrate high-order interaction information and the global environmental features to obtain frame-level spatial fusion risk features.

[0046] The fifth module is used to perform time series modeling on frame-level spatial fusion risk features, predict the probability of collision events in the current traffic scenario, and output collision risk assessment results.

[0047] The beneficial effects of the method and system of this invention are as follows: This invention acquires a continuous traffic scene image sequence during the driving process of an autonomous vehicle, detects and extracts features from traffic participants in the traffic scene image sequence, constructs entity node features corresponding to each traffic participant, and forms a traffic scene node set based on several entity node features; further, it constructs a soft hypergraph structure based on the traffic scene node set, generates a node-hyperedge soft correlation matrix to represent the high-order interaction relationship of multiple traffic participants, performs risk correlation assessment on candidate hyperedges and performs sparse screening to obtain a sparse soft hypergraph structure, and uses the hypergraph structure to characterize the high-order interaction relationship of multiple traffic participants; at the same time, it uses a continuous value soft correlation matrix to describe the participation degree of nodes and hyperedges, which can accurately represent the strength and uncertainty of the interaction, avoid the division error of hard hypergraphs that are either 0 or 1, and has stronger adaptability to complex traffic flow and multi-participant mixed traffic scenarios. The risk correlation assessment of candidate hyperedges... The system dynamically filters edges, retaining core hyperedges with high-risk associations and eliminating redundant low-value interactions. This reduces computational complexity without losing key risk information, effectively balancing model evaluation accuracy and inference speed to meet the real-time operational requirements of onboard autonomous driving systems. Furthermore, it uses a sparse soft hypergraph structure for hypergraph message propagation, transmitting group interaction information from multiple traffic participants to corresponding entity node features, resulting in traffic risk features fused with higher-order interaction information. Next, it acquires global environmental features corresponding to the traffic scenario and adaptively fuses the local traffic risk features fused with higher-order interaction information with the global environmental features through cross-attention, obtaining spatiotemporal fused risk features that simultaneously cover both human-vehicle interaction risks and environmental trigger risks, avoiding evaluation bias caused by a single feature dimension. Finally, it performs time-series modeling on the frame-level fused risk features to predict the probability of a collision event in the current traffic scenario and outputs the collision risk assessment results. The spatial dimension aggregates multi-agent interaction information through hypergraph message propagation and integrates global environmental features through a cross-attention mechanism. The temporal dimension combines a temporal convolutional network and a long-range temporal encoder to capture short-term motion changes and long-term temporal dependencies, respectively, to achieve spatiotemporal joint modeling of continuous traffic scenarios. It can capture the dynamic evolution trend of risks and is more forward-looking than single-frame static assessment. It has a higher accuracy in collision risk prediction and can improve the accuracy of collision risk assessment in complex traffic scenarios. It can also output structured risk interpretation information based on key nodes and key hyperedges. Attached Figure Description

[0048] Figure 1 This is a flowchart of the steps of the autonomous driving collision risk assessment method based on sparse soft hypergraph learning in this invention;

[0049] Figure 2 This is a block diagram of the autonomous driving collision risk assessment system based on sparse soft hypergraph learning of the present invention;

[0050] Figure 3 This is a schematic diagram of the structural framework of an autonomous driving collision risk assessment method based on sparse soft hypergraph learning provided in a specific embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram of a visual risk factor prediction result based on the perspective of a vehicle's front-facing camera, provided by a specific embodiment of the present invention. Detailed Implementation

[0052] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0053] Reference Figure 1 This invention provides a method for assessing collision risks in autonomous driving based on sparse soft hypergraph learning, which includes the following steps:

[0054] S100. Obtain a continuous traffic scene image sequence during the autonomous vehicle's driving process, detect and extract features of traffic participants in the traffic scene image sequence, construct entity node features corresponding to each traffic participant, and form a traffic scene node set based on several entity node features.

[0055] S110. Use an object detection network to identify objects in each image frame of the traffic scene image sequence and determine the location areas of vehicles, pedestrians, non-motorized vehicles and other road participants.

[0056] In this embodiment, a continuous sequence of traffic scene images is acquired by the autonomous vehicle during its operation. The traffic scene image sequence can be obtained from an onboard forward-facing camera, a surround-view camera, or other sensors capable of capturing images of the traffic environment.

[0057] S120. Extract the location parameters, category information and visual feature information of the corresponding traffic participants from each location area. The location parameters include at least one or more of the following: target center coordinates, width, height and detection confidence.

[0058] In this embodiment, entity node features are constructed based on the location parameters, category information, and visual appearance information of each traffic participant. The location parameters may include the center coordinates, width, height, and detection confidence of the target bounding box; the category information can be represented by a learnable category embedding vector; and the visual appearance information can be obtained by feature extraction from the target image region. The entity node features of the corresponding traffic participant are obtained by fusing the spatial geometric features, category semantic features, and visual appearance features.

[0059] S200. Construct a soft hypergraph structure based on the traffic scene node set, generate a node-hyperedge soft correlation matrix to represent the high-order interaction relationship of multiple traffic participants, conduct risk correlation assessment on candidate hyperedges and perform sparse screening to obtain a sparse soft hypergraph structure.

[0060] S210. Based on the entity node characteristics of each traffic participant in the traffic scene node set, generate a dynamic superedge representation of the corresponding candidate superedge.

[0061] S220. Calculate the similarity between the features of each entity node and the representation of each dynamic hyperedge to obtain the association weight between the corresponding node and the hyperedge.

[0062] S230. Normalize all association weights to obtain a node-hyperedge soft association matrix. The elements in the node-hyperedge soft association matrix are continuous values, which are used to characterize the degree of participation of the corresponding traffic participant node in the corresponding hyperedge.

[0063] In this embodiment, a soft hypergraph structure is constructed based on the entity nodes. The soft hypergraph structure includes a set of nodes, a set of hyperedges, and a node-hyperedge soft association matrix. Unlike a regular graph where an edge connects only two nodes, a hyperedge in a hypergraph can connect multiple nodes simultaneously, thus representing higher-order interaction relationships formed by multiple traffic participants.

[0064] S240. Calculate the risk activation weight of each candidate hyperedge based on the association weight distribution of each candidate hyperedge in the node-hyperedge soft correlation matrix.

[0065] S250. Sort all candidate superedges from high to low according to their risk activation weights;

[0066] S260. Select candidate hyperedges whose risk activation weights meet the preset threshold conditions as target hyperedges, and eliminate candidate hyperedges with low correlation to form a sparse soft hypergraph structure.

[0067] Furthermore, candidate hyperedges, including fixed and dynamic hyperedges, are generated, and node-hyperedge association weights are calculated. Specifically, a learnable initial hyperedge prototype can be set, and scene-adaptive hyperedge prototype offsets are generated based on the information of all node features in the current frame. Node features are linearly projected and their similarity to the dynamic hyperedge prototype is calculated to obtain the affinity between nodes and candidate hyperedges. After normalizing the affinity, a node-hyperedge soft association matrix is ​​obtained.

[0068] Top-K algorithm is used to sparsely select candidate hyperedges. For each dynamic hyperedge, the overall correlation strength between the hyperedge and each entity node is calculated to obtain the hyperedge activation score. Candidate hyperedges are then screened based on their activation level or risk correlation, eliminating redundant hyperedges and retaining key hyperedges to form a sparse soft hypergraph structure.

[0069] S300: Hypergraph message propagation is performed based on sparse soft hypergraph structure to transmit and update the group interaction information of multiple traffic participants to the corresponding entity node features, thereby obtaining traffic risk features that integrate high-order interaction information.

[0070] S310. For the information aggregation process from node to hyperedge, according to the association weights corresponding to the node-hyperedge soft association matrix, the entity node features of each traffic participant within the same target hyperedge are weighted and fused to generate hyperedge interaction features.

[0071] S320. For the information propagation process from the hyperedge to the node, according to the participation level of the corresponding node, the hyperedge interaction features are transmitted to the entity node features of each traffic participant, the node features of each entity node are updated, and traffic risk features that integrate high-order interaction information are obtained.

[0072] In this embodiment, hypergraph message propagation is performed based on a sparse soft hypergraph structure, including an information aggregation process from nodes to hyperedges and an information diffusion process from hyperedges to nodes. The information aggregation process from nodes to hyperedges is used to weight and fuse the features of multiple entity nodes belonging to the same hyperedge according to the degree of node-hyperedge participation to obtain hyperedge interaction features; the information diffusion process from hyperedges to nodes is used to feed back the group interaction information represented by the hyperedge to the corresponding entity nodes, thereby enhancing the nodes' ability to express higher-order interaction relationships.

[0073] S400: Obtain the global environmental features corresponding to the traffic scene, and perform attention fusion between the traffic risk features that integrate high-order interaction information and the global environmental features to obtain frame-level spatial fusion risk features.

[0074] S410. Use traffic risk features that integrate high-level interactive information as query vectors and global environmental features as key and value vectors.

[0075] S420. The association weights of the query vector, key vector and value vector are calculated through the cross-attention mechanism, and the global environmental features are weighted and filtered to obtain the weighted global environmental features.

[0076] S430. The weighted global environmental features are spliced ​​and fused with the traffic risk features of the fused high-order interactive information to obtain frame-level spatial fusion risk features. The global environmental features include at least one of road structure information, illumination information, weather information, background environment information and unidentified potential risk information.

[0077] In this embodiment, global environmental features of the traffic scene are extracted. These global environmental features can be obtained by convolutional neural networks, visual Transformers, or other visual coding networks.

[0078] A cross-attention mechanism is employed to fuse local high-order interaction features with global environment features. Specifically, the local interaction features obtained after sparse soft hypergraph message propagation are used as query vectors, and the global environment features are used as key and value vectors. An attention mechanism is used to retrieve environmental information related to the current risk interaction from the global scene, resulting in a fused frame-level risk representation.

[0079] S500 performs time-series modeling on frame-level spatial fusion risk features to predict the probability of collision events in the current traffic scenario and outputs collision risk assessment results.

[0080] S510. Utilize a temporal convolutional network to extract short-term motion change features of frame-level spatial risk features corresponding to consecutive image frames;

[0081] S520. Based on short-term motion change characteristics, a long-distance temporal encoder is used to capture the temporal dependencies in the long-term temporal dimension to obtain temporal correlation risk characteristics.

[0082] S530. Perform a fully connected mapping on the temporal correlation risk features and output the probability of a collision event occurring in the current traffic scenario.

[0083] In this embodiment, the fusion risk features of consecutive frames are time-series modeled, and the probability of a collision event in a traffic scene is output by a classification prediction head.

[0084] Output structured risk interpretation information. Based on the selected hyperedges in the sparse soft hypergraph, hyperedge sensitivity scores, and node participation in hyperedges, identify traffic participants and group interactions that significantly impact collision risk prediction results, and output this as risk source inspection information. This risk interpretation information can be used to assist autonomous driving decision-making systems, driver warning systems, or accident recap systems in understanding the basis of model predictions.

[0085] Finally, the specific implementation process of the embodiments of the present invention will be explained and described in conjunction with the accompanying drawings:

[0086] The following sections will provide a detailed description and explanation of the solutions in this application embodiment, using specific application examples. In the following embodiments, continuous traffic video from the vehicle's perspective, captured by an in-vehicle camera, will be used as input. In practical applications, the input can also be a surround-view camera, a multi-camera stitched image, image features fused from a camera and millimeter-wave radar or lidar, or other time-series data that reflects the traffic scene status.

[0087] First, a sequence of traffic scene images is acquired based on the viewpoint of the vehicle's front-facing camera. During operation, the autonomous vehicle uses its onboard camera to capture continuous image frames, forming video clips. ,in Indicates the first Frame image, This indicates the length of the video segment. This video segment corresponds to one collision risk prediction result. , This indicates the probability that the video clip poses a collision risk.

[0088] Traffic participant detection is performed on each frame of the image. The object detection module detects vehicles, pedestrians, cyclists, buses, trucks, motorcycles, and other objects that may be involved in traffic interactions in the image, and outputs the bounding box position, detection confidence score, and class label for each object. The bounding box positions can be normalized to the image size range so that a unified representation can be shared between images of different resolutions.

[0089] Construct the entity node feature matrix for the current frame. For the The first detected in the frame image Obtain the geometric features of each traffic participant. Category characteristics and appearance features Geometric features consist of the bounding box center coordinates, width, height, and confidence score; class features are generated by a learnable embedding layer; appearance features are obtained by visual feature extraction of the target bounding box region. These features are concatenated and mapped through a multilayer perceptron to obtain node features. .

[0090] If the current frame detects For each traffic participant, its node features are combined into a node feature matrix. If the number of detected objects differs across frames, a batch input with uniform dimensions can be constructed using padding nodes and masking mechanisms, ignoring the padding nodes in subsequent pooling or attention calculations.

[0091] Candidate hyperedges for a soft hypergraph are generated, and the node-hyperedge soft correlation matrix is ​​calculated. M candidate hyperedge prototypes are preset, which may include fixed and dynamic hyperedge prototypes. The fixed hyperedge prototype serves as a stable interaction basis, while the dynamic hyperedge prototype adaptively adjusts according to the node distribution of the current frame. A scene description vector is obtained by performing average pooling and max pooling on node features, and a hyperedge prototype offset is generated using an offset prediction network, thus obtaining a dynamic hyperedge prototype that matches the current traffic scene.

[0092] Node features are projected onto the same feature space as the hyperedge prototype, and the similarity between the node features and the hyperedge prototype is calculated. To improve expressive power, a multi-head similarity calculation method can be used, dividing the node features and the hyperedge prototype into multiple subspaces, calculating the similarity in each subspace, and then fusing them. Specifically, the mathematical calculation process for the similarity S in a single subspace can be expressed as:

[0093]

[0094] in, It refers to the node features within the current subspace. It is the prototype representation of the hyperedge of the current subspace. It is the dimension within each subspace.

[0095] Normalize the fusion similarity along the node dimension to obtain the soft correlation matrix A. Indicates the first The node participates in the first The degree of candidate superedges. Specifically, The calculation can be expressed as:

[0096]

[0097] Sparse hyperedge selection is performed. For dynamic candidate hyperedges, the overall correlation strength between them and all nodes in the current frame is calculated, which is used as the activation score of the hyperedge. Dynamic candidate hyperedges are sorted according to their activation scores, and the top K dynamic hyperedges are selected, which, together with the fixed hyperedges, form the sparse hyperedge set for the current frame. This yields the sparse soft correlation matrix. .

[0098] The hypergraph node information update process includes information aggregation from nodes to hyperedges and information diffusion from hyperedges to nodes. Specifically, for each hyperedge in the sparse hyperedge set, the node features are weighted and summed according to the degree of node participation in that hyperedge to obtain the group interaction representation of that hyperedge. For the m-th soft hyperedge, this information aggregation process can be represented as:

[0099]

[0100] in, Represents a node For superedge The level of participation. For the first Aggregation features of superedges.

[0101] Subsequently Perform linear transformations and nonlinear activations to obtain the updated hyperedge features. The specific mathematical process can be expressed as follows:

[0102]

[0103] For each entity node, based on the association weights of the multiple hyperedges it participates in, the corresponding hyperedge features are weighted and fed back to the node, resulting in a node representation carrying group interaction information. For the ... The information diffusion process from a node to a node via a hyperedge can be expressed as:

[0104]

[0105]

[0106] in, It is the updated hyperedge representation. Represents a node For superedge The level of participation It is a learnable transformation matrix. This is the activation function. This bidirectional information transfer process ensures that the updated node includes not only its own geometric, category, and appearance attributes, but also risk relationship information within its multi-agent interaction combination.

[0107] Extract global environment context features. (The sentence appears to be incomplete and requires further context.) A complete frame of image is input into a global visual feature extraction network to obtain global environmental features. And by projecting it onto the same dimension as the local interaction representation through a multilayer perceptron, we obtain... Global environmental features are used to represent road layout, traffic density, lighting conditions, weather conditions, background occlusion, and potential risk information outside the target bounding box.

[0108] Local and global features are fused through cross-attention. Local feature representations are used as query vectors, and global environment features are used as key vector K and value vector V, respectively. Cross-attention fusion is then performed to obtain a frame-level fused representation. .

[0109] Specifically, mathematical operations can be expressed as:

[0110]

[0111] in, This represents the local interaction of the current frame. This refers to global environmental characteristics.

[0112] Then and Perform residual connections and normalization to obtain a frame-level fusion risk representation. :

[0113]

[0114] Perform temporal local motion enhancement. Represent the fusion risk for consecutive frames. One-dimensional convolution is performed along the time dimension to capture short-term motion change features between adjacent frames. These short-term motion change features include, but are not limited to, risk precursors such as sudden braking, rapid approach, lateral movement, sudden appearance of pedestrians, and removal of target occlusion. Local motion enhancement can be represented as:

[0115]

[0116] in and These are the learnable parameters of the temporal convolution module. Then, the temporal convolution output is added to the original frame-level representation to obtain the motion-enhanced frame-level features. .

[0117] Temporal risk modeling is performed. The motion-enhanced frame-level feature sequence is input into a temporal Transformer encoder. Learnable class labels are added before the input sequence, and positional encoding is incorporated to enable the model to perceive temporal order. The resulting sequence can be represented as... Subsequently, the temporal Transformer models the long-term dependencies between different time frames through a self-attention mechanism, outputting a video-level representation.

[0118] Output risk probability and structured interpretability risk assessment results. The video-level representation is input into a linear classifier or multilayer perceptron classifier, and the collision risk probability is obtained through a sigmoid function. .when When the value exceeds a preset threshold, a warning result indicating a collision risk is output; when... If the risk level is below a preset threshold, an assessment result indicating that the current segment has a low risk is output. This threshold can be configured based on vehicle safety policies, false alarm tolerance, and application scenarios.

[0119] In general, refer to Figure 3The interpretable collision risk assessment method based on sparse soft hypergraph provided in this embodiment consists of three main modules: local context modeling, global context modeling, and temporal modeling. Finally, it obtains the risk judgment result and structured risk factor location result of the vehicle video.

[0120] Specifically, refer to Figure 4 The structured risk factor localization results determine key traffic participants and key hyperedge combinations based on the activation scores, sensitivity scores, and node-hyperedge participation levels of selected hyperedges in the sparse soft hypergraph. The system can output key objects as bounding boxes, highlighted markers, bird's-eye view nodes, or text prompts to aid in interpreting the model's risk prediction results.

[0121] Furthermore, in one training implementation, a driving video dataset containing pre-collision segments and non-accident segments is acquired. For pre-collision segments, only video frames prior to the collision event are used as input to train the model to perform risk assessment before the accident occurs. For non-accident segments, training samples can be constructed using random cropping or fixed-length sampling to avoid the model learning shortcuts solely based on sequence length.

[0122] During training, video clips are labeled at the video level: a label of 1 indicates a clip at pre-collision risk, and a label of 0 indicates a clip not involved in an accident. The model can learn the sparse soft hypergraph structure and risk prediction parameters through video-level classification loss without requiring node-level, hyperedge-level, or frame-level risk source annotations. When there is class imbalance, weighted cross-entropy loss, focus loss, or other loss functions that can alleviate class imbalance can be used.

[0123] In one implementation, the trained model is deployed on the onboard computing platform of an autonomous vehicle or on a roadside edge computing device. The system acquires real-time traffic image sequences in front of or around the vehicle and outputs risk probabilities according to a preset time window. When the risk probability exceeds a safety threshold, the system can send risk warnings to the path planning module, emergency braking module, or human-machine interaction module.

[0124] In another application, the present invention can also be used for offline traffic accident review, autonomous driving test data screening, hazardous scenario mining, and early warning of driver assistance systems. By outputting key risk entities and key interaction hyperedges, the system can help engineers analyze combinations of risk objects of interest to the model, thereby improving the efficiency of data annotation, model debugging, and safety verification.

[0125] In summary, the operation flow of this embodiment of the invention is as follows:

[0126] 1) Acquire a continuous sequence of traffic scene images collected by the autonomous vehicle during its operation. The traffic scene image sequence can be obtained by an onboard forward-facing camera, a surround-view camera, or other sensors capable of collecting traffic environment images.

[0127] 2) Perform traffic participation object detection on each frame of the image sequence to obtain the location parameters, category information and detection confidence of the road participants.

[0128] 3) Construct entity node features based on the location parameters, category information, and visual appearance information of each traffic participant.

[0129] 4) Construct a soft hypergraph structure based on the entity nodes. The soft hypergraph structure includes a set of nodes, a set of hyperedges, and a node-hyperedge soft association matrix. In this invention, the node-hyperedge relationship does not use binary hard association, but continuous soft association.

[0130] 5) Calculate the similarity between the learnable initial hyperedge prototype and the current frame node features, generate candidate hyperedges, and calculate the node-hyperedge association weights and dynamic hyperedge prototype P.

[0131] 6) Perform Top-K sparse selection on candidate superedges. This invention filters candidate superedges based on their activation level or risk relevance, eliminating weakly correlated or security-irrelevant interactions, retaining key superedges, and forming a sparse soft hypergraph structure.

[0132] 7) Hypergraph message propagation based on sparse soft hypergraph structure. The hypergraph message propagation includes a node-to-hyperedge information aggregation process and a hyperedge-to-node information diffusion process. The node-to-hyperedge information aggregation process is used to weight and fuse the features of multiple entity nodes belonging to the same hyperedge according to the node-hyperedge participation degree to obtain hyperedge interaction features; the hyperedge-to-node information diffusion process is used to feed back the group interaction information represented by the hyperedge to the corresponding entity node, thereby enhancing the node's ability to express higher-order interaction relationships.

[0133] 8) Extract global environmental features of the traffic scene. The global environmental features can be obtained by convolutional neural networks, visual Transformers, or other visual coding networks.

[0134] 9) A cross-attention mechanism is used to fuse local high-order interaction features with global environment features. Specifically, the local interaction features obtained after sparse soft hypergraph message propagation are used as query vectors, and the global environment features are used as key and value vectors. An attention mechanism is used to retrieve environmental information related to the current risk interaction from the global scene, resulting in a fused frame-level risk representation.

[0135] 10) Perform temporal modeling on the fused risk features of consecutive frames to output the collision risk probability. The temporal modeling may include local temporal convolution and temporal Transformer encoding. Finally, output the probability of a collision event occurring in the traffic scene through a classification prediction head.

[0136] 11) Output structured risk interpretation information. Based on the selected hyperedges, hyperedge sensitivity scores, and node participation in the hyperedge in the sparse soft hypergraph, identify the traffic participants and group interactions that significantly impact collision risk prediction results, and output this as risk source inspection information. This risk interpretation information can be used to assist autonomous driving decision-making systems, driver warning systems, or accident recap systems in understanding the basis of model predictions.

[0137] This invention acquires a continuous sequence of traffic scene images during the operation of an autonomous vehicle, detects traffic participants such as vehicles and pedestrians in each frame, and constructs entity node features based on position parameters, category information, and visual appearance information. A soft hypergraph is constructed based on these entity nodes, using a continuous node-hyperedge association matrix to represent high-order interaction relationships between multiple traffic participants, and key risk interactions are retained through sparse hyperedge selection. Furthermore, hypergraph message propagation is performed, fusing local high-order interaction features with global environmental features through cross-attention, and temporal modeling is performed on continuous frame features to output collision risk probabilities. This improves the accuracy of collision risk assessment in complex traffic scenarios and can output structured risk explanation information based on key nodes and key hyperedges.

[0138] Reference Figure 2 An autonomous driving collision risk assessment system based on sparse soft hypergraph learning includes:

[0139] The first module 201 is used to acquire a continuous traffic scene image sequence during the driving process of an autonomous vehicle, detect and extract features of traffic participants in the traffic scene image sequence, construct entity node features corresponding to each traffic participant, and form a traffic scene node set based on several entity node features.

[0140] The second module 202 is used to construct a soft hypergraph structure based on the traffic scene node set, generate a node-hyperedge soft correlation matrix to represent the high-order interaction relationship of multiple traffic participants, conduct risk correlation assessment on candidate hyperedges and perform sparse screening to obtain a sparse soft hypergraph structure.

[0141] The third module 203 is used for hypergraph message propagation based on sparse soft hypergraph structure, which transmits and updates the group interaction information of multiple traffic participants to the corresponding entity node features, and obtains traffic risk features that integrate high-order interaction information.

[0142] The fourth module 204 is used to obtain the global environmental features corresponding to the traffic scene. It performs attention fusion between the traffic risk features that integrate high-order interaction information and the global environmental features to obtain frame-level spatial fusion risk features.

[0143] The fifth module 205 is used to perform time series modeling on frame-level spatial fusion risk features, predict the probability of collision events in the current traffic scenario, and output collision risk assessment results.

[0144] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0145] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this is not intended to limit the scope of the embodiments of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. An autonomous driving collision risk assessment method based on sparse soft hypergraph learning, characterized in that, Includes the following steps: Acquire a continuous sequence of traffic scene images during the driving process of an autonomous vehicle, detect and extract features of traffic participants in the traffic scene image sequence, construct entity node features corresponding to each traffic participant, and form a traffic scene node set based on several entity node features. A soft hypergraph structure is constructed based on the traffic scene node set. A node-hyperedge soft correlation matrix is ​​generated to represent the high-order interaction relationship of multiple traffic participants. The candidate hyperedges are assessed for risk correlation and sparse screening is performed to obtain the sparse soft hypergraph structure. Hypergraph message propagation is performed based on sparse soft hypergraph structure to transmit and update the group interaction information of multiple traffic participants to the corresponding entity node features, thereby obtaining traffic risk features that integrate high-order interaction information. The global environmental features corresponding to the traffic scene are obtained, and the traffic risk features that integrate high-order interaction information are fused with the global environmental features by attention fusion to obtain frame-level spatial fusion risk features. Time series modeling is performed on frame-level spatial fusion risk features to predict the probability of collision events in the current traffic scenario and output collision risk assessment results.

2. The autonomous driving collision risk assessment method based on sparse soft hypergraph learning according to claim 1, characterized in that, The step of detecting and extracting features from traffic participants in the traffic scene image sequence specifically includes: The target detection network is used to identify targets in each image frame of a traffic scene image sequence to determine the location regions of vehicles, pedestrians, non-motorized vehicles and other road participants; Extract the location parameters, category information and visual feature information of the corresponding traffic participants from each location area. The location parameters include at least one or more of the following: target center coordinates, width, height and detection confidence.

3. The autonomous driving collision risk assessment method based on sparse soft hypergraph learning according to claim 2, characterized in that, The step of constructing the entity node features corresponding to each traffic participant specifically includes: Generate spatial geometric features based on the location parameters of traffic participants; Generate semantic category features based on the category information of traffic participants; Extracting visual appearance features based on visual feature information of traffic participants; By performing feature splicing and mapping on spatial geometric features, semantic category features, and visual appearance features, entity node features of each traffic participant are obtained.

4. The autonomous driving collision risk assessment method based on sparse soft hypergraph learning according to claim 3, characterized in that, The step of generating a node-hyperedge soft correlation matrix to represent the high-order interaction relationships among multiple traffic participants specifically includes: Based on the entity node characteristics of each traffic participant in the traffic scene node set, a dynamic superedge representation of the corresponding candidate superedge is generated. Calculate the similarity between the features of each entity node and the representation of each dynamic hyperedge to obtain the association weight between the corresponding node and the hyperedge; All association weights are normalized to obtain a node-hyperedge soft association matrix. The elements in the node-hyperedge soft association matrix are continuous values, which are used to characterize the degree of participation of the corresponding traffic participant node in the corresponding hyperedge.

5. The autonomous driving collision risk assessment method based on sparse soft hypergraph learning according to claim 4, characterized in that, The step of assessing the risk correlation of candidate hyperedges and performing sparse screening to obtain a sparse soft hypergraph structure specifically includes: Based on the association weight distribution of each candidate hyperedge in the node-hyperedge soft correlation matrix, calculate the risk activation weight of each candidate hyperedge; All candidate superedges are sorted from high to low according to their risk activation weight; Candidate hyperedges whose risk activation weights meet the preset threshold conditions are selected as target hyperedges, and low-correlation candidate hyperedges are eliminated to form a sparse soft hypergraph structure.

6. The autonomous driving collision risk assessment method based on sparse soft hypergraph learning according to claim 5, characterized in that, The hypergraph message propagation includes the information aggregation process from nodes to hyperedges and the information propagation process from hyperedges to nodes, wherein: For the information aggregation process from nodes to hyperedges, the entity node features of each traffic participant within the same target hyperedge are weighted and fused according to the association weights corresponding to the node-hyperedge soft association matrix to generate hyperedge interaction features. For the information propagation process from the hyperedge to the node, the hyperedge interaction features are transmitted to the entity node features of each traffic participant according to the participation level of the corresponding node, and the node features of each entity node are updated to obtain traffic risk features that integrate high-order interaction information.

7. The autonomous driving collision risk assessment method based on sparse soft hypergraph learning according to claim 6, characterized in that, The step of attention fusion, which integrates traffic risk features with high-level interactive information and global environmental features, specifically includes: Traffic risk features that integrate high-level interactive information are used as query vectors, and global environmental features are used as key and value vectors. The association weights of the query vector, key vector, and value vector are calculated through a cross-attention mechanism, and the global environmental features are weighted and filtered to obtain the weighted global environmental features. The weighted global environmental features are spliced ​​and fused with traffic risk features that integrate high-order interactive information to obtain frame-level spatial fusion risk features. The global environmental features include at least one of road structure information, illumination information, weather information, background environment information, and unidentified potential risk information.

8. The autonomous driving collision risk assessment method based on sparse soft hypergraph learning according to claim 7, characterized in that, The step of performing time-series modeling on frame-level spatial fusion risk features to predict the probability of a collision event in the current traffic scenario and outputting a collision risk assessment result specifically includes: Temporal convolutional networks are used to extract short-term motion change features of frame-level spatial risk features corresponding to consecutive image frames; Based on short-term motion change characteristics, a long-distance temporal encoder is used to capture the temporal dependencies in the long-term temporal dimension to obtain temporal correlation risk characteristics; Perform a fully connected mapping on the temporally correlated risk features to output the probability of a collision event occurring in the current traffic scenario.

9. The autonomous driving collision risk assessment method based on sparse soft hypergraph learning according to claim 8, characterized in that, Also includes: Based on the risk activation weights of each target hyperedge in the sparse soft hypergraph structure and the degree of participation of each entity node feature in the corresponding hyperedge, the key traffic participants and key interaction relationships that affect the collision risk prediction results are located. Based on the key traffic participants and key interaction relationships that affect the collision risk prediction results, the risk association structure explanation information of the corresponding traffic scenario is output.

10. An autonomous driving collision risk assessment system based on sparse soft hypergraph learning, characterized in that, Includes the following modules: The first module is used to acquire a continuous sequence of traffic scene images during the driving process of autonomous vehicles, detect and extract features of traffic participants in the traffic scene image sequence, construct entity node features corresponding to each traffic participant, and form a traffic scene node set based on several entity node features. The second module is used to construct a soft hypergraph structure based on the traffic scene node set, generate a node-hyperedge soft correlation matrix to represent the high-order interaction relationship of multiple traffic participants, conduct risk correlation assessment on candidate hyperedges and perform sparse screening to obtain a sparse soft hypergraph structure. The third module is used for hypergraph message propagation based on sparse soft hypergraph structure, which transmits and updates the group interaction information of multiple traffic participants to the corresponding entity node features, and obtains traffic risk features that integrate high-order interaction information. The fourth module is used to obtain the global environmental features corresponding to the traffic scene. It performs attention fusion between the traffic risk features that integrate high-order interaction information and the global environmental features to obtain frame-level spatial fusion risk features. The fifth module is used to perform time series modeling on frame-level spatial fusion risk features, predict the probability of collision events in the current traffic scenario, and output collision risk assessment results.

Images