Intelligent safety decision control system and method fusing spatiotemporal big data and multi-modal perception

Abstract

The application belongs to the technical field of autonomous system safety decision control, and discloses an intelligent safety decision control system and method fusing space-time big data and multi-modal perception. In view of the problems that the existing modules transfer coarse granularity, are difficult to cope with multi-modal data heterogeneous asynchronous missing and fixed boundary stalemate, the application calculates the reliability of the front-end computing mode and implements double-branch causal backdoor intervention to isolate false association, uses a selective structured state space sequence model to complete long-time long-range time series prediction; in the rear end, a smooth anisotropic space-time tube is generated based on conditional risk value tail sampling, and an additional tightening margin dynamically expanding with relative kinetic energy and uncertainty is embedded in the high-order control barrier function, and sparse quadratic programming and hierarchical rollback mechanism are used to solve the optimal instruction. The application seamlessly connects perception uncertainty and physical boundaries, and is suitable for intelligent networked vehicles, unmanned aerial vehicles and intelligent mine heavy machinery.

Description

Technical Field

[0001] This invention relates to the field of autonomous system safety decision-making and control technology, and in particular to an intelligent safety decision-making and control system and method that integrates spatiotemporal big data and multimodal perception. Background Technology

[0002] With the deep integration of artificial intelligence and automation control technologies, autonomous systems such as self-driving vehicles, multi-agent collaborative robots, unmanned aerial vehicles, and automated heavy machinery in smart mines are increasingly being deployed in highly dynamic and unpredictable open environments to perform safety-critical tasks. To build a comprehensive understanding of their surroundings, these autonomous systems typically integrate various sensing devices, including inertial navigation and wheeled odometers, vehicle-mounted cameras, LiDAR, global navigation satellite systems, and vehicle-to-infrastructure (V2I) communication. They rely on domain controllers or edge computing platforms with parallel computing capabilities to analyze the environment and output physical execution commands along the entire chain of "multimodal perception—trajectory prediction—planning and decision-making—motion control."

[0003] However, in the real physical world, data output from multiple sensors is highly asynchronous and structurally diverse. This results in temporal shifts and spatial misalignments due to clock asynchrony and installation location differences, as well as high data loss rates due to network latency and momentary equipment failures. Existing spatiotemporal fusion prediction frameworks, such as those based on Long Short-Term Memory networks or simple cascades of graph convolutional networks and self-attention networks, often employ superficial feature vector concatenation strategies when processing such heterogeneous data. They lack the ability to mine deep semantic interactions across modalities and are prone to falling into the trap of statistical correlation, failing to extract true causal relationships. When severe weather such as heavy rain or fog simultaneously causes a decline in visual sensor performance and changes in traffic conditions, these models often incorrectly establish a strong correlation between visual noise and target trajectory stagnation. This spurious correlation caused by unobserved common-mode confounding factors drastically reduces prediction accuracy when facing previously unseen edge scenarios. Furthermore, the global self-attention computation for long-term, high-dimensional data introduces a computational burden that increases quadratically with the sequence length, limiting the real-time response capability of the prediction process on embedded edge platforms.

[0004] In the planning and decision-making stage, the control layer needs to solve for safe trajectories based on environmental prediction information containing uncertainties. Although the control barrier function can provide the system with mathematical safety guarantees of forward invariant sets, traditional high-order control barrier functions and dynamic control barrier functions generally rely on static geometric boundaries or hard-coded safe distance thresholds. Faced with other traffic participants whose intentions are unclear and highly dynamic, if the safety threshold is set too high, the system will exhibit excessive conservatism, resulting in a "frozen" phenomenon of refusing to advance in narrow scenarios such as unsignalized intersections or congested work areas; if the threshold is reduced in pursuit of traffic efficiency, it will be difficult to cover the tail risks transmitted from the perception and prediction stage, leading to collisions when unforeseen trajectory deviations occur.

[0005] Therefore, it is evident that in the entire chain of perception, prediction, planning, and control, existing autonomous systems transmit data in a coarse-grained manner between modules. The front end struggles to output predictive results that remove spurious correlations and meet real-time requirements under asynchronous, high-missing, and confounded multimodal data, while the back end is constrained by fixed safety thresholds, making it difficult to strike a balance between excessive conservatism and potential collisions. Consequently, the uncertainty at the perception end cannot be translated into a control safety boundary that dynamically expands and contracts with physical risks. How to integrate perception, prediction, and decision control under these conditions, thereby ensuring both safety and traffic efficiency, is the technical problem this application aims to solve. Summary of the Invention

[0006] The purpose of this invention is to provide an intelligent safety decision-making control system and method that integrates spatiotemporal big data and multimodal perception, in order to solve the technical problems of existing autonomous systems, such as the fragmented transmission of various modules in the perception, prediction and planning control links, the inability to cope with the asynchronous high missing data and common mode mixed factor interference, and the excessive conservatism or potential collision caused by fixed safety thresholds. It achieves seamless translation of uncertainty at the perception end to the dynamic physical control boundary, ensuring the bottom line of safety while taking into account the passage efficiency.

[0007] To achieve the above objectives, the present invention provides an intelligent security decision-making and control system that integrates spatiotemporal big data and multimodal perception, comprising: The data access and spatiotemporal alignment module is used to concurrently acquire multi-source asynchronous sensor data through a circular buffer, map heterogeneous data to a standard time step based on a timestamp alignment operator, eliminate the influence of self-motion through vehicle pose compensation, and construct a unified feature tensor and validity mask. The multimodal causal fusion module is used to deweight the failed channel through a cross-modal adaptive gating attention mechanism, use a dual-branch causal backdoor to remove spurious correlations introduced by confounding factors, and output the trajectory prediction expectation and prediction covariance based on a selective structured state-space sequence model. The risk field and spatiotemporal tube generation module is used to perform conditional risk value sampling based on the predicted covariance matrix, generate anisotropic spatiotemporal tube repulsion envelopes through physical information neural networks, and adjust the safety margin in real time based on system-level perceived quality. The risk-adaptive barrier decision module is used to construct the control barrier function for encoding the spatiotemporal boundary and attitude stability domain. It introduces a risk-adaptive tightening margin into the fixed slope barrier inequality and issues control commands in real time through a quadratic programming solver.

[0008] As a preferred embodiment of the present invention, the data access and spatiotemporal alignment module is equipped with a data reliability evaluation unit for calculating the packet loss rate for each modality. Delay and obsolescence and normalized new information And generate a credibility coefficient based on the parameters. :

[0009] in, It is a non-negative weight, used to reduce and isolate failed channels at the front end, thereby improving the system's fault tolerance and robustness under harsh conditions.

[0010] As a preferred embodiment of the present invention, the multimodal causal fusion module includes a hybrid factor prototype dictionary, used to calculate the prototype posterior through auxiliary branches and apply orthogonal constraints to the main prediction branch. This cuts off false causal backdoor paths, thereby effectively eliminating background noise interference and ensuring the purity of long-term high-dimensional feature extraction.

[0011] As a preferred embodiment of the present invention, the risk field and the spacetime tube envelope generated by the spacetime tube generation module adopt a truncated Fourier radial function. Indicates, and satisfies the azimuth angle The anisotropic distribution is truncated, and geometric projection clipping is performed on the radius in each direction, shrinking it to the clearance distance with the boundary of the local drivable domain. This ensures that the envelope boundary is smooth and differentiable everywhere mathematically, preventing gradient breakage in subsequent stages.

[0012] As a preferred technical solution of the present invention, the risk adaptive barrier decision module embeds a hierarchical backoff mechanism in the quadratic programming. When the solver fails to converge within the budget time, it performs soft constraint relaxation or uses the backup control sequence of the previous cycle, thereby strictly ensuring forward invariance when hard constraints are feasible and achieving graceful degradation of safety when hard constraints are not feasible.

[0013] This invention also provides an intelligent security decision-making and control method that integrates spatiotemporal big data and multimodal perception, comprising the following steps: Step S1: Concurrently access and align the multi-source asynchronous sensor data with timestamps, and compensate historical observations to a unified coordinate system through rigid body transformation; Step S2: Calculate the modal confidence coefficient and combine it with the dual-branch causal backdoor intervention to remove spurious associations, and perform time series prediction through a selective structured state-space sequence model; Step S3: Tail sampling is performed based on the predicted covariance to generate anisotropic spatiotemporal tubes, and the safety envelope is expanded in real time according to the system-level perception quality. Step S4: Construct a continuous time-varying barrier inequality and introduce an adaptive tightening margin. The optimal control command is solved by discretization difference constraints and quadratic programming.

[0014] As a preferred embodiment of the present invention, step S2 employs a selectively structured state-space sequence model, which generates discrete parameters in real time through input projection. , and It completes long-range temporal memory transfer with linear complexity and outputs the expected predicted trajectory. With predicted covariance This significantly reduces the computational burden and greatly improves the system's real-time inference response capability on embedded edge platforms.

[0015] As a preferred embodiment of the present invention, in step S4, an additional tightening allowance is added. The calculation formula is:

[0016] in, For system-level perceived quality, The value is sent after being processed by asymmetric rate limiting with dead zone to suppress frequent switching of control quantities, avoid high-frequency reciprocating chatter of the actuator, and balance the smoothness and stability of control.

[0017] As a preferred embodiment of the present invention, in step S4, the control barrier function Defined as a smooth signed distance field based on anisotropic radial margin, its Lie derivative inequality discretization is:

[0018] in, For a fixed slope, To provide additional margin values ​​for the corresponding prediction steps, the constraints are transformed into values ​​related to the control inputs. The linear form ensures the convexity of the quadratic programming and eliminates the risk of numerical divergence under sudden operating conditions.

[0019] As a preferred embodiment of the present invention, in step S4, during a single optimization loop, the margin sequence... Treated as a known constant parameter, and refreshed periodically according to the current measured state within the closed-loop rolling time domain, the risk adaptive convergence is achieved through the closed-loop feedback in the rolling time domain, which maintains the convexity of the single-step solution and ensures the dynamic adaptive adjustment at the closed-loop level.

[0020] Compared with the prior art, the present invention has the following beneficial effects: The beneficial effects of this invention compared to the prior art are as follows: In the multimodal feature fusion stage, this invention uses cross-modal adaptive gated attention combined with the credibility coefficients of each modality's real-time evaluation to pre-weight failed channels. Furthermore, it employs a dual-branch causal backdoor to remove false correlations introduced by common-mode hybrid factors such as sudden weather changes and collective sensor degradation. This ensures that trajectory prediction remains focused on target evolution with real physical causality even under adverse weather conditions and high data loss, avoiding a sharp drop in prediction accuracy in edge scenarios. Simultaneously, it aggregates spatial topological features using Chebyshev multinomial graph convolution and completes temporal evolution using a selectively structured state-space sequence model. This reduces the computational complexity of temporal modeling from a quadratic relationship with sequence length to a linear relationship, enabling the prediction stage to complete inference on embedded edge platforms at millisecond intervals and maintaining continuous obstacle trajectory prediction even with high data loss due to communication disruptions.

[0021] In the decision-making and control stage, this invention uses conditional risk value to tail-sample the prediction covariance, generating a spatiotemporal tube that continuously changes over time for each obstacle. The risk-adaptive safety margin calculates compensation coefficients in real time based on relative kinetic energy, prediction uncertainty, perceived quality, and braking potential. This is done using a fixed class-κ... In the inequality of the higher-order control barrier function represented by the slope, an additional safety margin term that expands adaptively with risk is introduced. This allows the control safety constraints to tighten immediately under high risk, and the safety envelope to expand synchronously with the space control radius to ensure physical isolation. Under low risk, the constraints are automatically relaxed to release planning space. This overcomes the binary deadlock between overly conservative and potential collisions caused by fixed thresholds. It avoids "freezing" and false braking caused by excessively large thresholds, and covers the tail risks transmitted by perception and prediction. At the same time, attitude-motion coupled attitude stability constraints are embedded in sparse quadratic programming to avoid rollover or skidding caused by overly aggressive steering in avoidance actions. Asymmetric rate limiting and dead zone suppression suppress the high-frequency back-and-forth between conservative and aggressive safety domains. With the help of a graded backoff mechanism, it is ensured that each control cycle can output legally executable control commands within a millisecond budget time. When hard constraints are feasible, the forward invariance of the safety set is strictly maintained. When hard constraints are not feasible within the budget, the safety is gracefully degraded by minimizing constraint violations and prioritizing the use of certified safety controls.

[0022] Furthermore, the front-end causal deconfusion reduces the divergence of prediction uncertainty, establishing a reasonable upper bound for the expansion of the back-end adaptive margin and preventing excessive collapse or unnecessary expansion of the safety domain. The back-end amplifies the corresponding spatiotemporal control radius and safety margin by aggregating the perception quality evaluated modally by the front end at the system level, ensuring that perception degradation is translated into additional physical isolation margin before the prediction covariance is fully reflected. Thus, perception accuracy and control flexibility are interconnected within the same high-frequency closed loop, enabling the system to stably output control commands that balance safety baselines and traffic efficiency even in complex dynamic scenarios with asynchronous multimodal data, high missing data, and mixed interference. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall architecture of the intelligent safety decision control system that integrates spatiotemporal big data and multimodal perception according to the present invention; Figure 2 This is a flowchart illustrating the intelligent safety decision-making and control method that integrates spatiotemporal big data and multimodal perception according to the present invention.

[0024] Figure labeling: 100, Data access and spatiotemporal alignment module; 101, Data ingestion submodule; 102, Spatiotemporal alignment submodule; 200, Multimodal causal fusion module; 201, Gated attention submodule; 202, Causal backdoor intervention submodule; 203, Spatiotemporal coding submodule; 300, Risk field and spatiotemporal management generation module; 301, Spatiotemporal management modeling submodule; 302, Safety margin calculation submodule; 400, Risk adaptive barrier decision module; 401, Dynamic mapping submodule; 402, Barrier function submodule; 403, Quadratic programming optimization submodule. Detailed Implementation

[0025] The specific implementation of the present invention will be further described in detail below with reference to the accompanying drawings, so that those skilled in the art can reproduce the complete system and implement the corresponding method accordingly. The sampling frequency, buffer depth, number of network layers, field of view length, confidence level, and various weight and gain values ​​given in the following description are exemplary settings listed for ease of understanding of the technical solution and do not constitute a limitation on the scope of protection; those skilled in the art can make adaptive substitutions without departing from the concept of the present invention. For ease of writing, all values ​​under the following headings are used in this specification. Labeled operators (such as) , Both ) represent the "spatiotemporal" alignment operator, and their subscripts are... It serves only as an identifier for spatiotemporal mapping and does not participate in numerical calculations.

[0026] The intelligent safety decision-making and control system that integrates spatiotemporal big data and multimodal perception provided in this embodiment has the following overall architecture: Figure 1 As shown. Along the data processing direction, the system consists of a data access and spatiotemporal alignment module 100, a multimodal causal fusion module 200, a risk field and spatiotemporal management generation module 300, and a risk adaptive barrier decision module 400, all cascaded together. The entire system is hosted on a domain controller or edge computing server with parallel computing capabilities and shares the same global clock reference and a unified vehicle centroid coordinate system. The corresponding method flow is as follows: Figure 2 As shown, from the moment multimodal data is accessed, it goes through causal inference, risk field mapping, and constrained optimization, until the control command is issued, forming a closed loop that runs at a high frequency with a cycle of 10ms. This method is executed cycle by cycle by the aforementioned domain controller or edge computing server.

[0027] The data access and spatiotemporal alignment module 100 serves as the initial data filtering and normalization engine for the entire chain. It is responsible for recombining highly asynchronous and structurally diverse multi-source observations into normalized tensors that can be directly consumed by subsequent stages, within a unified non-Euclidean spatiotemporal reference coordinate system. This module internally includes a data acquisition submodule 101 and a spatiotemporal alignment submodule 102. The data acquisition submodule 101 configures an independent multi-threaded circular buffer for each sensor signal, with a buffer depth of, for example, 256 frames. The corresponding producer thread advances the write pointer and adds an arrival timestamp tag when a data frame arrives, while the main control loop advances the read pointer as a consumer. Updates to the read and write pointers are performed using atomic operations, thus achieving concurrent queuing and non-blocking reading of signals without the need for global locks. The data sources concurrently received by this submodule include at least: a sequence of local time-phased pose matrices output at a high frequency of approximately 100Hz by the inertial navigation and wheeled odometer systems. (in (This represents the number of time-sampled frames contained in the pose sequence within the current buffer window), with each row containing the three-dimensional position and three-axis attitude components in the geodetic coordinate system; a sequence of video image frames acquired by an onboard camera array at an intermediate frequency of approximately 30Hz. 3D point cloud features obtained by periodic scanning of lidar ; and traffic status beacons or text alerts that arrive at low frequency and irregularly via vehicle-to-infrastructure (V2I) wireless channels. The physical timing of each signal is different, and due to network jitter and equipment status fluctuations, they exhibit irregular arrival characteristics. The circular buffer provides a pool of original materials with timestamps for subsequent unified alignment.

[0028] The spatiotemporal alignment submodule 102 establishes a unified spatiotemporal reference on this material pool. Considering that applying linear interpolation to asynchronous signals would smooth out abrupt high-frequency components with diagnostic value, this submodule abandons interpolation and adopts a nearest-neighbor matching alignment strategy. In the time dimension, a set of discrete standard time steps is defined based on the master control cycle period (e.g., 20ms, corresponding to 50Hz). For any mode, for each standard time step By time alignment operator

[0029] All original arrival timestamps in the modal circular buffer Selected with The operator selects the most recent observation in time; its output is the original timestamp corresponding to the selected observation, which the system then uses to assign the entire data value carried by that observation to the standard time step. The assignment process does not perform any smoothing to preserve the abrupt changes in the original signal. Spatially, a discretized geographic topology grid of the drivable area is constructed using the vehicle's centroid and heading at the current (latest) standard time step as the origin, with a grid resolution of, for example, 0.5m. It is important to emphasize that, because the vehicle moves continuously at high speed in the physical world, the grid of the vehicle's centroid is essentially a non-inertial coordinate system. If the occupancy relationships at different times within the historical window are directly unioned and cascaded along the time axis without compensation, the same stationary target will fall into different nodes at different historical moments due to origin drift, causing temporal slices with the same node index in the tensor to be mixed with observations physically located at different geographical locations. Therefore, this submodule performs vehicle motion compensation before grid assignment: using the local pose sequence accessed by module 101 at a high frequency of approximately 100Hz. Calculate each historical standard step relative to the current standard step Rigid body relative pose transformation (Including translation and rotation), and all spatially attributed observations collected during this historical step will be processed. The system is rigidly transformed to the same common reference frame anchored to the current vehicle pose, and then grid assignment is performed. This eliminates the vehicle's self-motion, ensuring that stationary targets maintain consistent node indices throughout the entire history window, with their temporal slices corresponding to the same geographical location. Dynamic targets retain only their actual motion relative to the ground, thus restoring the physical consistency of spatiotemporal correspondence. This avoids erroneous topology extraction by subsequent graph convolutions and prevents the temporal manifold of selectively structured state space sequences from being torn apart. After the above compensation, the spatial attribute objects identified by each sensor at the current standard time step and each compensated and aligned historical step are processed. (Including the centroid of the point cloud cluster and the center of the bottom surface of the image object detection box), determined by the Euclidean distance finder.

[0030] Map each object to the nearest grid node in spatial location. Regarding the selection of key spatial nodes participating in tensor construction, the system does not traverse all grids. Instead, it takes the union of grid nodes within the current historical window that are occupied by at least one object through the above mapping, and adds the node where the vehicle's centroid is located and several nodes along the global reference path. At the same time, it removes idle nodes where no object falls within the entire window, thereby constraining the number of key spatial nodes participating in the calculation to a finite number. The selection rule takes as input the object-node mapping relationship within the window and outputs a set of indices of the nodes to be retained. After the above spatiotemporal mapping, Data for each modality is retained. Projected onto key spatial nodes and stitched together to form a unified multimodal feature tensor. ,in Set the historical time window step size (e.g., 20 steps). This represents the sum of the initial feature dimensions for each modality. It should be noted that due to the set of key nodes... The location is determined once by the union of the entire historical window's occupancy, while dynamic obstacles continuously move within the window, thus being partially included at certain time steps. The absence of observation data at nodes creates time-varying holes within the spatiotemporal tensor. To address this, the system uniformly fills all node-time step entries with zeros and simultaneously generates a new entry that is identical in both the node and time dimensions. Homomorphic binary validity mask Each entry is marked as either a true observation (set to 1) or zero-padding (set to 0). This mask is passed along with the tensor to subsequent stages. In the subsequent neighborhood aggregation of graph convolution and the temporal propagation of the state space sequence, the filling entries are masked so that they do not participate in message aggregation or contribute gradients to backpropagation, thereby avoiding the contamination of spatiotemporal coding with invalid gradients by hole entries. (In addition to the mask, it is also possible to choose to pre-fill the holes with the features of effective nodes in the neighborhood or the most recent effective value in the time dimension. Both are used in conjunction with the masking mechanism to further stabilize the values.)

[0031] Because clock drift, network packet loss, and momentary device failures in real-world links can cause some modes to have no valid observations falling within a certain number of standard time steps, the alignment submodule 102 maintains a rolling data reliability assessment for each mode while performing mapping, and passes this assessment as a reliability label for the tensor node to subsequent stages along with the tensor. For ease of description, let's call the mode... After the above alignment, in the standard step The normalized structured observation feature vector obtained per spatial node is Each component is a low-dimensional numerical descriptor of the mode at the corresponding node, such as the occupancy count and relative offset of the cluster centroid of a point cloud mode, or the coordinates of the bottom center of the detection box and the detection confidence of an image mode; That is, the unified tensor Mid-mode In standard steps The slices are structured numerical quantities after rasterization reduction, not the original pixel matrix or the disordered point cloud itself. Let the sliding evaluation window length be... A standard time step (e.g., 10 steps), record For modality In the window The number of time steps identified as missing by the inner alignment operator is used to define its packet loss rate. ;remember For modality The timestamp corresponding to the most recent valid observation Define the delay staleness for the nominal sampling period of this mode. Then, based on the most recent effective structured observation features... Calculate its short-term jump. Considering... The dimensions and numerical scales of the components are different (e.g., the occupancy count is a pure number, the centroid relative offset and the bottom center of the detection frame are lengths, and the detection confidence is a dimensionless probability). If the Euclidean norm is directly calculated without processing, it will lose its physical meaning due to the mixed dimensions. Therefore, we first perform channel-by-channel standardization (diagonal whitening) according to the characteristic scale of each channel: a channel-by-channel scale vector is constructed using the characteristic reference scale corresponding to each component (e.g., the rolling standard deviation of the channel in the sliding window or a preset fixed nominal scale). Standardized observation features ( This represents element-wise division, which transforms each component into a comparable quantity under the same dimensionless scale; then, it uses the standardized most recent effective observation characteristics. Compared with its single-step forecast value Normalized new information between

[0032] Characterizing short-duration abrupt jumps in the signal, among which It is obtained by one-step linear extrapolation from several effective standardized observation features of this mode (e.g., taking the first-order extrapolation). , (These are the two most recent valid standard steps). Since standardization and extrapolation only apply to the aforementioned low-dimensional structured numerical descriptors, not the original image or point cloud, and all components have been assigned to the same dimensionless scale, their algebraic linear extrapolation and Euclidean norm are mathematically well-defined and physically clear. To avoid false alarm divergence in open-field conditions, the denominator is not a small positive number approaching zero, but rather... Set a lower bound with physical meaning. (Its magnitude is taken as the normal norm of the standardized observation characteristics under typical target operating conditions, rather than...) (Minimum quantity at the level of standardization): When there are no dynamic obstacles within a local grid and the observation is standardized. When the whole approaches the zero vector, the denominator is bounded by the lower bound. Clamped, the unavoidable high-frequency digital noise in the molecule is divided by The result is only a small amount approaching zero and will not be amplified hundreds of thousands of times. No divergence, no false triggering of anomalies; only when the signal is significant ( Only when this condition is met does the expression degenerate into a relatively innovative form. Furthermore, when the observed energy of this mode in a local region is below the lower limit of occupancy (i.e., the empty-field condition), its innovation term is directly applied. Setting it to zero, since there are no detectable jumps when there is no observed structure, further eliminates the possibility of a zero-reliability coefficient. Accidental collapse under normal open working conditions. Combining the above three factors, the modal structure is obtained. At any moment Credibility coefficient

[0033] in Non-negative weights One that allows minor aging, not exceeding one nominal cycle, to go unpunished. The value falls between 0 and 1, approaching 1 when the data is fresh, complete, and continuous, and approaching 0 when the modality is severely degraded. This coefficient then participates in the modality weighting of the front-end fusion and is also converted into the adjustment basis for safety margin and spatiotemporal control radius in the back-end, thus incorporating the physical fact of the current perceived quality throughout the entire decision-making process.

[0034] The unified spatiotemporal tensor, after being normalized and labeled with reliability, enters the multimodal causal fusion module 200, and is processed sequentially by the gating attention submodule 201, the causal backdoor intervention submodule 202, and the spatiotemporal coding submodule 203 within it.

[0035] The gated attention submodule 201 first feeds the tensor into a lightweight encoder that is independent of each mode, for example, by using a two-layer fully connected network to obtain the tensor. Modal characterization (Note: Here) This is a high-dimensional representation output by the encoder, distinct from the structured observation features in the alignment stage of module 100. (These two belong to different processing stages). Subsequently, a gating network calculates the retention confidence of each modality node by node: for each node, the modal features of each node are concatenated along the feature dimension to form a length. The row vector is then right-multiplied by a gated projection vector shared across all nodes. To obtain the mode of the node Scalar gating, gathering After each node, a node-by-node gating vector is formed.

[0036] in A sigmoid activation function to compress the input to the 0-1 range. This represents a cascade operation along the feature dimension. The splicing features of each node are stacked row by row. matrix, With bias The former is obtained by right-multiplying the latter and summing the broadcasts. Node-by-node gating. Because the gating projection vector is shared across all spatial nodes, and the concatenated feature row vectors of each node are uniformly right-multiplied without being bound to specific node positions, the gating effect is applied regardless of the current number of key nodes. How does it change dynamically? All dimensions can be applied row-by-row (node-by-node) without changing, thus naturally adapting to a variable number of nodes in the space. This is achieved through data-driven learning. This reflects the statistical significance at the feature level, and its response to sudden channel failures often lags behind the actual degradation at the physical level. Therefore, the modal-level reliability coefficients output by the alignment submodule 102 will be used. (Scalar) and Node-by-Node Gating Multiplication and broadcasting along the feature dimension constitute an effective gating system corrected for physical reliability, based on which a fusion characterization is obtained by weighted combination of each modality.

[0037] In the formula For modality The learnable projection matrix is ​​obtained by using the right multiplication convention, i.e. Indicates Each row (each node) (Eigenvectors) right-multiplied by the same matrix Applying the same characteristic linear transformation to each node will still yield the same result. The matrix is ​​thus consistent with the dimension of the gated tensor and can be operated on element by element; Represents the Hadamard product, node-by-node gating. After broadcasting along the feature dimension, and with that The results are multiplied element-wise. Thus, when a mode experiences a sudden increase in noise due to blizzard obscuration or electromagnetic interference, even if its characteristics are not yet fully realized... The above fully exposes the abnormality. The contribution of that mode will also be reduced first, so that the fusion representation can complete the isolation of the failed channel within milliseconds.

[0038] The fusion representation is then transferred to the causal backdoor intervention submodule 202 to break down spurious predictions caused by statistical correlation. This submodule uses a structural causal model to characterize the data generation process, denoting the unobserved common-mode interference that causes distribution drift as the confounding factor latent set. Considering Since these are unobservable variables and cannot be directly enumerated in the network, in engineering practice they are approximated by a finite and learnable dictionary of prototype confounding factors. A prototype representation ( For example, taking values ​​from 16 to 64). Based on this, construct... The two branches share a common input; their separation of background and causal components is not achieved through manually designed filter kernels or prior masks. Instead of pre-extracting features, learnable features are decoupled through their respective training objectives. Specifically, the auxiliary representation branch is a multilayer sensing head. ,by The input and output are the category posteriors (i.e., the normalized weights of each prototype) on the prototype dictionary, and are supervised during training to predict the confounding factor prototype to which the current sample belongs, thereby being driven to absorb background / style redundancy components related to the confounding factor into its own representation; the main prediction branch is another multilayer perceptron. , in the same The main features are used as inputs and outputs for subsequent prediction. (and Consistently for the aforementioned Each of the key spatial nodes holds one Dimensional decoupling of embedded features, As an embedded dimension, its spatial node axis is entirely aligned with modules 100 and 203. (Strict registration of dimensional grid nodes, preventing dimensional abrupt changes), and orthogonal constraints are applied during backpropagation.

[0039] In the formula To be Each prototype represents a matrix of mixed factor dictionaries stacked row by row. and Sharing the same embedding dimension Therefore, the correlation matrix Each element measures the inner product correlation between a node's main feature and a prototype (averaged over a mini-batch of samples during training); minimizing its Frobenius norm forces the main feature. The hybrid subspace spanned by each prototype is orthogonal in the embedding space, thus blocking the penetration of hybrid factors into the main predicted feature. Further backdoor adjustments are implemented in the forward computation flow.

[0040] After approximating with the prototype dictionary, for the implicit set The summation is specifically implemented as a summation of Finite weighted aggregation of prototypes: where the condition term Based on the characteristics of the main road, the first The prototype representation is modulated (e.g., by feature concatenation or channel-wise affine modulation) and then calculated by the prediction head; prior... The edge occurrence frequency of the prototype on the training set is taken. By weighted summation of the probabilities of all prototypes, the false backdoor path between confounding factors and predicted features is cut off, and the debiased dynamic intent probability model is output. In this way, visual noise caused by severe weather is no longer mistakenly strongly correlated with trajectory stagnation, and the model can still maintain prediction stability in unseen edge scenes.

[0041] Features that have undergone causal bias correction are then fed into the spatiotemporal coding submodule 203. In the spatial domain, an adjacency matrix is ​​generated based on the physical distances between nodes. A graph convolutional network approximated by Chebyshev polynomials is deployed to perform message passing and aggregation within the neighborhood. For example, the polynomial order is taken as 3rd order to obtain spatial domain features. This avoids dense calculations on the entire graph; Expanding along the time dimension yields a spatial feature sequence step-by-step, denoted as the first step. The step value is In the time domain, instead of using a global self-attention-based sequence modeling mechanism, a selective structured state-space sequence model is adopted to handle temporal propagation. As the state space layer in the th The input characteristics of the step. This model discretizes the continuous state-space equations into a zero-order hold.

[0042] The hidden state dimension is, for example, 64; to achieve the input-driven selectivity mechanism, the discrete parameters are generated in real time based on the current input: the input correlation matrix is ​​obtained by linear projection of the input features. With output matrix and with Generate channel-wise positive discrete step sizes; continuous state matrix It is a learnable diagonal structure matrix (which can be initialized in a structured state-space manner) and discretized according to the zeroth order. , This parameterization allows the transition matrix to change in real time with the input sequence, enabling long-range temporal memory transfer in a time complexity linear with the sequence length, in conjunction with a parallel scanning algorithm. Module 200 ultimately outputs the future sequence to the next stage. The expected sequence of predicted trajectories of dynamic obstacles in the environment within a continuous time step (e.g., 30 steps). And the set of predictive covariance matrices characterizing its dispersion. .

[0043] Furthermore, in response to extreme common-mode perception collapse conditions such as extreme cold and blizzards, heavy fog, and enhanced electromagnetic interference—where full-frame camera noise, strong scattering from lidar, and large-scale packet loss in vehicle-road cooperation occur simultaneously, causing the reliability coefficients of all modes to synchronously approach zero and the fusion representation to be compromised—further investigation is needed. Degradation to a near-zero matrix—This embodiment incorporates a self-holding and blind zone propulsion mechanism under full-channel degradation to prevent the selective state-space model from freezing due to zero-value feedthrough and to ensure uninterrupted continuous prediction. Firstly, to avoid… The fact that the matrix is ​​exactly zero causes the input-related parameters of Mamba to ( Locked to baseline constant, fusion employs normalized confidence weighting (each) (using relative weights rather than absolute multipliers), and in all When the input falls below a critical threshold, the zero input is replaced with a learnable "unobserved" default feature token, while the hidden states of the state space sequence are also updated. It carries the information along time without resetting it, thus preserving the temporal context accumulated before the collapse. Secondly, when the system-level perceived quality... Falling below the critical lower limit During (signal common-mode collapse), the prediction path smoothly switches from the self-learning fusion prediction branch to the deterministic blind zone propulsion feedforward: for each on-orbit obstacle, extrapolation is performed using its most recent reliable state (position and velocity) before collapse, based on a constant velocity or constant rotation rate kinematic model; for the vehicle, extrapolation is performed using inertial navigation and wheel odometry pose (high-frequency and independent of degraded external sensing channels), thereby maintaining a trajectory prediction with reduced confidence but continuous performance within the sensing blind zone. Thirdly, the prediction covariance during the blind zone... The uncertainty propagation model based on air propagation is growing rapidly, and through Lowering the threshold further expands the spatiotemporal radius in all directions, allowing the safety envelope to actively expand outward, and the system becomes conservative rather than blind. Once the perception quality recovers and exceeds the lower limit, the prediction path smoothly switches back to the learning branch. Thus, the beneficial effect of "maintaining continuous prediction even when communication is poor and data loss is high" can also be achieved under extreme conditions of full-channel degradation.

[0044] The risk field and spatiotemporal management generation module 300 is responsible for translating high-order statistical probability models into analytical geometric constraints acceptable to control theory. Internally, it includes a spatiotemporal management modeling submodule 301 and a safety margin calculation submodule 302. The spatiotemporal management modeling submodule 301 calculates the predicted covariance. Tail sectional sampling is performed using conditional value at risk, recording the obstacle relative to its predicted center. The deviation is The prediction center is directly taken from the expected predicted trajectory output by module 200, that is... , Then it obeys the output of module 200, with For the mean, The predicted distribution of covariance (e.g., Gaussian distribution) First, define the deviation amount. At confidence level The value at risk (e.g., taking 0.95) is its quantiles

[0045] because The distribution is induced by the above-mentioned predicted distribution. Can be directly self We obtain (for the Gaussian case, its value is determined by...) (The value of the Mahalanobis radius determined by the characteristic structure at the corresponding quantile is given). Based on this, the conditional value of risk is defined as the conditional expectation of the portion exceeding the value of risk threshold.

[0046] This tail metric is used to assess the deviation limit under extremely low-probability conditions, and based on this, a continuously changing repulsion envelope is constructed for each individual obstacle, i.e., air traffic control.

[0047] in The trajectory vector of the spacetime tube center. This refers to the safety risk radius. It needs to be clarified that this pertains to the spacetime management center. It is not regenerated by this module, but is directly taken from the expected sequence of the obstacle prediction trajectory output by module 200, i.e. The physical information neural network in this submodule is only responsible for generating a continuously changing, anisotropic statistical radius field under a given center. (along the center, each azimuth angle) Independent values ​​are selected to make the repulsion envelope a star-shaped region rather than an isotropic circle, thus simultaneously conforming to the anisotropic distribution of road network geometry and prediction covariance in all directions. This also ensures the physical feasibility of the resulting repulsion envelope in terms of dynamics and road network geometry, thereby avoiding functional overlap or data flow loops with the front-end prediction module in center trajectory generation. The network uses three factors as forward inputs: the predicted center of the current obstacle. Predicting covariance The network also includes drivable domain descriptors (including lane boundaries, drivable connectivity, and geographic hard boundaries) extracted from the grid or high-precision map occupied by module 100, representing the current local road network geometry. In terms of network topology, the network adopts a "dual-branch encoding-fusion-temporal decoding" structure: the geometric branch extracts geometric embeddings from the occupied grid topology of the local drivable domain using graph convolution (or grid convolution) to adapt to non-Euclidean road network structures; the statistical branch semi-vectorizes the structured prediction covariance. Flattened into a fixed-length vector, concatenated with the prediction center, and then processed by a multilayer perceptron to obtain statistical embeddings; the two branch embeddings are projected and aligned to the same dimension in the fusion layer, then concatenated and fused by a multilayer perceptron, and finally output by a temporal decoding head (e.g., a one-dimensional convolution or a multilayer perceptron) to produce a progressive angular radius profile within the prediction field of view. (i.e., the upper edge of each prediction step) Discrete directions The radius, for example, take Its anisotropy is determined by the predicted covariance. The directional dispersion and local road network geometry are shaped together, and the main axis and orientation can also be achieved with low parameterization. The eigenvalue decomposition is a directed ellipse whose scale depends on the conditional value at risk. It is particularly important to emphasize that... The discrete azimuth radii do not form a star-shaped polygon with vertices by connecting them with broken lines—the signed distance field of the polygon boundary is not differentiable at the vertices and the central axis (Voronoi skeleton), which will cause the gradient of the subsequent barrier function to break; therefore, in this embodiment, its angular profile is represented as azimuth angle. truncated triangular (Fourier) radial function (order) The network directly outputs the low-order Fourier coefficients. Or by The envelope is obtained by fitting a sample, and thus the resulting envelope is a smooth, vertex-free closed curve. (where its anisotropy is preserved and its boundaries are differentiable everywhere) – among which... Flattening the highly structured covariance and embedding the geometric branches into the same dimension is the specific approach to achieving feature alignment and fusion of covariance and non-Euclidean geometric descriptors at the input layer. Its physical information loss in the offline stage consists of three types of differentiable residual terms that apply to the input: one is the uncertainty propagation residual, which constrains the growth rate of the generated radius sequence over time to match the spread expansion that the predicted covariance should have according to the prediction dynamics; since the network directly outputs discrete angular radius profiles with a discrete-time decoding head... The time derivative is not for continuous time coordinates. Perform automatic differentiation (time) Instead of using the continuous input coordinates of the network, it is discretized and approximated by the forward difference between adjacent steps of the output sequence, i.e. (summation index taken to) The forward difference required Always within the maximum predictive field of view Within without crossing the boundary, the end of the first If the step needs to be preserved, then backward difference should be used instead. To measure the covariance along the azimuth The projection growth and prediction velocity are mapped to a differentiable operator of the rate of change of the azimuth radius, and the prediction velocity consumed by it... It is not an independent quantity output separately by the previous stage, but directly derived from the predicted center position sequence already output by module 200. Derived via first-order forward difference, i.e. (also only in) The calculation path is entirely within the disclosed data flow; the forward difference is a linear operation on the network output, and the gradient can be normally backpropagated through it, thus concretizing the "state space equation residual" into a computable covariance-radius propagation consistency constraint; the second is the road network hard boundary residual, which penalizes the pipe wall for going beyond the drivable domain boundary, and in order to avoid the isotropic scalar radius being hijacked and collapsed by the nearest boundary on one side, this residual is applied directionally: ,in From the center along the orientation The directed clearance distance to the nearest hard boundary of the road network; thus, the azimuth radius on the side adjacent to the curb is only tightened by the proximity constraint on that side, while the azimuth radius on the side facing the open square can still expand freely according to the tail spread of the predicted covariance, avoiding the global collapse of the entire envelope due to the proximity of one side to the boundary, preserving the ability of the conditional risk value to characterize the tail risk of the open side; thirdly, the initial value satisfies the residual, constraining the azimuth radius of each direction in the first step and the distance from the curb. The instantaneous statistical radii obtained by sampling along the corresponding orientation are consistent. ,in For deviation along azimuth The conditional risk value tail radius of the projection. The key point is that this offline training is not targeted at a specific intersection, but rather conducted on a sample distribution covering various road geometries and variable topological layouts. Furthermore, the road network boundary participates in the aforementioned residual constraints as network input rather than being fixed into the network weights. Therefore, the network learns a conditional mapping that "makes the envelope conform to the geometric boundary under any given geometry," thus possessing the ability to generalize to entirely new intersections and variable topological manifolds not encountered during training. During the runtime phase, the system no longer performs any online backpropagation or online training, but only uses the trained network to perform a single forward inference on each current obstacle, directly outputting the corresponding angular radius profile. To provide a hard guarantee independent of learning weights when facing a completely new geographical layout, the generated spatiotemporal tube is further subjected to geometric projection clipping according to orientation, that is, the radius of each orientation is clipped to the real-time clearance of that orientation. Within this range, the envelope is strictly contracted to the connected manifold defined by the current real-time drivable domain, thus preventing it from encroaching on buildings or medians even in unfamiliar intersection geometry. Since forward inference for a single obstacle involves only a few matrix multiplications, geometric clipping is a lightweight polygon intersection operation, and multiple obstacles in the environment can be processed in batches and in parallel along the tensor dimension, the generation and clipping of the spatiotemporal tube can be completed within a millisecond-level control budget, even in dense scenes with dozens of obstacles simultaneously. This ensures compatibility with the system deployed on an embedded edge platform and running on a 10ms cycle. To enable the spatiotemporal tube to reflect instantaneous changes in the overall perception quality of the front end, the confidence coefficients of each modality are assigned according to their nominal importance. Aggregation yields system-level perceived quality.

[0048] And a radius generation is introduced, allowing the radius in each direction to be... ,in Non-negative gain (equal expansion in all directions when sensing quality degradation, or expansion only in directions not constrained by near boundaries to avoid unnecessary contact with walls). When the link experiences large-scale packet loss... When the temperature drops, the spatiotemporal tube radius actively expands outward, so that the degradation of front-end perception is translated into additional physical isolation margin before the prediction covariance is fully reflected.

[0049] Safety margin calculation submodule 302 periodically polls the local bus to obtain the absolute speed of the vehicle. Maximum available deceleration And combined with the predicted target speed The additional safety margin, which varies continuously with time, is calculated based on the predicted covariance. To ensure that all physical quantities are added at the same scale and satisfy dimensional homogeneity, this submodule first normalizes the relevant physical quantities into dimensionless ratios using their respective characteristic reference scales. Then, a basic margin is used to uniformly bear the dimensions of this margin, resulting in...

[0050] in Basic margin, with additional safety margin The dimensions of (and the barrier inequality mentioned later) Consistent dimensions). , , Let these be positive characteristic reference scales for velocity, covariance trace, and acceleration, respectively, such that the relative kinetic energy term within the parentheses... Prediction uncertainty term Perceived quality items With braking capacity All are dimensionless quantities; , , , This corresponds to the dimensionless adjustable gain. Thus, all terms within the parentheses and the entire expression are dimensionless. Only carry The dimensions of the expression are homogeneous. When the relative collision kinetic energy increases sharply, the prediction result is highly ambiguous, the perceived mass decreases significantly, or the braking potential is insufficient (…), the dimensions of the expression are homogeneous. When (increases), Nonlinearity increases; however, when the vehicle's performance redundancy is sufficient, the environment is certain, and perception is complete, the risk items within the brackets approach zero. Automatically approaching its lower limit Considering that rapid fluctuations in risk factors near the threshold, if directly transmitted through the margin, would cause frequent switching of control inputs and induce actuator chatter, this submodule does not directly issue instantaneously calculated values, but instead applies an asymmetric rate limit with a dead zone. The issued value of the previous control cycle is recorded as follows: The control cycle is The dead zone width is The downlink speed limit is The actual value issued

[0051] That is, only when the absolute value of the difference between the instantaneous calculated value and the previous instantaneous value exceeds the dead zone width. Output is updated only when necessary; it follows up immediately to prioritize safety when risks rise and margins need to be expanded, and updates are made at a rate not exceeding [a certain threshold] when risks fall and margins shrink. The rate of release is slow. This fast expansion and slow contraction logic, combined with the dead zone, jointly suppresses the high-frequency oscillation between conservative and aggressive modes in the safety domain.

[0052] The risk adaptive barrier decision module 400 serves as the system's execution center, performing an online non-convex to convex constrained optimization within each control cycle (e.g., 10ms). Internally, it includes a dynamics mapping submodule 401, a barrier function submodule 402, and a quadratic programming optimization submodule 403. The dynamics mapping submodule 401 abstracts the system into a control affine form.

[0053] State vector Including position in geodetic coordinate system Heading angle yaw rate and longitudinal and lateral speeds Control input Desired front wheel deflection angle (Absolute quantity, not increment) and expected longitudinal acceleration This constitutes the absolute front wheel deflection angle. Directly access the affine control item It participates in the calculation of lateral force and yaw rate in the single-track / double-track model, making The dynamic evolution is mathematically closed-loop, eliminating the need for separate integral states to retain the turning angle. The smoothness of the turning and the turning rate are jointly imposed by the control increment penalty term and the upper and lower limits of the turning rate in the cost function (if an incremental velocity-type expression is adopted, the absolute turning angle of the previous step is equivalently expanded into a state, with the turning angle increment as the input, and the dynamic effects are consistent). A first-order Taylor expansion of the nonlinear equation is performed within a small time window, and the discrete state transition matrix is ​​obtained from the Jacobian matrix. With control input matrix The method for determining the linearization operating point for each prediction step needs to be specifically explained to avoid errors. Instability within the step-size field of view due to model drift: This embodiment does not uniformly fix each step across the entire field of view at the current step size. Instead of algebraically linearizing a single operating point at any given time step, the model is progressively linearized along a nominal trajectory (i.e., instantaneous linearized LTV modeling)—the nonlinear model is driven forward by a nominal control sequence (e.g., the optimal solution of the previous control cycle shifted one step forward, or a reference feedforward sequence) to obtain the nominal operating point corresponding to each prediction step. And at their respective work points, Jacobi is taken to gradually generate... The nominal trajectory is consistent with the aforementioned risk adaptive margin sequence. The arrangement shares the same forward rolling result. Thus, under high-speed, large-turn conditions, the discrete matrix of step 10 and subsequent steps still closely matches the neighborhood of the actual trajectory, avoiding physical model drift caused by single-point linearization. If necessary, one or two more iterations of sequential quadratic programming can be performed to re-linearize each operating point with the latest obtained trajectory, ensuring consistency between the prediction model and the optimal control sequence. Furthermore, robustness to nominal trajectory failure under sudden risks must be considered: when a sudden high-risk environmental disturbance (such as a vehicle suddenly appearing or another vehicle cutting in) reduces the margin... When the rapid expansion logic expands dramatically in the current step, the nominal trajectory shifted from the previous cycle may suddenly fall outside the feasible region under the newly tightened constraints. If linearization is directly applied at this illegal operating point, the extracted barrier gradient will be affected. Numerical jumps or sign flips can easily occur due to excessive approximation to the boundary, causing sequential quadratic programming to oscillate or even fail to converge from an initial point far from the feasible region. Therefore, this embodiment performs feasibility recovery when the warm-start nominal trajectory is detected to be infeasible under the updated constraints. However, this recovery does not involve solving a separate optimization problem (to avoid computational spikes in double-layer serial solutions at critical moments), but rather employs a closed-loop operation with a defined upper bound: that is, a dynamically feasible conservative nominal trajectory is generated by forward integration of the model driven by an admissible control law—for example, by applying an emergency deceleration compatible with the maximum available deceleration, or by superimposing a time-progressive air traffic control center steering law with a turning rate limit, starting from the current measured state along the discrete transfer equation. Step forward (only) (Step-by-step algebraic rolling, no iterative solution required); because each adjacent step of this nominal value is controlled by an allowable control falling within the actuator limits. The integral yields a trajectory that inherently satisfies the constraints of inholonomic vehicle dynamics and state transition, representing a dynamically feasible path, rather than an illegal spliced ​​sequence obtained by geometrically translating and clipping the poses at each step (the latter would prevent adjacent steps from being formed by any legal translation). Connect, and make Jacobi linear along it. Structural distortions cause the QP solution to diverge on real vehicles. Therefore, the LTV model obtained by redoing the time-varying linearization around the feasible nominal and the barrier discretization is physically consistent. Meanwhile, the main loop is always solved using a single constant feasible quadratic programming problem with relaxation variables (soft constraint relaxation is embedded within the same QP, rather than being solved separately). Therefore, the worst-case scenario under sudden operating conditions still requires a single QP solution rather than two sequential optimizations, and the computational complexity of feasibility recovery is strictly limited to... Algebraically calculate and allocate fixed sub-budgets, ensuring the overall budget does not exceed the limit. The sequential quadratic programming iteration employs a damped step size with a trust region or line search and applies regularized amplitude limiting to the barrier gradient, ensuring that the iteration step size remains bounded and does not diverge even when the initial point is close to or slightly beyond the feasible boundary. Only when this single always feasible solution is still insufficient within budget does it descend to the third-level emergency braking, thus avoiding both the delay spikes of double-nested optimization and the high-frequency triggering of graded backoff due to the instantaneous obsolescence of the nominal trajectory, preserving the synergistic advantages of control smoothness and multi-step prediction vision. Based on this, the barrier function submodule 402 constructs candidate control barrier functions. This allows it to simultaneously encode the vehicle's state and its spatiotemporal control. The signed distance field of the anisotropic smooth boundary and the local attitude stability domain; since the boundary has been represented by a truncated Fourier radial function as a smooth closed curve without vertices, this embodiment takes the control barrier function as a smooth signed radial margin. (in The azimuth angle from the center to the vehicle. (where the constant is a small regularity) is introduced. The softened norm replaces the naked Euclidean norm in order to eliminate central singularity—when a sudden obstacle causes the center of the newborn spacetime tube to be disrupted. Extremely close to, and even numerically covering, the vehicle's status. and At that time, the naked norm gradient This will cause an algebraic explosion due to the denominator approaching zero, leading to NaN locking in the QP linearizer; while the softened gradient... The entire space, including the center, is bounded (the modulus is always less than one), preventing numerical divergence; simultaneously, in... Within a central micronucleus smaller than a minimum threshold, the azimuth angle To maintain or degenerate the radius to isotropic within the core, in order to avoid The directional uncertainty at the center is present everywhere within the boundary neighborhood (outer thin shell) where the barrier actually functions. Differentiable, the unique non-smooth point (center) is deeply embedded inside the safe set and is never touched, hence its Lie derivative is... Continuity avoids gradient steps at the vertices and central axis of discrete orientation polygons; if it needs to be expressed using a per-orientation constraint set, then... Each directional margin is minimized using the log-sum-exponential (Log-Sum-Exp) soft minimum. polymerization( (where is the softening coefficient), this formula represents the smoothness with the true minimum margin ( This approach employs a conservative approximation method, eliminating non-differentiability of sharp ridges to ensure gradient continuity required for subsequent sparse quadratic programming and avoid the Marathos effect and line search stagnation. The barrier function also encodes the native attitude stability domain, which constrains the amplitude of avoidance maneuvers to prevent rollover or skidding due to overly aggressive steering, and defines a safe forward-invariant set. To ensure that the control law does not cause the state to escape the set at any given time interval, the evolution inequality of higher-order Lie derivatives must be satisfied. This embodiment uses a fixed class-κ slope. Based on the standard barrier inequality, an additional tightening margin, output by the safety margin calculation submodule 302, which expands with risk, is further introduced. ,get

[0054] in , Take and Consistent dimensions. This time-varying inequality constitutes a virtual force field wall that fluctuates with physical risk: when physical risk increases... As the equation increases, the right-hand side of the inequality is raised overall, affecting... By setting higher lower bound requirements, the feasible control set is correspondingly tightened, thereby eliminating control components that would cause the state to approach the boundary, forcing the vehicle to begin decelerating far from the spatiotemporal boundary and maintaining a greater dynamic safety distance; when the risk decreases... Falling back to near zero, the constraint degenerates into... The standard barrier condition for the slope frees up planning space. It should be noted that, due to... At the boundary When the inequality is given That is, the state is actively pushed away from the boundary, hence the safe set. The forward invariance holds strictly when the hard constraint is feasible; however, it is precisely because of the high-risk instantaneous... A significant increase could cause the feasible control set to shrink abruptly or even become temporarily empty, necessitating a subsequent tiered backoff mechanism as a fallback. It should also be noted that the above formula represents the safety barrier design principle in the continuous-time domain, while model predictive control seeks optimization within a discrete predictive view; therefore, in engineering, the discrete matrix is ​​not directly applied. Substitute continuous Lie derivatives Instead, the continuous condition is discretized via forward differencing (zero-order preserved) into a difference inequality that holds along the discrete prediction model: at each prediction step The above requirements ,Right now ,in For walking distance, for The value taken in this step, Discrete model given by submodule 401 It was disseminated. To After linearization at each operating point, the difference inequality is reduced to a condition concerning the decision variables. The linear inequalities allow for the legitimate construction of sparse quadratic programming barrier constraints along the entire prediction horizon. This achieves algebraic equivalence between the continuous domain security barrier and the discrete prediction horizon through zero-order preservation, thus bridging the gap between them. Further clarification is needed regarding the margin sequence in the future time domain. The method of determining the variables should be chosen to avoid coupling with decision variables and thus compromising the convexity of the problem: Note that Including relative kinetic energy term And the future speed of self-driving cars Within the field of vision, these are decision variables to be determined. If we let... During the solution process Implicit changes will cause the norm term to degenerate the barrier constraint into a non-convex constraint, thus disrupting the sparse quadratic programming structure. Therefore, this embodiment processes the entire margin sequence within a single solution. Treating it as a function of known constant parameters rather than decision variables: Pre-calculated based on the current measured state at the beginning of each control cycle. —Or freeze the current time value across the entire field of view. (Since the current relative kinetic energy represents the worst-case scenario within this period, the frozen value is conservative and safe), or it can be gradually determined along a nominal self-vehicle trajectory (e.g., shifting the optimal sequence of the previous period one step forward, or extrapolating the reference trajectory at constant speed), that is, evaluating the relative kinetic energy term on the nominal trajectory rather than the decision variable; thus In the current quadratic programming, all variables are constants, and the barrier constraints remain constant regarding... The affine nature of this. The claimed "adaptive tightening with risk" is achieved through periodic recalculation in the rolling time domain: the current state is remeasured and refreshed every 10ms. Therefore, when the vehicle actually decelerates and its relative kinetic energy decreases, the margin in subsequent cycles also decreases. The adaptive effect is truly valid in the closed-loop rolling time domain, rather than requiring self-consistency within a single open-loop view. If higher fidelity is required, the obtained trajectory can be recalculated. Iterate through one to two sequential quadratic programming iterations to make the margin and trajectory self-consistent, and its additional computational overhead is bounded.

[0055] The quadratic programming optimization submodule 403 has a length of (For example, taking 20) Constructing a cost function within the rolling optimization field of view.

[0056] in For global path point reference, The weight matrix represents the control increment between two adjacent prediction steps (used to suppress control chattering and constrain the angular rate). The constraints are respectively levied on trajectory deviation, energy consumption, and control chattering. The discrete kinematic equation constraints given by the dynamics mapping submodule 401, the hardware-inherent upper and lower bound constraints on rotation and acceleration, and the linearized barrier difference inequality constraints derived from the zero-order preserved discretization of the barrier function submodule 402, which hold along multiple sets of discrete prediction models, are simultaneously combined to reshape the feasible region of the quadratic programming problem. Then, a sparse quadratic programming solution engine customized based on the interior point method or the effective set method is invoked to obtain the feasible region within a millisecond-level budget time. The minimum optimal control sequence that satisfies all barrier constraints As mentioned earlier, the introduction of additional margins can significantly tighten the feasible region during high-risk moments, potentially making standard quadratic programming infeasible or difficult to converge within a given time budget. To ensure that legitimate instructions are issued in each cycle, this submodule sets up a tiered rollback mechanism and clearly defines the safety properties provided by each level as follows: In the first level, the solver engine first attempts a complete solution under hard constraints; if a solution satisfying all barrier inequalities is successfully obtained at this level, the issued control strictly maintains the safety set. Forward invariance. Second level, if time budget... If the condition returns infeasible or fails to reach convergence accuracy within 8ms (for example), the barrier inequality is relaxed to one with relaxation variables. Soft constraints are applied, and a penalty term is added to the cost function. , Choose a sufficiently large positive value to account for the violation of the barrier constraints while ensuring the feasibility of the problem. The first level compresses to a minimum; it does not claim strict forward invariance, but rather achieves graceful degradation of safety by minimizing safety constraint violations when hard constraints are temporarily infeasible. The cost is allowing the state to briefly approach or even slightly exceed the original safety boundary within a controlled and bounded range. The third level, if the remaining budget is still insufficient to solve the soft constraints, employs pre-calculated backup control. This involves shifting the previously verified feasible control sequence one step forward, or applying a bounded emergency deceleration compatible with the maximum available deceleration while maintaining the previous turning angle, ensuring the vehicle remains as close as possible to the neighborhood of the previously verified safe set. Thus, regardless of whether the online optimization converges within budget, the system outputs a legal and executable control variable in this cycle: this control variable strictly maintains forward invariance when hard constraints are feasible, and achieves graceful degradation of safety by minimizing safety constraint violations and prioritizing the use of verified safe controls when relaxation or backoff is required. After obtaining the control sequence, its first segment is extracted. The data is converted into Controller Area Network (CLAN) bus messages or Industrial Ethernet protocol frames, such as the steering angle request value of the steer-by-wire motor and the pressure build-up command of the electro-hydraulic brake, and distributed to the physical execution layer to complete the decision-making loop of this cycle.

[0057] This invention also provides an intelligent safety decision-making and control method corresponding to the above system, see [link to relevant documentation]. Figure 2 This method is executed periodically by the aforementioned domain controller or edge computing server and includes the following steps.

[0058] Step S100: Concurrent access and spatiotemporal alignment of multimodal data. This step further includes: Step S110: Concurrently receiving asynchronous signals from multiple sources, such as local pose, video images, 3D point clouds, and vehicle-road cooperative beacons, using a multi-threaded circular buffer, and adding an arrival timestamp to each frame; Step S120: Defining a set of discrete standard time steps based on the master control cycle period, and applying a time alignment operator to any modality. Select the raw observation closest to each standard step time and assign its data to that standard step, then first use the local pose sequence. Historical observations are compensated for by rigid body relative pose transformation and anchored to a common reference frame with the current vehicle pose (eliminating vehicle self-motion), followed by an Euclidean distance finder. Each object with spatial attributes is mapped to a geographic topology grid with the current vehicle pose as the origin, and the grid is determined according to the rule of "adding the vehicle node and reference path node to the union of occupied grid nodes within the window, and removing idle nodes". A key spatial node, and thus in Each node is concatenated into a unified multimodal feature tensor. Step S130, record the mode In standard steps Structured observation feature slices are For each mode, the packet loss rate is calculated by scrolling within a sliding window. Delay and obsolescence And first use a channel-by-channel feature scale Standardize the heterogeneous components to Recalculate the normalized new information (The denominator is the lower bound of the normal range) Clamping to prevent division by zero amplification of the empty field. It is obtained by one-step linear extrapolation of the preceding effective standardized features, under the empty field condition. Set to zero), press Obtain the modal confidence coefficients, along with the validity mask for zero-padding. As a reliability label, it is output along with the tensor.

[0059] Step S200: Causal de-obfuscation of multimodal spatiotemporal features and trajectory prediction. This step further includes: Step S210: obtaining characterizations for each modality using independent encoders. Calculate the gating confidence level for each spatial node. (The gated linear layer is shared among all nodes, and is related to the number of nodes) (Irrelevant), and link it to the modal level credibility coefficient Multiplying and broadcasting along the feature dimension constitutes an effective gating, according to Obtain fusion characterization ( For the learnable projection matrix, Applying right multiplication to each row still results in the same output. ), and first reduce the weight of the failure mode with a sudden increase in noise; step S220, with the inclusion A learnable dictionary of confounding factors for each prototype approximates a latent set. The class posterior on the prototype is output from the auxiliary branch, and orthogonal constraints are applied to the main branch. and adjust the backdoor Specifically, this is implemented as... A finite weighted aggregation of prototypes (condition terms are obtained by modulating each prototype with the main path features, and prototype edge frequencies are taken a priori) is used to cut off spurious associations; step S230, spatial topological features are aggregated by Chebyshev polynomial graph convolution to obtain Then, using a selectively structured state-space sequence model, with For step-by-step input, and generated in real time from the input projection. and obtained by zero-order preservation ,according to , Complete the time-series evolution with linear complexity and output the expected predicted trajectory. With the set of predicted covariance matrices .

[0060] Step S300: Dynamic risk field mapping and spatiotemporal tube generation. This step further includes: Step S310: Calculating the deviation of the predicted covariance. Value at Risk (Depend on (obtained from the induced prediction distribution), and then using conditional value of risk. Tail sampling is performed; the spatiotemporal center is directly taken from the expected predicted trajectory output by module 200, i.e. Then, the physical information neural network, trained offline on various road network geometric distributions, takes the prediction center, prediction covariance, and current local drivable domain geometric descriptor as input, and performs a single forward inference (multiple obstacles in batches and parallel) on each obstacle to generate anisotropic angular radius profiles. The obtained spatiotemporal tube is geometrically projected and trimmed in each direction according to its orientation, and the radius of each direction is shrunk to the real-time clearance of that direction. Within this framework, strict measures are taken to ensure that the road network boundaries are not encroached upon (the lateral azimuth near the boundary is tightened, while the lateral azimuth in open areas continues to expand according to the covariance tail, avoiding overall pipe collapse), and then quality is perceived at the system level. The safety risk radius is obtained by stepwise correction. Based on this, a time-space tube is generated for each obstacle. Step S320: Poll the local bus to obtain the vehicle speed and braking potential, and calculate the compensation margin according to the dimensionless normalized compensation margin formula mentioned above. The output value is then obtained through asymmetric rate limiting processing with a dead zone for rapid expansion and slow contraction. .

[0061] Step S400: Constrained optimization solution under attitude-motion coupling. This step further includes: Step S410: Modeling the system as a control affine form and progressively performing a first-order Taylor expansion (time-varying linearization) along the nominal trajectory, obtaining the discrete state transition matrix and control input matrix at the nominal operating point of each prediction step to avoid model drift under high-speed, large-turn conditions due to full-view single-point linearization; Step S420: Constructing the control barrier function for the anisotropic smooth boundary (truncated Fourier radial, differentiable everywhere) of the spatiotemporal tube and the attitude stability domain. (If necessary, use Log-Sum-Exp soft minimum aggregation for each directional margin to ensure gradient continuity and avoid step jumps at vertices and the midline), with a fixed class-κ slope. By introducing an additional tightening margin with risk inflation into the standard barrier inequality, we obtain the continuously time-varying barrier inequality. The model is then discretized using forward differencing (zero-order preserved) into a discrete prediction model. Establishment, Regarding Linear differential barrier constraints (where the margin sequence) At the beginning of each cycle, a constant is pre-calculated based on the current measured state and does not change with the decision variable. The adaptive effect is realized through cycle-by-cycle recalculation in the rolling time domain to maintain the constraint regarding... (affineness); Step S430, construct the cost function within the scrolling view. (including control increment) (Boottle penalty), solving sparse quadratic programming by combining kinematic constraints, actuator boundary constraints, and multiple sets of discrete barrier difference constraints (when sudden constraint tightening and warm-up start-up nominal infeasibility occurs, generating a dynamically feasible conservative deceleration nominal by forward integration of the admissible control law (stepwise forward pushing along the state transition equation, Algebraic, non-quadratic solution (no geometric translations that violate kinematics), soft constraint relaxation is embedded in the same consistently feasible QP, so the worst case is still a single solution and no two-level serial optimization is generated. When it is not feasible within the budget time, soft constraint relaxation solution (minimizing constraint violations with large penalties and achieving graceful degradation of safety) and backup control are executed in a hierarchical rollback.

[0062] Step S500: Control command issuance and closed-loop iteration. Extract the first segment of the optimal control sequence and convert it into a controller area network bus message or industrial Ethernet protocol frame, then send it to the execution layer such as drive-by-wire steering and electro-hydraulic braking. When the next clock interrupt arrives, return to step S100 to start a new round of perception and optimization.

[0063] To more clearly illustrate the collaborative working mechanism of the above system and method in dynamic and complex scenarios, a specific application scenario is used as an example. Suppose a vehicle is approaching an intersection without traffic lights at a moderate speed, with a building on one side of the intersection obstructing the view, and a short-term heavy rainfall occurs. In the early stage of the rainfall, the video image frames show local blurring and noise. The alignment submodule 102 lowers the confidence coefficient of the camera modality based on its packet loss rate and information anomalies. The gating link then reduces the visual contribution and instead relies on LiDAR and vehicle-to-infrastructure text beacons. At the same time, the causal dual-branch classifies the common modality factor of rainfall, which consistently causes the collective degradation of multimodal data, into the mixed factor prototype and orthogonally strips it, so that the prediction does not mistakenly establish a strong correlation between visual noise and the deceleration of the vehicle in front. When a non-motorized vehicle, previously obscured by a building, suddenly enters the intersection, the prediction covariance increases sharply in that direction. The spatiotemporal tube modeling submodule, through conditional value-at-risk tail sampling, rapidly expands the spatiotemporal tube radius of the obstacle. This, combined with the additional radius increment due to decreased perception quality, forms a significantly convex repulsive envelope from the control perspective. Simultaneously, the safety margin calculation submodule, based on the increase in relative kinetic energy and uncertainty, adds an additional tightening margin. The instantaneous expansion and subsequent smoothing with a dead zone cause the right-hand side of the Lie derivative inequality to rise, tightening the feasible control set. During the solution process, the decision module initially planned a smooth, straight trajectory, but due to approaching the expanded spatiotemporal boundary and being subject to a higher lower bound requirement after tightening, it triggered a steep penalty wall. The solver then modified the solution within the attitude stability limits, finding a feasible trajectory that decelerates and deviates slightly. If this cycle fails to converge within the budget due to the sudden shrinking of the feasible region, a tiered backoff mechanism either uses soft constraint relaxation (minimizing constraint violations with a large penalty) or uses the previously verified safe control sequence as a fallback to ensure timely issuance of steering and braking commands. After the non-motorized vehicles leave and the rainfall weakens and perception quality improves... As the spatiotemporal radius slowly decreases at a limited rate, the constraint degenerates into a standard barrier condition, and the system automatically releases the planned space and restores normal traffic efficiency.

[0064] As can be seen from the above system structure and methodology, the reliability coefficient of the real-time evaluation of each mode at the front end not only isolates the failed channel in the gating stage, but also amplifies the corresponding spatiotemporal radius and additional safety margin after system-level aggregation, so that the perception degradation is seamlessly translated into the expansion of the safety envelope and the tightening of the safety constraints; the prediction variance converged by causal deconfusion limits the reasonable upper bound of the margin expansion, avoiding excessive collapse or unnecessary expansion of the safety domain; the additional margin with risk expansion directly raises the right-hand side of the Lie derivative inequality, so that the feasible domain of the quadratic programming breathes and expands with the physical risk; the asymmetric rate limiting with dead zone and the graded backoff together ensure that this dynamic process is both solvable in real time and stable and undisturbed within the millisecond period, and strictly maintains forward invariance when hard constraints are feasible, and achieves graceful degradation of safety when hard constraints are temporarily infeasible. Those skilled in the art can reproduce the complete system and implement the corresponding methods based on the above-mentioned variable definitions, calculation relationships, network parameterization methods, triggering conditions, and instruction conversion paths. They can also make adaptive substitutions to the sampling frequency, buffer depth, network structure, prototype dictionary size, feature reference scale, field of view length, gain value, and solver implementation without departing from the concept of this invention. All of these should be covered within the protection scope of this invention.

Claims

1. An intelligent safety decision-making and control system integrating spatiotemporal big data and multimodal perception, characterized in that, include: The data access and spatiotemporal alignment module is used to concurrently acquire multi-source asynchronous sensor data through a circular buffer, map heterogeneous data to a standard time step based on a timestamp alignment operator, eliminate the influence of self-motion through vehicle pose compensation, and construct a unified feature tensor and validity mask. The multimodal causal fusion module is used to deweight the failed channel through a cross-modal adaptive gating attention mechanism, use a dual-branch causal backdoor to remove spurious correlations introduced by confounding factors, and output the trajectory prediction expectation and prediction covariance based on a selective structured state-space sequence model. The risk field and spatiotemporal tube generation module is used to perform conditional risk value sampling based on the predicted covariance matrix, generate anisotropic spatiotemporal tube repulsion envelopes through physical information neural networks, and adjust the safety margin in real time based on system-level perceived quality. The risk-adaptive barrier decision module is used to construct the control barrier function for encoding the spatiotemporal boundary and attitude stability domain. It introduces a risk-adaptive tightening margin into the fixed slope barrier inequality and issues control commands in real time through a quadratic programming solver.

2. The system according to claim 1, characterized in that, The data access and spatiotemporal alignment module is equipped with a data reliability assessment unit, which is used to calculate the packet loss rate for each modality. Delay and obsolescence and normalized new information And generate a credibility coefficient based on the parameters. : ； in, It is a non-negative weight, used to reduce the weight and isolate failed channels at the front end.

3. The system according to claim 1, characterized in that, The multimodal causal fusion module includes a heterogeneous factor prototype dictionary, used to calculate the prototype posterior through auxiliary branches and apply orthogonal constraints to the main prediction branch. In order to cut off the backdoor path of false causality.

4. The system according to claim 1, characterized in that, The risk field and the spacetime tube envelope generated by the spacetime tube generation module are expressed using a truncated Fourier radial function. Indicates, and satisfies the azimuth angle The anisotropic distribution is truncated, and geometric projection clipping is performed on the radius in each direction, shrinking it to the clearance distance with the boundary of the local drivable domain. Within.

5. The system according to claim 1, characterized in that, The risk adaptive barrier decision module embeds a hierarchical backoff mechanism in the quadratic programming. When the solver fails to converge within the budget time, it performs soft constraint relaxation or uses the backup control sequence from the previous cycle.

6. An intelligent safety decision-making and control method integrating spatiotemporal big data and multimodal perception, characterized in that, Includes the following steps: Step S1: Concurrently access and align the multi-source asynchronous sensor data with timestamps, and compensate historical observations to a unified coordinate system through rigid body transformation; Step S2: Calculate the modal confidence coefficient and combine it with the dual-branch causal backdoor intervention to remove spurious associations, and perform time series prediction through a selective structured state-space sequence model; Step S3: Tail sampling is performed based on the predicted covariance to generate anisotropic spatiotemporal tubes, and the safety envelope is expanded in real time according to the system-level perception quality. Step S4: Construct a continuous time-varying barrier inequality and introduce an adaptive tightening margin. The optimal control command is solved by discretization difference constraints and quadratic programming.

7. The method according to claim 6, characterized in that, In step S2, a selective structured state-space sequence model is used to generate discrete parameters in real time through input projection. , and It completes long-range temporal memory transfer with linear complexity and outputs the expected predicted trajectory. With predicted covariance .

8. The method according to claim 6, characterized in that, In step S4, an additional tightening margin is added. The calculation formula is: ； in, For system-level perceived quality, The value is sent after being processed by asymmetric rate limiting with dead zone to suppress frequent switching of control quantities.

9. The method according to claim 6, characterized in that, In step S4, the control barrier function Defined as a smooth signed distance field based on anisotropic radial margin, its Lie derivative inequality discretization is: ； in, For a fixed slope, To provide additional margin values ​​for the corresponding prediction steps, the constraints are transformed into values ​​related to the control inputs. The linear form guarantees the convexity of the quadratic programming.

10. The method according to claim 6, characterized in that, In step S4, during a single optimization loop, the margin sequence Treated as a known constant parameter, and refreshed periodically according to the current measured state within the closed-loop rolling time domain, the risk adaptive convergence is achieved through the closed-loop feedback of the rolling time domain.

Images