Unified vla model based spatio-temporal constraint end-to-end autonomous driving system and method

Abstract

The application relates to the field of artificial intelligence and automatic driving technology and discloses a space-time constraint end-to-end automatic driving system and method based on a unified VLA model, which comprises a unified encoder, a computing power management unit, a core neural network and an output decoder. In the application, physical time is mapped into a computing power budget vector, a gating operator embedded in the core neural network is used to calculate feature information entropy and computing power consumption in real time, and a topology reconstruction logic based on entropy and computing power joint constraint is used to dynamically trigger a computation graph collapse operation at an instruction level, so that the network reasoning depth is adaptively stretched according to the physical time constraint. The application establishes an isomorphic mapping of the computation model topology structure and the physical time constraint, determines the upper limit of the deterministic delay under extreme working conditions, solves the dimension mismatch problem of the existing static computation graph under the hard real-time constraint, and realizes optimal bit allocation of the computing resources in the time dimension.

Description

Technical Field

[0001] This invention relates to a spatiotemporal constrained end-to-end autonomous driving system based on a unified VLA model, belonging to the fields of artificial intelligence and autonomous driving technology. Background Technology

[0002] Current end-to-end autonomous driving systems based on deep neural networks employ large-scale neural network models, such as Transformer, to directly map multimodal data collected by onboard sensors into vehicle control commands. This integrated perception and decision-making design avoids information loss caused by manually defined interfaces in traditional modular architectures, and improves the system's ability to generalize to complex long-tail scenarios, making it a key path to achieving high-level autonomous driving. As autonomous driving technology is increasingly applied to unstructured, high-frequency game scenarios such as the last mile, existing technologies face a contradiction between static allocation of computing resources and dynamic constraints of physical time. In open road cruising scenarios, the decision-making time window is ample, while in extreme scenarios such as crowded markets, narrow alleys, or encountering sudden obstacles, the physical laws compress the decision-making deadline to the millisecond level. The drastic fluctuations in the physical time window pose a challenge to the real-time performance of computing systems.

[0003] Existing control methods are insufficient to meet extreme timeliness requirements. For example, Chinese invention patent application CN121005018A discloses a method and device for controlling autonomous vehicles. The solution introduces a VLA (Visual Language Action) model, which integrates user natural language commands and visual features to output a decision-making reason and process decision-making thought chain, thus solving the problems of fuzzy command recognition and lack of system interpretability. Industry attempts have tried to introduce multi-model switching or rule-based degradation strategies, such as switching to a lightweight small model when an emergency is detected. However, this discrete state machine-based switching method introduces additional state monitoring and switching overhead, and the inconsistent behavioral characteristics between models cause control commands to jump, failing to solve the problem of adaptive calculation of a single model under continuous dynamic time constraints.

[0004] Therefore, the technical problem to be solved by this invention is to construct a computer system with elastic depth and dynamic topology reconstruction capabilities, perceive the remaining time constraints of the external physical world, continuously adjust the neural network inference structure and computing power allocation in real time, ensure decision completeness while adhering to the deterministic delay boundary under extreme conditions. Summary of the Invention

[0005] To address the problems mentioned in the background art, the technical solution of this invention is as follows: A spatiotemporal constrained end-to-end autonomous driving system based on a unified VLA model, the system comprising a unified encoder, a computing power management unit, a core neural network, and an output decoder:

[0006] A unified encoder is used to map multimodal input signals into a sequence of feature tensors of a unified dimension.

[0007] The computing power management unit is used to read the hardware clock at the start of the inference cycle and map the remaining physical time until the hard decision deadline to the floating-point operation count budget vector of the current cycle. The floating-point operation count budget vector is used to define the maximum number of matrix operations allowed to be executed within the current inference cycle.

[0008] The core neural network is used to perform cascaded inference operations on feature tensor sequences, and contains multiple cascaded computational layers.

[0009] The system also includes gating operators embedded in the output of each computational layer of the core neural network and topology reconstruction logic connected to the gating operators. The gating operators are used to calculate the Shannon entropy of the output feature tensor of the current computational layer and, based on the pre-stored computational power consumption model of the core neural network, calculate the estimated power consumption required to execute subsequent computational layers. The topology reconstruction logic is used to trigger a computation graph collapse operation when the Shannon entropy is lower than a preset deterministic threshold or the value indicated by the floating-point operation count budget vector is less than the estimated power consumption. The computation graph collapse operation is used to cut off the connection path between the current computational layer and subsequent computational layers at the instruction level and directly project the output feature tensor of the current computational layer to the output decoder, thereby establishing a positive correlation between the actual inference depth of the core neural network and the physical remaining time.

[0010] Preferably, the core neural network further includes a hybrid expert module, which has a routing network and multiple expert networks configured in parallel; the routing network is used to receive the floating-point operation count budget vector as a sparsity constraint parameter; when the computing power margin indicated by the floating-point operation count budget vector is lower than a preset warning threshold, the routing network performs a computing power throttling operation; the computing power throttling operation includes downgrading the number of activated expert networks from the standard configuration to the minimum configuration, thereby reducing the computing power consumption of a single layer of computation.

[0011] Preferably, the unified encoder includes visual semantic beacon encoding logic for constructing micro-location indexes for unstructured environments; the visual semantic beacon encoding logic extracts semantic feature vectors of target objects from natural language instructions and performs cross-attention calculations with visual feature tensors as query vectors; the system extracts visual features of high-response regions as beacon feature tensors based on attention weight distributions and injects the beacon feature tensors into the feature tensor sequence to establish a relative spatial coordinate system centered on the target object in the core neural network.

[0012] Preferably, the decision logic for the gated operator to perform computation graph collapse operation satisfies any one of the following separation conditions: deterministic constraints: Computing power boundary constraints: ;in, The Shannon entropy of the output feature tensor of the i-th layer is calculated. This is a preset deterministic threshold; Estimate the number of floating-point operations remaining. To estimate the power consumption required to execute subsequent computation layers; when any of the above conditions are met, the topology reconstruction logic generates an interrupt instruction to trigger the computation graph collapse operation, causing the core neural network to terminate forward propagation in the current computation layer.

[0013] Preferably, the core neural network is configured to perform counterfactual reasoning operations based on causal attention masks; the counterfactual reasoning operations include inserting hypothetical action feature tensors into a sequence of feature tensors and applying a counterfactual mask matrix; the counterfactual mask matrix is ​​used to allow state feature tensors at future time steps to focus on hypothetical action feature tensors while masking the actual observation feature tensors at the current time step, thereby generating a future state prediction sequence representing the hypothetical intervention within the feature space of the core neural network.

[0014] Preferably, the core neural network internalizes world-state autoregressive prediction logic, which is used to extrapolate the dynamic changes of the physical environment in the continuous time domain. The world-state autoregressive prediction logic receives a time-conditional feature tensor containing arbitrary time interval values ​​and outputs a predicted state feature tensor corresponding to the time interval. The system also includes adaptive step size adjustment logic, which is used to calculate the current safety margin based on the predicted state feature tensor, and reduce the time interval value of the next prediction step when the safety margin is lower than a preset safety threshold, thereby improving the accuracy of physical extrapolation in the time dimension.

[0015] Preferably, the output decoder includes game-theoretic scoring logic for selecting the optimal execution strategy from multiple candidate trajectories. The game-theoretic scoring logic calculates the dynamic utility value of each candidate trajectory, with the calculation weight controlled by the floating-point operation budget vector. As the remaining time indicated by the floating-point operation budget vector decreases, the game-theoretic scoring logic non-linearly increases the weight of the efficiency benefit index in the dynamic utility value, thereby prioritizing the shortest-duration trajectory strategy under time-sensitive conditions. The system adopts a layered deployment architecture with cloud-edge collaboration. The core neural network includes a first computing model deployed at the edge and a second computing model deployed in the cloud. The first computing model independently performs inference when the floating-point operation budget vector meets real-time requirements. When the Shannon entropy output by the first computing model is higher than a preset uncertainty threshold and the floating-point operation budget vector allows for network latency, the system sends an inference request to the second computing model and uses the output of the second computing model as the final decision basis.

[0016] Preferably, the output decoder includes a causal explanation generation logic for generating natural language text representing the decision logic; the causal explanation generation logic extracts the state data of the gated operators that trigger the computation graph collapse operation in the core neural network, as well as the weight distribution data in the game adjudication scoring logic; the causal explanation generation logic maps the extracted data to a preset language template to generate an explanation text containing causal relationships, and the explanation text is used to describe the logical basis for the system to select a specific strategy under specific time constraints.

[0017] Preferably, the unified encoder also includes time budget encoding logic; the time budget encoding logic decomposes the floating-point operation count budget vector into strategic-level budget components, tactical-level budget components, and instantaneous-level budget components; the strategic-level budget components are used to constrain the activation state of long-term planning experts in the core neural network, the tactical-level budget components are used to constrain the activation state of local interactive game experts, and the instantaneous-level budget components are used as a direct threshold benchmark for gating operators to perform computation graph collapse operations.

[0018] A spatiotemporal constrained end-to-end autonomous driving method based on a unified VLA model, comprising the following steps:

[0019] A unified encoder is used to map multimodal input signals into a sequence of feature tensors of a unified dimension;

[0020] The computing power management unit reads the hardware clock at the start of the inference cycle and maps the remaining physical time until the hard decision deadline to a floating-point operation budget vector for the current cycle, so as to define the maximum number of matrix operations allowed to be executed within the current inference cycle.

[0021] Cascaded inference operations are performed on feature tensor sequences using a core neural network.

[0022] During the inference operation, the Shannon entropy of the output feature tensor of the current computing layer is calculated using the gate operator embedded in the output of each computing layer, and the estimated power consumption required to execute the subsequent computing layer is calculated in real time based on the pre-stored computing power consumption model of the core neural network.

[0023] The computation graph collapse operation is triggered when the Shannon entropy is lower than a preset deterministic threshold or the value indicated by the floating-point operation count budget vector is less than the estimated power consumption.

[0024] The computation graph collapse operation is performed to cut off the connection path between the current computation layer and the subsequent computation layers at the instruction level, and directly project the output feature tensor of the current computation layer to the output decoder. This establishes a positive correlation between the actual inference depth of the core neural network and the physical remaining time, ensuring that the inference latency is not higher than the physical remaining time.

[0025] Compared with the prior art, the beneficial effects of the present invention are:

[0026] 1. In the spatiotemporal constraint end of the unified VLA model, a computation graph topology is constructed to adaptively reconstruct the computer system according to physical time constraints. The remaining time in the external physical world is converted into a budget of floating-point operations in the internal computing system through a computing power mapping unit. This budget is encoded as a computing power budget feature vector and injected into the network. Combined with the real-time monitoring of the entropy of the output feature information of the current layer by the gate operator, a dynamic pruning mechanism based on confidence and computing power margin is established. This allows the neural network inference depth to elastically scale according to the urgency of decision-making and the uncertainty of information in the current scenario. When time is sufficient or the scenario is complex, deep networks and multi-expert modules are automatically called to obtain high-precision inference. When time is tight or the scenario is simple, subsequent unnecessary computation layers are physically bypassed. This solves the mismatch between fixed computing power consumption and dynamic physical time window from the bottom layer of the computing model, and realizes on-demand allocation of computing resources in the time dimension.

[0027] 2. Establish the deterministic response boundary for end-to-end autonomous driving under extreme conditions. By introducing a sparsity penalty term based on the remaining computing power budget in the hybrid expert network, the number of activated experts is forced to be reduced to the minimum configuration when the physical deadline of the routing network is approaching. This triggers the cascade collapse of the computation graph, ensuring that the system can terminate inference within the worst-case execution time and output the optimal solution allowed by the current computing power, regardless of the fluctuation in the complexity of external sensor input data. This eliminates the systemic risk of control signal interruption caused by computation timeout in traditional static models, and ensures hard real-time safety in high-frequency game scenarios such as the last mile of the vehicle.

[0028] 3. Achieve a unified computational approach that integrates visual perception, language understanding, action generation, and world modeling within a single converter architecture. By mapping multimodal sensor data, vehicle status, and time budget into a unified sequence of isomorphic feature vectors, it eliminates semantic loss and information bottlenecks caused by interface definitions between different task modules in traditional modular architectures. It utilizes an autoregressive prediction mechanism to dynamically evolve the implicit modeling environment within the network, and directly performs counterfactual reasoning and trajectory generation within the causal attention mask feature space. The end-to-end, micro-optimizable approach allows the decision-making module to directly utilize fine-grained environmental features from the raw perception data. In the absence of high-precision maps or unstructured environments with GPS signals, it achieves precise micro-positioning and navigation through visual semantic beacon attention interaction.

[0029] 4. The spatiotemporal constraint computation graph collapse mechanism of the present invention has universality of application scenarios. The core entropy computing power joint constraint logic is decoupled from the specific execution mechanism. In addition to autonomous driving scenarios, it can be extended to embodied intelligence fields with hard real-time constraints on decision delay, such as real-time grasping of industrial robots and dynamic obstacle avoidance of drones. Attached Figure Description

[0030] Figure 1 This is a system architecture diagram of the spatiotemporal constraints and computational graph dynamic reconstruction based on the unified VLA model of this invention;

[0031] Figure 2 This is a dynamic response curve of the multidimensional weights changing with physical time in the game adjudication strategy of this invention;

[0032] Figure 3 This is a diagram showing the cloud-edge collaborative dual-track deployment and data flow of the spatiotemporal computing power elastic hub of this invention. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] This invention provides a spatiotemporally constrained end-to-end autonomous driving system based on a unified VLA model, comprising four core logical entities: a unified encoder, a computing power management unit, a core neural network, and an output decoder. The unified encoder is used to homogenize heterogeneous external perception data; the computing power management unit is used to establish the physical boundaries of the current inference cycle; the core neural network, as the inference subject, is used to dynamically reconstruct its computational path based on the physical boundaries; and the output decoder is used to map high-dimensional features into specific vehicle control commands and explanatory text. Addressing the objective technical challenge of unstructured road environments, such as congested markets and old residential areas, where the physical time window fluctuates drastically, the core design logic of this system lies in establishing an isomorphic mapping between computational consumption and time constraints. The system, through the computing power management unit at the beginning of each inference cycle... It reads the high-precision system clock through a hardware interface and calculates the distance from the hard decision deadline to the current time. Physical remaining time The remaining physical time Instead of being directly used as feature input, it queries the real-time performance status table of the current in-vehicle computing platform, such as the NPU or GPU, including core frequency, memory bandwidth utilization, and temperature status. Deterministically map to a budget of the maximum number of floating-point operations (FLOPs) allowed to be consumed in the current frame, generating a floating-point operation budget vector. This vector defines the maximum number of matrix operations allowed to be performed within the current inference cycle, constituting the boundary constraint for all subsequent neural computation activities.

[0035] Floating-point operation count budget vector The generation originates from a pre-built dynamic performance feature library. This includes controlling the computing platform junction temperature at multiple discrete temperature points ranging from -40℃ to 85℃ in a constant-temperature testing environment. At each temperature point, the matrix operation load is gradually increased until thermal throttling protection is triggered. The actual number of floating-point operations per second and memory bandwidth data of the system's stable output are recorded at each temperature point. During online operation, the computing power management unit collects the current hardware temperature and voltage status and directly calculates the maximum matrix operation limit that the hardware can actually execute within the current physical time window using linear interpolation of the feature library. After obtaining the above budget constraints, the unified encoder performs isomorphic processing of multimodal signals. For point cloud data from LiDAR, image data from cameras, and inertial navigation data from the vehicle itself, the unified encoder uses the corresponding feature extraction backbone network to transform them into a unified-dimensional feature tensor sequence. In this process, the unified encoder also includes time budget encoding logic and budgets the above floating-point operation counts into a vector. It is decomposed into three dimensions: strategic budget component, which represents the computing power quota for long-term planning; tactical budget component, which represents the computing power quota for local interaction; and instantaneous budget component, whose value directly corresponds to the computing power margin required for the current millisecond-level response. These three components are encoded into specific time budget tokens, which are concatenated with visual tokens, language tokens and state tokens to form a long sequence input, which is then injected into the core neural network.

[0036] The core neural network is the main body of this system for performing inference tasks. It contains multiple cascaded Transformer computation layers, such as 24 or 48 layers. To address the decision truncation problem caused by computational overload in traditional static models under extreme and demanding conditions, this invention physically embeds a lightweight gating operator at the output of each layer or a specific interval layer of the core neural network. This gating operator does not participate in feature extraction but only performs an information-theoretic monitoring task. For the feature tensor output by the i-th computation layer... The gating operator calculates the Shannon entropy of the probability distribution. Shannon entropy In mathematics, entropy represents the degree of uncertainty in the model's current understanding of the scene. The lower the entropy value, the more certain the feature representation tends to be. At the same time, based on a pre-stored computing power consumption model, the gating operator calculates in real time the estimated power consumption required to execute all subsequent remaining computing layers, i.e., layers i+1 to N. The system includes a pre-stored computational power consumption model that records the standard FLOPs required for matrix multiplication at each layer of the network. Based on these calculations, the topology reconstruction logic connected to the gate operator performs a core computational graph collapse judgment, which follows a separation constraint logic based on determinism and computational power boundaries. Specifically, the system presets a deterministic threshold. Real-time comparison of gating operators: if the current entropy value This indicates that the model's understanding of the current scenario is sufficiently reliable, requiring no further computation and triggering collapse; if the current remaining budget This indicates that the physical time window is about to expire and cannot support the complete operation of subsequent computing layers. A forced collapse must be triggered. As long as any of the above conditions are met, the topology reconstruction logic will generate an interrupt instruction.

[0037] When this inequality holds, it indicates that the model has either obtained sufficiently definite feature representations or that the remaining computing power is insufficient to support complete deep inference. At this point, the topology reconstruction logic generates an interrupt instruction, triggering a computation graph collapse operation. This cuts off the connection path between the current computation layer i and the next computation layer i+1 at the instruction set level, redirecting the feature tensors that were originally flowing to the next layer. The shallow projection interface of the output decoder is directly redirected. The instruction-level execution logic of the computation graph collapse operation is directly controlled by the runtime instruction queue of the inference engine. When the topology reconstruction logic generates an interrupt instruction, the system writes a no-operation instruction to the underlying accelerator command stream buffer or redirects the input address pointer of the subsequent tensor operation kernel to a preset empty memory address. This physically blocks the subsequent matrix multiplication instruction issuance and memory read requests at the hardware execution flow level, not only shielding the output results at the software application layer. This ensures that cutting off the computation path does not occupy physical computing resources and achieves zero-latency inference termination. This mechanism establishes a positively correlated physical mapping: the more pressing the physical time, the lower the computing budget, the easier it is to satisfy the inequality, and the shallower the actual execution depth of the network. This ensures that the system can output the optimal solution allowed by the current computing power before any physical deadline arrives. In addition, In the computational layer where collapse is not triggered, the core neural network further performs fine-grained computational throttling through a hybrid expert module. The hybrid expert module contains multiple expert networks configured in parallel, and the routing network controls the flow of data. In this system, the routing network not only receives feature tensors as input, but also receives a floating-point operation count budget vector as a sparsity constraint parameter. When the computational power margin indicated by the budget vector is lower than a preset warning threshold, the routing network performs computational throttling. By injecting a mask of negative infinity or a high-intensity sparsity penalty term before the Softmax calculation of the routing weights, the strategy originally configured to activate Top-K, such as K=4 experts, is downgraded in real time to activate the minimum configuration, such as K=1 expert. This operation directly reduces the number of matrix multiplications that need to be performed at the hardware level, releasing microsecond-level computational resources.

[0038] For unstructured environments lacking clear geographical markers, such as old residential areas and roads without lane markings, the unified encoder integrates visual semantic beacon encoding logic. This logic parses user-input natural language commands, such as "stop under the red awning," and extracts semantic feature vectors representing the target object, i.e., the red awning, using a pre-trained language model. This semantic feature vector is used as a query vector and cross-attention is calculated with the visual feature tensor (key / value) of the current frame. The system analyzes the generated attention weight distribution map, extracts visual features corresponding to spatial regions with response values ​​higher than a preset threshold, and encapsulates them into a beacon feature tensor. This beacon feature tensor is re-injected into the feature sequence, constructing a relative spatial coordinate system with the target object as the origin within the latent space of the core neural network. This provides a basis for vehicle positioning even when GPS signals are lost or high-precision maps are unavailable. By using micro-location indexing based on visual features, centimeter-level precise parking is achieved. To enhance the system's causal reasoning capabilities in complex game scenarios, the core neural network is configured to perform counterfactual reasoning operations based on causal attention masks. When the system identifies potential game objects, such as pedestrians with unclear intentions, it inserts special feature tensors representing hypothetical actions, such as accelerating through or forcibly cutting in, into the input sequence. During the reasoning process, a specific counterfactual mask matrix is ​​applied. Its logical structure is designed to allow the future state prediction token to focus on the hypothetical action token through an attention mechanism, while simultaneously masking the current real vehicle state token during attention calculation. In this way, the core neural network can directly deduce how the world state will evolve if the action is executed within the feature space without relying on an external physical simulator, generating a future state prediction sequence representing the hypothetical intervention, thereby assessing the potential risks of different strategies.

[0039] The core neural network also internalizes autoregressive world-state prediction logic for physical deduction in the continuous time domain. Unlike traditional discrete-time step predictions, this logic accepts values ​​containing arbitrary time intervals. The system uses positional encoding techniques to convert the temporal conditional feature tensor into a positional feature tensor. The mapping is performed as a high-dimensional vector and fused with the current state features. To ensure physical safety in extreme spaces, such as narrow alleys, the system introduces adaptive step size adjustment logic. This logic monitors in real time the spatial safety margin implied by the predicted state feature tensor, such as the Euclidean distance between the vehicle bounding box and the obstacle bounding box. When the safety margin falls below a preset safety threshold, such as 30 centimeters, the logic module automatically reduces the time interval of the next prediction step, for example, from 100ms to 10ms. The model performs high-frequency, fine-grained state inferences to improve the temporal resolution of collision detection. The final decision output is completed by the game-theoretic scoring logic in the output decoder. Multiple candidate trajectories are decoded from the features generated by the core neural network, and the dynamic utility value of each trajectory is calculated. The utility value is calculated using a weighted summation function, which includes efficiency gains, safety costs, and comfort costs. The key is that the weight coefficients of each indicator are not fixed values, but are controlled by the floating-point operation budget vector. As the remaining time indicated by the budget vector decreases, i.e., time becomes increasingly tight, the game decision scoring logic is based on a non-linear function, such as an exponential function, to increase the weight of efficiency gains in the total utility value. This means that under time-pressured conditions, the system will rationally become more aggressive, prioritizing the strategy with the shortest execution time; while when time is ample, it will revert to conservatism, prioritizing comfort.

[0040] In terms of system deployment architecture, this invention adopts a layered cloud-edge collaborative model. The core neural network is divided into a first computing model deployed at the vehicle edge and a second computing model deployed in the cloud. The first computing model at the edge is used for continuous operation, and its output Shannon entropy is monitored in real time. It is lightweight and has low latency. When the Shannon entropy output by the first computing model is higher than a preset uncertainty threshold, and the current floating-point operation count budget vector shows that the remaining time is sufficient to cover the network round-trip time (RTT), the edge sends the current feature sequence (Context) to the cloud as an inference request. The cloud uses the second computing model to perform deep inference and then returns the result. This second computing model has full parameters and high precision. Based on these characteristics, the edge device updates the final decision. If network conditions are not met or the time budget is insufficient, the system uses the output from the edge device to ensure a safety baseline. In addition, the output decoder also contains causal explanation generation logic to improve the system's transparency. This logic does not perform additional reasoning, but extracts the state data of the gate operators that trigger the computation graph collapse operation in the core neural network, i.e., why the thinking stops, and the weight distribution data in the game decision scoring logic, i.e. why this trajectory is chosen. This data is mapped and filled into a preset natural language template to generate an explanation text, such as the system prioritizing the acceleration detour strategy due to tight time budget and high certainty of the road conditions ahead with low entropy, providing logical basis for users or regulatory agencies.

[0041] Example 1: In a specific unstructured old residential area road scenario, a Robotaxi equipped with the spatiotemporal constraint end-to-end autonomous driving system of this invention is traveling at a speed of 20 km / h. This scenario is characterized by a narrow road (approximately 3.5 meters wide), with haphazardly parked stationary vehicles on both sides and no lane markings. When the system encounters a sudden situation where an electric bicycle suddenly crosses the obstructed area from the right, the hard decision-making deadline imposed by physical laws is instantly compressed to less than 150ms. At this time, traditional autonomous driving models based on fixed computation graphs often cannot complete all deep inference within such a short time window, resulting in delayed or incomplete control signals, leading to insufficient emergency braking distance or a significantly increased risk of collision. When the aforementioned sudden situation is triggered, the computing power management unit within the system of this invention, at the start of the inference cycle... (Assuming it's the trigger time of frame N), the system clock is read through the underlying hardware interface to calculate the remaining time until the physical collision threshold. Within just 120ms, the computing power management unit immediately queries the current computing power status table of the onboard NPU, rigidly mapping this 120ms physical time to the budget of the maximum number of floating-point operations allowed to be consumed in the current frame. ,For example FLOPs, this budget value, is rapidly encoded into instantaneous time budget tokens and injected into the input sequence of the core neural network.

[0042] As the feature data propagates forward layer by layer in the core neural network, the gate operator embedded at the output of the 6th layer Transformer monitors the Shannon entropy of the current layer's output feature tensor in real time. The probability has been reduced to 0.3 (indicating a high confidence level in detecting targets crossing the line), and the estimated power consumption required to execute subsequent deep networks (layers 7 to 24) has been calculated. This will exceed the current remaining budget. Based on the boundary constraints of computing power ( The decision logic is established instantly under the extremely tight computing power constraints. The topology reconstruction logic then generates a high-priority hardware interrupt instruction, directly triggering a computation graph collapse operation at layer 6. This physically cuts off the computational path to layer 7, causing the feature tensors of layer 6 to collapse. The bypass projection is sent to the output decoder, where the game-theoretic scoring logic receives a budget vector containing extremely low remaining time information. Its internal controlled weighting function undergoes a non-linear shift, elevating the weights of efficiency gains (converted here to fastest collision avoidance) to a dominant position, while temporarily suppressing the weights of comfort costs. The decoder quickly shifts from... The features decoded a short-term trajectory of a slight left turn and full braking. Although this trajectory sacrifices some ride comfort (due to significant longitudinal acceleration), it ensures that the complete perception-decision closed loop is completed within the physical limit of 120ms. Ultimately, the vehicle safely stops in front of the electric bicycle crossing the trajectory. The actual decision response time is controlled within 115ms, with no computational timeouts or signal loss. This addresses the collision risk under extreme conditions and verifies the deterministic response capability of the time entropy-based dynamic computation graph cascade collapse mechanism under hard real-time constraints.

[0043] Example 2: To verify the effectiveness of the spatiotemporal constrained end-to-end autonomous driving system based on the unified VLA model of this invention in real high-dynamic scenarios, and its deterministic response capability under hard real-time constraints based on the computational graph collapse mechanism of time entropy, this experiment was designed and implemented. This experiment was built on a High-Fidelity closed-loop autonomous driving simulation platform, integrating the Carla physics engine and the SUMO traffic flow simulator, which can accurately simulate vehicle dynamics, sensor noise, and complex multi-agent interaction behavior. In the simulation environment, a virtual Robotaxi equipped with the system of this invention was deployed. The sensor suite included a 6-channel 1080P camera covering a 360° field of view, a set of 32-line LiDAR, and an inertial measurement unit. To simulate the performance boundary of a real vehicle-mounted edge computing platform, the computing power of the operating environment was strictly limited using Docker container technology, setting the peak floating-point operation capability to 150 TFLOPS, and limiting the memory bandwidth. During the data acquisition process, to closely resemble engineering reality, an intensity of [insert intensity here] was deliberately superimposed on the input stream of the visual sensor. Gaussian white noise was introduced, and a 5% random drop rate was added to the lidar point cloud to simulate signal degradation caused by severe weather or sensor aging.

[0044] The test scenario was set as a typical unprotected left-turn intersection. In this scenario, the vehicle needs to find an opportunity to turn left in the gaps between oncoming straight traffic. The key challenge is the high density and speed of the oncoming traffic, as well as the risk of "ghost pedestrians" obstructing the view by large vehicles. The vehicle is required to make extremely precise decisions within a millisecond-level time window. The core monitoring parameters selected for the test are decision response time and collision rate. The decision response time depends not only on the inference speed of the model itself, but also on the dynamic constraints of the current computing power budget. The collision rate directly reflects the safety and effectiveness of the decision. The setting of these two parameters aims to balance the core engineering contradiction between efficiency and safety: too short a response time may lead to insufficient inference, thereby increasing the risk of collision, while too long an inference time may lead to accidents due to missing the fleeting time window. Based on the above settings, a total of 1,000 independent simulation tests were performed in this test. In each test, the seed for generating oncoming traffic was randomly changed to cover as many working conditions as possible. During the test, the computing power budget token value, the actual inference layer of the core neural network, the entropy output of the gating operator, and the final vehicle control command were recorded in real time for each frame.

[0045] During the experiment, the elastic scaling phenomenon of the computation graph was observed. When the vehicle was in a waiting area and the road conditions were relatively stable, the remaining time indicated by the computing power budget token was ample, and the entropy value output by the gating operator was high (indicating that further confirmation of environmental information was needed). At this time, the core neural network tended to execute the complete 24-layer inference, activating all expert networks to obtain the finest-grained understanding of the environment. However, when the vehicle detected a left-turn gap and began to accelerate, and suddenly detected a high-speed object approaching in the obstructed area, the remaining physical time decreased sharply. The computing power management unit quickly injected urgent budget constraints into the network. At this time, the data showed that after only the first 6 layers of inference, the gating operator determined that the remaining computing power budget was insufficient to support subsequent calculations (i.e., satisfying the requirements). This triggers computation graph collapse, and the network directly outputs features to the decoder at layer 6, instantly compressing the inference latency from the usual 120ms to 35ms. See Table 1, which lists the comparison data of key system performance indicators under three different typical operating conditions. The normal operating condition refers to a scenario with sparse oncoming traffic and no obstructions; the urgent operating condition refers to a scenario with dense oncoming traffic and the need to rush through; and the extreme operating condition refers to a scenario where an obstruction suddenly enters during the rushing through.

[0046] Table 1: Comparison of System Response and Calculation Graph Depth under Different Operating Conditions

[0047]

[0048] As can be clearly seen from the data in Table 1, as the urgency of the working conditions increases (computing power budget decreases), the system automatically and non-linearly reduces the inference depth and the number of activated experts. Although the inference depth is only 25% of that in normal working conditions, the response time is significantly compressed to 38ms, successfully outputting an effective emergency braking command before a physical collision occurs, keeping the collision rate at an extremely low level of 0.5%. This data trend strongly confirms the core mechanism of the topology reconstruction logic based on entropy computing power joint constraints in ensuring hard real-time safety: that is, by actively sacrificing the inference accuracy at non-critical moments in exchange for the response speed at critical moments. To further verify the technical effect of the present invention, a control group experiment was set up. The control group adopted a fixed-depth Transformer model (fixed to 24 layers) without the computing power management unit and topology reconstruction logic. In the same extreme working condition test set, the average response time of the control group was locked at about 115ms. Since it could not complete the inference at the moment of sudden danger (usually only a 50-80ms collision avoidance window), the control group failed to output the braking command in time due to calculation timeout in many cases, resulting in a collision rate as high as 12.4%.

[0049] Example 3: This example combines Figures 1 to 3 This section describes the spatiotemporal constrained end-to-end autonomous driving system and method based on the unified VLA model, such as... Figure 1 As shown, physical time and NPU status are obtained through a hardware clock and status module. The data flows to the computing power management unit to map physical time into a computing power budget vector. This floating-point operation budget vector is then input into the unified encoder. Simultaneously, the computing power management unit outputs sparsity constraint parameters to the core neural network. The unified encoder simultaneously receives multimodal input signals from vision, LiDAR, inertial navigation, and natural language and maps them into feature tensor sequences and time budget codes. The core neural network performs cascaded inference, hybrid expert module calculation, and causal masking operations on the feature sequences. During this process, the output features of each computing layer are transmitted to the embedded gating operator to calculate feature information entropy and estimated power consumption in real time. The Shannon entropy and estimated consumption data are then transmitted to the topology reconstruction logic. The topology reconstruction logic determines whether to trigger a computation graph collapse operation based on the joint constraints of entropy and computing power. If triggered, the subsequent path is cut off through the computation graph collapse operation and directly connected to the output decoder. At the same time, the computing power management unit sends a weight distribution adjustment signal to the output decoder. The output decoder combines the weight distribution to perform game-theoretic scoring and trajectory generation, and finally outputs vehicle control commands containing steering, acceleration, and explanatory text, thus forming a closed-loop control.

[0050] like Figure 2As shown, the horizontal axis represents the remaining time in milliseconds (ms), with a scale range from 10ms to 500ms. The vertical axis represents the weighting coefficient, ranging from 0 to 1.0. The graph contains three curves: the efficiency gain weight, indicated by solid dots, is at a high level of approximately 0.9 at the extremely urgent moment with 10ms remaining time, and decreases monotonically as the remaining time increases; the safety cost weight, indicated by dashed dots, is at a low level at 10ms, and shows an approximately linear upward trend as time increases; the comfort cost weight, indicated by dotted dots, remains at a low level of 0.05 within 100ms, only increasing when there is ample time. Figure 3 As shown, the system adopts a cloud-edge collaborative deployment architecture. The data source consists of multimodal sensing input LiDAR, vision, inertial navigation, and physical time constraint source hardware clock and computing power status. All data is fed into a unified encoder and then transmitted to the spatiotemporal computing power elastic hub. Based on the judgment logic, the remaining time is compared with the estimated consumption and the data is distributed: when the judgment is that time is tight or the certainty is high, the data flows to the real-time fast track at the left vehicle edge, where the first calculation model is used to perform elastic deep inference and dynamic collapse, and control commands are directly generated through game-theoretic decision decoding. When the judgment is that the uncertainty is high and the delay is allowed, the data flows to the high-precision deep track at the right cloud computing center, where the second calculation model with full parameters and high precision is used for inference and the high-precision results are sent back to correct the edge decision. At the same time, the cloud is connected to an offline calibration system for parameter initialization and simulation.

[0051] Example 4: To address the issue of collapse threshold in traditional computational graph collapse logic To address the uncertainties inherent in the design and to eliminate parameter issues arising from the unclear contribution of different feature dimensions to computational power consumption during multimodal fusion, this embodiment provides a dynamic calibration method for the collapse threshold based on an adaptive feedback mechanism. This transforms the previously experience-based static threshold into a dynamic variable driven by both real-time system status and historical inference feedback, thereby ensuring the stability and optimality of the computation graph collapse operation under different operating conditions. Based on closed-loop feedback control logic, this embodiment introduces a posterior evaluation operator at the end of each inference cycle to calculate the ratio between the actual inference accuracy loss and the actual computational power saved in the current cycle, i.e., the efficiency ratio. The system uses the feedback results from high-precision offline teacher models or large cloud models as a benchmark to calculate the difference between the current online model output and the benchmark results, quantifying it as accuracy loss. Simultaneously, the difference between the actual number of floating-point operations consumed in the current inference process and the number of operations required for the complete inference is calculated and quantified as the amount of computing power saved. efficiency ratio Defined as ,in To prevent small constants with a denominator of zero.

[0052] Based on the calculated efficiency ratio The system uses a preset proportional-integral-derivative (PID) controller to dynamically adjust the collapse threshold for the next inference cycle. If the efficiency ratio of the current cycle If the performance falls below the preset target threshold, it indicates that the current collapse strategy is either too aggressive, resulting in excessive accuracy loss, or too conservative, resulting in insufficient computational power saving. The PID controller will then output an adjustment amount. The threshold for the next cycle is adjusted using the following formula: Among them, adjustment amount The calculation satisfies: ,in, This represents the change in deviation between the current period and the previous period. For the current deviation, The system uses the inference cycle time interval as a reference point. Through this continuous feedback adjustment, the system can adaptively find the optimal collapse threshold under the current specific working conditions, eliminating the limitations of manually setting a fixed threshold. Addressing the issue of unclear weights for the impact of each modality on computational power consumption during multimodal feature fusion, this embodiment introduces a gradient-based feature importance weighting mechanism. During the model training phase, the system records the gradient values ​​of each modal input feature, such as visual tokens and LiDAR tokens, on the total computational power consumption loss function during backpropagation. The absolute value of this gradient directly reflects the sensitivity of that modality feature to computational resources during inference. The system normalizes these gradient values ​​to obtain the computational power sensitivity weight vector for each modality. During the online inference phase, the computing power management unit calculates and estimates the power consumption. Instead of simply accumulating the standard FLOPs of subsequent levels, the aforementioned computing power sensitivity weight vector is introduced. Weighted adjustments are made, and the specific calculation formula is adjusted as follows: ,in This represents the activation intensity of this modal feature in the current layer. To adjust the coefficients, this mechanism makes the actual computing power contribution of different modal characteristics explicit, so that the predicted computing power consumption is closer to the actual hardware execution.

[0053] Example 5: To eliminate the potential for unstable cold-start performance caused by initial data distribution offsets in traditional data-driven systems when facing different geographical regions, vehicle models, and sensor configurations, this example is implemented based on a hierarchical offline calibration platform. Before the formal deployment of the system, a high-fidelity digital twin simulation environment is constructed. Parametric modeling is performed based on real road topology data, historical traffic flow statistics, and dynamic parameters of the target deployment area. In this simulation environment, a set of standard test cases covering scenarios from open roads to extreme congestion is set. The system uses an evolutionary algorithm with the objective function of minimizing the overall inference latency and collision rate. In the simulation environment, the floating-point operation budget mapping table in the computing power management unit is iteratively searched multiple times. By statistically analyzing the matching relationship between inference depth and physical time window under different operating conditions, an initial computing power budget mapping benchmark table is generated.

[0054] Furthermore, unsupervised statistical calibration based on real-vehicle data was implemented for the collapse threshold and Shannon entropy benchmark of the gating operators in the core neural network. Data acquisition kits were installed on the target vehicle, and long-distance uninterrupted driving tests were conducted in a controlled closed test range. A shadow pattern dataset containing raw data from multimodal sensors and human driver input was collected. This dataset was used to perform inference tests on the pre-trained core neural network. The Shannon entropy distribution characteristics of the output feature tensors at each level were statistically analyzed. Based on statistical principles, the mean and variance of the Shannon entropy were calculated, and the collapse threshold was set as the benchmark. Initialize to Where k is the confidence coefficient, this process ensures that the threshold setting is directly anchored to the statistical law of characteristic uncertainty under a specific sensor configuration, rather than a general empirical value. Finally, the edge adaptation procedure based on hardware-in-the-loop (HIL) is executed, and the above-calibrated model is deployed to the target vehicle computing platform. The sensor simulator is connected to conduct full-load operation tests, and the temperature, power consumption and actual inference latency of the NPU are monitored in real time. Through automated stress test scripts, the concurrency of input data is gradually increased, and the computing power decay curve under different thermal states is measured. Based on the curve, the temperature compensation coefficient in the computing power management unit is fitted and corrected to generate the final hardware adaptation parameter file. Through the above three-stage calibration procedure from simulation optimization, real vehicle statistics to hardware adaptation, this embodiment provides a reproducible and standardized initialization path for the large-scale deployment of the system, effectively avoiding the risk of cold start due to improper parameter settings.

[0055] Example 6: To eliminate the potential parameter drift risks caused by batch differences in onboard computing platform hardware, sensor installation errors, and changes in ambient temperature during actual system deployment and large-scale mass production, and to address the uncertainty of system cold start performance in the absence of standardized initialization procedures, this example implements a pre-deployment calibration and initialization procedure based on hardware-in-the-loop (HIL) and environmental adaptability self-testing. Through a standardized offline engineering operation process, a set of optimal parameters adapted to the specific hardware characteristics and initial environmental conditions is generated for each autonomous driving system about to be put into use, thereby ensuring the system's performance in the physical world. Each startup is at the optimal operating point. This embodiment is implemented based on an automated calibration workstation that performs a computing power benchmark calibration step. In a standard temperature-controlled room, for example, with an ambient temperature constant at 25°C, the onboard computing platform to be calibrated is connected to a test bench. A set of preset benchmark computing power loads is run, including typical operator combinations ranging from lightweight matrix multiplication to high-concurrency convolution operations. The system collects real-time data on the actual floating-point operation rate, memory bandwidth utilization, and power consumption of the computing platform, and plots a computing power-temperature characteristic curve. Based on this curve, the system automatically calculates and solidifies the benchmark computing power mapping coefficients in the computing power management unit to ensure the computing power budget vector... The generation can accurately reflect the real physical computing power boundary of the current hardware at a specific temperature, rather than relying on theoretical peak data.

[0056] The sensor-model alignment step involves using a precision robotic arm to control a calibration target with a specific geometry, such as a checkerboard or corner reflector, within a standardized optical and radar calibration field. The target moves along a preset trajectory within the sensor's field of view. The system collects raw data from the multimodal sensors and inputs it into a unified encoder that has not been fine-tuned. By calculating the deviation between the spatial position of the beacon features in the encoder's output feature tensor and the actual physical position of the calibration target, the system uses a gradient descent algorithm to fine-tune the spatial mapping matrix parameters in the encoder. This process continues until the positional deviation converges to a preset accuracy threshold, such as a Euclidean distance of less than 1 cm. This step ensures that the spatial coordinate system generated by the unified encoder is precisely aligned with the absolute coordinate system of the physical world in the initial state, eliminating the impact of installation errors on the accuracy of micro-position indexing.

[0057] Finally, the environmental adaptation initialization step is executed. The system loads a set of simulation scenario data packages containing different weather conditions, such as sunny, rainy, and low light conditions, and traffic flow densities, such as open and congested conditions. The core neural network is then tested with rapid inference. In each scenario, the system statistically analyzes the average Shannon entropy output of the gating operators and the activation distribution of the hybrid experts. Based on the statistical results, the system automatically generates an initial state lookup table, defining the collapse threshold under different environmental perception features, such as light intensity and obstacle density. With the expert's recommended initial values ​​for routing weights, when the vehicle starts on an actual road, the system reads the current environmental perception features, queries the lookup table, and loads the corresponding initial parameters.

[0058] Example 7: In a dynamic sorting scenario of an industrial flexible production line, a six-axis collaborative robot equipped with the core architecture of the spatiotemporal constraint end-to-end intelligent control system of this invention is performing a high-speed material grasping task. The characteristics of this scenario are that the conveyor belt speed is 0.8m / s, the material spacing is uneven, and there are irregularly shaped workpieces with random orientations. When the system faces a sudden situation where a workpiece with an abnormal orientation is about to leave the effective grasping window, the hard decision deadline imposed by physical laws is compressed to less than 80ms. At this time, traditional visual grasping models based on fixed computation graphs often cannot complete all deep inference within such a short time window, resulting in a sharp increase in grasping failure rate. When the above-mentioned sudden situation is triggered, the computing power management unit in the system of this invention reads the system clock through the underlying hardware interface at the beginning of the inference cycle and calculates that the remaining time before the workpiece leaves the effective grasping area is only 60ms. The computing power management unit then queries the current computing power status table of the edge computing platform and rigidly maps this 60ms physical time to the maximum floating-point operation budget allowed to be consumed in the current frame.

[0059] As the feature data propagates forward layer by layer in the core neural network, the gating operator embedded in the output of the 4th layer Transformer monitors in real time that the Shannon entropy of the current layer's output feature tensor has decreased to 0.25. At the same time, it calculates that the estimated power consumption required to execute subsequent deep networks will exceed the current remaining budget. The topology reconstruction logic then generates a high-priority hardware interrupt instruction, directly triggering a computation graph collapse operation in the 4th layer, physically cutting off the computation path to the 5th layer, and bypassing the feature tensor of the 4th layer to the output decoder. Meanwhile, the game-theoretic scoring logic in the output decoder receives a budget vector containing extremely low remaining time information. Its internal controlled weight function undergoes a nonlinear jump, elevating the weight coefficient of efficiency gains to a dominant position. It selects the shortest-duration lateral gripping strategy from multiple candidate gripping trajectories. Although this strategy sacrifices some gripping stability, it ensures that the complete perception-decision closed loop is completed within the physical limit of 60ms. Finally, the robotic arm successfully completes the gripping action before the workpiece leaves the effective gripping area. The actual decision response time is controlled at 55ms, and the gripping success rate reaches 97.2%.

[0060] Example 8: In a complex urban environment, a quadcopter drone equipped with the core architecture of the spatiotemporal constraint end-to-end autonomous driving system of this invention is performing a last-mile delivery task at a speed of 12 m / s. The characteristics of this scenario are that there are buildings, power lines and unpredictable dynamic obstacles around the flight path. When the system faces a sudden situation where a bird suddenly crosses the flight path from the side, the hard decision deadline imposed by physical laws is compressed to less than 100 ms. At this time, the traditional drone obstacle avoidance model based on a fixed computation graph often cannot complete all deep inference within such a short time window, resulting in delayed avoidance actions or a sharp increase in collision risk. When the above-mentioned sudden situation is triggered, the computing power management unit in the system of this invention reads the system clock through the underlying hardware interface at the beginning of the inference cycle and calculates that the remaining time to the potential collision point is only 80 ms. The computing power management unit then queries the current computing power status table of the airborne edge computing platform (computing power of about 50 TFLOPS) and rigidly maps this 80 ms physical time to the maximum number of floating-point operations allowed to be consumed in the current frame.

[0061] The core neural network receives multimodal inputs from a forward-looking binocular camera, a downward-looking depth camera, and an inertial measurement unit. A unified encoder maps these inputs into a sequence of feature tensors of a unified dimension. As the feature data propagates forward layer by layer in the core neural network, the gating operator embedded in the output of the 5th layer Transformer monitors in real time that the Shannon entropy of the current layer's output feature tensor has decreased to 0.28. At the same time, it calculates that the estimated energy consumption required to execute subsequent deep networks will exceed the current remaining budget. Based on the joint constraint decision logic, the topology reconstruction logic then generates a high-priority hardware interrupt instruction, directly triggering the computation graph collapse operation at the 5th layer. Simultaneously, the game-theoretic scoring logic in the output decoder receives a budget vector containing extremely low remaining time information. Its internal controlled weight function undergoes a nonlinear jump, elevating the weight coefficient of efficiency gains to a dominant position. It selects a rapid ascent path from the candidate avoidance trajectories. Although this path increases energy consumption, it ensures that the drone escapes the collision risk zone in the shortest possible time. Ultimately, the drone safely passes 1.2 meters above the bird's crossing trajectory, with the actual decision response time controlled within 72ms, successfully avoiding a collision.

[0062] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A spatiotemporally constrained end-to-end autonomous driving system based on a unified VLA model, characterized in that, The system includes a unified encoder, a computing power management unit, a core neural network, and an output decoder. A unified encoder is used to map multimodal input signals into a sequence of feature tensors of a unified dimension. The computing power management unit is used to read the hardware clock at the start of the inference cycle and map the remaining physical time until the hard decision deadline to the floating-point operation count budget vector of the current cycle. The floating-point operation count budget vector is used to define the maximum number of matrix operations allowed to be executed within the current inference cycle. The core neural network is used to perform cascaded inference operations on feature tensor sequences, and contains multiple cascaded computational layers. The system also includes gating operators embedded in the output of each computational layer of the core neural network and topology reconstruction logic connected to the gating operators; the gating operators are used to calculate the Shannon entropy of the output feature tensor of the current computational layer, and calculate the estimated power consumption required to execute subsequent computational layers based on the pre-stored computational power consumption model of the core neural network; the topology reconstruction logic is used to trigger a computation graph collapse operation when the Shannon entropy is lower than a preset deterministic threshold or the value indicated by the floating-point operation count budget vector is less than the estimated power consumption. The computation graph collapse operation is used to sever the connection path between the current computation layer and the subsequent computation layers at the instruction level, and directly project the output feature tensor of the current computation layer to the output decoder, thereby establishing a positive correlation between the actual inference depth of the core neural network and the physical remaining time. The output decoder includes game-theoretic scoring logic, used to select the optimal execution strategy from multiple candidate trajectories. This logic calculates the dynamic utility value of each candidate trajectory, with the calculation weight controlled by the floating-point operation budget vector. As the remaining time indicated by the floating-point operation budget vector decreases, the game-theoretic scoring logic non-linearly increases the weight of efficiency gains in the dynamic utility value, thus prioritizing the shortest-duration trajectory strategy under time-sensitive conditions. The system adopts a layered deployment architecture with cloud-edge collaboration. The core neural network includes a first computational model deployed at the edge and a second computational model deployed in the cloud. The first computational model independently performs inference when the floating-point operation budget vector meets real-time requirements. When the Shannon entropy output by the first computational model exceeds a preset uncertainty threshold and the floating-point operation budget vector allows for network latency, the system sends an inference request to the second computational model and uses the output of the second computational model as the final decision-making basis.

2. The spatiotemporal constrained end-to-end autonomous driving system based on a unified VLA model according to claim 1, characterized in that, The core neural network also includes a hybrid expert module, which has a routing network and multiple expert networks configured in parallel. The routing network is used to receive the floating-point operation count budget vector as a sparsity constraint parameter. When the computing power margin indicated by the floating-point operation count budget vector is lower than a preset warning threshold, the routing network performs a computing power throttling operation. The computing power throttling operation includes downgrading the number of activated expert networks from the standard configuration to the minimum configuration to reduce the computing power consumption of a single layer.

3. The spatiotemporal constrained end-to-end autonomous driving system based on a unified VLA model according to claim 1, characterized in that, The unified encoder includes visual semantic beacon encoding logic, which is used to construct micro-location indexes for unstructured environments. The visual semantic beacon encoding logic extracts semantic feature vectors of target objects from natural language instructions and uses the semantic feature vectors as query vectors to perform cross-attention calculations with visual feature tensors. The system extracts visual features of high-response regions as beacon feature tensors based on attention weight distribution and injects the beacon feature tensors into the feature tensor sequence to establish a relative spatial coordinate system centered on the target object in the core neural network.

4. The spatiotemporal constrained end-to-end autonomous driving system based on a unified VLA model according to claim 1, characterized in that, The decision logic for the gated operator to perform computation graph collapse operation satisfies any one of the following separation conditions: deterministic constraints: Computing power boundary constraints: ;in, The Shannon entropy of the output feature tensor of the i-th layer is calculated. This is a preset deterministic threshold; Estimate the number of floating-point operations remaining. To estimate the power consumption required to execute subsequent computation layers; when any of the above conditions are met, the topology reconstruction logic generates an interrupt instruction to trigger the computation graph collapse operation, causing the core neural network to terminate forward propagation in the current computation layer.

5. The spatiotemporal constrained end-to-end autonomous driving system based on a unified VLA model according to claim 1, characterized in that, The core neural network is configured to perform counterfactual reasoning operations based on causal attention masks. The counterfactual reasoning operations involve inserting hypothetical action feature tensors into a sequence of feature tensors and applying a counterfactual mask matrix. The counterfactual mask matrix is ​​used to allow state feature tensors at future time steps to focus on hypothetical action feature tensors while masking the actual observation feature tensors at the current time step, thereby generating a sequence of future state predictions representing the hypothetical intervention within the feature space of the core neural network.

6. The spatiotemporal constrained end-to-end autonomous driving system based on a unified VLA model according to claim 1, characterized in that, The core neural network internalizes world-state autoregressive prediction logic, which is used to extrapolate the dynamic changes of the physical environment in the continuous time domain. The world-state autoregressive prediction logic receives a time-conditional feature tensor containing arbitrary time interval values ​​and outputs a predicted state feature tensor corresponding to that time interval. The system also includes adaptive step size adjustment logic, which is used to calculate the current safety margin based on the predicted state feature tensor, and when the safety margin is lower than a preset safety threshold, reduces the time interval value of the next prediction step to improve the accuracy of physical extrapolation in the time dimension.

7. The spatiotemporal constrained end-to-end autonomous driving system based on a unified VLA model according to claim 1, characterized in that, The output decoder includes a causal explanation generation logic, which generates natural language text representing the decision-making logic. The causal explanation generation logic extracts the state data of the gated operators that trigger the computation graph collapse operation in the core neural network, as well as the weight distribution data in the game adjudication scoring logic. The causal explanation generation logic maps the extracted data to a preset language template to generate an explanation text containing causal relationships. The explanation text is used to describe the logical basis for the system to select a specific strategy under specific time constraints.

8. The spatiotemporal constrained end-to-end autonomous driving system based on a unified VLA model according to claim 1, characterized in that, The unified encoder also includes time budgeting logic; the time budgeting logic decomposes the floating-point operation count budget vector into strategic-level budget components, tactical-level budget components, and instantaneous-level budget components; the strategic-level budget components are used to constrain the activation state of long-term planning experts in the core neural network, the tactical-level budget components are used to constrain the activation state of local interactive game experts, and the instantaneous-level budget components are used as a direct threshold benchmark for gating operators to perform computation graph collapse operations.

9. A spatiotemporal constrained end-to-end autonomous driving method based on a unified VLA model, applied to the system described in claim 1, characterized in that, The method includes the following steps: A unified encoder is used to map multimodal input signals into a sequence of feature tensors of a unified dimension; The computing power management unit reads the hardware clock at the start of the inference cycle and maps the remaining physical time until the hard decision deadline to a floating-point operation budget vector for the current cycle, so as to define the maximum number of matrix operations allowed to be executed within the current inference cycle. Cascaded inference operations are performed on feature tensor sequences using a core neural network. During the inference operation, the Shannon entropy of the output feature tensor of the current computing layer is calculated using the gate operator embedded in the output of each computing layer, and the estimated power consumption required to execute the subsequent computing layer is calculated in real time based on the pre-stored computing power consumption model of the core neural network. The computation graph collapse operation is triggered when the Shannon entropy is lower than a preset deterministic threshold or the value indicated by the floating-point operation count budget vector is less than the estimated power consumption. The computation graph collapse operation is performed to cut off the connection path between the current computation layer and the subsequent computation layers at the instruction level, and directly project the output feature tensor of the current computation layer to the output decoder. This establishes a positive correlation between the actual inference depth of the core neural network and the physical remaining time, ensuring that the inference latency is not higher than the physical remaining time.

Images