A streaming gated waypoint generation reduction method for intelligent driving visual language action models

Abstract

The application discloses a kind of streaming gate point generation reduction methods for intelligent driving visual language action model, belong to trajectory planning and visual language model inference acceleration field.This method first trains first gate and second gate;Then carry out point token autoregressive decoding;Extract the implicit state representation at the boundary of second point and third point, and use the first gate to distinguish the termination decision generated at the second point;Again, use the first three generated points to construct a geometric feature vector, and use the second gate to distinguish the termination decision generated at the third point;For samples that stop early, estimate the vehicle motion state based on the generated points, complete the remaining points, form a complete planning trajectory, and output to the autonomous driving planning control module.The application can reduce the number of point generation while ensuring planning safety and trajectory quality, reduce the inference delay of visual language action model, and improve real-time planning efficiency.

Description

Technical Field

[0001] This invention relates to the field of trajectory planning and visual language model inference acceleration for autonomous driving algorithms based on large models, and in particular to a streaming gating waypoint generation reduction method for intelligent driving visual language action models. Background Technology

[0002] In recent years, autonomous driving planning methods based on visual language models have been able to generate future trajectories simultaneously using multi-view images, text navigation instructions, and vehicle state information. These methods typically discretize the trajectory for the next few seconds into multiple waypoints and further encode them into text tokens, which are then generated token-by-token by a language model using an autoregressive approach. While this mechanism possesses strong visual semantic modeling capabilities, it suffers from significant serial decoding overhead in real-time planning scenarios.

[0003] Taking a planning model that represents the trajectory over the next three seconds as six waypoints as an example, generating all waypoints requires multiple steps of language model decoding. Since each decoding step depends on the output of the previous step, waypoint tokens cannot be generated in complete parallelism, making language model decoding a major component of the overall inference latency. For autonomous driving systems requiring high-frequency closed-loop decision-making, this latency reduces planning response speed and limits the deployment of large model planners on the vehicle or edge.

[0004] One intuitive approach is to determine scene difficulty based on the output entropy of waypoint tokens and stop generation early in low-entropy scenes. However, in visual language action models, output token entropy cannot reliably represent scene complexity. Although the model's hidden state contains readable scene complexity information, this information does not necessarily translate to the waypoint token probability distribution. Simply adjusting the output-side loss or entropy threshold is insufficient to obtain a stable and reliable early stopping signal.

[0005] Another approach is to predict the entire generation budget using a single classifier before generation begins. This approach has two drawbacks: First, the hidden states before generation do not contain any information about generated waypoints, resulting in insufficient classification accuracy for samples that need to distinguish between medium generation budgets and fully generated samples. Second, the ability of a single multi-classifier to distinguish intermediate levels (such as stopping at the third waypoint) is significantly lower than that of the extreme levels, leading to a large number of samples being conservatively routed to full generation, resulting in limited time savings.

[0006] Furthermore, kinematic simplicity does not equate to safe driving scenarios. For example, scenarios such as waiting at low-speed intersections or yielding at turns may geometrically approximate a straight line or maintain low speed, but the gaps between surrounding objects are small, and the interaction risk is high. If only the degree of trajectory nonlinearity or trajectory error is used as the routing label, the gating system will incorrectly assign such scenarios to aggressively generated paths, causing the collision rate to decline. Experimental analysis shows that when the gating system is trained solely on kinematic complexity, the vast majority of newly added collision samples exhibit characteristics of turning intent, low-speed driving, and proximity to obstacles, and their scenario complexity is misjudged as low. Therefore, it is necessary to propose a streaming gating method that can reduce the number of waypoint generation attempts while explicitly considering collision and safety clearance constraints. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a streaming gating waypoint generation reduction method for intelligent driving visual language action models. The aim is to reduce the number of waypoint generation while explicitly considering collision and safety clearance constraints, thereby reducing the inference latency of the visual language action model and improving real-time planning efficiency.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0009] A streaming gated waypoint generation reduction method for visual language models includes the following steps:

[0010] S1: Construct a training sample set. For each sample in the training sample set, enumerate three candidate routes: stopping at the second waypoint, stopping at the third waypoint, and fully generated. Score each candidate route based on the security-aware scoring function and generate a security-aware route label. Use the security-aware route label to train the first gate and the second gate respectively.

[0011] S2: Acquire multi-view images of the vehicle's surroundings, navigation text commands, and historical vehicle status, and input them into the visual language action model to perform autoregressive decoding of waypoint tokens for a future preset time window;

[0012] S3: When the waypoint token is autoregressively decoded to the boundary position between the second waypoint and the third waypoint, the hidden state representation of the visual language action model at the boundary position token is extracted;

[0013] S4: Input the hidden state representation into the first gate. When the first gate determines that it stops at the second waypoint, terminate the decoding of subsequent waypoint tokens and execute S6. When the first gate determines that it continues to generate, continue decoding to the boundary position between the third waypoint and the fourth waypoint and execute step S5.

[0014] S5: Construct a geometric feature vector based on the first three waypoints and input it into the second gate; when the second gate decides to stop at the third waypoint, terminate the subsequent decoding and execute step S6; when the second gate decides to continue generating all waypoints, complete the generation of all waypoints and execute step S7.

[0015] S6. For samples that stop early, the vehicle speed is estimated based on the adjacent displacements between the generated waypoints, and the vehicle acceleration is estimated by the difference between adjacent displacements. The remaining waypoints are completed using the kinematic extrapolation method to form extrapolated waypoints.

[0016] S7, combine the generated waypoints and the extrapolated complete waypoints into a complete planned trajectory, and output it to the autonomous driving planning and control module.

[0017] Furthermore, in S1, both the first gating device and the second gating device are trained independently under the condition that the parameters of the visual language action model are frozen; the parameter scale of the first gating device and the second gating device is smaller than the parameter scale of the visual language action model, the time consumed by a single gating decision is less than the time consumed by a single-step waypoint token decoding, and the input and output protocol of the main model is not modified.

[0018] Furthermore, the security perception scoring function in S1 is:

[0019]

[0020] Where r represents a candidate route, L2_norm(r) is the normalized trajectory error, col(r) is the collision indication term, clr_penalty(r) is the safety gap penalty term, lat_cost(r) is the normalized delay cost term, and lambda, mu, and eta are weight parameters; the candidate routes include stopping at the second waypoint, stopping at the third waypoint, and complete generation.

[0021] Furthermore, the safety clearance penalty term is expressed as:

[0022]

[0023] in To establish a safe distance, The distance between the predicted trajectory obtained using candidate route r and surrounding targets is the minimum distance; the delay cost term is the linearly normalized value of the candidate route relative to the decoding delay.

[0024] Furthermore, when generating security-aware route labels in S1, a security-priority hard constraint is applied. Specifically, when a candidate route causes a collision and there is a candidate route that does not cause a collision, and the trajectory error increment of the collision-free candidate route relative to the collision candidate route does not exceed a preset tolerance threshold delta, the candidate route with the lowest score is forcibly selected from the collision-free candidate routes as the security-aware route label, and the candidate route that causes a collision is ignored.

[0025] Furthermore, the security-aware route label in S1 is generated using an optimal route priority strategy. Specifically, the route with the smallest trajectory error among all candidate routes is used as the base label, and only for samples where using the base label would cause a collision, the route is reselected according to the security priority hard constraint.

[0026] Furthermore, the hidden state representation in S3 is obtained by registering a forward hook on the normalization layer of the visual language action model, and is the hidden vector at the boundary position token. The dimension of the hidden state representation is consistent with the dimension of the hidden layer of the language model.

[0027] Furthermore, the first gate in S4 is a multilayer perceptron, including an input layer, a nonlinear activation layer, a random deactivation layer, and a binary classification output layer; when the stopping probability output by the first gate is greater than or equal to a preset confidence threshold, the decision stops at the second waypoint; otherwise, the decision continues to be generated.

[0028] Furthermore, the geometric feature vector in S5 includes the lateral and longitudinal coordinates of the first three waypoints, the displacement from the first waypoint to the second waypoint, the displacement from the second waypoint to the third waypoint, the lateral drift, and the acceleration measure; wherein the lateral drift is the absolute value of the difference between the lateral coordinates of the third waypoint and the lateral coordinates of the first waypoint, and the acceleration measure is the difference between two adjacent displacements.

[0029] Furthermore, S1-S7 are applied to a visual language action model that represents the trajectory of the next three seconds as six waypoints. The candidate generated paths include generating two waypoints and then extrapolating and completing, generating three waypoints and then extrapolating and completing, and generating all six trajectory points.

[0030] The present invention has the following beneficial effects:

[0031] 1) This invention embeds waypoint generation reduction decision into the autoregressive decoding process. The gating is triggered in a streaming manner at the waypoint boundary token, directly reading the hidden state of the main model and the generated waypoint information, instead of relying on the waypoint token output entropy, thus bypassing the problem of unstable correspondence between output entropy and scene complexity. At the same time, compared with the one-time decision before generation, the second gating can utilize the explicit geometric information of the generated waypoints to improve the routing quality of intermediate tiers.

[0032] 2) This invention generates route labels through a safety perception scoring function and safety-priority hard constraints, incorporating collision, target gap, and trajectory error into the same training objective, thus avoiding safety regression caused by early stopping decisions based solely on kinematic complexity. For scenarios with simple kinematics but high risks, such as low-speed intersections and yielding at turns, the gating system can learn more conservative route selections, ensuring that the collision index after acceleration is on par with the fully generated baseline.

[0033] 3) This invention adopts an optimal route-first label generation strategy, applying safe rerouting only to collision samples to avoid overall label distribution shift, thus ensuring the training accuracy of the gating system and the learnability of routing decisions.

[0034] 4) This invention utilizes already generated waypoints for kinematic extrapolation and completion after early stopping, avoiding the need to generate all waypoint tokens, thereby reducing the number of serial decoding steps. Both gating systems are lightweight multilayer perceptrons, with single-decision latency significantly lower than single-step token decoding latency. This method does not modify the main model parameters or input / output protocols, can be integrated into existing visual language action models as an external inference acceleration module, and can be extended to any planning model that generates trajectories using discrete token autoregression.

[0035] 5) The candidate routes of this invention cover different computational budgets and can adaptively allocate decoding resources according to the difficulty of the scene, so that simple scenes generate fewer routes and complex scenes generate complete routes, achieving a balance between planning quality and real-time performance. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the reasoning process for a streaming gated waypoint generation reduction method for visual language models according to the present invention.

[0037] Figure 2 This is a schematic diagram of the gating system training process for a streaming gating waypoint generation and reduction method for visual language models according to the present invention.

[0038] Figure 3 A schematic diagram of the connection structure between the cascaded dual gating system and the visual language planning model;

[0039] Figure 4 A schematic diagram illustrating the process of generating security-aware route labels;

[0040] Figure 5 To generate a complete comparison diagram with streaming gating in terms of trajectory error, collision rate, and inference latency. Detailed Implementation

[0041] The present invention will now be described in further detail with reference to the accompanying drawings and specific preferred embodiments.

[0042] In the description of this invention, it should be understood that the terms "left side," "right side," "upper part," "lower part," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. "First," "second," etc., do not indicate the importance of the components, and therefore should not be construed as a limitation of this invention. The specific dimensions used in this embodiment are only for illustrating the technical solution and do not limit the scope of protection of this invention.

[0043] Example

[0044] This embodiment presents a streaming gated waypoint generation reduction method for intelligent driving visual language action models. (See also...) Figure 1 , 2 As shown in 3 and 4, the steps include:

[0045] S1. Generate safety-aware training labels and train the gating system. Construct a training sample set. For each sample in the training sample set, generate trajectories corresponding to three candidate routes: stopping at the second waypoint, stopping at the third waypoint, and generating a complete route. Utilize a safety-aware scoring function to comprehensively evaluate the normalized trajectory error, collision risk, minimum safety gap, and latency cost of each candidate route to determine the optimal candidate route. Safety-aware route labels are generated using an optimal route-first strategy. Initially, the candidate route with the smallest trajectory error is used as the label. For samples where using the basic label would lead to a collision (i.e., when the candidate route has a collision risk and there is a collision-free candidate route whose trajectory error increment does not exceed the error tolerance threshold delta), the label is corrected to the route with the best score among the collision-free candidate routes. In other words, when a candidate route causes a collision and there is a candidate route that does not cause a collision, and the trajectory error increment of the collision-free candidate route relative to the collision candidate route does not exceed the preset tolerance threshold delta, the candidate route with the lowest score among the collision-free candidate routes is forcibly selected as the safety-aware route label, ignoring the candidate route that causes the collision. Based on the final determined routing labels, cascaded gating training labels are constructed. The first gating unit learns the decision between "stopping at the second waypoint" and "continuing to generate," while the second gating unit learns the decision between "stopping at the third waypoint" and "complete generation." The first and second gating units are trained separately under the condition that the visual language action model parameters are frozen. The parameter sizes of the first and second gating units are smaller than those of the visual language action model parameters. The time consumed for a single gating decision is less than the time consumed for single-step waypoint token decoding, and the input / output protocol of the main model is not modified.

[0046] The security perception scoring function is as follows:

[0047]

[0048] Where r represents a candidate route, L2_norm(r) is the normalized trajectory error, col(r) is the collision indication term, clr_penalty(r) is the safety gap penalty term, lat_cost(r) is the normalized delay cost term, and lambda, mu, and eta are weight parameters; the candidate routes include stopping at the second waypoint, stopping at the third waypoint, and complete generation.

[0049] In this embodiment, lambda is 5, mu is 0.5, eta is 0.05, the safe distance d_safe is 5.0 meters, the error tolerance threshold delta is 0.25 meters, L2_norm is normalized to the median of the three-second trajectory error of the validation set, and lat_cost is linearly normalized to a shortest route delay of 0 and a complete generation delay of 1.

[0050] The selection of the above weight parameters is based on the following criteria: lambda makes the collision cost significantly exceed the normalized trajectory error difference between any candidate routes, forming an effective soft constraint; mu makes the safety gap penalty assist without exceeding the collision term; eta only forms an efficiency preference when safety and trajectory error are the same; delta is higher than the maximum trajectory error difference between actual candidate routes, ensuring that rerouting at the cost of centimeter-level error is always executed.

[0051] The safety clearance penalty is expressed as follows:

[0052]

[0053] in To establish a safe distance, The minimum distance between the predicted trajectory obtained using candidate route r and surrounding targets; the delay cost term is the linearly normalized value of the candidate route relative to the decoding delay.

[0054] S2, Constructing Visual Language Planning Input. Acquire multi-view images of the vehicle's surroundings, navigation text commands, historical vehicle status, and vehicle coordinate system information. After processing by a visual encoder and a text encoder, these are used to form the input sequence for the visual language action model. After the main model completes pre-filling, it begins autoregressive decoding of waypoint tokens for a future preset time window.

[0055] S3, Capture the hidden state at the waypoint boundary. Register a forward hook in the normalization layer of the main model to listen for boundary token events during the decoding process. When decoding reaches the boundary token between the second and third waypoints, extract the hidden state representation of the visual language action model at the boundary token, which is the hidden vector at the boundary token. The dimension of the hidden state representation is consistent with the dimension of the hidden layer of the language model. In this embodiment, the hook captures the 896-dimensional hidden state vector at this position for use by the first gating controller.

[0056] S4, First Gating Decision. The captured 896-dimensional hidden state vector is input into the first gating. The first gating is a multilayer perceptron (MLP) structure (896→128→2), including an input layer, a nonlinear activation layer, a random deactivation layer, and a binary classification output layer; it has GELU activation and random deactivation layers, and the latency of a single forward computation is on the order of 0.1 milliseconds. The output is the probability of stopping at the second waypoint. When this probability is greater than or equal to the threshold of 0.85, subsequent decoding is terminated and proceed to S6; otherwise, generation continues and S5 is executed to continue generation.

[0057] S5, the second gating controller performs streaming decisions. When decoding reaches the boundary token between the third and fourth waypoints, ten-dimensional geometric features are calculated in real time from the coordinates of the first three waypoints: [x1, x2, x3, y1, y2, y3, delta_12, delta_23, lateral_trend, accel], where delta_12 is the displacement from the first waypoint to the second waypoint, delta_23 is the displacement from the second waypoint to the third waypoint, lateral_trend = |x3| − |x1| represents lateral drift, and accel = delta_23 − delta_12 represents acceleration / deceleration trends. The second gating controller is an MLP (10→128→2) structure, and the decision stops at the third waypoint (proceeding to S6) or all waypoints are fully generated.

[0058] S6, Complete the remaining waypoints. For samples that stop early, the system estimates the velocity and acceleration based on the generated waypoints, and uses a constant acceleration extrapolation method to complete the remaining waypoints, finally obtaining the complete future trajectory. The extrapolation formula is:

[0059] Where p_n is the last generated waypoint. When only two waypoints are obtained, the acceleration term is set to zero or the acceleration is estimated using historical vehicle states. It should be noted that extrapolation must be applied to the actual generated waypoint sequence: if the decoding framework fills in the waypoints by repeating the last waypoint when there are insufficient waypoints, it should first be truncated according to the actual number of generated waypoints before extrapolation is performed; otherwise, the extrapolation degenerates into constant position filling.

[0060] S7 combines the generated waypoints and extrapolated complete waypoints into a complete planned trajectory and outputs it to the autonomous driving planning and control module.

[0061] The following example, applying S1-S7 to a future three-second six-waypoint visual language action model, illustrates the technical solution of this invention. This model encodes six future waypoints as tokens and generates them progressively. The measured average end-to-end latency of the complete path generation is approximately 1208 milliseconds (this value varies under different hardware and batch processing conditions), with serial decoding of the language model accounting for the majority of the latency. The number of decoded tokens corresponding to stopping at the second waypoint and the third waypoint is approximately 22% and 58% of the total generated tokens, respectively.

[0062] In this application example, the first gating unit reads the 896-dimensional hidden state at the boundary of the second waypoint, and the second gating unit reads the 16-dimensional geometric features constructed from the first three waypoints and their kinematic dynamics. After training with safety-aware labels, on 6019 validation samples, approximately 31.1% of the samples were routed to stop at the second waypoint, approximately 13.0% of the samples were routed to stop at the third waypoint, and approximately 55.9% of the samples remained fully generated.

[0063] In this application example, the average end-to-end latency using the method of this invention is approximately 1000 milliseconds, a reduction of approximately 29.6% compared to the 1420 milliseconds of the fully generated path. The number of collision samples in three seconds is consistent with the fully generated baseline (both are 43 / 6019, approximately 0.714%), with no net new collisions; the average trajectory error in three seconds is essentially the same as the fully generated baseline (changing from 0.683 meters to 0.676 meters). See also Figure 5 The method can maintain planning accuracy and collision safety while reducing the cost of autoregressive generation.

[0064] In contrast, if safety-aware labels are not used and all samples are directly routed to stop at the third waypoint, the number of collision samples will increase from the baseline of 43 (approximately 0.714%) to 53 (an increase of 10). If all samples are further routed to stop at the second waypoint, the number of collision samples will increase to 57 (an increase of 14), and the new collision samples will be concentrated in turning, low-speed, and close-range obstacle scenarios. After adopting the safety-aware labels and hard-constraint rerouting of this invention, the final cascaded routing scheme maintains the early stopping ratio while adding zero collisions, verifying the necessity of the safety-aware label generation mechanism.

[0065] To verify the effectiveness of this invention under closed-loop deployment conditions, a log-playback-based closed-loop evaluation was conducted on 150 driving scenarios and a total of 5869 decision steps. This means that each step involved advancing the vehicle's pose using the predicted trajectory from the previous step before planning the next step. The results show that after adopting the streaming gating method of this invention, the early stopping rate reached 44.1% (approximately 31.1% of decision steps stopped at the second waypoint and 13.0% at the third waypoint); the open-loop trajectory error (average L2 of six future waypoints) decreased from 0.584 meters to 0.578 meters, remaining essentially unchanged; the chain-like cumulative drift of the vehicle's pose was completely consistent with the fully generated baseline; the number of three-second collision steps increased from 37 steps (out of 5869 steps) to 38 steps, and the number of collision scenarios increased from 19 to 21 (out of 150 scenarios), with limited increments. These results demonstrate that the streaming gating method can maintain low trajectory error accumulation and limited collision rate increments under closed-loop log-playback conditions.

[0066] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the protection scope of the present invention.

Claims

1. A streaming gating waypoint generation reduction method for intelligent driving visual language action models, characterized in that: Includes the following steps: S1: Construct a training sample set. For each sample in the training sample set, enumerate three candidate routes: stopping at the second waypoint, stopping at the third waypoint, and fully generated. Score each candidate route based on the security-aware scoring function and generate a security-aware route label. Use the security-aware route label to train the first gate and the second gate respectively. S2: Acquire multi-view images of the vehicle's surroundings, navigation text commands, and historical vehicle status, and input them into the visual language model to perform autoregressive decoding of waypoint tokens for a future preset time window; S3: When the waypoint token is autoregressively decoded to the boundary position between the second waypoint and the third waypoint, the hidden state representation of the visual language model at the boundary position token is extracted; S4: Input the hidden state representation into the first gate. When the first gate determines that it stops at the second waypoint, terminate the decoding of subsequent waypoint tokens and execute S6. When the first gate determines that it continues to generate, continue decoding to the boundary position between the third waypoint and the fourth waypoint and execute step S5. S5: Construct a geometric feature vector based on the first three waypoints and input it into the second gate; when the second gate decides to stop at the third waypoint, terminate the subsequent decoding and execute step S6; when the second gate decides to continue generating all waypoints, complete the generation of all waypoints and execute step S7. S6. For samples that stop early, the vehicle speed is estimated based on the adjacent displacements between the generated waypoints, and the vehicle acceleration is estimated by the difference between adjacent displacements. The remaining waypoints are completed using the kinematic extrapolation method to form extrapolated waypoints. S7, combine the generated waypoints and the extrapolated complete waypoints into a complete planned trajectory, and output it to the autonomous driving planning and control module.

2. The method for reducing the generation of streaming gated waypoints for intelligent driving visual language action models according to claim 1, characterized in that: In S1, both the first gating device and the second gating device are trained independently under the condition that the parameters of the visual language action model are frozen; the parameter scale of the first gating device and the second gating device is smaller than the parameter scale of the visual language action model, the time consumed by a single gating decision is less than the time consumed by a single-step waypoint token decoding, and the input and output protocol of the main model is not modified.

3. The streaming gating waypoint generation reduction method for intelligent driving visual language action models according to claim 1, characterized in that: The security perception scoring function in S1 is: ； Where r represents the candidate route, L2_norm(r) is the normalized trajectory error, col(r) is the collision indication term, clr_penalty(r) is the safety gap penalty term, lat_cost(r) is the normalized delay cost term, and lambda, mu, and eta are weight parameters. Candidate routes include routes that stop at the second waypoint, routes that stop at the third waypoint, and routes that are generated completely.

4. The streaming gating waypoint generation reduction method for intelligent driving visual language action models according to claim 3, characterized in that: The safety clearance penalty term is expressed as follows: ； in To establish a safe distance, The distance between the predicted trajectory obtained using candidate route r and surrounding targets is the minimum distance; the delay cost term is the linearly normalized value of the candidate route relative to the decoding delay.

5. The method for reducing the generation of streaming gated waypoints for intelligent driving visual language action models according to claim 1, characterized in that: When generating a security-aware route label in S1, a security-priority hard constraint is applied. Specifically, when a candidate route causes a collision and there is a candidate route that does not cause a collision, and the trajectory error increment of the non-collision candidate route relative to the collision candidate route does not exceed a preset tolerance threshold delta, the candidate route with the smallest score is forcibly selected from the non-collision candidate routes as the security-aware route label, and the candidate route that causes a collision is ignored.

6. The streaming gating waypoint generation reduction method for intelligent driving visual language action models according to claim 5, characterized in that: The security-aware route label in S1 is generated using the optimal route priority strategy. Specifically, the route with the smallest trajectory error among all candidate routes is used as the base label. Only for samples where using the base label would cause a collision, the route is reselected according to the security priority hard constraint.

7. The method for reducing the generation of streaming gated waypoints for intelligent driving visual language action models according to claim 1, characterized in that: The hidden state representation in S3 is obtained by registering a forward hook on the normalization layer of the visual language action model. It is the hidden vector at the boundary position token, and the dimension of the hidden state representation is consistent with the dimension of the hidden layer of the language model.

8. The method for reducing the generation of streaming gated waypoints for intelligent driving visual language action models according to claim 1, characterized in that: The first gate in S4 is a multilayer perceptron, including an input layer, a nonlinear activation layer, a random deactivation layer, and a binary classification output layer; when the stopping probability output by the first gate is greater than or equal to a preset confidence threshold, the decision stops at the second waypoint; otherwise, the decision continues to be generated.

9. The streaming gating waypoint generation reduction method for intelligent driving visual language action models according to claim 1, characterized in that: The geometric feature vector in S5 includes the lateral and longitudinal coordinates of the first three waypoints, the displacement from the first waypoint to the second waypoint, the displacement from the second waypoint to the third waypoint, the lateral drift, and the acceleration measure; wherein the lateral drift is the absolute value of the difference between the lateral coordinates of the third waypoint and the lateral coordinates of the first waypoint, and the acceleration measure is the difference between two adjacent displacements.

10. The method for reducing the generation of streaming gated waypoints for intelligent driving visual language action models according to claim 1, characterized in that: S1-S7 are applied to a visual language action model that represents the trajectory of the next three seconds as six waypoints. The candidate generated paths include generating two waypoints and then extrapolating and completing, generating three waypoints and then extrapolating and completing, and generating all six trajectory points.

Images