Off-road automatic driving decision method based on large model and conditional flow matching track

Abstract

The application provides an off-road automatic driving decision method based on a large model and a conditional flow matching track, comprising the following steps: splicing visual tokens corresponding to a current frame and a plurality of historical frames of a vehicle front-view camera, performing linear projection mapping on the spliced visual tokens to obtain hidden, performing scaling normalization on the hidden, fusing tokens of off-road semantic instruction texts, and obtaining a multimodal embedding sequence; inputting the multimodal embedding sequence into a mixed expert network, processing the multimodal embedding sequence through a self-attention layer, and respectively inputting the multimodal embedding sequence into a visual language expert, a proprioceptive expert and a motion expert; the visual language expert is used for understanding semantics and spatial environment, and visual features are obtained; the proprioceptive expert is used for aligning physical states, and proprioceptive features are obtained; the motion expert is used for calculating a track trend of a vehicle at a next moment, and motion features are obtained; and the visual features, the proprioceptive features and the motion features are aggregated and input into a motion decoder to obtain a target off-road driving track of the vehicle in a next time period, so that the target off-road driving track is executed by a controller of the vehicle.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving artificial intelligence technology, and in particular to an autonomous driving decision-making method, device, electronic device, computer-readable storage medium, and computer program product for off-road unstructured environments based on multimodal large model and conditional flow matching trajectory generation, for realizing end-to-end feedforward trajectory planning and action decision-making in unstructured off-road environments.

[0002] This invention belongs to the interdisciplinary field of embodied intelligence and autonomous driving artificial intelligence, specifically involving multimodal large language model (MLLM), vision-language-action (VLA) model, mixed-expert network (MoE) and off-road autonomous navigation system, specifically referring to the technical device and method for realizing end-to-end feedforward trajectory planning and action decision-making in unstructured off-road environment. Background Technology

[0003] Existing vision-language-action (VLA) models, as a new generation paradigm of embodied intelligence, combine the capabilities of large-scale pre-trained visual encoders with language models to achieve interpretable decision-making through language anchoring. VLA models, represented by PaliGemma, OpenEMMA, and DriveGPT4, have made significant progress in the field of autonomous driving on structured urban roads. Their core assumptions are that road geometry is well-defined, traffic rules are clearly followable, and perceptual inputs can be effectively simplified into a two-dimensional bird's-eye view representation. Multi-task cascaded architectures such as UniAD achieved state-of-the-art performance in urban environments through the cascaded integration of perception, prediction, and planning. However, they rely on high-precision maps and multi-sensor fusion, leading to a break in the perception link in off-road scenarios without high-precision map support. Off-road navigation methods, represented by the RUGD and RELLIS-3D datasets, mainly rely on the semantic segmentation results of two-dimensional images as the criteria for terrain navigability. They combine classical control theory to achieve line-following driving, but they have fundamental defects such as the semantic segmentation labels being unable to exhaustively list the categories of off-road obstacles, the lack of explicit modeling of the three-dimensional geometry of the terrain, and the inability to handle the distribution of long-tail obstacles.

[0004] A comprehensive analysis of the existing technologies reveals three fundamental technical barriers they face in unstructured off-road environments. First, the rigid dynamics assumption fails: the action output space of existing VLA urban driving models is typically a discrete lateral offset or a finite set of driving behaviors. This design assumption fails in off-road environments—the three-dimensional undulations of the terrain surface cause the vehicle's equivalent kinematic parameters to change dynamically, and existing models cannot embed terrain-sensitive dynamic kinematic constraints. Second, there is perceptual aliasing and an open vocabulary gap: the perceptual input in off-road scenarios exhibits highly aliased characteristics—the texture features of dense vegetation are highly similar to those of distant obstacles, and the surface features of gravel and rocks are almost identical to those of traversable sand; furthermore, there are numerous long-tailed obstacles that cannot be exhaustively enumerated with a single label, fundamentally challenging the perception paradigm based on classification labels. Third, decision-making is unstable under long-tail distribution: Off-road scenarios represent a typical long-tail distribution - the vast majority of samples appear on relatively flat "normal" terrain, while the marginal cases that truly determine the success or failure of the task occur very infrequently but are extremely risky. The "average terrain" prior formed by the existing model training on urban data leads to blurred decision boundaries and excessive variance in action output in off-road scenarios. Summary of the Invention

[0005] This invention proposes an end-to-end feedforward VLA architecture (off-road VLA) specifically designed for unstructured off-road environments. This architecture captures apparent changes between the current frame and historical frames through a dual-temporal forward-view image contrast encoder, explicitly modeling terrain ruggedness and vegetation dynamics; it inherits the open-vocabulary semantic understanding capability of the SigLIP visual tower through a PaliGemma pre-trained multimodal generalization encoder, generalizing to unseen obstacles without exhaustive label enumeration; it organizes visual language experts, proprioceptive experts, and action experts in parallel through a heterogeneous multimodal hybrid expert network, sharing an attention mechanism but separating the feedforward network; it explicitly models the dependency structure between the three modal tokens through a block-based causal attention mask matrix; it redefines off-road trajectory prediction as a conditional generation process from noise distribution to target distribution by using conditional flow matching action experts to model action distribution with continuous probability flow; it achieves dynamic coupling between diffusion time steps and network parameters through a time-step adaptive normalization modulator; it significantly reduces inference computation through an inactive expert key-value cache reuse mechanism; and it avoids contamination of normalization statistics by extreme terrain state values ​​through a 1 / 99 percentile pruning strategy using an off-road long-tailed split-bar normalizer; and it utilizes a Beta... Sampling at skewed time steps gives higher sampling probability at smaller time steps to improve accuracy in off-road boundary scenarios; a low-memory fine-tuning training strategy using heterogeneous parameter grouping supports optional LoRA and quantization to adapt to limited on-vehicle computing resources.

[0006] To address the shortcomings of existing technologies, such as Figure 7As shown, this invention proposes an off-road autonomous driving decision-making method based on large model and conditional flow matching trajectory, including:

[0007] Step 1: Obtain the current frame and multiple historical frames from the vehicle's forward-facing camera, and simultaneously acquire the multi-dimensional proprioceptive state sequence of the vehicle at multiple historical time steps.

[0008] Step 2: The current frame and the historical frame are fed into the visual encoder to obtain the visual token corresponding to each frame. The visual tokens are then concatenated, linearly projected and mapped to the hidden area, and then scaled and normalized before being fused with the token of the off-road semantic command text to obtain a multimodal embedding sequence.

[0009] Step 3: Construct a multimodal hybrid expert network that includes visual language experts, proprioceptive experts and action experts. Each expert processes in parallel and shares the query-key-value projection from the attention layer. Proprioceptive experts and action experts share the feedforward network weights.

[0010] Step 4: Input the multimodal embedding sequence into the hybrid expert network. After processing by the self-attention layer, it is input into the visual language expert, proprioception expert, and action expert respectively. The visual language expert is used to understand semantics and spatial environment to obtain visual features. The proprioception expert is used to map and embed the continuous physical values ​​of the vehicle's underlying layers (vehicle speed, pose, etc.) into the unified semantic space of the large model to align the physical state and obtain ontological features. If they are not aligned, the system will face the fundamental problem that sensor data cannot be correctly understood by the large model. The action expert is used to calculate the trajectory trend of the vehicle in the next moment to obtain action features. After aggregating the visual features, ontological features, and action features, the sequence is input into the action decoder to obtain the target off-road driving trajectory of the vehicle in the next time period, which serves as the decision result for off-road autonomous driving and is executed by the vehicle's controller.

[0011] The aforementioned off-road autonomous driving decision-making method based on large model and conditional flow matching trajectory includes a multidimensional proprioceptive state sequence for each historical time step containing velocity, three-axis acceleration, three-axis angular velocity, heading angle, and coordinates of the distant target point; the visual encoder is a SigLIP vision tower.

[0012] The aforementioned off-road autonomous driving decision-making method based on large model and conditional flow matching trajectory, wherein the hybrid expert network controls the information flow between three modal tokens through a block causal attention mask matrix: the visual language expert's visual text token only pays bidirectional attention within itself, the proprioception expert's token pays attention to the visual text token and itself, and the action expert's token pays attention to all preceding modalities and performs unidirectional causal attention within the action modality.

[0013] The off-road autonomous driving decision-making method based on large model and conditional flow matching trajectory described above injects noisy trajectory and time step embedding into the action encoder during training. After the action token is generated by the multilayer perceptron, it is sent to the action expert. The time step information is simultaneously modulated by the Transformer layer parameters through an adaptive layer normalization mechanism, wherein the time step sampling adopts a Beta (α=1.5, β=1.0) distribution.

[0014] The off-road autonomous driving decision-making method based on large model and conditional flow matching trajectory, step 4 includes: generating the target off-road driving trajectory through multi-step Euler integration, in each integration step, the visual language expert and the proprioceptive expert reuse the key value cached in the first step, and only the action expert performs the complete forward calculation.

[0015] like Figure 8 As shown, this invention also proposes an off-road autonomous driving decision-making device based on a large model and conditional flow matching trajectory, including:

[0016] Module 1 acquires the current frame and multiple historical frames from the vehicle's forward-looking camera, and simultaneously acquires the multi-dimensional proprioceptive state sequence of the vehicle at multiple historical time steps.

[0017] Module 2 sends the current frame and the historical frame into the visual encoder to obtain the visual token corresponding to each frame. The visual tokens are then concatenated, linearly projected and mapped to the hidden area, and then scaled and normalized before being fused with the token of the off-road semantic command text to obtain a multimodal embedding sequence.

[0018] Module 3 constructs a multimodal hybrid expert network that includes visual language experts, proprioceptive experts, and action experts. Each expert processes in parallel and shares the query-key-value projection from the attention layer. Proprioceptive experts and action experts share the feedforward network weights.

[0019] Module 4: The multimodal embedding sequence is input into the hybrid expert network. After processing by the self-attention layer, it is input into the visual language expert, proprioceptive expert, and action expert respectively. The visual language expert is used to understand semantics and spatial environment to obtain visual features; the proprioceptive expert is used to align physical states to obtain ontological features; and the action expert is used to calculate the trajectory trend of the vehicle in the next time step to obtain action features. The visual features, ontological features, and action features are aggregated and then input into the action decoder to obtain the target off-road driving trajectory of the vehicle in the next time period, which is then executed by the vehicle's controller.

[0020] The aforementioned off-road autonomous driving decision-making device based on large model and conditional flow matching trajectory includes a multidimensional proprioceptive state sequence for each historical time step, comprising velocity, three-axis acceleration, three-axis angular velocity, heading angle, and coordinates of a distant target point; the visual encoder is a SigLIP vision tower.

[0021] This hybrid expert network controls the information flow between three modal tokens through a block-based causal attention mask matrix: the visual language expert's visual text token only pays bidirectional attention within itself, the proprioception expert's token pays attention to the visual text token and itself, and the action expert's token pays attention to all preceding modalities and performs unidirectional causal attention within the action modality.

[0022] During training, noisy trajectories and time step embeddings are injected into the motion encoder, and motion token embeddings are generated by the multilayer perceptron and then sent to the motion expert. The time step information is simultaneously modulated by the Transformer layer parameters through an adaptive layer normalization mechanism, where the time step sampling adopts a Beta (α=1.5, β=1.0) distribution.

[0023] Module 4 includes: generating the target off-road driving trajectory through multi-step Euler integration. In each step of integration, the visual language expert and the proprioceptive expert reuse the key values ​​cached in the first module, and only the motion expert performs the complete forward calculation.

[0024] The present invention also proposes an electronic device, including the aforementioned off-road autonomous driving decision-making device based on a large model and conditional flow matching trajectory. The electronic device may be connected to an information display device, which is used to display the target off-road driving trajectory using user-set display parameters, attributes, or through an artificial intelligence model.

[0025] The present invention also proposes a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the off-road autonomous driving decision-making method based on large model and conditional flow matching trajectory.

[0026] The present invention also proposes a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, it implements the steps of the off-road autonomous driving decision-making method based on large model and conditional flow matching trajectory.

[0027] As can be seen from the above solutions, the advantages of the present invention are:

[0028] First, by leveraging the open-vocabulary generalization capability of the SigLIP pre-trained visual encoder and the dual-temporal image contrast encoding, the perceptual bottleneck of traditional closed-set semantic segmentation is broken, enabling the identification of various unseen objects in off-road scenarios without exhaustively listing obstacle labels.

[0029] Second, action prediction is remodeled as conditional generation of a continuous probability flow. The multi-step waypoint continuous trajectory output by the flow matching paradigm naturally satisfies the dynamic kinematic constraints required for off-road terrain.

[0030] Third, by combining Beta distribution skewed sampling, 1 / 99 percentile robust normalization, and large-batch training strategies, the decision instability caused by the scarcity of off-road data and the long-tail distribution is systematically alleviated. Fourth, by using an inactive expert key-value cache reuse mechanism, the computational cost of multi-step flow matching inference is reduced by about 67%. Combined with mixed precision training and optional LoRA quantization, the model is feasible for real-time operation on an embedded GPU in the vehicle. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the overall structure of the off-road VLA as shown in an embodiment of the present invention;

[0032] Figure 2 The diagram illustrates the heterogeneous multimodal MoE network structure and block-based causal attention mechanism as shown in the embodiments of the present invention.

[0033] Figure 3 This is a flowchart illustrating the conditional flow matching training and feedforward inference process in an embodiment of the present invention.

[0034] Figure 4 This is a schematic diagram of an inactive expert key-value cache reuse inference pipeline shown in an embodiment of the present invention;

[0035] Figure 5 This is a schematic diagram illustrating the dual optimizer training architecture and parameter grouping in an embodiment of the present invention;

[0036] Figure 6 This is a schematic diagram of an off-road VLA decision-making device module shown in an embodiment of the present invention;

[0037] Figure 7 This is a flowchart of the method of the present invention;

[0038] Figure 8 This is a block diagram of the device of the present invention;

[0039] Figure 9 This is a schematic diagram of the structure of the first electronic device of the present invention;

[0040] Figure 10 This is a schematic diagram of the application environment structure of the first electronic device of the present invention;

[0041] Figure 11 This is a schematic diagram of the structure of the second electronic device of the present invention.

[0042] Figure label:

[0043] A - First electronic device;

[0044] B-Off-road autonomous driving decision-making device based on large model and conditional flow matching trajectory;

[0045] C-Data acquisition equipment;

[0046] D-Information display device;

[0047] 1000 - Second electronic device;

[0048] Ⅰ-Computational Unit;

[0049] II-ROM;

[0050] III-RAM;

[0051] N-bus;

[0052] V-Interface;

[0053] VI - Input Unit;

[0054] VII - Output Unit;

[0055] VIII - Storage medium;

[0056] IX - Communication Unit. Detailed Implementation

[0057] It should be noted that, in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0058] In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0059] The processor described in this invention is the control center of an electronic device. It can be a single processor or a collective term for multiple processing elements. For example, it can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of this invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0060] Alternatively, the processor can perform various functions of the electronic device by running or executing software programs stored in memory and by calling data stored in memory.

[0061] In a specific implementation, as one example, the processor may include one or more CPUs. Each of these processors may be a single-core processor or a multi-core processor. Here, "processor" can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). Electronic devices may include servers, desktop computers, laptops, smartphones, tablets, embedded computers, etc., where the embedded computer includes vehicles and robots, etc.

[0062] The memory is used to store the software program that executes the solution of the present invention, and the execution is controlled by the processor. For specific implementation methods, please refer to the above method embodiments, which will not be repeated here.

[0063] It should be noted that the structure of the electronic device shown in the accompanying drawings of this invention does not constitute a limitation thereof. The actual knowledge structure recognition device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0064] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0065] It should also be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0066] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0067] It should also be understood that, in various embodiments of the present invention, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0068] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0069] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0070] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0071] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0072] First, let's explain the technical terms: VLA (Vision-Language-Action) model refers to a vision-language-action model that combines the capabilities of a visual encoder with a language model, achieving interpretable decision-making through language anchoring; MoE (Mixture-of-Experts) refers to a hybrid expert network, a neural network architecture that achieves efficient parameter expansion by sparsely activating multiple expert modules; Conditional Flow Matching refers to a continuous probability flow generation method that directly generates samples by integrating ordinary differential equations from the velocity field from the noise distribution to the target distribution; Proprioception refers to the intrinsic perception of the vehicle's own motion state, including kinematic parameters such as speed, acceleration, and angular velocity.

[0073] To achieve the above technical objectives, such as Figure 1 and Figure 6 As shown, the present invention proposes the following technical solution:

[0074] (1) Obtain the dual-temporal forward view image stream, historical proprioceptive state sequence, and off-road semantic command text in the off-road scenario. The image stream includes the current frame and historical frames (Δt=0.5 seconds), which are input into the visual encoder after normalization and standardization. The historical proprioceptive state sequence contains 10-dimensional state information of 5 historical time steps, including velocity, three-axis acceleration, three-axis angular velocity, heading angle, and coordinates of the distant target point. The robust normalization strategy of 1 / 99 percentile pruning is used to map the state values ​​to the interval [-1, +1]. The text command uses the fixed string "predict trajectory" to enable the model to predict the future motion path of the vehicle based on the historical state and environmental constraints.

[0075] (2) The standardized image is input into a visual encoder, such as the SigLIP visual tower, to extract visual features. After being mapped to the dimension matching the language model by a multimodal projector, the image is scaled and normalized. The off-road semantic command text is encoded by a tokenizer and embedded into the image token position. The normalized proprioceptive state is mapped to the hidden dimension by a linear projection. The embedding sequences of the three modalities are fed into the subsequent hybrid expert network.

[0076] (3) The fused multimodal embedding sequence is input into a heterogeneous multimodal hybrid expert network (denoted as JointModel). This network consists of three parallel expert modules: Visual Language Expert (VLM Expert), Proprio Expert, and Action Expert. These three experts are three different computational channels split from the same large language model. They share query-key-value projection parameters from the attention layer and remain independent only in the feedforward network parameters. Among them, the feedforward network weights of Proprio Expert and Action Expert are shared, realizing efficient knowledge transfer between modalities. Figure 2 As shown, the network controls the information flow between three modal tokens through a block-based causal attention mask matrix: visual / text tokens only pay attention bidirectionally within themselves, proprioceptive tokens can pay attention to visual / text and themselves but not to action tokens, and action tokens can pay attention to all preceding modalities (visual text, the vehicle's historical physical state) and perform unidirectional causal attention within the action modality.

[0077] (4) such as Figure 3As shown, the motion expert uses a conditional flow matching paradigm for motion generation. Noisy trajectories and time steps are embedded and injected into the motion encoder. After being processed by a multilayer perceptron to generate motion tokens, these tokens are embedded and fed into the motion expert of the JointModel. Simultaneously, time step information modulates the Transformer layer parameters through an adaptive layer normalization (adaLN / adaLN-Zero) mechanism. Time step sampling uses a Beta (α=1.5, β=1.0) distribution, where Beta is the probability distribution function, α is the first shape parameter affecting the probability density function shape at the right end of the interval (i.e., near 1.0), and β is the second shape parameter affecting the probability density function shape at the left end of the interval (i.e., near 0.0). The probability density curve of the aforementioned Beta (α=1.5, β=1.0) slopes towards t→1, significantly increasing the sampling frequency of "low-noise / close to real trajectory" data during training. This assigns a higher sampling probability to smaller time steps closer to the target trajectory, thereby accumulating a stronger gradient signal during backpropagation to enhance the gradient training signal for off-road boundary scenes.

[0078] The noisy trajectory refers to the following in continuous flow matching algorithms: the model needs to learn a "vector field" that guides the data to smoothly evolve / de-noise from pure random noise (high-noise state) into a clear and smooth real driving trajectory (noise-free state). During the training phase, the algorithm does not only show the model pure noise or a completely clean trajectory, but also samples various intermediate processes with different levels of noise. The data from these intermediate processes constitutes the noisy trajectory.

[0079] (5) In the reasoning stage, such as Figure 4 As shown, a non-active expert key-value caching and reuse mechanism is employed to reduce computational load. In the first integration step, the entire JointModel is executed forward, and the key-value states of VLM Expert and Proprio Expert in all layers are cached. In subsequent integration steps, only the Action Expert is computed forward, and the key-values ​​of VLM Expert and Proprio Expert are read directly from the cache. By directly integrating the ordinary differential equation using Euler integration, the trajectory can be generated in 10 steps, reducing the computational load by approximately 67% compared to the full forward pass.

[0080] (6) The action decoder maps the hidden state output by the action expert to action velocity prediction. It performs multi-step integration using the Euler Integration Step in Flow Matching: a_{t+Δt} = a_t + Δt· a_pred, to gradually denoise / update the trajectory along the predicted velocity field. Finally, the result of the last step is pruned and output as a 7-step future waypoint sequence. a_t is the action / trajectory state at the current time step, a_{t+Δt} is the next action / trajectory state after the update, Δt is the integration time step size, and a_pred is the velocity field vector (action feature) predicted by the action expert.

[0081] (7) A heterogeneous parameter grouping strategy is adopted during the training phase. For example... Figure 5 As shown, the VLM parameter set uses an independent learning rate and an optional LoRA configuration; the motion expert parameter set employs an 8-bit AdamW optimizer with weight decay set to zero to maintain accurate motion output under extreme terrain conditions. Training uses a mixed-precision strategy, with forward / backward computation in BF16 and optimizer state stored in FP32. The visual tower is loaded from the PaliGemma-3B-PT-224 pre-training checkpoint, and motion-related components are randomly initialized.

[0082] (8) Multi-layer combined image data augmentation is used during training to improve perceptual robustness, including random resizing and cropping, brightness perturbation, contrast perturbation, saturation perturbation and hue perturbation; at the same time, large batch training (global batch 1024) and long period cosine annealing strategy are used to alleviate the risk of overfitting under cross-country sparse data.

[0083] To make the above-mentioned features and effects of the present invention clearer and easier to understand, specific embodiments are described below in conjunction with the accompanying drawings. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are merely illustrative. The scope of protection of the present invention is not limited to the disclosed embodiments, but is defined by the appended claims.

[0084] The feedforward VLA decision-making method for off-road unstructured environments provided in this invention includes the following steps.

[0085] Step 1, Data Acquisition. Acquire the current frame and historical frames (Δt=0.5 seconds) from the forward-looking camera. Simultaneously acquire vehicle CAN bus data to calculate the 10-dimensional proprioceptive state over 5 historical time steps, including velocity, three-axis acceleration, three-axis angular velocity, heading angle, and coordinates of the distant target point. After pruning and normalizing to the 1 / 99 percentile, the state sequence is obtained.

[0086] Step two, multimodal coding. The standardized image is fed into the SigLIP visual tower (27 layers, hidden=1152), outputting 256 visual tokens per frame. Two frames are concatenated to form 512 tokens, which are then linearly projected to hidden=2048 for text alignment. After scaling and normalization, these tokens are embedded and fused with the fixed text "predict trajectory". Proprioceptive states are linearly projected (from 10-dimensional to 1024-dimensional) to form independent proprioToken sequences.

[0087] Step 3, heterogeneous MoE forward propagation. The fused multimodal embedding sequence (532 VLM Tokens + 5 Proprio Tokens + 7 Action Tokens) is input into an 18-layer JointModel Transformer. Each expert processes in parallel, sharing the self-attention query-key-value projection, and the Proprio Expert and Action Expert share the feedforward network weights.

[0088] Step four, conditional flow matching inference. After initializing the pure noise trajectory, a predicted trajectory is generated through 10 steps of Euler integration. 10 steps represent a golden balance between real-time performance and trajectory accuracy, achieved through engineering trade-offs. Fewer than 10 steps are faster but may compromise accuracy, while more than 10 steps offer better accuracy but reduced computational efficiency and speed. Integration refers to the numerical solution process of the exponential ordinary differential equation. In each integration step, the VLM Expert and Proprio Expert reuse the key-value pairs cached in the first step, while only the Action Expert performs the complete forward computation. Beta(1.5, 1.0) distribution sampling allows the model to obtain more sufficient gradient updates at key decision points close to the target. Time-step embedding modulates the Transformer layer parameters through the adaLN-Zero mechanism, degenerating into an identity mapping in the early stages of training to accelerate convergence.

[0089] Step 5, Action Decoding Output. The action decoder (flow matching, diffusion model, or multilayer perceptron MLP are also acceptable) maps the action expert's output to 6 future waypoints (x, y coordinates), each step corresponding to 0.5 seconds, covering a total of 3 seconds of off-road driving trajectory. The output is then pruned and inversely normalized to restore the physical coordinates before being sent to the vehicle controller for execution.

[0090] Step 6, Model Training. The overall loss function is a weighted combination of conditional flow matching loss and optional perceptual loss. The VLM parameter set and action expert parameter set use independent AdamW8bit optimizers and learning rate schedulers (action expert learning rate 5e-5, VLM parameter set optional freezing or LoRA fine-tuning, LoRA rank r=32). Training uses BF16 mixed precision, with a gradient clipping threshold max_norm=1.0.

[0091] The key hyperparameters of this invention in a specific training embodiment are as follows: total number of training samples: 28,130; global batch size: 1024; single-card batch size: 4; flow matching Beta distribution parameters: α=1.5, β=1.0; inference integration steps: 10; JointModel Transformer layers: 18; attention heads: 8; VLM Expert hidden: 2048; Proprio / Action Expert hidden: 1024; SigLIP visual tower: 27 layers, hidden: 1152; maximum sequence length: 532; action expert learning rate: 5e-5; VLM parameter group learning rate: 5e-5; LoRA can be optionally enabled (r=32).

[0092] This invention can be used for applications such as off-road autonomous driving closed-loop simulation, off-road robot training, unmanned vehicle perception algorithm verification, digital twins of complex outdoor scenes and offline data replay.

[0093] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0094] like Figure 8 As shown, this invention also proposes an off-road autonomous driving decision-making device based on a large model and conditional flow matching trajectory, including:

[0095] Module 1 acquires the current frame and multiple historical frames from the vehicle's forward-looking camera, and simultaneously acquires the multi-dimensional proprioceptive state sequence of the vehicle at multiple historical time steps.

[0096] Module 2 sends the current frame and the historical frame into the visual encoder to obtain the visual token corresponding to each frame. The visual tokens are then concatenated, linearly projected and mapped to the hidden area, and then scaled and normalized before being fused with the token of the off-road semantic command text to obtain a multimodal embedding sequence.

[0097] Module 3 constructs a multimodal hybrid expert network that includes visual language experts, proprioceptive experts, and action experts. Each expert processes in parallel and shares the query-key-value projection from the attention layer. Proprioceptive experts and action experts share the feedforward network weights.

[0098] Module 4: The multimodal embedding sequence is input into the hybrid expert network. After processing by the self-attention layer, it is input into the visual language expert, proprioceptive expert, and action expert respectively. The visual language expert is used to understand semantics and spatial environment to obtain visual features; the proprioceptive expert is used to align physical states to obtain ontological features; and the action expert is used to calculate the trajectory trend of the vehicle in the next time step to obtain action features. The visual features, ontological features, and action features are aggregated and then input into the action decoder to obtain the target off-road driving trajectory of the vehicle in the next time period, which is then executed by the vehicle's controller.

[0099] The aforementioned off-road autonomous driving decision-making device based on large model and conditional flow matching trajectory includes a multidimensional proprioceptive state sequence for each historical time step, comprising velocity, three-axis acceleration, three-axis angular velocity, heading angle, and coordinates of a distant target point; the visual encoder is a SigLIP vision tower.

[0100] This hybrid expert network controls the information flow between three modal tokens through a block-based causal attention mask matrix: the visual language expert's visual text token only pays bidirectional attention within itself, the proprioception expert's token pays attention to the visual text token and itself, and the action expert's token pays attention to all preceding modalities and performs unidirectional causal attention within the action modality.

[0101] During training, noisy trajectories and time step embeddings are injected into the motion encoder, and motion token embeddings are generated by the multilayer perceptron and then sent to the motion expert. The time step information is simultaneously modulated by the Transformer layer parameters through an adaptive layer normalization mechanism, where the time step sampling adopts a Beta (α=1.5, β=1.0) distribution.

[0102] Module 4 includes: generating the target off-road driving trajectory through multi-step Euler integration. In each step of integration, the visual language expert and the proprioceptive expert reuse the key values ​​cached in the first module, and only the motion expert performs the complete forward calculation.

[0103] like Figure 9 As shown, in another embodiment of the present invention, a first electronic device A is also proposed, which includes the aforementioned off-road autonomous driving decision-making device B based on a large model and conditional flow matching trajectory.

[0104] like Figure 10As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D through a wired or wireless information transmission scheme. The data acquisition device C is used to collect the current frame, historical frames and multi-dimensional proprioceptive state sequence, and the information display device D is used to display the off-road autonomous driving decision results obtained by the present invention.

[0105] The information display device D can process and organize the data output by the first electronic device A based on an information display mechanism to improve the readability of the data. This information display mechanism can be manually preset, for example, visualizing the data output by the first electronic device A. It can present the user with the specified key information based on user-defined display parameters and / or attributes, such as the data range and font, color, and scrolling options. Users can access this information more quickly without needing to navigate to secondary pages or scroll through pages, saving them time and effort. Alternatively, the information display mechanism can be an artificial intelligence (AI) display model that learns the user's key information interests based on past usage habits, such as viewing time, click count, and edit count, and automatically presents rich and necessary key information.

[0106] The present invention also provides a computer program product, which includes a computer program that can be stored on a readable storage medium. When the computer program is executed by a processor, the computer is able to execute the off-road autonomous driving decision-making method based on large model and conditional flow matching trajectory provided by the above methods.

[0107] In another embodiment of the present invention, a storage medium VIII is proposed for storing a computer program that executes the off-road autonomous driving decision-making method based on large model and conditional flow matching trajectory. It should be understood that the storage medium in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0108] Figure 11 A schematic block diagram of a second electronic device 1000 that can be used to implement embodiments of the present invention is shown. The second electronic device 1000 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein. The second electronic device 1000 may be the same as or different from the first electronic device A.

[0109] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from storage medium VIII into random access memory (RAM) III. The RAM III may also store various programs and data required for the operation of the device 1000. The computing unit I, ROM II, and RAM III are interconnected via bus IV. An input / output (I / O) interface V is also connected to bus IV.

[0110] Multiple components in the second electronic device 1000 are connected to I / O interface V, including: input unit VI, such as a keyboard, mouse, etc.; output unit VII, such as various types of displays, speakers, etc.; storage medium VIII, such as a disk, optical disk, etc.; and communication unit IX, such as a network card, modem, wireless transceiver, etc. Communication unit IX allows the second electronic device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0111] The computing unit I can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit I performs the various methods and processes described above, such as method steps S1-S4. For example, in some embodiments, the methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage medium VIII. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1000 via ROM II and / or communication unit IX. When the computer program is loaded into RAM III and executed by computing unit I, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, computing unit I can be configured to perform methods by any other suitable means (e.g., by means of firmware).

[0112] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A decision-making method for off-road autonomous driving based on large model and conditional flow matching trajectory, characterized in that, include: Step 1: Obtain the current frame and multiple historical frames from the vehicle's forward-facing camera, and simultaneously acquire the multi-dimensional proprioceptive state sequence of the vehicle at multiple historical time steps. Step 2: The current frame and the historical frame are fed into the visual encoder to obtain the visual token corresponding to each frame. The visual tokens are then concatenated, linearly projected and mapped to the hidden area, and then scaled and normalized before being fused with the token of the off-road semantic command text to obtain a multimodal embedding sequence. Step 3: Construct a multimodal hybrid expert network that includes visual language experts, proprioceptive experts and action experts. Each expert processes in parallel and shares the query-key-value projection from the attention layer. Proprioceptive experts and action experts share the feedforward network weights. Step 4: Input the multimodal embedding sequence into the hybrid expert network. After processing by the self-attention layer, it is input into the visual language expert, proprioceptive expert and action expert respectively. The visual language expert is used to understand semantics and spatial environment to obtain visual features. This proprioceptive expert is used to align physical states and obtain proprioceptive features; Motion experts calculate the trajectory trend of the vehicle in the next moment to obtain motion features; after aggregating the visual features, the ontological features and the motion features, they are input into the motion decoder to obtain the target off-road driving trajectory of the vehicle in the next time period, which is then executed by the vehicle's controller.

2. The off-road autonomous driving decision-making method based on large model and conditional flow matching trajectory as described in claim 1, characterized in that, The multidimensional proprioceptive state sequence at each historical time step includes velocity, three-axis acceleration, three-axis angular velocity, heading angle, and coordinates of the distant target point; the visual encoder is a SigLIP visual tower.

3. The off-road autonomous driving decision-making method based on large model and conditional flow matching trajectory as described in claim 1, characterized in that, This hybrid expert network controls the information flow between three modal tokens through a block-based causal attention mask matrix: the visual language expert's visual text token only pays bidirectional attention within itself, the proprioception expert's token pays attention to the visual text token and itself, and the action expert's token pays attention to all preceding modalities and performs unidirectional causal attention within the action modality.

4. The off-road autonomous driving decision-making method based on large model and conditional flow matching trajectory as described in claim 1, wherein during training, the noisy trajectory and time step embedding are injected into the action encoder, and the action token is generated by the multilayer perceptron and then sent to the action expert; the time step information is simultaneously modulated by the Transformer layer parameters through an adaptive layer normalization mechanism, wherein the time step sampling adopts a Beta (α=1.5, β=1.0) distribution.

5. The off-road autonomous driving decision-making method based on large model and conditional flow matching trajectory as described in claim 1, wherein step 4 includes: The off-road driving trajectory of the target is generated through multiple Euler integration steps. In each integration step, the visual language expert and the proprioceptive expert reuse the key values ​​cached in the first step, and only the motion expert performs the complete forward calculation.

6. An off-road autonomous driving decision-making device based on large model and conditional flow matching trajectory, characterized in that, include: Module 1 acquires the current frame and multiple historical frames from the vehicle's forward-looking camera, and simultaneously acquires the multi-dimensional proprioceptive state sequence of the vehicle at multiple historical time steps. Module 2 sends the current frame and the historical frame into the visual encoder to obtain the visual token corresponding to each frame. The visual tokens are then concatenated, linearly projected and mapped to the hidden area, and then scaled and normalized before being fused with the token of the off-road semantic command text to obtain a multimodal embedding sequence. Module 3 constructs a multimodal hybrid expert network that includes visual language experts, proprioceptive experts, and action experts. Each expert processes in parallel and shares the query-key-value projection from the attention layer. Proprioceptive experts and action experts share the feedforward network weights. Module 4: The multimodal embedding sequence is input into the hybrid expert network. After processing by the self-attention layer, it is input into the visual language expert, proprioceptive expert and action expert respectively. The visual language expert is used to understand semantics and spatial environment to obtain visual features. This proprioceptive expert is used to align physical states and obtain proprioceptive features; Motion experts calculate the trajectory trend of the vehicle in the next moment to obtain motion features; after aggregating the visual features, the ontological features and the motion features, they are input into the motion decoder to obtain the target off-road driving trajectory of the vehicle in the next time period, which is then executed by the vehicle's controller.

7. The off-road autonomous driving decision-making device based on large model and conditional flow matching trajectory as described in claim 6, characterized in that, The multidimensional proprioceptive state sequence at each historical time step includes velocity, three-axis acceleration, three-axis angular velocity, heading angle, and coordinates of the distant target point; the visual encoder is a SigLIP visual tower; This hybrid expert network controls the information flow between three modal tokens through a block-based causal attention mask matrix: the visual language expert's visual text token only pays bidirectional attention within itself, the proprioception expert's token pays attention to the visual text token and itself, and the action expert's token pays attention to all preceding modalities and performs unidirectional causal attention within the action modality. During training, noisy trajectories and time step embeddings are injected into the motion encoder, and motion token embeddings are generated by the multilayer perceptron and then sent to the motion expert. The time step information is simultaneously modulated by the Transformer layer parameters through an adaptive layer normalization mechanism, where the time step sampling adopts a Beta (α=1.5, β=1.0) distribution. Module 4 includes: generating the target off-road driving trajectory through multi-step Euler integration. In each step of integration, the visual language expert and the proprioceptive expert reuse the key values ​​cached in the first module, and only the motion expert performs the complete forward calculation.

8. An electronic device, characterized in that, The device includes an off-road autonomous driving decision-making device based on a large model and conditional flow matching trajectory as described in claim 6 or 7. The electronic device may be connected to an information display device, which is used to display the target off-road driving trajectory using user-set display parameters, attributes, or through an artificial intelligence model.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the off-road autonomous driving decision-making method based on a large model and conditional flow matching trajectory as described in any one of claims 1-5.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the off-road autonomous driving decision-making method based on large model and conditional flow matching trajectory as described in any of claims 1-5.

Images