Cross-country scene unmanned vehicle auxiliary decision-making method based on pre-training visual language model

Abstract

The invention discloses a pre-training visual language model-based cross-country scene unmanned vehicle auxiliary decision-making method, and belongs to the technical field of unmanned driving. Comprising the following steps: constructing a simulation data set for training a three-dimensional target detection network; carrying out cross-modal consistency fusion and cue word design; and generating and mapping driver element actions and decisions. According to the invention, by fusing a three-dimensional target detection network and a pre-trained visual language model, introducing a thinking chain type reasoning and selective spatial information injection mechanism, and combining a trigger type reasoning process and meta-action sequence generation, high-level understanding and auxiliary decision making of a cross-country scene are realized; therefore, the defects of a traditional geometric programming algorithm in an unstructured environment are overcome.

Description

A method for assisting decision-making in off-road scenarios for unmanned vehicles based on pre-trained visual language models Technical Field

[0001] This invention belongs to the field of autonomous driving technology, and in particular relates to a method for assisting autonomous driving decision-making in off-road scenarios for unmanned vehicles based on a pre-trained visual language model and utilizing visual information combined with 3D target detection. Background Technology

[0002] Autonomous driving in off-road scenarios is a crucial application area of ​​driverless technology, primarily targeting autonomous driving tasks in unstructured or natural environments. Compared to structured urban roads, off-road scenarios typically lack clear road boundaries, lane lines, or traffic signs. Road surfaces vary widely, potentially consisting of mud, sand, gravel, etc., and may be accompanied by irregularly shaped obstacles such as potholes, branches, rocks, and shrubs, resulting in a high degree of randomness and uncertainty. Furthermore, changes in lighting conditions, weather conditions, and differences in surface reflectivity can affect sensors such as cameras and lidar, reducing the stability of perception results. Therefore, accurately identifying environmental elements and generating reliable driving strategies in such unstructured scenarios remains a core challenge in off-road autonomous driving research.

[0003] Existing off-road autonomous driving systems generally employ geometric reconstruction and drivability analysis methods based on LiDAR or depth cameras. These methods generate drivable trajectories for the vehicle by constructing local grid maps, calculating terrain slope, and estimating curvature. While these methods tend to be relatively stable in scenarios with clear terrain boundaries and regular geometric features, they become susceptible to problems when facing common soft obstacles, irregular objects, or complex textured surfaces. These sensors are easily affected by noise, insufficient reflectivity, and occlusion, leading to inaccurate obstacle identification and incorrect judgments of drivable areas. Furthermore, traditional geometric planning methods primarily focus on spatial structure and lack effective modeling capabilities for semantic-level driving decisions such as "whether grass is drivable," "whether slowing down is necessary in mud," and "whether detours are needed around rocks." This can easily result in problems such as predicted pathpoint oscillations, redundant planning, and decision-making stagnation.

[0004] With the development of deep learning technology, vision-based scene understanding methods are gradually being applied to the field of off-road autonomous driving. Some studies attempt to use models based on convolutional networks (CNNs) or Transformers to analyze textures, semantic categories, and surface types in natural scenes, thereby mitigating some of the problems existing in traditional geometric planning. In recent years, Vision-Language Models (VLMs) have shown a new development trend in the field of autonomous driving due to their excellent scene understanding and semantic reasoning capabilities. Pre-trained VLMs can extract semantic information from single or multiple frames of images and generate interpretable scene descriptions and driving strategies through chain-of-thought (CoT) reasoning. However, large-scale VLMs typically have high inference costs and large inference latency, making them difficult to deploy on autonomous vehicle platforms with limited computing resources. In addition, VLMs still rely on image features for spatial geometric understanding and remain unstable in judging the physical attributes of objects such as distance, size, and orientation. In addition, existing methods mostly output predicted trajectory points or control commands directly in an end-to-end manner, lacking a mechanism to work in conjunction with traditional geometric planners and having poor interpretability, making it difficult to apply directly to autonomous driving systems that prioritize robustness and reliability.

[0005] In summary, existing off-road autonomous driving technologies still have shortcomings in semantic understanding, spatial geometry judgment, and high-level policy generation. Especially when facing complex and irregular obstacles and highly random terrain conditions in natural environments, traditional methods struggle to maintain stable and reliable decision-making performance. Therefore, there is a need for an auxiliary decision-making method that can operate under limited onboard computing resources, possesses semantic reasoning capabilities, and can collaborate with traditional planning modules to improve the autonomous vehicle's environmental understanding and overall decision-making level in off-road scenarios. This invention addresses these problems by integrating a 3D object detection network with a pre-trained visual language model, introducing a chain-of-thought reasoning and selective spatial information injection mechanism, and combining a triggered reasoning process with meta-action sequence generation to achieve high-level understanding and auxiliary decision-making in off-road scenarios, thereby compensating for the shortcomings of traditional geometric planning algorithms in unstructured environments. Summary of the Invention

[0006] Based on an investigation and analysis of existing technologies related to autonomous driving in off-road scenarios, this invention innovatively proposes an off-road scenario auxiliary decision-making method that integrates a lightweight pre-trained visual language model with 3D spatial perception. This method comprises a 3D object detection module, a cross-modal consistency fusion module, a visual language reasoning module, and an auxiliary decision generation module. The system comprises several modules: a 3D object detection module to obtain the spatial location, size, and orientation features of obstacles from the forward-facing RGB image; a cross-modal consistency fusion module to perform consistency matching between the 3D object detection results and the region of interest of the visual language model, and to supplement the visual language model's prompt word template with 3D information in the form of structured prompts when the match is reliable, thereby enhancing the model's spatial understanding ability; and a module to maintain the original semantic judgment of the visual language model when the match is unreliable, in order to suppress the interference of unreliable spatial information on the inference results; a visual language inference module to perform semantic parsing on the input forward-facing RGB image using a pre-trained visual language model, and to generate high-level semantic information such as scene description, accessibility assessment, and potential risks through thought chain inference, thereby guiding the model to generate subsequent driving decisions; and an auxiliary decision generation module to combine the above semantic information to output interpretable meta-action sequences, and to convert them into specific control commands through a meta-action mapping mechanism, thereby supporting the autonomous vehicle to perform strategy adjustments such as deceleration, detour, and observation, and improving the decision-making stability and environmental adaptability of the autonomous vehicle in off-road scenarios.

[0007] The technical solution of the present invention:

[0008] A method for assisting decision-making in off-road scenarios for unmanned vehicles based on a pre-trained visual language model, comprising the following steps:

[0009] Step 1: Construct a simulation dataset for training a 3D object detection network

[0010] A dataset for 3D target detection in off-road scenarios was collected and constructed using the CARLA simulation platform, and a 3D target detection model was trained based on this dataset.

[0011] Furthermore, during data acquisition, by randomly setting the position and attitude of the unmanned vehicle, the lighting conditions of the simulation environment, and the types of obstacles, diverse off-road scenarios containing various natural obstacles are automatically generated. The simulation engine can directly output the 3D detection bounding box parameters of each obstacle in the scene, including the center position of the target. , size and azimuth Based on the camera's intrinsic and extrinsic parameter matrices, the corner points of the 3D detection bounding boxes of obstacles are projected onto the 2D image plane, thus obtaining 2D detection bounding boxes that strictly correspond to the 3D annotations. This ensures that each data sample simultaneously contains an RGB image, a 2D detection bounding box, and 3D data annotations. In this way, training samples conforming to the KITTI format can be generated without manual annotation, thereby overcoming the difficulty in obtaining 3D target detection annotation data for unmanned vehicles in off-road scenarios.

[0012] The aforementioned 3D object detection model is constructed by introducing a 3D object parameter regression head onto YOLO11n as a 2D detection network. The dataset described above is input into the 3D object detection model for training, enabling it to jointly predict obstacle targets and ultimately obtain the center positions of the obstacle targets. ,size and azimuth The three-dimensional detection results, i.e., the three-dimensional geometric parameters During the training of the 3D object detection model, a joint loss function is introduced to optimize the model parameters. The joint loss function is expressed as:

[0013]

[0014] in This represents the loss in the three-dimensional dimensions of the target. This indicates the target orientation parameter loss. These are the weighting coefficients used to balance size loss and orientation loss.

[0015] Furthermore, the orientation parameter loss described in the formula It is constructed using a combination of classification and location methods, and its form can be represented as:

[0016]

[0017] in, The classification loss term represents the orientation interval and is used to measure the difference between the predicted orientation interval and the actual orientation interval. The positioning loss term represents the orientation residual, used to constrain the continuous offset of the predicted orientation within the corresponding interval; These are the weighting coefficients used to balance classification loss and localization loss.

[0018] The orientation residual positioning loss in the formula It can be represented as:

[0019]

[0020] in, Indicates the target's true facing angle. Indicates the central angle of the corresponding orientation interval. This represents the orientation residual predicted by the network. This represents the number of valid orientation intervals involved in the calculation. Constraining the angle residuals using cosine similarity can effectively improve the continuity and stability of orientation prediction.

[0021] Furthermore, the three-dimensional size loss The calculation method is shown in the following expression:

[0022]

[0023] in, Represents the true three-dimensional dimensions of the target. This represents the average size parameter of the corresponding category in the training dataset. This represents the size residuals predicted by the network. Indicates the number of samples.

[0024] By training on a dataset, the 2D detection network is able to predict the 3D spatial structure information of targets in off-road scenarios, providing spatial information for subsequent inference by the visual language model.

[0025] Step 2: Cross-modal consistency fusion and prompt word design

[0026] This step is used to build a visual language reasoning module suitable for semantic understanding and high-level driving strategy generation in off-road scenarios. This module uses a pre-trained lightweight visual language model, MobileVLM, to perform semantic understanding on RGB images from a front-facing camera and perform reasoning when trigger conditions are met. It is used to identify high-level environmental semantics such as terrain features, obstacle attributes, drivability, and potential risks, and uses thought chain reasoning to gradually generate high-level driving strategies.

[0027] Step 2.1: The MobileVLM model includes a visual encoder and a large language model; the input image is simultaneously input into both the MobileVLM model and the 3D object detection model; the 3D object detection model obtains the 3D detection result, and simultaneously, the visual encoder of the MobileVLM model obtains the visual feature vector. Large language models are based on visual feature vectors Generate a semantic representation of the scene, denoted as To enhance the model's spatial understanding capabilities, this step employs a selective injection mechanism in its cue word design: when the cross-modal consistency condition is met, the three-dimensional geometric parameters obtained in step 1 are injected. The prompts are injected into structured text and transformed into prompt vectors that can be processed by the MobileVLM model. .

[0028] Step 2.2: To determine whether the 3D detection results obtained in Step 2.1 can supplement the inference of the visual language model, the 3D detection bounding box of the 3D target is... Back projection onto a two-dimensional plane to obtain a two-dimensional detection bounding box and the visual feature vector obtained by the visual encoder Obstacle areas of concern The intersection-over-union ratio (IoU) is calculated and defined using an approximate IoU form:

[0029]

[0030] in This represents the area of ​​the intersection region of the two bounding boxes. This is used to measure the area of ​​the 2D region within the bounding box of a 3D detection. When the IoU value is greater than a threshold... If the obstacle types are identical, the two are considered consistent and classified as a matching object, serving as supplementary structured information. If the condition is not met, it is classified as a non-matching object, and the model's original semantic reasoning result is used.

[0031] Step 2.3: When the 3D detection result is consistent with the obstacle type as determined in Step 2.2, the 3D geometric parameters in the 3D detection result... Semantic representation combined with the scene Through text mapping function The text is converted into a structured text format that the MobileVLM model can process and then injected into the prompt word template to obtain structured prompt words. Furthermore, it supplements the spatial information of the large language model in the form of natural language, thereby enhancing the spatial understanding ability of the large language model.

[0032] Step 2.4, set the three-dimensional geometric parameters With structured prompts Inputting both into the MobileVLM model generates a high-level semantic sequence. Based on high-level semantic sequences Visual feature vectors Structured prompt words The conditional probability of generating the corresponding semantic unit under given input conditions is obtained as follows:

[0033]

[0034] in, This represents a high-level semantic sequence generated by a visual language model. This represents the i-th semantic unit in the semantic sequence. Let N represent the sequence of semantic units generated up to step i, and let N represent the total number of generation steps in the semantic sequence. It represents the conditional probability of generating the corresponding semantic unit under given input conditions; the language output at each step is completed under the joint constraints of visual and spatial information, enabling the model to infer the environmental structure, terrain category, obstacle attributes and possible passage risks based on the input conditions.

[0035] Step 2.5: Based on conditional probability, the final reasoning result is obtained by following the multi-stage reasoning path of "scenario understanding - risk judgment - driving intention".

[0036] Furthermore, to enable the model to obtain more interpretable reasoning paths in off-road scenarios, this step also designs a multi-stage reasoning path of "scenario understanding - risk assessment - driving intention," guiding the model to analyze the scenario step by step using a chain-like reasoning approach. The reasoning trajectory generated by the large language model can be represented as:

[0037]

[0038] This ensures that the final output of high-level semantic expressions is consistent and interpretable.

[0039] Furthermore, the prompt word template includes formatting constraints on the semantic structure of scene description, risk assessment, and driving intention. The prompt words about scene description are at the beginning of the prompt word template, the prompt words about risk assessment are in the middle of the prompt word template, and the prompt words about driving intention are at the end of the prompt word template. This guides the model to output a comprehensive understanding of the scene in a unified language format, thereby making the model's output more stable and controllable.

[0040] The prompt word template is constructed using natural language. The model first provides a general scene description of the image content, including terrain features, obstacle types, and the surrounding environment. Then, it combines semantic understanding to infer potential factors that may affect driving, such as the risk of vehicle bumps due to potholes or the risk of vehicle skidding due to muddy or slippery surfaces, thus forming risk warnings. Finally, it generates driving intentions based on the aforementioned semantic content, such as "slow down and observe" or "the current environment is complex and requires cautious driving," to guide the model's decision-making tendency. This prompt word template does not rely on retraining the model parameters. Instead, through lightweight constraints on the prompt structure, off-road domain terminology, and reasoning logic, the visual language model can more stably identify typical off-road elements such as gravel roads, mud, and potholes. It generates more consistent, stable, and interpretable semantic outputs according to a chain-like reasoning path of "scene understanding—potential risk—driving intention," enabling the model to provide reliable high-level semantic basis for subsequent meta-action generation modules.

[0041] Step 3: Driving Meta-Action Generation and Decision Mapping

[0042] Using the scenario description, potential risk warnings, and high-level driving intentions generated step by step in step 2, the large language model further generates a set of executable driving meta-action sequences based on the high-level semantic sequences, and maps them to specific low-level control commands according to the vehicle's operating status, so as to achieve auxiliary decision-making for traditional planners.

[0043] The mapping process is represented as follows:

[0044]

[0045] in This represents the underlying control commands that the autonomous vehicle can execute. This indicates that the vehicle is in a low-speed state, and it maps the original action 'a' to the corresponding low-speed control command using a mapping function. This indicates that the vehicle is in a high-speed state. The meta-action 'a' is mapped to the high-speed control command mapping function corresponding to the action. Through this simple control strategy mapping method, the model can run on the vehicle platform with limited computing resources. Thus, without changing the main planning architecture, it provides effective auxiliary decision-making for traditional planning methods when facing negative obstacles, complex terrain conditions, etc., and improves the decision-making stability and adaptability of unmanned vehicles in off-road scenarios.

[0046] Meta-actions refer to abstract driving behaviors such as acceleration, deceleration, stopping, and observation. Based on the semantic reasoning results output by the model, the system selects 2-3 coherent meta-actions from a preset action set to form a meta-action sequence, which serves as the system's decision suggestion for the current scenario.

[0047] To ensure that the sequence of actions can be translated into appropriate control commands under different vehicle operating conditions, this step employs a decision mapping strategy based on vehicle operating conditions. The autonomous vehicle's operating conditions are categorized into low-speed and high-speed modes, and different control strategy sets are used to map the actions to different modes. When the autonomous vehicle is in low-speed mode, actions are mapped to the low-speed control strategy set, and the actions are translated into more stable and detailed control commands. When the autonomous vehicle is in high-speed mode, actions are mapped to the high-speed control strategy set, and the mapping results prioritize the vehicle's dynamic stability and directional control capabilities. This mapping method ensures that the auxiliary decision-making module outputs control commands that meet requirements under different speed conditions.

[0048] Beneficial results of the present invention:

[0049] (1) A three-dimensional detection dataset for off-road scenarios was constructed based on a simulation platform.

[0050] Data sets required for 3D object detection in off-road scenarios are scarce in practical engineering. Real-world data acquisition faces challenges such as uncontrollable scenarios, difficulty in reproducing obstacles, and high annotation costs, often resulting in insufficient available data when training 3D object detection networks. This invention constructs a training dataset containing typical off-road terrain and obstacle types based on the CARLA simulation platform. By simulating various off-road scene elements such as gravel, mud, and small potholes, the simulation engine automatically outputs the center position, size, and orientation angle of obstacles, forming training data with standardized structure and consistent annotation. This provides a reliable training and validation foundation in situations where 3D object detection training data is scarce in off-road scenarios, offering necessary support for acquiring 3D geometric information in decision support methods.

[0051] (2) Innovativeness of the method

[0052] This invention proposes an autonomous vehicle decision-making assistance method for off-road scenarios. By combining a pre-trained lightweight visual language model with a 3D object detection network, it achieves collaborative reasoning of semantic understanding and 3D spatial perception. It innovatively introduces a dual-reasoning mechanism based on detection triggering and time triggering, as well as a selective fusion strategy between 3D object detection results and the region of interest of the visual language model. Employing a control strategy generation method based on meta-action mapping, it effectively improves the decision reliability of autonomous vehicles when using traditional LiDAR-based end-to-end methods in scenarios such as negative obstacles and unstructured terrain by combining semantic reasoning and 3D detection information. Specifically, by judging the similarity between the 3D bounding box projection and the model's region of interest, if the result is reliable, 3D spatial information is injected into the prompt word template in the form of structured text, thereby enhancing the spatial understanding ability of the visual language model under limited computational resources; if consistency is insufficient, the original semantic reasoning result of the visual language model is maintained to avoid introducing unreliable information. Unlike traditional methods that rely on visual language models to directly predict trajectory points several time steps in the future, this invention generates meta-action sequences based on semantic reasoning and enhances the model's scene understanding capabilities through structured prompts. This enables the system to improve decision-making reliability in unstructured scenarios such as complex terrain and negative obstacles. The design of this invention links spatial perception, semantic reasoning, and decision support, opening up new avenues for the practical deployment of visual language models in off-road autonomous vehicles.

[0053] (3) Auxiliary decision-making method based on meta-action mapping

[0054] This invention replaces the traditional visual language model-based autonomous driving method of directly outputting predicted trajectory points with a meta-action mapping control strategy. Compared to directly having the model output predicted trajectory points, this invention, after acquiring scene semantics, generates a set of meta-action sequences based on the inference results of the visual language model and maps them to control commands according to the vehicle's operating mode, thereby effectively reducing computational resource requirements. Especially when facing negative obstacles such as potholes or complex off-road terrain, this method provides important semantic supplementation to traditional end-to-end planning algorithms that rely on LiDAR, allowing for timely adjustments to vehicle behavior and improving the stability and environmental adaptability of overall decision-making. Because the meta-action mapping strategy has lower computational resource requirements, this invention can be deployed on autonomous vehicle platforms with limited computational resources, providing a feasible solution for the lightweight deployment of autonomous vehicles in off-road scenarios. Attached Figure Description

[0055] Figure 1 shows an example of the prompts and results when cross-modal consistency injection was not performed.

[0056] Figure 2 shows examples of prompts and results for injecting 3D information through cross-modal consistency matching.

[0057] Figure 3 is a schematic diagram of the auxiliary decision-making network structure of the present invention. Detailed Implementation

[0058] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0059] This invention utilizes a 3D target detection dataset for off-road scenarios generated using the CARLA simulation platform. During dataset construction, an off-road map scene containing unstructured roads is first loaded into the CARLA simulation environment. Typical off-road obstacle models such as rocks, potholes, and logs are randomly placed within the scene. Simultaneously, the initial position, orientation, and ambient lighting conditions of the unmanned vehicle are randomly sampled to generate diverse off-road driving scenarios.

[0060] During data acquisition, continuous RGB images are obtained by a forward-facing camera positioned at the front of the vehicle. The CARLA simulation engine can directly output the 3D geometric information of each obstacle in the scene in the world coordinate system, including the coordinates of the obstacle's center point, size parameters, and orientation angle. To obtain 3D annotation information consistent with the camera's viewpoint, a coordinate transformation relationship from the world coordinate system to the camera coordinate system is first constructed based on the vehicle's pose and camera parameters. Then, the 3D bounding boxes of the obstacles are transformed from the world coordinate system to the camera coordinate system using the camera extrinsic matrix. Finally, combined with the camera intrinsic matrix, the eight corner points of the 3D bounding boxes are projected onto the 2D image plane to obtain the corresponding 2D projected bounding boxes.

[0061] To ensure the effectiveness of data annotation, only obstacle targets that are within the camera's field of view after projection and have a positive depth are retained, while invalid targets that are severely occluded or outside the field of view are filtered out, thus ensuring consistency between 3D annotations and image content. Through this method, each data sample simultaneously includes a forward-view RGB image, a corresponding 2D projected bounding box, and complete 3D geometric annotation information, including the target's spatial location, size parameters, and orientation angle, achieving the standardized annotation format required for 3D target detection.

[0062] The constructed dataset was divided into training, validation, and test sets in an 8:1:1 ratio for training and performance evaluation of the 3D object detection network. During model training, the commonly used stochastic gradient descent (SGD) optimization method was employed, and a momentum mechanism was introduced to accelerate convergence and improve training stability. The batch size during training was set to 8, and the total number of training epochs was set to 100.

[0063] To address the varying importance of different parameters in 3D object detection tasks, this invention assigns different weight coefficients to each regression component in the loss function. Specifically, the weight of the 3D size regression loss is set to 0.6, and the weight of the target orientation angle regression loss is set to 0.4, balancing the influence of size estimation and pose estimation on the overall training process. The optimizer's initial learning rate is set to 0.0001, and the momentum coefficient is set to 0.9. Through this training strategy, the 3D object detection network can stably converge on a simulated off-road scene dataset and achieve reliable 3D spatial parameter prediction capabilities, providing accurate spatial information support for subsequent cross-modal semantic reasoning and decision support.

[0064] In the method of this invention, a single-frame RGB image acquired by a front-facing camera is first simultaneously input into a 3D object detection network and a pre-trained visual language model. To ensure the effectiveness and real-time performance of the auxiliary decision-making module in off-road scenarios, this invention employs a dual inference triggering mechanism: firstly, when the 3D object detection network detects a preset type of off-road obstacle, an inference is immediately triggered; secondly, when the system does not perform semantic inference within a set time window, an inference is periodically forced to be triggered, thereby avoiding the impact on decision reliability due to a prolonged lack of updated semantic information.

[0065] During the inference process, the obstacle regions of interest to the visual language model are first matched with the projected bounding boxes of the 3D target detection results on the 2D plane. When the matching result meets preset conditions, the 3D spatial information of the corresponding obstacle is injected into the prompt word template in the form of structured text to enhance the visual language model's understanding of spatial structure and distance relationships. When the matching result does not meet the conditions, the original semantic inference result of the visual language model based on image information is directly adopted to avoid introducing unreliable spatial information. Finally, based on the generated scene description, potential risks, and driving intentions, the visual language model further outputs a set of meta-action sequences. These meta-action sequences are mapped to vehicle control commands and used as auxiliary decision results in the mainline navigation algorithm.

[0066] This invention integrates as an auxiliary decision-making module into an autonomous vehicle navigation system based on the Neupan algorithm. Addressing the issues of insufficient semantic information regarding negative obstacles and unstructured terrain in off-road scenarios, the sensor configuration of the autonomous vehicle was proactively upgraded on the experimental platform. A forward-facing RGB camera and LiDAR were introduced to work collaboratively, enhancing the system's perception capabilities in complex environments. This invention does not replace the original Neupan navigation and control framework; rather, it supplements and corrects the planning results, thereby enhancing the system's generalization ability and decision-making stability in complex off-road environments.

[0067] On an off-road experimental platform built using the Songling LIMO series wheeled unmanned vehicles, multiple rounds of real-vehicle tests were conducted on various typical off-road scenarios, including gravel roads, mud, waterlogged potholes, and dynamic obstacles. Experimental results show that, under the same test conditions, the introduction of the auxiliary decision-making module of this invention significantly improves the task completion rate of the unmanned vehicle in off-road missions, with the proportion of vehicles successfully reaching the expected destination consistently exceeding 96%. This verifies the effective supplementary role of this invention to traditional Neupan algorithm-based navigation systems in complex off-road scenarios.

[0068] The development and verification equipment system for the method of this invention is Ubuntu 20.04, with an Intel Core i7-6700K CPU and an NVIDIA GeForce GTX1080Ti GPU with 8GB of video memory, capable of supporting the training of 3D object detection networks and the inference of lightweight visual language models. The final deployment environment is an unmanned platform equipped with an NVIDIA Orin-X chip.

Claims

1. A method for assisting decision-making in off-road scenarios for unmanned vehicles based on a pre-trained visual language model, characterized in that, The steps are as follows: Step 1: Construct a simulation dataset for training the 3D target detection network. A dataset for 3D target detection in off-road scenarios is collected and constructed using the CARLA simulation platform, and a 3D target detection model is trained based on this dataset. The 3D target detection model is constructed by introducing a 3D target parameter regression head on top of YOLO11n as a 2D detection network. The aforementioned dataset is input into the 3D target detection model to train it, enabling the 3D target detection model to jointly predict obstacle targets and ultimately obtain the center position of the obstacle target. ,size and azimuth The three-dimensional detection results, i.e., the three-dimensional geometric parameters During the training of the 3D object detection model, a joint loss function is introduced to optimize the model parameters. The joint loss function is expressed as follows: ;in This represents the loss in the three-dimensional dimensions of the target. This indicates the target orientation parameter loss. The weighting coefficients are used to balance the size loss and orientation loss; Step 2: Cross-modal consistency fusion and cue word design Step 2.1: The MobileVLM model includes a visual encoder and a large language model; The input image is simultaneously input into the MobileVLM model and the 3D object detection model; The 3D object detection model obtains the 3D detection result, and at the same time, the visual encoder of the MobileVLM model obtains the visual feature vector. Large language models are based on visual feature vectors Generate a semantic representation of the scene, denoted as ； Step 2.2: To determine whether the 3D detection results obtained in Step 2.1 can supplement the inference of the visual language model, the 3D detection bounding box of the 3D target is... Back projection onto a two-dimensional plane to obtain a two-dimensional detection bounding box and the visual feature vector obtained by the visual encoder. Obstacle areas of concern The intersection-over-union ratio (IoU) is calculated and defined using an approximate IoU form: ;in This represents the area of ​​the intersection region of the two bounding boxes. This represents the area of ​​the 2D region within the 3D detection bounding box; when the IoU value is greater than a threshold... If the obstacle types are consistent, they are considered to be consistent and classified as matching objects, serving as supplementary structured information; if the condition is not met, they are classified as non-matching objects, and the model's original semantic reasoning result is used; Step 2.3, when the 3D detection result is determined to be consistent with the obstacle type after step 2.2, the 3D geometric parameters in the 3D detection result are... Semantic representation combined with the scene Through text mapping function The text is converted into a structured text format that the MobileVLM model can process and then injected into the prompt word template to obtain structured prompt words. Furthermore, it supplements the spatial information of the large language model in the form of natural language, thereby enhancing the spatial understanding ability of the large language model; Step 2.4, set the three-dimensional geometric parameters With structured prompts Inputting both into the MobileVLM model generates a high-level semantic sequence. Based on high-level semantic sequences Visual feature vectors Structured prompt words The conditional probability of generating the corresponding semantic unit under given input conditions is obtained as follows: ;in, This represents a high-level semantic sequence generated by a visual language model. This represents the i-th semantic unit in the semantic sequence. Let N represent the sequence of semantic units generated up to step i, and let N represent the total number of generation steps in the semantic sequence. It represents the conditional probability of generating the corresponding semantic unit given the input conditions; Step 2.5: Based on conditional probability, the final reasoning result is obtained by sequentially following a multi-stage reasoning path of "scene understanding - risk judgment - driving intention"; Step 3: Driving meta-action generation and decision mapping. Utilizing the scene description, potential risk warnings, and high-level driving intentions gradually generated by the method in Step 2, the large language model further generates a set of executable driving meta-action sequences based on the high-level semantic sequence, and maps them to specific low-level control commands according to the vehicle's operating state to achieve auxiliary decision-making for traditional planners; The mapping process is expressed as: ;in This represents the underlying control commands that the autonomous vehicle can execute. This indicates that the vehicle is in a low-speed state, and it represents the mapping function that maps the original action 'a' to the corresponding low-speed control command. This indicates that the vehicle is in a high-speed state, and it is a mapping function that maps the original action 'a' to the high-speed control command corresponding to that action.

2. The method for assisting decision-making in off-road scenarios for unmanned vehicles based on a pre-trained visual language model according to claim 1, characterized in that, During data acquisition, by randomly setting the position and attitude of the unmanned vehicle, the lighting conditions of the simulation environment, and the types of obstacles, diverse off-road scenarios containing various natural obstacles are automatically generated. The simulation engine can directly output the 3D detection bounding box parameters of each obstacle in the scene, including the center position of the target. ,size and azimuth ； Based on the camera's intrinsic and extrinsic parameter matrices, the corner points of the obstacle's 3D detection bounding box are projected onto the 2D image plane, thus obtaining a 2D detection bounding box that strictly corresponds to the 3D annotation, so that each data sample simultaneously contains an RGB image, a 2D detection bounding box, and 3D data annotations.

3. The method for assisting decision-making in off-road scenarios for unmanned vehicles based on a pre-trained visual language model according to claim 1, characterized in that, Orientation parameter loss The structure, which combines classification and location, can be represented as follows: ;in, The classification loss term represents the orientation interval, which measures the difference between the predicted orientation interval and the actual orientation interval. The positioning loss term represents the orientation residual, used to constrain the continuous offset of the predicted orientation within the corresponding interval; These are the weighting coefficients used to balance classification loss and localization loss; the orientation residual localization loss in this formula... It can be represented as: ;in, Indicates the target's true facing angle. Indicates the central angle of the corresponding orientation interval. This represents the orientation residual predicted by the network. This indicates the number of valid orientation intervals involved in the calculation; additionally, the three-dimensional size loss... The calculation method is shown in the following expression: ;in, Represents the true three-dimensional dimensions of the target. This represents the average size parameter of the corresponding category in the training dataset. This represents the size residuals predicted by the network. Indicates the number of samples.

4. The method for assisting decision-making in off-road scenarios for unmanned vehicles based on a pre-trained visual language model according to claim 1, characterized in that, To enable the model to obtain more interpretable reasoning paths in off-road scenarios, this step also designs a multi-stage reasoning path of "scenario understanding - risk assessment - driving intention," guiding the model to analyze the scenario step by step using a chain-like reasoning approach; the reasoning trajectory generated by the large language model can be represented as: This ensures that the final output of high-level semantic expressions is consistent and interpretable.

5. The method for assisting decision-making in off-road scenarios for unmanned vehicles based on a pre-trained visual language model according to claim 1, characterized in that, The aforementioned prompt word template includes formatted constraints on the semantic structure of scene description, risk assessment, and driving intention. Specifically, prompt words related to scene description are placed at the beginning of the prompt word template, prompt words related to risk assessment are placed in the middle of the prompt word template, and prompt words related to driving intention are placed at the end of the prompt word template. This guides the model to output a comprehensive understanding of the scene in a unified language format, thereby making the model's output more stable and controllable.

Images