Autonomous driving vehicle feedforward navigation method and device based on visual language model

Abstract

This invention relates to a feedforward navigation method and apparatus for autonomous vehicles based on a visual language model. The method includes: acquiring measurement data from multiple sensors and an image stream from a forward-looking camera; performing forward-looking semantic analysis on the images using a fine-tuned visual language model to predict environmental events that may degrade sensor performance within a future time window and outputting semantic confidence scores; constructing a semantic-to-physical parameter mapping model to convert semantic confidence scores into reliability scores; and actively adjusting the measurement noise covariance using the reliability scores before abnormal data enters the filter by constructing a dynamic weighting function, thereby obtaining the final navigation and positioning result. Compared with existing technologies, this invention does not rely on high-precision maps, overcomes the limitation of traditional passive robust methods in response lag at environmental switching boundaries, and effectively improves the transient positioning accuracy and robustness of vehicles in complex dynamic environments such as tunnels, tree-lined roads, and speed bumps.

Description

Technical Field

[0001] This invention relates to a method for locating autonomous vehicles, and more particularly to a feedforward navigation method and apparatus for autonomous vehicles based on a visual language model. Background Technology

[0002] With the development of intelligent connected vehicles, continuous, reliable, and high-precision positioning capabilities across all scenarios are a prerequisite and key to achieving autonomous driving, and also a core technology for next-generation information technologies such as smart cities and the Internet of Things. Currently, inertial navigation systems / global navigation satellite systems (INS / GNSS) integrated navigation has become the basic framework for navigation and positioning of autonomous vehicles due to its ability to provide long-term stable absolute positioning information and high-frequency relative motion attitude.

[0003] However, in complex environments such as urban canyons, tunnels, and under overpasses, GNSS signals are highly susceptible to interference from multipath and non-line-of-sight effects, leading to a significant decrease or even interruption in positioning accuracy. To compensate for the shortcomings of INS / GNSS integrated navigation systems in challenging environments, lidar (LiDAR) and wheel speedometers (OD) are often used as auxiliary sensors for navigation and positioning in autonomous vehicles. However, all sensors in this fusion framework, including GNSS itself, will introduce transient errors under specific environmental boundary conditions, causing transient instability in the integrated navigation system. Specifically: when GNSS enters or exits urban canyons, tunnels, or overpasses, its positioning solution will experience drastic jumps or drifts due to the rapid changes in satellite geometry and the instantaneous enhancement of multipath effects; when LiDAR enters or exits sparsely characterized areas or encounters severe weather, its point cloud matching will temporarily fail, introducing transient pose jumps; when wheel speedometers pass over slippery roads, speed bumps, or potholes, the instantaneous slippage or bouncing of the wheels will produce pulse-like speed errors.

[0004] To address outlier issues in multi-sensor fusion, most current mainstream methods rely on passive response mechanisms based on post-event detection and adjustment. Whether within the Kalman filter (KF) or factor graph optimization (FGO) framework, the core of these methods lies in the statistical testing of measurement residuals, such as the chi-square test and M-estimation. However, both data detection and robust estimation suffer from a fundamental delay in these mainstream passive response mechanisms: they can only trigger the adjustment mechanism after receiving an anomalous measurement data point and calculating its associated residual. This means that the anomalous data is only detected after it has already negatively impacted state prediction. Especially when environmental states undergo drastic changes and trigger transient instability, this delayed response often makes it difficult to avoid significant losses in positioning accuracy due to the initial injection of outliers.

[0005] Furthermore, while end-to-end large-scale navigation solutions demonstrate a certain degree of intelligence, they typically function as a black box replacing the entire navigation stack, lacking physical interpretability and making it difficult to guarantee the reliability and transparency of the system in safety-critical tasks.

[0006] Chinese invention patent CN121932988A discloses a method and related apparatus for simultaneous localization and mapping (SLAM) using a multi-sensor fusion system. The method includes: extracting scene semantic information through a multimodal large model; generating weight matrices for visual odometry relative to IMU odometry and LiDAR odometry relative to IMU odometry using a visual conditional deep neural network and a point cloud conditional deep neural network, respectively; optimizing these matrices by combining visual odometry factors, LiDAR odometry factors, and IMU pre-integration factors; and finally outputting system state estimates and map point coordinate estimates. This scheme can adaptively adjust the fusion weights of visual-LiDAR-Inertial Odometry (SLAM) using scene semantic information, improving the robustness of the SLAM system in unstructured environments. However, the aforementioned solutions primarily target LiDAR-Vision-Inertial SLAM scenarios, adjusting the weight matrices of visual and LiDAR odometry relative to the IMU odometry. The optimization objectives simultaneously include system state and map point coordinates. They fail to address the transient measurement degradation issues arising from the switching of multiple navigation sensors (GNSS, LiDAR, wheel speedometers, etc.) in autonomous vehicle integrated navigation at environmental transition boundaries such as tunnels, tree-lined roads, sparsely characterized areas, speed bumps, or potholes. Therefore, existing technologies still lack a feedforward navigation method specifically designed for transient degradation scenarios in autonomous vehicle integrated navigation, capable of proactively adjusting the weights of various navigation measurement factors before abnormal measurements significantly impact state estimation.

[0007] In summary, there is currently a lack of a feedforward navigation method and device for autonomous vehicles based on visual language models to solve or partially solve the above problems. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a feedforward navigation method and device for autonomous vehicles based on a visual language model. This addresses or partially addresses the problem of decreased positioning accuracy in autonomous vehicle integrated navigation systems caused by transient degradation of navigation sensors such as GNSS, LiDAR, and wheel speedometers at environmental transition boundaries, such as tunnels, tree-lined roads, sparsely characterized areas, speed bumps, and potholes. The method outputs structured environmental semantic prediction results through a visual language model, generates physical reliability scores for each sensor using a semantic-physical parameter mapping model, and optimizes the noise covariance matrix of the corresponding measurement factors in the feedforward adjustment factor graph model before abnormal measurements significantly affect state estimation. This improves the transient positioning accuracy, robustness, and interpretability of the autonomous vehicle integrated navigation system.

[0009] The objective of this invention can be achieved through the following technical solutions: According to one aspect of the present invention, a feedforward navigation method for autonomous vehicles based on a visual language model is provided, the method comprising: S1. Acquire measurement data from the vehicle-mounted multi-source sensor, and a forward-looking camera image stream synchronized with the vehicle-mounted multi-source sensor in time; S2. Input the image stream into the fine-tuned visual language model, extract environmental features using a dual-frame visual input strategy, perform feedforward environmental semantic prediction based on the environmental features, and output the prediction result including semantic confidence. S3. Based on the measurement data obtained from the vehicle-mounted multi-source sensors, the prediction results are input into a pre-trained semantic-physical parameter mapping model to obtain the physical reliability score of each sensor. S4. Construct a dynamic weighting function based on the physical reliability score, and adjust the noise covariance matrix of the corresponding sensor measurement factor in the factor graph optimization model; S5. Based on the adjusted noise covariance matrix, solve the nonlinear least squares problem of the factor graph optimization model, and output the optimal position, speed and attitude estimates of the autonomous vehicle in the navigation coordinate system to realize feedforward navigation of the autonomous vehicle.

[0010] As a preferred technical solution, the process of outputting the prediction result of semantic confidence includes: A dual-frame visual input strategy is adopted to provide visual input to the fine-tuned visual language model. The visual input includes the current image frame and the previous image frame. Based on the visual input and guiding prompts, a structured JSON object is output as the prediction result using a fine-tuned visual language model. The structured JSON object includes the state category, reason, and confidence score.

[0011] As a preferred technical solution, the semantic-physical parameter mapping model includes a multilayer perceptron regression head, and the training process of the semantic-physical parameter mapping model includes: A reference state is generated using post-processing. At each timestamp of the training set, the raw measurement data of each sensor is compared with the reference state at the current moment to calculate the true measurement residual value. The real measurement residuals are converted into normalized true reliability scores using a robust cost function. A multilayer perceptron regression head is constructed and trained with the goal of minimizing the mean square error between the predicted physical reliability score and the true reliability score, resulting in a trained semantic-physical parameter mapping model.

[0012] As a preferred technical solution, the dynamic weighting function is expressed as follows: in, For semantic confidence, The preset confidence threshold, To control the coefficient of the weight descent slope, different confidence thresholds and weight descent slope coefficients are set for the Global Navigation Satellite System, LiDAR, and wheel speedometer, respectively, based on the sensitivity of different sensors under different degradation scenarios, to achieve differentiated weight adjustment.

[0013] As a preferred technical solution, the adjusted noise covariance matrix Represented as: in, The original measurement noise covariance matrix, For the first Semantic confidence of each measurement factor.

[0014] As a preferred technical solution, the measurement data from the vehicle-mounted multi-source sensors includes inertial measurement unit measurement data, global navigation satellite system measurement data, lidar measurement data, and wheel speedometer measurement data.

[0015] As a preferred technical solution, the objective function of the factor graph optimization model is expressed as: in, Let the state variable be to be estimated. For the residual terms of the inertial measurement unit, For the residuals of the global navigation satellite system, For the residual terms of the lidar, For the residual term of the wheel speed gauge, Represents the robust cost function, To optimize the time index within the window, To optimize the index of the last moment within the window, For the a priori residual term, Let Jacobian matrix be the matrix corresponding to the prior constraints. Let covariance be the matrix corresponding to the prior residuals. This is the covariance matrix corresponding to the pre-integrated residuals of the inertial measurement unit. These are the noise covariance matrices of the Global Navigation Satellite System, LiDAR, and Wheel Speed ​​Meter, respectively, adjusted by a dynamic weighting function. , , The first Measurements from the global navigation satellite system, lidar, and wheel speedometer at all times. , , , , These represent the weighted squared residuals corresponding to the prior residuals, inertial measurement unit pre-integration residuals, global navigation satellite system residuals, lidar residuals, and wheel speedometer residuals, respectively.

[0016] As a preferred technical solution, the residual term of the inertial measurement unit is expressed as follows: in, From Time's up The inertial measurement unit pre-integrated measurement quantity at time; , , These are the position pre-integration residual, velocity pre-integration residual, and attitude pre-integration residual, respectively. and These are the zero-bias residuals of the accelerometer and the zero-bias residuals of the gyroscope, respectively. and These are the zero bias values ​​for the accelerometer and gyroscope, respectively. From the navigation coordinate system to the first The direction cosine matrix of the carrier coordinate system at any given time; and The first Time and the The position of the carrier in the navigation coordinate system at any given time; and The first Time and the The speed of the carrier in the navigation coordinate system at any given moment; and The first Time and the The attitude of the carrier in the navigation coordinate system at any given moment; This is the gravitational acceleration vector in the navigation coordinate system; For the first Time to the The time interval between moments; , and These are the pre-integrated values ​​of the position, velocity, and attitude of the inertial measurement unit within the time interval, respectively. This indicates the extraction of the vector portion of the quaternion; , , , The first Time and the The zero bias of the accelerometer and the zero bias of the gyroscope at any given moment.

[0017] As a preferred technical solution, the visual language model runs asynchronously at a lower frequency than the factor graph optimization model to output environmental semantic prediction results within a future time window, and the factor graph optimization model updates the measurement noise covariance matrix using the most recent environmental semantic prediction results in a high-frequency manner.

[0018] According to another aspect of the present invention, a feedforward navigation device for autonomous vehicles based on a visual language model is provided, the device being used to perform the feedforward navigation method for autonomous vehicles based on a visual language model as described above, the device comprising: The sensing module is used to simultaneously acquire measurement data from the inertial measurement unit, global navigation satellite system, lidar and wheel speedometer, as well as image data from the forward-looking camera; The VLM prediction module has a built-in fine-tuned visual language model for processing image data and outputting semantic confidence scores for sensor performance degradation. The parameter mapping module is used to convert the semantic confidence into a physical reliability score using a pre-trained mapping network; An active fusion processor is used to update the noise covariance matrix in the factor graph in a feedforward manner based on the physical reliability score and to calculate the optimal state estimate of the carrier.

[0019] Compared with the prior art, the present invention has at least one of the following beneficial effects: (1) Proactive: This invention uses a visual language model to perform feedforward environmental semantic prediction and constructs a dynamic weight function. The weight is adjusted in advance before the vehicle enters the abnormal area, which completely eliminates the transient instability problem caused by the delayed detection of the traditional passive method. It significantly improves the positioning accuracy at the boundary of environmental switching and realizes the paradigm shift from the traditional passive response to the proactive adjustment.

[0020] (2) Smoothness and robustness: This invention constructs a dynamic weight function based on semantic confidence to adjust the noise covariance matrix of the corresponding sensor measurement factors in the graph optimization model, thereby achieving smooth and continuous adjustment of sensor weights. This avoids the problem of state estimation oscillation caused by traditional hard threshold switching, and achieves the technical effect of effectively suppressing error divergence and maintaining high-precision positioning output in complex scenarios such as signal obstruction, feature sparsity and road bumps.

[0021] (3) Interpretability: The present invention adopts a modular design, uses a visual language model for semantic prediction, uses a semantic-physical parameter mapping model to obtain physical reliability scores, constructs a dynamic weight function to adjust the noise parameters of the physical model, and finally solves the factor graph optimization model output position results. This design retains the intelligence of the end-to-end large model, and maintains the interpretability, reliability and boundary constraints of the factor graph, which is more secure than the pure end-to-end solution.

[0022] (4) Not dependent on high-precision maps: The present invention is based entirely on real-time perceived visual semantic information for feedforward adjustment, without relying on pre-built high-precision maps, and achieves the technical effect of effectively dealing with transient and non-static environmental changes that high-precision maps cannot predict, and has stronger environmental adaptability. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall process of the method of the present invention; Figure 2 This is a schematic diagram of the factor graph structure of the active sensor fusion method based on VLM enhancement in this invention; Figure 3 This is a schematic diagram of the dynamic weight function curve of the feedforward adaptive fusion mechanism in this invention; Figure 4 This is a schematic diagram of the structure of the device of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] Example 1 To address the problems existing in the aforementioned prior art, this embodiment provides a feedforward navigation method for autonomous vehicles based on a visual language model, such as... Figure 1 As shown, it includes the following steps: S1. Acquire measurement data from the vehicle-mounted multi-source sensors, as well as a forward-looking camera image stream that is time-synchronized with the vehicle-mounted multi-source sensors.

[0026] The onboard computing platform synchronously acquires measurement data from multiple sensors via sensor interfaces. These sensors include an inertial measurement unit (IMU), a global navigation satellite system (GNSS), a lidar radar (LiDAR), and wheel speed sensors (OD). Simultaneously, a high-definition image stream, strictly time-aligned with the sensor data, is acquired via an onboard forward-facing camera. In this embodiment, the IMU output frequency is set to 100Hz, providing angular and linear velocity measurements; the output frequencies of the GNSS, LiDAR, OD, and camera are set to 10Hz.

[0027] S2. Input the image stream into the fine-tuned visual language model, extract environmental features using a dual-frame visual input strategy, perform feedforward environmental semantic prediction based on the environmental features, and output the prediction results including semantic confidence.

[0028] This step utilizes a fine-tuned Visual Language Model (VLM) to analyze the image stream. To enhance the model's ability to perceive dynamic changes in the environment, a dual-frame visual input strategy is employed.

[0029] At any moment Visual input provided to the model Defined as: in, For the current image frame, This is the image frame from the previous moment.

[0030] The output of the visual language model A structured JSON object, including state category, reason, and confidence score, represented as: in, This represents the model parameters after fine-tuning. These are guiding prompts.

[0031] The specific structure is as follows: in, Predictions of environmental conditions related to sky obstruction or satellite visibility; The prediction results are for road conditions such as road surface adhesion, road surface smoothness, speed bumps, or potholes. This section presents environmental state prediction results related to LiDAR availability. Each state prediction result includes a state category, a cause, and a confidence score. The state category characterizes the type of environmental degradation within the current or future time window, the cause explains the visual semantic basis that triggers the state judgment, and the confidence score quantifies the certainty of the visual language model for that state prediction result.

[0032] Each state This includes the state category, the cause, and a confidence score, for example: Among them, confidence score The certainty of VLM's own predictions was quantified.

[0033] Compared to object detection models based on predefined categories, visual language models not only identify explicit targets such as tunnels, speed bumps, and sparsely characterized regions, but also combine factors like road structure, sky occlusion, road texture, road boundaries, obstacle occlusion, and scene context to generate structured semantic predictions for navigation sensor degradation risks. Because visual language models possess cross-category semantic understanding and combined scene generalization capabilities, they can predict sensor degradation risks that are not predefined or are caused by multiple environmental factors. This reduces the system complexity associated with designing multiple independent detection models for different degradation scenarios, improving the adaptability of feedforward navigation methods in long-tailed complex environments.

[0034] In a preferred embodiment, the visual language model does not run synchronously with the factor graph optimization model at the same frequency. Instead, it outputs environmental semantic prediction results for a future preset time window in a low-frequency asynchronous manner. The factor graph optimization model performs state estimation in a high-frequency manner and calls the most recent environmental semantic prediction result output by the visual language model during each optimization to update the noise covariance matrix of the corresponding measurement factor. By decoupling low-frequency semantic prediction from high-frequency state estimation, the impact of visual language model inference latency on the main navigation loop can be reduced, while ensuring the real-time performance of navigation state estimation.

[0035] S3. Based on the measurement data acquired from the vehicle-mounted multi-source sensors, the prediction results are input into the pre-trained semantic-physical parameter mapping model to obtain the physical reliability score of each sensor.

[0036] To transform the abstract semantic confidence output in S2 into the quantitative physical parameters required by the navigation filter, this step introduces a pre-trained semantic-physical parameter mapping model. This model is a multilayer perceptron (MLP) regression head. The offline training process of the mapping model is as follows: 1. Generate realistic reliability labels: On the complete training dataset, we first run a high-precision post-processing algorithm to generate a set of reference states. Subsequently, at each timestamp of the training set... The raw measurement values ​​of each sensor Reference state at that moment By comparison, the true measurement residuals can be calculated. : in, This is the observation model for the corresponding sensor.

[0037] 2. Label Normalization: The robust cost function is used to convert the actual measurement residuals into normalized true reliability scores. : in, These are preset hyperparameters used to control the sensitivity of residuals to reliability scores.

[0038] 3. Supervised Training: Constructing a Multilayer Perceptron Regression Head To minimize the difference between the predicted value and the true reliability score The training is performed using the mean squared error between the two values ​​as the target, and the loss function is: in, Semantic features extracted from VLM For trainable parameters, N This represents the total number of training samples.

[0039] S4. Construct a dynamic weighting function based on the physical reliability score, and adjust the noise covariance matrix of the corresponding sensor measurement factors in the factor graph optimization model.

[0040] This embodiment employs a factor graph optimization framework to achieve feedforward active adaptive fusion and optimal state estimation through multi-source information fusion. The factor graph structure is as follows: Figure 2 As shown, this includes prior factors, IMU pre-integration factors, and measurement factors such as GNSS, LiDAR, and OD. The system state variables are defined as follows: in , Indicates the first Vehicle state variables at time t. Indicates the location of the carrier. Indicates the speed of the carrier, Indicates the attitude of the carrier. and This represents the zero bias of the IMU. It should be noted that the factor graph optimization model in this embodiment is geared towards the integrated navigation state estimation of autonomous vehicles. The variables to be estimated mainly include the vehicle's position, velocity, attitude in the navigation coordinate system, and the zero bias of the inertial measurement unit, rather than the coordinates of environmental map points. Therefore, this embodiment differs from the technical approach of jointly optimizing system state and map point coordinates in simultaneous localization and mapping methods, and is more suitable for application scenarios where autonomous vehicles perform continuous navigation and localization in existing road environments.

[0041] Specifically, such as Figure 2 As shown, multiple fusion status nodes The state sequence is constructed chronologically, with adjacent fused state nodes connected via IMU pre-integration factors to constrain the relative motion relationships between adjacent time points. Each fused state node is also connected to GNSS factors, wheel speed gauge factors, and lidar factors to introduce absolute position constraints, wheel speed constraints, and lidar matching constraints. LiDAR frame nodes and lidar state nodes characterize the constraint relationship between lidar measurements and vehicle states. The VLM module outputs environmental semantic confidence based on the forward-view image and applies this semantic confidence to the noise covariance matrix of the corresponding measurement factors, thereby achieving feedforward weight adjustment of measurement factors such as GNSS, LiDAR, and OD. Prior states are used to provide initial constraints for the optimization window. Thus, the factor graph maintains the interpretable structure of multi-source sensor fusion optimization while introducing the active adjustment effect of visual semantic information on sensor reliability.

[0042] To achieve proactive robust navigation, this step introduces a feedforward adaptive fusion mechanism to construct a dynamic weight function. : in, The semantic confidence obtained from the mapping. The preset confidence threshold, A coefficient used to control the slope of the weight decrease.

[0043] like Figure 3 As shown, Figure 3 This is a schematic diagram of the dynamic weight function curve of the feedforward adaptive fusion mechanism in this invention, wherein... For the semantic confidence of wheel speed measurement, For the semantic confidence of lidar, Confidence level of the inertial measurement unit. Confidence level of global navigation satellite systems.

[0044] The dynamic weighting function is used to dynamically adjust the noise covariance matrix of the measurement factors: in, The original measurement noise covariance matrix, For the first Semantic confidence of each measurement factor.

[0045] When predicting a decrease in sensor reliability, by reducing Make the covariance matrix The value is increased, thereby reducing the weight of the sensor in the fusion.

[0046] Finally, the nonlinear least squares objective function of the factor graph optimization model is constructed as follows: in, Let the state variable be to be estimated. For the residual terms of the inertial measurement unit, For the residuals of the global navigation satellite system, For the residual terms of the lidar, For the residual term of the wheel speed gauge, Represents the robust cost function, To optimize the time index within the window, To optimize the index of the last moment within the window, For the a priori residual term, Let Jacobian matrix be the matrix corresponding to the prior constraints. Let covariance be the matrix corresponding to the prior residuals. This is the covariance matrix corresponding to the pre-integrated residuals of the inertial measurement unit. These are the noise covariance matrices of the Global Navigation Satellite System, LiDAR, and Wheel Speed ​​Meter, respectively, adjusted by a dynamic weighting function. , , The first Measurements from the global navigation satellite system, lidar, and wheel speedometer at all times. , , , , These represent the weighted squared residuals corresponding to the prior residuals, inertial measurement unit pre-integration residuals, global navigation satellite system residuals, lidar residuals, and wheel speedometer residuals, respectively.

[0047] IMU factor residuals Specifically, it is expressed as follows: in, From Time's up The inertial measurement unit pre-integrated measurement quantity at time; , , These are the position pre-integration residual, velocity pre-integration residual, and attitude pre-integration residual, respectively. and These are the zero-bias residuals of the accelerometer and the zero-bias residuals of the gyroscope, respectively. and These are the zero bias values ​​for the accelerometer and gyroscope, respectively. From the navigation coordinate system to the first The direction cosine matrix of the carrier coordinate system at any given time; and The first Time and the The position of the carrier in the navigation coordinate system at any given time; and The first Time and the The speed of the carrier in the navigation coordinate system at any given moment; and The first Time and the The attitude of the carrier in the navigation coordinate system at any given moment; This is the gravitational acceleration vector in the navigation coordinate system; For the first Time to the The time interval between moments; , and These are the pre-integrated values ​​of the position, velocity, and attitude of the inertial measurement unit within the time interval; This indicates the extraction of the vector portion of the quaternion; , , , The first Time and the The zero bias of the accelerometer and the zero bias of the gyroscope at any given moment.

[0048] S5. Based on the adjusted noise covariance matrix, solve the nonlinear least squares objective function of the factor graph optimization model to obtain the optimal state estimate of the carrier. It outputs the optimal position, speed, and attitude estimates of the autonomous vehicle in the navigation coordinate system, realizing feedforward navigation for autonomous vehicles.

[0049] Example 2 Regarding the aforementioned embodiment of the feedforward navigation method for autonomous vehicles based on a visual language model, this embodiment provides a feedforward navigation device for autonomous vehicles based on a visual language model, used to execute the aforementioned feedforward navigation method for autonomous vehicles based on a visual language model, such as... Figure 4As shown, the device includes: The data acquisition module 201 is used to acquire IMU measurement data of the carrier during dynamic movement, as well as GNSS, LiDAR, wheel speed meter data and forward-looking camera image stream that are synchronized with the IMU data in time. The VLM prediction module 202 is connected to the data acquisition module. It has a built-in fine-tuned visual language model for prospective analysis of image streams and outputs environmental semantic prediction results and confidence levels for specific sensors. The parameter mapping module 203, connected to the VLM prediction module, stores pre-trained MLP network parameters and is used to map the semantic confidence of the VLM output to the sensor reliability score at the physical layer. The active fusion processor 204, connected to the parameter mapping module, runs a factor graph optimization algorithm to adjust the measurement noise covariance matrix in a feedforward manner based on the reliability score and solve for the optimal state estimate.

[0050] Example 3 In this embodiment, the experimental vehicle is equipped with an inertial measurement unit (IMU), a global navigation satellite system (GNSS) receiver, a lidar, a wheel speedometer, and a forward-looking camera. The IMU outputs at a frequency of 100Hz, while the GNSS, lidar, wheel speedometer, and forward-looking camera all output at 10Hz. The lidar is a 16-line lidar with a GNSS positioning error of 3m and a wheel speedometer measurement error of 0.5%. To obtain a reference trajectory, the raw sensor data is post-processed, and the post-processed kinematics and forward / backward smoothing are combined to generate a reference positioning result. The experiment selects three typical degradation scenarios for verification: tunnel obstruction, sparse features on tree-lined roads, and continuous speed bumps.

[0051] In this table, FGO represents the standard factor graph optimization method without robust processing and feedforward adjustment; FGO-R represents a method that introduces a passive robust fusion mechanism based on measurement residuals on the basis of the standard factor graph optimization method; FGO-M represents a robust factor graph optimization method that uses M-estimated robust cost function; and the method of this invention represents a navigation method that uses visual language model feedforward prediction and dynamic weight adjustment. In the experiments, the average level position error and maximum positioning error were used as evaluation indicators; the smaller the values ​​of both, the better the positioning accuracy and transient robustness. Table 1 shows the comparison results of position errors of different methods in typical degradation scenarios. Each data unit in the table represents the average level position error / maximum positioning error, in meters (m).

[0052] Table 1 Comparison of position errors of different methods in typical degradation scenarios As shown in Table 1, under the three typical degradation scenarios, the average position error and maximum positioning error of the method of this invention are lower than those of the comparative methods. Specifically, in the tunnel occlusion scenario, the method of this invention can reduce the weight of the global navigation satellite system measurement factor in advance before the vehicle enters the tunnel, thereby suppressing transient errors at the occlusion boundary; in the tree-lined road feature sparse scenario, the method of this invention can reduce the weight of the lidar measurement factor in advance, thereby reducing the impact of point cloud matching degradation on state estimation; in the continuous speed bump scenario, the method of this invention can reduce the weight of the wheel speed meter measurement factor in advance, thereby suppressing the impact of speed anomalies caused by road bumps on the positioning results. Experimental results show that the method of this invention can perform feedforward weight adjustment before abnormal measurements enter the factor map optimization model, effectively reducing transient positioning errors at the environment switching boundary and improving the robustness of the autonomous vehicle integrated navigation system.

[0053] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A feedforward navigation method for autonomous vehicles based on a visual language model, characterized in that, The method includes: S1. Acquire measurement data from the vehicle-mounted multi-source sensor, and a forward-looking camera image stream synchronized with the vehicle-mounted multi-source sensor in time; S2. Input the image stream into the fine-tuned visual language model, extract environmental features using a dual-frame visual input strategy, perform feedforward environmental semantic prediction based on the environmental features, and output the prediction result including semantic confidence. S3. Based on the measurement data obtained from the vehicle-mounted multi-source sensors, the prediction results are input into a pre-trained semantic-physical parameter mapping model to obtain the physical reliability score of each sensor. S4. Construct a dynamic weighting function based on the physical reliability score, and adjust the noise covariance matrix of the corresponding sensor measurement factor in the factor graph optimization model; S5. Based on the adjusted noise covariance matrix, solve the nonlinear least squares problem of the factor graph optimization model, and output the optimal position, speed and attitude estimates of the autonomous vehicle in the navigation coordinate system to realize feedforward navigation of the autonomous vehicle.

2. The feedforward navigation method for autonomous vehicles based on a visual language model according to claim 1, characterized in that, The process of outputting the prediction results of semantic confidence includes: A dual-frame visual input strategy is adopted to provide visual input to the fine-tuned visual language model. The visual input includes the current image frame and the previous image frame. Based on the visual input and guiding prompts, a structured JSON object is output as the prediction result using a fine-tuned visual language model. The structured JSON object includes the state category, reason, and confidence score.

3. The feedforward navigation method for autonomous vehicles based on a visual language model according to claim 1, characterized in that, The semantic-physical parameter mapping model includes a multilayer perceptron regression head, and the training process of the semantic-physical parameter mapping model includes: A reference state is generated using post-processing. At each timestamp of the training set, the raw measurement data of each sensor is compared with the reference state at the current moment to calculate the true measurement residual value. The real measurement residuals are converted into normalized true reliability scores using a robust cost function. A multilayer perceptron regression head is constructed and trained with the goal of minimizing the mean square error between the predicted physical reliability score and the true reliability score, resulting in a trained semantic-physical parameter mapping model.

4. The feedforward navigation method for autonomous vehicles based on a visual language model according to claim 1, characterized in that, The dynamic weighting function is expressed as follows: in, For semantic confidence, The preset confidence threshold, To control the coefficient of the weight descent slope, different confidence thresholds and weight descent slope coefficients are set for the Global Navigation Satellite System, LiDAR, and wheel speedometer, respectively, based on the sensitivity of different sensors under different degradation scenarios, to achieve differentiated weight adjustment.

5. The feedforward navigation method for autonomous vehicles based on a visual language model according to claim 4, characterized in that, The adjusted noise covariance matrix Represented as: in, The original measurement noise covariance matrix, For the first Semantic confidence of each measurement factor.

6. The feedforward navigation method for autonomous vehicles based on a visual language model according to claim 1, characterized in that, The measurement data from the vehicle-mounted multi-source sensors include inertial measurement unit measurement data, global navigation satellite system measurement data, lidar measurement data, and wheel speedometer measurement data.

7. The feedforward navigation method for autonomous vehicles based on a visual language model according to claim 1, characterized in that, The objective function of the factor graph optimization model is expressed as: in, For the state variables to be estimated, For the residual terms of the inertial measurement unit, For the residuals of the global navigation satellite system, For the residual terms of the lidar, For the residual term of the wheel speed gauge, Represents the robust cost function. To optimize the time index within the window, To optimize the index of the last moment within the window, For the a priori residual term, Let Jacobian matrix be the matrix corresponding to the prior constraints. Let covariance be the matrix corresponding to the prior residuals. This is the covariance matrix corresponding to the pre-integrated residuals of the inertial measurement unit. These are the noise covariance matrices of the Global Navigation Satellite System, LiDAR, and Wheel Speed ​​Meter, respectively, adjusted by a dynamic weighting function. , , The first Measurements from the global navigation satellite system, lidar, and wheel speedometer at all times. , , , , These represent the weighted squared residuals corresponding to the prior residuals, inertial measurement unit pre-integration residuals, global navigation satellite system residuals, lidar residuals, and wheel speedometer residuals, respectively.

8. The feedforward navigation method for autonomous vehicles based on a visual language model according to claim 7, characterized in that, The residual term of the inertial measurement unit is expressed as: in, From Time's up The inertial measurement unit pre-integrated measurement quantity at time; , , These are the position pre-integration residual, velocity pre-integration residual, and attitude pre-integration residual, respectively. and These are the zero-bias residuals of the accelerometer and the zero-bias residuals of the gyroscope, respectively. and These are the zero bias values ​​for the accelerometer and gyroscope, respectively. From the navigation coordinate system to the first The direction cosine matrix of the carrier coordinate system at any given time; and The first Time and the The position of the carrier in the navigation coordinate system at any given time; and The first Time and the The speed of the carrier in the navigation coordinate system at any given moment; and The first Time and the The attitude of the carrier in the navigation coordinate system at any given moment; This is the gravitational acceleration vector in the navigation coordinate system; For the first Time to the The time interval between moments; , and These are the pre-integrated values ​​of the position, velocity, and attitude of the inertial measurement unit within the time interval, respectively. This indicates the extraction of the vector portion of the quaternion; , , , The first Time and the The zero bias of the accelerometer and the zero bias of the gyroscope at any given moment.

9. The feedforward navigation method for autonomous vehicles based on a visual language model according to claim 1, characterized in that, The visual language model runs asynchronously at a lower frequency than the factor graph optimization model to output environmental semantic prediction results within a future time window. The factor graph optimization model updates the measurement noise covariance matrix at a high frequency using the most recent environmental semantic prediction results.

10. A feedforward navigation device for autonomous vehicles based on a visual language model, characterized in that, The apparatus is used to execute the feedforward navigation method for autonomous vehicles based on a visual language model as described in any one of claims 1-9, the apparatus comprising: The sensing module is used to simultaneously acquire measurement data from the inertial measurement unit, global navigation satellite system, lidar and wheel speedometer, as well as image data from the forward-looking camera; The VLM prediction module has a built-in fine-tuned visual language model for processing image data and outputting semantic confidence scores for sensor performance degradation. The parameter mapping module is used to convert the semantic confidence into a physical reliability score using a pre-trained mapping network; An active fusion processor is used to update the noise covariance matrix in the factor graph in a feedforward manner based on the physical reliability score and to calculate the optimal state estimate of the carrier.

Images