An unmanned vehicle positioning method and system based on multi-source information fusion

Abstract

This invention provides a method and system for autonomous vehicle positioning based on multi-source information fusion, relating to the fields of autonomous driving, computer vision, and integrated navigation. Addressing the issues of visual odometry failure and insufficient robustness of multi-source fusion in weak or no GNSS environments, this invention maps multiple fisheye images to a ground coordinate system via inverse perspective transformation to generate a panoramic overhead view. Ground texture features are extracted from this overhead view and inter-frame matching is performed. Vehicle motion is estimated using a two-dimensional planar rigid body model, and a matching quality index based on the number of feature-matched inliers and reprojection errors is constructed. Pre-integration is performed using inertial measurement unit data, and the visual observation weights are adaptively adjusted according to the matching quality within an extended Kalman filter framework to achieve dynamic fusion of multi-source information. Error correction is performed when GNSS is available, outputting high-precision vehicle position, velocity, and attitude information. This invention improves the continuity and stability of positioning in complex scenarios.

Description

Technical Field

[0001] This invention relates to the fields of autonomous driving, computer vision and integrated navigation technology, and in particular to a method and system for locating unmanned vehicles based on multi-source information fusion. Background Technology

[0002] With the rapid development of the intelligent connected vehicle industry, high-precision positioning has become a core capability of autonomous driving technology. Currently, autonomous vehicle positioning mainly relies on a combined navigation scheme using Global Navigation Satellite System (GNSS / GPS) and inertial navigation. However, in underground parking lots, tunnels, urban canyons, and areas obstructed by large buildings, GNSS signals are easily affected by obstruction, multipath effects, and non-line-of-sight propagation, resulting in a significant decrease in positioning accuracy, or even complete failure. Positioning schemes relying solely on GNSS are insufficient to meet the continuous and stable positioning requirements of these scenarios.

[0003] To address the positioning challenges in GNSS-constrained environments, Simultaneous Localization and Mapping (SLAM) technology has been extensively studied. Existing SLAM technologies are mainly divided into two categories: LiDAR SLAM and Visual SLAM. While LiDAR SLAM technology offers advantages in accuracy and stability, the high cost of LiDAR equipment makes it difficult to meet the cost requirements for large-scale commercialization; furthermore, LiDAR cannot acquire rich texture and color information, limiting its ability to perceive natural scenes.

[0004] In contrast, visual SLAM has the advantages of low hardware cost and rich environmental texture information, thus becoming an important development direction for vehicle positioning. However, existing visual SLAM solutions are mostly based on feature extraction from forward-looking perspective images, which has the following drawbacks: First, the field of view is limited and easily affected by dynamic targets such as pedestrians and vehicles; second, non-ground areas such as the sky and building facades can introduce unstable features, affecting pose estimation; third, visual matching is not robust enough in weak texture or high reflectivity environments such as white walls, glass curtain walls, and strong lighting changes; fourth, relying solely on visual odometry is prone to cumulative drift, resulting in insufficient long-term operational stability.

[0005] On the other hand, although existing visual-inertial fusion methods introduce IMU to improve short-term continuity, most methods still use a fixed visual measurement noise model when the visual observation quality deteriorates. They cannot adaptively adjust the visual observation weights according to the matching reliability of the current frame. Therefore, filtering divergence or positioning jumps may still occur under complex working conditions.

[0006] Therefore, there is an urgent need to propose a multi-source information fusion positioning method that is suitable for weak or no GNSS scenarios, has low hardware costs, high positioning accuracy, and strong operational continuity, so as to meet the practical application needs of unmanned vehicles in closed or semi-closed scenarios. Summary of the Invention

[0007] This invention proposes a positioning method and system for autonomous vehicles based on multi-source information fusion, in order to solve the problems of positioning failure of existing autonomous vehicles in weak or no GNSS scenarios, insufficient positioning accuracy and robustness of single sensors, and imperfect fusion mechanism of four-view overhead images and inertial navigation.

[0008] In a first aspect, the present invention provides a method for locating an autonomous vehicle based on multi-source information fusion, the method comprising the following steps:

[0009] Step S1: Acquire fisheye images of the vehicle in the front, rear, left, and right directions, as well as acceleration and angular velocity data output by the inertial measurement unit (IMU), and perform time alignment and spatial coordinate unification between the fisheye images and the data output by the inertial measurement unit.

[0010] Step S2: Perform distortion correction and inverse perspective transformation on the fisheye image, map it to a unified ground coordinate system, and generate a panoramic view of the vehicle from all angles.

[0011] Step S3: Extract ground texture feature points on the overhead panoramic image, perform feature matching on adjacent frames, use the random sampling consensus algorithm to estimate the relative displacement and heading change of the vehicle in the two-dimensional plane, and calculate the matching quality index characterizing the reliability of visual observation in the current frame based on the matching results.

[0012] Step S4: Pre-integrate the acceleration data and angular velocity data to obtain the state prediction quantity;

[0013] Step S5: Construct an extended Kalman filter model, using the state prediction as the predicted value, and the relative displacement and heading changes as observations for fusion and updating, and dynamically adjust the weight of the observations in the fusion and updating according to the matching quality index;

[0014] Step S6: When the global navigation satellite system signal is available, absolute position observation is introduced to correct the error of the fusion result, and the vehicle's position, speed and attitude information are output.

[0015] Furthermore, the specific method for acquiring fisheye images of the vehicle in the front, rear, left, and right directions, as well as acceleration and angular velocity data output by the inertial measurement unit, in step S1, and for performing time alignment and spatial coordinate unification between the fisheye images and the data output by the inertial measurement unit, includes:

[0016] For the A fisheye camera is used. Let the coordinates of a spatial point in the coordinate system of this camera be... Its optical axis angle is ;in These respectively represent the three-dimensional coordinate vector at the th... Coordinate components in the X, Y, and Z axes of a camera coordinate system;

[0017] The polar radius was modeled using the Kannala-Brandt fisheye imaging model:

[0018]

[0019] in, Indicates the first A fisheye camera at the angle The imaging polarity below, For the first Distortion parameters of a fisheye camera;

[0020] Further define the normalized directional components for:

[0021]

[0022] The corresponding pixel coordinates are:

[0023]

[0024] in, These represent the spatial points at the th, . The horizontal and vertical pixel coordinates of a fisheye image For focal length parameters, Principal point coordinates;

[0025] The extrinsic parameter matrices of each fisheye camera relative to the vehicle coordinate system can be obtained using a checkerboard or calibration board.

[0026]

[0027] in, Let be a rotation matrix. It is a translation vector;

[0028] Furthermore, the rigid body transformation relationship between the inertial measurement unit (IMU) coordinate system and the vehicle coordinate system is obtained through joint calibration, unifying all sensors under the vehicle coordinate system.

[0029] Furthermore, the specific method for performing distortion correction and inverse perspective transformation on the fisheye image in step S2, mapping it to a unified ground coordinate system, and generating a panoramic view of the vehicle from all angles includes:

[0030] Let the ground point in the vehicle coordinate system be:

[0031]

[0032] The ground satisfies , These are the coordinates of the ground point in the vehicle coordinate system. Axis (usually representing the longitudinal and longitudinal coordinates of a vehicle) and Axis (usually representing the left and right lateral coordinates of a vehicle);

[0033] The ground point is at the The homogeneous coordinates in the camera coordinate system can be expressed as:

[0034]

[0035] in, , indicating the vehicle coordinate system to the first Homogeneous coordinate transformation matrix for each camera coordinate system;

[0036] Will Substituting the above fisheye imaging model, the ground points can be obtained. Pixel coordinates in the original fisheye image Therefore, a mapping relationship is formed between the vehicle's ground coordinates and the pixels of each fisheye image:

[0037]

[0038] During the system initialization phase, each pixel in the target overhead image is viewed. Corresponding ground physical coordinates Perform offline mapping calculations and generate a lookup table:

[0039]

[0040] in, Indicates the first The path lookup table shows the pixel location in the image from above. The original fisheye image sampling coordinates stored at [location] They represent the first The horizontal and vertical sampled pixel coordinates corresponding to the path lookup table;

[0041] Let the first The overhead image after road mapping is The panoramic view obtained after fusing multiple images is ,in To visualize the image pixel locations from above, a weight-based smooth fusion method is used to stitch the images together within overlapping areas.

[0042]

[0043] Among them, weight Can be based on pixel point to the The distance from the center of the road camera's field of view or the distance to the stitching boundary is determined, preferably as follows:

[0044]

[0045] in, Indicates the distance, To prevent extremely small constants with a denominator of zero;

[0046] To reduce brightness discontinuities caused by differences in automatic exposure between multiple cameras, linear photometric compensation can be performed on each image stream:

[0047]

[0048] in, Indicates the first Pixel values ​​of the road image after photometric compensation. and The first Gain and bias compensation parameters for road images; after compensation and weighted fusion, a continuous and uniform panoramic view of the vehicle around the view is obtained.

[0049] Furthermore, the specific method for extracting ground texture feature points on the overhead panoramic image, performing feature matching on adjacent frames, estimating the relative displacement and heading change of the vehicle in the two-dimensional plane using the random sampling consensus algorithm, and calculating the matching quality index characterizing the reliability of the visual observation of the current frame based on the matching results in step S3 includes:

[0050] ORB or FAST feature points are extracted from the ground area of ​​the panoramic view as ground texture feature points; inter-frame feature matching between adjacent frames is performed using descriptors.

[0051] Let the first The feature point coordinates of the frame are The matched first Frame feature points are The equation of motion can be expressed as:

[0052]

[0053] in, This represents the translation along the X-axis; This represents the translation along the Y-axis; This represents the change in heading angle;

[0054] By using the Random Sample Consensus (RANSAC) algorithm to eliminate mismatches and solving the above equations, the inter-frame relative motion can be obtained. ;

[0055] Calculate matching quality index based on feature matching results :

[0056]

[0057] in, The number of interior points after RANSAC. The expected threshold for the number of interior points; The average reprojection error of the interior points; These are the weighting coefficients.

[0058] Furthermore, the specific method for pre-integrating the acceleration data and angular velocity data to obtain the state prediction quantity in step S4 includes:

[0059] The inertial measurement and navigation module provides vehicle acceleration and angular velocity measurement data at a high frequency of no less than 200 Hz;

[0060] Let the acceleration measured by the inertial measurement unit (IMU) be... angular velocity is ;exist Zero bias at time and Gaussian white noise The kinematic model is as follows:

[0061]

[0062]

[0063] in, Indicates time Rotation matrix from the carrier coordinate system to the global coordinate system; Indicates time The time derivative of the vehicle's velocity in the global coordinate system; Represents the gravity vector in the global coordinate system; Indicates time Vehicle attitude quaternion and its time derivative; symbol Represents quaternion multiplication;

[0064] In two visual keyframes and The time period between Within, relative position of pre-integral quantity Relative velocity Relative posture The calculations will be separated from the initial state:

[0065]

[0066]

[0067]

[0068] in, Indicates time Relative to keyframe Rotation matrix of the reference coordinate system; and These represent keyframes. The accelerometer and gyroscope used here are zero bias; Relative to keyframe Instantaneous attitude increment of the reference coordinate system.

[0069] Furthermore, the specific method for constructing the extended Kalman filter model in step S5, using the state prediction as the predicted value, and using the relative displacement and heading changes as observations for fusion and updating, and dynamically adjusting the weights of the observations in the fusion and updating according to the matching quality index, includes:

[0070] The system's state vector Defined as:

[0071]

[0072] in, These represent the vehicle's position, velocity, and attitude quaternions in the global frame, as well as the acceleration zero bias and gyroscope zero bias of the inertial measurement unit (IMU).

[0073] The pre-integrated quantity of the inertial measurement unit (IMU) is used as the state prediction driver; based on the IMU pre-integration model, the covariance matrix of the previous time step is... To spread:

[0074]

[0075] in, Let Jacobian matrix be the system state transition matrix. The noise Jacobian matrix, The noise covariance matrix of the inertial measurement unit (IMU);

[0076] The relative motion output from the overhead image odometer As an observation, it is used for fusion and updating; measurement residuals are constructed. :

[0077]

[0078] here This represents the error calculation on the manifold;

[0079] Matching quality index output by the overhead image odometry module Real-time calculation of adaptive measurement noise covariance matrix :

[0080]

[0081] in, Using the baseline covariance, To adjust the gain;

[0082] When visual features are insufficient, i.e. When it shrinks, The Kalman gain increases exponentially. The system will automatically reduce the weight of visual data and place more trust on high-frequency inertial measurement unit (IMU) prediction data.

[0083] Secondly, the present invention also provides an autonomous vehicle positioning system based on multi-source information fusion, including a four-eye fisheye camera module, an inertial measurement module, a time synchronization and calibration module, an overhead panoramic generation module, an overhead image odometry module, and an adaptive multi-source fusion module.

[0084] The four-eye fisheye camera module and the inertial measurement module are respectively connected to the time synchronization and calibration module;

[0085] The output of the four-eye fisheye camera module is connected to the input of the overhead panoramic generation module, and the output of the overhead panoramic generation module is connected to the input of the overhead image odometer module.

[0086] The output terminals of the overhead image odometry module and the inertial measurement module are respectively connected to the input terminal of the adaptive multi-source fusion module;

[0087] The output of the adaptive multi-source fusion module is used to output the vehicle's position, speed, and attitude information.

[0088] Furthermore, the overhead panoramic generation module includes a distortion correction unit, an inverse perspective transformation unit, an image stitching unit, and a photometric consistency correction unit; wherein, the output of the distortion correction unit is connected to the input of the inverse perspective transformation unit, the output of the inverse perspective transformation unit is connected to the input of the image stitching unit, and the output of the image stitching unit is connected to the input of the photometric consistency correction unit.

[0089] Furthermore, the adaptive multi-source fusion module includes an inertial measurement unit (IMU) pre-integration unit, an extended Kalman filter unit, and an adaptive weight adjustment unit; wherein the output terminals of the IMU pre-integration unit and the adaptive weight adjustment unit are respectively connected to the input terminal of the extended Kalman filter unit.

[0090] Compared with the prior art, the present invention has the following advantages:

[0091] 1. This invention effectively overcomes the problem of reduced positioning accuracy or failure caused by GPS signal obstruction in environments such as underground parking lots, urban canyons, and tunnels by integrating four monocular fisheye cameras and an inertial measurement unit (IMU). The system effectively avoids the technical bottleneck of traditional visual SLAM in strong light changes and textureless areas (such as white walls and glass curtain walls) by converting images into a unified top view and extracting stable ground texture features for matching, thus enhancing the positioning robustness in complex scenarios.

[0092] 2. This system primarily relies on a low-cost four-eye surround-view camera and an IMU for multi-source information fusion, replacing expensive LiDAR equipment. This low-cost hardware solution overcomes the limitations of existing LiDAR and high-cost multi-sensor systems in large-scale mass production on consumer-grade automotive platforms, making it highly suitable for large-scale commercial applications such as autonomous valet parking (AVP) and park logistics.

[0093] 3. This invention innovatively designs an adaptive multi-source fusion module, with extended Kalman filtering as its core, to deeply fuse the high-frequency predictive driving quantity provided by the IMU with the relative motion quantity output by the overhead image odometry. The system can adaptively adjust the weight of visual observations according to the image matching quality index and introduce global error correction when GNSS is available, thereby effectively suppressing the cumulative drift error caused by long-term operation of a single visual odometry, ensuring high positioning accuracy and continuity.

[0094] 4. The overhead panoramic generation module of this invention pre-calculates and stores the complex fisheye distortion correction mapping table and inverse perspective transformation lookup table during the system initialization phase. During system operation, image transformation is performed directly by looking up the tables, which significantly reduces the computational load and saves computing resources, ensuring the real-time performance of the system processing and enabling the output frequency of high-precision position, velocity, and attitude fusion positioning to remain stable at a level of no less than 100 Hz.

[0095] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0096] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, with the same or corresponding reference numerals denoteing the same or corresponding parts, wherein:

[0097] Figure 1This is a block diagram of the overall structure of the present invention;

[0098] Figure 2 The flowchart shows the vehicle-mounted real-time surround view method based on a four-channel fisheye camera according to the present invention.

[0099] Figure 3 This is a schematic diagram of the hardware installation of the present invention;

[0100] Figure 4 A schematic diagram of the original images captured by the four-eye surround-view sensor module in the front, back, left, and right directions;

[0101] Figure 5 This is a schematic diagram of the stitched result of the vehicle panoramic view output by the panoramic view generation module. Detailed Implementation

[0102] The exemplary embodiments disclosed in this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art. Unless otherwise specified, the technical means used in the embodiments are conventional means well known to those skilled in the art.

[0103] Existing technologies reveal different approaches to multi-camera fusion, but none address the problems addressed by this invention. Patent CN117830529A utilizes four fisheye cameras to generate a 540-degree surround-view image for assisted driving. Its core technology lies in image stitching and display, but it does not involve using the surround-view image for SLAM localization and mapping, thus failing to provide continuous, high-precision vehicle pose estimation. While patent CN113155140A describes a visual-inertial SLAM method, its visual features are still extracted from traditional forward-looking perspective images, making it prone to failure in textureless areas. More importantly, its multi-source fusion mechanism calculates the weight adjustment of visual observations based on the pixel displacement distance after projection of a single feature point. This point-based weight allocation strategy struggles to accurately reflect the reliability of the overall visual matching in the current frame, resulting in insufficient global consistency and robustness in complex scenarios.

[0104] This invention proposes a localization method for autonomous vehicles based on multi-source information fusion. Addressing the technical challenge of localization failure for autonomous vehicles in enclosed environments with weak GPS or no satellite signals (such as underground parking lots, urban canyons, and tunnels), this system constructs a low-cost, high-precision visual SLAM localization architecture based on multi-source heterogeneous data from four monocular fisheye cameras and an inertial measurement unit (IMU). The device mainly consists of six parts: a four-eye surround-view sensor module, an inertial measurement and navigation module, a time synchronization and calibration module, an overhead panoramic generation module, an overhead image odometry module, and an adaptive multi-source fusion module.

[0105] Compared with traditional visual SLAM systems that rely solely on forward-facing monocular or binocular cameras, the core innovation of this invention lies in: extracting stable ground textures by generating a panoramic view (BEV), thus avoiding feature loss caused by dynamic object interference and textureless areas (such as white walls and glass curtain walls); at the same time, it innovatively introduces an adaptive extended Kalman filter (EKF) framework based on visual matching quality assessment, deeply integrates high-frequency IMU pre-integration, and significantly reduces computational overhead by using a pre-computed inverse perspective transformation lookup table (LUT).

[0106] The following details the specific implementation algorithms and logic of each module.

[0107] (a) Time synchronization and calibration module

[0108] This module is a prerequisite for multi-source information fusion. Its main function is to ensure that the data from the four cameras and the IMU are accurately aligned on a unified time axis, and to determine the spatial geometric relationship between the sensors through calibration.

[0109] In this embodiment, the four fisheye cameras and the IMU are synchronized using a unified hardware clock. The IMU's high-frequency clock is used as the master clock, and a synchronization exposure trigger signal is sent to the four cameras via the controller, establishing a definite correspondence between the image sampling time and the IMU sampling sequence. Wheel speed and odometer data are aligned to the IMU or image time using timestamp interpolation.

[0110] For the A fisheye camera is used. Let the coordinates of a spatial point in the coordinate system of this camera be... Its optical axis angle is .in These respectively represent the three-dimensional coordinate vector at the th... The coordinate components of the camera coordinate system along the X, Y, and Z axes.

[0111] The polar radius was modeled using the Kannala-Brandt fisheye imaging model:

[0112]

[0113] in, Indicates the first A fisheye camera at the angle The imaging polarity below, For the first Distortion parameters of a fisheye camera.

[0114] Further define the normalized directional components for

[0115]

[0116] The corresponding pixel coordinates are

[0117]

[0118] in, These represent the spatial points at the th, . The horizontal and vertical pixel coordinates of a fisheye image For focal length parameters, The coordinates of the main point.

[0119] The external parameter matrices of each fisheye camera relative to the vehicle coordinate system can be obtained using a checkerboard or calibration board.

[0120]

[0121] in, Let be a rotation matrix. This is the translation vector. Further, through joint calibration, the rigid body transformation relationship between the IMU coordinate system and the vehicle coordinate system is obtained, thereby unifying all sensors to the vehicle coordinate system.

[0122] (ii) Panoramic View Generation Module

[0123] This invention utilizes the near-planar nature of the ground to convert four fisheye images into a top-down view in a unified ground coordinate system, generating a seamless panoramic image. To meet the requirements of low-cost computing platforms for mass production, this system employs a lookup table (LUT) method for efficient projection.

[0124] Let the ground point in the vehicle coordinate system be...

[0125]

[0126] The ground satisfies .

[0127] The ground point is at the The homogeneous coordinates in the coordinate system of each camera can be expressed as:

[0128]

[0129] in , indicating the vehicle coordinate system to the first Homogeneous coordinate transformation matrix of each camera coordinate system.

[0130] Will Substituting the above fisheye imaging model, the ground points can be obtained. Pixel coordinates in the original fisheye image Therefore, a mapping relationship is established between the vehicle's ground coordinates and the pixels in each fisheye image:

[0131]

[0132] During the system initialization phase, each pixel in the target overhead image is viewed. Corresponding ground physical coordinates Perform offline mapping calculations and generate a lookup table:

[0133]

[0134] Indicates the first The path lookup table shows the pixel location in the image from above. The original fisheye image sampling coordinates stored at [location] They represent the first The path lookup table corresponds to the horizontal and vertical sampled pixel coordinates.

[0135] At runtime, it only needs to read the corresponding original image coordinates from the lookup table and obtain the pixel values ​​through bilinear interpolation to complete fisheye distortion correction and inverse perspective projection, thus avoiding the repeated execution of complex nonlinear mapping calculations in real time.

[0136] Let the first The overhead image after road mapping is The panoramic view obtained after fusing multiple images is ,in To provide an overview of the image pixel locations, a weight-based smooth fusion method is used to stitch the images together within overlapping areas:

[0137]

[0138] Among them, weight Can be based on pixel point to the The distance from the center of the camera's field of view or the distance to the stitching boundary is determined, preferably...

[0139]

[0140] in Indicates the distance, To prevent extremely small constants with a denominator of zero.

[0141] To reduce brightness discontinuities caused by differences in automatic exposure between multiple cameras, linear photometric compensation can be performed on each image stream:

[0142]

[0143] in Indicates the first Pixel values ​​of the road image after photometric compensation. and The first Gain and bias compensation parameters for the road image. After compensation and weighted fusion, a continuous and uniform panoramic view of the vehicle around the camera is obtained.

[0144] (III) Overhead View Odometer Module

[0145] Traditional visual odometry typically extracts features from raw camera images, which can easily capture unstable or dynamic objects such as the sky, moving vehicles, and trees, and is prone to failure in white-walled environments. This module performs calculations on a stitched panoramic view, forcing the system to focus only on the most stable road textures (such as lane lines, road surface patches, landmark symbols, etc.).

[0146] Extract stable ground texture feature points (such as ORB or FAST features) from the overhead panoramic image. Utilize descriptors to perform adjacent frame (the first frame) analysis. Frame and the Inter-frame feature matching (frames).

[0147] Since the top view has been mapped onto the real two-dimensional physical scale ground, the vehicle's motion in a short time can be simplified to two-dimensional planar rigid body motion (translation along the X-axis). Translation along the Y-axis and changes in heading angle ).

[0148] Let the first The feature point coordinates of the frame are The matched first Frame feature points are The equation of motion can be expressed as:

[0149]

[0150] By using the RANSAC (Random Sample Consensus) algorithm to eliminate mismatched points and solving the above equations, the relative motion between frames can be obtained. .

[0151] To provide a reliable basis for subsequent adaptive multi-source fusion modules, this method defines a comprehensive quality index reflecting the current matching reliability. :

[0152]

[0153] in, The number of interior points after RANSAC. The expected threshold for the number of interior points; The average reprojection error of the interior points; These are the weighting coefficients. The closer the value is to 1, the richer the current road surface texture and the more reliable the relative motion estimation of the visual odometry.

[0154] (iv) Inertial Measurement and Pre-Integration Module

[0155] The inertial measurement and navigation module provides vehicle acceleration and angular velocity measurement data at a high frequency of no less than 200 Hz. In order to fuse high-frequency IMU data with relatively low-frequency visual odometry (typically around 30 Hz) and avoid repeated integration calculations during optimization or filtering, this invention employs an IMU pre-integration unit for processing.

[0156] Let the acceleration measured by the IMU be... angular velocity is Considering Zero bias at time. and Gaussian white noise The true kinematic model is:

[0157]

[0158]

[0159] in, Indicates time Rotation matrix from the carrier coordinate system to the global coordinate system; Indicates time The time derivative of the vehicle's velocity in the global coordinate system; Represents the gravity vector in the global coordinate system; Indicates time Vehicle attitude quaternion and its time derivative; symbol This represents quaternion multiplication.

[0160] In two visual keyframes and The time period between Inside, pre-integral quantity (relative position) Relative velocity Relative posture The calculation will be separated from the initial state:

[0161]

[0162]

[0163]

[0164] in, Indicates time Relative to keyframe Rotation matrix of the reference coordinate system; and These represent keyframes. The accelerometer and gyroscope used here are zero bias; Relative to keyframe Instantaneous attitude increment of the reference coordinate system.

[0165] The strapdown inertial navigation algorithm is used to calculate the vehicle's short-term, high-precision position, velocity, and attitude in real time, providing predictive driving forces for the fusion module.

[0166] (v) Adaptive Multi-Source Fusion Module

[0167] This section is the core data convergence hub of this invention. This invention innovatively designs an adaptive multi-source fusion module, with the extended Kalman filter (EKF) as its core, to deeply fuse the high-frequency predictive drive of the IMU with the relative motion output from the overhead image odometry.

[0168] The system's state vector Defined as:

[0169]

[0170] These represent the vehicle's position, velocity, and attitude quaternions in the global system, as well as the IMU's acceleration bias and gyroscope bias.

[0171] The IMU pre-integral value is used as the state prediction driver. Based on the IMU pre-integral model, the covariance matrix of the previous time step is... To spread:

[0172]

[0173] in, Let Jacobian matrix be the system state transition matrix. The noise Jacobian matrix, This is the IMU noise covariance matrix.

[0174] The relative motion output from the overhead image odometer The measurements are then fused and updated. Measurement residuals are constructed. :

[0175]

[0176] here This represents the error calculation on the manifold.

[0177] Adaptive weight adjustment: Traditional EKF visual measurement noise covariance matrix This is usually a fixed constant. However, when ground texture is missing, the visual odometry error increases sharply. This module utilizes the matching quality index output by the overhead image odometry module. Real-time calculation of adaptive measurement noise covariance matrix :

[0178]

[0179] in Using the baseline covariance, To adjust the gain. When visual features are insufficient ( When it becomes smaller, The Kalman gain increases exponentially. The system will automatically reduce the weight of visual data and place more trust in high-frequency IMU prediction data, thereby maintaining localization continuity when visual matching fails.

[0180] The adaptive multi-source fusion module includes a GNSS-assisted correction unit. When GNSS signals are available before the vehicle enters an underground parking garage or on an open road, absolute position observations are introduced for global error correction. This effectively suppresses the cumulative drift error caused by long-term operation of a single visual odometer, ensuring high positioning accuracy and continuity.

[0181] (vi) Summary and Demonstration of Beneficial Effects

[0182] In terms of overall connectivity: the four-eye surround-view sensor module achieves clock alignment and coordinate system unification with the inertial measurement and navigation module through the time synchronization and calibration module; after the image input overhead panoramic generation module generates a panoramic image, it is sent to the overhead image odometer module; finally, the odometer output result and the IMU high-frequency data are input together into the adaptive multi-source fusion module, and the positioning output frequency is stabilized at a level of not less than 100 Hz.

[0183] This system mainly consists of six parts: a four-eye surround-view sensor module, an inertial measurement and navigation module, a time synchronization and calibration module, an overhead panoramic generation module, an overhead image odometry module, and an adaptive multi-source fusion module.

[0184] The four-eye surround-view sensor module includes a front camera, a rear camera, a left camera, and a right camera; the inertial measurement and navigation module includes a three-axis accelerometer, a three-axis gyroscope, and an inertial navigation calculation unit; the time synchronization and calibration module includes a hardware trigger controller, a fisheye intrinsic parameter calibration unit, a multi-camera extrinsic parameter calibration unit, and a camera IMU joint calibration unit; the overhead panoramic generation module includes a fisheye distortion correction unit, an inverse perspective transformation unit, a multi-channel image stitching unit, and a photometric consistency correction unit; the overhead image odometry module includes a ground feature extraction unit, an inter-frame feature matching unit, a relative motion estimation unit, and a matching quality evaluation unit; and the adaptive multi-source fusion module includes an IMU pre-integration unit, an extended Kalman filter unit, an adaptive weight adjustment unit, and a GNSS auxiliary correction unit.

[0185] The overall connection is as follows: the four-eye surround-view sensor module is installed in the front, rear, left, and right directions of the vehicle, and achieves clock alignment and coordinate system unification with the inertial measurement and navigation module through the time synchronization and calibration module; the images collected by the four-eye cameras are input into the overhead panoramic generation module to generate a panoramic view in real time; the overhead panoramic view is input into the overhead image odometer module, which outputs the relative motion between frames and the matching quality index; the high-frequency acceleration and angular velocity data output by the inertial measurement and navigation module and the output results of the overhead image odometer module are jointly input into the adaptive multi-source fusion module, and after being fused by extended Kalman filtering, the high-precision position, velocity, and attitude estimation results are output, with a positioning output frequency of not less than 100 Hz.

[0186] The main function of the four-eye surround-view sensor module of this invention is to acquire ground images covering 360° around the vehicle from four directions: front, rear, left, and right, providing raw data for generating an overhead panoramic view. The four cameras use wide-angle fisheye lenses with a horizontal field of view of not less than 190°, and the horizontal field of view overlap angle between adjacent cameras is not less than 30°, ensuring sufficient common field of view in the stitching area.

[0187] The main function of the inertial measurement and navigation module of this invention is to provide vehicle acceleration and angular velocity measurement data at a high frequency of not less than 200 Hz, and to calculate the vehicle's short-term high-precision position, velocity and attitude in real time through the strapdown inertial navigation algorithm, so as to provide predictive driving force for the fusion module and maintain positioning continuity when visual matching fails.

[0188] The main function of the time synchronization and calibration module of this invention is to ensure that the data of the four cameras and the IMU are accurately aligned on a unified time axis, and to determine the intrinsic parameters of each camera, the relative extrinsic parameters between each camera, and the coordinate transformation relationship between the camera and the IMU through calibration, so as to provide an accurate geometric basis for subsequent image processing and fusion calculation.

[0189] The main function of the overhead panoramic generation module of this invention is to sequentially process four fisheye images through distortion correction and inverse perspective transformation, converting them into an overhead view in a unified ground coordinate system. Then, through weighted blending and stitching and photometric consistency correction, a real-time, seamless vehicle panoramic overhead view is generated. The aforementioned distortion correction mapping table and inverse perspective transformation lookup table are pre-calculated and stored during system initialization, and executed by table lookup during runtime to ensure real-time processing.

[0190] The main function of the overhead image odometer module of this invention is to extract stable ground texture feature points on the overhead panoramic image, estimate the two-dimensional relative displacement and heading changes of the vehicle between frames through adjacent frame feature matching and RANSAC geometric verification, and output a comprehensive quality index reflecting the current matching reliability, providing a basis for the adaptive weight adjustment of the fusion module.

[0191] The main function of the adaptive multi-source fusion module of this invention is to use extended Kalman filtering as the core, IMU pre-integrated quantity as the state prediction driver, and the relative motion quantity output by the overhead image odometry as the observation quantity for fusion and updating. It also adaptively adjusts the visual observation weights according to the matching quality index, introduces absolute position observations for global error correction when GNSS signals are available, and outputs high-precision continuous positioning results.

[0192] This invention relies on the above six components to achieve high-precision autonomous positioning of unmanned vehicles in various scenarios such as open roads, underground parking lots, and tunnels.

[0193] Appendix Figure 1 The overall structural block diagram of the present invention is shown, reflecting the connection relationships and data flow between the four-eye surround-view sensor module, the time synchronization and calibration module, the inertial measurement and navigation module, the overhead panoramic generation module, the overhead image odometry module, and the adaptive multi-source fusion module; Appendix Figure 2 This paper illustrates the real-time surround view and positioning processing flow of a vehicle based on a four-channel fisheye camera, including pre-calculation of the distortion correction mapping table and inverse perspective transformation lookup table during system initialization, fisheye distortion correction, inverse perspective transformation, table lookup and bilinear interpolation during runtime, multi-channel image stitching, overlapping area smoothing fusion, photometric consistency correction, ground feature extraction and matching, and RANSAC geometric verification and motion estimation steps; (See attached image) Figure 3 The installation positions of the front, rear, left, and right fisheye cameras and IMU on the vehicle body are shown, as well as the field-of-view coverage and overlap of each camera; Appendix Figure 4 The original panoramic images captured by four fisheye cameras are shown to illustrate that the original input image has a large field of view and obvious fisheye distortion characteristics; (Attached) Figure 5 The image shows a panoramic view of the vehicle obtained after distortion correction, inverse perspective transformation, stitching, and photometric correction of four fisheye images, with the central black area corresponding to the vehicle body area.

[0194] The specific implementation of this system overcomes the shortcomings of existing LiDAR and high-cost multi-sensor systems, which are difficult to mass-produce on consumer-grade automotive platforms. Complex algorithms can be implemented using inexpensive fisheye cameras and low-power automotive-grade chips. Its feature extraction strategy based on ground texture, combined with an IMU pre-integration and adaptive weight EKF filtering architecture, perfectly covers the autonomous positioning requirements of typical application scenarios such as underground parking lots and tunnels, providing a solid technological foundation for large-scale applications such as autonomous valet parking (AVP) and park logistics.

[0195] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for positioning an unmanned vehicle based on multi-source information fusion, characterized in that, The positioning method includes the following steps: Step S1: Acquire fisheye images of the vehicle in the front, rear, left, and right directions, as well as acceleration and angular velocity data output by the inertial measurement unit (IMU), and perform time alignment and spatial coordinate unification between the fisheye images and the data output by the inertial measurement unit. Step S2: Perform distortion correction and inverse perspective transformation on the fisheye image, map it to a unified ground coordinate system, and generate a panoramic view of the vehicle from all angles. Step S3: Extract ground texture feature points on the overhead panoramic image, perform feature matching on adjacent frames, use the random sampling consensus algorithm to estimate the relative displacement and heading change of the vehicle in the two-dimensional plane, and calculate the matching quality index characterizing the reliability of visual observation in the current frame based on the matching results. Step S4: Pre-integrate the acceleration data and angular velocity data to obtain the state prediction quantity; Step S5: Construct an extended Kalman filter model, using the state prediction as the predicted value, and the relative displacement and heading changes as observations for fusion and updating, and dynamically adjust the weight of the observations in the fusion and updating according to the matching quality index; Step S6: When the global navigation satellite system signal is available, absolute position observation is introduced to correct the error of the fusion result, and the vehicle's position, speed and attitude information are output. 2.The method of claim 1, wherein, The specific method for acquiring fisheye images of the vehicle in the front, rear, left, and right directions, as well as acceleration and angular velocity data output by the inertial measurement unit, in step S1, and for performing time alignment and spatial coordinate unification between the fisheye images and the data output by the inertial measurement unit, includes: For the A fisheye camera is used. Let the coordinates of a spatial point in the coordinate system of this camera be... Its optical axis angle is ;in These respectively represent the three-dimensional coordinate vector at the th... Coordinate components in the X, Y, and Z axes of a camera coordinate system; The polar radius was modeled using the Kannala-Brandt fisheye imaging model: in, Indicates the first A fisheye camera at the angle The imaging polarity below, For the first Distortion parameters of a fisheye camera; Further define the normalized directional components for: The corresponding pixel coordinates are: in, These represent the spatial points at the th, . The horizontal and vertical pixel coordinates of a fisheye image For focal length parameters, Principal point coordinates; The extrinsic parameter matrices of each fisheye camera relative to the vehicle coordinate system can be obtained using a checkerboard or calibration board. in, For rotation matrix, It is a translation vector; It is a zero matrix; Furthermore, the rigid body transformation relationship between the inertial measurement unit (IMU) coordinate system and the vehicle coordinate system is obtained through joint calibration, unifying all sensors under the vehicle coordinate system.

3. The autonomous vehicle positioning method based on multi-source information fusion according to claim 1, characterized in that, The specific method for performing distortion correction and inverse perspective transformation on the fisheye image in step S2, mapping it to a unified ground coordinate system, and generating a panoramic view of the vehicle from all angles includes: Let the ground point in the vehicle coordinate system be: The ground satisfies ; These are the coordinates of the ground point in the vehicle coordinate system. Axis coordinates and Axis coordinates; The ground point is at the The homogeneous coordinates in the camera coordinate system can be expressed as: in, , indicating the vehicle coordinate system to the first Homogeneous coordinate transformation matrix for each camera coordinate system; Will Substituting the above fisheye imaging model, the ground points can be obtained. Pixel coordinates in the original fisheye image Therefore, a mapping relationship is formed between the vehicle's ground coordinates and the pixels of each fisheye image: During the system initialization phase, each pixel in the target overhead image is viewed. Corresponding ground physical coordinates Perform offline mapping calculations and generate a lookup table: in, Indicates the first The path lookup table shows the pixel location in the image from above. The original fisheye image sampling coordinates stored at [location] They represent the first The horizontal and vertical sampled pixel coordinates corresponding to the path lookup table; Let the first The overhead image after road mapping is The panoramic view obtained after fusing multiple images is ,in To visualize the image pixel locations from above, a weight-based smooth fusion method is used to stitch the images together within overlapping areas. Among them, weight Can be based on pixel point to the The distance from the center of the road camera's field of view or the distance to the stitching boundary is determined, preferably as follows: in, Indicates the distance, To prevent extremely small constants with a denominator of zero; To reduce brightness discontinuities caused by differences in automatic exposure between multiple cameras, linear photometric compensation can be performed on each image stream: in, Indicates the first Pixel values ​​of the road image after photometric compensation. and The first Gain and bias compensation parameters for road images; after compensation and weighted fusion, a continuous and uniform panoramic view of the vehicle around the camera is obtained.

4. The autonomous vehicle positioning method based on multi-source information fusion according to claim 1, characterized in that, The specific methods described in step S3, which involve extracting ground texture feature points from the overhead panoramic image, performing feature matching on adjacent frames, estimating the relative displacement and heading changes of the vehicle in the two-dimensional plane using the random sampling consensus algorithm, and calculating a matching quality index characterizing the reliability of visual observation in the current frame based on the matching results, include: ORB or FAST feature points are extracted from the ground area of ​​the panoramic view as ground texture feature points; inter-frame feature matching between adjacent frames is performed using descriptors. Let the first The feature point coordinates of the frame are The matched first Frame feature points are The equation of motion can be expressed as: in, This represents the translation along the X-axis; This represents the translation along the Y-axis; This represents the change in heading angle; By using the Random Sample Consensus (RANSAC) algorithm to eliminate mismatches and solving the above equations, the inter-frame relative motion can be obtained. ; Calculate matching quality index based on feature matching results : in, The number of interior points after RANSAC. The expected threshold for the number of interior points; The average reprojection error of the interior points; These are the weighting coefficients.

5. The autonomous vehicle positioning method based on multi-source information fusion according to claim 1, characterized in that, The specific method for pre-integrating the acceleration data and angular velocity data to obtain the state prediction quantity in step S4 includes: The inertial measurement and navigation module provides vehicle acceleration and angular velocity measurement data at a high frequency of no less than 200 Hz; Let the acceleration measured by the inertial measurement unit (IMU) be... angular velocity is ;exist Zero bias at time and Gaussian white noise The kinematic model is as follows: in, Indicates time Rotation matrix from the carrier coordinate system to the global coordinate system; Indicates time The time derivative of the vehicle's velocity in the global coordinate system; Represents the gravity vector in the global coordinate system; Indicates time Vehicle attitude quaternion and its time derivative; symbol Represents quaternion multiplication; In two visual keyframes and The time period between Within, relative position of pre-integral quantity Relative velocity Relative posture The calculations will be separated from the initial state: in, Indicates time Relative to keyframe Rotation matrix of the reference coordinate system; and These represent keyframes. The accelerometer and gyroscope used here are zero bias; Relative to keyframe Instantaneous attitude increment of the reference coordinate system.

6. The autonomous vehicle positioning method based on multi-source information fusion according to claim 1, characterized in that, The specific method for constructing the extended Kalman filter model in step S5, using the state prediction as the predicted value, and the relative displacement and heading changes as observations for fusion and updating, and dynamically adjusting the weights of the observations in the fusion and updating according to the matching quality index, includes: The system's state vector Defined as: in, These represent the vehicle's position, velocity, and attitude quaternions in the global frame, as well as the acceleration zero bias and gyroscope zero bias of the inertial measurement unit (IMU). The pre-integrated quantity of the inertial measurement unit (IMU) is used as the state prediction driver; based on the IMU pre-integration model, the covariance matrix of the previous time step is... To spread: in, Let Jacobian matrix be the system state transition matrix. The noise Jacobian matrix, The noise covariance matrix of the inertial measurement unit (IMU); The relative motion output from the overhead image odometer As an observation, it is used for fusion and updating; measurement residuals are constructed. : here This represents the error calculation on the manifold; Matching quality index output by the overhead image odometry module Real-time calculation of adaptive measurement noise covariance matrix : in, Using the baseline covariance, To adjust the gain; When visual features are insufficient, i.e. When it shrinks, The Kalman gain increases exponentially. The system will automatically reduce the weight of visual data and place more trust on high-frequency inertial measurement unit (IMU) prediction data.

7. A system for implementing the autonomous vehicle positioning method based on multi-source information fusion as described in claim 1, characterized in that, It includes a four-eye fisheye camera module, an inertial measurement module, a time synchronization and calibration module, an overhead panoramic generation module, an overhead image odometry module, and an adaptive multi-source fusion module; The four-eye fisheye camera module and the inertial measurement module are respectively connected to the time synchronization and calibration module; The output of the four-eye fisheye camera module is connected to the input of the overhead panoramic generation module, and the output of the overhead panoramic generation module is connected to the input of the overhead image odometer module. The output terminals of the overhead image odometry module and the inertial measurement module are respectively connected to the input terminal of the adaptive multi-source fusion module; The output of the adaptive multi-source fusion module is used to output the vehicle's position, speed, and attitude information.

8. The autonomous vehicle positioning system based on multi-source information fusion according to claim 7, characterized in that, The overhead panoramic generation module includes a distortion correction unit, an inverse perspective transformation unit, an image stitching unit, and a photometric consistency correction unit; wherein, the output of the distortion correction unit is connected to the input of the inverse perspective transformation unit, the output of the inverse perspective transformation unit is connected to the input of the image stitching unit, and the output of the image stitching unit is connected to the input of the photometric consistency correction unit.

9. The autonomous vehicle positioning system based on multi-source information fusion according to claim 7, characterized in that, The adaptive multi-source fusion module includes an inertial measurement unit (IMU) pre-integration unit, an extended Kalman filter unit, an adaptive weight adjustment unit, and a GNSS auxiliary correction unit; wherein the output terminals of the IMU pre-integration unit, the adaptive weight adjustment unit, and the GNSS auxiliary correction unit are respectively connected to the input terminal of the extended Kalman filter unit.

Images