A real-time positioning method in GPS signal blind area based on SLAM-IMU coupling

Abstract

The application discloses a real-time positioning method in a GPS signal blind area based on SLAM-IMU coupling, and is based on a positioning device comprising a binocular camera provided with a SLAM sensor and an IMU element; the positioning method comprises the following steps: the binocular camera provided with the SLAM sensor transmits an image sequence captured at different discrete time points to a computer; the IMU element transmits real-time angular velocity and acceleration collected to the computer; the computer combines the image sequence, the real-time angular velocity and the acceleration, establishes a nonlinear objective function containing multiple optimization variables, and makes the optimization variables reach a global optimal solution through multiple iterations, so that a real-time position and a movement trajectory are obtained. The application couples IMU data with a visual SLAM sensor through pre-integration processing and performs iterative optimization, constructs a real-time positioning algorithm based on multi-source information coupling, and solves the problem of worker autonomous positioning in a GPS signal blind area.

Description

Technical Field

[0001] This invention relates to the field of SLAM positioning technology, and in particular to a real-time positioning method in GPS signal blind zones based on SLAM-IMU coupling. Background Technology

[0002] Existing positioning technologies primarily rely on signals emitted by positioning tags worn by workers. These signals are received and processed by intelligently deployed positioning base stations to obtain the workers' location information, enabling precise positioning and management. This technology is based on signal transmission between external fixed base stations and mobile targets on-site. However, it is worth noting that this positioning technology is highly sensitive to environments with severe obstruction and strong signal interference, which to some extent affects its positioning accuracy and stability.

[0003] SLAM (Simultaneous Localization and Mapping) refers to a method for simultaneously locating oneself in an unknown environment and building a map of that environment. The term SLAM originated in the field of robotics, and its main function is to simultaneously estimate the posture of an object and the 3D structure of the environment in an unknown setting. SLAM is a technology that utilizes visual information or laser and sensor data to achieve simultaneous localization and mapping. Visual SLAM uses cameras to perceive the environment. In an unknown environment, autonomously moving humans or robots use data captured by cameras to make preliminary estimates of their own position and the surrounding environment. As the robot moves, it performs precise localization based on real-time position estimates and data perceived by the cameras, continuously building and updating the map while rationally planning its path. It is widely used in 3D modeling, virtual reality (VR), robotics, autonomous driving, and other fields. Its successful application in these fields makes precise positioning of workers in scenarios where satellite aerial triangulation is ineffective possible. By equipping personnel with visual cameras, the current position can be determined by changes in the surrounding environment, which can be used in scenarios with poor satellite positioning. Chinese invention application CN114136311A discloses a laser SLAM localization method based on IMU pre-integration, which combines IMU and laser SLAM to solve the pose. However, laser SLAM lacks the ability to detect loop closures, and it is difficult to eliminate accumulated errors. Summary of the Invention

[0004] The purpose of this invention is to solve the problems in the prior art.

[0005] The technical solution adopted by this invention to solve its technical problem is: to provide a real-time positioning method in GPS signal blind zones based on SLAM-IMU coupling, based on a positioning device including a binocular camera equipped with a SLAM sensor and an IMU element, the positioning method including the following steps:

[0006] The binocular camera transmits the captured image sequences at different discrete time points to the computer;

[0007] The IMU element transmits the collected real-time angular velocity and acceleration to the computer;

[0008] The computer combines image sequences, real-time angular velocity, and acceleration to establish a nonlinear objective function containing multiple optimization variables. Through multiple iterations, the optimization variables reach the global optimum, thereby obtaining the real-time position and trajectory.

[0009] Preferably, the process by which the computer obtains the real-time location and movement trajectory of the staff includes the following steps:

[0010] The acquired image sequence is subjected to corner point extraction, corner point optimization, and corner point matching to establish visual information; the corner point refers to a significant feature point.

[0011] Based on angular velocity and acceleration Construct an IMU motion model and pre-integrate the IMU motion model to obtain a simplified IMU motion model;

[0012] A nonlinear objective function with multiple optimization variables is established by combining visual information and a simplified IMU motion model;

[0013] The initial iterative values ​​of the optimization variables are determined by approximate values ​​based on sensor measurements. The global optimal solution is obtained through multiple iterations, thus obtaining the real-time location and movement trajectory of the staff.

[0014] Preferably, the process of the computer establishing visual information includes the following steps:

[0015] Corner extraction is performed using the Shi-Tomasi algorithm, which selects the best corners to improve the quality of feature points.

[0016] Corner optimization employs interpolation for sub-pixel corner detection, using linear interpolation along the object's edge to avoid abrupt changes in pixel values.

[0017] Corner matching uses optical flow to track and match corner points, and uses the pixel brightness changes between adjacent frames to calculate the camera's displacement and rotation, estimating the camera's motion in real time, thereby providing positioning information during map building.

[0018] Preferably, the interpolation method uses linear interpolation to solve for pixel gray values, combining the known pixel gray values ​​around the desired point with a straight line of two known quantities to find the pixel gray value of the desired point.

[0019] Preferably, for the desired point P(x,y), the pixel gray values ​​of four points surrounding P(x,y) are known, including the pixel gray values ​​of A(x1,y2), B(x2,y2), C(x1,y1), and D(x2,y1), where y2 > y > y1, and x2 > x > x1; the process of calculating the pixel gray value of P(x,y) includes:

[0020] Insert pixels F = (x, y1) and E = (x, y2) between AB and CD respectively;

[0021] The pixel value of point P is calculated based on the pixel values ​​of E and F, and is expressed as:

[0022]

[0023] Where f(·) represents the pixel value.

[0024] Preferably, the corner matching uses optical flow to track and match corner points. When m spatial points in an m×m window move in unison within a continuous time interval, it is represented as:

[0025]

[0026] Where μ represents the velocity vector of the pixel along the x-axis, and v represents the velocity vector of the pixel along the y-axis; F x express F y express F t express F is represented as F(x,y,t), which refers to the pixel grayscale value of the point at coordinates (x,y) at time t.

[0027] Preferably, the IMU motion model is represented as follows:

[0028]

[0029] in, Indicates the position at time j. Represents the velocity at time j. This represents the attitude at time j; This represents the position at time i. Let represent the velocity at time i, and Δt represent the time interval from i to j. Indicates the attitude at time t; G represents the actual value of acceleration at time t. w δ represents the gravity vector in the global coordinate system. t Indicates a very short time interval;

[0030] This represents the actual value of the angular velocity at time t; This represents four-element multiplication.

[0031] Preferably, the pre-integration of the IMU motion model specifically involves aligning the IMU data of the first frame with the image of the a-th frame, then aligning the IMU data of the last frame with the image of the b-th frame, and finally integrating the data to obtain a simplified IMU motion model representation:

[0032]

[0033] in, This represents the position of the IMU coordinate system relative to the world coordinate system at time k+1. This represents the velocity of the IMU coordinate system relative to the world coordinate system at time k+1. This represents the attitude from the IMU coordinate system to the world coordinate system at time k+1; Let Δt represent the position, velocity, and attitude of the IMU coordinate system relative to the world coordinate system at time k. k This represents the time interval from k to k+1. This indicates the accelerometer bias at time t. This indicates that the gyroscope bias occurs at time t. This indicates the accelerometer measurement value in the Body system. This represents the gyroscope measurement value in the Body system, b g and b a This refers to the zero bias of the gyroscope and the zero bias of the accelerometer. It is the rotation matrix of the IMU's Body coordinate system relative to the world coordinate system.

[0034] Preferably, the step of establishing a nonlinear objective function by combining visual information and a simplified IMU motion model includes the following steps:

[0035] Construct a state vector, including sensor state, extrinsic parameters, and inverse depth of spatial landmarks. The state vector X is represented as:

[0036]

[0037] Where x0, x1, ..., x n The sensor state is represented by the sensor's monitored parameters (coordinates, velocity, rotation angle), accelerometer deviation, and gyroscope readings, where the state variable x for the k-th frame is... k Including translation velocity quantity Rotation b a b represents the deviation of the accelerometer in the IMU. g Represents the bias of the gyroscope in the IMU; λ0,λ1,…,λm This represents the inverse depth of a spatial landmark, which is the reciprocal of the landmark's depth. The extrinsic parameters representing the camera and IMU include the translation and rotation relationships between the camera and the IMU;

[0038] Constraints are imposed on the IMU model, mainly the changes in displacement, velocity, rotation, acceleration deviation, and gyroscope deviation between two frames, resulting in the IMU measurement residuals expressed as follows:

[0039]

[0040] Where δa is the position pre-integration error, δθ is the angle pre-integration error, δβ is the velocity pre-integration error, and δb is the position pre-integration error. a For zero bias error in acceleration, δb g This represents the zero bias error of the gyroscope; among which, To represent four-element multiplication, [·] xyz A three-dimensional vector that takes only the imaginary part (x, y, z) of a quaternion; This represents the rotation matrix in the IMU body coordinate system from frame k to frame (k+1). The position of the IMU coordinate system relative to the world coordinate system at time k+1. The position of the IMU coordinate system relative to the world coordinate system at time k. The velocity of the IMU coordinate system relative to the world coordinate system at time k. Acceleration in the IMU body coordinate system from frame k to k+1 Indicates from frame b k to frame b k+1 Rotation in quaternion form in IMU body coordinate system The inverse of the rotation quaternion from the world coordinate system to the IMU body coordinate system at time k. The rotation quaternion (attitude) for the transformation from the body coordinate system to the world coordinate system at time k+1. Let represent the rotation matrix from the IMU body coordinate system to the world coordinate system at time k. This represents the velocity of the IMU coordinate system relative to the world coordinate system at time k+1. The deviation from the IMU body coordinate system at time k to the IMU body coordinate system at time k+1. This represents the acceleration deviation of the IMU body coordinate system at time k+1. This represents the acceleration error of the IMU body coordinate system at time k. This represents the angular velocity deviation of the IMU body coordinate system at time k+1. Angular velocity deviation of the IMU body coordinate system at time k;

[0041] Constraints are imposed on visual information to obtain the ideal sensor pose by minimizing the visual residual caused by pixel reprojection. The pixel coordinates of the first landmark point, initially projected into the i-th camera coordinate system, are transformed and mapped to the j-th coordinate system. The difference between the transformed pixel coordinates and the directly observed corresponding pixel coordinates constitutes the visual residual, expressed as:

[0042]

[0043] in, Projected onto the Cth j The x-value in the camera coordinate system. Projected onto the Cth j The y-value in the camera coordinate system. Projected onto the Cth j The z-value in the camera coordinate system. This represents the inverse homogeneous transformation matrix for transforming from the IMU coordinate system back to the camera coordinate system. Let be the inverse matrix of the homogeneous transformation matrix from the world coordinate system to the j-th IMU coordinate system. T represents the inverse of the homogeneous transformation matrix from the world coordinate system to the j-th IMU coordinate system. bc This represents the homogeneous transformation matrix from the camera coordinate system to the IMU coordinate system; the feature point is at the Cth position. i The pixel coordinates on the frame are λ represents the inverse depth;

[0044] Taking into account prior information, visual data, and IMU constraint information, the optimal state vector integrating multi-sensor information is obtained through a nonlinear optimization method, expressed as:

[0045]

[0046] Where, ||r p -J p X|| 2 It is marginalized prior information. It is the IMU measurement residual. It is visual residual. This represents the noise covariance of the IMU pre-integration. This represents the noise covariance generated by visual perception.

[0047] Preferably, the step of achieving a globally optimal solution for the optimization variables through multiple iterations includes the IMU measurement residuals. Visual residual IMU pre-integrated noise covariance Covariance of noise generated by vision The initial iterative values ​​of the optimization variables are determined based on approximate values ​​measured by the sensors.

[0048] The present invention has the following beneficial effects:

[0049] (1) The site 3D graphics constructed by binocular vision SLAM are optimized by Shi-Tomasi corner detection algorithm, interpolation method and optical flow tracking algorithm, and the SLAM and IMU coupled positioning algorithm is further improved to enhance the integrity, accuracy and stability of SLAM vision method and provide scale information, so as to realize the real-time positioning of on-site workers; among them, corner optimization adopts interpolation method to detect sub-pixel corners. By performing linear interpolation along the edge of the object, the value abrupt change caused by pixel value abrupt change can be avoided, so that the pixel gray value of the final obtained point is more accurate;

[0050] (2) Compared with laser SLAM positioning technology, the visual SLAM used in this invention uses a large amount of redundant texture information, making loop closure detection easier. Even if a certain amount of error accumulates at the front end, the error can still be eliminated through loop closure correction. Furthermore, the price of laser radar is much higher than that of visual sensors, resulting in high deployment costs and limited operability in practical applications. In contrast, the visual SLAM used in this invention has advantages such as low cost, convenient deployment, and high visualization, making it more operable in practical applications.

[0051] (3) Compared with adding IMU residual terms, point feature and line feature residual terms to BA joint optimization pose calculation, this invention adopts the sliding window method to optimize the calculation process and at the same time uses the Schur complement method to convert the constraints of the frame outside the window and the frame inside the window into prior error information and couple it into the target function of the coupled positioning, so as to reduce the computational amount and computation time of the coupled algorithm.

[0052] (4) This invention combines IMU and laser SLAM to achieve real-time positioning of personnel, which effectively solves the problem of inaccurate positioning caused by signal blind spots in the existing positioning management system. It helps to protect the life safety of workers at the construction site, improves the management efficiency of supervisors at the construction site, and provides a favorable guarantee for the safety and stability of the entire construction process.

[0053] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. Attached Figure Description

[0054] Figure 1 This is a diagram illustrating the method steps of an embodiment of the present invention;

[0055] Figure 2 This is a flowchart illustrating an embodiment of the present invention;

[0056] Figure 3This is a schematic diagram of corner point optimization using the interpolation method in an embodiment of the present invention. Detailed Implementation

[0057] See Figure 1 and Figure 2 The diagram shown is a flowchart and step diagram of a method according to an embodiment of the present invention, including the following steps:

[0058] S101, a positioning device including a binocular camera with SLAM sensors and an IMU element is installed on the construction worker;

[0059] S102, the binocular camera equipped with a SLAM sensor transmits the captured image sequences at different discrete time points to the computer, and the IMU element transmits the acquired real-time angular velocity and acceleration to the computer.

[0060] S103: The computer combines the acquired image sequence, real-time angular velocity, and acceleration to establish a nonlinear objective function containing multiple optimization variables. Through multiple iterations, the optimization variables reach the global optimal solution, thereby obtaining the real-time location and movement trajectory of the construction personnel.

[0061] Specifically, S103 includes the following steps:

[0062] S1031, corner point extraction, corner point optimization, and corner point matching are performed on the acquired image sequence to establish visual information; the corner point refers to a significant feature point.

[0063] S1032, based on angular velocity and acceleration Construct an IMU motion model and pre-integrate the IMU motion model to obtain a simplified IMU motion model;

[0064] S1033, a nonlinear objective function containing multiple optimization variables is established by combining visual information and a simplified IMU motion model;

[0065] S1034 determines the initial iterative value of the optimization variable by using approximate values ​​based on sensor measurements, and obtains the global optimal solution through multiple iterations, thus obtaining the real-time location and movement trajectory of the staff.

[0066] Specifically, S1031 performs corner point (significant feature point) extraction, corner point optimization, and corner point matching on the acquired image, and inversely calculates the camera's motion trajectory and pose changes based on the motion information of these feature points.

[0067] Specifically, the Shi-Tomasi algorithm is used to extract corner points, and the quality of feature points is improved by selecting the best corner points.

[0068] Specifically, corner optimization employs interpolation for sub-pixel corner detection. By performing linear interpolation along the object's edge, it avoids abrupt changes in pixel values, resulting in more accurate pixel grayscale values. This invention uses linear interpolation to solve for pixel grayscale values. Linear interpolation is a method that combines the known pixel grayscale values ​​around the desired point with a straight line representing two known quantities to find the value of the unknown quantity. See [link to related documentation]. Figure 3 As shown, to find the pixel value of point P given the grayscale values ​​of four pixels ABCD, first, insert two pixels F = (x, y1) and E = (x, y2) into AB and CD respectively on the x-axis. Then, along the y-axis, obtain the pixel value of point P based on the pixel values ​​of points E and F. The pixel value of point P is calculated as follows:

[0069]

[0070] Here, f() represents the pixel value of a point.

[0071] Specifically, corner matching uses optical flow to track and match corners. It calculates camera displacement and rotation by utilizing pixel brightness changes between adjacent frames, estimating camera motion in real time to provide positioning information during map building. The image information during optical flow tracking can be viewed as a time-dependent function F(t), with pixel grayscale values ​​F(x,y,t). The velocity vector of the pixel along the x-axis is then considered. Let μ be the velocity vector along the y-axis. Let it be ν. At the same time, let... Let it be F x , Let it be F y , Let it be F t When m quantities move in unison within an m×m window over a continuous time interval, we can obtain m... 2 Equations:

[0072]

[0073] The above formula can Its speed of motion is

[0074] Specifically, S1032 constructs an IMU motion model based on the collected personnel acceleration, angular velocity, etc., and pre-integrates the IMU motion model to obtain a simplified IMU motion model; including the following steps:

[0075] S21, the measurement model for angular velocity and acceleration in the IMU is as follows:

[0076]

[0077] Here, the symbol w refers to the Global Positioning System. and This represents the measured values ​​of angular velocity and acceleration; ω b and a w These are the actual values ​​of angular velocity and acceleration. bg and b a This refers to the zero bias of gyroscopes and accelerometers; n g and n a This is the noise from the gyroscope and accelerometer. w =[0,0,g] T This represents the gravity vector in the global coordinate system.

[0078] S22, then taking the derivatives of position, velocity, and attitude with respect to time, we obtain the following formulas:

[0079]

[0080] Where P, v, and q represent the position, velocity, and attitude of the IMU, respectively. This is a four-element multiplication.

[0081] S23, integrate the position, velocity, and attitude measurements collected at time i to obtain the position, velocity, and attitude at time j, as shown in the following formula:

[0082]

[0083] Wherein, δt represents a very short time interval, and it is assumed that the bias remains unchanged within δt. In this embodiment of the invention, the value of δt ranges from 0.001 to 0.002.

[0084] S24. Pre-integration is performed on the IMU data. Pre-integration is a technique used to process two consecutive frames of IMU data, integrating acceleration and angular velocity to more accurately derive the camera's relative position and attitude. However, when processing image sequences, if there are multiple IMU data points between consecutive frames (i.e., image a and image b), the entire calculation process becomes quite complex. Therefore, to simplify the model, this study introduces a simplification technique: first, the IMU data of the first frame is aligned with the image of frame a; second, the IMU data of the last frame is aligned with the image of frame b; and finally, the data is integrated to accurately determine the camera's position and attitude. The pre-integration formulas for position, velocity, and attitude in continuous time in the IMU method are as follows:

[0085]

[0086] in, This indicates the accelerometer measurement value in the Body system. This represents the gyroscope measurement value in the Body system, b g and b a This refers to the zero bias of the gyroscope and accelerometer. This is the rotation matrix of the IMU's body coordinate system relative to the world coordinate system. Furthermore, in all the above formulas, the ^ sign indicates that the measured value is affected by zero bias and noise, such as... Without representing the actual value, such as a w ω b .

[0087] Specifically, step S1033 establishes a nonlinear objective function by constructing the residual variance of visual information and IMU information, which includes multiple optimization variables of the image and IMU. This includes the following steps:

[0088] S31, construct the state vector; the state vector includes n+1 sensor states, one extrinsic parameter, and m+1 inverse depths of spatial landmarks. The sensor monitoring parameters include basic information such as coordinates, velocity, and rotation angle, while also considering accelerometer deviation and gyroscope readings. The extrinsic parameter includes the translation and rotation relationship between the camera and the IMU. The inverse depth of the spatial landmarks is the reciprocal of the landmark depth. X represents the state vector, x... k The state variables for the k-th frame include five variables: translation, velocity, rotation, accelerometer deviation, and gyroscope deviation. This represents the extrinsic parameters of the camera and IMU. The state vector is as follows:

[0089]

[0090] S32, the constraints on the IMU, mainly concern the changes in displacement, velocity, rotation, acceleration deviation, and gyroscope deviation between two frames. The specific formulas are as follows:

[0091]

[0092] The above figures, in order, represent the position (P), velocity (v), rotation (q), and two biases (ba and bw) at times k and k+1 in the IMU. δa represents the position pre-integration error, δθ represents the angle pre-integration error, and δβ represents the velocity pre-integration error. δb a For zero bias error in acceleration, δb g This represents the gyroscope's zero-bias error. This represents a four-element multiplication, which involves the state variables from frame k to frame (k+1), totaling 10 variables. In the above errors, displacement, velocity, and offset are obtained by direct subtraction. [·] xyz It is a three-dimensional vector that takes only the imaginary part (x,y,z) of the quaternion.

[0093] S33, regarding visual constraints, the ideal sensor pose can be obtained by minimizing the visual residual caused by pixel reprojection. The pixel coordinates of the first landmark point, initially projected into the i-th camera coordinate system, are transformed and mapped to the j-th coordinate system. The difference between the transformed pixel coordinates and the directly observed corresponding pixel coordinates constitutes the visual residual. The calculation formula is as follows:

[0094]

[0095] Wherein, the state variable to be estimated is the three-dimensional spatial coordinates (x, y, z) of the feature point. T Observed values ​​(u,v) T The coordinates of the feature in the camera normalized plane.

[0096] There are three variables to be optimized: extrinsic parameters, state variables, and inverse depth. The values ​​projected onto the camera coordinate system in the j-th frame are:

[0097]

[0098] Among them, T bc b is the extrinsic parameter for the rotation from the camera to the IMU. i Transformation matrix from frame IMU frame to world frame b j Transformation matrix from frame IMU frame to world frame Feature points at the Cth i The pixel coordinates on the frame are This represents the inverse depth.

[0099] S34. When establishing the overall objective function, prior information, visual data, and IMU constraint information were comprehensively considered. The optimal state vector integrating multi-sensor information was obtained through a nonlinear optimization method, as shown in the following formula:

[0100]

[0101] in, This represents the noise covariance of the IMU pre-integration. r represents the noise covariance generated by vision. p -J p X is marginalized prior information. It is the IMU measurement residual. This refers to visual residuals. Covariance is used to adjust the confidence level of the IMU and vision sensors during data coupling. When the noise level of one sensor is significantly higher than that of the other, it means that the data from the first sensor is less reliable. Therefore, during data coupling, it is necessary to appropriately reduce the weight ratio of the first sensor while increasing the weight ratio of the other sensor to enhance the relative reliability of the data from each sensor.

[0102] Specifically, step S1034 determines the initial iterative values ​​of the optimization variables based on approximate values ​​measured by sensors, and then performs in-depth optimization processing on this basis to ensure that the optimization variables can reach the global optimum as much as possible and avoid getting trapped in local optima. Through the coupling of IMU and SLAM, the location and movement trajectory of personnel can be located in real time on a computer.

[0103] As can be seen, the present invention provides a real-time positioning method for GPS signal blind zones based on SLAM-IMU coupling. This method uses a visual SLAM sensor to perceive the environment and perform corner matching to construct a three-dimensional image. Then, it couples and optimizes pre-integrated IMU data to build a real-time positioning algorithm based on multi-source information coupling. This solves the problem of workers' autonomous positioning in signal blind zones, helps supervisors understand the workers' location in real time, provides safety guidance for staff, and reduces the occurrence of accidents.

[0104] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A real-time positioning method based on SLAM-IMU coupling in GPS signal blind zones, characterized in that, The localization method, based on a localization device including a binocular camera with a SLAM sensor and an IMU element, includes the following steps: The binocular camera transmits the captured image sequences at different discrete time points to the computer; The IMU element transmits the collected real-time angular velocity and acceleration to the computer; The computer combines image sequences, real-time angular velocity, and acceleration to establish a nonlinear objective function containing multiple optimization variables. Through multiple iterations, the optimization variables reach the global optimal solution, thereby obtaining the real-time position and trajectory. The process by which the computer obtains the real-time location and movement trajectory of the staff includes the following steps: The acquired image sequence is subjected to corner point extraction, corner point optimization, and corner point matching to establish visual information; the corner point refers to a significant feature point. Based on angular velocity and acceleration Construct an IMU motion model and pre-integrate the IMU motion model to obtain a simplified IMU motion model; A nonlinear objective function with multiple optimization variables is established by combining visual information and a simplified IMU motion model; The initial iterative values ​​of the optimization variables are determined by approximate values ​​based on sensor measurements. The global optimal solution is obtained through multiple iterations, thus obtaining the real-time location and movement trajectory of the staff. The IMU motion model is represented as follows: in, Indicates the position at time j. Represents the velocity at time j. This represents the attitude at time j; This represents the position at time i. Let represent the velocity at time i, and Δt represent the time interval from i to j. Indicates the attitude at time t; G represents the actual value of acceleration at time t. w δ represents the gravity vector in the global coordinate system. t Indicates a short time interval; This represents the actual value of the angular velocity at time t; This represents four-element multiplication; The pre-integration of the IMU motion model specifically involves aligning the IMU data of the first frame with the image of the a-th frame, then aligning the IMU data of the last frame with the image of the b-th frame, and finally integrating the data to obtain a simplified IMU motion model representation: in, This represents the position of the IMU coordinate system relative to the world coordinate system at time k+1. This represents the velocity of the IMU coordinate system relative to the world coordinate system at time k+1. This represents the attitude from the IMU coordinate system to the world coordinate system at time k+1; Let Δt represent the position, velocity, and attitude of the IMU coordinate system relative to the world coordinate system at time k. k This represents the time interval from k to k+1. This indicates the accelerometer bias at time t. This indicates that the device is biased by the gyroscope at time t; This indicates the accelerometer measurement in the Body system. This represents the gyroscope measurement value in the Body system, b g and b a These refer to the zero bias of the gyroscope and the zero bias of the accelerometer, respectively. It is the rotation matrix of the IMU's Body coordinate system relative to the world coordinate system.

2. The real-time positioning method for GPS signal blind zones based on SLAM-IMU coupling according to claim 1, characterized in that, The process of establishing visual information by the computer includes the following steps: Corner extraction is performed using the Shi-Tomasi algorithm, which selects the best corners to improve the quality of feature points. Corner optimization employs interpolation for sub-pixel corner detection, using linear interpolation along the object's edge to avoid abrupt changes in pixel values. Corner matching uses optical flow to track and match corner points, and uses the pixel brightness changes between adjacent frames to calculate the camera's displacement and rotation, estimating the camera's motion in real time, thereby providing positioning information during map building.

3. The real-time positioning method for GPS signal blind zones based on SLAM-IMU coupling according to claim 2, characterized in that, The interpolation method uses linear interpolation to solve for pixel gray values. It combines the known pixel gray values ​​around the desired point with a straight line of two known quantities to find the pixel gray value of the desired point.

4. The real-time positioning method for GPS signal blind zones based on SLAM-IMU coupling according to claim 3, characterized in that, For the desired point P(x,y), the pixel gray values ​​of the four points surrounding P(x,y) are known, including the pixel gray values ​​of A(x1,y2), B(x2,y2), C(x1,y1), and D(x2,y1), where y2 > y > y1 and x2 > x > x1. The process of calculating the pixel gray value of P(x,y) includes: Insert pixels F = (x, y1) and E = (x, y2) between AB and CD respectively; The pixel value of point P is calculated based on the pixel values ​​of E and F, and is expressed as: Where f(·) represents the pixel value.

5. The real-time positioning method for GPS signal blind zones based on SLAM-IMU coupling according to claim 3, characterized in that, The corner matching uses optical flow to track and match corner points. When m spatial points in an m×m window move in unison within a continuous time interval, it is represented as: Where μ represents the velocity vector of the pixel along the x-axis, and v represents the velocity vector of the pixel along the y-axis; F x express F y express F t express F is represented as F(x,y,t), which refers to the pixel grayscale value of the point at coordinates (x,y) at time t.

6. The real-time positioning method for GPS signal blind zones based on SLAM-IMU coupling according to claim 1, characterized in that, The process of establishing a nonlinear objective function containing multiple optimization variables by combining visual information and a simplified IMU motion model includes the following steps: Construct a state vector, including sensor state, extrinsic parameters, and inverse depth of spatial landmarks. The state vector X is represented as: Where x0, x1, ..., x n The sensor status is represented by the sensor's monitoring parameters, accelerometer deviation, and gyroscope readings. The monitoring parameters include coordinates, velocity, and rotation angle; where the state variable x for the k-th frame... k Including translation velocity quantity Rotation b a b represents the deviation of the accelerometer in the IMU. g Represents the bias of the gyroscope in the IMU; λ0,λ1,…,λ m This represents the inverse depth of a spatial landmark, which is the reciprocal of the landmark's depth. The extrinsic parameters representing the camera and IMU include the translation and rotation relationships between the camera and the IMU; Constraints are imposed on the IMU model, mainly the changes in displacement, velocity, rotation, acceleration deviation, and gyroscope deviation between two frames, resulting in the IMU measurement residuals expressed as follows: Where δa is the position pre-integration error, δθ is the angle pre-integration error, δβ is the velocity pre-integration error, and δb is the position pre-integration error. a For zero bias error in acceleration, δb g This represents the zero bias error of the gyroscope; among which, To represent four-element multiplication, [·] xyz A three-dimensional vector that takes only the imaginary part of a quaternion; This represents the rotation matrix in the IMU body coordinate system from frame k to frame (k+1). This represents the position of the IMU coordinate system relative to the world coordinate system at time k+1. This represents the position of the IMU coordinate system relative to the world coordinate system at time k. The velocity representing the position of the IMU coordinate system relative to the world coordinate system at time k. This represents the acceleration in the IMU body coordinate system from frame k to frame (k+1). Indicates from frame b k to frame b k+1 Rotation in quaternion form in IMU body coordinate system This represents the inverse of the rotation quaternion from the world coordinate system to the IMU body coordinate system at time k. This represents the rotation quaternion used to transform from the body coordinate system to the world coordinate system at time k+1. Let represent the rotation matrix from the IMU body coordinate system to the world coordinate system at time k. This represents the velocity of the IMU coordinate system relative to the world coordinate system at time k+1. This represents the deviation from the IMU body coordinate system at time k to the IMU body coordinate system at time k+1. This represents the acceleration deviation of the IMU body coordinate system at time k+1. This represents the acceleration error of the IMU body coordinate system at time k. This represents the angular velocity deviation of the IMU body coordinate system at time k+1. This represents the angular velocity deviation of the IMU body coordinate system at time k; Constraints are imposed on visual information to obtain the ideal sensor pose by minimizing the visual residual caused by pixel reprojection. The pixel coordinates of the first landmark point, initially projected into the i-th camera coordinate system, are transformed and mapped to the j-th coordinate system. The difference between the transformed pixel coordinates and the directly observed corresponding pixel coordinates constitutes the visual residual, expressed as: in, Indicates projection onto the Cth... j The x-value in the camera coordinate system. Indicates projection onto the Cth... j The y-value in the camera coordinate system. Indicates projection onto the Cth... j The z-value in the camera coordinate system. This represents the inverse homogeneous transformation matrix for transforming from the IMU coordinate system back to the camera coordinate system. Let be the inverse matrix of the homogeneous transformation matrix from the world coordinate system to the j-th IMU coordinate system. Let T represent the homogeneous transformation matrix from the world coordinate system to the i-th IMU coordinate system. bc This represents the homogeneous transformation matrix from the camera coordinate system to the IMU coordinate system; the feature point is at the Cth position. i The pixel coordinates on the frame are λ represents the inverse depth; Taking into account prior information, visual data, and IMU constraint information, the optimal state vector integrating multi-sensor information is obtained through a nonlinear optimization method, expressed as: Where, ||r p -J p X|| 2 It is marginalized prior information. It is the IMU measurement residual. It is visual residual. This represents the noise covariance of the IMU pre-integration. This represents the noise covariance generated by visual perception.

7. The real-time positioning method for GPS signal blind zones based on SLAM-IMU coupling according to claim 6, characterized in that, The process involves iterating through multiple iterations to reach a globally optimal solution for the optimization variables, which include the IMU measurement residuals. Visual residual IMU pre-integrated noise covariance Covariance of noise generated by vision The initial iterative values ​​of the optimization variables are determined based on approximate values ​​measured by the sensors.

Images