Inertial navigation calibration method based on vision

Abstract

The invention relates to the technical field of calibration measurement, in particular to an inertial navigation calibration method based on vision. The method comprises the following steps: measuring target visual image data and target inertial navigation motion data corresponding to a target carrier in real time through a visual sensor and an inertial navigation sensor; performing time dimension alignment and static and dynamic motion feature fusion analysis on the target visual image data and the target inertial navigation motion data to obtain static and dynamic motion visual features of the target carrier; performing nearest neighbor feature point pair search and visual motion estimation on the static and dynamic motion visual features of the target carrier to obtain a visual motion translation vector of the target carrier and a visual motion rotation matrix of the target carrier; and performing vision-inertial navigation calibration on the error measurement corresponding to the inertial navigation sensor to obtain an error calibration compensation result corresponding to the inertial navigation sensor. According to the invention, high-precision calibration of inertial navigation can be realized, and the positioning precision and reliability of inertial navigation are improved.

Description

Technical Field

[0001] The present invention relates to the field of calibration measurement technology, and in particular to a vision-based inertial navigation calibration method. Background Art

[0002] The inertial navigation system is an important autonomous navigation system that measures the acceleration and angular velocity of the carrier and uses integral operations to infer the position, velocity and attitude information of the carrier. The traditional inertial navigation calibration method is mainly based on turntable tests. By testing the inertial navigation system in multiple positions and attitudes on a high-precision turntable, the error parameters of the inertial navigation sensor are estimated using the known turntable motion information. However, during the long-term operation of the inertial navigation system, the accumulation of sensor errors (such as gyroscope drift, accelerometer zero bias, etc.) will cause the navigation error to continue to increase. At the same time, the current method of combining visual technology with inertial navigation calibration is not perfect, and there is a lack of systematic and efficient inertial navigation calibration, which makes it impossible to accurately compensate for the errors of inertial navigation sensors. Summary of the invention

[0003] Based on this, it is necessary for the present invention to provide a vision-based inertial navigation calibration method to solve at least one of the above technical problems.

[0004] To achieve the above purpose, a vision-based inertial navigation calibration method includes the following steps: Step S1: installing and deploying corresponding visual sensors and inertial navigation sensors on the target carrier, and using the visual sensors and inertial navigation sensors to measure the target visual image data and target inertial navigation motion data corresponding to the target carrier in real time; aligning the target visual image data and the target inertial navigation motion data in time dimension by adding corresponding timestamps, so as to obtain the corresponding target visual image sequence and target inertial navigation motion sequence in the same time dimension; Step S2: Perform static and dynamic motion feature fusion analysis on the target visual image corresponding to the target visual image sequence at each time point to obtain static and dynamic motion visual features of the target carrier; Step S3: searching for the nearest neighbor feature point pairs of the static and dynamic motion visual features of the target carrier to obtain the target carrier motion matching feature point pairs; performing visual motion estimation on the target carrier based on the target carrier motion matching feature point pairs to obtain the target carrier visual motion state variables, including the target carrier visual motion translation vector and the target carrier visual motion rotation matrix; Step S4: Based on the visual motion state variables of the target carrier and the target inertial navigation motion sequence, visual-inertial navigation calibration is performed on the error measurement corresponding to the inertial navigation sensor to obtain the error calibration compensation result corresponding to the inertial navigation sensor.

[0005] Further, step S1 includes the following steps: Step S11: installing and deploying corresponding visual sensors and inertial navigation sensors on the target carrier; Step S12: Using a built-in synchronization signal generator and a hardware synchronization trigger mode to synchronously measure the target carrier using a visual sensor and an inertial navigation sensor to obtain target visual image data and target inertial navigation motion data; Step S13: cleaning and standardizing the target visual image data and the target inertial navigation motion data, so as to enhance contrast, remove noise and standardize pixels for the target visual image data, and remove duplicate data, outliers, compensate for missing values ​​and standardize motion parameters for the target inertial navigation motion data, so as to obtain target visual standard image data and target inertial navigation standard data; Step S14: aligning the target visual standard image data and the target inertial navigation standard data in time dimension by adding corresponding timestamps to obtain the corresponding target visual image sequence and target inertial navigation motion sequence in the same time dimension.

[0006] Further, step S11 includes the following steps: Conduct material and dynamic analysis on the target carrier to obtain the corresponding structural material, motion characteristics and environmental characteristics of the target carrier; Based on the structural material, motion characteristics and environmental characteristics of the target carrier, the sensor adaptation planning analysis of the visual sensor and the inertial navigation sensor is carried out to adapt the type, accuracy requirements and protection level of the visual sensor and the inertial navigation sensor to obtain the target carrier sensor adaptation planning results; Based on the target carrier sensor adaptation planning results, the corresponding visual sensors and inertial navigation sensors to be installed are designed for sensor installation topology, so as to determine the corresponding optimal installation angle and position of the visual sensor according to the change of the motion field of view of the target carrier, and ensure that the target area is covered to the greatest extent under various motion states. For the inertial navigation sensor, it is installed at the position with the least vibration and the easiest to sense motion according to the corresponding dynamic characteristics of the target carrier, so as to generate the target carrier sensor installation topology structure; Based on the target carrier sensor installation topology, the corresponding visual sensors and inertial navigation sensors are installed on the target carrier and sensor calibration is performed. The corresponding internal parameters of the visual sensors, including focal length and distortion coefficient, are calibrated in combination with the optical imaging principle. Gravity and turntable calibration equipment is used to calibrate the corresponding zero bias and scale factor of the inertial navigation sensor according to the gravity field and the law of rotational motion, so as to realize the corresponding calibration work of the visual sensors and inertial navigation sensors.

[0007] Furthermore, the visual sensor includes a binocular camera or a monocular camera combined with a laser radar.

[0008] Furthermore, the inertial navigation sensor includes a gyroscope and an accelerometer.

[0009] Further, step S12 includes the following steps: The built-in synchronization signal generator of the target carrier is used as the core node to synchronize the trigger topology with the visual sensor and the inertial navigation sensor in a star structure, and the feedback signals corresponding to each sensor are aggregated to the synchronization signal generator using a bus structure to achieve two-way communication, so as to generate the hardware synchronization trigger topology of the target carrier; Based on the hardware synchronous trigger topology of the target carrier, a hardware synchronous trigger method is used to send a synchronous trigger signal to the corresponding visual sensor and inertial navigation sensor to generate a target carrier synchronous trigger signal pulse; The target carrier is synchronized with the trigger signal pulse response to control the corresponding visual sensor and inertial navigation sensor, and the transmission delay of the synchronization signal pulse from the generator to the sensor is measured. At the same time, the corresponding measurement time is fine-tuned based on the transmission delay to perform synchronous real-time measurement of the target carrier to obtain the target visual image data and target inertial navigation motion data.

[0010] Further, step S2 includes the following steps: Step S21: obtaining the target visual images corresponding to the adjacent time frames before and after the time point through the target visual image corresponding to the target visual image sequence at each time point; Step S22: performing target carrier change analysis between target visual images corresponding to the corresponding time points based on the target visual images corresponding to the previous and next adjacent time frames at the time point, so as to compare and analyze the target edge and texture difference changes between the target visual images corresponding to the previous and next adjacent time frames at the time point, so as to obtain the target visual image difference changes between each time point and the previous and next adjacent time frames; Step S23: Based on the difference change of the target visual image between each time point and the adjacent time frames, the target visual image corresponding to the corresponding time point is judged as a static or dynamic image. If there is a difference change between each time point and the adjacent time frames, the target visual image corresponding to the time point is determined as a dynamic visual image; if there is no difference change between each time point and the adjacent time frames, the target visual image corresponding to the time point is determined as a static visual image. Step S24: performing static motion feature analysis on the static visual image to obtain static motion features of the target carrier; performing dynamic motion feature analysis on the dynamic visual image to obtain dynamic motion features of the target carrier; Step S25: performing static and dynamic motion feature fusion analysis on the static motion features of the target carrier and the dynamic motion features of the target carrier to obtain static and dynamic motion visual features of the target carrier.

[0011] Further, the static motion feature analysis of the static visual image in step S24 includes the following steps: Perform Gaussian blur and down-sampling operations on the static visual image to obtain the corresponding target static visual sub-image at each scale; The target edge structure feature analysis is performed on the target static visual sub-image corresponding to each scale, so as to extract the target edge contour and regional shape corresponding to the static visual image by edge detection and morphological operation, and obtain the static visual edge structure feature of the target carrier corresponding to each scale; Based on the static visual edge structure features of the target carrier at each scale, the static visual key points of the target static visual sub-image at each scale are extracted to extract the corresponding corner points and edge intersections, and obtain the key point set of the static visual image of the target carrier; Based on the key point set of the static visual image of the target carrier, the static motion feature analysis of the static visual image is performed by using scale-invariant feature transformation and accelerated robust features to obtain the static motion features of the target carrier, including the shape, texture and color features corresponding to the target carrier.

[0012] Furthermore, the dynamic motion feature analysis of the dynamic visual image in step S24 includes the following steps: Treat each dynamic visual image as a space-time cube, and analyze the grayscale changes and gradient changes between adjacent frames of dynamic visual images in the time dimension; Based on the grayscale changes and gradient changes between the dynamic visual images of adjacent frames, the motion clues between the dynamic visual images are statistically analyzed to obtain the dynamic motion direction and the dynamic motion speed of the target carrier; Based on the dynamic motion direction and dynamic motion speed of the target carrier and combined with the sparse optical flow method based on the Lucas-Kanade algorithm, the dynamic motion characteristics of the target carrier corresponding to the dynamic visual image are analyzed to obtain the dynamic motion characteristics of the target carrier, including the corresponding motion trajectory and speed changes of the target carrier during the movement.

[0013] Further, step S3 includes the following steps: Step S31: searching for the nearest neighbor feature point pairs of the static and dynamic motion visual features of the target carrier to obtain the target carrier motion matching feature point pairs; Step S32: performing a temporal and spatial feature point correlation analysis on the target carrier based on the target carrier motion matching feature point pairs and combining triangulation and epipolar geometry principles to analyze the relationship between the angle and side length changes of the triangles formed by the feature points in adjacent frame images of the target carrier, and obtain the temporal and spatial correlation relationship between the feature points of the target carrier; Step S33: performing a statistical analysis of the movement speed and angle of the movement trajectory corresponding to the target carrier based on the spatiotemporal correlation between the characteristic points of the target carrier, so as to obtain the movement speed and movement rotation angle corresponding to the target carrier on the three spatial axes; Step S34: Perform visual motion estimation on the target carrier based on the corresponding motion speed and motion rotation angle of the target carrier on the three axes of space to obtain the visual motion state variable of the target carrier, including the visual motion translation vector of the target carrier and the visual motion rotation matrix of the target carrier.

[0014] Further, step S31 includes the following steps: Performing motion visual deep deconstruction on the static and dynamic visual features of the target carrier, using Fourier transform and wavelet transform to analyze the frequency components and spatial distribution characteristics of static features, and establishing a time series model to analyze the corresponding change trends at different time points for dynamic features, so as to obtain the static and dynamic feature deconstruction data of the target carrier; Based on the static and dynamic feature deconstruction data of the target carrier, a spatiotemporal coding feature point index is constructed for each feature point in the static and dynamic motion visual feature of the target carrier, so that the three-dimensional spatial coordinates of the feature point are mapped into a code using a coding method based on the Hilbert curve in spatial coding, and the corresponding time code is generated according to the time sequence of the feature points in time coding, so as to obtain the static and dynamic visual index feature of the target carrier after spatiotemporal coding; According to the static and dynamic visual index features of the target carrier after spatiotemporal coding, the neighborhood of the feature points is divided, so that the corresponding static and dynamic visual feature points of the target carrier are divided into corresponding triangular neighborhoods based on the Delaunay triangulation part. For the feature points at the edge and corner of the target carrier and with obvious feature changes, a large-scale neighborhood is defined, and for the general feature points, a small-scale neighborhood is defined, so as to obtain the neighborhood division result of the multi-scale feature points of the target carrier; Based on the multi-scale feature point neighborhood division result of the target carrier, the feature points corresponding to the corresponding scale neighborhood within the static and dynamic motion visual features of the target carrier are screened for the nearest neighbor candidate points, so as to obtain the target carrier motion matching feature point pairs by calculating the corresponding energy values ​​and visual similarities between each feature point in the feature point neighborhood.

[0015] Furthermore, the target carrier visual motion translation vector described in step S34 is specifically: ; ; ; ; in, represents the target carrier visual motion translation vector, represents the translation component along the X axis, represents the translation component along the Y axis, represents the translation component along the Z axis, represents the motion measurement time, Indicates the movement speed of the target carrier on the X-axis. Indicates the movement speed of the target carrier on the Y axis. Indicates the movement speed of the target carrier on the Z axis.

[0016] Furthermore, the target carrier visual motion rotation matrix described in step S34 is specifically: The corresponding rotation matrix around the X axis is: ; The corresponding rotation matrix around the Y axis is: ; The corresponding rotation matrix around the Z axis is: ; in, Indicates the corresponding motion rotation angle of the target carrier on the X-axis. Indicates the corresponding motion rotation angle of the target carrier on the Y axis. Indicates the corresponding movement rotation angle of the target carrier on the Z axis.

[0017] Further, step S4 includes the following steps: Step S41: performing dual-sensor motion parameter characteristic analysis on the target carrier visual motion state variables and the target inertial navigation motion sequence to obtain a dual-sensor motion parameter characteristic data set; Step S42: performing dual sensor data fusion on the visual motion state variables of the target carrier and the target inertial navigation motion sequence based on the dual sensor motion parameter characteristic data set, so as to use the motion measurement of the inertial navigation sensor in a short time to assist in estimating the motion parameters corresponding to the fusion visual sensor, and obtain a dual sensor motion parameter fusion vector; Step S43: Based on the dual-sensor motion parameter fusion vector and using the extended Kalman filter, the error measurement corresponding to the inertial navigation sensor is calibrated with vision-inertial navigation, so that the error parameters corresponding to the inertial navigation sensor are used as state variables, including gyroscope drift and accelerometer zero bias, and the measurement values ​​of the visual sensor and the measurement values ​​of the inertial navigation sensor in the dual-sensor motion parameter fusion vector are used as observation variables. At the same time, the state variables are continuously updated through the filtering algorithm to realize real-time estimation and calibration compensation of the error measurement corresponding to the inertial navigation sensor, so as to obtain the error calibration compensation result corresponding to the inertial navigation sensor.

[0018] Furthermore, the process of dual-sensor motion parameter characteristic analysis in step S41 is specifically to analyze the translation error range between the translation vector corresponding to the visual sensor and the motion trajectory in different directions, and to analyze the rotation angle accuracy corresponding to the rotation matrix in different directions. For the motion measurement parameters corresponding to the inertial navigation sensor, the dynamic model is used to analyze the response characteristics of the angular velocity measured by the gyroscope and the acceleration measured by the accelerometer under different motion states, and to evaluate its zero bias stability and scale factor error.

[0019] Beneficial effects of the present invention: The vision-based inertial navigation calibration method proposed in the present invention has the beneficial effect of measuring the visual image data and inertial navigation motion data of the target on the target carrier in real time by installing and deploying the visual sensor and the inertial navigation sensor, and realizing the synchronization of the visual image and the inertial navigation data in the same time dimension by timestamping and aligning these data. This synchronization process ensures the timeliness and accuracy of the sensor data, which is the basis for subsequent data fusion and motion estimation. First, the visual sensor can provide a wealth of external environment information for the target, such as images, video data, etc., and the inertial navigation sensor measures the acceleration and angular velocity of the target through the accelerometer and gyroscope, and provides accurate dynamic information for the motion state of the target. Due to the different data acquisition frequencies and measurement dimensions of these two types of sensors, relying on any one of them alone may not be able to fully describe the motion of the target. Therefore, by aligning the timestamp, the visual image and the inertial navigation data are fused, and more accurate motion estimation can be achieved, which provides a basis for subsequent static and dynamic feature extraction, feature point matching and error correction, thereby laying a solid foundation for subsequent multi-sensor fusion and motion state estimation. Secondly, by fusing and analyzing the static and dynamic motion features of the target visual image at each time point, the visual image sequence contains information such as the position, posture, shape change, etc. of the target in a dynamic environment, and this information is usually manifested as the time change of visual features. By fusing static and dynamic motion features, the motion information of the target at different time points can be effectively extracted. Static features usually refer to the fixed attributes of the target such as the geometric shape and texture, while dynamic features represent the target's motion trajectory, speed, rotation and other behaviors that change over time. Fusion analysis of the two can improve the recognition and modeling accuracy of the target's motion state. The key to this step is the precise extraction and fusion of static and dynamic information in the target image sequence, so that the inertial navigation system can better understand the target's motion trajectory and its behavioral characteristics in three-dimensional space. By comprehensively analyzing static and dynamic features, the details of the target's motion can be more accurately captured in complex environments, thereby providing more accurate visual basis for subsequent feature point matching and motion estimation.Then, the nearest neighbor feature point pairs are searched for the static and dynamic motion visual features of the target carrier, and based on this, the visual motion of the target carrier is estimated. In the target visual image sequence, the feature points can be corners, edges or other image areas with unique identification in the image. These feature points can reflect the position change of the target in three-dimensional space. By matching these feature points, the target visual images at different time points can be linked to obtain the motion trajectory of the target. Using the matched feature points, combined with the motion estimation method based on the geometric model, the visual motion state of the target can be estimated, which usually includes the translation vector and the rotation matrix. The translation vector describes the movement of the target in space, while the rotation matrix describes the posture change of the target. Through this process, the motion trajectory of the target can be accurately inferred in a complex dynamic environment, and on this basis, the next step of error correction and fusion can be carried out, thereby improving the accuracy of target state estimation. Finally, the inertial navigation sensor errors of the target carrier are corrected through visual-inertial navigation calibration. The inertial navigation sensor itself will have certain errors, such as accelerometer bias, gyroscope noise, etc. These errors will affect the system's estimation accuracy of the target motion state. By combining the visual motion state variables of the target carrier with the data of the inertial navigation sensor, visual-inertial navigation calibration can be performed to effectively identify and compensate for these errors. The visual motion state variables provide the translation vector and rotation matrix of the target. This information can be compared with the measurement results of the inertial navigation sensor. The sensor errors are modeled and corrected through the optimization algorithm, which can accurately compensate for the errors of the inertial navigation sensor, thereby improving the positioning and navigation accuracy of the overall system, and significantly improving the motion state tracking and navigation performance of the target carrier. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments thereof made with reference to the following drawings: Figure 1 Schematic diagram of the process flow of the vision-based inertial navigation calibration method of the present invention; Figure 2 for Figure 1 Detailed step flow diagram of step S1; Figure 3 for Figure 1 Detailed step flow chart of step S2 in FIG. DETAILED DESCRIPTION

[0021] The technical method of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by technicians in this field without creative work are within the scope of protection of the present invention.

[0022] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. The functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor methods and / or microcontroller methods.

[0023] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.

[0024] To achieve this, please refer to Figures 1 to 3 The present invention provides a vision-based inertial navigation calibration method, the method comprising the following steps: Step S1: installing and deploying corresponding visual sensors and inertial navigation sensors on the target carrier, and using the visual sensors and inertial navigation sensors to measure the target visual image data and target inertial navigation motion data corresponding to the target carrier in real time; aligning the target visual image data and the target inertial navigation motion data in time dimension by adding corresponding timestamps, so as to obtain the corresponding target visual image sequence and target inertial navigation motion sequence in the same time dimension; Step S2: Perform static and dynamic motion feature fusion analysis on the target visual image corresponding to the target visual image sequence at each time point to obtain static and dynamic motion visual features of the target carrier; Step S3: searching for the nearest neighbor feature point pairs of the static and dynamic motion visual features of the target carrier to obtain the target carrier motion matching feature point pairs; performing visual motion estimation on the target carrier based on the target carrier motion matching feature point pairs to obtain the target carrier visual motion state variables, including the target carrier visual motion translation vector and the target carrier visual motion rotation matrix; Step S4: Based on the visual motion state variables of the target carrier and the target inertial navigation motion sequence, visual-inertial navigation calibration is performed on the error measurement corresponding to the inertial navigation sensor to obtain the error calibration compensation result corresponding to the inertial navigation sensor.

[0025] In the embodiment of the present invention, please refer to Figure 1FIG. 1 is a schematic diagram of a flow chart of a vision-based inertial navigation calibration method of the present invention. In this example, the vision-based inertial navigation calibration method includes the following steps: Step S1: installing and deploying corresponding visual sensors and inertial navigation sensors on the target carrier, and using the visual sensors and inertial navigation sensors to measure the target visual image data and target inertial navigation motion data corresponding to the target carrier in real time; aligning the target visual image data and the target inertial navigation motion data in time dimension by adding corresponding timestamps, so as to obtain the corresponding target visual image sequence and target inertial navigation motion sequence in the same time dimension; In an embodiment of the present invention, according to the structure and motion characteristics of the target carrier, the visual sensor (such as a binocular camera) and the inertial navigation sensor (such as a high-precision gyroscope and accelerometer) are installed at appropriate positions respectively. For the visual sensor, a special bracket is used to install it at the front end of the target carrier to ensure the coverage of the field of view; the inertial navigation sensor is installed at a position close to the center of gravity of the carrier and with small vibration to ensure the measurement accuracy. After the installation is completed, the sensor is started to start real-time measurement. The visual sensor shoots the surrounding environment of the target carrier at a set frame rate to generate target visual image data; the inertial navigation sensor continuously measures the acceleration, angular velocity and other information of the carrier to obtain the target inertial navigation motion data. During the data acquisition process, the synchronization signal generator records the precise timestamp when sending a trigger signal, and adds it to the target visual image data and the target inertial navigation motion data respectively. Subsequently, the data is sorted and matched according to the timestamp. If the timestamps are inconsistent, linear interpolation is used for fine-tuning, and finally the target visual image sequence and target inertial navigation motion sequence under the same time dimension are obtained.

[0026] Step S2: Perform static and dynamic motion feature fusion analysis on the target visual image corresponding to the target visual image sequence at each time point to obtain static and dynamic motion visual features of the target carrier; In an embodiment of the present invention, for a target visual image corresponding to each time point in a target visual image sequence, its static and dynamic properties are judged by comparing the images of adjacent time frames before and after the time point, so as to analyze the target edge and texture difference changes respectively by using Canny edge detection and grayscale co-occurrence matrix. If there is a difference, it is a dynamic visual image, otherwise it is a static visual image. For a static visual image, Gaussian blur and down sampling are first performed to construct an image pyramid, and then Canny edge detection and morphological operations are used to extract edge contours and regional shapes, Harris corner detection is used to extract key points, and finally scale-invariant feature transform (SIFT) and speeded-up robust features (SURF) are used to analyze shape, texture and color features. For a dynamic visual image, a space-time cube is constructed, and the grayscale and gradient changes of adjacent frames are analyzed. Based on this, motion clues are statistically analyzed to obtain motion direction and speed, and the sparse optical flow method of the Lucas-Kanade algorithm is combined to analyze the motion trajectory and speed changes. Finally, the static and dynamic motion features are associated and fused to finally obtain the static and dynamic motion visual features of the target carrier.

[0027] Step S3: searching for the nearest neighbor feature point pairs of the static and dynamic motion visual features of the target carrier to obtain the target carrier motion matching feature point pairs; performing visual motion estimation on the target carrier based on the target carrier motion matching feature point pairs to obtain the target carrier visual motion state variables, including the target carrier visual motion translation vector and the target carrier visual motion rotation matrix; In an embodiment of the present invention, feature points are extracted from the static and dynamic motion visual features of the target carrier, and the Euclidean distance between the feature points is calculated using a feature descriptor (such as a SIFT descriptor). A distance threshold is set. When the Euclidean distance is less than the threshold, two feature points are determined as a pair of nearest neighbor feature points. All feature points are searched and matched, and unstable or ambiguous feature point pairs are excluded, thereby obtaining target carrier motion matching feature point pairs. Based on these matching feature point pairs, in combination with triangulation and epipolar geometry principles, the angle and side length change relationship of the triangle formed by the feature points in adjacent frame images is analyzed, the spatiotemporal correlation between the feature points is determined, and the displacement and angle change of the feature points between adjacent frames are calculated based on the spatiotemporal correlation. The displacement is projected onto the three axes of space to obtain the motion speed, the rotation angle change is statistically calculated, and the target carrier visual motion translation vector is constructed using the motion speed. The target carrier visual motion rotation matrix is ​​obtained by conversion using the Rodriguez formula according to the rotation axis and the rotation angle, thereby obtaining the target carrier visual motion state variable.

[0028] Step S4: Based on the visual motion state variables of the target carrier and the target inertial navigation motion sequence, visual-inertial navigation calibration is performed on the error measurement corresponding to the inertial navigation sensor to obtain the error calibration compensation result corresponding to the inertial navigation sensor.

[0029] In an embodiment of the present invention, the state vector of the extended Kalman filter is defined by combining the visual motion state variables (including the visual motion translation vector and the visual motion rotation matrix) of the target carrier with the target inertial navigation motion sequence, taking the gyroscope drift and the accelerometer zero bias as state variables, taking the visual measurement values ​​in the visual motion state variables of the target carrier and the inertial navigation measurement values ​​in the target inertial navigation motion sequence as observation variables, in the prediction step of the extended Kalman filter, the state variables at the next moment are predicted according to the dynamic model of the inertial navigation sensor, in the update step, the error covariance between the observation variable and the predicted state variable is calculated, the Kalman gain is calculated by the Kalman gain formula, and then the state variable is adjusted, the prediction and update steps are repeated continuously, and the real-time estimation of the error measurement of the inertial navigation sensor is realized, the error parameters of the inertial navigation sensor are compensated according to the estimation result, and finally the error calibration compensation result corresponding to the inertial navigation sensor, such as the compensated gyroscope drift value and the accelerometer zero bias value, is obtained.

[0030] Further, step S1 includes the following steps: Step S11: installing and deploying corresponding visual sensors and inertial navigation sensors on the target carrier; Step S12: Using a built-in synchronization signal generator and a hardware synchronization trigger mode to synchronously measure the target carrier using a visual sensor and an inertial navigation sensor to obtain target visual image data and target inertial navigation motion data; Step S13: cleaning and standardizing the target visual image data and the target inertial navigation motion data, so as to enhance contrast, remove noise and standardize pixels for the target visual image data, and remove duplicate data, outliers, compensate for missing values ​​and standardize motion parameters for the target inertial navigation motion data, so as to obtain target visual standard image data and target inertial navigation standard data; Step S14: aligning the target visual standard image data and the target inertial navigation standard data in time dimension by adding corresponding timestamps to obtain the corresponding target visual image sequence and target inertial navigation motion sequence in the same time dimension.

[0031] As an embodiment of the present invention, refer to Figure 2 As shown, Figure 1 Detailed step flow diagram of step S1 in FIG. 1 , in this embodiment, step S1 includes the following steps: Step S11: installing and deploying corresponding visual sensors and inertial navigation sensors on the target carrier; In an embodiment of the present invention, by performing a detailed analysis of the structure and motion characteristics of the target carrier, for visual sensors, if the target carrier is a small drone, taking into account its field of view requirements during flight, a binocular camera with a wide-angle lens is selected and installed at the front end of the drone using a dedicated bracket, ensuring that the screws are tightened to ensure that the installation is firm and the angle can cover the main field of view in front of the drone. For inertial navigation sensors, such as high-precision gyroscopes and accelerometers, they are installed in a position close to the center of gravity and with less vibration according to the center of gravity position and vibration characteristics of the drone, and are fixed using welding and glue to ensure that the sensor can accurately sense the motion state of the drone.

[0032] Step S12: Using a built-in synchronization signal generator and a hardware synchronization trigger mode to synchronously measure the target carrier using a visual sensor and an inertial navigation sensor to obtain target visual image data and target inertial navigation motion data; In an embodiment of the present invention, a stable hardware connection structure is formed by connecting a built-in synchronization signal generator with a visual sensor and an inertial navigation sensor through a signal line. The synchronization signal generator sends a synchronization trigger signal at a preset time interval and frequency, and the signal is simultaneously transmitted to the visual sensor and the inertial navigation sensor. When the visual sensor receives the trigger signal, the shooting function is immediately started to capture the visual image around the target carrier, and the image data is stored in the data storage module in the form of a digital signal. At the same time, after receiving the trigger signal, the inertial navigation sensor starts to measure the acceleration, angular velocity and other motion parameters of the target carrier, and records these data in real time, and finally obtains the target visual image data and the target inertial navigation motion data.

[0033] Step S13: cleaning and standardizing the target visual image data and the target inertial navigation motion data, so as to enhance contrast, remove noise and standardize pixels for the target visual image data, and remove duplicate data, outliers, compensate for missing values ​​and standardize motion parameters for the target inertial navigation motion data, so as to obtain target visual standard image data and target inertial navigation standard data; In an embodiment of the present invention, a histogram equalization algorithm is used to enhance the contrast of the target visual image data so that the brightness distribution of the image is more uniform, and a median filtering algorithm is used to remove salt and pepper noise in the image. The algorithm replaces the grayscale value of each pixel with the median of the grayscale values ​​of the pixels in its neighborhood. Then, the pixel values ​​of the image are standardized by subtracting the pixel mean and dividing by the pixel standard deviation so that the pixel values ​​conform to the standard normal distribution. For the target inertial navigation motion data, duplicate data is found and deleted through data comparison, and outliers are identified and removed using statistical methods, such as a standard deviation-based method. For missing values, linear interpolation is used to compensate. Finally, the motion parameters are standardized so that their mean is 0 and their standard deviation is 1, and finally, target visual standard image data and target inertial navigation standard data are obtained.

[0034] Step S14: aligning the target visual standard image data and the target inertial navigation standard data in time dimension by adding corresponding timestamps to obtain the corresponding target visual image sequence and target inertial navigation motion sequence in the same time dimension.

[0035] In an embodiment of the present invention, during the data acquisition process, the synchronization signal generator records a precise timestamp while sending a trigger signal, and the timestamp is added to the corresponding target visual standard image data and target inertial navigation standard data respectively. In subsequent processing, the two sets of data are sorted and matched based on the timestamp. If the timestamps are not completely consistent, the linear interpolation method is used to fine-tune the data so that the target visual standard image data and the target inertial navigation standard data correspond one to one in the time dimension, and finally the corresponding target visual image sequence and target inertial navigation motion sequence in the same time dimension are obtained.

[0036] Further, step S11 includes the following steps: Conduct material and dynamic analysis on the target carrier to obtain the corresponding structural material, motion characteristics and environmental characteristics of the target carrier; In an embodiment of the present invention, the structural material of the target carrier is detected by using a material analysis instrument, such as using a spectrometer to determine the elemental composition of each part of the carrier, and then clarifying its material type. The actual movement of the target carrier is simulated by a dynamic testing device, and its acceleration, velocity, displacement and other parameters are measured. Its motion characteristics, including the smoothness and periodicity of the motion, are analyzed. At the same time, the environment in which the target carrier is located is monitored on the spot, and the temperature and humidity of the environment are measured using a temperature and humidity sensor, the air pressure is measured using a barometer, and the wind speed is measured using an anemometer, so as to obtain the temperature, humidity, air pressure, wind speed and other characteristics of the environment in which the target carrier is located.

[0037] Preferably, a sensor adaptation planning analysis is performed on the visual sensor and the inertial navigation sensor based on the structural material, motion characteristics and environmental characteristics of the target carrier, so as to adapt and plan the type, accuracy requirements and protection level of the visual sensor and the inertial navigation sensor, and obtain the target carrier sensor adaptation planning result; In the embodiment of the present invention, if the target carrier has a complex structure, a fast movement speed, and a large change in ambient light, and considering the need to accurately obtain three-dimensional spatial information, a binocular camera is planned to be used as a visual sensor, and its accuracy requirement is set to millimeter level, and the protection level is set to IP67 to adapt to complex environments. For inertial navigation sensors, due to the drastic changes in carrier movement, high-precision gyroscopes and accelerometers are selected, and the accuracy requirements reach 0.01° / h and 0.001m / s². The protection level is set to be able to withstand a certain degree of vibration and impact. If the target carrier moves relatively smoothly and the environment is relatively simple, a monocular camera combined with a laser radar can be planned as a visual sensor, and the accuracy and protection level can be determined according to actual needs. In this way, the sensor adaptation planning analysis is completed, and finally the target carrier sensor adaptation planning result is obtained.

[0038] Preferably, based on the target carrier sensor adaptation planning result, the corresponding visual sensor and inertial navigation sensor to be installed are subjected to sensor installation topology design, so as to determine the corresponding optimal installation angle and position of the visual sensor according to the change of the motion field of view corresponding to the target carrier, and ensure that the target area is covered to the greatest extent under various motion states, and the inertial navigation sensor is installed at the position with the least vibration and the easiest to sense motion according to the corresponding dynamic characteristics of the target carrier, so as to generate the target carrier sensor installation topology structure; In an embodiment of the present invention, by using a computer to simulate the change of the field of view of the target carrier in different motion states for the visual sensor, for example, if the target carrier is an aircraft, its take-off, cruising, landing and other states are simulated, and the optimal installation angle and position of the visual sensor are determined by analyzing the simulation results. For example, the binocular camera is installed at the front end of the aircraft with an elevation angle of 30° and a horizontal angle of 60° to ensure that the front target area can be covered to the greatest extent under various flight attitudes. For the inertial navigation sensor, according to the dynamic model of the target carrier, the part with the smallest vibration and accurate perception of movement is found, such as installing a gyroscope and an accelerometer near the center of gravity of the aircraft, and finally a sensor installation topology structure of the target carrier is generated.

[0039] Preferably, the corresponding visual sensors and inertial navigation sensors are installed on the target carrier based on the target carrier sensor installation topology and sensor calibration is performed to calibrate the corresponding internal parameters of the visual sensors, including focal length and distortion coefficient, in combination with the optical imaging principle, and use gravity and turntable calibration equipment to calibrate the corresponding zero bias and scale factor of the inertial navigation sensor according to the gravity field and the law of rotational motion to achieve the corresponding calibration work of the visual sensors and inertial navigation sensors.

[0040] In an embodiment of the present invention, the visual sensor and the inertial navigation sensor are accurately installed on the target carrier using a dedicated installation tool according to the target carrier sensor installation topology. For the visual sensor, the optical imaging principle is used to shoot a standard chessboard image and use the Zhang Zhengyou calibration method to calculate the focal length and distortion coefficient of the visual sensor, and then adjust and calibrate. For the inertial navigation sensor, the target carrier is placed on a gravity and turntable calibration device, and the zero bias and scale factor of the gyroscope and accelerometer are measured and calculated according to the direction and size of the gravity field and the rotational motion law of the turntable, and then they are calibrated and adjusted, thereby finally achieving the calibration of the two sensors.

[0041] Furthermore, the visual sensor includes a binocular camera or a monocular camera combined with a laser radar.

[0042] Furthermore, the inertial navigation sensor includes a gyroscope and an accelerometer.

[0043] Further, step S12 includes the following steps: The built-in synchronization signal generator of the target carrier is used as the core node to synchronize the trigger topology with the visual sensor and the inertial navigation sensor in a star structure, and the feedback signals corresponding to each sensor are aggregated to the synchronization signal generator using a bus structure to achieve two-way communication, so as to generate the hardware synchronization trigger topology of the target carrier; In an embodiment of the present invention, a synchronization signal generator is used as a core node in a target carrier. When connected in a star structure, independent lines are drawn from the synchronization signal generator to connect the visual sensor and the inertial navigation sensor respectively. Each line has stable electrical characteristics to ensure the accuracy of signal transmission. At the same time, a bus structure is adopted to connect the feedback signal lines of the visual sensor and the inertial navigation sensor to the same bus, and then the bus is connected to the synchronization signal generator. In this way, the trigger signal emitted by the synchronization signal generator can be independently and accurately transmitted to each sensor, and the feedback signal of the sensor can also be aggregated to the synchronization signal generator through the bus to achieve two-way communication, and finally form a hardware synchronization trigger topology of the target carrier. For example, signal lines of specific specifications are used for connection to ensure the stability and anti-interference ability of signal transmission.

[0044] Preferably, based on the hardware synchronization trigger topology of the target carrier, a synchronization trigger signal is sent to the corresponding visual sensor and inertial navigation sensor in a hardware synchronization trigger mode to generate a synchronization trigger signal pulse of the target carrier; In an embodiment of the present invention, after the hardware synchronization trigger topology structure of the target carrier is built, the synchronization signal generator is started. The synchronization signal generator generates a synchronization trigger signal according to a preset time interval and signal strength. These signals are transmitted to the visual sensor and the inertial navigation sensor respectively through the star-structured line. The synchronization trigger signal has a specific pulse width and frequency to ensure that the sensor can be accurately triggered to work. For example, the pulse width of the signal is set to 100 microseconds and the frequency is 10 Hz. During the signal transmission process, the electrical characteristics of the line and the signal enhancement equipment are used to ensure the strength and stability of the signal, thereby generating a target carrier synchronization trigger signal pulse, so that the visual sensor and the inertial navigation sensor can start working at the same time.

[0045] Preferably, the target carrier is synchronized with the trigger signal pulse response to control the corresponding visual sensor and inertial navigation sensor, and the transmission delay of the synchronization signal pulse from the generator to the sensor is measured. At the same time, the corresponding measurement time is fine-tuned based on the transmission delay to perform synchronous real-time measurement of the target carrier to obtain target visual image data and target inertial navigation motion data.

[0046] In an embodiment of the present invention, after the target carrier synchronizes the trigger signal pulse to the visual sensor and the inertial navigation sensor, the sensor starts to respond. In order to measure the transmission delay of the synchronization signal pulse from the generator to the sensor, a high-precision time measurement device is set at the sensor end. When the sensor receives the signal pulse, the time is recorded and compared with the time when the signal generator sends the signal to calculate the transmission delay. For example, the signal generator sends a signal at time t1, and the visual sensor receives the signal at time t2. The transmission delay is t2-t1. According to the measured transmission delay, the measurement time of the sensor is fine-tuned. If the transmission delay of the visual sensor is large, its measurement time is advanced by the corresponding delay time to ensure that the two sensors can perform measurements synchronously. After such synchronous real-time measurement, the visual sensor collects the target visual image data, and the inertial navigation sensor collects the target inertial motion data.

[0047] Further, step S2 includes the following steps: Step S21: obtaining the target visual images corresponding to the adjacent time frames before and after the time point through the target visual image corresponding to the target visual image sequence at each time point; Step S22: performing target carrier change analysis between target visual images corresponding to the corresponding time points based on the target visual images corresponding to the previous and next adjacent time frames at the time point, so as to compare and analyze the target edge and texture difference changes between the target visual images corresponding to the previous and next adjacent time frames at the time point, so as to obtain the target visual image difference changes between each time point and the previous and next adjacent time frames; Step S23: Based on the difference change of the target visual image between each time point and the adjacent time frames, the target visual image corresponding to the corresponding time point is judged as a static or dynamic image. If there is a difference change between each time point and the adjacent time frames, the target visual image corresponding to the time point is determined as a dynamic visual image; if there is no difference change between each time point and the adjacent time frames, the target visual image corresponding to the time point is determined as a static visual image. Step S24: performing static motion feature analysis on the static visual image to obtain static motion features of the target carrier; performing dynamic motion feature analysis on the dynamic visual image to obtain dynamic motion features of the target carrier; Step S25: performing static and dynamic motion feature fusion analysis on the static motion features of the target carrier and the dynamic motion features of the target carrier to obtain static and dynamic motion visual features of the target carrier.

[0048] As an embodiment of the present invention, refer to Figure 3 As shown, Figure 1 Detailed step flow diagram of step S2 in FIG. 1 , in this embodiment, step S2 includes the following steps: Step S21: obtaining the target visual images corresponding to the adjacent time frames before and after the time point through the target visual image corresponding to the target visual image sequence at each time point; In an embodiment of the present invention, the target visual image sequence is stored in chronological order, each time point corresponds to a frame of the target visual image, and for a specific time point t in the sequence, the corresponding target visual image is I_t. Through the time index, the target visual image I_{t - 1} corresponding to the previous frame and the target visual image I_{t + 1} corresponding to the next frame of the time point are directly obtained. For example, if the target visual image sequence has a total of 100 frames, when the time point t = 50, through the index operation, the 49th frame image is read from the storage as I_{49}, the 50th frame image is read as I_{50}, and the 51st frame image is read as I_{51}, so as to obtain the target visual images corresponding to the adjacent time frames before and after each time point in this way.

[0049] Step S22: performing target carrier change analysis between target visual images corresponding to the corresponding time points based on the target visual images corresponding to the previous and next adjacent time frames at the time point, so as to compare and analyze the target edge and texture difference changes between the target visual images corresponding to the previous and next adjacent time frames at the time point, so as to obtain the target visual image difference changes between each time point and the previous and next adjacent time frames; In an embodiment of the present invention, by comparing the target visual image I_t with I_{t-1} and I_{t+1} at each time point t, first, the Canny edge detection algorithm is used to extract the target edges of I_t, I_{t-1} and I_{t+1} respectively, and then the difference between the edge images is calculated, for example, by calculating the absolute value of the difference between the corresponding pixel positions, and counting the number and distribution of the difference pixels. For texture differences, the gray level co-occurrence matrix method is used to calculate the texture features of I_t, I_{t-1} and I_{t+1}, such as contrast, correlation, etc., and then the differences in these texture feature values ​​are compared, the edge differences and texture differences are combined, and finally the target visual image difference changes between each time point and the adjacent time frames are obtained.

[0050] Step S23: Based on the difference change of the target visual image between each time point and the adjacent time frames, the target visual image corresponding to the corresponding time point is judged as a static or dynamic image. If there is a difference change between each time point and the adjacent time frames, the target visual image corresponding to the time point is determined as a dynamic visual image; if there is no difference change between each time point and the adjacent time frames, the target visual image corresponding to the time point is determined as a static visual image. In the embodiment of the present invention, by setting the edge difference threshold and the texture difference threshold, for each time point t, if the number of edge difference pixels between the target visual image I_t and I_{t - 1}, I_{t + 1} exceeds the edge difference threshold, or the difference in texture feature values ​​exceeds the texture difference threshold, it is determined that there is a difference change between the time point and the adjacent time frames before and after, and I_t is determined to be a dynamic visual image. On the contrary, if the difference in the number of edge difference pixels and the difference in texture feature values ​​do not exceed the corresponding thresholds, it is determined that there is no difference change, and I_t is determined to be a static visual image. For example, the edge difference threshold is set to 100 pixels, and the texture feature difference threshold is set to 0.1. When the number of edge difference pixels between I_t and I_{t - 1} is 150, I_t is a dynamic visual image.

[0051] Step S24: performing static motion feature analysis on the static visual image to obtain static motion features of the target carrier; performing dynamic motion feature analysis on the dynamic visual image to obtain dynamic motion features of the target carrier; In an embodiment of the present invention, a Gaussian blur and down-sampling operation is first performed on a static visual image to construct an image pyramid of different scales. At each scale, Canny edge detection and morphological operations are used to extract the edge contour and regional shape of the target. The Harris corner detection algorithm is used to extract corner points and edge intersections to obtain a key point set. Then, the scale-invariant feature transform (SIFT) and speeded-up robust feature (SURF) algorithms are used to analyze the shape, texture and color features of the target carrier to obtain the static motion features of the target carrier. For a dynamic visual image, it is constructed into a space-time cube, and the grayscale and gradient changes of adjacent frames are analyzed in the time dimension. Based on these changes, a motion clue statistical analysis is performed to obtain the dynamic motion direction and speed of the target carrier. The motion trajectory and speed change of the target carrier are analyzed in combination with the sparse optical flow method based on the Lucas-Kanade algorithm, and finally the dynamic motion features of the target carrier are obtained.

[0052] Step S25: performing static and dynamic motion feature fusion analysis on the static motion features of the target carrier and the dynamic motion features of the target carrier to obtain static and dynamic motion visual features of the target carrier.

[0053] In an embodiment of the present invention, the shape, texture and color features in the static motion features of the target carrier are integrated with the motion direction, speed, motion trajectory and other features in the dynamic motion features of the target carrier. For the shape features, it is analyzed whether the target shape changes during the dynamic motion. If there is a change, the situation and law of the change are recorded. For the texture features, the changes in the texture during the dynamic motion are observed, such as the flow or deformation of the texture. For the color features, the distribution and changes of the color in the dynamic process are statistically analyzed, and the dynamic motion information is associated with the static feature information. For example, the motion trajectory is combined with the shape and color features of the target, and the state and changes of the target carrier at different times are comprehensively analyzed, and finally the static and dynamic motion visual features of the target carrier are obtained.

[0054] Further, the static motion feature analysis of the static visual image in step S24 includes the following steps: Perform Gaussian blur and down-sampling operations on the static visual image to obtain the corresponding target static visual sub-image at each scale; In an embodiment of the present invention, a Gaussian filter is used to perform blur processing on a static visual image. The kernel size of the Gaussian filter is set to 5x5, and the standard deviation is 1.0. The Gaussian filter is applied to each pixel of the image, and a new pixel value is calculated through a convolution operation to smooth the image and reduce the influence of noise. Then, a downsampling operation is performed, and an alternate row and column sampling method is adopted, that is, a pixel is selected every other row and column, and the size of the image is reduced to half of the original size. The above Gaussian blurring and downsampling process is repeated to construct image pyramids of different scales. For example, the initial image is 512x512 pixels, and a sub-image of 256x256 pixels is obtained after the first processing, and a sub-image of 128x128 pixels is obtained after the second processing, and so on, and finally the corresponding target static visual sub-image at each scale is obtained.

[0055] Preferably, the target edge structure feature analysis is performed on the target static visual sub-image corresponding to each scale, so as to extract the target edge contour and region shape corresponding to the static visual image by edge detection and morphological operation, and obtain the target carrier static visual edge structure feature corresponding to each scale; In an embodiment of the present invention, edge detection is performed using the Canny edge detection algorithm for a static visual sub-image of a target at each scale. First, Gaussian smoothing is performed on the image. Then, the gradient amplitude and direction of the image are calculated. Next, non-maximum suppression is performed to remove non-edge pixels. Finally, the true edge pixels are determined by double threshold processing. After the edge image is obtained, morphological operations are used for optimization. A dilation operation is used to fill small gaps in the edge to make the edge more continuous. An erosion operation is then used to remove small noise points around the edge. The edge contour and regional shape of the target are extracted by searching for connected areas. For example, for a sub-image containing a circular target, these operations can accurately extract the edge contour and regional shape of the circle, and finally the static visual edge structural features of the target carrier corresponding to each scale are obtained.

[0056] Preferably, based on the static visual edge structure features of the target carrier at each scale, static visual key points are extracted from the target static visual sub-image at each scale to extract the corresponding corner points and edge intersections, and obtain a static visual image key point set of the target carrier; In an embodiment of the present invention, the Harris corner detection algorithm is used to extract corner points according to the static visual edge structure characteristics of the target carrier at each scale. The Harris corner detection algorithm calculates the grayscale change rate of the image in different directions to find areas with large grayscale changes. These areas are corner points. For the extraction of edge intersections, they are determined by analyzing the intersection of edges in the edge image. Specifically, each pixel in the edge image is traversed to check the edge conditions of the surrounding pixels. When multiple edges are found to intersect here, the point is marked as an edge intersection. The corner points and edge intersections extracted at each scale are summarized to obtain a key point set of the static visual image of the target carrier. For example, 20 corner points and 10 edge intersections are extracted from a sub-image of one scale. Such points at all scales are merged to form a complete key point set, and finally a key point set of the static visual image of the target carrier is obtained.

[0057] Preferably, static motion feature analysis is performed on the static visual image based on the key point set of the static visual image of the target carrier and using scale-invariant feature transformation and accelerated robust features to obtain the static motion features of the target carrier, including shape, texture and color features corresponding to the target carrier.

[0058] In an embodiment of the present invention, a static motion feature analysis is performed based on a set of key points of a static visual image of a target carrier, using a scale-invariant feature transform (SIFT) and a speeded up robust feature (SURF) algorithm. For the SIFT algorithm, a local area is constructed around each key point, and the gradient amplitude and direction in the area are calculated to generate a feature descriptor. The SURF algorithm extracts features around the key points by calculating the integral image of the image. The shape features of the target carrier are analyzed through these feature descriptors. For example, the distribution and connection relationship of the feature points are used to determine whether the target is a circle, a rectangle or other shape. For texture features, the gradient variation law in the feature descriptor is analyzed to determine the thickness, density and other features of the texture. In terms of color features, the pixel color information at the location of the key points is combined to statistically analyze the distribution and proportion of the colors, thereby obtaining the shape, texture and color features corresponding to the target carrier, and finally obtaining the static motion features of the target carrier.

[0059] Furthermore, the dynamic motion feature analysis of the dynamic visual image in step S24 includes the following steps: Treat each dynamic visual image as a space-time cube, and analyze the grayscale changes and gradient changes between adjacent frames of dynamic visual images in the time dimension; In the embodiment of the present invention, a series of dynamic visual images are arranged in time order to construct a space-time cube, in which the spatial dimension is composed of the length and width of the image, and the time dimension is represented by the number of frames of the image. For two adjacent frames of dynamic visual images in the time dimension, the grayscale difference of their corresponding pixels is first calculated. For example, the grayscale difference of the previous frame of image is calculated. and the next frame At the same coordinates Subtract the grayscale value of the pixel at At the same time, the Sobel operator is used to calculate the gradients of the two frames in the horizontal and vertical directions respectively, and then the difference of the gradients at the corresponding positions between adjacent frames is calculated to analyze the gradient changes. Through these operations, the grayscale and gradient change characteristics of adjacent frame images in the time dimension are fully understood.

[0060] Preferably, based on the grayscale change and gradient change between the dynamic visual images of adjacent frames, a motion clue statistical analysis is performed between each dynamic visual image to obtain the dynamic motion direction and the dynamic motion speed of the target carrier; In the embodiment of the present invention, motion clues are counted according to the grayscale changes and gradient changes of the dynamic visual images of adjacent frames. The area with large grayscale changes often indicates that the target carrier has obvious movement. For the gradient change, its direction and magnitude can reflect the movement trend of the target. By counting the grayscale and gradient changes of multiple areas in the image, the approximate movement direction of the target carrier is determined. For example, if the grayscale value of a certain area continues to increase and the gradient direction points to a specific direction, it can be determined that the target is moving in this direction. When calculating the movement speed, the amplitude and time interval of the grayscale change are used for estimation. Assuming that the time interval between adjacent frames is , the grayscale change of a certain area is , by establishing an empirical model of grayscale change and motion speed ( is a coefficient determined by experiment), calculate the movement speed of the target carrier in different areas, and obtain the overall dynamic movement speed of the target carrier by combining the conditions of each area.

[0061] Preferably, the dynamic motion characteristics of the target carrier corresponding to the dynamic visual image are analyzed based on the dynamic motion direction and the dynamic motion speed of the target carrier and combined with the sparse optical flow method based on the Lucas-Kanade algorithm to obtain the dynamic motion characteristics of the target carrier, including the corresponding motion trajectory and speed changes of the target carrier during the movement.

[0062] In an embodiment of the present invention, a sparse optical flow method based on the Lucas-Kanade algorithm is used to analyze dynamic visual images based on the dynamic motion direction and dynamic motion speed of the target carrier. First, some representative feature points, such as corner points, are selected in the initial frame image. Then, based on the assumption of the Lucas-Kanade algorithm, that is, pixels in a local area have the same motion, the grayscale information around these feature points in adjacent frame images is used to calculate the optical flow. The optical flow represents the motion vector of the feature point between adjacent frames, and its direction and size correspond to the motion direction and speed of the target, respectively. By tracking the motion of these feature points in multiple frame images and connecting the positions of the feature points in each frame, the motion trajectory of the target carrier is obtained. At the same time, the change in the optical flow size of each feature point in different frames is recorded to analyze the speed change of the target carrier, and finally the dynamic motion characteristics of the target carrier are fully obtained.

[0063] Further, step S3 includes the following steps: Step S31: searching for the nearest neighbor feature point pairs of the static and dynamic motion visual features of the target carrier to obtain the target carrier motion matching feature point pairs; In an embodiment of the present invention, static and dynamic motion visual feature points are extracted from a multi-frame image sequence of a target carrier, for example, a SIFT (Scale Invariant Feature Transform) algorithm is used to extract feature points and their feature descriptors. For feature points in each frame image, the nearest neighbor feature points are searched in adjacent frame images, and the Euclidean distance of the feature descriptors between the feature points is calculated. The smaller the distance, the more similar the features are. A distance threshold is set. When the Euclidean distance is less than the threshold, the two feature points are determined as a pair of nearest neighbor feature points. Such search and matching operations are performed on all feature points to exclude those feature point pairs with unstable or ambiguous matching, and finally the target carrier motion matching feature point pairs are obtained. For example, in an image frame containing 1,000 feature points, 500 pairs of motion matching feature point pairs are obtained after matching and screening.

[0064] Step S32: performing a temporal and spatial feature point correlation analysis on the target carrier based on the target carrier motion matching feature point pairs and combining triangulation and epipolar geometry principles to analyze the relationship between the angle and side length changes of the triangles formed by the feature points in adjacent frame images of the target carrier, and obtain the temporal and spatial correlation relationship between the feature points of the target carrier; In an embodiment of the present invention, by utilizing the target carrier motion matching feature point pairs and combining the triangulation principle, for the matching feature point pairs in adjacent frame images, a triangle composed of the feature points and the camera optical center is constructed according to the camera's intrinsic and extrinsic parameters, and the side length and angle of the triangle are calculated by the known camera parameters and the pixel coordinates of the feature points in the image. At the same time, based on the epipolar geometry principle, the epipolar constraint relationship between the feature points in adjacent frame images is determined, that is, the epipolar line where the feature points appear in another frame image, and the changes in the angle and side length of the triangle formed by the feature points in the adjacent frame images are analyzed, such as the increase or decrease in the side length, the increase or decrease in the angle, and based on these change relationships, the spatiotemporal correlation relationship between the feature points is established, and the corresponding relationship between the feature points in different time and space is clarified, such as how a certain feature point moves and changes in adjacent frames, and finally the spatiotemporal correlation relationship between the feature points of the target carrier is obtained.

[0065] Step S33: performing a statistical analysis of the movement speed and angle of the movement trajectory corresponding to the target carrier based on the spatiotemporal correlation between the characteristic points of the target carrier, so as to obtain the movement speed and movement rotation angle corresponding to the target carrier on the three spatial axes; In an embodiment of the present invention, the displacement and angle change of each feature point between adjacent frames are determined according to the spatiotemporal correlation between the feature points of the target carrier. For each feature point, its position change between adjacent frames is calculated, and the movement speed of the feature point is obtained in combination with the time interval. The movement speeds of all feature points are projected onto the three spatial axes (X, Y, and Z axes), and the projected speed values ​​are statistically analyzed, such as calculating the average value, to obtain the movement speed of the target carrier on the three spatial axes. At the same time, the rotation of the feature points between adjacent frames is analyzed, and the rotation angle is obtained by calculating the direction change of the feature points. The rotation angle of all feature points is statistically analyzed to obtain the movement rotation angle of the target carrier in space. For example, after analyzing the movement of 100 feature points, the average movement speed of the target carrier on the X-axis is 0.5 m / s, on the Y-axis is 0.3 m / s, and on the Z-axis is 0.2 m / s, and the movement rotation angle is 10 degrees. Finally, the corresponding movement speed and movement rotation angle of the target carrier on the three spatial axes are obtained.

[0066] Step S34: Perform visual motion estimation on the target carrier based on the corresponding motion speed and motion rotation angle of the target carrier on the three axes of space to obtain the visual motion state variable of the target carrier, including the visual motion translation vector of the target carrier and the visual motion rotation matrix of the target carrier.

[0067] In the embodiment of the present invention, the visual motion translation vector of the target carrier is constructed according to the corresponding motion speed of the target carrier on the three axes of space, and the motion speed on the X, Y, and Z axes are respectively used as the three components of the translation vector, for example, the translation vector ,in , , are the movement speeds on the X, Y, and Z axes respectively. For the movement rotation angle of the target carrier, according to the rotation axis and rotation angle, the Rodriguez formula is used to convert it into a rotation matrix. The rotation axis can be determined by analyzing the rotation direction of the feature point, and the rotation angle is the movement rotation angle obtained previously. Through these calculations, the visual motion state variables of the target carrier are obtained, that is, the visual motion translation vector of the target carrier and the visual motion rotation matrix of the target carrier, which are used for subsequent inertial navigation calibration and other operations. The specific visual motion translation vector of the target carrier is: ; ; ; ; in, represents the target carrier visual motion translation vector, represents the translation component along the X axis, represents the translation component along the Y axis, represents the translation component along the Z axis, represents the motion measurement time, Indicates the movement speed of the target carrier on the X-axis. Indicates the movement speed of the target carrier on the Y axis. Indicates the motion speed of the target carrier on the Z axis; further, the visual motion rotation matrix of the target carrier is specifically: The corresponding rotation matrix around the X axis is: ; The corresponding rotation matrix around the Y axis is: ; The corresponding rotation matrix around the Z axis is: ; in, Indicates the corresponding motion rotation angle of the target carrier on the X-axis. Indicates the corresponding motion rotation angle of the target carrier on the Y axis. It represents the corresponding motion rotation angle of the target carrier on the Z axis, and finally obtains the visual motion state variable of the target carrier.

[0068] Further, step S31 includes the following steps: Performing motion visual deep deconstruction on the static and dynamic visual features of the target carrier, using Fourier transform and wavelet transform to analyze the frequency components and spatial distribution characteristics of static features, and establishing a time series model to analyze the corresponding change trends at different time points for dynamic features, so as to obtain the static and dynamic feature deconstruction data of the target carrier; In an embodiment of the present invention, by considering the static features of the target carrier, the visual image of the target carrier is regarded as a two-dimensional signal, and a Fourier transform is performed on it to convert the image from the spatial domain to the frequency domain. The distribution of different frequency components is determined by analyzing the spectrum diagram. For example, low-frequency components may correspond to the overall contour of the image, and high-frequency components correspond to the detailed texture of the image. Then, the image is decomposed into sub-bands of different scales and directions using wavelet transform, and the distribution characteristics of static features at different spatial positions are further analyzed, such as the spatial distribution of features such as edges and corners. For dynamic features, a visual image sequence of the target carrier over a period of time is collected, and the position and grayscale value of the feature points are used as the observation values ​​of the time series. An autoregressive integrated moving average (ARIMA) model is established to predict the feature change trend at future time points based on historical observations. For example, by analyzing the position changes of feature points at different time points, the movement speed and direction changes of the target carrier are determined, and finally the static and dynamic feature deconstruction data of the target carrier are obtained.

[0069] Preferably, a spatiotemporal coding feature point index is constructed for each feature point in the static and dynamic motion visual feature of the target carrier based on the static and dynamic feature deconstruction data, so that the three-dimensional spatial coordinates of the feature points are mapped into codes using a coding method based on the Hilbert curve in spatial coding, and corresponding time codes are generated according to the time sequence of the feature points in time coding, so as to obtain the static and dynamic visual index features of the target carrier after spatiotemporal coding; In an embodiment of the present invention, the three-dimensional spatial coordinates of each feature point are determined by deconstructing data based on the static and dynamic features of the target carrier. For spatial coding, the Hilbert curve coding method is adopted to divide the three-dimensional space into a number of small cube grids, each small cube corresponds to a unique Hilbert code value, and the small cube in which the feature point is located is determined according to the three-dimensional coordinates of the feature point. The Hilbert code value of the small cube is used as the spatial coding of the feature point. In terms of time coding, the time point when the feature point first appears in the image sequence is recorded, and an increasing integer is assigned to each feature point in chronological order as the time code. For example, the time code of the first feature point that appears is 1, the time code of the second feature point that appears is 2, and so on. The spatial coding and time coding are combined together to obtain the static and dynamic visual index features of the target carrier after spatiotemporal coding, which is convenient for subsequent rapid retrieval and analysis of the feature points.

[0070] Preferably, feature point neighborhood division is performed according to the static and dynamic visual index features of the target carrier after spatiotemporal encoding, so as to divide the corresponding static and dynamic visual feature points of the target carrier into corresponding triangular neighborhoods based on the Delaunay triangulation part, and define large-scale neighborhoods for feature points at the edge and corner of the target carrier and with obvious feature changes, and define small-scale neighborhoods for general feature points, so as to obtain the target carrier multi-scale feature point neighborhood division result; In an embodiment of the present invention, the three-dimensional spatial coordinates of the feature points are obtained by utilizing the spatial coding information in the static and dynamic visual index features of the target carrier after spatiotemporal coding, and the Delaunay triangulation algorithm is used to connect all the feature points into non-overlapping triangles, so that the circumscribed circle of each triangle does not contain other feature points. For the feature points at the edge and corner positions of the target carrier, by analyzing the distribution and changes of the surrounding feature points, it is determined that their features have changed significantly. For example, the grayscale value around the edge feature point changes greatly, and the gradient of the corner feature point changes significantly in multiple directions. A large-scale neighborhood is defined for these feature points, such as a circular area with a radius of 5 feature point spacings with the feature point as the center. For general feature points, that is, feature points with relatively gentle changes in the surrounding features, a small-scale neighborhood is defined, such as a circular area with a radius of 2 feature point spacings with the feature point as the center, and finally a multi-scale feature point neighborhood division result of the target carrier is obtained.

[0071] Preferably, based on the result of the multi-scale feature point neighborhood division of the target carrier, the feature points corresponding to the corresponding scale neighborhood within the static and dynamic motion visual features of the target carrier are screened for the nearest neighbor candidate points, so as to calculate the corresponding energy values ​​and visual similarities between each feature point in the feature point neighborhood, and screen out feature points with high energy values ​​and high visual similarities as the nearest neighbor candidate points, so as to obtain the target carrier motion matching feature point pairs.

[0072] In an embodiment of the present invention, based on the neighborhood division result of the multi-scale feature points of the target carrier, for the corresponding scale neighborhood of each feature point, the energy value and the visual similarity between the feature points in the neighborhood are calculated. For the calculation of the energy value, the gray value change rate of the feature point is used as the measure of the energy. For example, the gradient amplitude of the gray value within a certain range around the feature point is calculated. The larger the gradient amplitude, the higher the energy value. For the calculation of the visual similarity, the Euclidean distance between the feature descriptors (such as SIFT descriptors) of the feature points is used as a measurement standard. The smaller the Euclidean distance, the higher the visual similarity. For each feature point, the feature points with energy values ​​higher than a set threshold (such as the top 30% of the energy value) and visual similarities higher than the set threshold (such as the top 30% of the Euclidean distance) are screened out in its neighborhood as the nearest neighbor candidate points. These nearest neighbor candidate points are combined in pairs to finally obtain the target carrier motion matching feature point pairs for subsequent motion analysis and inertial navigation calibration.

[0073] Further, step S4 includes the following steps: Step S41: performing dual-sensor motion parameter characteristic analysis on the target carrier visual motion state variables and the target inertial navigation motion sequence to obtain a dual-sensor motion parameter characteristic data set; In an embodiment of the present invention, the visual sensor is used to take the motion trajectory of the target carrier in the actual scene as a reference. For example, in a set rectangular motion trajectory, the translation vectors of the visual sensor in the three directions of X, Y, and Z are recorded, and the real motion trajectory is obtained by a high-precision measuring device. The translation vector measured by the visual sensor is compared, and the translation error range between the real trajectory and the real trajectory in different directions is calculated. For the rotation matrix, in the process of rotation in different directions, the actual rotation angle is recorded by a precise angle measuring instrument, and compared with the rotation angle corresponding to the rotation matrix output by the visual sensor, the rotation angle accuracy is obtained. For inertial navigation sensors, dynamic models such as Newton's second law are used to install gyroscopes and accelerometers on a test bench that simulates different motion states (such as uniform linear motion, accelerated linear motion, uniform circular motion, etc.). In each motion state, the angular velocity measured by the gyroscope and the acceleration measured by the accelerometer are recorded. Through long-term observation and data analysis, the zero bias stability of the gyroscope, that is, the drift of the measured value in a static state, is evaluated; the scale factor error of the accelerometer is calculated, for example, when the standard acceleration is known, the difference between the measured value of the accelerometer and the standard value is compared, and finally it is compiled into a dual-sensor motion parameter characteristic data set.

[0074] Step S42: performing dual sensor data fusion on the visual motion state variables of the target carrier and the target inertial navigation motion sequence based on the dual sensor motion parameter characteristic data set, so as to use the motion measurement of the inertial navigation sensor in a short time to assist in estimating the motion parameters corresponding to the fusion visual sensor, and obtain a dual sensor motion parameter fusion vector; In the embodiment of the present invention, in the dual-sensor data fusion process, since the inertial navigation sensor has a high measurement accuracy in a short time, taking the accelerated linear motion of the target carrier in a short time as an example, the accelerometer of the inertial navigation sensor can quickly and accurately measure the acceleration value, and the gyroscope can also accurately measure the angular velocity. According to the dual-sensor motion parameter characteristic data set, the error of the translation vector and the rotation matrix of the visual sensor in this short-time rapid motion state is understood, and the acceleration and angular velocity information measured by the inertial navigation sensor is converted into displacement and angle information through kinematic formulas. For example, using integral operation, according to The acceleration value measured by the accelerometer is used to calculate the displacement increment in a short period of time, and the current position is obtained by combining the initial position information. Then, this position and angle information is fused with the translation vector and rotation matrix measured by the visual sensor. The weighted average method is used to assign different weights to the data of the inertial navigation sensor and the visual sensor according to the accuracy performance of each sensor in the dual-sensor motion parameter characteristic data set under different motion states. For example, in short-term rapid motion, the weight of the inertial navigation sensor data is set to 0.7, and the weight of the visual sensor data is set to 0.3. After weighted fusion of the two data, the dual-sensor motion parameter fusion vector is finally obtained.

[0075] Step S43: Based on the dual-sensor motion parameter fusion vector and using the extended Kalman filter, the error measurement corresponding to the inertial navigation sensor is calibrated with vision-inertial navigation, so that the error parameters corresponding to the inertial navigation sensor are used as state variables, including gyroscope drift and accelerometer zero bias, and the measurement values ​​of the visual sensor and the measurement values ​​of the inertial navigation sensor in the dual-sensor motion parameter fusion vector are used as observation variables. At the same time, the state variables are continuously updated through the filtering algorithm to realize real-time estimation and calibration compensation of the error measurement corresponding to the inertial navigation sensor, so as to obtain the error calibration compensation result corresponding to the inertial navigation sensor.

[0076] In the embodiment of the present invention, the state vector of the extended Kalman filter is defined, and the gyroscope drift and the accelerometer zero bias are used as state variables. For example, the state vector ,in is the gyroscope drift, For the accelerometer zero bias, the measurement values ​​of the visual sensor (such as translation vector, rotation matrix) and the measurement values ​​of the inertial navigation sensor (such as acceleration, angular velocity) in the dual-sensor motion parameter fusion vector are used as observation variables. In the prediction step of the extended Kalman filter, the state variables at the next moment are predicted according to the dynamic model of the inertial navigation sensor. For example, the gyroscope drift and accelerometer zero bias at the next moment are predicted according to the error model of the gyroscope and accelerometer. In the update step, the error covariance between the observed variable and the predicted state variable is calculated, and the state variable is adjusted by the Kalman gain. For example, the Kalman gain formula is used. ,in is the forecast error covariance, is the observation matrix, is the observation noise covariance, and the Kalman gain is calculated , and then update the state variables by calculating according to the above parameters ,in It is the observed value. By repeating the prediction and update steps, the real-time estimation of the inertial navigation sensor error measurement is realized, and finally the error calibration compensation result corresponding to the inertial navigation sensor is obtained, such as the compensated gyroscope drift value and accelerometer zero bias value.

[0077] Furthermore, the process of dual-sensor motion parameter characteristic analysis in step S41 is specifically to analyze the translation error range between the translation vector corresponding to the visual sensor and the motion trajectory in different directions, and to analyze the rotation angle accuracy corresponding to the rotation matrix in different directions. For the motion measurement parameters corresponding to the inertial navigation sensor, the dynamic model is used to analyze the response characteristics of the angular velocity measured by the gyroscope and the acceleration measured by the accelerometer under different motion states, and to evaluate its zero bias stability and scale factor error.

[0078] Therefore, the embodiments should be regarded as illustrative and non-restrictive from all points, and the scope of the present invention is limited by the appended claims rather than the above description, and it is therefore intended that all changes falling within the meaning and range of equivalent elements of the application documents are included in the present invention.

[0079] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.

Claims

1. A vision-based inertial navigation calibration method, characterized in that: The following steps are involved: Step S1: installing and deploying corresponding visual sensors and inertial navigation sensors on the target carrier, and using the visual sensors and inertial navigation sensors to measure the target visual image data and target inertial navigation motion data corresponding to the target carrier in real time; aligning the target visual image data and the target inertial navigation motion data in time dimension by adding corresponding timestamps, so as to obtain the corresponding target visual image sequence and target inertial navigation motion sequence in the same time dimension; Step S2: Perform static and dynamic motion feature fusion analysis on the target visual image corresponding to the target visual image sequence at each time point to obtain static and dynamic motion visual features of the target carrier; Step S3: searching for the nearest neighbor feature point pairs of the static and dynamic motion visual features of the target carrier to obtain the target carrier motion matching feature point pairs; Based on the target carrier motion matching feature point pairs, the target carrier visual motion is estimated to obtain the target carrier visual motion state variables, including the target carrier visual motion translation vector and the target carrier visual motion rotation matrix; Step S4: Based on the visual motion state variables of the target carrier and the target inertial navigation motion sequence, visual-inertial navigation calibration is performed on the error measurement corresponding to the inertial navigation sensor to obtain the error calibration compensation result corresponding to the inertial navigation sensor.

2. The vision-based inertial navigation calibration method according to claim 1, characterized in that: Step S1 includes the following steps: Step S11: installing and deploying corresponding visual sensors and inertial navigation sensors on the target carrier; Step S12: Using a built-in synchronization signal generator and a hardware synchronization trigger mode to synchronously measure the target carrier using a visual sensor and an inertial navigation sensor to obtain target visual image data and target inertial navigation motion data; Step S13: cleaning and standardizing the target visual image data and the target inertial navigation motion data, so as to enhance contrast, remove noise and standardize pixels for the target visual image data, and remove duplicate data, outliers, compensate for missing values ​​and standardize motion parameters for the target inertial navigation motion data, so as to obtain target visual standard image data and target inertial navigation standard data; Step S14: aligning the target visual standard image data and the target inertial navigation standard data in time dimension by adding corresponding timestamps to obtain the corresponding target visual image sequence and target inertial navigation motion sequence in the same time dimension.

3. The vision-based inertial navigation calibration method according to claim 2, characterized in that: Step S11 includes the following steps: Conduct material and dynamic analysis on the target carrier to obtain the corresponding structural material, motion characteristics and environmental characteristics of the target carrier; Based on the structural material, motion characteristics and environmental characteristics of the target carrier, the sensor adaptation planning analysis of the visual sensor and the inertial navigation sensor is carried out to adapt the type, accuracy requirements and protection level of the visual sensor and the inertial navigation sensor to obtain the target carrier sensor adaptation planning results; Based on the target carrier sensor adaptation planning results, the corresponding visual sensors and inertial navigation sensors to be installed are designed for sensor installation topology, so as to determine the corresponding optimal installation angle and position of the visual sensor according to the change of the motion field of view of the target carrier, and ensure that the target area is covered to the greatest extent under various motion states. For the inertial navigation sensor, it is installed at the position with the least vibration and the easiest to sense motion according to the corresponding dynamic characteristics of the target carrier, so as to generate the target carrier sensor installation topology structure; Based on the target carrier sensor installation topology, the corresponding visual sensors and inertial navigation sensors are installed on the target carrier and sensor calibration is performed. The corresponding internal parameters of the visual sensors, including focal length and distortion coefficient, are calibrated in combination with the optical imaging principle. Gravity and turntable calibration equipment is used to calibrate the corresponding zero bias and scale factor of the inertial navigation sensor according to the gravity field and the law of rotational motion, so as to realize the corresponding calibration work of the visual sensors and inertial navigation sensors.

4. The vision-based inertial navigation calibration method according to claim 3, characterized in that: The visual sensor includes a binocular camera or a monocular camera combined with a laser radar.

5. The vision-based inertial navigation calibration method according to claim 3, characterized in that: The inertial navigation sensor includes a gyroscope and an accelerometer.

6. The vision-based inertial navigation calibration method according to claim 2, characterized in that: Step S12 includes the following steps: The built-in synchronization signal generator of the target carrier is used as the core node to synchronize the trigger topology with the visual sensor and the inertial navigation sensor in a star structure, and the feedback signals corresponding to each sensor are aggregated to the synchronization signal generator using a bus structure to achieve two-way communication, so as to generate the hardware synchronization trigger topology of the target carrier; Based on the hardware synchronous trigger topology of the target carrier, a hardware synchronous trigger method is used to send a synchronous trigger signal to the corresponding visual sensor and inertial navigation sensor to generate a target carrier synchronous trigger signal pulse; The target carrier is synchronized with the trigger signal pulse response to control the corresponding visual sensor and inertial navigation sensor, and the transmission delay of the synchronization signal pulse from the generator to the sensor is measured. At the same time, the corresponding measurement time is fine-tuned based on the transmission delay to perform synchronous real-time measurement of the target carrier to obtain the target visual image data and target inertial navigation motion data.

7. The vision-based inertial navigation calibration method according to claim 1, characterized in that: Step S2 includes the following steps: Step S21: obtaining the target visual images corresponding to the adjacent time frames before and after the time point through the target visual image corresponding to the target visual image sequence at each time point; Step S22: performing target carrier change analysis between target visual images corresponding to the corresponding time points based on the target visual images corresponding to the previous and next adjacent time frames at the time point, so as to compare and analyze the target edge and texture difference changes between the target visual images corresponding to the previous and next adjacent time frames at the time point, so as to obtain the target visual image difference changes between each time point and the previous and next adjacent time frames; Step S23: Based on the difference change of the target visual image between each time point and the adjacent time frames, the target visual image corresponding to the corresponding time point is judged as a static or dynamic image. If there is a difference change between each time point and the adjacent time frames, the target visual image corresponding to the time point is determined as a dynamic visual image; if there is no difference change between each time point and the adjacent time frames, the target visual image corresponding to the time point is determined as a static visual image. Step S24: performing static motion feature analysis on the static visual image to obtain static motion features of the target carrier; performing dynamic motion feature analysis on the dynamic visual image to obtain dynamic motion features of the target carrier; Step S25: performing static and dynamic motion feature fusion analysis on the static motion features of the target carrier and the dynamic motion features of the target carrier to obtain static and dynamic motion visual features of the target carrier.

8. The vision-based inertial navigation calibration method according to claim 7, characterized in that: The static motion feature analysis of the static visual image in step S24 includes the following steps: Perform Gaussian blur and down-sampling operations on the static visual image to obtain the corresponding target static visual sub-image at each scale; The target edge structure feature analysis is performed on the target static visual sub-image corresponding to each scale, so as to extract the target edge contour and regional shape corresponding to the static visual image by edge detection and morphological operation, and obtain the static visual edge structure feature of the target carrier corresponding to each scale; Based on the static visual edge structure features of the target carrier at each scale, the static visual key points of the target static visual sub-image at each scale are extracted to extract the corresponding corner points and edge intersections, and obtain the key point set of the static visual image of the target carrier; Based on the key point set of the static visual image of the target carrier, the static motion feature analysis of the static visual image is performed by using scale-invariant feature transformation and accelerated robust features to obtain the static motion features of the target carrier, including the shape, texture and color features corresponding to the target carrier.

9. The vision-based inertial navigation calibration method according to claim 7, characterized in that: The dynamic motion feature analysis of the dynamic visual image described in step S24 includes the following steps: Treat each dynamic visual image as a space-time cube, and analyze the grayscale changes and gradient changes between adjacent frames of dynamic visual images in the time dimension; Based on the grayscale changes and gradient changes between the dynamic visual images of adjacent frames, the motion clues between the dynamic visual images are statistically analyzed to obtain the dynamic motion direction and the dynamic motion speed of the target carrier; Based on the dynamic motion direction and dynamic motion speed of the target carrier and combined with the sparse optical flow method based on the Lucas-Kanade algorithm, the dynamic motion characteristics of the target carrier corresponding to the dynamic visual image are analyzed to obtain the dynamic motion characteristics of the target carrier, including the corresponding motion trajectory and speed changes of the target carrier during the movement.

10. The vision-based inertial navigation calibration method according to claim 1, characterized in that: Step S3 includes the following steps: Step S31: searching for the nearest neighbor feature point pairs of the static and dynamic motion visual features of the target carrier to obtain the target carrier motion matching feature point pairs; Step S32: performing a temporal and spatial feature point correlation analysis on the target carrier based on the target carrier motion matching feature point pairs and combining triangulation and epipolar geometry principles to analyze the relationship between the angle and side length changes of the triangles formed by the feature points in adjacent frame images of the target carrier, and obtain the temporal and spatial correlation relationship between the feature points of the target carrier; Step S33: performing a statistical analysis of the movement speed and angle of the movement trajectory corresponding to the target carrier based on the spatiotemporal correlation between the characteristic points of the target carrier, so as to obtain the movement speed and movement rotation angle corresponding to the target carrier on the three spatial axes; Step S34: Perform visual motion estimation on the target carrier based on the corresponding motion speed and motion rotation angle of the target carrier on the three axes of space to obtain the visual motion state variable of the target carrier, including the visual motion translation vector of the target carrier and the visual motion rotation matrix of the target carrier.

11. The vision-based inertial navigation calibration method according to claim 10, characterized in that: Step S31 includes the following steps: Performing motion visual deep deconstruction on the static and dynamic visual features of the target carrier, using Fourier transform and wavelet transform to analyze the frequency components and spatial distribution characteristics of static features, and establishing a time series model to analyze the corresponding change trends at different time points for dynamic features, so as to obtain the static and dynamic feature deconstruction data of the target carrier; Based on the static and dynamic feature deconstruction data of the target carrier, a spatiotemporal coding feature point index is constructed for each feature point in the static and dynamic motion visual feature of the target carrier, so that the three-dimensional spatial coordinates of the feature point are mapped into a code using a coding method based on the Hilbert curve in spatial coding, and the corresponding time code is generated according to the time sequence of the feature points in time coding, so as to obtain the static and dynamic visual index feature of the target carrier after spatiotemporal coding; According to the static and dynamic visual index features of the target carrier after spatiotemporal coding, the neighborhood of the feature points is divided, so that the corresponding static and dynamic visual feature points of the target carrier are divided into corresponding triangular neighborhoods based on the Delaunay triangulation part. For the feature points at the edge and corner of the target carrier and with obvious feature changes, a large-scale neighborhood is defined, and for the general feature points, a small-scale neighborhood is defined, so as to obtain the neighborhood division result of the multi-scale feature points of the target carrier; Based on the multi-scale feature point neighborhood division result of the target carrier, the feature points corresponding to the corresponding scale neighborhood within the static and dynamic motion visual features of the target carrier are screened for the nearest neighbor candidate points, so as to obtain the target carrier motion matching feature point pairs by calculating the corresponding energy values ​​and visual similarities between each feature point in the feature point neighborhood.

12. The vision-based inertial navigation calibration method according to claim 10, characterized in that: The target carrier visual motion translation vector described in step S34 is specifically: ； ； ； ； in, represents the target carrier visual motion translation vector, represents the translation component along the X axis, represents the translation component along the Y axis, represents the translation component along the Z axis, represents the motion measurement time, Indicates the movement speed of the target carrier on the X-axis. Indicates the movement speed of the target carrier on the Y axis. Indicates the movement speed of the target carrier on the Z axis.

13. The vision-based inertial navigation calibration method according to claim 10, characterized in that: The target carrier visual motion rotation matrix described in step S34 is specifically: The corresponding rotation matrix around the X axis is: ; The corresponding rotation matrix around the Y axis is: ; The corresponding rotation matrix around the Z axis is: ; in, Indicates the corresponding motion rotation angle of the target carrier on the X-axis. Indicates the corresponding motion rotation angle of the target carrier on the Y axis. Indicates the corresponding movement rotation angle of the target carrier on the Z axis.

14. The vision-based inertial navigation calibration method according to claim 1, characterized in that: Step S4 includes the following steps: Step S41: performing dual-sensor motion parameter characteristic analysis on the target carrier visual motion state variables and the target inertial navigation motion sequence to obtain a dual-sensor motion parameter characteristic data set; Step S42: performing dual sensor data fusion on the visual motion state variables of the target carrier and the target inertial navigation motion sequence based on the dual sensor motion parameter characteristic data set, so as to use the motion measurement of the inertial navigation sensor in a short time to assist in estimating the motion parameters corresponding to the fusion visual sensor, and obtain a dual sensor motion parameter fusion vector; Step S43: Based on the dual-sensor motion parameter fusion vector and using the extended Kalman filter, the error measurement corresponding to the inertial navigation sensor is calibrated with vision-inertial navigation, so that the error parameters corresponding to the inertial navigation sensor are used as state variables, including gyroscope drift and accelerometer zero bias, and the measurement values ​​of the visual sensor and the measurement values ​​of the inertial navigation sensor in the dual-sensor motion parameter fusion vector are used as observation variables. At the same time, the state variables are continuously updated through the filtering algorithm to realize real-time estimation and calibration compensation of the error measurement corresponding to the inertial navigation sensor, so as to obtain the error calibration compensation result corresponding to the inertial navigation sensor.

15. The vision-based inertial navigation calibration method according to claim 14, characterized in that: The process of dual-sensor motion parameter characteristic analysis described in step S41 is specifically to analyze the translation error range between the translation vector corresponding to the visual sensor and the motion trajectory in different directions, analyze the rotation angle accuracy corresponding to the rotation matrix in different directions, and use the dynamic model to analyze the response characteristics of the angular velocity measured by the gyroscope and the acceleration measured by the accelerometer under different motion states for the motion measurement parameters corresponding to the inertial navigation sensor, and evaluate its zero bias stability and scale factor error.

Images