An inertial navigation device and system based on multi-modal data analysis

Abstract

The application discloses an inertial navigation device and system based on multi-modal data analysis, and relates to the field of navigation, comprising an IMU measurement unit, an RTC, a PMIC module, a safety module, a processor module, a storage module, an interface module, an NPU, a visual sensor and a terminal. The application adds a visual sensor and an NPU to a conventional inertial navigation system, cooperates with corresponding algorithms, realizes correction of inertial navigation information by using real-time visual information combined with an AI model before output of the inertial navigation information, and improves the accuracy of output information of the inertial navigation system.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of navigation, in particular to an inertial navigation device and system based on multi-modal data analysis. BACKGROUND

[0002] Inertial navigation, as an important pillar in the field of modern navigation, has a long history and deep technical foundation. Its core advantage lies in high autonomy, which enables it to break away from dependence on external information sources and still operate stably in extreme complex environments such as satellite signal loss or severe electromagnetic interference, providing solid protection for precise navigation of various unmanned devices and transportation vehicles, and relying on good concealment.

[0003] Current inertial navigation systems take the following main measures: one is regular calibration strategy, two is combined navigation scheme, and three is optimization from the system itself.

[0004] Current inertial navigation systems belong to the category of post-error correction or pre-optimization of sensor performance. In the actual operation process of the inertial navigation system, the real-time and dynamic correction means for errors are relatively lacking, and it is difficult to correct the errors in time and accurately in the navigation process. SUMMARY

[0005] Therefore, the purpose of the present application is to provide an inertial navigation device and system based on multi-modal data analysis to solve the technical problems.

[0006] To achieve the above-mentioned purpose, the present application provides the following technical scheme: an inertial navigation device based on multi-modal data analysis, comprising an IMU measurement unit, an RTC, a PMIC module, a safety module, a processor module, a storage module, an interface module, an NPU, a visual sensor, and a terminal. The IMU measurement unit is electrically connected with the PMIC module, and the IMU measurement unit is signal connected with the processor module. The RTC is electrically connected with the PMIC module, and the RTC is signal connected with the processor module. The PMIC module is electrically connected with the safety module, and the PMIC module is electrically connected with the processor module. The PMIC module is electrically connected with the storage module, and the PMIC module is electrically connected with the interface module. The PMIC module is electrically connected with the NPU, and the PMIC module is electrically connected with the visual sensor. The safety module is signal connected with the processor module, and the safety module is signal connected with the interface module. The processor module is signal connected with the storage module, and the processor module is signal connected with the NPU. The processor module is signal connected with the visual sensor, and the storage module is signal connected with the visual sensor. The interface module is signal connected with the terminal, and the NPU is signal connected with the visual sensor.

[0007] The processor module is further configured to undertake data processing of the IMU measurement unit, parameter correction model calculation and data processing in the inertial navigation process, and overall control of the whole system.

[0008] The IMU measurement unit is configured to measure acceleration and angular velocity of the carrier in real time, and calculate information of the carrier pose, speed and position by cooperating with a corresponding algorithm.

[0009] The visual sensor is configured to provide an external scene within a fixed range around the carrier, and provide real-time visual information for the AI inference model.

[0010] The NPU is configured to accelerate a large number of matrix operations in the image AI model, efficiently process image data, and cooperate with the AI model to infer position, attitude and speed information.

[0011] The security module is configured to perform security isolation on software, prevent external intrusion into the inertial navigation system through the interface module software, and protect data information security by using data isolation.

[0012] The RTC is configured to provide a real-time clock for the whole inertial navigation system.

[0013] The storage module is configured to store acquired related data, execution programs and algorithm programs.

[0014] The PMIC module is configured to supply power to different components of the whole inertial navigation system.

[0015] The interface module is configured to provide various common external interfaces, and facilitate access of various mobile terminal devices.

[0016] The system flow is specifically as follows: S1: the inertial navigation system is started, the processor module starts to load a driver program, configure predetermined parameters, configure corresponding navigation modes, i.e., a two-dimensional plane navigation mode and a three-dimensional space navigation mode, then the accelerometer and the gyroscope in the IMU measurement unit and the visual sensor are self-checked, the accelerometer and the gyroscope are initially aligned, and the visual sensor is automatically visually corrected.

[0017] S2: the accelerometer and the gyroscope in the IMU measurement unit and the visual sensor start to work synchronously, and start to measure acceleration and angular velocity of the carrier and images of a preset range environment around the carrier, respectively.

[0018] S3: According to the measured angular velocity, acceleration, and the preset carrier mode, the corresponding algorithm is configured, the corresponding attitude angle is calculated through the gyro measurement data, including the carrier's pitch angle, roll angle and heading angle, often also need to convert the coordinates to get the updated carrier's attitude information in the reference coordinate system, and then the acceleration information measured by the accelerometer and the updated attitude information are used to calculate the carrier's position information and velocity information in the navigation coordinate system, to get the carrier's rough inertial navigation information, the carrier's updated position, attitude and velocity information, after obtaining the carrier's position information, the corresponding algorithm reasoning is carried out through the vision sensor, a parameter correction model is constructed, and the final carrier's inertial navigation information is obtained.

[0019] S4: After a minimum carrier inertial navigation information update, the error compensation model of the IMU measurement unit and the vision-based parameter correction model are multi-modal fusion, the corresponding AI model is used for accelerated reasoning, the accelerometer and the gyroscope in the IMU measurement unit are compensated and error corrected, and the subsequent S, S and S processes are continuously carried out until the terminal closes the inertial navigation system.

[0020] The application further provides that the step S3 adds a vision sensor and an NPU module cooperating with AI model reasoning in the inertial navigation system, and constructs an inertial navigation parameter correction model, which can compensate and correct the carrier's inertial navigation information in real time during the inertial navigation process, thereby improving the accuracy of the inertial navigation information.

[0021] The application further provides that the step S4 is multi-modal fusion through the error compensation model of the traditional IMU measurement unit and the vision-based parameter correction model, and the corresponding AI model is used for accelerated reasoning to realize rapid sensor parameter correction, thereby reducing the cumulative error caused by the inertial navigation time, so as to further improve the accuracy and effective time length of the inertial navigation.

[0022] In summary, the application mainly has the following beneficial effects:

[0023] The application adds a visual sensor and an NPU in a traditional inertial navigation system, cooperates with a corresponding algorithm, realizes that inertial navigation error will accumulate with time in the process of inertial navigation, uses real-time visual information to correct the inertial navigation information before the inertial navigation information is output, improves the accuracy of the output information of the inertial navigation system, and calibrates the related parameters of the IMU measurement unit in time after the output of the minimum inertial navigation data, thereby reducing the error of the IMU measurement unit accumulated with the inertial navigation, further improving the accuracy of the inertial navigation information and prolonging the effective time of the inertial navigation, being more accurate, intelligent and effective than the traditional scheme, and making the application precision of the inertial navigation higher, the application time longer, and the application scene more. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 It is a system framework diagram of the application.

[0025] Figure 2 It is a running flowchart of the application.

[0026] In the figure: 1, IMU measurement unit; 2, RTC; 3, PMIC module; 4, safety module; 5, processor module; 6, storage module; 7, interface module; 8, NPU; 9, visual sensor; 10, terminal. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. The embodiments described below with reference to the drawings are exemplary and are used to explain the application, and cannot be understood as a limitation of the application.

[0028] The embodiments of the application will be described below according to the overall structure of the application.

[0029] A kind of inertial navigation device and system based on multi-modal data analysis, such as Figure 1As shown, it comprises an IMU measurement unit 1, an RTC 2, a PMIC module 3, a security module 4, a processor module 5, a storage module 6, an interface module 7, an NPU 8, a visual sensor 9, and a terminal 10. The IMU measurement unit 1 is electrically connected to the PMIC module 3, and is signal connected to the processor module 5. The RTC 2 is electrically connected to the PMIC module 3, and is signal connected to the processor module 5. The PMIC module 3 is electrically connected to the security module 4, and is electrically connected to the processor module 5, and is electrically connected to the storage module 6, and is electrically connected to the interface module 7, and is electrically connected to the NPU 8, and is electrically connected to the visual sensor 9. The security module 4 is signal connected to the processor module 5, and is signal connected to the interface module 7. The processor module 5 is signal connected to the storage module 6, and is signal connected to the NPU 8, and is signal connected to the visual sensor 9. The storage module 6 is signal connected to the visual sensor 9. The interface module 7 is signal connected to the terminal 10. The NPU 8 is signal connected to the visual sensor 9.

[0030] The IMU measurement unit 1, the RTC 2, the PMIC module 3, the processor module 5, the storage module 6, the NPU 8, and the visual sensor 9 can realize inertial navigation based on multi-modal data analysis.

[0031] It should be pointed out that the operator of the terminal opens the inertial navigation function through the terminal 10. This instruction will be transmitted to the security module 4 through the interface module 9 after certain security processing, and corresponding identification and security verification will be performed. After identification and verification, the processor module 5 will issue an instruction, the whole system will enter pre-start, the IMU measurement unit 1 and the visual sensor 9 will start preloading and perform corresponding correction, the driver in the storage module 6 will be preloaded, the corresponding model will be loaded and started, and the processor module 5 will start deploying AI inference algorithm to the NPU 8. When the pre-start work is completed, the inertial navigation system will perform self-checking, and after the self-checking is completed, the IMU measurement unit will collect data and enter the inertial navigation process.

[0032] When the inertial navigation system enters the inertial navigation process, the gyroscope and accelerometer in IMU measurement unit 1 collect angular velocity and acceleration, respectively. The collected data is stored in memory module 6. Processor module 5 uses the angular velocity measured by the gyroscope and corresponding algorithms to calculate the current pitch, roll, and yaw angle parameters of the vehicle. Then, through coordinate system transformation matrix and related calculations, the updated attitude information of the vehicle can be obtained. Subsequently, the acceleration measured by the accelerometer in IMU measurement unit 1 needs to be transformed from the vehicle coordinate system to the navigation coordinate system. Then, using the acceleration in the navigation coordinate system and the updated vehicle attitude information, vector calculation is performed to obtain the position and velocity information of the vehicle in the navigation coordinate system. At this point, a rough inertial navigation information of the vehicle is obtained. At the same time, the real-time image information collected by the visual sensor is combined with the corresponding AI model. The NPU8 is then used to accelerate the inference of the AI ​​model. The AI ​​model outputs correction data, which is used to correct the previous rough inertial navigation information of the vehicle, ultimately resulting in more accurate inertial navigation information.

[0033] Example 2

[0034] Processor module: It is responsible for data processing of the IMU measurement unit, parameter correction model calculation, data processing, and overall control of the entire system during inertial navigation.

[0035] IMU Measurement Unit 1: Real-time measurement of the carrier's acceleration and angular velocity, combined with corresponding algorithms to calculate the carrier's pose, velocity, and position information.

[0036] Visual sensor 9: Provides external scene information within a fixed range around the carrier, providing real-time visual information for the AI ​​inference model. Visual sensor 9 is usually fixed on the carrier or has a fixed coordinate system correspondence. First, the image coordinate system and the carrier coordinate system are bound together to obtain the transformation matrix between the two. Then, the real-time image is combined with the corresponding dynamic pose estimation algorithm, and then the NPU8 is used for accelerated inference to update the carrier's pose information in real time. Then, this pose information is used to adjust and correct the previously obtained coarse inertial navigation information of the carrier, and finally output a more accurate inertial navigation information.

[0037] NPU8: Accelerates a large number of matrix operations in image AI models, efficiently processes image data, and works with AI models to infer position, pose and velocity information.

[0038] Security Module 4: Provides security isolation in the execution software to prevent external intrusion into the inertial navigation system through the interface module software, and protects data information security by means of data isolation.

[0039] RTC2: Provides a real-time clock for the entire inertial navigation system.

[0040] Storage module 6: store the relevant data obtained, execution programs, algorithm programs, etc.

[0041] PMIC module 3: power supply for different components of the entire inertial navigation system.

[0042] Interface module 7: provides various common external interfaces, facilitating the access of various mobile terminal devices. It can be understood that the IMU measurement unit 1, RTC 2, PMIC module 3, processor module 5, storage module 6, NPU 8, and visual sensor 9 can realize inertial navigation based on multi-modal data analysis.

[0043] Example three

[0044] The system flow is as follows: S1: the inertial navigation system is started, the processor module starts to load the driver program, configures the predetermined parameters, configures the corresponding navigation mode two-dimensional plane navigation mode and three-dimensional space navigation mode, and then the accelerometer and gyroscope in the IMU measurement unit and the visual sensor perform self-checking, while the accelerometer and gyroscope perform initial alignment and the visual sensor performs automatic visual correction.

[0045] S2: the accelerometer and gyroscope in the IMU measurement unit and the visual sensor start working synchronously, respectively starting to measure the acceleration and angular velocity of the carrier and the image of the environment within the preset range around the carrier.

[0046] S3: according to the measured angular velocity, acceleration, and preset carrier mode, the corresponding algorithm is configured, the attitude angle is calculated through the gyroscope measurement data, including the carrier's pitch angle, roll angle, and heading angle, often also needing to convert the coordinates to obtain the updated carrier attitude information in the reference coordinate system, and then the acceleration information measured by the accelerometer and the updated attitude information are used to calculate the position information and velocity information of the carrier in the navigation coordinate system, to obtain the carrier's rough inertial navigation information, the carrier's updated position, attitude, and velocity information, after obtaining the carrier's position and attitude information, the corresponding algorithm reasoning is performed through the visual sensor to construct a parameter correction model, and the final carrier's inertial navigation information is obtained.

[0047] S4: after a minimum carrier inertial navigation information update, the error compensation model of the IMU measurement unit and the visual-based parameter correction model are fused, the corresponding AI model is used for accelerated reasoning, the accelerometer and gyroscope in the IMU measurement unit are compensated and error corrected, and the subsequent S2, S3, and S4 processes are continuously performed until the terminal closes the inertial navigation system.

[0048] It should be noted that the visual sensor and the NPU module cooperating with the AI model reasoning are added in the inertial navigation system in step S3, an inertial navigation parameter correction model is constructed, the model can compensate and correct the carrier inertial navigation information in real time during the inertial navigation process, so as to improve the accuracy of the inertial navigation information; in step S4, the multi-modal fusion is carried out through the error compensation model of the traditional IMU measurement unit and the parameter correction model based on vision, the corresponding AI model is used for accelerated reasoning, the rapid sensor parameter correction is realized, so as to reduce the cumulative error caused by the inertial navigation time, so as to further improve the accuracy of the inertial navigation and the effective time of the inertial navigation. It can be understood that: when the inertial navigation system measures the angular velocity and acceleration through the gyroscope and accelerometer in the IMU measurement unit 1, the processor module 5 calculates the preliminary carrier update information according to the corresponding algorithm, then the visual sensor 9 is used to collect the carrier environment data in real time, then the corresponding reasoning algorithm is matched, the NPU 8 is matched to accelerate the deduction, and then the correction data is deduced, the preliminary carrier inertial navigation information is corrected by using the correction data, and finally more accurate carrier inertial navigation information is obtained. At the same time, after the minimum carrier update information is output once, the errors of the gyroscope and the accelerometer in the IMU measurement unit are compensated according to the deduced correction data. Compared with the previous inertial navigation system, the present application is more accurate, intelligent and has a longer effective time, so that the application precision of the inertial navigation is higher, the application time is longer, and the application scene is more.

[0049] Although the embodiments of the present application have been shown and described, the specific embodiments are only an explanation of the present application, and are not a limitation of the application. The specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner, and those skilled in the art can make modifications, replacements and variations of the embodiments without creative contribution after reading the specification, as long as they are within the scope of the claims of the present application.

Claims

1. An inertial navigation system based on multimodal data analysis, characterized in that: The specific system flow is as follows: S1: The inertial navigation system starts up, the processor module starts loading the driver, configures the predetermined parameters, configures the corresponding navigation mode, two-dimensional planar navigation mode, three-dimensional spatial navigation mode, then the accelerometer and gyroscope in the IMU measurement unit and the vision sensor perform self-test, at the same time the accelerometer and gyroscope perform initial alignment, and the vision sensor performs automatic visual correction. S2: The accelerometer, gyroscope and vision sensor in the IMU measurement unit start working synchronously, respectively measuring the acceleration and angular velocity of the carrier and the environmental image of the preset range around the carrier; S3: Based on the measured angular velocity and acceleration, as well as the preset carrier mode and the configured corresponding algorithm, the corresponding attitude angles are calculated using gyroscope measurement data, including the carrier's pitch angle, roll angle, and yaw angle. The coordinates need to be transformed to obtain updated carrier attitude information in the reference coordinate system. Then, the position and velocity information of the carrier in the navigation coordinate system are calculated using the acceleration information measured by the accelerometer and the updated attitude information, to obtain the carrier's rough inertial navigation information, updated position, attitude, and velocity information. After obtaining the carrier's pose information, the corresponding algorithm reasoning is performed through the visual sensor to construct a parameter correction model and obtain the final carrier's inertial navigation information. S4: After a minimum carrier inertial navigation information update, the error compensation model and vision-based parameter correction model of the IMU measurement unit are fused in a multimodal manner. The corresponding AI model is used to accelerate inference and compensate and correct the error of the accelerometer and gyroscope in the IMU measurement unit. The S2, S3 and S4 processes are then continued until the terminal shuts down the inertial navigation system. In step S3, a visual sensor and an NPU module that works with the AI ​​model for inference are added to the inertial navigation system to build an inertial navigation parameter correction model. This model can compensate and correct the carrier's inertial navigation information in real time during the inertial navigation process, thereby improving the accuracy of the inertial navigation information. In step S4, multimodal fusion is performed using the error compensation model of the traditional IMU measurement unit and the vision-based parameter correction model. The corresponding AI model is used to accelerate inference and achieve rapid sensor parameter correction, thereby reducing the cumulative error caused by inertial navigation time and further improving the accuracy and effective duration of inertial navigation.

2. An inertial navigation device based on multimodal data analysis, applied to an inertial navigation system based on multimodal data analysis as described in claim 1, comprising an IMU measurement unit (1), an RTC (2), a PMIC module (3), a security module (4), a processor module (5), a storage module (6), an interface module (7), an NPU (8), a vision sensor (9), and a terminal (10), characterized in that: The IMU measurement unit (1) is electrically connected to the PMIC module (3), the IMU measurement unit (1) is signal-connected to the processor module (5), the RTC (2) is electrically connected to the PMIC module (3), the RTC (2) is signal-connected to the processor module (5), the PMIC module (3) is electrically connected to the security module (4), the PMIC module (3) is electrically connected to the processor module (5), the PMIC module (3) is electrically connected to the storage module (6), the PMIC module (3) is electrically connected to the interface module (7), and the PMIC module (3) is electrically connected to the N The PMIC module (3) is electrically connected to the vision sensor (9), the security module 4 is signal-connected to the processor module 5, the security module 4 is signal-connected to the interface module (7), the processor module (5) is signal-connected to the storage module (6), the processor module (5) is signal-connected to the NPU (8), the processor module (5) is signal-connected to the vision sensor (9), the storage module (6) is signal-connected to the vision sensor (9), the interface module (7) is signal-connected to the terminal (10), and the NPU (8) is signal-connected to the vision sensor (9).

3. An inertial navigation device based on multimodal data analysis according to claim 2, characterized in that: The processor module is responsible for data processing of the IMU measurement unit, parameter correction model calculation, data processing, and overall control of the entire system during inertial navigation. IMU measurement unit (1): measures the acceleration and angular velocity of the carrier in real time, and calculates the carrier's pose, velocity and position information in conjunction with the corresponding algorithm; Visual sensor (9): Provides external scene information within a fixed range around the carrier, providing real-time visual information for AI inference models; NPU(8): Accelerates a large number of matrix operations in image AI models, efficiently processes image data, and works with AI models to infer position, pose and velocity information; Security module (4): Security isolation in the execution software to prevent external intrusion into the inertial navigation system through the interface module software, and data isolation to protect data information security; RTC(2): Provides a real-time clock for the entire inertial navigation system; Storage module (6): Stores the acquired relevant data, execution program, and algorithm program; PMIC module (3): provides power to the different components of the entire inertial navigation system; Interface module (7): Provides various common external interfaces to facilitate access for various mobile terminal devices.

Images