Fusion positioning methods, devices, systems and flying cars

Abstract

This application relates to a fusion positioning method, apparatus, system, and flying car. The method includes: acquiring local pose data output by a visual inertial navigation odometry system (VIS), wherein the local pose data is obtained by tightly coupling fusion of visual data and IMU data, wherein the IMU data is obtained by processing first IMU data from the VIS and second IMU data from a navigation device; acquiring global pose data output by the navigation device; loosely coupling fusion of the local pose data and the global pose data; and outputting the resulting positioning data as the positioning result. The solution provided by this application can provide relatively accurate and stable positioning in outdoor flight scenarios.

Description

Technical Field

[0001] This application relates to the field of flying car technology, and in particular to a fusion positioning method, device, system and flying car. Background Technology

[0002] Currently, SLAM (Simultaneous Localization and Mapping) is considered a core technology for achieving autonomous operation of mobile robots and has been widely applied in fields such as drones, unmanned vehicles, and virtual reality. Visual sensors are prone to blurring during high-speed motion, and their estimation of high-speed and rotational motion is inherently insufficient. IMU (Inertial Measurement Unit) incorporates accelerometers and gyroscopes, making it highly accurate for short-term high-speed motion detection, but it suffers from cumulative errors over long periods. VIO (Visual-Inertial Odometry) combines the characteristics of both IMUs and visual sensors, primarily using input data from IMUs and visual sensors such as cameras to estimate the current motion state information of the vehicle, including velocity, attitude, and position.

[0003] Currently, most visual odometry applications are concentrated in indoor scenes because indoor scenes are relatively stable, with relatively stable lighting, aircraft motion, and feature textures (point features and line features). Among related technologies, visual odometry applications for aerial scenes such as outdoor flight scenarios are relatively few. Aerial scenes are more sensitive to all six degrees of freedom, forming a true 3D scene, and the motion at high altitudes is more complex, with more dramatic changes in perspective, making accurate and stable positioning more difficult.

[0004] Therefore, for outdoor flight scenarios, there is a need to provide more suitable and accurate positioning methods for aircraft such as flying cars. Summary of the Invention

[0005] To address or partially address the problems existing in related technologies, this application provides a fusion positioning method, device, system, and flying car that can provide relatively accurate and stable positioning in outdoor flight scenarios.

[0006] The first aspect of this application provides a fusion positioning method, including:

[0007] The local pose data output by the visual inertial navigation odometry is acquired, wherein the local pose data is obtained by tightly coupling and fusing visual data and IMU data, wherein the IMU data is obtained by processing the first IMU data of the visual inertial navigation odometry and the second IMU data of the navigation device;

[0008] Obtain the global pose data output by the navigation device;

[0009] The local pose data and the global pose data are loosely coupled and fused, and the resulting positioning data is output as the positioning result.

[0010] In one embodiment, the visual inertial odometry includes a camera and a first IMU device, wherein the visual data comes from the camera and the first IMU data comes from the first IMU device;

[0011] The navigation device includes a second IMU device, and the data from the second IMU device comes from the second IMU device.

[0012] In one embodiment, the IMU data is obtained by processing first IMU data from a visual inertial odometry system and second IMU data from a navigation device, including:

[0013] If the reception of the second IMU data by the second IMU device of the navigation device is normal, the second IMU data is selected as the IMU data;

[0014] If the reception of the second IMU data of the second IMU device of the navigation device is abnormal, the first IMU data is selected as the IMU data.

[0015] In one embodiment, the step of loosely coupling and fusing the local pose data with the global pose data, and outputting the resulting localization data as the localization result, includes:

[0016] When the navigation device experiences a signal abnormality, the positioning data is determined based on the local pose data output by the visual inertial navigation odometry, and the positioning data is output as the positioning result.

[0017] When the visual inertial navigation odometry exhibits visual underconstraint, positioning data is determined based on the global pose data output by the navigation device, and the positioning data is output as the positioning result.

[0018] In one embodiment, when the navigation device experiences a signal abnormality, determining positioning data based on the local pose data output by the visual-inertial odometry system, and outputting the positioning data as the positioning result, includes:

[0019] When the covariance matrix of the navigation device is abnormal, obtain the transformation relationship value between the local pose data of the visual inertial odometry and the global pose data after the local pose data of the visual inertial odometry is converted to the global coordinate system;

[0020] The local pose data output by the visual inertial odometry is multiplied by the transformation relationship value to obtain the positioning data, and the positioning data is output as the positioning result.

[0021] In one embodiment, when the visual-inertial odometry exhibits visual underconstraint, determining positioning data based on the global pose data output by the navigation device and outputting the positioning data as the positioning result includes:

[0022] When the covariance matrix of the visual inertial navigation odometry is abnormal, the transformation relationship value between the local pose data of the visual inertial navigation odometry after converting it to the global coordinate system and the global pose data is obtained, and the difference between the transformation relationship value and the true position transformation relationship value is determined based on the transformation relationship value.

[0023] The positioning data is obtained by multiplying the global pose data output by the navigation device with the inverse of the difference, and the positioning data is output as the positioning result.

[0024] In one embodiment, acquiring the global pose data output by the navigation device includes:

[0025] The global pose data output by the navigation device using carrier phase differential positioning (RTK) is obtained.

[0026] In one embodiment, the visual-inertial odometry is initialized based on visual data, IMU data, and global pose data output by the navigation device during initialization; wherein,

[0027] PnP pose calculation is performed on the visual data, and pose calculation is performed after integral processing of the IMU data;

[0028] The pose calculation results of the visual data and the pose calculation results of the IMU data are filtered together to obtain updated pose transformation data;

[0029] The filtered pose transformation data is combined with the global pose data output by the navigation device as feedback to correct the visual data of the previous frame of the visual inertial navigation odometry.

[0030] A second aspect of this application provides a fusion positioning device, comprising:

[0031] The first input module is used to acquire local pose data output by the visual inertial navigation odometry, wherein the local pose data is obtained by tightly coupling and fusing visual data and IMU data, and wherein the IMU data is obtained by processing the first IMU data of the visual inertial navigation odometry and the second IMU data of the navigation device.

[0032] The second input module is used to acquire the global pose data output by the navigation device;

[0033] The fusion positioning module is used to loosely fuse the local pose data and the global pose data, and output the positioning data obtained after fusion processing as the positioning result.

[0034] In one embodiment, the fusion positioning module includes:

[0035] The first processing submodule is used to determine positioning data based on the local pose data output by the visual inertial navigation odometry when the navigation device experiences signal abnormality, and output the positioning data as the positioning result.

[0036] The second processing submodule is used to determine positioning data based on the global pose data output by the navigation device when the visual inertial navigation odometry shows visual underconstraint, and output the positioning data as the positioning result.

[0037] A third aspect of this application provides a fusion positioning system, comprising:

[0038] Visual-inertial odometry is used to output local pose data;

[0039] Navigation equipment used to output global pose data and second IMU data;

[0040] A computing platform is used to acquire local pose data output by a visual inertial navigation odometry system, wherein the local pose data is obtained by tightly coupling and fusing visual data and IMU data, wherein the IMU data is obtained by processing the first IMU data of the visual inertial navigation odometry system and the second IMU data of the navigation device; acquire global pose data output by the navigation device; perform loose coupling and fusing processing on the local pose data and the global pose data, and output the resulting positioning data as the positioning result.

[0041] The fourth aspect of this application provides a flying car, including the fusion positioning device described above.

[0042] The fifth aspect of this application provides a flying car, comprising:

[0043] Processor; and

[0044] A memory that stores executable code, which, when executed by the processor, causes the processor to perform the method described above.

[0045] A sixth aspect of this application provides a computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method described above.

[0046] The technical solution provided in this application may include the following beneficial effects:

[0047] The technical solution of this application adopts a fusion positioning method combining visual inertial odometry (VIO) with navigation equipment. In the process of tightly coupling and fusing visual data and IMU data to obtain local pose data, the second IMU data of the navigation equipment is also considered to improve the positioning accuracy of the VIOMA. Then, the VIOMA and navigation equipment are loosely coupled, that is, the local pose data and the global pose data are loosely coupled and fused. Finally, the positioning data obtained after fusion is output as the positioning result. Through the above processing, the navigation equipment participates in the entire positioning process, including participating in the tightly coupled fusion of front-end visual data and IMU data, and participating in the loosely coupled process of the back-end VIOMA and navigation equipment. This solves the problems of unstable VIOMA output and inaccurate positioning in outdoor flight scenarios, providing more accurate and stable positioning in outdoor flight scenarios, and providing more accurate and stable visual positioning and navigation support for the stable flight of aircraft such as flying cars.

[0048] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0049] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments thereof.

[0050] Figure 1 This is a schematic flowchart illustrating the fusion positioning method in an embodiment of this application;

[0051] Figure 2 This is a flowchart illustrating a fusion positioning method according to another embodiment of this application;

[0052] Figure 3 This is a schematic diagram of the hardware composition of the fusion positioning system shown in the embodiments of this application;

[0053] Figure 4 This is a schematic diagram illustrating the application of the fusion positioning method shown in the embodiments of this application;

[0054] Figure 5 This is a schematic diagram of the initialization process in the fusion positioning method shown in the embodiments of this application;

[0055] Figure 6 This is a schematic diagram comparing forced switching and fusion in the fusion positioning method shown in the embodiments of this application;

[0056] Figure 7 This is a schematic diagram of the device state in the fusion positioning method shown in the embodiments of this application;

[0057] Figure 8 This is a schematic diagram of the fusion positioning device shown in the embodiments of this application;

[0058] Figure 9 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application. Detailed Implementation

[0059] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0060] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0061] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0062] In related technologies, visual odometry applications for aerial scenarios such as outdoor flight are relatively limited. For outdoor flight scenarios, there is a need to provide more suitable and accurate positioning methods for aircraft such as flying cars. This application provides a fusion positioning method that can provide more accurate and stable positioning for aircraft in outdoor flight scenarios.

[0063] The technical solutions of the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0064] Figure 1This is a flowchart illustrating the fusion positioning method in an embodiment of this application.

[0065] See Figure 1 The method includes:

[0066] S101. Acquire local pose data output by the visual inertial navigation odometry, wherein the local pose data is obtained by tightly coupling and fusing visual data and IMU data, wherein the IMU data is obtained by processing the first IMU data of the visual inertial navigation odometry and the second IMU data of the navigation device.

[0067] The visual inertial odometry system includes a camera and a first IMU device, with visual data coming from the camera and first IMU data coming from the first IMU device; the navigation device includes a second IMU device, with second IMU data coming from the second IMU device. The camera can be a binocular camera.

[0068] The IMU data is obtained by processing the first IMU data from the visual inertial navigation odometry and the second IMU data from the navigation device, including: if the reception of the second IMU data of the navigation device is normal, the second IMU data is selected as the IMU data; if the reception of the second IMU data of the navigation device is abnormal, the first IMU data is selected as the IMU data.

[0069] During initialization, the visual inertial navigation odometry (VIO) process initializes based on visual data, IMU data, and global pose data output by the navigation device. This includes PnP pose calculation on the visual data and integration processing of the IMU data followed by pose calculation. The pose calculation results from the visual data and the IMU data are then filtered together to obtain updated pose transformation data. This filtered pose transformation data is then combined with the global pose data output by the navigation device as feedback to correct the visual data from the previous frame of the VIO.

[0070] S102. Obtain the global pose data output by the navigation device.

[0071] Among them, the global pose data output by the navigation device is obtained using the RTK (Real-time kinematic) method.

[0072] S103. Perform loosely coupled fusion processing on the local pose data and the global pose data, and output the resulting positioning data as the positioning result.

[0073] Specifically, when the navigation device experiences signal anomalies, positioning data is determined based on the local pose data output by the visual inertial navigation odometry (VIO) system, and this positioning data is output as the positioning result. For example, when the covariance matrix of the navigation device is abnormal, the transformation relationship value between the local pose data of the VIOMS system (after converting it to the global coordinate system) and the global pose data is obtained; the positioning data is obtained by multiplying the local pose data output by the VIOMS system with the transformation relationship value, and this positioning data is output as the positioning result.

[0074] When visual underconstraints occur in the visual-inertial odometry (VIO), positioning data is determined based on the global pose data output by the navigation device, and this positioning data is output as the positioning result. For example, when the covariance matrix of the VIOMA is abnormal, the transformation relationship value between the local pose data of the VIOMA after transformation to the global coordinate system and the global pose data is obtained. The difference between the transformation relationship value and the actual position transformation relationship value is determined based on the transformation relationship value. The positioning data is obtained by multiplying the global pose data output by the navigation device and the inverse of the difference, and this positioning data is output as the positioning result.

[0075] As can be seen from this embodiment, the technical solution of this application adopts a fusion positioning method combining visual inertial odometry (VIO) with navigation equipment. In the process of tightly coupling and fusing visual data and IMU data to obtain local pose data, the second IMU data of the navigation equipment is also considered to improve the positioning accuracy of the VIOMA. Then, the VIOMA and navigation equipment are loosely coupled, that is, the local pose data and global pose data are loosely coupled and fused. Finally, the positioning data obtained after fusion is output as the positioning result. Through the above processing, the navigation equipment participates in the entire positioning process, including participating in the tightly coupled fusion process of front-end visual data and IMU data, and participating in the loosely coupled process of back-end VIOMA and navigation equipment. This solves the problems of unstable VIOMA output and inaccurate positioning in outdoor flight scenarios, providing more accurate and stable positioning in outdoor flight scenarios, and providing more accurate and stable visual positioning and navigation support for the stable flight of aircraft such as flying cars.

[0076] Figure 2 This is a flowchart illustrating a fusion positioning method according to another embodiment of this application. This embodiment describes the fusion positioning method of this application through the interaction between a visual inertial odometry system, a navigation device, and a computing platform for fusion processing.

[0077] This application employs a two-level fusion approach using three sensors—camera, IMU, and satellite navigation equipment—to address the issue of unstable visual odometry output in large outdoor scenarios such as high-altitude ground-to-air interactions, providing visual positioning and navigation support for stable flight of high-altitude aircraft. In this application embodiment, the satellite navigation equipment participates throughout the positioning process, not only in the loose combination at the backend but also in the visual inertial navigation odometry initialization process, to enhance the stability of visual feature constraints. Furthermore, the output information of the satellite navigation equipment in this application embodiment includes attitude information in addition to latitude, longitude, and altitude information.

[0078] Please also see Figure 3 and Figure 4 The fusion positioning system illustrated in this application includes a visual-inertial odometry (VIO), a navigation device, and a computing platform. The VIOMA is used to output local pose data; the navigation device is used to output global pose data and second IMU data; the computing platform is used to acquire the local pose data output by the VIOMA, wherein the local pose data is obtained by tightly coupling fusion of visual data and IMU data, and the IMU data is obtained by processing the first IMU data from the VIOMA and the second IMU data from the navigation device; acquire the global pose data output by the navigation device; loosely couple the local pose data and the global pose data, and output the resulting positioning data as the positioning result.

[0079] The visual inertial odometry system includes a camera and an IMU device (which can be referred to as the first IMU device), while the navigation equipment includes a GNSS (Global Navigation Satellite System) board and an IMU device (which can be referred to as the second IMU device).

[0080] The camera can be a stereo camera or a monocular camera. Taking a stereo camera as an example, it can include two camera sensors and one built-in IMU. The NAV (Navigation) device can have a built-in high-precision IMU and a satellite GNSS board. The navigation device can be a combined navigation device or a device with an independent external IMU and RTK. By adopting a multi-IMU redundancy design, it can adapt to dynamic aerial scenarios and improve the level of safety.

[0081] The left and right images from the binocular cameras, along with IMU data, can form a tightly coupled VINS (visual-inertial system). Tight coupling typically uses raw data from both sensors to jointly estimate a set of variables, and sensor noise also influences each other. This VINS system can use nonlinear optimization methods to iteratively optimize the output pose, but is not limited to this. This process is a local odometry; by aligning it with the global pose of the machine at startup, the transformation relationship between the local and global coordinate systems can be obtained, i.e., the transformation relationship between the local pose and the global pose.

[0082] The navigation device triggers the stereo camera via a PPS (Pulse per second) synchronization trigger line, and simultaneously sends the navigation device's GPRMC (Recommended Positioning Information) time to the camera port via a serial port, thus completing time synchronization between the camera's image data, IMU data, and the navigation device internally. Besides the PPS pulse method, time synchronization between the camera and the navigation device can also be achieved through PTP (Precision Time Protocol) network port synchronization, among other methods.

[0083] The computing platform receives image data and IMU data from the visual inertial odometry system, and global pose data and high-precision IMU data from the navigation device. This embodiment uses visual data as image data, but is not limited to this. The global pose estimation result or local pose estimation result output by the computing platform can be sent to the receiving device. The entire fused visual odometry calculation method runs on an embedded platform such as an ARM (Advanced RISC Machine). The binocular camera transmits image data and IMU data to the computing platform (ARM embedded platform) via a data port (USB (Universal Serial Bus) or Ethernet port, depending on the camera). The global pose data and high-precision IMU data (e.g., measured acceleration and angular velocity) from the navigation device are sent to the computing platform via a CAN (Controller Area Network) port. After fusion is completed on the computing platform, the fused positioning result is finally sent via the CAN port to any device terminal that wants to use the result.

[0084] To ensure normal operation in complex scenarios, this embodiment of the application sets a loose combination relationship between the output of the local visual inertial odometry (camera-IMU visual odometry) and the global pose of the navigation device (position converted to NED NE-G coordinate system due to latitude, longitude, and altitude). The loose combination can be based on nonlinear optimization or filtering. The purposes of setting the loose combination relationship are: 1) to maximize the positioning accuracy of the entire visual inertial odometry when the satellite signal of the navigation device is good; 2) to achieve trajectory convergence by relying on the global position constraints of the navigation device's satellites even when the visual odometry is under-constrained (failure of high-speed motion visual detection and tracking or in scenarios with clear texture); 3) to achieve trajectory convergence and maintain stable positioning output by using the visual and IMU constraints of the visual inertial odometry even when the satellite signal of the navigation device is poor.

[0085] See Figure 2 The method includes:

[0086] S201, the visual inertial odometry acquires local image data and local first IMU data, and receives second IMU data and global pose data from the navigation device.

[0087] Visual odometry can acquire left and right images captured by a binocular camera, as well as first IMU data measured by the built-in first IMU. The left and right images from the binocular camera, combined with the IMU data, can form a tightly coupled visual odometry system.

[0088] The visual inertial odometry receives second IMU data measured and collected by the second IMU built into the navigation device, as well as global pose data output by the navigation device.

[0089] S202, the visual inertial navigation odometry is initialized based on image data, IMU data and global pose data output by the navigation device.

[0090] The initialization process can be found in [reference]. Figure 5 The initialization process mainly involves setting a relatively stable initial value for the entire camera-IMU system, i.e., the visual-inertial odometry system. This initial value may include, for example, the coordinate accuracy of 3D points used in PnP (Perspective-n-Point) calculations, and the accuracy of the IMU's zero-bias. PnP is a method for solving the motion of 3D to 2D point pairs.

[0091] To enhance system security, this application example incorporates at least two IMU sensors: one located in the binocular camera and the other in the navigation device. Therefore, during operation, the two IMU data streams are selectively used. Considering the high accuracy of the navigation device's second IMU, if the second IMU data reception is normal, it is used as the input; otherwise, the system switches to using the camera's built-in first IMU data.

[0092] During initialization, the visual inertial odometry (VIO) system initializes based on visual data, IMU data, and global pose data output from the navigation device. This initialization process includes PnP pose calculation on the visual data and integration processing of the IMU data (including acceleration, angular velocity, etc.) followed by pose calculation. The pose calculation results from both the visual and IMU data are then filtered to obtain updated pose transformation data. This filtered pose transformation data is then combined with the global pose data output from the navigation device as feedback to correct the visual data from the previous frame of the VIOS. Alternatively, the filtered pose transformation data can be used as feedback to correct the visual data from the previous frame of the VIOS. The previous frame refers to the image frame previously acquired by the VIOS.

[0093] During the selection process of two IMU data sets, the embodiments of this application can align and synchronize the two IMU data sets in real time. For example, the pose calculated by the IMU and the pose calculated by the camera PnP are filtered together to update a pose transformation. This updated pose transformation, combined with the pose provided by the satellite, is then used as feedback to correct the 3D points (3D landmarks) of the previous frame. This feedback adjustment can improve the accuracy of the visual PnP at the next moment. The filtering can be, for example, Kalman filtering, but is not limited to it. Kalman filtering is an algorithm that uses the state equation of a linear system to make an optimal estimate of the system state through the system input and output observation data. Since the observation data includes the influence of noise and interference in the system, the optimal estimation can also be regarded as a filtering process. Kalman filtering can use the dynamic information of the target to try to remove the influence of noise and obtain a good estimate of the target position. This estimate can be an estimate of the current target position (filtering), an estimate of the future position (prediction), or an estimate of the past position (interpolation or smoothing).

[0094] In the initialization phase, this embodiment of the application involves the integration of IMU, vision, and satellite positioning. RTK positioning can be used to calculate the relative transformation between two frames to assist in constructing and improving the accuracy of 3D landmark points.

[0095] Because the initialization process involves continuous iterative convergence, the output is highly unstable. Therefore, during the initialization phase, the global positioning of the satellite navigation device can be directly and simply used as the output of the entire visual-inertial odometry system until initialization is complete. Since the initialization time is usually short, the deviation between the first output pose of the visual-inertial odometry and the pose of the satellite navigation device after initialization will not be too large. At this point, the body coordinate system of the visual-inertial odometry can be transformed to the global coordinate system based on the attitude of the starting point.

[0096] S203, the visual inertial odometer outputs local pose data to the computing platform.

[0097] After initialization, the visual-inertial odometry (VIO) processes image data and IMU data to obtain local pose data. Related processing methods can be used to configure and process the image data and IMU data to obtain the local pose data; this application does not limit this approach. The IMU data at this time is either the first IMU data or the second IMU data selected based on the reception status of the navigation device's second IMU data. If the reception of the navigation device's second IMU data is normal, the second IMU data is selected as the IMU data; if the reception of the navigation device's second IMU data is abnormal, the first IMU data is selected as the IMU data.

[0098] The visual inertial odometry system outputs the processed local pose data to the computing platform.

[0099] S204. The navigation device outputs the second IMU data and global pose data to the visual inertial navigation odometer, and outputs the global pose data to the computing platform.

[0100] In this embodiment, the navigation device participates in the entire positioning process, not only in the loose combination at the backend but also in the initialization of the visual inertial odometry (VIO) to enhance the stability of visual feature constraints. Therefore, the navigation device outputs second IMU data and global pose data to the IVO for reference.

[0101] The navigation device processes the second IMU data and other acquired navigation information to obtain global pose data, and then outputs the global pose data to the computing platform. Related processing methods can be used to process the second IMU data and other acquired navigation information to obtain the global pose data; this application does not limit this approach. The navigation device of this application obtains the global pose data using carrier phase differential positioning (RTK) and then sends it to the computing platform.

[0102] Carrier phase differential positioning (RTK) technology is a real-time differential method that processes carrier phase observations from two measurement stations. The carrier phase data acquired by the base station is sent to the user receiver for differential calculation of coordinates. This is a new and commonly used satellite positioning measurement method. Previous static, rapid static, and dynamic measurements all required post-processing to achieve centimeter-level accuracy, while RTK can achieve centimeter-level positioning accuracy in real-time in the field. It employs a dynamic real-time differential carrier phase method, bringing new measurement principles and methods to engineering layout, topographic mapping, and various control surveys, greatly improving operational efficiency.

[0103] It should be noted that steps S204 and S201 are not related in any order.

[0104] S205 The computing platform receives local pose data output from the visual inertial navigation odometry and global pose data output from the navigation device.

[0105] In this embodiment, a computing platform is configured to fuse the pose data of the visual inertial odometer and the navigation device. Therefore, the computing platform receives the local pose data output by the visual inertial odometer and the global pose data output by the navigation device.

[0106] S206. The computing platform performs loosely coupled fusion processing on local pose data and global pose data, and outputs the resulting positioning data as the positioning result.

[0107] Due to the specific characteristics of each sensor and factors such as noise, the output pose and trajectory of the IMU+camera combination in a visual inertial odometry system cannot perfectly coincide with the satellite positioning of the navigation device, even after conversion to a unified coordinate system for alignment. The navigation device can use RTK positioning. Because RTK positioning is highly accurate, it can be used when the navigation device receives a strong signal. However, when the satellite signal weakens or is lost (e.g., due to obstruction from tall buildings), the entire system needs to switch to the visual inertial odometry output. Since there are inherent discrepancies between the two, this instantaneous switching will cause momentary jitter and fluctuations. This discrepancy includes: 1) coordinate system transformation error; 2) the noise models of each sensor are different, resulting in unavoidable deviations in the relative pose estimation results.

[0108] See Figure 6If the navigation device were to switch directly to the visual inertial odometry (VIO) state when the satellite signal is lost, a state jump would occur, which is unacceptable for the control system. This embodiment employs a loose combination of the navigation device's global satellite pose and the VIOMA. When the satellite signal is good, the system will adjust the VIOMA's calculated value towards the satellite's global pose. However, when the satellite signal is lost or weakened, the system will maintain a smooth trajectory, and in the short term, this smooth trajectory will have a relatively small error compared to the true value.

[0109] In the embodiments of this application, the loose combination of the backend navigation device and the visual inertial odometer can be implemented using a nonlinear optimization method, but is not limited to this.

[0110] In this application, the transformation relationship between the local pose data of the visual inertial navigation odometry (VIO) and the pose data in the global coordinate system is denoted as T_. This transformation relationship value T_ also has a difference with the actual position transformation relationship value, denoted as T_gps. In other words, the local pose data of the VIOS is transformed into a transformed pose data in the global coordinate system, and this transformed pose data has a transformation relationship value T with the global pose data of the navigation device. The actual position transformation relationship value can be obtained from relevant technologies, and this application does not limit it.

[0111] In this embodiment, when visual underconstraint occurs in the visual inertial odometry, positioning data is determined based on the global pose data output by the navigation device, and the positioning data is output as the positioning result. For example, when the covariance matrix of the visual inertial odometry is abnormal, the transformation relationship value between the local pose data of the visual inertial odometry after transformation to the global coordinate system and the global pose data is obtained. The difference between the transformation relationship value and the actual position transformation relationship value is determined based on the transformation relationship value. The positioning data is obtained by multiplying the global pose data output by the navigation device and the inverse of the difference, and the positioning data is output as the positioning result. That is, when the variance matrix of the visual inertial odometry is abnormal (the covariance is too large, indicating that it has entered an underconstrained state), the pose data of the visual inertial odometry is discarded, the optimization calculation is stopped, and the last stable transformation T_ and T_gps are maintained. The system output pose = (inverse of T_gps) * satellite navigation output pose, until the visual constraints are reconstructed. Covariance represents the linear correlation between two random variables. It is a population parameter used to measure the magnitude of the cooperative variation between two variables, that is, the magnitude of the mutual influence between the two variables. The larger the absolute value of the covariance, the greater the mutual influence between the two variables. When the covariance exceeds a set threshold, it is considered that the covariance is too large. This threshold can be determined empirically and is not limited in this application.

[0112] In this embodiment, when the navigation device experiences signal anomalies, positioning data is determined based on the local pose data output by the visual inertial navigation odometry (VIO) system, and this positioning data is output as the positioning result. For example, when the covariance matrix of the navigation device is abnormal, the transformation relationship value between the local pose data of the VIOMS system (after converting it to the global coordinate system) and the global pose data is obtained; the positioning data is obtained by multiplying the local pose data output by the VIOMS system with the transformation relationship value, and this positioning data is output as the positioning result. In other words, if the covariance of the satellite navigation is too large, the system output pose = (T_) * VIOMS output pose until the satellite signal improves again.

[0113] See Figure 7 When all sensors are functioning correctly, this loosely coupled fusion process uses satellite global positioning observations from the navigation device to provide constraints for correcting the visual-inertial odometry (VIO) estimates. When the navigation device loses satellite signals, the entire trajectory constraint from the point of loss is constructed by the camera and IMU of the VIOS. When the tightly coupled camera + IMU system in the VIOS experiences image blurring and tracking loss (the output will diverge in a purely visual odometry system), the entire motion state constraint is uniquely constructed using the satellite positioning observations from the navigation device. In VIOS, the tight coupling of the camera + IMU can be achieved using either nonlinear optimization or filtering-based methods.

[0114] Therefore, using the scheme of this application, trajectory convergence can be achieved through the global position constraints of the navigation device's satellites even when the visual odometry is under-constrained. When the satellite signal from the navigation device is poor, trajectory convergence can be achieved through the visual and IMU constraints of the visual odometry, maintaining stable positioning output. This second-order loose combination of visual inertial navigation and satellite navigation positioning ensures that the failure or under-constraint of data from either component will not cause the overall system to diverge. Even if visual feature tracking is lost or temporarily lacks texture, stable output can still be achieved through satellite constraints.

[0115] Corresponding to the aforementioned application function implementation method embodiments, this application also provides a fusion positioning device, a flying car, an electronic device, and corresponding embodiments.

[0116] Figure 8 This is a schematic diagram of the fusion positioning device shown in the embodiments of this application.

[0117] See Figure 8 The fusion positioning device 80 provided in this application embodiment includes: a first input module 81, a second input module 82, and a fusion positioning module 83.

[0118] The first input module 81 is used to acquire local pose data output by a visual inertial navigation odometry (VIO). This local pose data is obtained by tightly coupling and fusing visual data and IMU data. The IMU data is obtained by processing first IMU data from the VIOMA and second IMU data from the navigation device. The VIOMA includes a camera and a first IMU device; the visual data comes from the camera, and the first IMU data comes from the first IMU device. The navigation device includes a second IMU device, and the second IMU data comes from the second IMU device. The camera can be a stereo camera. The processing of the IMU data includes: if the reception of the second IMU data from the navigation device is normal, selecting the second IMU data as the IMU data; if the reception of the second IMU data from the navigation device is abnormal, selecting the first IMU data as the IMU data.

[0119] The second input module 82 is used to acquire the global pose data output by the navigation device. Specifically, the second input module 82 acquires the global pose data obtained by the navigation device using the RTK method.

[0120] The fusion positioning module 83 is used to perform loosely coupled fusion processing on local pose data and global pose data, and output the positioning data obtained after fusion processing as the positioning result.

[0121] The fusion positioning module 83 includes: a first processing submodule 831 and a second processing submodule 832.

[0122] The first processing submodule 831 is used to determine positioning data based on the local pose data output by the visual inertial navigation odometry when the navigation device experiences signal abnormalities, and output the positioning data as the positioning result. For example, when the covariance matrix of the navigation device is abnormal, the transformation relationship value between the local pose data of the visual inertial navigation odometry after transformation to the global coordinate system and the global pose data is obtained; the positioning data is obtained by multiplying the local pose data output by the visual inertial navigation odometry and the transformation relationship value, and the positioning data is output as the positioning result.

[0123] The second processing submodule 832 is used to determine positioning data based on the global pose data output by the navigation device when visual underconstraint occurs in the visual inertial odometry, and output the positioning data as the positioning result. For example, when the covariance matrix of the visual inertial odometry is abnormal, the transformation relationship value between the local pose data of the visual inertial odometry after transformation to the global coordinate system and the global pose data is obtained, and the difference between the transformation relationship value and the actual position transformation relationship value is determined based on the transformation relationship value; the positioning data is obtained by multiplying the global pose data output by the navigation device and the inverse of the difference, and the positioning data is output as the positioning result.

[0124] The technical solution of this application adopts a fusion positioning method combining visual inertial odometry (VIO) with navigation equipment. In the process of tightly coupling and fusing visual data and IMU data to obtain local pose data, the second IMU data of the navigation equipment is also considered to improve the positioning accuracy of the VIO. Then, the VIO and navigation equipment are loosely coupled, that is, the local pose data and global pose data are loosely coupled and fused. Finally, the fused positioning data is output as the positioning result. Through the above processing, the navigation equipment participates in the entire positioning process, including participating in the tightly coupled fusion of front-end visual data and IMU data, and participating in the loose coupling process of the back-end VIO and navigation equipment. This solves the problems of unstable VIO output and inaccurate positioning in outdoor flight scenarios, providing more accurate and stable positioning in outdoor flight scenarios, and providing more accurate and stable visual positioning and navigation support for the stable flight of aircraft such as flying cars.

[0125] This application also provides a flying car, including, for example... Figure 8 The fusion positioning device 80 in the middle.

[0126] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0127] Figure 9 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application. The electronic device may be, for example, a flying car, but is not limited to this.

[0128] See Figure 9 The electronic device 1000 includes a memory 1010 and a processor 1020.

[0129] The processor 1020 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0130] Memory 1010 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by processor 1020 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 1010 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 1010 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, a high-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0131] The memory 1010 stores executable code, which, when processed by the processor 1020, can cause the processor 1020 to execute part or all of the methods described above.

[0132] Furthermore, the method according to this application can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing some or all of the steps in the method described above.

[0133] Alternatively, this application may be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) storing executable code (or computer program or computer instruction code) thereon, which, when executed by a processor of an electronic device (or server, etc.), causes the processor to perform part or all of the steps of the methods described above according to this application.

[0134] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A fusion positioning method, characterized in that, include: The local pose data output by the visual inertial navigation odometry is acquired, wherein the local pose data is obtained by tightly coupling and fusing visual data and IMU data, wherein the IMU data is obtained by processing the first IMU data of the visual inertial navigation odometry and the second IMU data of the navigation device; The visual inertial odometry includes a camera and a first IMU device, wherein the visual data comes from the camera and the first IMU data comes from the first IMU device; the navigation device includes a global navigation satellite system board and a second IMU device, wherein the second IMU data comes from the second IMU device; Obtain the global pose data output by the navigation device; The local pose data output by the visual inertial navigation odometry and the global pose data output by the navigation device are loosely coupled and fused. The resulting positioning data is then output as the positioning result. This includes: when the navigation device experiences a signal abnormality, determining the positioning data based on the local pose data output by the visual inertial navigation odometry and outputting the positioning data as the positioning result. When the navigation device experiences a signal anomaly, determining positioning data based on the local pose data output by the visual inertial odometer and outputting the positioning data as the positioning result includes: when the covariance matrix of the navigation device is abnormal, obtaining the transformation relationship value between the local pose data of the visual inertial odometer converted to the global coordinate system and the global pose data output by the navigation device; multiplying the local pose data output by the visual inertial odometer with the transformation relationship value to obtain the positioning data, and outputting the positioning data as the positioning result.

2. The method according to claim 1, characterized in that, The IMU data is obtained by processing the first IMU data from the visual inertial odometry system and the second IMU data from the navigation device, including: If the reception of the second IMU data by the second IMU device of the navigation device is normal, the second IMU data is selected as the IMU data; If the reception of the second IMU data of the second IMU device of the navigation device is abnormal, the first IMU data is selected as the IMU data.

3. The method according to claim 1, characterized in that, The step of loosely coupling and fusing the local pose data with the global pose data, and outputting the resulting localization data as the localization result, further includes: When the visual inertial odometry exhibits visual underconstraint, positioning data is determined based on the global pose data output by the navigation device, and the positioning data is output as the positioning result.

4. The method according to claim 3, characterized in that, When visual underconstraint occurs in the visual-inertial odometry, the positioning data is determined based on the global pose data output by the navigation device, and the positioning data is output as the positioning result, including: When the covariance matrix of the visual inertial navigation odometry is abnormal, the transformation relationship value between the local pose data of the visual inertial navigation odometry after it is transformed to the global coordinate system and the global pose data output by the navigation device is obtained, and the difference between the transformation relationship value and the actual position transformation relationship value is determined based on the transformation relationship value. The positioning data is obtained by multiplying the global pose data output by the navigation device with the inverse of the difference, and the positioning data is output as the positioning result.

5. The method according to claim 1, characterized in that, The step of acquiring the global pose data output by the navigation device includes: The global pose data output by the navigation device using carrier phase differential positioning (RTK) is obtained.

6. The method according to any one of claims 1 to 5, characterized in that: During initialization, the visual-inertial odometry system initializes itself based on visual data, IMU data, and global pose data output by the navigation device; wherein, PnP pose calculation is performed on the visual data, and pose calculation is performed after integral processing of the IMU data; The pose calculation results of the visual data and the pose calculation results of the IMU data are filtered together to obtain updated pose transformation data; The filtered pose transformation data is combined with the global pose data output by the navigation device as feedback to correct the visual data of the previous frame of the visual inertial navigation odometry.

7. A fusion positioning device, characterized in that, include: The first input module is used to acquire local pose data output by the visual inertial navigation odometry, wherein the local pose data is obtained by tightly coupling and fusing visual data and IMU data, and wherein the IMU data is obtained by processing the first IMU data of the visual inertial navigation odometry and the second IMU data of the navigation device. The visual inertial odometry includes a camera and a first IMU device, wherein the visual data comes from the camera and the first IMU data comes from the first IMU device; the navigation device includes a global navigation satellite system board and a second IMU device, wherein the second IMU data comes from the second IMU device; The second input module is used to acquire the global pose data output by the navigation device; The fusion positioning module is used to loosely fuse the local pose data output by the visual inertial navigation odometry with the global pose data output by the navigation device, and output the positioning data obtained after fusion processing as the positioning result. The fusion positioning module includes: a first processing submodule, used to determine positioning data based on the local pose data output by the visual inertial navigation odometry when the navigation device experiences signal abnormality, and output the positioning data as a positioning result; The first processing submodule is specifically used to obtain the transformation relationship value between the local pose data of the visual inertial navigation odometry after it is transformed to the global coordinate system and the global pose data output by the navigation device when the covariance matrix of the navigation device is abnormal; multiply the local pose data output by the visual inertial navigation odometry and the transformation relationship value to obtain positioning data, and output the positioning data as the positioning result.

8. The apparatus according to claim 7, characterized in that, The fusion positioning module also includes: The second processing submodule is used to determine positioning data based on the global pose data output by the navigation device when the visual inertial navigation odometry shows visual underconstraint, and output the positioning data as the positioning result.

9. A fusion positioning system, characterized in that, include: Visual-inertial odometry is used to output local pose data; Navigation equipment used to output global pose data and second IMU data; A computing platform is used to acquire local pose data output by a visual inertial navigation odometry system, wherein the local pose data is obtained by tightly coupling and fusing visual data and IMU data, wherein the IMU data is obtained by processing a first IMU data from the visual inertial navigation odometry system and a second IMU data from the navigation device. The visual inertial odometry includes a camera and a first IMU device, the visual data comes from the camera, and the first IMU data comes from the first IMU device; the navigation device includes a global navigation satellite system board and a second IMU device, the second IMU data comes from the second IMU device; the global pose data output by the navigation device is acquired; The local pose data output by the visual inertial navigation odometry (VIO) and the global pose data output by the navigation device are loosely coupled and fused. The resulting positioning data is then output as the positioning result. This includes: when the navigation device experiences a signal anomaly, determining positioning data based on the local pose data output by the VIO, and outputting this positioning data as the positioning result; specifically, when the covariance matrix of the navigation device is abnormal, the computing platform obtains the transformation relationship value between the local pose data of the VIO, after converting it to the global coordinate system, and the global pose data output by the navigation device; multiplying the local pose data output by the VIO and the transformation relationship value to obtain positioning data, and outputting this positioning data as the positioning result.

10. A flying car, characterized in that, Includes the fusion positioning device as described in any one of claims 7 to 8.

Images