Method for determining vehicle position through multi-sensor fusion

Abstract

A method for determining the position of a vehicle includes receiving a sequence of camera images based on a camera carried by the vehicle and determining a camera pose based on the sequence of camera images. Determining a global positioning system position based on a global positioning system receiver carried by the vehicle, determining an inertial movement signal based on an inertial measurement unit carried by the vehicle, and receiving a wheel encoder signal from the wheels of the vehicle. The method further includes determining at least one of a vehicle position and a vehicle speed based on at least two of a camera pose, a global positioning system position, an inertial movement signal, and a wheel encoder signal that are synchronized in time.

Description

Technical Field

[0001] The present disclosure relates to autonomous vehicle positioning, and more particularly to improving vehicle positioning accuracy through multi-sensor fusion. Background Art

[0002] Currently, vehicle positioning relies on the Global Positioning System (GPS) / Inertial Measurement Unit (IMU). In environments where GPS signals are reduced or absent, such as tunnels and underground garages, the use of GPS / IMU for vehicle positioning may be affected. Summary of the Invention

[0003] An example method for determining a vehicle position and a vehicle speed includes receiving a sequence of camera images by a camera carried by the vehicle and determining a camera pose based on the sequence of camera images. Determining a global positioning system position based on a global positioning system receiver carried by the vehicle, determining an inertial movement signal based on an inertial measurement unit carried by the vehicle, and receiving a wheel encoder signal from a wheel of the vehicle. The method further includes determining at least one of the vehicle position and the vehicle speed based on at least two of the camera pose, the global positioning system position, the inertial movement signal, and the wheel encoder signal that are time synchronized.

[0004] Another example method for determining a vehicle position and a vehicle speed includes receiving a sequence of camera images by a camera carried by the vehicle and determining a camera pose based on the sequence of camera images. The method includes determining a global positioning system position based on a global positioning system receiver carried by the vehicle, and determining an inertial movement signal based on an inertial measurement unit carried by the vehicle. The method further includes receiving a wheel encoder signal from a wheel of the vehicle and initializing the vehicle position and the vehicle speed based on a fusion of the camera pose, the global positioning system position, the inertial movement signal, and the wheel encoder signal. Brief Description of the Drawings

[0005] In the drawings:

[0006] Figure 1 is a first example system diagram according to an embodiment of the present disclosure;

[0007] Figure 2 is a second example system diagram according to an embodiment of the present disclosure;

[0008] Figure 3 is an example of a global navigation satellite system / inertial measurement unit receiver output according to an embodiment of the present disclosure;

[0009] Figure 4An example of visual odometry output according to an embodiment of the present disclosure;

[0010] Figure 5 An example of wheel encoder output according to an embodiment of the present disclosure;

[0011] Figure 6 An example method of sensor fusion for automatic vehicle positioning according to an embodiment of the present disclosure;

[0012] Figure 7 An example of visual odometry initialization according to an embodiment of the present disclosure;

[0013] Figure 8 An example of wheel encoder initialization according to an embodiment of the present disclosure;

[0014] Figure 9 An example method of bundle adjustment for a camera and a wheel encoder according to an embodiment of the present disclosure;

[0015] Figure 10 An example method of bundle adjustment for a camera and a global positioning system / inertial measurement unit receiver according to an embodiment of the present disclosure;

[0016] Figure 11 An example of visual odometry initialization in a system having a global positioning system / inertial measurement unit receiver, a camera, and a wheel encoder according to an embodiment of the present disclosure;

[0017] Figure 12 An example method of signal processing for a global positioning system / inertial measurement unit receiver, a camera, visual odometry, and a wheel encoder according to an embodiment of the present disclosure;

[0018] Figure 13 A first example method according to an embodiment of the present disclosure; and

[0019] Figure 14 A second example method according to an embodiment of the present disclosure. Detailed Description

[0020] The embodiments listed below are written only for illustrative purposes of the application of the present apparatus and method and do not limit its scope. Equivalent modifications of such apparatus and method should fall within the scope of the claims.

[0021] Throughout the following description and claims, certain terms are used to refer to particular system components. As those skilled in the art will appreciate, different groups may use different names to refer to components and / or methods. This document is not intended to distinguish between components and / or methods that have different names but the same function.

[0022] In the following description and claims, the terms "comprising" and "including" are used in an open-ended manner and thus can be interpreted to mean "including but not limited to...". Additionally, the term "coupled" or "coupling" is intended to mean an indirect or direct connection. Thus, if a first device is connected to a second device, that connection can be a direct connection or an indirect connection via other devices and connections.

[0023] Figure 1 An example hybrid computing system 100 is depicted, and the hybrid computing system 100 can be used to implement a neural network associated with the operation of one or more parts or steps of the process described in the appendix. Figures 13 to 14 In this example, the processors associated with the hybrid system include a Field Programmable Gate Array (FPGA) 122, a Graphical Processor Unit (GPU) 120, and a Central Processing Unit (CPU) 118.

[0024] The CPU 118, GPU 120, and FPGA 122 have the ability to provide a neural network. The CPU is a general-purpose processor that can execute many different functions, and the generality of the CPU results in the ability to perform a variety of different tasks. However, the CPU's processing of multiple data streams is limited, and the CPU's capabilities are limited with respect to neural network functions. The GPU is a graphics processor that has many small processing cores capable of processing parallel tasks sequentially. The FPGA is a field-programmable device that has the ability to be reconfigured and perform any function that can be programmed into the CPU or GPU in the form of hard-wired circuitry. Since the FPGA is programmed in the form of circuitry, its speed is many times faster than that of the CPU and significantly faster than that of the GPU.

[0025] The system may also include other types of processors, such as an Accelerated Processing Unit (APU) and a Digital Signal Processor (DSP). The APU includes a CPU with an on-chip GPU element, and the DSP is designed for high-speed digital data processing. An Application Specific Integrated Circuit (ASIC) can also perform the hard-wired functions of the FPGA. However, the lead time for designing and manufacturing an ASIC is approximately several quarters of a year, as opposed to the quick-turnaround implementation time available for programming the FPGA.

[0026] The graphics processing unit 120, the central processing unit 118, and the field programmable gate array 122 are connected to each other, and the graphics processing unit 120, the central processing unit 118, and the field programmable gate array 122 are connected to the memory interface controller 112. The FPGA is connected to the memory interface controller 112 through the programmable logic circuit to memory interconnect 130. This additional device is utilized because the FPGA operates at a very large bandwidth and to minimize the circuitry utilized by the FPGA to perform storage tasks. The memory interface controller 112 is additionally connected to the permanent memory disk 110, the system memory 114, and the read only memory (ROM) 116.

[0027] Figure 1 The system can be used to program and train the FPGA. The GPU works well with unstructured data and can be used for training. Once the data is trained, a deterministic inference model can be found, and the CPU can program the FPGA using the model data determined by the GPU.

[0028] The memory interface and controller are connected to the central interconnect 124, and the central interconnect is additionally connected to the GPU 120, the CPU 118, and the FPGA 122. The central interconnect 124 is additionally connected to the input and output interface 128 and the network interface 126.

[0029] Figure 2 A second example hybrid computing system 200 is depicted, and the hybrid computing system 200 can be used to implement a neural network related to the operation of one or more parts or steps of the process 1000. In this example, the processors related to the hybrid system include a field programmable gate array (FPGA) 210 and a central processing unit (CPU) 220.

[0030] The FPGA is electrically connected to the FPGA controller 212, and the FPGA controller 212 is interfaced with the Direct Memory Access (DMA) 218. The DMA is connected to the input buffer 214 and the output buffer 216, and the input buffer 214 and the output buffer 216 are coupled to the FPGA to buffer data into the FPGA and buffer data out of the FPGA, respectively. The DMA 218 includes two First In First Out (FIFO) buffers, one for the main CPU and the other for the FPGA. The DMA allows data to be written to the appropriate buffer and read from the appropriate buffer.

[0031] On the CPU side of the DMA is the main switch 228, which transfers data and commands to and from the DMA. The DMA is also connected to the SDRAM controller 224, which allows data to be transferred from the CPU 220 to the FPGA and allows data to be transferred from the FPGA to the CPU 220. The SDRAM controller is also connected to the external SDRAM 226 and the CPU 220. The main switch 228 is connected to the peripheral device interface 230. The flash controller 222 controls the persistent memory and is connected to the CPU 220.

[0032] The proposed vehicle positioning solution uses multiple sensors such as the Global Positioning System (GPS), Inertial Measurement Unit (IMU), camera-based Visual Odometry (VO), and wheel encoder-based Wheel Odometry (WO) to provide positioning for autonomous vehicles.

[0033] Figure 3 An example set of Global Positioning System (GPS) and / or Global Navigation Satellite System (GNSS) and Inertial Measurement Unit (IMU) receiver outputs 300 is depicted. GPS / GNSS is a set of satellite-based transmitters that transmit satellite positions, orbits, and precise time. This transmitted data is used to determine the ground position of an object based on differential timing of signal reception. GPS relies on access to GPS satellites or GPS ground-based signals. Using GPS and IMU 310, the position 312, speed 314, and attitude 316 of the receiver can be determined. When GPS signals are lost, the IMU uses internal sensors such as accelerometers, gyroscopes, magnetometers, and barometers to continue positioning the vehicle. A major drawback of IMUs is that they are affected by cumulative errors and are ineffective for use in navigation.

[0034] Figure 4 An example of a camera-based visual odometry output 400 is depicted. Visual Odometry (VO) allows the position and orientation of a camera to be determined based on images obtained from the camera. The visual odometry output from the camera 410 allows the camera pose, represented by rotation 412 and translation 414, to be determined. Visual odometry uses image sequences to estimate the motion of the camera in real time. A drawback of a visual odometry system alone is that the motion scale is unobservable.

[0035] Figure 5Illustrates an example 500 of wheel odometry (WO) based on a wheel encoder. The wheel encoder 510 allows wheel odometry to estimate the vehicle linear velocity 512 and yaw rate 514 based on wheel encoder measurements. The wheel encoder measures the radians the wheel travels and, using the wheel radius, can determine the distance traveled over time, i.e., the vehicle speed. One drawback of only the WO system is that only the distance traveled, speed, and yaw rate can be determined.

[0036] A fusion-based vehicle positioning solution can consist of sensors such as Global Positioning System (GPS), Inertial Measurement Unit (IMU), wheel encoder-based wheel odometry (WO), and camera-based visual odometry (VO). The proposed solution combines the sensors to overcome the drawbacks of each individual type of sensor used in autonomous vehicles.

[0037] In situations where Global Positioning System (GPS) signals are blocked or become very weak, a receiver integrated with an Inertial Measurement Unit can output positioning information through dead reckoning; however, this may cause cumulative errors. A possible solution is to utilize camera-based visual odometry and wheel encoder-based wheel odometry to supplement the Inertial Measurement Unit receiver and reduce IMU cumulative errors. In situations where multiple sensors (e.g., GPS and camera-based odometry in a very dark parking garage) may not function, the Inertial Measurement Unit receiver can receive the wheel odometry output to allow autonomous navigation to continue.

[0038] The proposed method combines sensors such as Global Positioning System, Inertial Measurement Unit, camera-based visual odometry, and wheel encoder-based wheel odometry to provide a positioning result for an autonomous vehicle via a Kalman filter. Due to the sensor fusion, if data is not available for multiple types of sensors, the method allows autonomous navigation to continue. This sensor fusion allows for a continued navigation function in challenging environments for autonomous vehicle positioning such as tunnels, city cannons, and underground garages. In the case where Global Positioning System signals are blocked, the visually odometry output of wheel encoder calibration allows for position detection through dead reckoning. In the case where both Global Positioning System and camera-based visual odometry are offline, the wheel encoder can output a positioning result to the IMU during the transition.

[0039] Figure 6Depicts an example system for sensor fusion for autonomous vehicle positioning 600. The system includes sensors 610, which are, for example, GPS / IMU 612, a camera 614 that allows visual odometry output, and a wheel encoder 616 that allows wheel odometry output. The sensor signals are sent to a sensor fusion module 618 to determine the vehicle position 620 and speed 622 using an Extended Kalman Filter (EKF).

[0040] A receiver integrated with a global positioning system and an inertial measurement unit can provide an initial reference positioning result for initializing camera-based visual odometry. Camera-based visual odometry scale drift can be calibrated through GPS / IMU distance measurements.

[0041] Figure 7 Depicts an example of visual odometry initialization 700. Position data and speed data are output by GPS / IMU 710, and Earth-Centered Earth-Fixed (ECEF) coordinates are transformed 714 into a set of East-North-Up (ENU) coordinates and / or a set of North-East-Down (NED) coordinates. The camera 712 outputs a sequence of camera images that serve as the basis for visual odometry 716. An initialization process 718 occurs based on the input of the transformed GPS / IMU coordinates for visual odometry. The output visual odometry 720 is transformed into a set of East-North-Up coordinates and / or a set of North-East-Down coordinates 722.

[0042] A receiver integrated with a global positioning system and an inertial measurement unit can provide an initial reference positioning result for initializing wheel encoder-based wheel odometry. Wheel encoder external parameters based on the wheel encoder can be calibrated through GPS / IMU distance measurements.

[0043] Figure 8 Depicts an example of wheel encoder initialization 800. Position data and speed data are output by GPS / IMU 810, and the Earth-Centered Earth-Fixed coordinates are transformed 814 into a set of East-North-Up coordinates and / or a set of North-East-Down coordinates. The wheel encoder 812 outputs data that serves as the basis for wheel odometry 816. An initialization process 818 occurs based on the input of the transformed GPS / IMU coordinates and wheel odometry 816. Wheel encoder external parameters can be determined during the initialization process. The output wheel odometry 820 is transformed into a set of East-North-Up coordinates and / or a set of North-East-Down coordinates 822.

[0044] When vision odometry based on an independent camera is used in an autonomous vehicle, due to the cumulative scale drift error that may affect the vision odometry result, the vision odometry based on the independent camera may not accurately identify the vehicle position. Wheel odometry based on a wheel encoder can allow calibration of the vision odometry scale by determining the external parameters of the wheel encoder to zero the scale drift.

[0045] Figure 9 An example method 900 for bundle adjustment 910 of a camera and a wheel encoder is depicted. The camera pose is based on a sequence of camera images 914 from the camera to output vision odometry. The wheel encoder 912 outputs wheel odometry. The wheel encoder 912 can provide distance measurements between two adjacent camera image 914 epochs. Time synchronization 916 allows determination of pose optimization 920. The wheel encoder external parameters 918 link the vision odometry scale to global coordinates based on the wheel encoder and camera calibration process. The camera sensor can provide a sequence of image data, and the wheel encoder measurements can be used to constrain the camera pose in a local bundle adjustment.

[0046] A receiver integrated with a global positioning system and an inertial measurement unit can provide pose optimization results for camera-based vision odometry. The scale drift of the camera-based vision odometry can be calibrated by GPS / IMU distance measurements.

[0047] Figure 10 An example method 1000 for bundle adjustment 1010 of camera images 1014 and data 1012 from a global positioning system and an inertial measurement unit receiver is depicted. Time synchronization 1016 allows determination of camera pose optimization 1018. The global positioning system inertial measurement unit receiver can provide accurate position information, which can be used to constrain the camera pose in a local bundle adjustment.

[0048] The time-synchronized sensor data sets can be processed by the vision odometry system in a simultaneous or near-simultaneous manner during initialization. The camera images can perform a vision odometry initialization process before GPS / IMU initialization or wheel odometry based on the wheel encoder. The global positioning system and inertial measurement unit receiver can provide global position information for the vision odometry system. The camera pose output by the vision odometry can be synchronized with the global positioning system position information. After the vision odometry initialization process, the camera pose can be converted to global positioning system global coordinates based on simultaneous localization and mapping global coordinates. The wheel encoder measurements can be determined between two initialized camera image epochs and can be initialized to be synchronized with the global positioning system global coordinates. Thus, when the global positioning system, inertial measurement unit receiver, and camera have interrupted the data stream, the wheel encoder can independently output global positioning information.

[0049] Figure 11 Illustrates an example method 1100 for visual odometry initialization 1110. The system initialization process can be based on global positioning system and inertial measurement unit receiver data 1112, camera-based visual odometry using a sequence of camera images 1114, and wheel odometry using data from wheel encoders 1116. In this method, visual odometry initialization 1118 is completed before system initialization 1120. The system initialization receives GPS / IMU data, visual odometry data, and wheel odometry data, and uses these three data sets to initialize the system.

[0050] During navigation, a time-synchronized sensor data set can be processed by the visual odometry system in a simultaneous or near-simultaneous manner. The global positioning system and inertial measurement unit receiver, as well as wheel odometry based on wheel encoders, can provide global position information to the visual odometry system. The camera pose output by the visual odometry can be synchronized with the global positioning system position information. After the visual odometry initialization process, the camera pose can be converted to global positioning system global coordinates based on simultaneous localization and mapping of global coordinates. Wheel encoder measurements can be determined between two initialized camera image epochs and can output position results synchronized with the global positioning system global coordinates. Thus, when the data streams of the global positioning system, inertial measurement unit receiver, and camera have been interrupted, the wheel encoder can independently output global positioning information.

[0051] Figure 12 Illustrates an example method 1200 for signal processing of a global positioning system and inertial measurement unit receiver, camera, visual odometry, and wheel encoder. After system initialization ( Figure 11 、1100) is completed ( Figure 11 、1120, Figure 12 、1210), visual odometry tracking 1212 can begin. In this example method, a sequence of camera images 1216 is used to determine an initial camera pose 1218 based on visual odometry. Data from the GPS / IMU receiver 1214 and wheel odometry based on wheel encoder data 1220 are routed to a visual odometry scale drift calibration module 1222. The data from the GPS / IMU, visual odometry data, and wheel odometry data are fused 1224 and subjected to an extended Kalman filter. The visually odometry output calibrated by the wheel encoder and the outputs of the global positioning system and inertial measurement unit receiver can be processed together by a Kalman filter for sensor fusion. The data from the extended Kalman filter is used to determine vehicle position information 1226 represented in GPS global coordinates.

[0052] The camera pose between two adjacent camera image sequences can be determined by simultaneous or near-simultaneous localization and mapping. The wheel encoder measurements between two adjacent camera image sequences can be used to assist in calibrating the visual odometry scale. The global positioning system and inertial measurement unit receivers can provide positioning information, which can be used to calibrate the visual odometry represented in global coordinates.

[0053] In the absence of GPS data reception, wheel odometry based on the wheel encoder can be used to initialize the visual odometry scale represented in global coordinates. The wheel encoder can be used to calibrate the visual odometry represented in global coordinates because it has been initialized with the global positioning system in global coordinates during the system initialization process. By using the wheel odometry measurements based on the wheel encoder in the visual odometry tracking loop, due to the scale drift error, the visual odometry scale can be made more accurate than visual odometry alone.

[0054] If the global positioning system / inertial measurement unit receiver and the camera have data interruptions, the wheel odometry based on the wheel encoder can assist the visual odometry tracking loop to output positioning information. When the visual odometry system recovers from tracking loss, the previous pose of the wheel encoder can be used to initialize the visual odometry system. When the global positioning system / inertial measurement unit receiver has data recovery, a bundle adjustment can be performed on the system.

[0055] An example method for determining the vehicle position includes: receiving 1310 a camera image sequence based on a camera carried by the vehicle, and determining 1312 the camera pose based on the camera image sequence. The method includes determining 1314 the global positioning system position based on a global positioning system receiver carried by the vehicle, determining 1316 an inertial movement signal based on an inertial measurement unit carried by the vehicle, and receiving 1318 a wheel encoder signal from the wheels of the vehicle. The method further includes determining 1320 at least one of the vehicle position and the vehicle speed based on at least two of the camera pose, the global positioning system position, the inertial movement signal, and the wheel encoder signal that are synchronized in time.

[0056] The method may further include correcting the camera pose based on at least one of global positioning system (GPS) position, inertial motion signal, and wheel encoder signal filtering and / or extended Kalman filtering of at least one of vehicle position and vehicle speed. The method may further include converting at least one of vehicle position and vehicle speed into GPS position, and / or converting a set of GPS earth-centered earth-fixed coordinates into a set of north-east-down coordinates, and / or converting a set of GPS earth-centered earth-fixed coordinates into a set of north-east-earth coordinates. The method may further include correcting the camera pose based on external parameters of the wheel encoder. The method may further include initializing at least one of vehicle position and vehicle speed based on the fusion of camera pose, GPS position, inertial motion signal, and wheel encoder signal.

[0057] Another example method for determining vehicle position includes: receiving a sequence of camera images 1410 based on a camera carried by the vehicle, and determining 1412 the camera pose based on the sequence of camera images. The method includes determining 1414 the GPS position based on a GPS receiver carried by the vehicle, and determining 1416 the inertial motion signal based on an inertial measurement unit carried by the vehicle. The method further includes receiving 1418 a wheel encoder signal from the wheels of the vehicle, and initializing 1420 the vehicle position and vehicle speed based on the fusion of camera pose, GPS position, inertial motion signal, and wheel encoder signal.

[0058] The method may further include correcting the camera pose based on at least one of global positioning system (GPS) position, inertial motion signal, and wheel encoder signal filtering and / or extended Kalman filtering of at least one of vehicle position and vehicle speed. The method may further include converting at least one of vehicle position and vehicle speed into GPS position, and / or converting a set of GPS earth-centered earth-fixed coordinates into a set of north-east-down coordinates, and / or converting a set of GPS earth-centered earth-fixed coordinates into a set of north-east-earth coordinates. The method may further include correcting the camera pose based on external parameters of the wheel encoder.

[0059] Those skilled in the art will appreciate that the various illustrative blocks, modules, elements, components, methods, and algorithms described herein may be implemented as electronic hardware, computer software, or a combination of both. To illustrate this interchangeability of hardware and software, the various illustrative blocks, modules, elements, components, methods, and algorithms have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the system. Skilled artisans may implement the described functionality in varying ways for each particular application. Without departing from the scope of the subject technology, the various components and blocks may be differently arranged (e.g., arranged in a different order or partitioned in a different manner).

[0060] It should be understood that the specific order or hierarchy of steps in the disclosed processes is an illustration of example methods. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the processes can be rearranged. Some steps can be performed simultaneously. The accompanying method claims present the elements of the various steps in an example order and are not meant to be limited to the specific order or hierarchy presented.

[0061] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The foregoing description provides various examples of the subject technology, and the subject technology is not limited to these examples. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects. Thus, it is not intended that the claims be limited to the aspects shown herein, but rather that the full scope be consistent with the language of the claims, where elements recited in the singular are not intended to mean "one and only one" unless explicitly so stated, but rather "one or more." Unless otherwise expressly stated, the term "some" means one or more. Masculine pronouns (e.g., "his") include feminine and neuter genders (e.g., "her" and "its"), and vice versa. Headings and subheadings, if any, are for convenience only and do not limit the invention. The recitation of the terms "configured to," "operable to," and "programmed to" does not mean any specific tangible or intangible modification to an object, but rather is intended to be used interchangeably. For example, a processor configured to monitor and control an operation or component can also mean that the processor is programmed to monitor and control the operation or that the processor is operable to monitor and control the operation. Similarly, a processor configured to execute code can be interpreted as a processor programmed to execute code or a processor operable to execute code.

[0062] Phrases such as "aspect" do not imply that such an aspect is essential to the claimed subject matter or that such an aspect is applicable to configurations of the claimed subject matter. Disclosure related to an aspect may apply to a configuration, or to one or more configurations. An aspect may provide one or more examples. Phrases such as "aspect" may refer to one or more aspects, and vice versa. Phrases such as "embodiment" do not imply that such an embodiment is essential to the claimed subject matter or that such an embodiment is applicable to configurations of the claimed subject matter. Disclosure related to an embodiment may apply to an embodiment, or to one or more embodiments. An embodiment may provide one or more examples. Phrases such as "embodiment" may refer to one or more embodiments, and vice versa. Phrases such as "configuration" do not imply that such a configuration is essential to the claimed subject matter or that such a configuration is applicable to configurations of the claimed subject matter. Disclosure related to a configuration may apply to a configuration, or to one or more configurations. A configuration may provide one or more examples. Phrases such as "configuration" may refer to one or more configurations, and vice versa.

[0063] The term "example" is used herein to mean "serving as an example or illustration". Any aspect or design described herein as an "example" is not necessarily to be construed as more preferred or advantageous than other aspects or designs.

[0064] Structural and functional equivalents of elements of the various aspects described throughout this disclosure that are known or later become known to those of ordinary skill in the art are hereby expressly incorporated by reference and are intended to be covered by the claims. Moreover, the disclosure herein is not intended to be dedicated to the public even if the disclosure is not expressly recited in the claims. No element of any claim is to be construed under the provisions of 35 U.S.C. § 112, sixth paragraph, unless the element is expressly recited using the phrase "means for" or, in the case of a method claim, the phrase "step for" to recite the element. Further, to the extent that the specification or claims use the terms "including", "having", and the like, such terms are intended to be inclusive in a manner similar to the term "comprising" as interpreted when "comprising" is used as a transitional word in a claim.

[0065] References to "one embodiment", "an embodiment", "some embodiments", "various embodiments", etc. indicate that a particular element or feature is included in at least one embodiment of the invention. Although these phrases may appear in various places, these phrases do not necessarily refer to the same embodiment. In light of the disclosure herein, one of ordinary skill in the art will be able to design and combine any of the various mechanisms suitable for implementing the above functions.

[0066] It should be understood that the present disclosure only teaches an example of an illustrative embodiment, and those skilled in the art can easily design many variations of the present invention after reading the present disclosure, and the scope of the present invention will be determined by the following claims.

Claims

1. A method for determining a vehicle position, comprising: Receiving a sequence of camera images from a camera carried by the vehicle; The sequence of camera images determines a camera pose based on visual odometry of the camera to determine the position and orientation of the camera on the vehicle, and corrects the visual odometry of the camera based on at least one of a global positioning system position, an inertial movement signal, and a wheel encoder signal to correct the camera pose; Wherein the global positioning system position is determined based on a global positioning system receiver carried by the vehicle; The inertial movement signal is determined based on an inertial measurement unit carried by the vehicle; Receiving the wheel encoder signal from the wheels of the vehicle; And Determining at least one of a vehicle position and a vehicle speed based on a fusion of one type of sensor data synchronized in time with at least one different type of sensor data, the types of sensor data being selected from the group consisting of the camera pose, the global positioning system position, the inertial movement signal, and the wheel encoder signal; Wherein visual odometry based on the camera and wheel odometry based on the wheel encoder are used to supplement the inertial measurement unit receiver and reduce the cumulative error of the inertial measurement unit; and Wherein, due to scale drift error, the wheel odometry measurement based on the wheel encoder in the visual odometry tracking loop enables the visual odometry scale to be more accurate than the visual odometry alone.

2. The method according to claim 1, wherein Further comprising filtering at least one of the vehicle position and the vehicle speed.

3. The method according to claim 1, characterized in that, Further comprising performing an extended Kalman filter on at least one of the vehicle position and the vehicle speed.

4. The method according to claim 1, wherein Further comprising converting at least one of the vehicle position and the vehicle speed into the global positioning system position.

5. The method according to claim 1, characterized in that, Further comprising converting a set of global positioning system earth-centered earth-fixed coordinates into a set of north-east-down coordinates.

6. The method according to claim 1, wherein Further comprising converting a set of global positioning system earth-centered earth-fixed coordinates into a set of north-east-earth coordinates.

7. The method according to claim 1, wherein Further comprising initializing at least one of the vehicle position and the vehicle speed based on a fusion of one type of sensor data with at least one different type of sensor data, the types of sensor data being selected from the group consisting of the camera pose, the global positioning system position, the inertial movement signal, and the wheel encoder signal.

8. The method according to claim 1, characterized in that Further comprising correcting the camera pose based on wheel encoder external parameters.

9. The method according to claim 1, characterized in that, In a case where multiple sensors do not work, the inertial measurement unit receiver can receive the output of the wheel odometry to allow automatic navigation to continue.

10. The method according to claim 1, characterized in that, In a case where the global positioning system signal is blocked, the visual odometry output corrected by the wheel encoder signal allows position detection by dead reckoning. In a case where both the global positioning system and the camera-based visual odometry are offline, the wheel encoder can output a positioning result to the inertial measurement unit during the transition period.

11. The method according to claim 1, characterized in that, In a case where no global positioning system data is received, the wheel odometry based on the wheel encoder can be used to initialize the visual odometry scale.

12. The method according to claim 1, wherein If the Global Positioning System / Inertial Measurement Unit receiver and the camera have data interruptions, the wheel odometry based on the wheel encoder can assist the visual odometry tracking loop to output positioning information.

13. The method according to claim 12, wherein When the visual odometry system recovers from a tracking loss, the previous attitude of the wheel encoder can be used to initialize the visual odometry system, and when the Global Positioning System / Inertial Measurement Unit receiver has data recovery, bundle adjustment can be performed on the system.

14. A method for determining a vehicle position, comprising: Receiving a sequence of camera images from a camera carried by the vehicle; The sequence of camera images determines the camera attitude based on the visual odometry of the camera to determine the position and orientation of the camera on the vehicle, and corrects the visual odometry of the camera based on at least one of a Global Positioning System position, an inertial movement signal, and a wheel encoder signal to correct the camera attitude; Wherein the Global Positioning System position is determined based on a Global Positioning System receiver carried by the vehicle; The inertial movement signal is determined based on an Inertial Measurement Unit carried by the vehicle; Receiving the wheel encoder signal from the wheels of the vehicle; And Determining the vehicle position and vehicle speed based on the fusion of one type of sensor data and at least one different type of sensor data, the types of sensor data being selected from the group consisting of the camera attitude, the Global Positioning System position, the inertial movement signal, and the wheel encoder signal; Wherein the visual odometry based on the camera and the wheel odometry based on the wheel encoder are used to supplement the Inertial Measurement Unit receiver and reduce the cumulative error of the Inertial Measurement Unit; and Wherein, due to scale drift error, the wheel odometry measurement based on the wheel encoder in the visual odometry tracking loop can make the visual odometry scale more accurate than the visual odometry alone.

15. The method according to claim 14, characterized in that Further comprising filtering at least one of the vehicle position and the vehicle speed.

16. The method according to claim 14, wherein Further comprising performing an extended Kalman filter on at least one of the vehicle position and the vehicle speed.

17. The method according to claim 14, wherein Further comprising converting at least one of the vehicle position and the vehicle speed to the Global Positioning System position.

18. The method according to claim 14, wherein Further comprising converting a set of Global Positioning System Earth-Centered Earth-Fixed coordinates into a set of North-East-Down coordinates.

19. The method according to claim 14, characterized in that, Further comprising converting a set of Global Positioning System Earth-Centered Earth-Fixed coordinates into a set of North-East-Earth coordinates.

20. The method according to claim 14, wherein Further comprising correcting the camera attitude based on the external parameters of the wheel encoder.

21. The method according to claim 14, wherein In a situation where multiple sensors do not work, the Inertial Measurement Unit receiver can receive the output of the wheel odometry to allow automatic navigation to continue.

22. The method according to claim 14, wherein In a situation where the Global Positioning System signal is blocked, the visual odometry output corrected by the wheel encoder signal allows position detection by dead reckoning. In a situation where both the Global Positioning System and the camera-based visual odometry are offline, the wheel encoder can output positioning results to the Inertial Measurement Unit during the transition period.

23. The method according to claim 14, wherein In a situation where no Global Positioning System data is received, the wheel odometry based on the wheel encoder can be used to initialize the visual odometry scale.

24. The method according to claim 14, characterized in that, If the global positioning system / inertial measurement unit receiver and the camera have data interruptions, wheel odometry based on wheel encoders can assist the visual odometry tracking loop to output positioning information.

25. The method according to claim 24, wherein When the visual odometry system recovers from a tracking loss, the previous pose of the wheel encoder can be used to initialize the visual odometry system. When the global positioning system / inertial measurement unit receiver has data recovery, bundle adjustment can be performed on the system.

Images