System and method for calibrating an inertial test unit and a camera

Abstract

The present application relates to a system for calibrating an inertial test unit and a camera of an autonomous vehicle. The system includes at least one storage medium including instructions for calibrating the inertial test unit and the camera, and at least one processor in communication with the storage medium. When executing the instructions, the at least one processor is configured to: obtain a trajectory (510) of the autonomous vehicle traveling in a straight line; determine an inertial test unit posture (520) of the inertial test unit relative to a first coordinate system; determine a camera posture (530) of the camera relative to a second coordinate system; determine a relative coordinate posture (540) between the first coordinate system and the second coordinate system; and determine a relative posture (550) between the camera and the inertial test unit based on the inertial test unit posture, the camera posture, and the relative coordinate posture. The present application also discloses a method for calibrating an inertial test unit and a camera of an autonomous vehicle and a non-transitory readable medium.

Description

Technical Field

[0001] The present application generally relates to systems and methods for autonomous driving, and more particularly to systems and methods for calibrating inertial measurement units (IMUs) and cameras of autonomous vehicles. Background Art

[0002] Autonomous vehicles, incorporating a variety of sensors, are becoming increasingly popular. Onboard inertial measurement units (ITUs) and cameras play a crucial role in autonomous driving. However, in some cases, calibration between the ITU and cameras is complex or indirect. Therefore, it is desirable to provide a system and method for calibrating ITUs and cameras in a simple and straightforward manner. Summary of the Invention

[0003] One aspect of the present application describes a system for calibrating an inertial test unit (ITU) and a camera of an autonomous vehicle. The system may include at least one storage medium including a set of instructions for calibrating the ITU and the camera; and at least one processor in communication with the storage medium, wherein when executing the instructions, the at least one processor is configured to: obtain a trajectory of the autonomous vehicle traveling in a straight line; determine an ITU pose of the ITU relative to a first coordinate system; determine a camera pose relative to a second coordinate system; determine a relative coordinate pose between the first coordinate system and the second coordinate system; and determine a relative pose between the camera and the ITU based on the ITU pose, the camera pose, and the relative coordinate pose.

[0004] In some embodiments, the at least one processor is further configured to determine the first coordinate system based on a trajectory of the autonomous vehicle.

[0005] In some embodiments, to determine the inertial test unit posture, the at least one processor is further configured to: obtain inertial test unit data from the inertial test unit; and determine the inertial test unit posture based on the inertial test unit data and the first coordinate system.

[0006] In some embodiments, the at least one processor is further configured to: acquire camera data from a camera; and determine the second coordinate system based on the camera data.

[0007] In some embodiments, to determine the camera pose, the at least one processor is further configured to: determine the camera pose based on the camera data and a second coordinate system.

[0008] In some embodiments, in order to determine the second coordinate system, the at least one processor is also used to: determine a second ground normal vector based on camera data and three-dimensional reconstruction technology; determine a second driving direction of the camera based on the camera data; and determine the second coordinate system based on the second ground normal vector and the second driving direction of the camera.

[0009] In some embodiments, the 3D reconstruction technique is a structure from motion (SFM) method.

[0010] In some embodiments, in order to determine the relative coordinate pose, the at least one processor is further used to: align a first ground normal vector of the first coordinate system with a second ground normal vector of the second coordinate system; align a first driving direction of the inertial test unit with a second driving direction of the camera; and determine the relative coordinate pose between the first coordinate system and the second coordinate system.

[0011] According to another aspect of the present application, a method for calibrating an inertial test unit (ITU) and a camera of an autonomous vehicle may include obtaining a trajectory of the autonomous vehicle traveling in a straight line; determining an ITU pose of the ITU relative to a first coordinate system; determining a camera pose relative to a second coordinate system; determining a relative coordinate pose between the first coordinate system and the second coordinate system; and determining a relative pose between the camera and the ITU based on the ITU pose, the camera pose, and the relative coordinate pose.

[0012] According to another aspect of the present application, a non-transitory computer-readable medium includes at least one set of instructions for calibrating an inertial test unit and a camera of an autonomous vehicle. When executed by at least one processor of an electronic device, the at least one set of instructions instructs the at least one processor to perform a method. The method may include obtaining a trajectory of the autonomous vehicle traveling in a straight line; determining an inertial test unit pose of the inertial test unit relative to a first coordinate system; determining a camera pose relative to a second coordinate system; determining a relative coordinate pose between the first coordinate system and the second coordinate system; and determining a relative pose between the camera and the inertial test unit based on the inertial test unit pose, the camera pose, and the relative coordinate pose.

[0013] According to another aspect of the present application, a system for calibrating an inertial test unit and a camera of an autonomous vehicle may include a trajectory acquisition module, configured to acquire a trajectory of the autonomous vehicle traveling in a straight line; an inertial test unit posture determination module, configured to determine the inertial test unit posture of the inertial test unit relative to a first coordinate system; a camera posture determination module, configured to determine the camera posture of the camera relative to a second coordinate system; a relative coordinate posture determination module, configured to determine the relative coordinate posture between the first coordinate system and the second coordinate system; and a relative posture determination module, configured to determine the relative posture between the camera and the inertial test unit based on the inertial test unit posture, the camera posture and the relative coordinate posture.

[0014] Some additional features of the present application may be explained in the following description. Some additional features of the present application will be apparent to those skilled in the art through study of the following description and accompanying drawings, or through understanding the production or operation of the embodiments. The features of the present application may be realized and achieved through practice or use of the methods, means, and combinations of various aspects of the specific embodiments described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The method of the present application will be further described by way of exemplary embodiments. These exemplary embodiments will be described in detail with reference to the accompanying drawings. The drawings are not drawn to scale. These embodiments are not limiting, and in these embodiments, like numbers in the various figures represent similar structures, wherein:

[0016] Figure 1 is a schematic diagram of an exemplary autonomous driving system according to some embodiments of the present application;

[0017] Figure 2 is a schematic diagram of exemplary hardware and / or software components of an exemplary computing device according to some embodiments of the present application;

[0018] Figure 3 is a schematic diagram of exemplary hardware and / or software components of an exemplary mobile device according to some embodiments of the present application;

[0019] Figure 4 is a block diagram of an exemplary processing device according to some embodiments of the present application;

[0020] Figure 5 is a flowchart of an exemplary process for calibrating an inertial test unit and a camera of an autonomous vehicle according to some embodiments of the present application;

[0021] Figure 6 is a schematic diagram of a relative posture between a camera and an inertial test unit according to some embodiments of the present application;

[0022] Figure 7 is a flowchart of an exemplary process for determining an inertial test unit posture relative to a first coordinate system according to some embodiments of the present application;

[0023] Figure 8 is a flowchart of an exemplary process for determining a camera pose relative to a second coordinate system according to some embodiments of the present application;

[0024] Figure 9 is a flowchart of an exemplary process of determining a second coordinate system according to some embodiments of the present application; and

[0025] Figure 10 is a flowchart of an exemplary process for determining a relative coordinate posture between a first coordinate system and a second coordinate system according to some embodiments of the present application. DETAILED DESCRIPTION

[0026] The following description is intended to enable one of ordinary skill in the art to implement and utilize the present application, and is provided in the context of a specific application scenario and its requirements. It will be apparent to one of ordinary skill in the art that various modifications may be made to the disclosed embodiments, and that the general principles defined herein may be applied to other embodiments and application scenarios without departing from the principles and scope of the present application. Therefore, the present application is not limited to the described embodiments, but should be accorded the broadest scope consistent with the claims.

[0027] The terms used in this application are only used to describe specific exemplary embodiments and do not limit the scope of this application. For example, the singular forms "a", "an", and "the" used in this application may also include the plural forms, unless the context clearly indicates an exception. It should also be understood that, as used in this application, the terms "comprise" and "include" only indicate the presence of the features, wholes, steps, operations, components and / or parts, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, components, parts and / or their combinations.

[0028] These and other features and characteristics of the present application, as well as the functions and methods of operation of the related structural elements, as well as the assembly of components and manufacturing economies, will become more apparent from the following description of the accompanying drawings, which form a part of this specification. However, it should be understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of the present application. It should be understood that the drawings are not drawn to scale.

[0029] Flowcharts are used in this application to illustrate operations performed by systems according to some embodiments of the present application. It should be understood that the operations in the flowcharts may not be performed in order. Instead, the various steps may be performed in reverse order or simultaneously. Furthermore, one or more additional operations may be added to these flowcharts. One or more operations may also be deleted from the flowcharts.

[0030] In addition, although the systems and methods disclosed in this application are mainly related to calibrating inertial test units and cameras in autonomous driving systems, it should be understood that this is only an exemplary embodiment. The systems and methods of the present application can be applied to any other type of transportation system. For example, the systems and methods of the present application can be applied to transportation systems in different environments, including land, sea, aerospace, etc., or any combination thereof. The autonomous driving vehicles of the transportation system may include taxis, private cars, ride-sharing, buses, trains, bullet trains, high-speed railways, subways, ships, airplanes, spacecraft, hot air balloons, etc., or any combination thereof.

[0031] One aspect of the present application relates to a system and method for calibrating an inertial test unit and a camera of an autonomous vehicle. When the autonomous vehicle is traveling in a straight line, the system and method can define two coordinate systems. One coordinate system is used to determine the posture of the inertial test unit, and the other coordinate system is used to determine the posture of the camera. Although the posture of the inertial test unit and the posture of the camera are in two different coordinate systems, the system and method can determine the relative posture of the two coordinate systems. In this way, the system and method can determine the relative posture between the inertial test unit and the camera to calibrate them. According to the system and method described in the present application, the inertial test unit and the camera can be calibrated in a simple and direct manner.

[0032] Figure 1 FIG1 is a schematic diagram of an exemplary autonomous driving system 100 according to some embodiments of the present application. In some embodiments, the autonomous driving system 100 may include a vehicle 110 (e.g., vehicles 110 - 1 , 110 - 2 , ..., and / or 110 - n ), a server 120 , a terminal device 130 , a storage device 140 , a network 150 , and a positioning and navigation system 160 .

[0033] The vehicle 110 may be any type of autonomous vehicle, unmanned aerial vehicle, or the like. An autonomous vehicle or unmanned aerial vehicle may refer to a vehicle that is capable of achieving a certain degree of driving automation. Exemplary driving automation levels may include a first level where the vehicle is primarily supervised by a human and has specific autonomous functions (e.g., autonomous steering or acceleration), a second level where the vehicle has one or more advanced driver assistance systems (ADAS) (e.g., adaptive cruise control, lane keeping system) that can control the vehicle's braking, steering, and / or acceleration, a third level where the vehicle is capable of autonomous driving when one or more specific conditions are met, a fourth level where the vehicle can operate without human input or oversight but is still subject to certain limitations (e.g., confined to a certain area), a fifth level where the vehicle can operate autonomously in all situations, or any combination thereof.

[0034] In some embodiments, the vehicle 110 may have an equivalent structure that enables the vehicle 110 to move or fly. For example, the vehicle 110 may include the structure of a traditional vehicle, such as a chassis, a suspension, a steering device (e.g., a steering wheel), a braking device (e.g., a brake pedal), an accelerator, etc. For another example, the vehicle 110 may have a body and at least one wheel. The body may be any body type, such as a sports car, a coupe, a sedan, a pickup truck, a station wagon, a sports utility vehicle (SUV), a minivan, or a convertible. At least one wheel may be configured as all-wheel drive (AWD), front-wheel drive (FWR), rear-wheel drive (RWD), etc. In some embodiments, the intended vehicle 110 may be an electric vehicle, a fuel cell vehicle, a hybrid vehicle, a traditional internal combustion engine vehicle, etc.

[0035] In some embodiments, vehicle 110 is capable of sensing its environment and navigating using one or more detection units 112. At least two of the detection units 112 may include a global positioning system (GPS) module, a radar (e.g., a laser radar (LiDAR)), an inertial measurement unit (IMU), a camera, or any combination thereof. The radar (e.g., LiDAR) may be configured to scan the surrounding environment and generate point cloud data. This point cloud data can then be used to create a digital three-dimensional representation of one or more objects surrounding vehicle 110. A GPS module may refer to a device capable of receiving geolocation and time information from GPS satellites and then calculating the device's geographic location. An inertial measurement unit sensor may refer to an electronic device that uses various inertial sensors to measure and provide information about the vehicle's specific forces, angular rates, and sometimes the magnetic field surrounding the vehicle. These various inertial sensors may include accelerometers (e.g., piezoelectric sensors), velocity sensors (e.g., Hall effect sensors), distance sensors (e.g., radar, laser radar, infrared sensors), steering angle sensors (e.g., tilt sensors), traction-related sensors (e.g., force sensors), and the like. The camera may be configured to capture one or more images associated with an object (eg, a person, animal, tree, roadblock, building, or vehicle) within range of the camera.

[0036] In some embodiments, the server 120 may be a single server or a server group. The server group may be centralized or distributed (for example, the server 120 may be a distributed system). In some embodiments, the server 120 may be local or remote. For example, the server 120 may access information and / or data stored in the terminal device 130, the detection unit 112, the vehicle 110, the storage device 140 and / or the positioning navigation system 160 via the network 150. For another example, the server 120 may be directly connected to the terminal device 130, the detection unit 112, the vehicle 110 and / or the storage device 140 to access the stored information and / or data. In some embodiments, the server 120 may be implemented on a cloud platform or on an on-board computer. By way of example only, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-layer cloud, or the like, or any combination thereof. In some embodiments, the server 120 may be in the present application. Figure 2 The described computing device 200 may include one or more components.

[0037] In some embodiments, the server 120 may include a processing device 122. The processing device 122 may process information and / or data associated with autonomous driving to perform one or more functions described in this application. For example, the processing device 122 may calibrate an inertial test unit and a camera. In some embodiments, the processing device 122 may include one or more processing engines (e.g., a single-chip processing engine or a multi-chip processing engine). By way of example only, the processing device 122 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a graphics processing unit (GPU), a physical processing unit (PPU), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a controller, a microcontroller unit, a reduced instruction set computer (RISC), a microprocessor, etc., or any combination thereof. In an embodiment, the processing device 122 may be integrated into the vehicle 110 or the terminal device 130.

[0038] In some embodiments, the terminal device 130 may include a mobile device 130-1, a tablet computer 130-2, a laptop computer 130-3, a built-in device in a vehicle 130-4, a wearable device 130-5, etc., or any combination thereof. In some embodiments, the mobile device 130-1 may include a smart home device, a wearable device, a smart mobile device, a virtual reality device, an augmented reality device, etc., or any combination thereof. In some embodiments, the smart home device may include smart lighting devices, smart appliance control devices, smart monitoring devices, smart TVs, smart cameras, intercoms, etc., or any combination thereof. In some embodiments, the wearable device may include a smart bracelet, smart shoes and socks, smart glasses, smart helmets, smart watches, smart clothing, smart backpacks, smart accessories, etc., or any combination thereof. In some embodiments, the smart mobile device may include a smart phone, a personal digital assistant (PDA), a gaming device, a navigation device, a sales terminal (POS), etc., or any combination thereof. In some embodiments, the virtual reality device and / or the augmented virtual reality device may include a virtual reality helmet, virtual reality glasses, virtual reality goggles, augmented reality helmets, augmented reality glasses, augmented reality goggles, etc., or any combination thereof. For example, the virtual reality device and / or the augmented reality device may include Google TM Glass, Oculus Rift, HoloLens, GearVR, etc. In some embodiments, the in-vehicle device 130-4 may include an in-vehicle computer, an in-vehicle television, etc. In some embodiments, the server 120 may be integrated into the terminal device 130. In some embodiments, the terminal device 130 may be a device with positioning technology for locating the position of the terminal device 130.

[0039] Storage device 140 can store data and / or instructions. In some embodiments, storage device 140 can store data acquired from vehicle 110, detection unit 112, processing device 122, terminal device 130, positioning and navigation system 160, and / or external storage devices. For example, storage device 140 can store inertial test unit data acquired from an inertial test unit in detection unit 112. For another example, storage device 140 can store camera data acquired from a camera in detection unit 112. In some embodiments, storage device 140 can store data and / or instructions that server 120 can execute or use to perform the exemplary methods described herein. For example, storage device 140 can store instructions that processing device 122 can execute or use to calibrate the inertial test unit and camera. In some embodiments, storage device 140 can include mass storage, removable storage, volatile read-write memory, read-only memory (ROM), or any combination thereof. Exemplary mass storage can include magnetic disks, optical disks, solid-state disks, and the like. Exemplary removable storage can include flash drives, floppy disks, optical disks, memory cards, compact disks, magnetic tapes, and the like. Exemplary volatile read-write memory may include random access memory (RAM). Exemplary random access memory may include dynamic random access memory (DRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), static random access memory (SRAM), thyristor random access memory (T-RAM), and zero capacitance random access memory (Z-RAM). Exemplary read-only memory may include mask read-only memory (MROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), and digital versatile disk read-only memory. In some embodiments, the storage device 140 may be executed on a cloud platform. By way of example only, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-layer cloud, or any combination thereof.

[0040] In some embodiments, the storage device 140 can be connected to the network 150 to communicate with one or more components of the autonomous driving system 100 (e.g., the server 120, the terminal device 130, the detection unit 112, the vehicle 110, and / or the positioning and navigation system 160). One or more components of the autonomous driving system 100 can access the data or instructions stored in the storage device 140 via the network 150. In some embodiments, the storage device 140 can be directly connected to or communicate with one or more components of the autonomous driving system 100 (e.g., the server 120, the terminal device 130, the detection unit 112, the vehicle 110, and / or the positioning and navigation system 160). In some embodiments, the storage device 140 can be part of the server 120. In some embodiments, the storage device 140 can be integrated into the vehicle 110.

[0041] Network 150 can facilitate the exchange of information and / or data. In some embodiments, one or more components of autonomous driving system 100 (e.g., server 120, terminal device 130, detection unit 112, vehicle 110, storage device 140, or positioning and navigation system 160) can transmit information and / or data to other components of autonomous driving system 100 via network 150. For example, server 120 can obtain inertial test unit data or camera data from vehicle 110, terminal device 130, storage device 140, and / or positioning and navigation system 160 via network 150. In some embodiments, network 150 can be any form of wired or wireless network, or any combination thereof. By way of example only, network 150 can include a cable network, a wired network, a fiber optic network, a telecommunications network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, a near-field communication (NFC) network, the like, or any combination thereof. In some embodiments, network 150 can include one or more network access points. For example, the network 150 may include wired or wireless network access points (eg, 150 - 1 , 150 - 2 ) through which one or more components of the automated driving system 100 may connect to the network 150 to exchange data and / or information.

[0042] The positioning and navigation system 160 can determine information associated with an object, such as a terminal device 130, a vehicle 110, or the like. In some embodiments, the positioning and navigation system 160 may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a compass navigation system (COMPASS), a BeiDou navigation satellite system, a Galileo positioning system, a quasi-zenith satellite system (QZSS), or the like. The information may include the location, altitude, speed or acceleration, current time, or the like of the object. The positioning and navigation system 160 may include one or more satellites, such as satellite 160-1, satellite 160-2, and satellite 160-3. The satellites 160-1 to 160-3 may determine the above information independently or collectively. The satellite positioning and navigation system 160 may send the above information to the network 150, the terminal device 130, or the vehicle 110 via a wireless connection.

[0043] Those skilled in the art will appreciate that when an element (or component) of the autonomous driving system 100 executes, the element may execute via electrical signals and / or electromagnetic signals. For example, when the terminal device 130 sends a request to the server 120, the processor of the terminal device 130 may generate an electrical signal encoding the request. The processor of the terminal device 130 may then send the electrical signal to an output port. If the terminal device 130 communicates with the server 120 via a wired network, the output port may be physically connected to a cable, which further transmits the electrical signal to an input port of the server 120. If the terminal device 130 communicates with the server 120 via a wireless network, the output port of the terminal device 130 may be one or more antennas that convert the electrical signal into an electromagnetic signal. In electronic devices such as the terminal device 130 and / or the server 120, when their processors process instructions, issue instructions, and / or perform actions, the instructions and / or actions are carried out via electrical signals. For example, when the processor retrieves or saves data from a storage medium (e.g., storage device 140), it may send an electrical signal to a read / write device of the storage medium, which may read or write structured data in the storage medium. The structured data may be transmitted to the processor in the form of an electrical signal via a bus of the electronic device. Here, the electrical signal may refer to one electrical signal, a series of electrical signals, and / or multiple discrete electrical signals.

[0044] Figure 2 2 is a schematic diagram of exemplary hardware and / or software components of an exemplary computing device according to some embodiments of the present application. In some embodiments, server 120 and / or terminal device 130 may be implemented on computing device 200. For example, processing device 122 may be implemented on computing device 200 and configured to perform the functions of processing device 122 disclosed in this application.

[0045] The computing device 200 may be used to implement any component of the automated driving system 100 of the present application. For example, the processing device 122 of the automated driving system 100 may be executed on the computing device 200 via its hardware, software program, firmware, or a combination thereof. Although only one such computer is shown for convenience, the computer functions described herein in connection with the automated driving system 100 may be implemented in a distributed manner across multiple similar platforms to distribute the processing load.

[0046] The computing device 200 may include a communication (COM) port 250 connected to a network (e.g., network 150) connected thereto to facilitate data communication. The computing device 200 may also include a processor (e.g., processor 220) for executing program instructions in the form of one or more processors (e.g., logic circuits). For example, the processor may include an interface circuit and a processing circuit therein. The interface circuit may be configured to receive electrical signals from the bus 210, wherein the electrical signal encoding is used for processing structured data and / or instructions of the circuit. The processing circuit may perform logical calculations and then determine the conclusion, result, and / or instruction encoding as electrical signals. The interface circuit may then send electrical signals from the processing circuit via the bus 210.

[0047] The computing device 200 may also include various forms of program memory and data memory, including, for example, a disk 270, a read-only memory (ROM) 230, or a random access memory (RAM) 240, for storing various data files processed and / or transmitted by the computing device 200. The exemplary computing device 200 may also include program instructions stored in the read-only memory 230, the random access memory 240, and / or other types of non-transitory storage media executed by the processor 220. The methods and / or processes of the present application may be implemented in the form of program instructions. The computing device 200 also includes an input / output component 260 that supports input / output between the computing device 200 and other components therein. The computing device 200 may also receive programming and data via network communications.

[0048] For illustrative purposes only, only one processor is described in computing device 200. However, it should be noted that computing device 200 in this application may also include multiple processors, and thus, operations performed by one processor described in this application may also be performed jointly or individually by multiple processors. For example, a processor of computing device 200 performs operations A and B. For another example, operations A and B may also be performed jointly or individually by two different processors in computing device 200 (e.g., a first processor performs operation A, a second processor performs operation B, or the first and second processors jointly perform operations A and B).

[0049] Figure 31 is a schematic diagram of exemplary hardware and / or software components of an exemplary mobile device according to some embodiments of the present application. In some embodiments, the terminal device 130 can be applied on the mobile device 300. Figure 3 As shown, mobile device 300 may include a communication platform 310, a display 320, a graphics processing unit (GPU) 330, a central processing unit (CPU) 340, input / output (I / O) 350, memory 360, a mobile operating system (OS) 370, and storage 390. In some embodiments, any other suitable components, including but not limited to a system bus or controller (not shown), may also be included in mobile device 300.

[0050] In some embodiments, the mobile operating system 370 (e.g., iOS TM 、Android TM 、Windows Phone TM ) and one or more applications 380 can be loaded from storage 390 into memory 360 for execution by CPU 340. Applications 380 may include a browser or any other suitable mobile application for receiving and presenting information related to positioning or other information from processing device 122. User interaction with the information stream can be achieved through input / output 350 and provided to processing device 122 and / or other components of autonomous driving system 100 via network 150.

[0051] To implement the various modules, units, and their functions described herein, a computer hardware platform may be used as the hardware platform for one or more of the components described herein. A computer with a user interface component may be used to implement a personal computer (PC) or any other type of workstation or terminal device. If the computer is appropriately programmed, the computer may also be used as a server.

[0052] Figure 4 FIG1 is a block diagram of an exemplary processing device 122 according to some embodiments of the present application. The processing device 122 may include a trajectory acquisition module 410, an inertial measurement unit pose determination module 420, a camera pose determination module 430, a relative coordinate pose determination module 440, and a relative pose determination module 450.

[0053] The trajectory acquisition module 410 may be configured to acquire a straight-line driving trajectory of the autonomous driving vehicle.

[0054] Inertial test unit posture determination module 420 can be configured to determine an inertial test unit posture of the inertial test unit relative to the first coordinate system. For example, inertial test unit posture determination module 420 can obtain inertial test unit data from the inertial test unit and determine the first coordinate system. For another example, inertial test unit posture determination module 420 can determine the inertial test unit posture based on the inertial test unit data and the first coordinate system.

[0055] The camera pose determination module 430 may be configured to determine the camera pose of the camera relative to the second coordinate system. For example, the camera pose determination module 430 may obtain camera data from the camera and determine the second coordinate system based on the camera data. For another example, the camera pose determination module 430 may determine the camera pose based on the camera data and the second coordinate system.

[0056] Relative coordinate pose determination module 440 can be configured to determine a relative coordinate pose between the first coordinate system and the second coordinate system. For example, relative coordinate pose determination module 440 can align a first ground normal vector of the first coordinate system with a second ground normal vector of the second coordinate system, and align a first driving direction of the inertial test unit with a second driving direction of the camera. Relative coordinate pose determination module 440 can also determine a relative coordinate pose between the first coordinate system and the second coordinate system.

[0057] Relative pose determination module 450 may be configured to determine a relative pose between the camera and the inertial test unit based on the inertial test unit pose, the camera pose, and the relative coordinate pose.

[0058] The modules in the processing device 122 can be connected or communicate with each other via a wired connection or a wireless connection. The wired connection may include a metal cable, an optical cable, a hybrid cable, or the like, or any combination thereof. The wireless connection may include a local area network (LAN), a wide area network (WAN), Bluetooth, a ZigBee network, a near field communication (NFC), or the like, or any combination thereof. Two or more modules can be combined into a single module, and any module can be split into two or more units. For example, the processing device 122 may include a storage module (not shown) for storing information and / or data related to the inertial test unit and the camera (e.g., inertial test unit data, camera data, etc.).

[0059] Figure 5 FIG. 5 is a flow chart of an exemplary process 500 for calibrating an inertial test unit and a camera of an autonomous vehicle according to some embodiments of the present application. In some embodiments, the process 500 may be implemented as a set of instructions (e.g., an application) stored in the read-only memory 230 or the random access memory 240. The processor 220 and / or Figure 4The modules in the process 500 may execute a set of instructions, and when executing the instructions, the processor 220 and / or the modules may be configured to perform the process 500. The operations of the process shown below are for illustration purposes only. In some embodiments, the process 500 may be accomplished using one or more additional operations not described and / or one or more operations not discussed herein. In addition, as Figure 5 The order in which the operations of the processes shown and described below are not limiting.

[0060] In 510 , the processing device 122 (e.g., the trajectory acquisition module 410 , the interface circuit of the processor 220 ) may acquire the trajectory of the autonomous driving vehicle traveling in a straight line.

[0061] In some embodiments, an inertial test unit and a camera can be installed on an autonomous vehicle to sense the environment around the autonomous vehicle and navigate the autonomous vehicle. In some embodiments, the autonomous vehicle can be controlled (by a driver or processing device 122) to travel a predetermined distance in a straight line. In some embodiments, the predetermined distance can be a default value stored in the system 100 (storage device such as storage device 140, read-only memory 230, random access memory 240, etc.), or determined by the system 100 or its operator according to different application scenarios. For example, the predetermined distance can be 50 meters, 100 meters, 200 meters, 1000 meters, etc. The processing device 122 can obtain the trajectory of the autonomous vehicle when the autonomous vehicle is traveling in a straight line.

[0062] At 520 , processing device 122 (eg, inertial test unit pose determination module 420 ) may determine an inertial test unit pose of the inertial test unit relative to the first coordinate system.

[0063] In some embodiments, the inertial test unit posture relative to the first coordinate system can reflect the orientation, position, posture or rotation of the inertial test unit relative to the first coordinate system. In some embodiments, the inertial test unit posture can be represented as Euler angles, rotation matrices, orientation quaternions, etc., or any combination thereof. For example, the inertial test unit posture can be represented as follows: Figure 6 The rotation matrix shown Wherein R may represent a matrix, I may represent an inertial test unit, and s1 may represent a first coordinate system.

[0064] Figure 6 FIG is a schematic diagram of an exemplary relative posture between a camera and an inertial test unit according to some embodiments of the present application. Figure 6 As shown, C can represent the origin of the camera, and X I 、Y I and Z ICan represent the three axes of the camera respectively. I can represent the origin of the inertial test unit, X I 、Y I and Z I X can represent the three axes of the inertial test unit. O1 and O2 can represent the origins of the first coordinate system S1 and the second coordinate system S2 respectively. I 、Y I and Z I X2, Y2, and Z2 may represent the three axes of the first coordinate system S1, respectively. X2, Y2, and Z2 may represent the three axes of the second coordinate system S2, respectively. The relative pose of the camera relative to the inertial measurement unit (IMU) can be represented. The relative pose of the camera relative to the second coordinate system S2 can be represented. The relative pose of the IMU relative to the first coordinate system S1 can be represented. The relative pose of the second coordinate system S2 relative to the first coordinate system S1 can be represented.

[0065] In some embodiments, the first coordinate system may be a defined three-dimensional coordinate system. For example, when the autonomous vehicle is traveling in a straight line, processing device 122 may determine a first ground normal vector and a first driving direction of the autonomous vehicle. Processing device 122 may determine the first coordinate system using the first ground normal vector and the first driving direction as two axes of the first coordinate system according to the right-hand rule.

[0066] In some embodiments, when the autonomous vehicle is traveling in a straight line, the inertial test unit can use various inertial sensors to detect and output acceleration, rotation rate, and sometimes the magnetic field around the inertial test unit. For example, the various inertial sensors may include one or more accelerometers, one or more gyroscopes, one or more magnetometers, etc., or any combination thereof. The processing device 122 can use the acceleration, rotation rate and / or magnetic field to calculate the inertial test unit attitude. The process or method for determining the first coordinate system and / or the inertial test unit attitude can be found elsewhere in this application (e.g., Figure 7 and its description).

[0067] At 530 , the processing device 122 (eg, the camera pose determination module 430 ) may determine a camera pose of the camera relative to the second coordinate system.

[0068] In some embodiments, the camera pose relative to the second coordinate system may reflect the orientation, position, attitude, or rotation of the camera relative to the second coordinate system. In some embodiments, the camera pose may be represented as Euler angles, a rotation matrix, an orientation quaternion, or any combination thereof. For example, the camera pose may be represented as Figure 6 The rotation matrix shown Where R may represent a matrix, R may represent a camera, and s2 may represent a second coordinate system.

[0069] In some embodiments, the second coordinate system may be a defined three-dimensional coordinate system associated with the camera. For example, when the autonomous vehicle is traveling in a straight line, the camera may capture a video or image within the camera's field of view. The processing device 122 may establish the second coordinate system based on the video or image captured by the camera. For example, the processing device 122 may obtain at least two images from the video or image and process the at least two images using a three-dimensional reconstruction technique. The processing device 122 may obtain a second ground normal vector in the three-dimensional scene. The processing device 122 may determine the second coordinate system using the second ground normal vector of the camera and the second driving direction as the two axes of the second coordinate system according to the right-hand rule.

[0070] In some embodiments, the processing device 122 can determine the camera pose based on a 3D reconstruction technique. For example, the processing device 122 can input at least two images and / or internal parameters of the camera into the 3D reconstruction technique. The 3D reconstruction technique can output the camera pose relative to the second coordinate system and the 3D structure data of the scene captured by the camera. The process or method for determining the second coordinate system and / or the camera pose can be found elsewhere in this application (e.g., Figure 8-9 and its description).

[0071] At 540 , the processing device 122 (eg, the relative coordinate pose determination module 440 ) may determine a relative coordinate pose between the first coordinate system and the second coordinate system.

[0072] In some embodiments, the relative coordinate pose between the first coordinate system and the second coordinate system can reflect the orientation, position, attitude or rotation of the first coordinate system relative to the second coordinate system. In some embodiments, the relative coordinate pose can be expressed as Euler angles, rotation matrices, orientation quaternions, etc., or any combination thereof. For example, the relative coordinate pose can be expressed as a rotation matrix like Figure 6 As shown, R can represent a matrix, s1 can represent a first coordinate system, and s2 can represent a second coordinate system.

[0073] The first coordinate system and the second coordinate system are both defined coordinate systems and are essentially two different representations of the same coordinate system. In some embodiments, the processing device 122 can determine the relative coordinate pose by rotating and aligning the axes of the first coordinate system and the second coordinate system. For example, the processing device 122 can align the first ground normal vector of the first coordinate system with the second ground normal vector of the second coordinate system, and align the second driving direction of the inertial test unit and the second driving direction of the camera around the ground normal vector to determine the relative coordinate pose. In some embodiments, the processing device 122 can determine the relative coordinate pose based on the same reference coordinate system. For example, the processing device 122 can determine the first relative pose of the first coordinate system relative to the world coordinate system, respectively and the second relative posture of the second coordinate system relative to the world coordinate system The processing device 122 can be configured to Multiply by the second relative attitude Determine the relative coordinate pose of the first coordinate system relative to the second coordinate system The process or method for determining relative coordinate pose can be found elsewhere in this application (e.g., Figure 10 and its description).

[0074] At 550 , processing device 122 (eg, relative pose determination module 450 ) may determine a relative pose between the camera and the inertial test unit based on the inertial test unit pose, the camera pose, and the relative coordinate pose.

[0075] In some embodiments, the relative pose between the camera and the inertial test unit can reflect the direction, position, attitude or rotation of the camera relative to the inertial test unit. In some embodiments, the relative pose can be expressed as Euler angles, rotation matrices, orientation quaternions, etc., or any combination thereof. For example, the relative pose can be expressed as Euler angles α, β, and γ. α, β, and γ can represent rotation angles around the X-axis, Y-axis, and Z-axis, respectively. For another example, the relative pose can be expressed as Figure 6 The rotation matrix shown Where R can represent the matrix, C can represent the camera, and I can represent the inertial measurement unit. Rotation matrix It can be around three axes R x 、R Y , and R Z The product of the three rotation matrices of . Among them, and

[0076]

[0077] In some embodiments, the processing device 122 may measure the attitude of the inertial test unit based on the Camera pose and relative coordinate pose Determine the relative pose of the camera with respect to the inertial test unit For example, the processing device 122 can determine the relative pose according to the following equation (1):

[0078]

[0079] Where, is the transposed matrix of the relative coordinate attitude, and is the transposed matrix of the inertial test unit attitude.

[0080] In some embodiments, the relative pose between the camera and the inertial test unit can be used to navigate the autonomous vehicle. For example, while the autonomous vehicle is driving, the processing device 122 can calculate the position of a three-dimensional target captured by the autonomous vehicle's lidar as seen by the camera. With the help of the inertial test unit, the processing device 122 can first transform the three-dimensional target captured by the lidar into the inertial test unit coordinate system. Then, using the relative pose between the camera and the inertial test unit, the processing device 122 can transform the three-dimensional target back to the camera coordinate system.

[0081] It should be noted that the above description is provided for illustrative purposes only and is not intended to limit the scope of the present application. It will be apparent to those skilled in the art that various changes and modifications may be made based on the description of the present application. However, these changes and modifications will not deviate from the scope of the present application. For example, one or more other optional operations (e.g., storage operations) may be added elsewhere in process 500. In a storage operation, the processing device 122 may store information and / or data (e.g., the relative posture between the camera and the inertial test unit) in a storage device (e.g., storage device 140) disclosed elsewhere in the present application.

[0082] Figure 7 FIG. 7 is a flow chart of an exemplary process 700 for determining an inertial test unit posture relative to a first coordinate system according to some embodiments of the present application. In some embodiments, process 700 may be implemented by a set of instructions (e.g., an application) stored in read-only memory 230 or random access memory 240. Processor 220 and / or Figure 4 The modules in the process 700 may execute a set of instructions, and when executing the instructions, the processor 220 and / or the modules may be configured to perform the process 700. The operations of the process shown below are for illustration purposes only. In some embodiments, the process 700 may be completed with one or more additional operations not described and / or one or more operations not discussed herein. In addition, Figure 7 The order in which the operations of the processes shown in and described below are not limiting.

[0083] At 710 , processing device 122 (eg, inertial test unit attitude determination module 420 , interface circuitry of processor 220 ) may obtain inertial test unit data from the inertial test unit.

[0084] In some embodiments, the inertial test unit may include at least two inertial sensors, such as one or more accelerometers, one or more gyroscopes, one or more magnetometers, or any combination thereof. The inertial test unit may output inertial test unit data using the at least two inertial sensors. For example, the inertial test unit data may include acceleration, rotation rate, the magnetic field around the autonomous vehicle, or any combination thereof. Processing device 122 may obtain the inertial test unit data from the inertial test unit while the autonomous vehicle is traveling.

[0085] At 720 , the processing device 122 (e.g., the inertial test unit attitude determination module 420 ) may determine a first coordinate system based on the trajectory of the autonomous vehicle.

[0086] In some embodiments, when the autonomous vehicle is traveling in a straight line, the processing device 122 may obtain a trajectory of the autonomous vehicle. The processing device 122 may determine a first ground normal vector and a first driving direction of the autonomous vehicle from the trajectory of the autonomous vehicle. Figure 6 As shown, the processing device 122 can use the first ground normal vector and the first driving direction as two axes of the first coordinate system S1 (for example, the first ground normal vector is X1 and the first driving direction is Y1), and determine the third axis (for example, Z1) of the first coordinate system S1 according to the right-hand rule.

[0087] At 730 , processing device 122 (eg, inertial test unit pose determination module 420 ) may determine the inertial test unit pose based on the inertial test unit data and the first coordinate system.

[0088] In some embodiments, processing device 122 may calculate an inertial test unit pose relative to the first coordinate system based on acceleration, rotation rate, and / or a magnetic field surrounding the autonomous vehicle. For example, processing device 122 may fuse the acceleration, rotation rate, and / or magnetic field according to a fusion algorithm to determine the inertial test unit pose. Exemplary fusion algorithms may include a complementary filtering method, a conjugate gradient filtering method, an extended Kalman filtering method, an unscented Kalman filtering method, or the like, or any combination thereof.

[0089] It should be noted that the above is provided for illustrative purposes only and is not intended to limit the scope of this application. For those skilled in the art, various changes and modifications can be made based on the description of this application. However, these changes and modifications do not deviate from the scope of this application. For example, one or more other optional operations (e.g., storage operations) can be added elsewhere in process 700. In the storage operation, the processing device 122 can store information and / or data (e.g., inertial test unit data) related to the inertial test unit in the storage device (e.g., storage device 140) disclosed elsewhere in this application.

[0090] Figure 8 FIG8 is a flowchart of an exemplary process 700 for determining a camera pose relative to a second coordinate system according to some embodiments of the present application. In some embodiments, the process 800 may be implemented by a set of instructions (e.g., an application) stored in the read-only memory 230 or the random access memory 240. The processor 220 and / or Figure 4 The modules in the process 800 may execute a set of instructions, and when executing the instructions, the processor 220 and / or the modules may be configured to perform the process 800. The operations of the process shown below are for illustration purposes only. In some embodiments, the process 800 may be completed with one or more additional operations not described and / or one or more operations not discussed herein. In addition, Figure 8 The order in which the operations of the processes are shown and described below is not limiting.

[0091] At 810 , the processing device 122 (eg, the camera pose determination module 430 , an interface circuit of the processor 220 ) may obtain camera data from a camera.

[0092] In some embodiments, when the autonomous vehicle is traveling in a straight line, the camera can acquire camera data (e.g., video or images) within the range of the autonomous vehicle. The processing device 122 can acquire the camera data from the camera.

[0093] At 820 , the processing device 122 (eg, the camera pose determination module 430 ) may determine a second coordinate system based on the camera data.

[0094] In some embodiments, the processing device 122 may input the camera data into a 3D reconstruction technology to obtain a 3D scene. Exemplary 3D reconstruction technologies may include a shape from texture (SFT) method, a shading reconstruction 3D shape method, a multi-view stereo (MVS) method, a structure from motion (SFM) method, a time of flight (ToF) method, a structured light method, a moiré method, etc., or any combination thereof. In the 3D scene, the processing device 122 may obtain a second ground normal vector and a second driving direction of the camera. Figure 6As shown, the processing device 122 can use the second ground normal vector and the second driving direction as two axes of the second coordinate system S2 (for example, the second ground normal vector is used as X2, and the second driving direction is used as Y2 of the second coordinate system S2 to determine the third axis (for example, Z2)) according to the right-hand rule. The process or method for determining the second coordinate system can be found elsewhere in this application (for example, Figure 10 and its description).

[0095] At 830 , the processing device 122 (eg, the camera pose determination module 430 ) may determine a camera pose based on the camera data and the second coordinate system.

[0096] In some embodiments, the processing device 122 may use three-dimensional reconstruction technology to determine the camera pose. For example, the processing device 122 may input camera data and / or internal parameters of the camera into a motion recovery method. The motion recovery method can use the video or image captured by the camera to automatically recover the movement of the camera and the three-dimensional structure of the scene captured by the camera. For example, in the motion recovery method, a set of two-dimensional feature points in the video or image can be tracked to obtain the trajectory of the feature points over time. Then, using the trajectory of the feature points over time, the position of the camera and / or the three-dimensional position of the feature points can be derived. Using the position of the camera and / or the three-dimensional position of the feature points, the rotation matrix between the camera and the second coordinate system can be determined. The processing device 122 can determine the camera pose of the camera relative to the second coordinate system based on the rotation matrix.

[0097] It should be noted that the above is provided for illustrative purposes only and is not intended to limit the scope of the present application. It will be apparent to those skilled in the art that various changes and modifications may be made based on the description of the present application. However, such changes and modifications will not depart from the scope of the present application. For example, one or more other optional operations (e.g., storage operations) may be added elsewhere in process 800. In the storage operation, processing device 122 may store information and / or data (e.g., camera data) related to the camera in a storage device (e.g., storage device 140) disclosed elsewhere in this application.

[0098] Figure 9 FIG. 9 is a flow chart of an exemplary process 900 for determining a second coordinate system according to some embodiments of the present application. In some embodiments, the process 900 may be implemented by a set of instructions (eg, an application) stored in the read-only memory 230 or the random access memory 240. The processor 220 and / or Figure 4The modules in the process 900 may execute a set of instructions, and when executing the instructions, the processor 220 and / or the modules may be configured to perform the process 900. The operations of the process shown below are for illustration purposes only. In some embodiments, the process 900 may be completed with one or more additional operations not described and / or one or more operations not discussed herein. In addition, Figure 9 The order in which the operations of the processes shown and described below are not limiting.

[0099] At 910 , the processing device 122 (eg, the camera pose determination module 430 ) may determine a second ground normal vector based on the camera data and three-dimensional reconstruction techniques.

[0100] In some embodiments, the processing device 122 may input the camera data into a 3D reconstruction technique (eg, a kinematic mechanism method) to obtain a 3D scene. The processing device 122 may obtain a ground normal vector in the 3D scene as a second ground normal vector.

[0101] At 920 , the processing device 122 (eg, the camera pose determination module 430 ) may determine a second direction of travel for the camera based on the camera data.

[0102] The processing device 122 may obtain the driving direction of the camera in the three-dimensional scene as the second driving direction.

[0103] At 930 , the processing device 122 (eg, the camera pose determination module 430 ) may determine a second coordinate system based on the second ground normal vector and the second driving direction of the camera.

[0104] In some embodiments, as Figure 6 As shown, the processing device 122 can use the second ground normal vector and the second driving direction as two axes of the second coordinate system S2 (for example, the second ground normal vector is X2 and the second driving direction is Y2) according to the right-hand rule to determine the third axis (for example, Z2) of the second coordinate system S2. The processing device 122 can use the second ground normal vector, the second driving direction, and the determined third axis as X2, Y2, and Z2, respectively, to determine the second coordinate system.

[0105] It should be noted that the above is provided for illustrative purposes only and is not intended to limit the scope of the present application. It will be apparent to those skilled in the art that various changes and modifications may be made based on the description of the present application. However, such changes and modifications will not depart from the scope of the present application. For example, one or more other optional operations (e.g., storage operations) may be added elsewhere in process 900. In the storage operation, processing device 122 may store information and / or data (e.g., camera data) related to the camera in a storage device (e.g., storage device 140) disclosed elsewhere in this application.

[0106] Figure 10 FIG1 is a flowchart of an exemplary process 900 for determining a relative coordinate pose between a first coordinate system and a second coordinate system according to some embodiments of the present application. In some embodiments, the process 1000 may be implemented by a set of instructions (e.g., an application) stored in the read-only memory 230 or the random access memory 240. The processor 220 and / or Figure 4 The modules in the process 1000 may execute a set of instructions, and when executing the instructions, the processor 220 and / or the modules may be configured to perform the process 1000. The operations of the process shown below are for illustration purposes only. In some embodiments, the process 1000 may be completed with one or more additional operations not described and / or one or more operations not discussed herein. In addition, Figure 10 The order in which the operations of the processes shown and described below are not limiting.

[0107] At 1010 , the processing device 122 (eg, the relative coordinate pose determination module 440 ) may align a first ground normal vector of a first coordinate system with a second ground normal vector of a second coordinate system.

[0108] In some embodiments, processing device 122 may translate and / or rotate the first coordinate system toward the second coordinate system. Processing device 122 may align the first ground normal vector with the second ground normal vector after the translation and / or rotation. In some embodiments, processing device 122 may record the translation and / or rotation in the form of Euler angles, a rotation matrix, an orientation quaternion, or the like, or any combination thereof.

[0109] At 1020 , processing device 122 (eg, relative coordinate pose determination module 440 ) may align the first direction of travel of the inertial test unit with the second direction of travel of the camera.

[0110] In some embodiments, after aligning the first ground normal vector with the second ground normal vector, processing device 122 may further rotate the first driving direction of the inertial measurement unit and the second driving direction of the camera to align around the aligned ground normal vector. In some embodiments, processing device 122 may record the rotation in the form of Euler angles, a rotation matrix, an orientation quaternion, or any combination thereof.

[0111] At 1030, the processing device 122 (e.g., the relative coordinate pose determination module 440) may determine Figure 6 The relative coordinate pose between the first coordinate system and the second coordinate system described in .

[0112] In some embodiments, the processing device 122 may determine the relative coordinate pose based on the translation and / or rotation. In some embodiments, the relative coordinate pose may be represented as Euler angles, a rotation matrix, an orientation quaternion, or any combination thereof.

[0113] It should be noted that the above is provided for illustrative purposes only and is not intended to limit the scope of the present application. For those skilled in the art, various changes and modifications can be made according to the description of the present application. However, these changes and modifications will not deviate from the scope of the present application. For example, one or more other optional operations (e.g., storage operations) can be added elsewhere in process 1000. For another example, the processing device 122 can determine the relative coordinate posture between the first coordinate system and the second coordinate system based on the same reference coordinate system (e.g., world coordinate system).

[0114] The basic concepts have been described above. It will be apparent to those skilled in the art after reading this application that the above disclosures are merely illustrative and do not constitute limitations on this application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and amendments to this application. Such modifications, improvements, and amendments are suggested in this application and remain within the spirit and scope of the exemplary embodiments of this application.

[0115] At the same time, this application uses specific terms to describe the embodiments of this application. For example, "one embodiment," "an embodiment," and / or "some embodiments" refer to a certain feature, structure, or characteristic related to at least one embodiment of this application. Therefore, it should be emphasized and noted that "one embodiment," "an embodiment," or "an alternative embodiment" mentioned twice or more in different places in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of this application may be appropriately combined.

[0116] In addition, it will be understood by those skilled in the art that various aspects of the present application can be illustrated and described by a number of patentable categories or situations, including any new and useful process, machine, product or combination of substances, or any new and useful improvements thereto. Accordingly, various aspects of the present application can be performed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software can all be referred to as "units", "modules" or "systems". In addition, various aspects of the present application can take the form of a computer program product embodied in one or more computer-readable media, wherein computer-readable program code is contained therein.

[0117] A computer-readable signal medium may include a propagated data signal embodying computer program code, for example, in baseband or as part of a carrier wave. Such propagated signals may take a variety of forms, including electromagnetic, optical, or any suitable combination. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium that can be connected to an instruction execution system, apparatus, or device to communicate, propagate, or transfer a program for use. Program code on a computer-readable signal medium may be propagated via any suitable medium, including radio, electrical cable, fiber optic cable, radio frequency, or any combination of the foregoing.

[0118] The computer program coding required for the operation of each part of the application can be written in any one or more programming languages, including subject-oriented programming languages such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc., conventional procedural programming languages such as C language, Visual Basic, Fortran 2003, Perl, COBOL 2002, PHP, ABAP, dynamic programming languages such as Python, Ruby and Groovy, or other programming languages. The program code can be run entirely on the user's computer, or run on the user's computer as an independent software package, or run partly on the user's computer and partly on a remote computer, or run entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any network form, such as a local area network (LAN) or a wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service such as software as a service (SaaS).

[0119] In addition, unless expressly stated in the claims, the order of the processing elements and sequences described in this application, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of this application. Although the above disclosure discusses some of the invention embodiments currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this application. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing server or mobile device.

[0120] Similarly, it should be noted that, in order to simplify the presentation of this disclosure and thereby facilitate understanding of one or more embodiments of the invention, the foregoing descriptions of the embodiments of this disclosure sometimes combine multiple features into a single embodiment, figure, or description thereof. However, this approach should not be interpreted as reflecting an intention that the claimed object material to be scanned requires more features than those expressly recited in each claim. In practice, an embodiment may have fewer features than the totality of the features of a single disclosed embodiment.

Claims

1. A system for calibrating an inertial test unit and camera of an autonomous vehicle, comprising: at least one storage medium comprising a set of instructions for calibrating the inertial test unit and the camera; as well as at least one processor in communication with the storage medium, wherein when executing the set of instructions, the at least one processor is configured to: Obtaining a straight-line trajectory of the autonomous driving vehicle; determining an inertial test unit pose of the inertial test unit relative to a first coordinate system, the inertial test unit pose reflecting an orientation, position, attitude, or rotation of the inertial test unit relative to the first coordinate system; acquiring camera data from the camera; Determining a second coordinate system based on the camera data specifically includes: Determining a second ground normal vector based on the camera data and 3D reconstruction technology; determining a second driving direction of the camera based on the camera data; and determining the second coordinate system based on the second ground normal vector and the second driving direction of the camera; Determining a camera pose of the camera relative to the second coordinate system using a three-dimensional reconstruction technique based on the camera data and the second coordinate system specifically includes: Tracking a set of two-dimensional feature points in a video or image based on the camera data and / or internal parameters of the camera to obtain a trajectory of the feature points over time; Determining a camera position and / or a three-dimensional position of the feature point based on the feature point trajectory over time; Determining a rotation matrix between the camera and a second coordinate system based on the position of the camera and / or the three-dimensional position of the feature point; Determining a camera pose of the camera relative to a second coordinate system based on the rotation matrix, wherein the camera pose reflects an orientation, position, attitude, or rotation of the camera relative to the second coordinate system; determining a relative coordinate pose between the first coordinate system and the second coordinate system, the first coordinate system being different from the second coordinate system; and A relative pose between the camera and the inertial test unit is determined based on the inertial test unit pose, the camera pose, and the relative coordinate pose.

2. The system according to claim 1, wherein: The at least one processor is further configured to: Based on the trajectory of the autonomous vehicle, the first coordinate system is determined.

3. The system according to claim 2, characterized in that To determine the inertial test unit attitude, the at least one processor is further configured to: acquiring inertial test unit data from the inertial test unit; and The inertial test unit pose is determined based on the inertial test unit data and the first coordinate system.

4. The system according to claim 1, wherein: The 3D reconstruction technology is a structure-from-motion (SFM) method.

5. The system according to any one of claims 1 or 4, characterized in that To determine the relative coordinate pose, the at least one processor is further configured to: aligning a first ground normal vector of the first coordinate system with the second ground normal vector of the second coordinate system; aligning a first direction of travel of the inertial test unit with the second direction of travel of the camera; as well as The relative coordinate pose between the first coordinate system and the second coordinate system is determined.

6. A method for calibrating an inertial test unit and a camera of an autonomous vehicle, the method being implemented on a computing device comprising at least one storage medium containing a set of instructions and at least one processor in communication with the storage medium, the method comprising: Obtaining a straight-line trajectory of the autonomous driving vehicle; determining an inertial test unit pose of the inertial test unit relative to a first coordinate system, the inertial test unit pose reflecting an orientation, position, attitude, or rotation of the inertial test unit relative to the first coordinate system; acquiring camera data from the camera; Determining a second coordinate system based on the camera data specifically includes: Determining a second ground normal vector based on the camera data and 3D reconstruction technology; Determining a second driving direction of the camera based on the camera data; and determining a second coordinate system based on the second ground normal vector and the second driving direction of the camera; and determining a camera pose of the camera relative to the second coordinate system using a three-dimensional reconstruction technique based on the camera data and the second coordinate system, specifically comprising: Tracking a set of two-dimensional feature points in a video or image based on the camera data and / or internal parameters of the camera to obtain a trajectory of the feature points over time; Determining a camera position and / or a three-dimensional position of the feature point based on the feature point trajectory over time; Determining a rotation matrix between the camera and a second coordinate system based on the position of the camera and / or the three-dimensional position of the feature point; determining a camera pose of the camera relative to a second coordinate system based on the rotation matrix, the camera pose reflecting an orientation, position, attitude, or rotation of the camera relative to the second coordinate system, the first coordinate system being different from the second coordinate system; determining a relative coordinate pose between the first coordinate system and the second coordinate system, the first coordinate system being different from the second coordinate system; and A relative pose between the camera and the inertial test unit is determined based on the inertial test unit pose, the camera pose, and the relative coordinate pose.

7. The method according to claim 6, characterized in that Also includes: Based on the trajectory of the autonomous vehicle, the first coordinate system is determined.

8. The method according to claim 7, characterized in that Determining the inertial test unit posture further includes: acquiring inertial test unit data from the inertial test unit; and The inertial test unit pose is determined based on the inertial test unit data and the first coordinate system.

9. The method according to claim 6, characterized in that The 3D reconstruction technology is a structure-from-motion (SFM) method.

10. The method according to any one of claims 6 or 9, characterized in that: Determining the relative coordinate posture includes: aligning a first ground normal vector of the first coordinate system with the second ground normal vector of the second coordinate system; aligning a first direction of travel of the inertial testing unit with the second direction of travel of the camera; and The relative coordinate pose between the first coordinate system and the second coordinate system is determined.

11. A non-transitory readable medium comprising at least one set of instructions for calibrating an inertial test unit and a camera of an autonomous vehicle, wherein: When executed by at least one processor of an electronic device, the at least one set of instructions instructs the at least one processor to perform a method comprising: Obtaining a straight-line trajectory of the autonomous driving vehicle; determining an inertial test unit pose of the inertial test unit relative to a first coordinate system, the inertial test unit pose reflecting an orientation, position, attitude, or rotation of the inertial test unit relative to the first coordinate system; acquiring camera data from the camera; Determining a second coordinate system based on the camera data specifically includes: Determining a second ground normal vector based on the camera data and 3D reconstruction technology; determining a second driving direction of the camera based on the camera data; and determining a second coordinate system based on the second ground normal vector and the second driving direction of the camera; Determining a camera pose of the camera relative to the second coordinate system using a three-dimensional reconstruction technique based on the camera data and the second coordinate system specifically includes: Tracking a set of two-dimensional feature points in a video or image based on the camera data and / or internal parameters of the camera to obtain a trajectory of the feature points over time; Determining a camera position and / or a three-dimensional position of the feature point based on the feature point trajectory over time; Determining a rotation matrix between the camera and a second coordinate system based on the position of the camera and / or the three-dimensional position of the feature point; determining a camera pose of the camera relative to a second coordinate system based on the rotation matrix, the camera pose reflecting an orientation, position, attitude, or rotation of the camera relative to the second coordinate system; determining a relative coordinate pose between the first coordinate system and the second coordinate system, the first coordinate system being different from the second coordinate system; and A relative pose between the camera and the inertial test unit is determined based on the inertial test unit pose, the camera pose, and the relative coordinate pose.

12. The non-transitory readable medium according to claim 11, wherein: The method further comprises: The first coordinate system is determined based on the trajectory of the autonomous vehicle.

13. The non-transitory readable medium according to claim 12, wherein: Wherein, determining the posture of the inertial test unit further comprises: acquiring inertial test unit data from the inertial test unit; and The inertial test unit pose is determined based on the inertial test unit data and the first coordinate system.

14. A system for calibrating an inertial test unit and a camera of an autonomous vehicle, comprising: a trajectory acquisition module, configured to acquire a trajectory of the autonomous driving vehicle traveling in a straight line; an inertial test unit attitude determination module, configured to determine an inertial test unit attitude of the inertial test unit relative to a first coordinate system, wherein the inertial test unit attitude reflects an orientation, position, attitude, or rotation of the inertial test unit relative to the first coordinate system; The camera pose determination module is configured as follows: acquiring camera data from the camera; Determining a second coordinate system based on the camera data specifically includes: Determining a second ground normal vector based on the camera data and 3D reconstruction technology; determining a second driving direction of the camera based on the camera data; and determining a second coordinate system based on the second ground normal vector and the second driving direction of the camera; Determining a camera pose of the camera relative to the second coordinate system using a three-dimensional reconstruction technique based on the camera data and the second coordinate system specifically includes: Tracking a set of two-dimensional feature points in a video or image based on the camera data and / or internal parameters of the camera to obtain a trajectory of the feature points over time; Determining a camera position and / or a three-dimensional position of the feature point based on the feature point trajectory over time; Determining a rotation matrix between the camera and a second coordinate system based on the position of the camera and / or the three-dimensional position of the feature point; determining a camera pose of the camera relative to a second coordinate system based on the rotation matrix, the camera pose reflecting the orientation, position, pose or rotation of the camera relative to the second coordinate system; a relative coordinate pose determination module configured to determine a relative coordinate pose between the first coordinate system and the second coordinate system, the first coordinate system being different from the second coordinate system; and The relative posture determination module is configured to determine the relative posture between the camera and the inertial test unit based on the inertial test unit posture, the camera posture and the relative coordinate posture.

Images