Sensor angle alignment
Through the dual control point calibration algorithm and iterative rotation angle calculation, the accuracy of sensor calibration in finite space and time is solved, fast and accurate sensor calibration is achieved, and the accuracy of vehicle navigation is improved.
Patent Information
- Application Number
- CN202311438094.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-06-05
- Filing Date
- 2023-10-31
- Publication Date
- 2024-12-06
AI Technical Summary
Existing vehicle sensor calibration technology requires multiple control points and a long time, making it difficult to accurately calibrate external sensor parameters in limited space and compact periods, affecting the accuracy of vehicle navigation.
Using the dual control point calibration algorithm, the calibration value is determined by iterative rotation angle, the camera and processor capture images, calculate rotation orientation, generate calibration values, and control the vehicle through the vehicle controller to achieve sensor alignment in minimum space and time .
It realizes rapid and accurate calibration of sensor data in large-scale vehicle assembly facilities, reduces calibration time and space requirements, and improves vehicle navigation accuracy.
Smart Images

Figure CN119085730A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates generally to a vehicle sensor alignment system in a vehicle calibration facility, and more particularly to a method and apparatus for a vehicle sensor alignment algorithm that uses iterative rotation angle determination in response to a dual control point calibration target. Background Art
[0002] Modern vehicles are increasingly equipped with more and more electronic systems to improve the safety and performance of the vehicle. These electronic systems may include sensors to enable vehicles, such as autonomous and semi-autonomous vehicles, to sense their environment and navigate with little or no user input. Autonomous vehicles use sensing devices such as radar, lidar, image sensors, etc. to sense their environment. Autonomous vehicle systems also use information from global positioning system (GPS) technology, navigation systems, vehicle-to-vehicle communications, vehicle-to-infrastructure technology, and / or drive-by-wire systems to navigate the vehicle.
[0003] Although autonomous and semi-autonomous vehicles offer many potential advantages over conventional vehicles, in some cases it may be desirable to improve the operation of the vehicle. For example, autonomous vehicle features rely on sensor data being accurately and precisely converted to real-world coordinates for data association and sensor fusion. Due to manufacturing variations and tolerances, these sensors must be calibrated to ensure that estimates of the positions of objects near the vehicle are accurate. Calibration factors can be determined to compensate for angular changes as well as rotational changes. Typically, manufacturing alignment has limited space and tight cycle time requirements, so target-based calibrations requiring many control points use more space and longer cycle times than are available during typical production. Therefore, it is desirable to provide a lightweight algorithm to accurately calibrate sensor external parameters with a minimum of control points, thereby reducing the required calibration time and space. In addition, in conjunction with the accompanying drawings and the aforementioned technical field and background technology, other desired features and characteristics of the present invention will become apparent from the subsequent detailed description and the appended claims. Summary of the invention
[0004] The present disclosure provides a calibration system and method for a visual system or imaging system of a vehicle. A vehicle control system executing a vehicle control algorithm is operable to receive data from a vehicle sensor, solve alignment as a mathematical optimization problem given two or more control points in an iterative process, to generate sensor data and sensor alignment correction values with minimal space and time requirements in a large-volume vehicle assembly facility, etc. According to an exemplary embodiment, a vehicle sensor system includes: a camera configured to capture a first image of a first field of view; a data interface configured to receive a translation vector, a first position of a first calibration point and a second position of a second calibration point, and wherein the first calibration point and the second calibration point are within the first field of view; a processor configured to determine an expected rotational orientation in response to the first position of the first calibration point and the second position of the second calibration point, determine a first position of the first calibration point within the first image and a second position of the second calibration point within the first image, determine a calculated rotational orientation in response to the first position of the first calibration point within the first image and the second position of the second calibration point within the first image, and generate a calibration value in response to the calculated rotational orientation; a memory configured to store the calibration value; and a vehicle controller configured to control a host vehicle in response to the calibration value and subsequent images captured by the camera.
[0005] According to another aspect of the exemplary embodiment, the translation vector, the first position of the first calibration point and the second position of the second calibration point are coordinates within a three-dimensional coordinate system, and wherein the processor is also operable to convert the translation vector, the first position of the first calibration point and the second position of the second calibration point into a two-dimensional coordinate system.
[0006] According to another aspect of the exemplary embodiment, the translation vector is transmitted from the calibration station to the data interface via a wireless network.
[0007] According to another aspect of the exemplary embodiment, the translation vector is determined in response to vehicle design parameters and an orientation of the vehicle within a calibration environment.
[0008] According to another aspect of the exemplary embodiment, the first calibration point and the second calibration point are located on a calibration target within the first field of view.
[0009] According to another aspect of the exemplary embodiment, a two-point algorithm is used to calculate the calculated rotational orientation.
[0010] According to another aspect of the exemplary embodiment, the processor is further operable to determine a subsequently calculated rotational orientation in response to an error in the calculated rotational orientation being greater than a threshold.
[0011] According to another aspect of the exemplary embodiment, the first position of the first calibration point and the second position of the second calibration point are determined in a two-dimensional coordinate system.
[0012] According to another aspect of the exemplary embodiment, an inertial measurement unit is further included for measuring vehicle attitude, and wherein the vehicle attitude and the translation vector are used to calculate the expected rotational orientation.
[0013] According to another aspect of an exemplary embodiment, a method includes: acquiring a translation vector, a first calibration point position, and a second calibration point position via a network interface; determining an expected rotational orientation in response to the translation vector, the first calibration point position, and the second calibration point position; capturing an image of the first calibration point and the second calibration point through a camera; determining a first position of the first calibration point and a second position of the second calibration point through an image recognition algorithm in response to the image; calculating a calculated rotational orientation in response to the first position of the first calibration point and the second position of the second calibration point; determining a calibration value in response to the calculated rotational orientation; storing the calibration value in a memory; and controlling a vehicle in response to the calibration value and subsequent images captured by the camera.
[0014] According to another aspect of the exemplary embodiment, the step further includes converting the translation vector, the first calibration point location, and the second calibration point location from a three-dimensional coordinate system to a two-dimensional coordinate system.
[0015] According to another aspect of the exemplary embodiment, a translation vector is pre-determined in response to vehicle size and camera position.
[0016] According to another aspect of the exemplary embodiment, a translation vector is determined in response to an image and a lidar depth map.
[0017] According to another aspect of the exemplary embodiment, a translation vector is determined in response to a vehicle attitude value detected by an inertial measurement unit.
[0018] According to another aspect of the exemplary embodiment, the calculated rotational orientation is determined in response to a two-point algorithm.
[0019] According to another aspect of the exemplary embodiment, including recalculating the calibration value in response to a subsequent angular difference between a subsequently calculated rotational orientation and the calculated rotational orientation exceeding a threshold.
[0020] According to another aspect of the exemplary embodiment, the first calibration point and the second calibration point are located on a calibration target within a manufacturing calibration test station.
[0021] According to another aspect of the exemplary embodiment, the calculated rotational orientation is determined in response to a small angle approximation.
[0022] According to another aspect of an exemplary embodiment, a camera calibration system includes: an optical sensor for capturing a two-dimensional image of a field of view; a network interface for receiving a translation vector, a first position of a first calibration point, and a second position of a second calibration point, and wherein the first calibration point and the second calibration point are within the field of view; a processor for determining an expected rotational orientation in response to the first position of the first calibration point and the second position of the second calibration point, for determining a first position of the first calibration point within the two-dimensional image and a second position of the second calibration point within the two-dimensional image, for determining a calculated rotational orientation in response to the first position of the first calibration point within the two-dimensional image and the second position of the second calibration point within the two-dimensional image; and for generating a calibration value in response to the calculated rotational orientation; a memory for storing the calibration value; and a vehicle controller for controlling a host vehicle in response to the calibration value and a subsequent two-dimensional image captured by the optical sensor.
[0023] According to another aspect of the exemplary embodiment, the translation vector, the first position of the first calibration point and the second position of the second calibration point are coordinates within a three-dimensional coordinate system, and wherein the processor is also operable to convert the translation vector, the first position of the first calibration point and the second position of the second calibration point into a two-dimensional coordinate system. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Exemplary embodiments will be described below with reference to the following drawings, wherein like reference numerals denote like elements, and wherein:
[0025] Figure 1 A vehicle system using a sensor alignment algorithm according to various embodiments is shown;
[0026] Figure 2 illustrates an exemplary environment for performing a sensor alignment algorithm using a dual control point calibration arrangement in accordance with various embodiments;
[0027] Figure 3 An exemplary block diagram of a system for performing a sensor alignment algorithm using a dual control point calibration arrangement according to various embodiments is shown;
[0028] Figure 4 A flow chart illustrating a method of performing a sensor alignment algorithm using a dual control point calibration arrangement according to various embodiments is shown; and
[0029] Figure 5 An exemplary environment for executing a multi-sensor alignment algorithm in a vehicle application is shown in accordance with various embodiments. DETAILED DESCRIPTION
[0030] The following detailed description is merely exemplary in nature and is not intended to limit application and use. In addition, it is not intended to be bound by any express or implied theory presented in the previous technical field, background technology, summary of the invention or the following detailed description. As used herein, the term module refers to any hardware, software, firmware, electronic control component, processing logic and / or processor device, either alone or in any combination, including but not limited to: application specific integrated circuits (ASICs), electronic circuits, processors (shared, dedicated or groups) and memories that execute one or more software or firmware programs, combinational logic circuits and / or other suitable components that provide the described functions.
[0031] Now turn to Figure 1 , an exemplary system 100 for robust sensor alignment is shown according to various embodiments. The exemplary system 100 includes a vehicle 10 having a plurality of sensing devices 40a to 40n, a propulsion system 20, a transmission system 22, a steering system 24, a braking system 26, a sensor system 28, an actuator system 30, at least one data storage device 32, at least one controller 34, and a communication system 36.
[0032] like Figure 1 As shown in FIG. 1 , the vehicle 10 generally includes a chassis 12, a body 14, front wheels 16, and rear wheels 18. The body 14 is disposed on the chassis 12 and substantially surrounds the components of the vehicle 10. The body 14 and the chassis 12 may together form a frame. The wheels 16-18 are each rotatably coupled to the chassis 12 near a respective corner of the body 14.
[0033] In various embodiments, the vehicle 10 is an autonomous vehicle, and the control system 100 is incorporated into the autonomous vehicle 10 (hereinafter referred to as the autonomous vehicle 10). For example, the autonomous vehicle 10 is a vehicle that is automatically controlled to transport passengers from one location to another. In the illustrated embodiment, the vehicle 10 is shown as a passenger car, but it should be understood that any other means of transportation (including motorcycles, trucks, sports utility vehicles (SUVs), recreational vehicles (RVs), marine vessels, airplanes, etc.) can also be used. In an exemplary embodiment, the autonomous vehicle 10 is a so-called level 4 or level 5 automation system. The level 4 system indicates "high automation", which means that the autonomous driving system is in a driving mode-specific performance in all aspects of the dynamic driving task even if the human driver does not respond appropriately to the intervention request. The level 5 system indicates "full automation", which means the full-time performance of the autonomous driving system in all aspects of the dynamic driving task under all road and environmental conditions that the human driver can manage. It can be understood that in various embodiments, the vehicle 10 can be a non-autonomous vehicle and is not limited to this example.
[0034] As shown, the vehicle 10 generally includes a propulsion system 20, a transmission system 22, a steering system 24, a braking system 26, a sensor system 28, an actuator system 30, at least one data storage device 32, at least one controller 34 and a communication system 36. In various embodiments, the propulsion system 20 may include an internal combustion engine, an electric motor (such as a traction motor) and / or a fuel cell propulsion system. The transmission system 22 is configured to transmit power from the propulsion system 20 to the wheels 16-18 according to a selectable speed ratio. According to various embodiments, the transmission system 22 may include a stepped automatic transmission, a continuously variable transmission or other suitable transmission. The braking system 26 is configured to provide braking torque to the wheels 16-18. In various embodiments, the braking system 26 may include a friction brake, a line brake, a regenerative braking system (such as an electric motor) and / or other suitable braking systems. The steering system 24 affects the position of the wheels 16-18. Although shown as including a steering wheel for illustrative purposes, in some embodiments expected within the scope of the present disclosure, the steering system 24 may not include a steering wheel.
[0035] Sensor system 28 includes one or more sensing devices 40a-40n that sense observable conditions of the external environment and / or the internal environment of autonomous vehicle 10. Sensing devices 40a-40n may include, but are not limited to, radar, lidar, global positioning system, optical camera, thermal camera, ultrasonic sensor, and / or other sensors.
[0036] In various embodiments, the sensing devices 40a to 40n are disposed at different locations of the vehicle 10. In the exemplary embodiments described herein, one or more of the sensing devices 40 to 40n are implemented as a lidar device. In this regard, each of the sensing devices 40a to 40n may include or incorporate one or more lasers, scanning components, optical arrangements, photodetectors, and other components that are appropriately configured to horizontally and rotatably scan the environment near the vehicle 10 at a specific angular frequency or rotational speed. In the exemplary embodiments described herein, one or more of the sensing devices 40a to 40n are implemented as an optical camera that is configured to capture images of the environment near the vehicle 10.
[0037] The actuator system 30 includes one or more actuator devices 42a-42n that control one or more vehicle features, such as, but not limited to, the propulsion system 20, the transmission system 22, the steering system 24, and the braking system 26. In various embodiments, the vehicle features may also include interior vehicle features and / or exterior vehicle features, such as, but not limited to, doors, trunks, and cabin features, such as air, music, lighting, etc. (not numbered).
[0038] Still reference Figure 1 In an exemplary embodiment, the communication system 36 is configured to wirelessly transmit information to and from other entities, such as, but not limited to, other vehicles (“V2V” communications), infrastructure (“V2I” communications), remote systems, personal devices, and / or calibration stations (reference Figure 2 In an exemplary embodiment, the communication system 36 is a wireless communication system configured to communicate via a wireless local area network (WLAN) using the IEEE 802.11 standard or by using cellular data communications. However, additional or alternative communication methods, such as dedicated short-range communication (DSRC) channels, are also considered within the scope of the present disclosure. A DSRC channel refers to a one-way or two-way short-range to medium-range wireless communication channel designed specifically for automotive use and a corresponding set of protocols and standards.
[0039] The data storage device 32 stores data for automatically controlling the autonomous vehicle 10. In various embodiments, the data storage device 32 stores a definition map of the navigable environment. In various embodiments, the definition map can be predefined by a remote system and obtained from the remote system. For example, the definition map can be assembled by a remote system and transmitted to the autonomous vehicle 10 (wirelessly and / or in a wired manner) and stored in the data storage device 32. In various embodiments, the data storage device 32 stores calibrations for aligning the sensing devices 40a to 40n. In various embodiments, using the methods and systems described herein, one or more calibrations are estimated as external parameters. It will be understood that the data storage device 32 can be part of the controller 34, separate from the controller 34, or part of the controller 34 and part of a separate system.
[0040] The controller 34 includes at least one processor 44 and a computer readable storage device or medium 46. The processor 44 can be any custom or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), a secondary processor in a number of processors associated with the controller 34, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, any combination thereof, or generally any device for executing instructions. For example, the computer readable storage device or medium 46 can include volatile and non-volatile storage in read-only memory (ROM), random access memory (RAM), and keep-alive memory (KAM). KAM is a permanent or non-volatile memory that can be used to store various operating variables when the processor 44 is powered off. The computer readable storage device or medium 46 can be implemented using any of a variety of known memory devices, such as PROM (programmable read-only memory), EPROM (electrical PROM), EEPROM (electrically erasable PROM), flash memory, or any other electrical, magnetic, optical, or combination memory device capable of storing data, some of which represents executable instructions used by the controller 34 in controlling the autonomous vehicle 10.
[0041] The instructions may include one or more separate programs, each of which includes an ordered list of executable instructions for implementing logical functions. When executed by processor 44, these instructions receive and process signals from sensor system 28, execute logic, calculations, methods, and / or algorithms for automatically controlling components of autonomous vehicle 10, and generate control signals for actuator system 30 based on the logic, calculations, methods, and / or algorithms to automatically control components of autonomous vehicle 10. Although in Figure 1 Only one controller 34 is shown, but embodiments of autonomous vehicle 10 may include any number of controllers 34 that communicate via any suitable communication medium or combination of communication mediums and cooperate to process sensor signals, execute logic, calculations, methods, and / or algorithms, and generate control signals to automatically control features of autonomous vehicle 10. In various embodiments, one or more instructions of controller 34 are included in control system 100 and, when executed by processor 44, cause processor 44 to perform methods and systems that dynamically align sensor devices by updating calibrations stored in data storage device 32, as described in more detail below.
[0042] According to various embodiments, the controller 34 implements an autonomous driving system (ADS). The software and / or hardware components of the controller 34 (e.g., the processor 44 and the computer-readable storage device 46) are used to provide an autonomous driving system for use in conjunction with the vehicle 10, for example, automatically controlling the various actuators 30 on the vehicle 10 to control vehicle acceleration, steering, and braking, respectively, without human intervention.
[0043] In various embodiments, the instructions for the autonomous driving system can be organized by function or system. For example, the autonomous driving system can include a computer vision system, a positioning system, a guidance system, and a vehicle control system. It will be appreciated that in various embodiments, the instructions can be organized into any number of systems (e.g., combined, further segmented, etc.), as the present disclosure is not limited to the current examples.
[0044] In various embodiments, the computer vision system synthesizes and processes sensor data and predicts the presence, location, classification, and / or path of objects and characteristics of the environment of the vehicle 10. In various embodiments, the computer vision system can combine information from multiple sensors, including but not limited to cameras, lidar, radar, and / or any number of other types of sensors. In various embodiments, the computer vision system receives information from and / or implements the control system 100 described herein.
[0045] The positioning system processes the sensor data and other data to determine the location of the vehicle 10 relative to the environment (e.g., local location relative to a map, precise location relative to a road lane, vehicle heading, speed, etc.). The guidance system processes the sensor data and other data to determine a path to be followed by the vehicle 10. The vehicle control system generates control signals for controlling the vehicle 10 according to the determined path.
[0046] In various embodiments, the controller 34 implements machine learning techniques to assist in functions of the controller 34 , such as feature detection / classification, obstacle mitigation, route traversal, mapping, sensor integration, ground truth determination, and the like.
[0047] According to some exemplary embodiments, a control system, generally shown at 100, is associated with a vehicle 10. In general, the control system 100 selectively aligns two sensors of the vehicle 10 by estimating external parameters. As will be discussed in more detail, the estimation is based on a method that utilizes a mathematical optimization problem given a set of lidar-camera control points with highly flexible 3D-2D correspondence requirements. In various embodiments, the two sensors include a lidar sensor and a camera sensor. It will be appreciated that other sensors may be implemented in various embodiments.
[0048] Now turn to Figure 2, an exemplary environment 200 for a robust sensor alignment algorithm is shown according to various embodiments. The exemplary environment 200 may be a calibration station in a vehicle manufacturing facility and may include a test vehicle 210 and a calibration target 220 having a first calibration point 230 and a second calibration point 232. In some exemplary embodiments, the calibration target 220 may be a checkerboard pattern calibration target or any calibration target having two or more calibration points at known locations within the exemplary environment 200. In this exemplary description, a camera employed in a vehicle platform is described, but the proposed systems and methods may be used to calibrate any number of sensors in any number of platforms and are not limited to cameras in automotive applications.
[0049] In some exemplary embodiments, optical sensor calibration in a manufacturing environment may be performed using a calibration target 220 with a test vehicle 210 at a known position and orientation. The calibration target 220 may be a specially designed object having a known shape and size and may be located at a known position relative to the test vehicle position. In addition, the calibration target 220 will have a plurality of calibration points, such as a first calibration point 230 and a second calibration point 232 having known positions on the calibration target 220. In some exemplary embodiments, the optical sensor may be a camera for capturing an image of the calibration target and the resulting data for adjusting the calibration settings of the camera.
[0050] The system may first be configured to acquire a translation vector corresponding to the test vehicle 210 and the calibration environment 220. The translation vector is a 3D vector that describes the position of the projection center of the sensor relative to the calibration environment coordinate system. The translation vector may be obtained from a calibration station controller, etc., and may be determined in response to vehicle design parameters, such as a designed sensor position and orientation determined using a vehicle design specification, etc., and a test vehicle position and orientation within the calibration environment 220. The test vehicle position may be determined in response to an external sensor that is communicatively coupled to the calibration station controller to determine precise position and orientation information of the test vehicle 210. In response to the determined vehicle position and orientation information, the calibration station controller may determine the translation vector and couple the translation vector to a sensor system processor in the test vehicle 210.
[0051] In some exemplary embodiments, the translation vector may be calculated by a processor 44 within the test vehicle 210 in response to one or more images or other sensor data captured by the test vehicle sensor using a set of known 3D points and their corresponding 2D image points. The known 3D points are placed at known locations in the calibration environment, and the corresponding 2D image points are obtained by capturing images or detecting the known 3D points with one or more sensors. The sensor calibration algorithm may then use these known points to estimate the intrinsic and extrinsic parameters of the camera. Once the intrinsic and extrinsic parameters are estimated, the calibration translation vector may be calculated. The calibration translation vector may be the difference between the coordinates of the known 3D points and their corresponding coordinates in the camera coordinate system.
[0052] After obtaining the translation vector, the sensor integrated into the test vehicle 210 is used to capture an image of the calibration target 220. Image processing techniques are performed on the image to detect the location of the first calibration point 230 and the second calibration point 232 within the image. The first calibration point 230 and the second calibration point 232 are then located in camera image coordinates, and the small angle approximation and the two-point algorithm are applied to calculate the rotational orientation angles of the first calibration point 230 and the second calibration point 232. If the angle in question is small, the small angle approximation can be used to approximate the values of the main trigonometric functions. The equation representing the rotational orientation angle of each calibration point is used to solve for the remaining unknown values using a least squares method, etc. In some exemplary embodiments, the rotational orientation angle corresponds to the pitch, roll, and yaw of the sensor, and / or a rotation around a three-dimensional coordinate system derived from the sensor and / or vehicle position.
[0053] In some exemplary embodiments, the pose of the test vehicle 210 can be determined in response to vehicle inertial measurement unit (IMU) data, and the expected rotational orientation angle between the first calibration point 230 and the second calibration point 232 is determined. The calculated rotational orientation angle is compared with the expected rotational orientation angle to determine the calibration data. In the case of a large initial error, the system is configured to reconvert the 3D target to camera image coordinates with a known translation vector and a newly determined rotational orientation angle, capture a new image and perform two-dimensional feature extraction, and reapply the two-point algorithm to determine the precise rotational orientation angle. This process can be repeated until the difference between the determined rotational orientation angle and the newly determined rotational orientation angle is less than a threshold. In some exemplary embodiments, the threshold can be between one thousandth of a degree and one millionth of a degree. The calibration data is then used to compensate for any difference between the expected rotational orientation angle and the calculated rotational orientation angle.
[0054] Now turn to Figure 3, an exemplary block diagram of a system 300 for a robust sensor alignment algorithm is shown according to various embodiments. The exemplary system 300 includes a camera 310, an image processor 315, a sensor system processor 330, a memory 340, a network interface 345, an IMU 355, and a vehicle controller 350. When images captured by a vehicle camera are used to determine the position of an object for an assisted driving application, camera orientation and rotation errors are introduced due to manufacturing differences in the camera or body 14, and errors introduced due to inconsistencies in vehicle positioning in the installation and calibration test station. To address this problem, a robust camera alignment calibration method is provided. This alignment algorithm is not limited to cameras, but can be used for calibration of various optical sensors, etc. The vehicle controller 350 can be communicatively coupled to the sensor system processor 330 and the IMU 355. The IMU 355 can be configured to determine one or more accelerations, pitch, roll, and yaw of the test vehicle. These values can be coupled to the vehicle controller 350 for determining the test vehicle posture. The test vehicle posture can be calculated by the sensor processing system 330 and can be used to establish a translation vector and / or an expected rotation angle.
[0055] The camera 310 may be an optical sensing device having one or more charge coupled devices (CCD) or complementary metal oxide semiconductor (CMOS) sensors for converting light entering the camera into an electrical signal that can be stored as an image file or presented on an image display or monitor. The size of the sensor, pixel size, noise, dynamic range, and color reproduction all affect the image quality of a digital camera sensor. The camera 310 is typically recessed or mounted to an exterior surface of the vehicle and is used to capture light from the camera's field of view (FOV). Typically, a vehicle will have multiple cameras with overlapping FOVs so that a 360-degree surround view can be generated by stitching together images from multiple cameras. The onboard camera 310 may be used for traffic sign recognition, video-based distance control, road guidance, object detection, and the like. In some exemplary embodiments, the camera 310 may capture high-definition images that are then coupled to an image processor 315 for object detection. The image processor 315 may run an image processing algorithm in response to the image for detecting and classifying objects within the FOV. In some exemplary embodiments, image processor 315 may be integrated into camera 310, may be a discrete component, or may form part of sensor system processor 330 or the like.
[0056] The network interface 345 can be configured to communicatively couple to a data network, such as a wireless local area network, for transmitting and receiving data. In some exemplary embodiments, the network interface 345 can be configured to receive a translation vector from a calibration control station or other data source via a wireless network. The translation vector is a 3D vector describing the center of projection of the camera relative to the calibration environment coordinate system.
[0057] The detected image information may be coupled from the image processor 315 to the sensor system processor 330. In some exemplary embodiments, the sensor system processor 330 or a similar processor or system for processing sensor data may receive the extracted control points from the image processor 315. The sensor system processor 330 is configured to obtain the translation vector via the network interface 345. In response to obtaining the translation vector, the sensor system processor 330 may detect two or more control points in response to the control points detected by the image processor 315. The two control points may be points on a calibration target or the like with a known position within the sensor calibration environment. The rotation angle is determined in response to the two calibration points, the calibration point positions in the known 3D coordinate system, and the calibration points in the 2D camera coordinate system. Then, the sensor rotation angle is determined in response to the equations generated from each calibration point to solve for the remaining unknown values. The resulting calibration data may be stored in the memory 340 and then may be accessed by the vehicle controller 350 for controlling the vehicle according to advanced assisted driving algorithms, such as lane centering operations, adaptive cruise control, or fully autonomous vehicle operations. In the event that there is a large initial error between the determined rotation angle and the initial or expected rotation angle, the system is configured to capture a subsequent image of the calibration target, determine the positions of the two or more control points on the calibration target, and determine a subsequent rotation angle. The subsequent rotation angle is compared to the previously determined rotation angle, and calibration data is generated in response to the comparison. The process can be repeated until the difference between the determined rotation orientation angle and the newly determined rotation orientation angle is less than a threshold value. In some exemplary embodiments, the threshold value can be between one thousandth of a degree and one millionth of a degree.
[0058] In some exemplary embodiments, the vehicle controller 350 can be communicatively coupled to the sensor system processor 330 and the IMU 355. The IMU 355 can be configured to determine one or more accelerations, pitch, roll, and yaw of the test vehicle. These values can be coupled to the vehicle controller 350 for determining the test vehicle pose. The test vehicle pose can be calculated by the sensor processing system 330 and can be used to establish a translation vector and / or an expected rotation angle.
[0059] Now go to Figure 4 , a flow chart illustrating an exemplary implementation of a method 400 for performing a robust sensor alignment algorithm according to various embodiments is shown. Although the following method 400 is described as a series of sequential steps, some steps may be performed in parallel or out of order, such as the capture and processing of image data and the acquisition of translation vectors.
[0060] The method 400 is first operable to obtain 410 a translation vector and position of a target in a calibration environment. The translation vector is a vector determined in response to a physical relationship between a vehicle sensor (such as a camera) and a coordinate system in which the vehicle's surroundings reside. The expected translation vector corresponds to an expected field of view center of the sensor and is determined in response to vehicle design parameters, such as a designed sensor position and orientation determined using a vehicle design specification, etc. The position of the target and / or the position of the calibration point may be provided in a 3D coordinate system.
[0061] The method 400 can then be operated to transform 420 the 3D target position into a camera coordinate system with a known translation vector and an expected rotational orientation between two or more calibration points. Knowing the 3D coordinates of each target position and the position and orientation of the camera within the calibration environment, the calibration points can be projected onto a 2D plane. In response to each calibration point, the projection produces an expected translation vector and an initial rotational orientation.
[0062] The method 400 then captures 430 an image of the calibration environment including the calibration target and one or more calibration points. An image processing algorithm is executed to extract features from the image and determine a 2D position in the camera coordinate system for each calibration point. The method 400 then executes 440 an algorithm to determine an orientation angle in response to the calibration points. In some exemplary embodiments, the orientation angle may correspond to the yaw, roll, and pitch of the sensor and / or vehicle. In some exemplary embodiments, the orientation angle corresponds to a rotation around the X, Y, and Z coordinates of 3D space.
[0063] In some exemplary embodiments, the orientation angle may be determined by assuming a small angle approximation and applying a two-point algorithm to calculate the orientation angle in response to the following relationship for each calibration point.
[0064] in,
[0065] is a known 3D target in vehicle coordinates,
[0066] T is the known transformation vector from vehicle to camera coordinates, assuming
[0067] K is a known camera intrinsic parameter,
[0068] are the known projected 2D image coordinates,
[0069] To simplify the equation,
[0070] make
[0071] Solving for the rotation vector [R] for a single calibration point results in two equations with three unknown values. Solving for two calibration points results in four equations with three unknowns. The three unknowns (such as pitch, roll, and yaw) can then be solved using methods such as least squares to determine the orientation angle.
[0072] In the case of a large initial error R between the translation vector and the initial rotation vector, the method is configured to recalculate 420 the target position using the determined orientation angle in response to the determined translation vector and the expected orientation angle. The method 400 then captures 430 subsequent images and recalculates subsequent orientation angles. If the determined error between the expected orientation angle and the determined orientation angle is less than a threshold error value 445, the method 400 stores 450 a calibration value calculated in response to the error value. The calibration value is used to compensate for any changes in the rotational alignment of the camera mounted on the surface of the vehicle.
[0073] Now go to Figure 5 , shows an exemplary environment 500 for simultaneously calibrating multiple vehicle surface mounted cameras in a calibration station in a manufacturing environment according to various embodiments. The exemplary environment 500 shows a test vehicle 510 having multiple optical sensors with a first field of view 525, a second field of view 535, and a third field of view 545. The first field of view 525 may cover a first calibration target 520, the second field of view 535 covers a second calibration target 530, and the third field of view 545 covers a third calibration target 540.
[0074] In some exemplary embodiments, the sensor system processor of the test vehicle 510 obtains a first determined translation vector and calibration point position for each optical sensor and / or calibration target 520, 530, 540. For each of the corresponding cameras, the calibration point position is converted from the 3D coordinate system to the camera coordinate system. An expected rotational orientation is then calculated for each pair of calibration targets for each camera. In some exemplary embodiments, a first translation vector and a first expected rotational orientation are determined for the first camera, a second determined translation vector and a second expected rotational orientation are determined for the second camera, and a third determined translation vector and a third expected rotational orientation are determined for the third camera.
[0075] The images are captured by a first camera, a second camera, and a third camera of a first field of view 525, a second field of view 535, and a third field of view 545, respectively. An image recognition algorithm is performed on each image to determine the position of the calibration points of each calibration target in 2D camera coordinates. A rotational orientation is calculated for each set of calibration points, and an error between the calculated rotational orientation and their corresponding expected rotational orientation is calculated. In response to the calculated error, calibration values are calculated for each camera. These calibration values are subsequently stored in a memory and used by a sensor system processor, etc., for compensating for orientation changes of each camera when used to determine the position of objects in the environment near the test vehicle 510, such as for executing advanced driver assistance algorithms, etc.
[0076] Although at least one exemplary embodiment has been proposed in the foregoing detailed description, it should be understood that there are a large number of variations. It should also be understood that one or more exemplary embodiments are merely examples and are not intended to limit the scope, applicability or configuration of the present disclosure in any way. On the contrary, the foregoing detailed description will provide a convenient roadmap for implementing one or more exemplary embodiments for those skilled in the art. It should be understood that various changes may be made to the functions and arrangements of the elements without departing from the scope of the present disclosure as set forth in the attached claims and their legal equivalents.
Claims
1. A device comprising: a camera configured to capture a first image of a first field of view; a data interface configured to receive a translation vector, a first position of a first calibration point, and a second position of a second calibration point, and wherein the first calibration point and the second calibration point are within the first field of view; a processor configured to determine an expected rotational orientation responsive to the first position of the first calibration point and the second position of the second calibration point, determine a first position of the first calibration point within the first image and a second position of the second calibration point within the first image, determine a calculated rotational orientation responsive to the first position of the first calibration point within the first image and the second position of the second calibration point within the first image, and generate a calibration value responsive to the calculated rotational orientation; a memory configured to store the calibration value; and A vehicle controller is configured to control a host vehicle in response to the calibration value and subsequent images captured by the camera.
2. The device according to claim 1, wherein: The translation vector, the first position of the first calibration point, and the second position of the second calibration point are coordinates within a three-dimensional coordinate system, and wherein the processor is further operable to convert the translation vector, the first position of the first calibration point, and the second position of the second calibration point into a two-dimensional coordinate system.
3. The device according to claim 1, wherein: The translation vector is transmitted from a calibration station to the data interface via a wireless network.
4. The device according to claim 1, wherein: The translation vector is determined in response to vehicle design parameters and an orientation of the vehicle within a calibration environment.
5. The device according to claim 1, wherein: The first calibration point and the second calibration point are located on a calibration target within the first field of view.
6. The device according to claim 1, wherein: A two-point algorithm is used to calculate the rotational orientation of the calculation.
7. The device according to claim 1, wherein: The processor is further operable to determine a subsequently calculated rotational orientation in response to an error in the calculated rotational orientation being greater than a threshold.
8. The device according to claim 1, wherein: The first position of the first calibration point and the second position of the second calibration point are determined in a two-dimensional coordinate system.
9. The apparatus of claim 1 further comprising an inertial measurement unit for measuring vehicle attitude, and wherein: The vehicle pose and the translation vector are used to calculate the expected rotational orientation.
10. A method comprising: Acquiring a translation vector, a first calibration point position, and a second calibration point position via a network interface; determining an expected rotational orientation in response to the translation vector, the first calibration point location, and the second calibration point location; capturing images of the first calibration point and the second calibration point by a camera; In response to the image, determining a first position of the first calibration point and a second position of the second calibration point by an image recognition algorithm; calculating a calculated rotational orientation in response to the first position of the first calibration point and the second position of the second calibration point; determining a calibration value responsive to the calculated rotational orientation; storing the calibration value in a memory; as well as A vehicle is controlled in response to the calibration value and subsequent images captured by the camera.
