Targetless calibration of cameras and inertial sensors in vehicles

Abstract

This invention relates to targetless calibration of a camera and an inertial sensor in a vehicle. A calibration system for a camera and an inertial sensor, respectively rigidly fixed to a vehicle, includes a controller having a processor and a tangible, non-transitory memory thereon for recording instructions. The camera is adapted to capture one or more images, and the inertial sensor is adapted to acquire motion data, including the angular velocity vector of the vehicle. The controller is configured to receive corresponding data from the camera and the inertial sensor. The controller is configured to perform adaptive feature selection on the one or more images, including extracting one or more regions of interest from the images respectively and calculating a corresponding optimal quality level for each region of interest. Based in part on this adaptive feature selection and the corresponding data from the inertial sensor, the camera is aligned with the inertial sensor.

Description

Technical Field

[0001] This disclosure generally relates to calibration systems for sensor pairs used in vehicles. More specifically, this disclosure relates to targetless calibration of a camera and an inertial sensor, which are respectively rigidly fixed to the vehicle. Background Technology

[0002] Advanced driver assistance systems (ADAS) and autonomous vehicles typically combine various systems for efficient operation, such as anti-lock braking systems (ABS), electronic stability control systems (ESCs), blind spot information systems (BSAs), lane departure warning systems (LDSs), adaptive cruise control systems (ACCSs), and traction control systems (TCSs). These systems rely on input from multiple complex sensing devices, including various types of imaging sensors, optical sensors, and computer vision applications. Data from the sensors is converted into real-world coordinates for data correlation and sensor fusion applications. The quality of sensor alignment and calibration affects the overall accuracy of these systems. Summary of the Invention

[0003] This document discloses a calibration system for a camera and an inertial sensor, each rigidly fixed to a vehicle. The calibration system includes a controller having a processor and a tangible, non-transitory memory for recording instructions thereon. The controller is configured to receive corresponding data from the camera and the inertial sensor, including one or more images captured by the camera and motion data (e.g., acceleration and angular velocity) acquired by the inertial sensor. The controller is configured to perform adaptive feature selection on the one or more images, including extracting one or more regions of interest from the one or more images and calculating a corresponding optimal quality level for each of the one or more regions of interest. Based in part on this adaptive feature selection and the corresponding data from the inertial sensor, the camera is aligned with the inertial sensor.

[0004] Calculating the corresponding optimal quality level may include obtaining multiple pre-selected parameters, including the expected number of feature points, the maximum number of feature points, and the minimum spacing between feature points. Calculating the corresponding optimal quality level may also include obtaining the number of feature points detected in the region of interest. In some embodiments, the controller is programmed to calculate the corresponding optimal quality level based on a previous quality level in a previous loop, an adjustable adaptive gain factor, the maximum number of feature points, the expected number of feature points, and the number of feature points detected. The controller may be programmed to determine the corresponding optimal quality level as: QL prior M is the previous quality level, D is the maximum number of feature points, G is the expected number of feature points, and F is the adjustable adaptive gain factor.

[0005] In some embodiments, the controller is configured to perform a feature detection routine on the one or more regions of interest, in part based on the current quality level, the plurality of pre-selected parameters, and the number of detections of the feature points. The feature detection routine may be adapted to generate an updated value for the number of detections of the feature points. In one example, the feature detection routine is a Shi-Tomasi corner detector. Performing the adaptive feature selection may include detecting a corresponding plurality of feature points in the one or more regions of interest, in part based on the corresponding optimal quality level. In some embodiments, the controller is configured to remove one or more corresponding anomalous features from the one or more regions of interest.

[0006] In some embodiments, the motion data includes an angular velocity vector with corresponding components, the corresponding components including a first angular velocity about a first axis, a second angular velocity about a second axis, and a third angular velocity about a third axis. The controller may be programmed to exclude the calibration result of the current cycle when at least two of the corresponding components of the angular velocity vector are below a predetermined threshold. The controller may also be programmed to include the calibration result of the current cycle when at least two of the corresponding components of the angular velocity vector are greater than or equal to the predetermined threshold.

[0007] In some embodiments, the controller is programmed to determine whether to trigger a diagnostic assessment based on at least one of a predetermined time, user input, and one or more predetermined events. The controller may be programmed to perform a six-degree-of-freedom calibration when the diagnostic assessment is triggered. The controller may be programmed to perform a three-degree-of-freedom calibration when the diagnostic assessment is not triggered.

[0008] This document discloses a method for calibrating a camera and an inertial sensor in a vehicle having a controller with a processor and tangible non-transitory memory, wherein the camera and the inertial sensor are rigidly fixed to the vehicle. The method includes receiving corresponding data from the camera and the inertial sensor, the corresponding data including one or more images captured by the camera and motion data acquired by the inertial sensor. The method includes performing adaptive feature selection on the one or more images, including extracting one or more regions of interest in the one or more images and calculating a corresponding optimal quality level for the one or more regions of interest. The camera is aligned with the inertial sensor, partly based on the adaptive feature selection and the corresponding data from the inertial sensor.

[0009] The present invention also includes the following technical solutions.

[0010] Option 1. A calibration system for rigidly fixing a camera and an inertial sensor to a vehicle, respectively, the calibration system comprising:

[0011] A controller having a processor and tangible, non-transitory memory thereon storing instructions, which, through execution by the processor, cause the controller to:

[0012] Data is received from the camera and the inertial sensor, the data including one or more images captured by the camera and motion data acquired by the inertial sensor;

[0013] Perform adaptive feature selection on the one or more images, including extracting one or more regions of interest in the one or more images and calculating the corresponding optimal quality level for the one or more regions of interest; and

[0014] The camera is aligned with the inertial sensor in part based on the adaptive feature selection and the corresponding data from the inertial sensor.

[0015] Option 2. The calibration system according to Option 1, wherein calculating the corresponding optimal quality level includes:

[0016] Multiple pre-selected parameters are obtained, including the expected number of feature points, the maximum number of feature points, and the minimum distance between the feature points; and

[0017] Obtain the number of detected feature points in the one or more regions of interest.

[0018] Option 3. The calibration system according to Option 2, wherein:

[0019] The controller is programmed to calculate the corresponding optimal quality level based on the previous quality level in the previous loop, the adjustable adaptive gain factor, the maximum number of feature points, the expected number of feature points, and the number of detected feature points.

[0020] Option 4. The calibration system according to Option 3, wherein:

[0021] The controller is programmed to determine the corresponding optimal quality level as follows: QL prior M is the previous quality level, D is the maximum number of feature points, G is the expected number of feature points, and F is the adjustable adaptive gain factor.

[0022] Option 5. The calibration system according to Option 2, wherein:

[0023] The controller is configured to perform feature detection routines on the one or more regions of interest, in part based on the current quality level, the plurality of pre-selected parameters, and the number of detections of the feature points; and

[0024] The feature detection routine is adapted to generate an updated value for the number of detected feature points.

[0025] Option 6. The calibration system according to Option 5, wherein:

[0026] The feature detection routine is the Shi-Tomasi corner detector.

[0027] Option 7. The calibration system according to Option 1, wherein:

[0028] Performing the adaptive feature selection involves detecting a plurality of corresponding feature points in one or more regions of interest, in part based on the corresponding optimal quality level.

[0029] Option 8. The calibration system according to Option 7, wherein:

[0030] The controller is configured to remove one or more corresponding anomalous features from the one or more regions of interest.

[0031] Option 9. The calibration system according to Option 1, wherein:

[0032] The motion data includes angular velocity vectors with corresponding components, the corresponding components including a first angular velocity about a first axis, a second angular velocity about a second axis, and a third angular velocity about a third axis; and

[0033] The controller is programmed to exclude the calibration result of the current cycle when at least two of the corresponding components of the angular velocity vector are below a predetermined threshold.

[0034] Option 10. The calibration system according to Option 9, wherein:

[0035] The controller is programmed to include the calibration result of the current cycle when at least two of the corresponding components of the angular velocity vector are greater than or equal to the predetermined threshold.

[0036] Option 11. The calibration system according to Option 1, wherein:

[0037] The controller is programmed to determine whether to trigger a diagnostic assessment based on at least one of a predetermined time, user input, and one or more predetermined events; and

[0038] The controller is programmed to perform a six-degree-of-freedom calibration when the diagnostic assessment is triggered.

[0039] Option 12. The calibration system according to Option 11, wherein:

[0040] The controller is programmed to perform a three-degree-of-freedom calibration when the diagnostic assessment is not triggered.

[0041] Option 13. A method for calibrating a camera and an inertial sensor in a vehicle, the vehicle having a controller with a processor and tangible non-transitory memory, the camera and the inertial sensor being rigidly fixed to the vehicle, the method comprising:

[0042] Data is received from the camera and the inertial sensor, the data including one or more images captured by the camera and motion data acquired by the inertial sensor;

[0043] Perform adaptive feature selection on the one or more images, including extracting one or more regions of interest in the one or more images and calculating the corresponding optimal quality level for the one or more regions of interest; and

[0044] The camera is aligned with the inertial sensor in part based on the adaptive feature selection and the corresponding data from the inertial sensor.

[0045] Option 14. The method according to Option 13, wherein calculating the corresponding optimal quality level includes:

[0046] Multiple pre-selected parameters are obtained, including the expected number of feature points, the maximum number of feature points, and the minimum spacing between feature points;

[0047] Obtain the number of detected feature points in the one or more regions of interest; and

[0048] The current quality level is calculated based on the previous quality level in the previous loop, the maximum number of feature points, the expected number of feature points, and the number of detected feature points.

[0049] Option 15. The method according to Option 14, wherein calculating the current quality level includes:

[0050] The current quality level is calculated as follows: QL prior M is the previous quality level, D is the maximum number of feature points, G is the desired number of feature points, and F is the number of detected feature points.

[0051] Option 16. The method according to Option 14 further includes:

[0052] The controller performs feature detection routines on the one or more regions of interest, in part based on the current quality level, the plurality of pre-selected parameters, and the number of detections of the feature points; and

[0053] An updated value for the number of detected feature points is generated via the feature detection routine.

[0054] Option 17. The method according to Option 13, wherein performing the adaptive feature selection includes:

[0055] Detecting corresponding multiple feature points in one or more regions of interest, in part based on the corresponding optimal quality level; and

[0056] Remove one or more corresponding anomalous features from the one or more regions of interest.

[0057] Option 18. The method according to Option 13, wherein the motion data includes an angular velocity vector with corresponding components, and the method further includes:

[0058] When at least two of the corresponding components of the angular velocity vector are higher than a predetermined threshold, the corresponding data for the current cycle is excluded. The corresponding components of the angular velocity vector include a first angular velocity about a first axis, a second angular velocity about a second axis, and a third angular velocity about a third axis.

[0059] When at least two of the corresponding components of the angular velocity vector are less than or equal to the predetermined threshold, the corresponding data of the current cycle is included.

[0060] Option 19. The method according to Option 13 further includes:

[0061] The controller determines whether to trigger a diagnostic assessment based on at least one of a predetermined time, user input, and one or more predetermined events.

[0062] When the diagnostic assessment is triggered, a six-degree-of-freedom calibration is performed; and

[0063] When the diagnostic assessment is not triggered, perform a three-degree-of-freedom calibration.

[0064] The foregoing features and advantages, as well as other features and advantages, of this disclosure will become apparent when considered in conjunction with the accompanying drawings and the following detailed description of the best mode for carrying out this disclosure. Attached Figure Description

[0065] Figure 1This is a schematic partial diagram of a calibration system for a vehicle equipped with a camera, inertial sensor, and controller.

[0066] Figure 2 It is by Figure 1 A schematic example image captured by the camera;

[0067] Figure 3 It is an operation Figure 1 A flowchart of the calibration system method;

[0068] Figure 4 It shows in detail Figure 3 A flowchart of a part of the method; and

[0069] Figure 5 It shows in detail Figure 3 The flowchart for another part of the method.

[0070] Representative embodiments of this disclosure are shown by way of non-limiting example in the accompanying drawings and are described in more detail below. However, it should be understood that the novelty of this disclosure is not limited to the specific forms shown in the drawings listed above. Rather, this disclosure will cover modifications, equivalents, combinations, sub-combinations, substitutions, groupings, and alternatives that fall within the scope of this disclosure as covered, for example, by the appended claims. Detailed Implementation

[0071] Referring to the accompanying drawings, where the same reference numerals refer to the same parts, Figure 1 A calibration system 10 for a sensor pair 12 is schematically illustrated, the sensor pair 12 including a camera 14 and an inertial sensor 16. The camera 14 and the inertial sensor 16 are each rigidly fixed, connected, or attached (directly or indirectly) to a mobile platform 20, referred to herein as vehicle 20. The respective positions of the camera 14 and the inertial sensor 16 on / in vehicle 20 may vary depending on the application at hand. Vehicle 20 may include, but is not limited to, passenger cars, SUVs, light trucks, heavy vehicles, minivans, buses, transport vehicles, bicycles, mobile robots, agricultural implements (e.g., tractors), sports-related equipment (e.g., golf carts), boats, aircraft, trains, or other mobile platforms having the sensor pair 12. Vehicle 20 may be an electric vehicle, which may be fully electric or hybrid / partially electric. It is to be understood that vehicle 20 may take many different forms and have additional components.

[0072] refer to Figure 1 The calibration system 10 includes a controller C having at least one processor P and at least one memory M (or a non-transitory, tangible computer-readable storage medium) on which instructions are recorded for execution as described below. Figure 3The calibration method 100 for sensor 12 is described in detail. Memory M stores a set of controller-executable instructions, and processor P executes the set of controller-executable instructions stored in memory M. The calibration system 10 adapts to different types of lighting conditions and does not require expensive targets and calibration setups.

[0073] Figure 1 The camera 14 is suitable for acquiring visual data and can be combined with various types of optical sensors and photodetectors available to those skilled in the art. The camera 14 can be a stereo camera suitable for acquiring left and right image pairs. The inertial sensor 16 is a device that integrates multiple sensors to estimate the spatial orientation of an object. (Reference) Figure 1 Inertial sensor 16 is configured to acquire motion data of vehicle 20 in multiple directions, such as linear acceleration, linear velocity, and rotational angular velocity, along the X, Y, and Z axes, respectively. Inertial sensor 16 may include a corresponding accelerometer, gyroscope, and magnetometer (not shown) for each axis to determine rotational acceleration along the X, Y, and Z axes, referred to as roll, pitch, and yaw, respectively. Inertial sensor 16 defines an inertial reference frame F1, while camera 14 defines a camera reference frame F2, both relative to a global reference frame F3.

[0074] For accurate navigation and measurement, the inertial reference frame F1 and the camera reference frame F2 are aligned with the global reference frame F3 in a process sometimes referred to as extrinsic parameter calibration. During this process, coordinate transformation relationships are established between the various reference frames, which typically uses a calibration target. Calibration system 10 (via execution of method 100) provides a solution for performing extrinsic calibration to support sensor alignment requirements without using a specific calibration target. Method 100 is based on matching feature points (e.g., roadside features) detected in images acquired by camera 14 with vehicle motion data obtained from inertial sensor 16.

[0075] Figure 1 The controller C can be an integral part of other controllers of vehicle 20 or a separate module operatively connected to other controllers of vehicle 20. For example, controller C can be an electronic control unit (ECU) of vehicle 20. Controller C can receive input from other sensors operatively connected to vehicle 20, such as navigation sensor 22 and radar unit 24. Navigation sensor 22, which can be a Global Positioning Satellite (GPS) sensor, is configured to obtain the position coordinates of vehicle 20, such as latitude and longitude values. (Reference) Figure 1 The controller C can access map database 26, which may be a public or commercial information source available to those skilled in the art, such as Google Earth. Alternatively, map database 26 may be loaded into the memory M of the controller C.

[0076] refer to Figure 1 Various components of the calibration system 10 can communicate with the controller C (and with each other) via a wireless network 30, which can be a short-range or long-range network. The wireless network 30 can be a communication bus, which may be in the form of a Serial Controller Area Network (CAN-BUS). The wireless network 30 can be combined with Bluetooth. TM Wireless local area networks (LANs), wireless metropolitan area networks (MANs), or wireless wide area networks (WANs) that connect multiple devices using wireless distribution methods. Other types of connections are also possible.

[0077] The exemplary image 50 (or image frame) captured by camera 14 in Figure 2 It is shown in the figure and described below. Reference Figure 2 Image 50 is covered with feature points 52 detected by controller C, for example via feature detection routine 32. Feature detection routine 32 may employ edge detection, corner detection, ridge detection, scale-invariant transform, or other types of techniques. Feature points 52 are separated by spacing 54. Image 50 may be divided into one or more regions of interest 56, each having a corresponding plurality of feature points 52. As described below, method 100 includes methods for extracting features from an image (e.g., Figure 2 The mechanism in the image 50 adaptively selects the highest quality feature points 52 and ensures the uniform distribution of feature points 52. Method 100 prevents the detection of low-quality feature points, or the detection of too few (i.e., a small number) feature points 52, which could potentially introduce errors in the calibration parameters.

[0078] Now for reference Figure 3 The diagram illustrates a flowchart of method 100. Method 100 can be implemented by storing... Figure 1 On the controller C and can be controlled by Figure 1 The controller C section executes computer-readable code or instructions. The method may be executed in real time, continuously, systematically, sporadically, and / or at regular intervals, for example, every 10 milliseconds during normal and continuous operation of vehicle 20.

[0079] Figure 3 Method 100 begins at block 101 and ends at block 103, and includes subroutines or modules 102, 202, 302, 402, 502, 602, and 702. Method 100 does not need to be applied in the specific order described herein. Furthermore, it should be understood that some modules (or some blocks within modules) may be eliminated.

[0080] According to module 102, controller C is programmed to receive corresponding data from camera 14 and inertial sensor 16, including but not limited to one or more images captured within a predetermined time interval and angular velocity data (e.g., angular velocity vector). From module 102, method 100 can simultaneously continue to modules 202, 302, and 402.

[0081] according to Figure 3 Module 202, in which controller C is programmed to perform adaptive feature selection, includes detecting high-quality features robust to ambient lighting conditions. Module 202 includes blocks 204, 206, 208, 210, and 212. Module 202 begins at block 204, where controller C is programmed to extract one or more regions of interest 56 from image 50 (see [link to module 202]). Figure 2 For example, the first area of ​​interest is 58 and the second area of ​​interest is 60.

[0082] Next, according to block 206, controller C is programmed to adaptively calculate the corresponding optimal quality level for each region of interest 56. The quality level is a parameter characterizing the lowest acceptable quality of image feature points, such as edges or corners. In some embodiments, the quality level is a normalized number between 0 and 1. Feature points (e.g., corners) with corresponding scores or quality measurements below this quality level are rejected. In some cases, such as in low-light conditions, a relatively low quality level is appropriate, while in others, a relatively high quality level is appropriate. Figure 4 An exemplary implementation of block 206 is shown, which has a feedback loop mechanism. (See reference...) Figure 4 According to subblock 252, controller C is adapted to obtain multiple pre-selected parameters, including the expected number (D) of feature points 52 per frame, the maximum number (M) of feature points 52 per frame, and the minimum value of the spacing 54 between feature points 52. These multiple pre-selected parameters can be pre-programmed or selected by the user. According to subblock 254, controller C is adapted to obtain the number (F) of feature points 52 detected in each iteration or loop. Feature detection routines 32, available to those skilled in the art, can be employed, employing edge detection, corner detection, ridge detection, scale-invariant transform, or other types of techniques. For example, feature detection routine 32 can be a Shi-Tomasi corner detector, a Harris corner detector, or a Susan detector. Feature detection routine 32 outputs or generates an updated value for the number (F) of feature points 52 detected. For the first iteration or loop, an initial value for the number (F) of detections can be pre-selected.

[0083] refer to Figure 4 Sub-block 256 receives inputs from sub-blocks 252 and 254. Based on sub-block 256, controller C is adapted to adjust the input based on the previous quality level (QL) from previous iterations. priorThe current quality level (QL) is calculated using the maximum number (M), expected number (D), and number of detections (F) of feature points 52. current Current Quality Level (QL) current ) can be determined as: , where G is the adaptive gain factor. For example, the value of the adaptive gain factor (G) can be chosen to be 0.1. However, it should be understood that the value of the adaptive gain factor (G) can be adjusted based on the application at hand. For the first iteration, the previous quality level (QL) can be selected. prior The initial value of ). Next, according to block 258, controller C is programmed to obtain an updated value of the number of detections (F) of feature points 52, where the current quality level (QL) is . current The quality level is determined in sub-block 256 (via feature detection routine 32). As mentioned above, the quality level is a parameter characterizing the lowest acceptable quality of feature points in a frame, where features below the current quality level (QL) are considered acceptable. current The feature points of the corresponding quality measurement results were rejected.

[0084] The adaptive feature selection algorithm can determine the optimal quality level for each individual frame / scene. For example, if vehicle 20 moves from a bright scene to a darker scene, controller C adaptively reduces the feature quality level based on the number of currently detected feature points 52 relative to the expected number of feature points 52. This enables the detection of more feature points 52 and improves calibration performance. Optionally, the execution of block 206 can continue to decision block 260, where controller C is adapted to determine the difference (ΔQ = QL) between the current quantity level and the previous quantity level. current - QL prior The difference between the current and previous quality levels is checked against a predetermined value. If the difference is within the predetermined value (decision block 260 = yes), block 206 ends. In a non-limiting example, block 206 ends when the current level and the previous level are within 1% of each other. If not (decision block 260 = no), the loop repeats with a feedback loop. In other words, as shown in line 262, the number (F) of detections of feature points 52 determined in sub-block 258 is used to update sub-block 254, and this process is repeated. For each region of interest 56, controller C can be configured to set the corresponding optimal quality level to the current quality level when the difference between the previous level and the current quality level is within a predetermined value.

[0085] Return to reference Figure 3Module 202 proceeds from block 206 to block 208. According to block 208, controller C is programmed to extract feature points 52 detected for each region of interest 56, where the corresponding optimal quality level was determined in block 206. Module 202 continues to block 210, where outlier features 62 can be removed from the feature points 52 in each region of interest 56. As those skilled in the art will understand, the data may include “inliers” that can be interpreted by certain model parameters and “outliers” that are not suitable for the model. An exemplary routine that can be used to remove outlier features 62 is a random sample consensus routine called RANSAC.

[0086] According to block 212, controller C is programmed to track feature point 52 and obtain the orientation and scaled position of camera 14 in each image 50 relative to camera 14 in other images. Here, controller C is programmed to determine the spatial and geometric relationships of feature point 52 (extracted in block 208) by means of the movement of camera 14, via optical flow methods, from structure of motion (SFM) techniques / routines, and other 3-D reconstruction routines available to those skilled in the art.

[0087] Now for reference Figure 3 In module 302, controller C is programmed to detect degenerate motion and exclude the results of the current iteration in certain scenarios. Module 302 begins at block 304, where controller C is programmed to determine whether at least two of the corresponding components of the angular velocity vector of vehicle 20 are greater than or equal to a predetermined threshold. This allows for sufficient or strong signal excitation for calibration purposes. In a non-limiting example, the predetermined threshold may be set between approximately 0.03 and 0.04 radians per second. The angular velocity vector is obtained from data acquired by inertial sensor 16 and includes three components: a first angular velocity (ω) about a first axis (e.g., the X-axis). X ), and the second angular velocity (ω) around the second axis (e.g., the Y-axis). Y ) and the third triangular velocity (ω) around the third axis (e.g., the Z-axis). Z ).

[0088] If block 304 = is (i.e., (ω) X ω Y ω ZIf at least two of the values ​​in block 304 are greater than or equal to a predetermined threshold, then module 302 proceeds to block 306, where the calibration result of the current iteration or loop is included in the dataset output to module 502. If block 304 = No, then module 302 proceeds to block 308, where the calibration result of the current iteration or loop is excluded from the dataset output to module 502. Although the calibration result in the current loop is excluded from the optimization, the previous calibration results remain unchanged at the current timestamp, iteration, or loop during the optimization phase. Module 302 ends at blocks 306 and 308.

[0089] according to Figure 3 Module 402, controller C, is programmed to integrate propagation data from various sensors in inertial sensor 16. This propagation data includes the position and velocity of vehicle 20, the distance vehicle 20 has moved within a specified time, and the orientation of vehicle 20. (Reference) Figure 3 After modules 202, 302 and 402 are executed, method 100 continues to module 502.

[0090] according to Figure 3 Module 502, controller C is programmed to receive a calibration dataset (from modules 202, 302, and 402). Controller C is adapted to perform nonlinear optimization on the calibration dataset to align detected / tracked feature points 52 and vehicle motion / orientation data. Estimating the motion and orientation of vehicle 20 may involve using visual rangefinding techniques on a sequence of images captured by moving camera 14. For example, feature points detected in a first image frame are matched with those in a second image frame. This information is used to construct an optical flow field for the feature points detected in both images. This optical flow field describes the divergence from a single point, thus indicating the direction of motion of camera 14, and therefore providing an estimate of the motion of camera 14 (and vehicle 20). This information is matched with vehicle motion data from inertial sensor 16 (integrated in module 402) to generate alignment data. This process may be repeated for multiple sets of adjacent images. In some embodiments, a machine learning algorithm may be employed to minimize a cost function based on the projection error between two adjacent images.

[0091] Proceeding from module 502, method 100, to module 602. According to... Figure 3 In module 602, controller C is programmed to perform calibration against inertial reference frame F1 and camera reference frame F2 based on alignment data generated in module 502. Specifically, a coordinate transformation relationship is established between inertial reference frame F1, camera reference frame F2, and global reference frame F3. From module 602, method 100 continues to module 702. Figure 3 In module 702, controller C is programmed to detect erroneous position calibration. Figure 5 An exemplary implementation of module 702 is shown in the figure. Reference is now made to... Figure 5 Module 702 begins with block 704, where controller C is programmed to track feature point 52 (an example of which is in...). Figure 2 (As shown in the diagram). Module 702 proceeds to decision block 706 to determine whether a diagnostic assessment should be performed based on a predetermined time, user input, sensor replacement, and other factors. For example, the diagnostic assessment may be configured to be performed at specific time intervals, when a specific event (e.g., an engine event, scheduled maintenance) occurs, or upon user request.

[0092] refer to Figure 5 If it is determined that a diagnostic assessment should not be performed (decision block 706 = No), module 702 proceeds to block 708, where a three-degree-of-freedom calibration is performed. From block 708, module 702 proceeds to block 710, where the orientation calibration is updated and module 702 terminates. If it is determined that a diagnostic assessment should be performed (decision block 706 = Yes), module 702 proceeds to block 712, where vehicle 20 is driven multiple laps in the parking lot (e.g., at least two laps), and proceeds to block 714, where a six-degree-of-freedom calibration is performed. This six-degree-of-freedom calibration covers translation along three vertical axes, combined with orientational changes caused by rotation about these three vertical axes, referred to as yaw (normal axis), pitch (lateral axis), and roll (longitudinal axis).

[0093] refer to Figure 5 From block 714, module 702 proceeds to block 710 (as shown in line 716), where the orientation calibration is updated, and then proceeds to decision block 718 to determine if the existing position calibration is valid. In other words, the existing position calibration (from module 602) is compared with the result of the six-DOF calibration (performed in block 714). If the existing position calibration is sufficiently close or valid (decision block 718 = Yes), then module 702 terminates. If the existing position calibration is invalid (decision block 718 = No), then module 702 proceeds to block 720. (See reference...) Figure 5 According to block 720, the existing position calibration is updated based on the six-degree-of-freedom calibration performed in block 714, and module 702 ends.

[0094] In summary, calibration system 10 (via execution of method 100) allows for precise calibration of sensor 12 by utilizing feature detection to track the image sequence captured by camera 14 and match them with data from inertial sensor 16. Calibration system 10 minimizes calibration errors due to degenerative motion, for example, by stopping calibration when there is no two-axis rotation. Method 100 includes a decision scheme (by...) Figure 5(Illustrated in module 702) to select between rapid three-degree-of-freedom dynamic calibration for the orientation of sensor pair 12 and six-degree-of-freedom dynamic calibration for the position and orientation of sensor pair 12. Method 100 maintains a certain number of detected high-quality features to achieve robustness to ambient lighting conditions, thereby ensuring the selection of the minimum number of feature points with the highest quality and maintaining a uniform distribution of these feature points 52.

[0095] Figure 1 The controller C includes a computer-readable medium (also called a processor-readable medium) that includes a non-transitory (e.g., tangible) medium involved in providing data (e.g., instructions) that can be read by a computer (e.g., by the computer's processor). Such media can take many forms, including but not limited to non-volatile and volatile media. For example, non-volatile media may include optical discs or magnetic disks and other persistent storage. For example, volatile media may include dynamic random access memory (DRAM), which may constitute main memory. Such instructions can be transmitted via one or more transmission media, including coaxial cables, copper wires, and optical fibers, including wires that include a system bus coupled to the computer's processor. For example, some forms of computer-readable media include floppy disks, hard disks, magnetic tapes, other magnetic media, CD-ROMs, DVDs, other optical media, physical media with perforated patterns, RAM, PROMs, EPROMs, FLASH-EEPROMs, other memory chips or cassette tapes, or other computer-readable media.

[0096] The lookup tables, databases, data repositories, or other data stores described herein may include various mechanisms for storing, accessing, and retrieving a variety of data, including hierarchical databases, a set of files in a file-based rechargeable energy storage system, application databases in proprietary formats, relational database energy management systems (RDBMS), and so on. Each such data store may be contained within a computing device employing a computer operating system such as one of those mentioned above, and may be accessed via a network in one or more of a variety of ways. The file system may be accessed from the computer operating the rechargeable energy storage system and may include files stored in various formats. In addition to languages ​​used for creating, storing, editing, and executing stored programs, such as the PL / SQL language mentioned above, the RDBMS may also employ Structured Query Language (SQL).

[0097] Figure 3-5The flowcharts illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code, including one or more executable instructions for implementing a specified logical function. It will also be noted that each block in the block diagram and / or flowchart illustrations, and combinations of blocks in the block diagram and / or flowchart illustrations, may be implemented by a rechargeable energy storage system based on purpose-specific hardware that performs a specific function or action, or by a combination of purpose-specific hardware and computer instructions. These computer program instructions may also be stored in a computer-readable medium that can direct a controller or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable medium produce an article of art including instructions for implementing the functions / actions specified in the flowchart and / or block diagram blocks.

[0098] Numerical values ​​of parameters (e.g., quantities or conditions) in this specification, including the appended claims, should be understood to be modified by the term "about" in the respective instances, regardless of whether "about" actually precedes the numerical value. "About" indicates that the numerical value allows for some slight imprecision (the accuracy of the value by some method; approximately or reasonably close to the value; almost). If the imprecision provided by "about" is not understood in this common sense in the art, then "about" as used herein at least indicates variations that may arise from common methods of measuring and using these parameters. Furthermore, the disclosure of ranges includes the disclosure of each value as well as ranges further subdivided throughout the range. Each value within the range and the endpoints of the range are disclosed herein as separate embodiments.

[0099] Detailed description and figures or drawings are provided to support and describe this disclosure, but the scope of this disclosure is defined only by the claims. While some best modes and other embodiments for implementing the claimed disclosure have been described in detail, various alternative designs and embodiments exist for practicing the disclosure as defined in the appended claims. Furthermore, features of the embodiments shown in the drawings or the various embodiments mentioned in this specification are not necessarily to be construed as embodiments independent of each other. Rather, it is possible that each feature described in an example of one embodiment may be combined with one or more other desired features from other embodiments, resulting in other embodiments that are not described in words or with reference to the drawings. Therefore, such other embodiments fall within the framework of the appended claims.

Claims

1. A calibration system for rigidly fixing a camera and an inertial sensor to a vehicle, respectively, the calibration system comprising: A controller having a processor and tangible, non-transitory memory thereon storing instructions, which, through execution by the processor, cause the controller to: Data is received from the camera and the inertial sensor, the data including one or more images captured by the camera and motion data acquired by the inertial sensor; Perform adaptive feature selection on the one or more images, including extracting one or more regions of interest in the one or more images and calculating the corresponding optimal quality level for the one or more regions of interest; as well as The camera is aligned with the inertial sensor, partly based on the adaptive feature selection and the corresponding data from the inertial sensor. The controller is programmed to determine whether to trigger a diagnostic assessment based on at least one of a predetermined time, user input, and one or more predetermined events. as well as The controller is programmed to perform a six-degree-of-freedom calibration when the diagnostic assessment is triggered.

2. The calibration system of claim 1, wherein, Calculating the corresponding optimal quality level includes: Multiple pre-selected parameters are obtained, including the expected number of feature points, the maximum number of feature points, and the minimum distance between the feature points; and Obtain the number of detected feature points in the one or more regions of interest.

3. The calibration system according to claim 2, wherein: The controller is programmed to calculate the corresponding optimal quality level based on the previous quality level in the previous loop, the adjustable adaptive gain factor, the maximum number of feature points, the expected number of feature points, and the number of detected feature points.

4. The calibration system according to claim 3, wherein: The controller is programmed to determine the respective optimal quality level as: where QL prior is the previous quality level, M is the maximum number of feature points, D is the desired number of feature points, G is the tunable adaptive gain factor, and F is the detected number of feature points.

5. The calibration system according to claim 2, wherein: The controller is configured to perform feature detection routines on the one or more regions of interest, in part based on the current quality level, the plurality of pre-selected parameters, and the number of detections of the feature points; as well as The feature detection routine is adapted to generate an updated value for the number of detected feature points.

6. The calibration system according to claim 5, wherein: The feature detection routine is the Shi-Tomasi corner detector.

7. The calibration system according to claim 1, wherein: Performing the adaptive feature selection involves detecting a plurality of corresponding feature points in one or more regions of interest, in part based on the corresponding optimal quality level.

8. The calibration system according to claim 7, wherein: The controller is configured to remove one or more corresponding anomalous features from the one or more regions of interest.

9. The calibration system according to claim 1, wherein: The motion data includes angular velocity vectors with corresponding components, the corresponding components including a first angular velocity about a first axis, a second angular velocity about a second axis, and a third angular velocity about a third axis; as well as The controller is programmed to exclude the calibration result of the current cycle when at least two of the corresponding components of the angular velocity vector are below a predetermined threshold.

10. The calibration system according to claim 9, wherein: The controller is programmed to include the calibration result of the current cycle when at least two of the corresponding components of the angular velocity vector are greater than or equal to the predetermined threshold.

11. The calibration system according to claim 1, wherein: The controller is programmed to perform a three-degree-of-freedom calibration when the diagnostic assessment is not triggered.

12. A method for calibrating a camera and an inertial sensor in a vehicle, the vehicle having a controller with a processor and tangible non-transitory memory, the camera and the inertial sensor being rigidly fixed to the vehicle, the method comprising: Data is received from the camera and the inertial sensor, the data including one or more images captured by the camera and motion data acquired by the inertial sensor; Perform adaptive feature selection on the one or more images, including extracting one or more regions of interest in the one or more images and calculating the corresponding optimal quality level for the one or more regions of interest; The camera is aligned with the inertial sensor in part based on the adaptive feature selection and the corresponding data from the inertial sensor; The controller determines whether to trigger a diagnostic assessment based on at least one of a predetermined time, user input, and one or more predetermined events. as well as When the diagnostic assessment is triggered, a six-degree-of-freedom calibration is performed.

13. The method according to claim 12, wherein, Calculating the corresponding optimal quality level includes: Multiple pre-selected parameters are obtained, including the expected number of feature points, the maximum number of feature points, and the minimum spacing between feature points; Obtain the number of detected feature points in the one or more regions of interest; and The current quality level is calculated based on the previous quality level in the previous loop, the maximum number of feature points, the expected number of feature points, and the number of detected feature points.

14. The method according to claim 13, wherein, Calculating the current quality level includes: The current quality level is calculated as follows: QL prior M is the previous quality level, D is the maximum number of feature points, G is the desired number of feature points, and F is the number of detected feature points.

15. The method of claim 13, further comprising: The controller performs feature detection routines on the one or more regions of interest, in part based on the current quality level, the plurality of pre-selected parameters, and the number of detections of the feature points. as well as An updated value for the number of detected feature points is generated via the feature detection routine.

16. The method according to claim 12, wherein, Performing the adaptive feature selection includes: Detecting corresponding multiple feature points in one or more regions of interest, in part based on the corresponding optimal quality level; and Remove one or more corresponding anomalous features from the one or more regions of interest.

17. The method according to claim 12, wherein, The motion data includes angular velocity vectors with corresponding components, and the method further includes: When at least two of the corresponding components of the angular velocity vector are higher than a predetermined threshold, the corresponding data for the current cycle is excluded. The corresponding components of the angular velocity vector include a first angular velocity about a first axis, a second angular velocity about a second axis, and a third angular velocity about a third axis. When at least two of the corresponding components of the angular velocity vector are less than or equal to the predetermined threshold, the corresponding data of the current cycle is included.

18. The method of claim 12, further comprising: When the diagnostic assessment is not triggered, perform a three-degree-of-freedom calibration.

Images