Safety camera based inertial measurement unit calibration for stationary systems

Abstract

Techniques and systems for safety camera based IMU calibration for stationary systems, including vehicles, are described. Existing vehicle camera systems are employed with enhanced security to prevent malicious attempts by hackers to try and cause the vehicle to enter an IMU calibration mode. IMU calibration occurs when the calibration system determines that the vehicle is parked in a controlled environment; calibration targets are positioned at different perspectives to the vehicle camera to act as a source of optical patterns of encoded data. The features of the patterns are for security as well as for alignment functionality. Images of the calibration targets enable the vehicle coordinate system to be inferred from which calculations for IMU installation error compensation are performed. The relative rotation between the IMU and vehicle coordinate systems is applied to the IMU data to compensate for the relative rotation between the vehicle and IMU, improving vehicle grade and bank measurements.

Description

Background Technology

[0001] Vehicle guidance systems typically rely on inertial measurement unit (IMU) technology to determine vehicle orientation or movement. Careful IMU positioning ensures proper operability. Original equipment manufacturers (OEMs) can specify IMU mounting accuracy within half a degree of the vehicle's body coordinate system, allowing IMU data to be compatible with the vehicle's functionality. Existing manufacturing techniques (e.g., soldering) have coplanarity limitations when applied to mounting IMUs on printed circuit boards (PCBs). Even minor installation errors in vehicle IMU mounting can lead to significant errors in the IMU data, potentially affecting vehicle performance, including causing uncomfortable or unsafe handling. Summary of the Invention

[0002] This document describes the techniques and systems for security camera-based IMU calibration of stationary systems, including vehicles. Existing vehicle camera systems are employed with enhanced security to prevent malicious attempts by hackers to force the vehicle into IMU calibration mode. IMU calibration occurs when the calibration system determines that the vehicle is parked in a controlled environment; the calibration target is positioned at different viewpoints to the vehicle camera to serve as a source of an optical pattern for encoded data. The pattern is characterized for security and for alignment functionality. The image of the calibration target can be used to infer the vehicle coordinate system from which calculations for IMU installation error compensation are performed. The relative rotation between the IMU and vehicle coordinate systems is applied to the IMU data to compensate for the relative rotation between the vehicle and the IMU, thereby improving vehicle gradient and tilt measurements.

[0003] In one example, the method includes: receiving an image of a first calibration target within the field of view of a first camera of a stationary vehicle from a first camera of the first camera, and receiving an image of a second calibration target within the field of view of a second camera of the second camera of the stationary vehicle from a second camera of the second camera. The field of view of the second camera is opposite to the field of view of the first camera. The method further includes: determining whether the first calibration target and the second calibration target are valid for calibrating an inertial measurement unit (IMU) of a stationary vehicle based on data encoded in the image of the first calibration target and data encoded in the image of the second calibration target; and in response to determining that the first calibration target and the second calibration target are valid for calibrating the IMU, establishing a vehicle coordinate system relative to ground-truth conditions outside the vehicle based on the images of the first calibration target and the second calibration target. The method further includes: outputting information such that the stationary vehicle can compensate for IMU installation errors in the vehicle by calibrating the IMU to align IMU data output from the IMU with the vehicle coordinate system.

[0004] These and other described techniques can be performed by hardware or a combination of hardware and software executing thereon. For example, a computer-readable storage medium (CRM) may have instructions stored thereon, and when those instructions are executed, configure a processor to perform the described techniques. A system may include means for performing the described techniques. A processor or processor unit may be part of a system configured to perform the methods and techniques described herein.

[0005] Through implementations of these and other examples contemplated by this disclosure, safe and accurate IMU calibration can be achieved to eliminate, for example, IMU installation irregularities in vehicles (e.g., trucks, cars), as further explained with reference to the detailed embodiments and accompanying drawings. This summary describes camera-based IMU calibration for stationary systems. This summary is not intended to identify essential features of the claimed subject matter, nor is it intended to define the scope of the claimed subject matter. Attached Figure Description

[0006] This document describes the details of IMU calibration for a stationary system based on a security camera with reference to the accompanying drawings. The drawings may use the same numbers to refer to similar features and components, and hyphen numbers to indicate variations of these similar features and components. The drawings are organized as follows:

[0007] Figure 1 A conceptual diagram of an example environment for IMU calibration based on a security camera for a stationary system is shown.

[0008] Figure 2 A conceptual diagram of a calibration unit configured to perform IMU calibration based on a security camera for a stationary system is shown.

[0009] Figure 3 A conceptual diagram of a calibration component configured to perform IMU calibration based on a security camera for a stationary system is shown; and

[0010] Figure 4 A flowchart is shown for an example procedure to perform IMU calibration based on a security camera for a stationary system. Detailed Implementation

[0011] IMU data can be used to enable certain vehicle functions. IMU positioning is mandated to be highly accurate to ensure alignment with the vehicle's coordinate system. Accurate IMU data is particularly important for achieving safe autonomous and semi-autonomous driving, as well as providing reliable active safety features. Because even minor installation errors in the vehicle's IMU installation can lead to significant errors in the IMU data, IMU calibration may be necessary after installation in and between different parts of the vehicle.

[0012] Some existing techniques achieve IMU calibration while the vehicle is in motion. However, these techniques are complex, rely on expensive computers, are time-consuming, and can sometimes even render the vehicle inoperable (e.g., while waiting for the IMU calibration to be exported). These techniques are also limited in their effectiveness in removing installation errors between the IMU and the vehicle's coordinate system.

[0013] Other techniques enable IMU calibration when the vehicle is stationary and not moving. Vehicles typically include camera systems with multiple cameras. Images captured by these systems are used to calibrate the IMU; however, this form of calibration presents security challenges. Existing camera-based IMU calibration techniques are vulnerable to hacking. Vehicle cameras can be tricked when the calibration target is outside the parked vehicle. If the vehicle's IMU calibration is compromised and the IMU is calibrated incorrectly, autonomous or semi-autonomous control may malfunction, leading to unstable or unsafe vehicle behavior.

[0014] Overview

[0015] A technique and system for safety camera-based IMU calibration for stationary systems, including vehicles, are described. Existing vehicle camera installations for enhanced safety are employed for stationary IMU calibration. This calibration is not performed every time a vehicle is parked, but only when the vehicle is parked in a controlled calibration environment (e.g., in a vehicle factory, vehicle service facility, or other controlled environment) where safety vehicle data (e.g., IMU calibration mapping) of the parked vehicle, including IMUs mounted to PCB assemblies, is accessible.

[0016] To prevent malicious attempts by hackers to force a vehicle into IMU calibration mode, IMU calibration is restricted to being performed by a computer located within the vehicle itself, or by a computer with access to the vehicle and the calibration environment. For example, consumers are not permitted to recalibrate their vehicle's IMU in their own garages. Specialized equipment acts as the calibration system and needs to communicate with vehicle components to align the IMU with the vehicle's coordinate system.

[0017] Within a controlled environment, the calibration target is positioned externally to the vehicle, at a different perspective from the vehicle's existing camera system. The calibration target may include a computer-controlled source of optical patterns sensed by the vehicle's cameras. For example, each camera may have a corresponding calibrated optical liquid crystal display (LCD) screen placed nearby. However, for enhanced security, each generated optical pattern may be individually encoded in a manner specifically designed to prevent spoofing attempts (e.g., in the case of IMU calibration intrusion). The calibration target may enable the vehicle to maintain specific functions and information (e.g., IMU calibration) that are uniquely privileged for a specific vehicle serial number, specific vehicle service facilities, etc. This may complement other identifiers or coded features that can be embedded in the optical patterns captured by the vehicle's cameras.

[0018] The camera system includes at least two cameras, typically mounted at different locations and heights; additional cameras can improve calibration results. The cameras themselves are well-spaced on the vehicle. At least one first camera (e.g., a front-facing camera) has a field of view projected from a first side (e.g., the front end) of the vehicle. At least one second camera (e.g., a rear-facing camera) has an opposing field of view projected from a second side (e.g., the rear end) of the vehicle opposite the first side. By collecting images from two or more cameras located at opposite parts, sides, or ends of the vehicle, installation errors associated with the IMU can be accurately checked and compensated for.

[0019] The calibration system obtains the corresponding height of the camera (e.g., from the vehicle controller). The calibration system uses these corresponding heights to determine the vehicle coordinate system relative to the ground truth conditions. This remains effective even when tire inflation or other vehicle components cause imperfections in the vehicle coordinate system. The transformation between the IMU coordinate system and the actual vehicle coordinate system can be applied to the IMU data because it is generated to compensate for the relative rotation that exists directly between the vehicle and the IMU. This compensation aligns the IMU data with the vehicle coordinate system, regardless of irregularities in the alignment between the IMU or the vehicle coordinate system and the ground truth conditions. The accuracy of vehicle measurements derived from the IMU readings (such as vehicle bank and slope) is improved. As an example, when two opposite (e.g., front and rear) cameras with a small relative height difference are selected, IMU mount calibration with half-degree accuracy can be achieved on a four-meter-long vehicle. For example, the relative height difference could be approximately 34.9 mm, calculated by multiplying four meters by the tangent of half a degree.

[0020] These calibration techniques can be part of a robust overall adaptive sensing system that can then automatically handle specific changes in camera position based on other detailed vehicle information, such as varying tire pressure. However, according to the described techniques, calibrating the IMU to compensate for vehicle installation errors does not require detailed vehicle information at the time of camera inspection.

[0021] Example Environment

[0022] Figure 1 A conceptual diagram of an example environment 100 for IMU calibration based on a security camera for a stationary system is shown. Environment 100 includes a vehicle 102, which is undergoing IMU installation calibration by a calibration system 104. Environment 100 represents a controlled environment (such as a vehicle factory or vehicle service station) where the components of vehicle 102 are initialized to the original baseline of the setup or reprogrammed using an official update to the initial setup. Although shown as a passenger vehicle, vehicle 102 could represent other types of vehicles, including other types of road vehicles (e.g., cars, motorcycles, buses, tractors, semi-trailers), rail vehicles (e.g., trains), water vehicles (e.g., boats), aircraft (e.g., airplanes), spacecraft (e.g., satellites), etc.

[0023] Vehicle 102 includes an IMU 106 and a camera system 108. IMU 106 generates IMU data, which is compensated using IMU calibration data 110, generated by analyzing camera images 112 captured by at least two cameras of camera system 108. For ease of description, IMU 106 is described as a single IMU. Often, vehicle 102 includes multiple IMUs. IMU 106 may include more than one IMU, each positioned at a different location on vehicle 102 to obtain accurate IMU data for different vehicle areas. Similarly, multiple IMUs of IMU 106 may be positioned at the same location on vehicle 102 to increase safety, for example, by providing redundancy in the IMU data generated on the vehicle.

[0024] Camera system 108 is used by vehicle 102 to detect objects present on or near the road from camera images 112, objects that could affect how or whether vehicle 102 can continue driving or parking. Each camera of camera system 108 is configured to capture images of at least a portion of a specific region of interest surrounding vehicle 102, referred to as the field of view. To obtain the desired field of view, components of camera system 108 may be mounted on, attached to, or integrated with any part of vehicle 102, such as the front, rear, top, bottom, corner, or side of vehicle 102, a bumper, side mirror, part of headlights and / or taillights, tailgate, trunk lid, or any other outward-facing location of vehicle 102.

[0025] Camera system 108 includes cameras 108-1, 108-2, 108-3, 108-4, and 108-n. Camera 108-1 is a camera of vehicle 102 facing a first end (e.g., the front end), and camera 108-2 is a camera of vehicle 102 facing a second, opposite end (e.g., the rear end). Camera 108-3 is a camera of vehicle 102 facing a first side (e.g., the driver's side), and camera 108-4 is a camera of vehicle 102 facing a second, opposite side (e.g., the passenger side). Camera 108-n is a centrally located camera of vehicle 102. In other examples, cameras 108-1 and 108-2, or cameras 108-3 and 108-4, are corner cameras placed at different corner locations on vehicle 102. In some examples, cameras 108-1 and 108-2 are positioned at opposite corners of vehicle 102. Similarly, cameras 108-3 and 108-4 can be positioned at different, opposite corners. For the purposes of this disclosure, an opposite corner refers to a combination of the front passenger-side corner and the rear passenger-side corner of the vehicle 102, or a combination of the front driver-side corner and the rear passenger-side corner of the vehicle 102. Opposite sides include the front and rear ends of the vehicle 102, or a combination of the passenger side or the driver side.

[0026] Camera 108-1 is configured to provide a field of view 120-1, camera 108-2 is configured to provide a field of view 120-2, and so on, with camera 108-n configured to provide a field of view 120-n (not shown). The camera system 108 may have five or more cameras or fewer cameras.

[0027] In addition to detecting objects to facilitate driving functions, camera system 108 can also be used to calibrate IMU 106, for example, to compensate for installation errors caused by misalignment of IMU 106 relative to the vehicle coordinate system of vehicle 102. However, careful selection of the cameras used to capture camera images 112 is required. To generate camera images 112 sufficient to generate IMU calibration data 110, camera system 108 includes at least two cameras with opposite fields of view. Opposite fields of view are considered to be fields of view projected from cameras positioned on opposite sides or opposite corners of vehicle 102. For example, cameras 108-1 and 108-2 project opposite fields of view, where field of view 120-1 is projected in front of vehicle 102, while field of view 120-2 is projected beyond the rear of vehicle 102. Other considerations in selecting cameras for camera system 108 to allow the calculation of IMU calibration data 110 include selecting cameras with the minimum separation distance and / or maximum separation height between them. This is in other... Figures 2-4 A more detailed description can be found in the view.

[0028] Camera system 108 may include its own computer hardware, or may be at least partially controlled by computer hardware of vehicle 102, which may be the same as or different from the vehicle components controlling IMU 106. For simplicity, these components are not shown in the figures. However, vehicle 102 includes computer hardware (e.g., controller, control unit, processor, programmable logic) configured to coordinately control camera system 108 and IMU 106 to capture and manage camera images 112 stored on vehicle 102. IMU calibration data 110 may be stored by vehicle 102 in a different CRM than the camera images 112. In other examples, vehicle 102 may maintain camera images 112 and IMU calibration data 110 on the same CRM. After generating IMU calibration data 110 based on camera images 112, the computer hardware of vehicle 102 can use the IMU calibration data 110 to calibrate IMU 106.

[0029] Vehicle 102 may include hardware or logic for generating IMU calibration data 110 on vehicle 102. However, in Figure 1 In the example, IMU calibration data 110 is generated offline using different hardware within environment 100 (e.g., outside the computer hardware of vehicle 102). The computer hardware of calibration system 104 is configured to acquire camera images 112 from vehicle 102 and generate IMU calibration data 110, which is sent back to vehicle 102 so that the computer hardware of vehicle 102 calibrates IMU 106.

[0030] IMU 106 can generate IMU data, and IMU calibration data 110 can be used by the computer hardware of vehicle 102 to compensate for the IMU data before it is output for use by vehicle functions. To this end, the computer hardware of vehicle 102 can perform operations that result in the generation of IMU data that is closely aligned with the vehicle coordinate system of vehicle 102, thereby preventing any installation errors associated with IMU 106 from damaging the IMU data. The compensated IMU data from IMU 106 is output in preparation for performing other functions of vehicle 102.

[0031] The calibration system 104 includes a calibration unit 114, a communication link 116, and multiple calibration targets 118. The communication link 116 enables data transmission between the computer hardware of the vehicle 102 and the calibration system 104. The communication link 116 can be wired or wireless. The communication link 116 can be a secure link utilizing encryption and authentication technologies to prevent malicious access to the vehicle 102 and / or the calibration system 104. When the communication link 116 is established, the vehicle 102 can send camera images 112 captured by the camera system 108 and, in response, receive IMU calibration data 110 generated by the calibration system 104.

[0032] Calibration targets 118 are referred to as calibration target 118-1, calibration target 118-2, calibration target 118-3, and calibration target 118-4. Each of the calibration targets 118 is placed in the field of view of a different camera of the camera system 108. Calibration target 118-1 is located in field of view 120-1, calibration target 118-2 is located in field of view 120-2, calibration target 118-3 is located in field of view 120-3, and calibration target 118-4 is located in field of view 120-4. Each of the calibration targets 118 is configured to display content (including optically encoded data) that enables recalibration of the IMU 106. This content is displayed by each of the calibration targets 118 at a precise position relative to the vehicle 102 and the camera of the camera system 108. This allows alignment marks within the encoded data to be used to determine the vehicle coordinate system based on the camera image 112.

[0033] Furthermore, the encoded data prevents the calibration target 118 from being spoofed, and further prevents the calibration system 104 from incorrectly generating IMU calibration data 110 for incorrect types of vehicle 102 and / or IMU 106 (e.g., brand, model, year, serial number, vehicle identification number). For example, if the features of the encoded data cannot be identified from the camera image 112 captured by the calibration target 118, the calibration system 104 can avoid communicating with the vehicle 102 and not generate IMU calibration data 110.

[0034] The calibration unit 114 of the calibration system 104 controls the content presented on the calibration target 118 and generates IMU calibration data 110 based on the camera image 112 of that content. For example, the calibration unit 114 is configured to receive the camera image 112 of the calibration target 118 captured by the camera system 108. Based on the camera image 112 received via the communication link 116, the calibration unit 114 is configured to determine the realignment of the coordinate system of the IMU 106. For example, the realignment includes rotational correction of the axes of the IMU coordinate system to align the reference system of the IMU 106 with the reference system of the vehicle 102. The calibration unit 114 causes the IMU calibration data 110 to be transmitted to the IMU 106 via the communication link 116. This results in the realignment of the coordinate system of IMU 106 so that even when vehicle 102 leaves the controlled conditions of environment 100 (e.g., is parked on a road), the future IMU data generated by IMU 106 is precisely aligned with the coordinate system of vehicle 102.

[0035] Example calibration unit

[0036] Figure 2 A conceptual diagram of a calibration unit 114-1 configured to perform IMU calibration for a stationary system based on a security camera is shown. Calibration unit 114-1 is an example of calibration unit 114 and... Figure 1 The context is described below. For example, calibration unit 114-1 is considered to be operatively and communicatively coupled to at least two of calibration targets 118, each calibration target having its own corresponding screen 200 for displaying images or other optically identifiable content. When connected to more than two calibration targets 118, calibration unit 114-1 obtains sufficient information from the content on the screen 200 to improve calculations inferred using IMU 106 (e.g., vehicle measurements). For example, both vehicle gradient calculations and vehicle tilt calculations can be improved using at least three carefully positioned calibration targets 118, or at least three images 112 from the corresponding screens 200 of two different carefully positioned calibration targets 118.

[0037] The calibration unit 114-1 includes a processor 202 and a CRM 204. The CRM 204 stores instructions and / or data that, when executed by the processor 202, enable the calibration component 206, the camera image interface 208, and the IMU data interface 210. The calibration unit 114-1 shares a communication link 212 with each of the calibration targets 118.

[0038] When an image of calibration target 118 appears in two opposite fields of view from at least two cameras of camera system 108, camera image interface 208 configures calibration unit 114-1 to receive the image of calibration target 118. For example, via communication link 116, camera image interface 208 receives one of the camera images 112 captured by camera 108-1, which belongs to calibration target 118-1 within field of view 120-1. Furthermore, as captured by camera 108-2, camera image interface 208 receives at least one more camera image 112 that is an image of calibration target 118-2 within field of view 120-2, opposite to field of view 120-1. Similarly, when calibration targets 118-3 and 118-4 appear in fields of view 120-3 and 120-4 respectively, camera image interface 208 can receive images of other calibration targets 118 when images of other calibration targets 118 appear in images captured by cameras 108-3 and 108-4.

[0039] IMU data interface 210 configures calibration unit 114-1 to transmit IMU calibration data 110 to vehicle 102 via communication link 116. For example, calibration component 206 generates IMU calibration data 110 based on camera image 112 received at camera image interface 208. IMU calibration data 110 is transmitted from IMU data interface 210 to vehicle 102 to compensate for IMU data generated by IMU 106.

[0040] The calibration component 206 is responsible for generating IMU calibration data 110 based on information analyzed from the input buffer of the camera image interface 208 to fill the output buffer of the IMU data interface 210. To generate the IMU calibration data 110, the calibration component 206 selects specific cameras to be used for IMU mounting calibration, such that specific content is displayed in their respective fields of view, and precisely aligns the IMU 106 with the vehicle coordinate system of the vehicle 102 based on data encoded in the camera image 112 captured at the calibration target 118.

[0041] To select cameras for camera system 108, calibration component 206 obtains information about the position, viewing angle, and field of view size of each camera in camera system 108 to identify suitable camera pairs with opposite or relative fields of view. Not all cameras in camera system 108 can be suitable pairs; a minimum height similarity between two cameras can be used to ensure that their relative positioning satisfactorily accounts for small installation errors (e.g., angular errors) in IMU 106 due to trigonometric mathematics. For example, calibration component 206 can obtain the relative separation height between camera 108-1 and camera 108-2. Calibration component 206 can also obtain the relative separation distance between camera 108-1 and camera 108-2. If the separation distance between cameras 108-1 and 108-2 on vehicle 102 is too close (e.g., not on opposite sides, not at opposite corners, not at opposite ends), they may not be suitable for calibrating IMU 106 to improve vehicle tilt and roll calculations. Similarly, if cameras 108-1 and 108-2 are too far apart in height relative to the vehicle coordinate system, they may not be suitable for calibrating IMU 106 to improve vehicle tilt and roll calculations. However, if the distance is sufficiently large and the height is sufficiently close, cameras 108-1 and 108-2 can be selected for calibration to improve the vehicle tilt and roll calculations of IMU 106. That is, calibration component 206 can select cameras 108-1 to 108-n from multiple different cameras 108-1 to 108-n of vehicle 102 for calibrating IMU 106 based on whether the relative separation height between cameras 108-1 and 108-2 meets a height threshold (e.g., less than one-hundredth of a meter). Alternatively, the calibration component 206 may select cameras 108-1 and 108-2 from multiple different cameras 108-1 to 108-n of the vehicle 102 to calibrate the IMU 106 based on whether the relative separation distance between cameras 108-1 and 108-2 meets a distance threshold (e.g., more than a few meters).

[0042] The calibration component 206 communicates with the calibration target 118 to display content on the corresponding screen 200, which appears in the field of view 120 and camera image 112 of the selected camera from the camera system 108. The calibration component 206 controls the presentation and format of the content displayed at each calibration target 118. Each of the corresponding screens 200 can display the same, similar, or different content. The calibration component 206 can output a first optical pattern for display on the corresponding screen 200 of calibration target 118-1. The calibration component 206 can output a second optical pattern for display on the corresponding screen 200 of calibration target 118-2. The optical pattern can differ from one calibration target 118 to the next; however, each calibration target 118 may include a portion of encoded data. When captured by cameras 108-1 and 108-2 and / or 108-3 and 108-4, the encoded data appears in the pixels of the camera image 112.

[0043] The encoded data is used for multiple purposes. The comparison of the intended encoded data with the encoded data detected in camera image 112 activates or adjusts IMU 106 calibration. Furthermore, the encoded data is used as alignment features to generate IMU calibration data 110 to calibrate IMU 106. Thus, the content appearing on the corresponding screen 200 of calibration target 118 provides for security purposes to prevent unintended or malicious calibration, as well as for functional purposes for calibrating IMU 106.

[0044] For example, as shown in the seemingly random pattern on the corresponding screen 200, the calibration target 118 can be controlled by the calibration component to display a similar pattern. Figure 2The content shown is as follows. Included in this optical pattern example is encoded data, including a reference alignment area with embedded security features, including additional noise areas. The alignment area can be used to generate IMU calibration data 110. The optical pattern can be displayed or projected by calibration target 118 onto a plane perpendicular to the horizontal plane (which includes one of the cameras 108-1, 108-2, etc. and vehicle 102). This can optimize the detection of vehicle tilt and roll angles during vehicle coordinate system alignment with the IMU coordinate system and IMU 106 calibration. Embedded security features can be used to authenticate potential calibrations being performed. For example, an embedded key can be presented by calibration target 118 as a public key for authenticating and verifying whether the calibration process is permitted to be performed. In some examples, a barcode or QR code can indicate a secure website or Internet Protocol (IP) address to allow bidirectional communication between calibration component 206 and vehicle 102. Additional security measures may include periodically changing calibration target 118 to further prevent unauthorized reprogramming or calibration of IMU 106. This ensures that IMU 106 is not recalibrated unless calibration component 206 can validate that a particular parameter is verifiable from camera image 112. In this way, calibration component 206 is less likely to be compromised by spoofing attacks that attempt to mimic encoded data located on spoofing targets positioned around vehicle 102, for example, when vehicle 102 is parked outside environment 100.

[0045] Example calibration component

[0046] Figure 3 A conceptual diagram of a calibration component 206-1 configured to perform IMU calibration for a stationary system based on a security camera is shown. Calibration component 206-1 is an example of calibration component 206 and... Figure 1 and Figure 2 The calibration component 206-1 is described in the context of [the previous sentence]. It is configured to compensate for IMU installation errors in the vehicle 102 by generating IMU calibration data 110 for calibrating IMU 106.

[0047] Continue in Figure 2In the example, in response to receiving camera images 112 of calibration targets 118-1 and 118-2 and / or 118-3 and 118-4, calibration component 206-1 checks whether to continue or avoid generating IMU calibration data 110. Based on the data encoded in the image of calibration target 118-1 and the data encoded in the image of calibration target 118-2, it is determined whether calibration targets 118-1 and 118-2 are valid for calibrating the IMU 106 of vehicle 102. In response to determining that calibration targets 118-1 and 118-2 are invalid for calibrating IMU 106 (e.g., one or more of the camera images 112 do not contain the encoded data that calibration component 206-1 expects to see), calibration component 206-1 avoids performing the function of generating IMU calibration data 110. Otherwise, for example, in response to verifying calibration targets 118-1 and 118-2 according to camera images 112, calibration component 206-1 invokes the function of generating IMU calibration data 110.

[0048] IMU calibration data 110, generated by calibration component 206-1, removes the relative rotation between IMU coordinate system 302 and vehicle coordinate system 304, unaffected by the ground truth condition 306 of environment 100. IMU calibration data 110 may include adjustments to parameters of IMU 106 to align IMU coordinate system 302 with vehicle coordinate system 304 after isolating the relative rotations of each of the IMU coordinate system 302 and vehicle coordinate system 304 relative to the ground truth condition 306 (e.g., gravity vector).

[0049] Ground truth condition 306 includes the true gravity vector of environment 100 (e.g., as measured by an altitude-calibrated IMU located outside vehicle 102 within environment 100) and is operatively coupled to calibration component 206-1. Ground truth condition 306 can be obtained by calibration component 206-1 in one or more ways (e.g., from input by a technician, from input from documentation concerning environment 100, from other equipment or calibration system 104 within environment 100).

[0050] Calibration component 206-1 obtains IMU data from IMU 106 as a reference frame to determine the IMU coordinate system 302 relative to the ground truth condition 306. Based on images of calibration targets 118-1 and 118-2, a vehicle coordinate system 304 is also established relative to the ground truth condition 306. By comparing the vehicle coordinate system 304 with the IMU coordinate system 302, the relative rotation between the two coordinate systems 304 and 302 can be calculated, isolated from the influence of the ground truth condition 306. For example, the first relative rotation between the IMU coordinate system 302 (m) and the ground truth condition 306 (g) can be calculated. Similarly, the second relative rotation between the vehicle coordinate system 304(v) and the ground truth condition 306(g) is calculated. Based on the first relative rotation Second relative rotation Calculate the third relative rotation between only the vehicle coordinate system 304(v) and the IMU coordinate system 302(m). The third relative rotation The relative rotation effect of ground truth condition 306(g) is removed, and the relative rotation between the axes of IMU coordinate system 302 and vehicle coordinate system 304 is indicated separately. IMU calibration data 110 is determined by determining adjustments to the parameters of IMU 106 that minimize the third relative rotation between IMU coordinate system 302 and vehicle coordinate system 304 as much as possible, or reduce the third relative rotation to at least an acceptable tolerance level (e.g., half a degree in any direction).

[0051] IMU calibration data 110 is provided to vehicle 102 to allow vehicle 102 to calibrate the IMU data output from IMU 106 for use in functions aligned with vehicle coordinate system 304. Using IMU data interface 210, calibration component 206-1 enables calibration system 104 to transmit IMU calibration data 110 to vehicle 102 via communication link 116. IMU calibration data 110 indicates how to compensate IMU data to align the IMU coordinate system 302 of IMU 106 with the vehicle coordinate system 304 of vehicle 102. This effectively prevents erroneous IMU data due to installation errors in the component installation of vehicle 102 and / or IMU 106. The computer hardware of vehicle 102 can compensate for IMU installation errors by reprogramming the settings and parameters of IMU 106 by calibrating IMU 106 using IMU calibration data 110. After the IMU data is aligned with the vehicle coordinate system 304 and the IMU installation error in the vehicle 102 is compensated, the vehicle 102 can use the IMU data output from the IMU 106 to perform vehicle functions that act according to the IMU data.

[0052] Example calibration process

[0053] Figure 4 A flowchart of an example process 400 for performing a security camera-based IMU calibration for a stationary system is shown. For ease of description, process 400 is primarily described in a scenario performed by a calibration system 104, which uses a communication link 116 to exchange information with a vehicle 102 that is parked and stationary in environment 100.

[0054] Processor 202, with access to CRM 204, can execute instructions to perform process 400, and to perform calibrations on components 206 and 206-1, camera image interface 208, and IMU data interface 210. In this example, the operations (also referred to as steps) of process 400 are numbered from 402 to 414. However, this numbering does not necessarily imply a specific order of operations. The steps of process 400 can be related to... Figure 4 The diagram shows different ways to rearrange, skip, repeat, or execute specific methods.

[0055] At 402, an image of a first calibration target within the field of view of a first camera on a stationary vehicle is received. At 404, an image of a second calibration target within the field of view of a second camera opposite to the field of view of the first camera is received from a second camera on the stationary vehicle. For example, calibration system 104 can select at least one pair of cameras from vehicle 102 to calibrate IMU 106. Calibration system 104 selects camera 108-1, which is a front-facing camera and has a calibration target 118-1 within a field of view 120-1. To pair with camera 108-1 for performing IMU calibration, calibration system selects camera 108-2, which is a rear-facing camera and has a calibration target 118-1 within a field of view 120-2 opposite to the field of view 120-1.

[0056] In some cases, more than two calibration targets 118 and / or more than two cameras of camera system 108 are used. Using at least two images, more preferably three images, to determine the slope and tilt angle, the vehicle coordinate system 304 can be accurately determined. For example, at 402, an image of a third calibration target within the field of view of the first camera can be received by processor 202. An image of a fourth calibration target within the field of view of the third calibration target or the second camera can be received by processor 202. Using images of two, three, or even more calibration targets allows for the calculation of the vehicle coordinate system 304 with greater accuracy and / or higher confidence due to the addition of alignment reference indicators and coded data available from the additional camera images.

[0057] At 406, data encoded in the image of the first calibration target and data encoded in the image of the second calibration target are identified. For example, the calibration system 104 communicates via communication link 116 so that the vehicle 102 controls the camera system 108 to capture camera images 112. Within the camera images 112 captured by cameras 108-1 and 108-2, the calibration system 104 identifies features within the optical patterns of calibration targets 118-1 and 118-2, which include encoded data for determining the vehicle coordinate system 304.

[0058] For example, camera 108-1 may have a first viewpoint to calibration target 118-1, and camera 108-2 may have a second viewpoint to calibration target 118-2. Depending on the viewpoints used by calibration targets 118-1 and 118-2, the optical pattern will appear different in camera image 112. The first and second viewpoints may be the same or different. In either case, the calibration system sets the first and second viewpoints to precise angles that result in unique, corresponding optical patterns in camera image 112. By setting the viewpoints to precise angles, deception of the calibration target and calibration system 104 can be prevented, and a precise vehicle coordinate system can be inferred as the reference frame used to calibrate IMU 106.

[0059] At 408, based on data encoded in the image of the first calibration target and data encoded in the image of the second calibration target, it is determined whether the first and second calibration targets are valid for calibrating the IMU of a stationary object. For example, the calibration system 104 compares the corresponding optical pattern at each of the calibration targets 118-1 and 118-2 with the corresponding expected pattern of that particular calibration target to determine whether that particular calibration target is valid for calibrating the IMU. If the calibration targets 118-1 and 118-2 are not precisely positioned near the cameras 108-1 and 108-2, where a precise viewpoint is applied, the optical pattern 112 within the camera image 112 may not include identifiable features that can be used to determine the vehicle coordinate system.

[0060] Within camera image 112, encoded data may indicate brand, model, serial number, vehicle identification number, or other identifiers for vehicle 102 and the IMU 106 being calibrated. If the encoded data within camera image 112 is not verified as authentic by calibration system 104, IMU 106 is not recalibrated. For example, if at 408 it is determined that the first and second calibration targets are valid for calibrating the IMU, a "yes" branch is adopted and process 400 continues to step 410. Otherwise, if it is determined that the first and second calibration targets are invalid for calibrating the IMU, a "no" branch is adopted and process 400 continues to step 414. The calibration of IMU 106 can be adjusted in one or more ways, such as (e.g., based on whether the information received from the controller of vehicle 102 meets the entry conditions for IMU calibration).

[0061] The computer hardware of vehicle 102 sends information to calibration system 104 indicating whether vehicle 102 is in a state acceptable for IMU calibration. When vehicle 102 is in environment 100, it is in a controlled environment in which it can be placed in a special programming mode that cannot be accessed from outside environment 100 or without the use of special equipment (such as calibration system 104). Information about the vehicle's status (e.g., whether vehicle 102 is parked, whether the controller and / or vehicle 102 is in programming mode, whether vehicle 102 is in environment 100) can indicate to calibration system 104 whether IMU 106 is safe for calibration. Calibration system 104 is authorized when the information meets the entry conditions for IMU calibration. Thus, calibration system 104 avoids generating IMU calibration data 110 for calibrating IMU 106 unless doing so is absolutely safe. Based on the information received from the controller of vehicle 102, and in response to the authorized calibration of IMU 106, IMU 106 can be calibrated to align the IMU data output from IMU 106 with the vehicle coordinate system 304.

[0062] At 410, based on the images of the first and second calibration targets, a vehicle coordinate system relative to the ground truth condition outside the vehicle is determined. This includes, for example, a second relative rotation between the vehicle coordinate system 304 and the ground truth condition 306. In addition, the first relative rotation between the IMU coordinate system 302 and the ground truth condition 306 is also calculated. Based on the first relative rotation Second relative rotation Calculate the third relative rotation between the vehicle coordinate system 304 and the IMU coordinate system 302. Third relative rotation A direct relationship is provided between IMU 106 and the vehicle reference system, regardless of the deviation of each coordinate system relative to the ground truth condition 306. The vehicle reference system (i.e., vehicle coordinate system 304) is related to the vehicle chassis (e.g., vehicle frame). Therefore, vehicle coordinate system 304 is independent of changes in the position of the vehicle components due to factors such as variable suspension, uneven tire inflation, variable torque, or other variables that do not cause relative installation errors between IMU 106 and the vehicle chassis and vehicle coordinate system 304. Relative rotations between the axes of IMU coordinate system 302 and vehicle coordinate system 304 are calculated to generate IMU calibration data 110, which, when applied by vehicle 102 to the parameters of IMU 106, ensures that the IMU data to be output is aligned with vehicle coordinate system 304, regardless of installation errors in the installation of IMU 106.

[0063] At 412, output information enables a stationary vehicle to calibrate the IMU, thereby aligning the IMU data output from the IMU with the vehicle's coordinate system by compensating for IMU installation errors within the vehicle. For example, calibration system 104 is configured to output information to vehicle 102. This output includes IMU calibration data 110, which is transmitted from calibration system 104 to vehicle 102, where the computer hardware of vehicle 102 adjusts IMU 106 to be rotated and aligned with vehicle coordinate system 304. Thus, during vehicle operation, the IMU data output from IMU 106 is compensated for alignment with vehicle coordinate system 304, allowing vehicle 102 to directly use the IMU data with the assurance that it is error-free. This is correct even if the installation of IMU 106 exceeds the tolerances specified for the PCB assembly that mounts IMU 106 to vehicle 102.

[0064] Further examples

[0065] The following provides some additional examples of IMU calibration based on a security camera for stationary systems.

[0066] Example 1: A method comprising: receiving from a first camera of a stationary vehicle an image of a first calibration target within the field of view of the first camera; receiving from a second camera of the stationary vehicle an image of a second calibration target within the field of view of the second camera, the field of view of the second camera being opposite to the field of view of the first camera; determining, based on encoded data in the image of the first calibration target and encoded data in the image of the second calibration target, whether the first calibration target and the second calibration target are valid for calibrating an inertial measurement unit (IMU) of the stationary vehicle; in response to determining that the first calibration target and the second calibration target are valid for calibrating the IMU, establishing a vehicle coordinate system based on the images of the first calibration target and the second calibration target, relative to ground truth conditions outside the vehicle; and outputting information such that the stationary vehicle can compensate for IMU installation errors in the vehicle by calibrating the IMU to align IMU data output from the IMU with the vehicle coordinate system.

[0067] Example 2: Any of the methods in the preceding examples, wherein the first camera includes a front-facing camera of the vehicle, and the second camera includes a rear-facing camera of the vehicle.

[0068] Example 3: Any of the methods in the preceding examples, wherein the first camera includes a camera of the vehicle facing a first side, and the second camera includes a camera of the vehicle facing the opposite side.

[0069] Example 4: Any of the methods in the preceding examples, wherein the first camera includes a first corner front camera of the vehicle, and the second camera includes a relative corner rear camera of the vehicle.

[0070] Example 5: Any of the methods in the foregoing examples further includes: receiving an image of a third calibration target within the field of view of a third camera of a stationary vehicle; receiving an image of a fourth calibration target within the field of view of a fourth camera of a stationary vehicle, the field of view of the fourth camera being opposite to the field of view of the third camera; determining, based on the encoded data in the image of the third calibration target and the encoded data in the image of the fourth calibration target, whether the third calibration target and the fourth calibration target are valid for the calibration IMU; and in response to determining that the third calibration target and the fourth calibration target are valid for the calibration IMU, further establishing a vehicle coordinate system based on the images of the third calibration target and the fourth calibration target.

[0071] Example 6: The method of any of the foregoing examples, wherein calibrating the IMU to align the IMU data output from the IMU with the vehicle coordinate system includes: receiving information from the controller of the stationary vehicle indicating that the stationary vehicle is parked in a controlled calibration environment; authorizing calibration of the IMU based on the information received from the controller; and calibrating the IMU to align the IMU data output from the IMU with the vehicle coordinate system in response to the authorization of calibration of the IMU.

[0072] Example 7: A method of any of the foregoing examples, wherein a first camera has a first viewpoint to a first calibration target and a second camera has a second viewpoint to a second calibration target, the method further comprising: setting the first and second viewpoints to precise angles resulting in a unique corresponding optical pattern, the unique corresponding optical pattern comprising encoded data at each of the first and second calibration targets; and comparing the corresponding optical pattern at each of the first and second calibration targets with a corresponding expected pattern for the particular calibration target to determine whether the particular calibration target is valid for calibrating the IMU.

[0073] Example 8: Any of the methods in the preceding examples, wherein the first calibration target and the second calibration target include corresponding screens for displaying the first calibration target and the second calibration target.

[0074] Example 9: The method of Example 8 further includes: outputting a first optical pattern for display on a corresponding screen of a first calibration target, the first optical pattern including encoded data appearing in an image of the first calibration target; and outputting a second optical pattern for display on a corresponding screen of a second calibration target, the second optical pattern including encoded data appearing in an image of the second calibration target.

[0075] Example 10: Any of the methods in the foregoing examples further includes: obtaining the relative separation height between the first camera and the second camera; and selecting the first camera and the second camera from a plurality of different cameras of the vehicle for use in calibrating the IMU based on whether the relative separation height between the first camera and the second camera meets a height threshold.

[0076] Example 11: Any of the methods in the foregoing examples further includes: obtaining the relative separation distance between the first camera and the second camera; and further selecting the first camera and the second camera for calibrating the IMU based on whether the relative separation distance between the first camera and the second camera satisfies a distance threshold for correcting the error of the installation angle of the IMU relative to a stationary vehicle.

[0077] Example 12: Any of the methods in the preceding examples, wherein outputting information to enable a stationary vehicle to calibrate the IMU to align the IMU data output from the IMU with the vehicle coordinate system includes: sending information to the stationary vehicle for compensating the IMU data for relative rotation between the vehicle coordinate system and the IMU coordinate system.

[0078] Example 13: Any of the methods in the preceding examples, wherein after the IMU data is aligned with the vehicle coordinate system and IMU installation errors in the vehicle are compensated, the IMU data is output from the IMU for use in performing the function of a stationary vehicle acting according to the IMU data.

[0079] Example 14: A system including a processor configured to perform any of the methods in the foregoing examples.

[0080] Example 15: A computer-readable storage medium including instructions that, when executed, cause a processor to perform any of the methods of the foregoing examples.

[0081] Example 16: A system comprising means for performing any of the methods described in the preceding examples.

[0082] Conclusion

[0083] While various embodiments of the present disclosure have been described in the foregoing description and illustrated in the accompanying drawings, it should be understood that the present disclosure is not limited thereto, but can be practiced in various ways within the scope of the following claims. It will be apparent from the foregoing description that various modifications can be made without departing from the scope of the present disclosure as defined by the appended claims. Problems associated with IMU mounting alignment can occur in other systems besides transportation systems, not just transportation. Therefore, although described as for improving IMU alignment in transportation vehicles, the techniques described above can also be adapted and applied to other problems to effectively calibrate the IMU to align with the main coordinate systems of other objects, devices, and systems.

[0084] Unless the context clearly specifies otherwise, the use of "or" and grammatically related terms indicates an unrestricted, non-exclusive alternative. As used herein, the phrase referring to "at least one" of a list of items means any combination of those items, including a single member. For example, "at least one of a, b, or c" is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

Claims

1. A method for a stationary system, comprising: receiving, from a first camera of a stationary vehicle, an image of a first calibration target within a first camera field of view of the first camera; receiving, from a second camera of the stationary vehicle, an image of a second calibration target within a second camera field of view of the second camera, the second camera field of view being opposite the first camera field of view; determining, based on encoded data in the image of the first calibration target and encoded data in the image of the second calibration target, whether the first and second calibration targets are valid for calibrating an inertial measurement unit (IMU) of the stationary vehicle; in response to determining that the first and second calibration targets are valid for calibrating the IMU, establishing, based on the image of the first calibration target and the image of the second calibration target, a vehicle coordinate system relative to ground truth conditions external to the vehicle; and outputting information to enable the stationary vehicle to compensate for IMU installation errors in the vehicle by calibrating the IMU to align IMU data output from the IMU with the vehicle coordinate system.

2. The method of claim 1, wherein, the first camera comprises a front-facing camera of the vehicle and the second camera comprises a rear-facing camera of the vehicle.

3. The method of claim 1, wherein, the first camera comprises a camera facing a first side of the vehicle and the second camera comprises a camera facing an opposite side of the vehicle.

4. The method of claim 1, wherein, the first camera comprises a first corner front-facing camera of the vehicle and the second camera comprises an opposite corner rear-facing camera of the vehicle.

5. The method of any of claims 1-4, further comprising: receiving, from a third camera of the stationary vehicle, an image of a third calibration target within a third camera field of view; receiving, from a fourth camera of the stationary vehicle, an image of a fourth calibration target within a fourth camera field of view, the fourth camera field of view being opposite the third camera field of view; determining, based on encoded data in the image of the third calibration target and encoded data in the image of the fourth calibration target, whether the third and fourth calibration targets are valid for calibrating the IMU; and in response to determining that the third and fourth calibration targets are valid for calibrating the IMU, establishing the vehicle coordinate system further based on the image of the third calibration target and the image of the fourth calibration target.

6. The method of any one of claims 1-4, wherein, calibrating the IMU to align the IMU data output from the IMU with the vehicle coordinate system comprises: receiving, from a controller of the stationary vehicle, information indicating that the stationary vehicle is parked in a controlled calibration environment; authorizing calibration of the IMU based on the information received from the controller; and in response to authorizing calibration of the IMU, calibrating the IMU to align the IMU data output from the IMU with the vehicle coordinate system.

7. The method of any of claims 1-4, wherein: the first camera has a first view relative to the first calibration target; the second camera has a second view relative to the second calibration target; and the method further comprises: setting the first view and the second view to precise angles that result in respective unique optical patterns that include encoded data at each of the first calibration target and the second calibration target; and comparing the respective optical pattern at each of the first calibration target and the second calibration target to a corresponding expected pattern for that calibration target to determine whether that calibration target is valid for calibrating the IMU.

8. The method of any one of claims 1-4, wherein, the first calibration target and the second calibration target include respective screens that display the first calibration target and the second calibration target.

9. The method of claim 8, further comprising: outputting a first optical pattern for display on the respective screen of the first calibration target, the first optical pattern including encoded data that appears in an image of the first calibration target; and outputting a second optical pattern for display on the respective screen of the second calibration target, the second optical pattern including encoded data that appears in an image of the second calibration target.

10. The method of any of claims 1-4, further comprising: obtaining a relative separation height between the first camera and the second camera; and selecting the first camera and the second camera from a plurality of different cameras of the vehicle for calibrating the IMU based on whether the relative separation height between the first camera and the second camera satisfies a height threshold.

11. The method of claim 10, further comprising: obtaining a relative separation distance between the first camera and the second camera; and selecting the first camera and the second camera for calibrating the IMU further based on whether the relative separation distance between the first camera and the second camera satisfies a distance threshold for correcting an error in a mounting angle of the IMU relative to the stationary vehicle.

12. The method of any one of claims 1-4, wherein, outputting information to enable the stationary vehicle to calibrate the IMU to align the IMU data output from the IMU with the vehicle coordinate system includes sending information to the stationary vehicle for compensating the IMU data for a relative rotation between the vehicle coordinate system and a coordinate system of the IMU.

13. The method of any one of claims 1-4, wherein, after the IMU data is aligned with the vehicle coordinate system and compensated for the IMU mounting error in the vehicle, the IMU data is output from the IMU for performing a function of the stationary vehicle that acts in accordance with the IMU data.

14. A system comprising a processor configured for performing the method of any of claims 1-8, 10, or 12.

15. The system of claim 14, wherein, the first camera comprises a front-facing camera of the vehicle and the second camera comprises a rear-facing camera of the vehicle. the first camera comprises a front-facing camera of the vehicle and the second camera comprises a rear-facing camera of the vehicle.

16. The system of claim 14, wherein, The first camera comprises a first side-facing camera of the vehicle and the second camera comprises an opposite side-facing camera of the vehicle.

17. The system of claim 14, wherein, The first camera comprises a first corner front-facing camera of the vehicle and the second camera comprises an opposite corner rear-facing camera of the vehicle.

18. The system of any one of claims 14-17, wherein, The processor is further configured to display the first and second calibration targets on respective screens.

19. The system of claim 18, wherein, The processor is further configured to: output a first optical pattern for display on a respective screen of the first calibration target, the first optical pattern comprising encoded data appearing in an image of the first calibration target; and output a second optical pattern for display on a respective screen of the second calibration target, the second optical pattern comprising encoded data appearing in an image of the second calibration target.

20. A computer-readable storage medium comprising instructions that, when executed, cause a processor to perform the method of any one of claims 1-8, 10, or 12.

Images