METHOD FOR CONTROLLING A VEHICLE AND ACCORDINGLY CONTROLLED VEHICLE

The method improves sensor calibration in autonomous vehicles by filtering non-horizontal edges and using transformation parameters to enhance data correlation, addressing calibration inconsistencies and anomalies for accurate navigation and operation.

DE102018116107B4Active Publication Date: 2024-12-05GM GLOBAL TECHNOLOGY OPERATIONS LLC
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
DE102018116107
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Priority Date
2017-07-06
Filing Date
2018-07-03
Publication Date
2024-12-05
Estimated Expiration
2038-07-03

AI Technical Summary

Technical Problem

Existing autonomous vehicles face challenges in accurately calibrating relationships between different types of sensors due to varying sampling rates and viewpoints, leading to inconsistencies in data correlation and potential calibration anomalies, which affect the vehicle's ability to accurately navigate and operate autonomously.

Method used

A method for calibrating vehicle sensors by detecting a steady state, obtaining and filtering image and distance measurement data, and determining transformation parameters to convert between sensor frames, focusing on non-horizontal edges to reduce noise and improve accuracy, allowing for real-time calibration and anomaly detection.

Benefits of technology

Enhances the accuracy of data correlation and object detection in autonomous vehicles by reducing noise and enabling fast, on-the-fly calibration of multiple sensors, while detecting and mitigating calibration anomalies for improved autonomous operation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

Vehicle, including: a first device on board the vehicle that provides initial data; a second device on board the vehicle that provides second data; one or more sensors on board the vehicle; one or more actuators on board the vehicle; and A controller which, through a processor, detects a steady state based on the output of one or more sensors, receives a first set of the first data from the first device during the steady state, filters out the horizontal boundary regions from the first set, resulting in a filtered set of the first data, receives a second set of the second data during the steady state, determines one or more transformation parameter values ​​based on a relationship between the second set and the filtered set, and subsequently operates the one or more actuators in the vehicle autonomously in a manner influenced by the one or more transformation parameter values.
Need to check novelty before this filing date? Find Prior Art

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The subject matter described herein relates to the subject matter described in US patent application US 2019 / 0 012 808 A1, filed concurrently herewith. TECHNICAL FIELD

[0002] The present invention relates generally to motor vehicles and to a method for autonomous control of onboard components using calibrated relationships between different types of data representative of the operating environment, and to a correspondingly controlled vehicle.

[0003] DE 10 2016 117 710 A1 discloses a method in which a fusion system comprising a radar system and a vision system is calibrated. Points of a target object detected in the radar system are mapped to the coordinate system of the vision system using a mapping function and thus calibrated. The system targets points of a point cloud within an area of ​​interest encompassing the target object in the sensor field of view.

[0004] Furthermore, DE 10 2016 009 327 A1 discloses a method for calibrating a camera. It involves projecting a pattern using a lidar sensor, determining distance values ​​for this pattern, and having the camera record this pattern. The camera can then be calibrated based on these distance values. BACKGROUND

[0005] An autonomous vehicle is a vehicle capable of sensing its surroundings and navigating with little or no user input. An autonomous vehicle senses its surroundings using sensor devices such as radar, lidar, image sensors, and the like. The autonomous vehicle system further utilizes information from global positioning systems (GPS), navigation systems, vehicle-to-vehicle communications, vehicle infrastructure technologies, and / or wired systems to navigate the vehicle.

[0006] Vehicle automation was categorized by numerical levels from zero, corresponding to no automation with full human control, to five, corresponding to full automation without human control. Various automated driver assistance systems, such as cruise control, adaptive cruise control, and parking assistance systems, correspond to lower levels of automation, while true "driverless" vehicles correspond to higher levels of automation.

[0007] To achieve high-level automation, vehicles are often equipped with an increasing number of different types of devices for analyzing the environment around the vehicle, such as cameras or other imaging devices for capturing images of the environment, radar or other ranging devices for measuring or detecting features in the environment, and the like. In practice, the various on-board devices are located at different locations on board the vehicle and typically run at different sampling or refresh rates, resulting in different types of data being collected at different times from different viewpoints or perspectives.Accordingly, it is desirable to provide systems and methods for calibrating relationships between different devices to improve the ability to accurately determine correlations between different data types, which in turn facilitates the extraction of more accurate information about the environment from the data and / or the assignment of attributes to objects or features within the environment, thus improving autonomous vehicle control. Furthermore, it is also desirable to detect potential calibration anomalies that may affect calibration and implement appropriate remedial action(s) to mitigate any potential calibration impairment.Other desirable functions and features will become apparent from the following detailed description and the appended claims, taken in conjunction with the accompanying drawings, as well as the technical field and background explained above. SUMMARY

[0008] Systems and methods are provided for controlling a vehicle. In one embodiment, a method for controlling a vehicle includes detecting a steady state of the vehicle based on the output of an in-vehicle sensor system, and in response to the steady state, obtaining a first image from an in-vehicle imaging device during the steady state and obtaining a first set of ranging data corresponding to a field of view of the imaging device from an in-vehicle ranging device during the steady state.The method continues by filtering horizontal edge regions from the first image and determining one or more transformation parameter values ​​for converting between a reference frame associated with the imaging device and a reference frame associated with the ranging device based at least in part on the filtered image data and the first set of ranging data.The method then obtains a second image from the imaging device, obtains a second set of range measurement data corresponding to the field of view of the imaging device at a time associated with the second image from the imaging device, assigns one or more attributes to the second image based at least in part on the second set of range measurement data and the one or more transformation parameter values, and autonomously operates one or more actuators in the vehicle in a manner influenced by the one or more attributes associated with the second image.

[0009] In another embodiment, an autonomous vehicle is provided, including: a first in-vehicle device providing first data, a second in-vehicle device providing second data, one or more sensors in the vehicle, one or more actuators in the vehicle, and a controller configured, through a processor, to detect a steady state based on the output of the one or more sensors, obtain a first set of first data from the first device during the steady state, filter the horizontal edge regions from the first set, resulting in a filtered set of first data, obtain a second set of second data during the steady state,determines one or more transformation parameter values ​​based on a relationship between the second set and the filtered set and then autonomously operates the one or more actuators in the vehicle in a manner influenced by the one or more transformation parameter values.

[0010] In another embodiment, a method for controlling a vehicle includes: detecting, by an in-vehicle controller, a steady-state condition of the vehicle in response to the steady-state condition, obtaining one or more images acquired during the steady-state condition by one or more in-vehicle imaging devices and a first set of range measurement data output during the steady-state condition by a first in-vehicle range measurement device, wherein one or more subsets of the first set of range measurement data are correlated with one or more vertical edge regions in a first image of the one or more images, determining one or more parameter values ​​associated with the conversion between a first frame of reference associated with a first imaging device and a second frame of reference,which is associated with the first imaging device based on a relationship between the one or more subsets of the first set of range measurement data and the one or more vertical edge regions of the first image, and after determining the one or more parameter values: obtaining an updated image from the first imaging device, obtaining an updated subset of range measurement data corresponding to the updated image, determining depths associated with the updated image based on the updated subset of range measurement data and one or more parameter values, and determining a command to operate one or more actuators in the vehicle in a manner influenced by the depths associated with the updated image. DESCRIPTION OF THE DRAWINGS

[0011] The exemplary embodiments are described below in conjunction with the following drawings, wherein like reference numerals denote like elements, and wherein: Fig. 1 is a functional block diagram illustrating an autonomous vehicle according to various embodiments; Fig. 2 shows a functional block diagram illustrating a transport system with one or more autonomous vehicles from Fig. 1 according to various embodiments; Fig. 3 is a schematic block diagram of an automated drive system (ADS) that is suitable for implementation by the vehicle in accordance with several embodiments. Fig. 1 is suitable; Fig. 4 is an illustration of an arrangement of a plurality of imaging devices and a plurality of distance measuring devices on board a vehicle, such as the autonomous vehicle in Fig. 1 in accordance with various embodiments; Fig. 5 is a block diagram of a process module for implementation on board the vehicle in Fig. 1 or Fig. 4 in accordance with one or more exemplary embodiments; Fig. 6 is a flowchart illustrating a data collection process for obtaining data sets for subsequent calibration or verification according to one or more exemplary embodiments; Fig. Figure 7 illustrates an exemplary image of an environment representative of a recorded field of view of a camera located in the vehicle; Fig. 8 illustrates an exemplary filtered version of the image of Fig. 7, which describes the filtering of horizontal edges and the retention of non-horizontal edges of image data intended for use in connection with the data collection process of Fig. 6 are suitable; Fig. Figure 9 illustrates an exemplary representation of uncalibrated distance measurement data in relation to an underlying image of a recorded field of view from an in-vehicle camera, which is used for correlation in connection with the data collection process of Fig. 6 is suitable; Fig. Figure 10 is a flowchart illustrating a calibration process for calibrating a pair of instruments using data sets collected according to the data collection process of Fig. 6 according to one or more exemplary embodiments; Fig. 11 illustrates an exemplary image of an environment that is suitable for a recorded field of view of a camera located in the vehicle and overlying distance measurement data of a camera connected to the camera according to the calibration process of Fig. 10, which is representative of a distance measuring device calibrated according to one or more exemplary embodiments; and Fig. 12 is a flowchart illustrating a calibration verification process for validating an existing calibration of a pair of devices and detecting anomalous conditions with respect to one or more of the devices, according to one or more example embodiments. DETAILED DESCRIPTION

[0012] The following detailed description is provided for illustrative purposes only. Furthermore, there is no intention to be bound by any theory expressed or implied in the foregoing technical field, background, brief summary, or the following detailed description. As used herein, the term "module" refers to any hardware, software, firmware product, electronic control component, processing logic, and / or processor device, individually or in combination, including, but not limited to, an application-specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or grouped), and memory that executes one or more software or firmware programs, combinational logic circuitry, and / or other suitable components that provide the described functionality.

[0013] Embodiments of the present invention may be described herein as functional and / or logical block components and various processing steps. It should be noted that such block components may be constructed from any number of hardware, software, and / or firmware components configured to perform the required functions. For example, an embodiment of the present invention of a system or component may employ various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, look-up tables, or the like, capable of performing multiple functions under the control of one or more microprocessors or other control devices.Furthermore, those skilled in the art will recognize that the exemplary embodiments of the present invention may be used in connection with any number of systems, and that the system described herein represents merely one exemplary embodiment of the present invention.

[0014] For the sake of brevity, conventional techniques associated with signal processing, data transmission, signaling, imaging, ranging, synchronization, calibration, controllers, and other functional aspects of the systems (and the individual control elements of the systems) may not be described in detail herein. Furthermore, the connecting lines depicted in the various figures are intended to represent exemplary functional relationships and / or physical connections between the various elements. It should be noted that many alternative or additional functional relationships or physical connections may be present in an embodiment of the present invention.

[0015] In one or more of the exemplary embodiments described herein, a vehicle capable of a certain level of autonomous operation includes a number of different devices that capture images or otherwise generate data depicting a scene or environment near the vehicle from different perspectives and at different sampling or refresh rates. Image data from an onboard imaging device may be correlated with ranging data from a onboard ranging device based on the relationship between the line of sight and / or field of view of the imaging device and the time instances at which the ranging device captures or samples the field of view associated with the imaging device.In exemplary embodiments, the relationship between a particular imaging device and a particular ranging device is calibrated to obtain conversion parameter values ​​that can be used to convert the ranging data from the ranging device into an image captured by the imaging device, and vice versa. For example, coordinate locations or positions can be assigned to specific parts of the image using the ranging data and the conversion parameter values ​​to translate coordinates or points within the reference frame of the ranging data into the reference frame of the image data. In this way, relative depths or other dimensional or physical features can be assigned to regions within the image, which can then be used to detect or locate specific object types within the image.to detect, to determine distances between the objects and the vehicle, and to determine commands to steer or otherwise operate the vehicle in a manner influenced by the type and relative position of the objects in the vehicle's environment. For purposes of illustration, the subject matter herein may be described primarily in the context of the imaging devices used as cameras and the ranging devices as light detection and ranging (lidar) devices. Nevertheless, it should be appreciated that the subject matter described herein is not necessarily limited to use with optical cameras and lidars, and alternative embodiments may utilize other types of cameras or imaging devices, radar or other types of ranging devices or sensor assemblies (e.g., sonar, light pattern sensors, parallax algorithms, etc.).) or any number of combinations of imaging devices and distance measuring devices.

[0016] As will be explained below, especially in connection with the Fig. 4-11, in one exemplary embodiment, a camera is calibrated with a lidar by determining or selecting the data generated or otherwise output by each device during a period when the vehicle is stationary. In this regard, a steady state of the vehicle may be detected or otherwise identified before the calibration data collection is performed. An image captured by the camera while the vehicle is stationary and a set of ranging data output by the lidar while the vehicle is stationary are also collected, wherein the set of ranging data corresponds to the field of view of the camera. In one or more exemplary embodiments, wherein the camera captures multiple images while stationary, the image closest to the time at which the lidar scans the camera's field of view is used for calibration.

[0017] Once an image and corresponding rangefinding data have been obtained, the image data is processed or otherwise analyzed to detect or otherwise identify horizontal edges within the image. For example, one or more image processing techniques may be performed on the image data to identify edges within the image, which are filtered such that substantially horizontal edges, which manifest as discontinuities or changes in image characteristics when moving vertically along a vertical scan line or pixel column from one pixel row to another, are generally excluded or otherwise removed from further consideration.In an exemplary embodiment, Gaussian smoothing is performed on the image data to reduce noise, and then a Sobel kernel is used to quantify the magnitude of the difference or discontinuity between pixels with respect to image intensity. In this regard, the Sobel kernel is configured to quantify or detect intensity changes along rows of pixels, as well as to quantify or detect intensity changes in diagonal directions (albeit with a reduced weighting), while substantially filtering out intensity changes along columns of pixels. In this regard, the Sobel kernel function filters out horizontal edges (e.g., discontinuities along a vertical scan line or column of pixels) and isolates, emphasizes, or identifies only those edges within the image that are substantially vertically or diagonally aligned with a horizontal reference plane of the image (e.g.,discontinuities along a horizontal row of pixels or diagonally from one pixel to another).

[0018] The non-horizontal edge regions are selected or otherwise retained for correlation with range data corresponding to those edge regions. In this regard, using the position of the edge regions within the image, the positions of the corresponding regions within the ranging data can be determined or otherwise identified based on the relationship between the camera's field of view and the lidar's orientations, where ranging data for the field of view is obtained.In general, a non-horizontal edge in an image manifests itself as a change in intensity or other property of the image upon horizontal movement from one pixel to the left or right to another pixel across a substantially horizontal row of pixels, or upon diagonal movement from one pixel to another, whereas a non-horizontal edge in the ranging data manifests itself as changes in depth or distance upon horizontal movement from one point to another point across a series of points within the lidar point cloud dataset.

[0019] After identifying subsets of the range measurement data corresponding to the subsets of the edge regions of the imaging data, the subsets of the non-horizontal edge regions of the imaging data and the subsets of the non-horizontal edge regions of the range measurement data can be stored or otherwise retained and then used to calibrate the relationship between the images acquired by that camera and the lidar scans of that camera's field of view. For example, translational and rotational conversion parameter values ​​for converting range data to the image reference frame and vice versa can be determined by solving for parameter values ​​that minimize a cost function associated with the conversion.Then, for a subsequent set of range measurement data generated by the lidar for the camera field of view, the conversion parameter values ​​can be used to convert the lidar points from the lidar reference frame to the image reference frame. Once the lidar points have been converted to the image reference frame, the lidar points can be projected into pixel positions using the camera's internal parameters (e.g., focal length, principal point, distortion parameters, etc.), enabling the mapping of depth or other physical or dimensional attributes to pixel positions (or ranges or sets thereof).In this regard, object detection, classification, and analysis based on the image data can be improved using depth, distance, dimensions, or other information derived from the calibrated and correlated lidar data. This in turn improves the autonomous operation of the vehicle, as the commands for operating the vehicle are influenced by these objects in the vehicle's environment. For example, more precise correlation of pixels of image data with projected lidar points can determine a more precise bounding box of an object in the image, which in turn improves object classification when a neural network is applied to the image data contained in the bounding box to determine what the object is (e.g., a car, a pedestrian, a traffic sign, etc.).

[0020] In some embodiments, multiple image pairs and corresponding data sets are used to calibrate a camera-lidar pair. For example, after obtaining a first set of non-horizontal edge regions of image data and ranging data for a first vehicle position, the vehicle may be driven or otherwise moved until a subsequent steady-state condition is detected. Once the steady-state condition is detected, an updated set of ranging data acquired by the lidar while scanning the camera's field of view while the vehicle was stationary is determined and paired with an updated image acquired by the camera while the vehicle is scanning the camera's field of view near the time the lidar scanned the camera's field of view.Here, too, non-horizontal edge regions of image data are identified in the updated image and linked to corresponding non-horizontal edge regions of the updated range measurement data. The paired non-horizontal edge regions of the updated data can also be saved or otherwise maintained and used to calibrate the camera-lidar pair, for example, by minimizing the cost function, taking into account both the association between the originally paired non-horizontal edge region data from the original vehicle position and the association between the updated pair of non-horizontal edge region data from the updated vehicle position.

[0021] Furthermore, in some embodiments, the paired non-horizontal edge region data may be stored or otherwise maintained in association with the vehicle position at which the corresponding image and range measurement data were obtained, thereby accounting for changes in vehicle position between paired camera-lidar datasets. For example, in some embodiments, a minimum threshold difference exists between the vehicle positions associated with the camera-lidar datasets to ensure that the underlying data of the camera-lidar datasets is sufficiently dissimilar (e.g., to avoid the use of substantially redundant data in calibration).In still further embodiments, the cost function may consider the relative differences between paired camera-lidar datasets, for example, to improve calibration accuracy when portions of the images overlap or capture a common object or area, or the like. For example, when substantially the same scene or environment around the vehicle is captured from different vehicle positions or orientations (e.g., when the difference in vehicle position between paired camera-lidar datasets is less than the minimum threshold difference), calibration may be improved by incorporating or otherwise considering the vehicle position and the corresponding camera and lidar positions and orientations in the cost function.

[0022] Filtering out horizontal edges and limiting calibration to substantially non-horizontal peripheries of image and ranging data reduces the effects of noise in the ranging data and can also reduce the computational resources and time required to calibrate a camera-lidar pair. In this regard, multiple camera-lidar pairs can be calibrated essentially simultaneously. For example, if the vehicle has multiple different cameras with different fields of view that are stationary for a sufficient period of time to sample one complete rotation of the lidar around the vehicle, each of the cameras can be calibrated to the lidar in a similar manner, as described above for each camera (e.g.,By correlating non-horizontal edge regions of an image acquired by the respective camera while the vehicle is moving with corresponding regions in the portion of the ranging data corresponding to the lidar scanning the respective camera's field of view. By filtering out horizontal edges, calibration can be performed faster and with less noise, enabling the on-the-fly calibration of multiple cameras with respect to a lidar device. For example, in one embodiment, each time the vehicle starts moving, the onboard cameras can first be calibrated with the lidar device while the vehicle is initially in a stationary position, and then the calibrations can be updated the next time the vehicle is stationary.This allows calibrations for multiple cameras related to a specific lidar device to be updated regularly without the need for special calibration procedures or maneuvers.

[0023] As described below mainly in connection with Fig. 12, in the exemplary embodiments described herein, the continued validity or accuracy of the existing calibration of each camera-lidar pair may be periodically checked during autonomous operation of the vehicle to detect or otherwise identify an anomalous condition related to one of the cameras or the lidar and to take one or more remedial actions to mitigate the anomalous condition. For example, the cost function and conversion parameter values ​​associated with a camera-lidar pair may be re-evaluated using a most recently obtained pairing of image and range measurement data to ensure that the conversion parameter values ​​from the previous calibration minimize the cost function.In response to determining that the values ​​of the conversion parameters no longer minimize the cost function, edges within this final pairing of the image and range measurement data are analyzed to identify or otherwise determine whether an anomalous condition exists with respect to one of the devices. For example, non-horizontal edges within the image may be identified, and the presence of corresponding non-horizontal edges in the lidar point cloud data may be verified or confirmed. Conversely, non-horizontal edges within the lidar point cloud data may be identified, and the presence of corresponding non-horizontal edges in the image data may be verified or confirmed.It should be noted, however, that the calibration verification procedures described herein are not necessarily limited to discontinuities and may, in practice, use other types of alignment algorithms or correlations that do not necessarily relate to horizontal discontinuities.

[0024] In response to the absence of a non-horizontal edge region in the image data corresponding to a non-horizontal edge region in the lidar point cloud data, that camera may be flagged or otherwise indicated as potentially anomalous. For example, a user notification may be generated or otherwise provided to inform a vehicle operator, owner, or passenger of a potential abnormal condition related to the camera. Subsequently, manual inspection or maintenance may be performed to restore camera operation (e.g., cleaning the lens, securing electrical connections, or the like). In addition, one or more additional remedial actions may be performed, such as temporarily suspending use of the camera or reducing the influence of that camera's image data relative to others (e.g.,by adjusting one or more weighting factors in an algorithm). Similarly, in response to the absence of a non-horizontal edge region in the lidar data that corresponds to a non-horizontal edge region in the image data, the lidar may be flagged or otherwise identified as potentially anomalous, and one or more additional mitigation actions may also be performed (e.g., temporarily suspending use of the lidar if another lidar or radar is present in the vehicle).

[0025] With reference to Fig. 1, an autonomous vehicle control system, generally depicted at 100, is associated with a vehicle 10 according to various embodiments. In general, the control system 100 determines a movement plan for autonomous operation of the vehicle 10 along a route that takes into account the objects or obstacles detected by the sensors 28, 40, as described in more detail below. As described above and in more detail below, a control module onboard the autonomous vehicle 10 calibrates different types of sensors 28, 40 with respect to one another such that the data from these different sensors 28, 40 onboard the vehicle are spatially associated or otherwise mutually associated based on the calibration, which improves the detection of objects, their classification, and the resulting autonomous operation of the vehicle 10.

[0026] As in Fig. 1, the vehicle 10 generally includes a chassis 12, a body 14, front wheels 16, and rear wheels 18. The body 14 is disposed on the chassis 12 and substantially encloses the other components of the vehicle 10. The body 14 and the chassis 12 may together form a frame. The wheels 16-18 are each pivotally connected to the chassis 12 near a respective corner of the body 14.

[0027] In various embodiments, the vehicle 10 is an autonomous vehicle, and the control system 100 is integrated into the autonomous vehicle 10 (hereinafter referred to as the autonomous vehicle 10). The autonomous vehicle 10 is, for example, a vehicle that is automatically controlled to transport passengers from one location to another. The vehicle 10 is shown as a passenger car in the illustrated embodiment, but it should be noted that any other vehicle, including motorcycles, trucks, sport utility vehicles (SUVs), recreational vehicles (RVs), vessels, aircraft, etc., may be used. In one exemplary embodiment, the autonomous vehicle 10 is a so-called Level Four or Level Five automation system.A Level Four system indicates "high automation," referring to the driving mode-specific performance by an automated driving system of all aspects of the dynamic driving task, even when a human driver fails to respond appropriately to a request. A Level Five system indicates "full automation," referring to the full-time performance of an automated driving system of all aspects of the dynamic driving task under all road and environmental conditions manageable by a human driver.

[0028] As illustrated, the autonomous vehicle 10 generally includes a propulsion system 20, a transmission system 22, a steering system 24, a braking system 26, a sensor system 28, an actuator system 30, at least one data storage device 32, at least one controller 34, and a communication system 36. The propulsion system 20, in various embodiments, may include an internal combustion engine, an electric machine such as a traction motor, and / or a fuel cell propulsion system. The transmission system 22 is configured to transfer power from the propulsion system 20 to the vehicle wheels 16-18 according to selectable gear ratios. According to various embodiments, the transmission system 22 may include a stepped-ratio automatic transmission, a continuously variable transmission, or other suitable transmission. The braking system 26 is configured to provide braking torque to the vehicle wheels 16-18.The braking system 26 may, in various embodiments, include friction brakes, brake-by-wire, a regenerative braking system such as an electric machine, and / or other suitable braking systems. The steering system 24 influences the position of the vehicle wheels 16-18. While illustrated as a steering wheel in some embodiments within the scope of the present invention, the steering system 24 may not include a steering wheel.

[0029] The sensor system 28 includes one or more sensor devices 40a-40n that sense observable conditions of the external environment and / or the internal environment of the autonomous vehicle 10. The sensor devices 40a-40n may include, but are not limited to, radars, lidars, global positioning systems, optical cameras, thermal imaging cameras, ultrasonic sensors, and / or other sensors. The actuator system 30 includes one or more actuator devices 42a-42n that control one or more vehicle features, such as, but are not limited to, the drive system 20, the transmission system 22, the steering system 24, and the braking system 26. In various embodiments, the vehicle features may further include, but are not limited to, interior and / or exterior vehicle features, such as doors, a trunk, and interior features such as air, music, lighting, etc. (unnumbered).

[0030] The data storage device 32 stores data for use in automatically controlling the autonomous vehicle 10. In various embodiments, the data storage device 32 stores defined maps of the navigable environment. In various embodiments, the defined maps are predefined and retrieved from a remote system (in further detail with respect to Fig. 2). For example, the defined maps may be assembled by the remote system and communicated to the autonomous vehicle 10 (wirelessly and / or wired) and stored in the data storage device 32. As can be seen, the data storage device 32 may be part of the controller 34, separate from the controller 34, or part of the controller 34 and part of a separate system.

[0031] The controller 34 includes at least one processor 44 and a computer-readable storage device or media 46. The processor 44 may be a custom-made or off-the-shelf processor, a central processing unit (CPU), a graphics processing unit (GPU) among multiple processors connected to the controller 34, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, a combination thereof, or generally any device for executing instructions. The computer-readable storage device or media 46 may include volatile and non-volatile memory in read-only memory (ROM), random access memory (RAM), and keep-alive memory (KAM). KAM is persistent or non-volatile memory that can be used to store various operating variables while the processor 44 is powered off.The computer-readable storage device or media 46 may be implemented using any of a number of known storage devices, such as PROMs (programmable read-only memory), EPROMs (electrical PROMs), EEPROMs (electrically erasable PROMs), flash memory, or any other electrical, magnetic, optical, or combination storage devices capable of storing data, some of which represent executable instructions used by the controller 34 in controlling the autonomous vehicle 10.

[0032] The instructions may include one or more separate programs, each comprising an ordered list of executable instructions for implementing logical functions. The instructions, when executed by processor 44, receive and process signals from sensor system 28, perform logic, calculations, methods, and / or algorithms to automatically control the components of autonomous vehicle 10, and generate control signals to actuator system 30 to automatically control the components of autonomous vehicle 10 based on the logic, calculations, methods, and / or algorithms. Although in Fig. 1 only one controller 34 is shown, embodiments of the autonomous vehicle 10 may include any number of controllers 34 that communicate and cooperate via a suitable communication medium or combination of communication media to process the sensor signals, perform logic, calculations, methods and / or algorithms, and generate control signals to automatically control the functions of the autonomous vehicle 10.

[0033] In various embodiments, one or more instructions of controller 34 are integrated into control system 100 (e.g., in data storage element 46) and, when executed by processor 44, result in processor 44 detecting or identifying a steady state of vehicle 10 based on output data from one or more vehicle sensors 40 (e.g., a speed sensor, an attitude sensor, or the like), acquiring data acquired or generated by imaging and ranging devices 40 while vehicle 10 is in the steady state, and calibrating the relationships between the respective imaging device 40 and the respective ranging devices 40 using non-horizontal edges, as further described below.Processor 44 can then use the calibrations to establish correlations and transformations between the data sets to assign attributes from one data set to another, thereby improving object detection, object classification, object prediction, and the like. The resulting objects and their classification and predicted behavior influence the schedules for autonomous control of vehicle 10, which in turn influence commands generated or otherwise provided by processor 44 for controlling actuators 42.

[0034] Furthermore, in one or more embodiments, the processor 44 detects or identifies a steady-state condition of the vehicle 10 and periodically checks or validates the accuracy of the existing calibrations. In response to an anomalous condition with respect to a calibration, the processor 44 determines whether the imaging device 40 or the ranging device 40 has an anomalous condition with non-horizontal edges. In the absence of an anomalous condition with respect to the imaging device 40 or the ranging device 40, the processor 44 may recalibrate the pair and resume the aforementioned operation of the vehicle 10 using the updated calibration.

[0035] As in Fig. 1, the communication system 36 is configured, in exemplary embodiments, to wirelessly transmit information to and from other entities 48, such as, but not limited to, other vehicles (“V2V” communication), infrastructure (“V2I” communication), remote systems, and / or personal devices (relating to Fig. 2). In an exemplary embodiment, the wireless communication system 36 is configured to communicate over a wireless local area network (WLAN) using the IEEE 802.11 standard, via Bluetooth, or by means of mobile data communication. However, additional or alternative communication methods, such as a dedicated short-range communication (DSRC) channel, are also contemplated within the scope of the present invention. DSRC channels refer to one-way or two-way short- to mid-range radio communication channels specifically developed for the automotive industry and a corresponding set of protocols and standards.

[0036] With further reference to Fig. 2 in various embodiments, the autonomous vehicle 10 described with reference to Fig. 1, may be suitable for use as part of a taxi or shuttle service within a specific geographic area (e.g., a city, a school or business campus, a shopping mall, an amusement park, an event center, or the like). For example, the autonomous vehicle 10 may be associated with an autonomous vehicle-based transportation system. Fig. Figure 2 illustrates an exemplary embodiment of an operating environment, shown generally at 50, including an autonomous vehicle-based transportation system 52, which, as described with reference to Fig. 1, is associated with one or more instances of autonomous vehicles 10a-10n. In various embodiments, the operating environment 50 further includes one or more user devices 54 that communicate with the autonomous vehicle 10 and / or the remote transportation system 52 via a communications network 56.

[0037] The communications network 56 supports communication between devices, systems, and components supported by the operating environment 50 (e.g., via physical communication links and / or wireless communication links). For example, the communications network 56 may include a wireless carrier system 60, such as a cellular phone system, including a plurality of cellular towers (not shown), one or more mobile switching centers (MSCs) (not shown), and any other network components necessary to connect the wireless carrier system 60 to the fixed network. Each cellular tower includes transmit and receive antennas and a base station, with the base stations of various cellular towers connected to the MSCs, either directly or through intermediate devices, such as a base station controller.The wireless carrier system 60 may implement any suitable communications technology, for example, digital technologies such as CDMA (e.g., CDMA2000), LTE (e.g., 4G LTE or 5G LTE), GSM / GPRS, or other current or emerging wireless technologies. Other cell tower / base station / MSC arrangements are possible and could be used with the wireless carrier system 60. For example, the base station and cell tower could be co-located or remote from each other, each base station could serve a single cell tower, a single base station could serve multiple cell towers, or multiple base stations could be coupled to a single MSC, to name a few of the possible arrangements.

[0038] In addition to using the wireless carrier system 60, a second wireless carrier system in the form of a satellite communications system 64 may be used to provide unidirectional or bidirectional communications with the autonomous vehicle 10a-10n. This may be accomplished using one or more communications satellites (not shown) and an uplink transmitting station (not shown). Unidirectional communications may include, for example, satellite radio services, where programmed content data (news, music, etc.) is received from the transmitting station, packaged for upload, and then transmitted to the satellite, which broadcasts the programming to subscribers. Bidirectional communications may include, for example, satellite telephone services that use the satellite to relay telephone communications between the vehicle 10 and the station.Satellite telephony can be used either in addition to or instead of the mobile operator system 60.

[0039] A fixed-line communication system 62 may include a conventional fixed-line telecommunications network connected to one or more landline telephones and connecting the wireless carrier system 60 to the remote transport system 52. For example, the fixed-line communication system 62 may be a public switched telephone network (PSTN) such as those used to provide wired telephony, packet-switched data communications, and the Internet infrastructure. One or more segments of the fixed-line communication system 62 could be implemented using a standard wired network, a fiber optic or other optical network, a cable network, power lines, other wireless networks such as wireless local area networks (WLANs) or networks providing wireless broadband access (BWA), or any combination thereof.Furthermore, the remote transport system 52 need not be connected via the fixed network communication system 62, but could include radiotelephone equipment so that it can communicate directly with a wireless network, such as the wireless carrier system 60.

[0040] Although in Fig. 2 illustrates only one user device 54, embodiments of the operating environment 50 may support any number of user devices 54, including multiple user devices 54 owned, operated, or otherwise used by a person. Each user device 54 supported by the operating environment 50 may be implemented using any suitable hardware platform. In this regard, the user device 54 may be implemented in a common form factor, including: a desktop computer; a mobile computer (e.g., a tablet computer, a laptop computer, or a netbook computer); a smartphone; a video game device; a digital media player; a component of a home entertainment device; a digital camera or video camera; a wearable computing device (e.g., a smart watch, smart glasses, smart clothing); or the like.Each user device 54 supported by operating environment 50 is embodied as a computer-implemented or computer-based device having the hardware, software, firmware, and / or processing logic necessary to perform the various techniques and methods described herein. For example, user device 54 includes a microprocessor in the form of a programmable device that includes one or more instructions stored in an internal memory structure and is operable to receive binary inputs and generate binary outputs. In some embodiments, user device 54 includes a GPS module that can receive GPS satellite signals and generate GPS coordinates based on those signals.In further embodiments, user device 54 includes cellular communication functionality such that the device performs voice and / or data communications over communication network 56 using one or more cellular communication protocols, as explained herein. In various embodiments, user device 54 includes a visual display, such as a graphical touchscreen display or other display.

[0041] The remote transportation system 52 includes one or more backend server systems that may be cloud-based, network-based, or resident at the particular campus or geographic location served by the transportation system 52. The remote transportation system 52 may be staffed by a live advisor, an automated advisor, or a combination of both. The remote transportation system 52 may communicate with the user devices 54 and the autonomous vehicles 10a-10n to schedule trips, relocate autonomous vehicles 10a-10n, and the like. In various embodiments, the remote transportation system 52 stores account information, such as subscriber authentication data, vehicle license plates, profile records, behavior patterns, and other related subscriber information.

[0042] According to a typical use case workflow, a registered user of the remote transportation system 52 may create a ride request via the user device 54. The ride request typically specifies the passenger's desired pickup location (or current GPS location), the desired destination (which may identify a predefined vehicle stop and / or a user-defined passenger destination), and a pickup time. The remote transportation system 52 receives the ride request, processes the request, and dispatches a selected one of the autonomous vehicles 10a-10n (when and if available) to pick up the passenger at the designated pickup location and time. The transportation system 52 may also generate and send an appropriately configured confirmation message or notification to the user device 54 to notify the passenger that a vehicle is en route.

[0043] As can be seen, the subject matter disclosed herein provides certain enhanced features and functionality for what may be considered a standard or baseline autonomous vehicle 10 and / or an autonomous vehicle-based transportation system 52. To this end, an autonomous vehicle-based transportation system may be modified, enhanced, or otherwise supplemented to provide the additional functionality described in more detail below.

[0044] Referring to Fig. 3, the controller 34 implements an autonomous driving system (ADS - Autonomous Driving System) 70, as in Fig. 3. That is, suitable software and / or hardware components of the controller 34 (e.g., the processor 44 and the computer-readable storage medium 46) are used to provide an autonomous driving system 70 used in conjunction with the vehicle 10, e.g., to automatically control various actuators 30 and thus control the vehicle's acceleration, steering, and braking behavior without human intervention.

[0045] In various embodiments, the instructions of the autonomous driving system 70 may be structured according to function or system. For example, the autonomous driving system 70 may, as shown in Fig. 3, include a sensor fusion system 74, a positioning system 76, a steering system 78, and a vehicle control system 80. As can be appreciated, in various embodiments, the instructions may be divided into any number of systems (e.g., combined, further subdivided, etc.), as the invention is not limited to the present examples.

[0046] In various embodiments, the sensor fusion system 74 synthesizes and processes sensor data and predicts the presence, location, classification, and / or history of objects and features of the environment of the vehicle 10. In various embodiments, the sensor fusion system 74 may include information from multiple sensors, including, but not limited to, cameras, lidars, radars, and / or any number of other types of sensors. In one or more exemplary embodiments described herein, the sensor fusion system 74 supports the system described hereinafter primarily in the context of Fig. 6-12. In exemplary embodiments, the sensor fusion system 74 correlates image data with lidar point cloud data using calibrated conversion parameter values ​​associated with the camera-lidar pair that is the respective source of the image and lidar point cloud data to assign depth to the image data, identify objects in one or more of the image data, or otherwise synthesize associated image data and other data, such as lidar data. In other words, the sensor output from the sensor fusion system 74 provided to the vehicle control system 80 (e.g., indicating detected objects and / or their locations relative to the vehicle 10) represents the calibrations and associations between the camera images, lidar point cloud data, and the like.

[0047] The positioning system 76 processes sensor data along with other data to determine a position (e.g., a local position relative to a map, an exact position relative to the lane of a road, vehicle direction, speed, etc.) of the vehicle 10 relative to its surroundings. The guidance system 78 processes sensor data along with other data to determine a route for the vehicle 10 to follow. The vehicle control system 80 generates control signals for controlling the vehicle 10 according to the determined route.

[0048] In various embodiments, the controller 34 implements machine learning techniques to support the functionality of the controller 34, such as feature detection / classification, obstacle mitigation, route traversal, mapping, sensor integration, ground truth determination, and the like.

[0049] Fig. 4 shows an exemplary vehicle 400 including a plurality of cameras 402 distributed throughout the vehicle 400, as well as a plurality of distance measuring devices 404 also distributed throughout the vehicle 400. The cameras 402 are arranged at different locations and provide different fields of view that capture different parts of the environment near the vehicle 400. For example, a first camera 402 is positioned on the front left (or driver's side) of the vehicle 400, and its field of view is oriented 45° counterclockwise relative to the longitudinal axis of the vehicle 400 in the direction of travel; another camera 402 may be positioned on the front right (or passenger side) of the vehicle 400, with its field of view 45° clockwise relative to the longitudinal axis of the vehicle 400.Additional cameras 402 are positioned on the rear left and right sides of the vehicle 400 and similarly oriented away from the longitudinal axis at 45° relative to the vehicle's longitudinal axis, along with cameras 402 on the left and right sides of the vehicle 400 and away from the longitudinal axis and perpendicular to the vehicle's longitudinal axis. The illustrated embodiment further includes a pair of cameras 402 at or near the vehicle's longitudinal axis and oriented to capture a forward-facing field of view along a line of sight substantially parallel to the vehicle's longitudinal axis.

[0050] In exemplary embodiments, the cameras 402 have viewing angles, focal lengths, and other attributes that may differ from those of the one or more cameras 402. For example, the cameras 402 on the right and left sides of the vehicle may have a viewing angle that is greater than that of the cameras 402 positioned at the front left, front right, rear left, or rear right of the vehicle. In some embodiments, the viewing angle of the cameras 402 is selected such that the field of view of different cameras 402 at least partially overlaps to ensure camera coverage at specific locations or orientations relative to the vehicle 400.

[0051] One or more ranging devices 404 are arranged at different locations within the vehicle 400, and in one embodiment, these are arranged symmetrically about the longitudinal axis of the vehicle 400 to achieve parallax. In the exemplary embodiments described herein, the ranging devices 404 are embodied as lidar devices. In this regard, each ranging device 404 may include one or more lasers, scanning components, optical assemblies, photodetectors, and other components configured to horizontally and rotationally scan the environment in the vicinity of the vehicle 400 at a specific angular frequency or rotational speed. For example, in one embodiment, each lidar device 404 is configured to rotate horizontally at a frequency of 10 Hertz (Hz) and to scan 360°.As used herein, a lidar scan should be understood to refer to a single revolution of a lidar device 404.

[0052] In the exemplary embodiments described herein, the cameras 402 autonomously and automatically capture images at a particular frequency, where the frequency or rate at which the cameras 402 capture images is greater than the angular frequency of the lidar devices 404. In one embodiment, the frequency or sampling rate of the cameras 402 is at least twice the angular frequency of the lidar device 404. For example, the cameras 402 may capture new image data corresponding to their respective fields of view at a rate of 30 Hz, while the lidar device 404 scans at a rate of 10 Hz and automatically provides updated data. Thus, each camera 402 can sample multiple images per lidar scan and capture the images at different times, regardless of the orientation of the lidar device 404 or the angular position within the scan.In such embodiments, an image from each respective camera 402 temporarily associated with the lidar point cloud data from a particular lidar scan may be selected or identified based on the relative time difference between the time the image was acquired and the time the angular position of the lidar scan coincides with the line of sight of a lidar device 404 and the line of sight or angle of view of the respective camera 402. In further embodiments, images obtained during a lidar scan may be fused or otherwise combined to obtain a representative image corresponding to the lidar scan.In still further embodiments, the cameras 402 may be triggered or otherwise controlled to capture images at the time instance during a lidar scan when the angular position of the lidar scan coincides with the line of sight of a lidar device 404 that is aligned with the line of sight or angle of view of the corresponding camera 402. In this regard, it should be understood that there are numerous different methods for synchronizing or temporarily associating different data sets, and that the subject matter described herein is not intended to be limited to any particular manner for identifying or associating image data of an area related to a lidar scan of that area or corresponding lidar data representing that area.

[0053] Fig. Figure 5 shows an embodiment of a processing module 500 (or control module) that can be implemented by the controller or integrated into the controller 34, the processor 44, and / or the sensor fusion system 74. The image processing module 500 is coupled to a camera 502 (e.g., one of the cameras 402) onboard the vehicle and a lidar device 504 (e.g., one of the lidar devices 404). It should be noted that, although Fig. 5 illustrates a single camera 502, however, in practice, the image processing module 500 may be coupled to multiple cameras 40, 402 on board a vehicle 10, 400 to temporally relate images from multiple cameras 40, 402 to the lidar point cloud data of a single scan of the lidar device 504 and to perform the calibration processes described herein with respect to multiple cameras 40, 402 substantially simultaneously or in parallel.Furthermore, the image processing module 500 can be coupled to additional lidar devices 40, 504 located in the vehicle 10, 400 to temporally link and correlate various images from the onboard cameras 40, 402, 502 with the lidar point cloud data from the scans of various lidar devices 40, 404, 504 and thereby perform the calibration-related processes described herein with respect to multiple cameras 40, 402 and multiple lidar devices 40, 404, 504 substantially simultaneously or in parallel.

[0054] The image processing module 500, in one or more embodiments, is configured to buffer, store, or otherwise maintain image data related to one or more images (or samples) acquired via lidar scan from the camera 502, and may select or otherwise identify a acquired image temporarily associated with the lidar device 504 that is within the field of view of the camera 502 (e.g., based on the timestamps associated with the images in the buffer).Based on a signal received from the lidar device 504 or an indication indicating when the lidar device 504 is aligned in its initial or reference orientation within a scan, for example, the angular frequency of the lidar device 504 and then the angle or orientation of the camera 502 is displayed, wherein the image processing module 500 can calculate a scan time at which the line of sight of the lidar device 504 is aligned with the camera 502, also referred to below as the lidar scan time, and can select the image data set with a timestamp closest to the lidar scan time to associate or correlate lidar data representing the camera's field of view.

[0055] The image processing module 500 is coupled to the lidar device 504 to retrieve or otherwise obtain the lidar point cloud data from a lidar scan, and then correlates at least a portion of the lidar point cloud data with the temporally associated image data. For example, the image processing module 500 may select or otherwise identify the subset of the lidar point cloud data corresponding to the lidar device 504 intersecting the viewpoint of the camera 502.

[0056] As described in more detail below, in some embodiments, the image processing module 500 calibrates the relationship between the camera 502 and the lidar device 504 and stores or maintains calibration data representative of the relationship in the data storage element 506. In this regard, the calibration data may include values ​​for variables of a function for spatially translating from the lidar point cloud reference frame to the data reference frame, or vice versa. The image processing module 500 may then use the calibration data when correlating subsequent image data with point cloud data to project the lidar data onto the image data and to assign depth, distance, or other magnitude properties to the image data.In this way, the correlation between image data and point cloud data can be used by the image processing module 500 to detect objects for later classification and prediction and to output these pre-processed to one or more further object or obstacle analysis modules, vehicle control modules, or the like.

[0057] With reference now to Fig. 6 and with further reference to the Fig. 1-5 illustrates a flowchart of various embodiments of a data collection process 600 incorporated into a controller 34 in the control system 100 in Fig. 1 and supports the ADS 70. In this regard, the data collection process 600 can be performed or implemented by one or more control modules 34, 44, 500 on board a vehicle 10, 400. As can be seen from the invention, the sequence of operations within the method is not limited to the Fig. 6, but may, where applicable, be performed in one or more different orders according to the present invention. In addition, one or more of the sequences described in the context of Fig. 6 can be omitted from a practical embodiment of the data collection process 600, while still retaining the intended overall functionality. In various embodiments, the data collection process 600 can be executed based on one or more predetermined events and / or continuously during operation of the vehicle 10, 400.

[0058] In exemplary embodiments, the data collection process 600 may be performed to collect pairs of image data and lidar data for calibrating the relationship between a corresponding camera and lidar device, or alternatively, for ongoing verification of the validity or accuracy of the calibration. The illustrated data collection process 600 begins by receiving or otherwise obtaining data from one or more vehicle sensors 602 onboard the vehicle and by detecting or otherwise identifying when the vehicle is in a steady state at 604. For example, a control module 34, 500 onboard the vehicle may be coupled to one or more vehicle sensors 40 onboard the vehicle and produce corresponding outputs that may be analyzed or monitored to determine whether the vehicle 10, 400 is in a steady state, such asSpeed ​​sensors, position sensors, throttle sensors, brake sensors, and the like. In this regard, the data collection process 600 can detect a steady state of the vehicle 10, 400 when the rotational speed or velocity of the vehicle 10, 400 is zero, the acceleration of the vehicle 10, 400 is zero, and the position of the vehicle 10, 400 is stable or unchanged. In this way, the data collection process 600 ensures that the acquired data sets used for calibration will not contain noise or other artifacts that could be introduced by the motion of the vehicle 10, 400.

[0059] In one or more embodiments, the data collection process 600 also verifies or confirms that one or more calibration criteria are met based on the current vehicle position at 606. In this regard, some embodiments of the data collection process 600 may impose one or more minimum requirements that the current vehicle position must differ from the vehicle positions associated with previously acquired data sets to ensure that the scene or environment represented by the acquired data sets is not substantially similar to that of the previously acquired data sets to ensure sufficiently different depth discontinuities.For example, the data collection process 600 may require that the distance between the current vehicle position and a previous vehicle position associated with one or more previously acquired data sets is greater than a minimum threshold distance, that the difference between the current orientation or direction associated with the current vehicle position and the orientations or directions associated with one or more previously acquired data sets is greater than a minimum threshold difference, that a time difference between the time associated with the current vehicle position and a previously acquired data set, or a combination thereof, is greater than a minimum time difference.In this way, the data collection process 600 can ensure that the vehicle position changes such that the scene or environment represented by the current data sets is likely to differ from that of the previously acquired data sets, or that the time difference between the data sets is likely to result in differences between the scene or environment represented by the current data sets and those of the previously acquired data sets.

[0060] In some embodiments, the data collection process 600 may use a map database to determine, based on the current vehicle position relative to the map, a likely environment around the vehicle (e.g., a busy intersection versus a parking lot, or the like), and then adjust the calibration criteria accordingly based on the type of environment or the characteristics of the environment. For example, if the vehicle is stationary at a traffic light at a busy intersection, the data collection process 600 may require only a minimal time difference (e.g., 1 second) between data sets without requiring a change in the vehicle's position. Conversely, if the stationary condition occurs in a parking lot, the data collection process 600 may require a minimal change in the vehicle's position between data sets.

[0061] If the data collection process 600 detects a steady state that meets the applicable calibration criteria, the data collection process 600 continues by receiving or otherwise obtaining lidar data corresponding to the field of view of the camera to be calibrated acquired during the steady state at 608, and receiving or otherwise obtaining image data corresponding to the field of view of the camera to be calibrated acquired during the steady state and temporally linked to the lidar data at 610. In this regard, the control module 34, 500 receives a subset of lidar data corresponding to the lidar device 504 traversing the field of view of the camera 502 and then also selects the image data from the camera 502 that most closely temporally corresponds to the instance(s) in the steady state in which the lidar device 504 traversed the field of view or line of sight of the camera 502.

[0062] Once the temporal association of the lidar and image data acquired during the steady state is achieved, the data collection process 600 continues by filtering or otherwise processing the image data to identify non-horizontal edge regions at 612 and thereby filter out or otherwise exclude horizontal edge regions from further analysis or use during calibration. For example, Fig. 7 shows an exemplary unfiltered image 700 captured by a camera of an environment in the camera's field of view, and Fig. 8 shows an exemplary filtered version of the image 800 that includes only non-horizontal edge regions 802. To obtain a filtered image 800 that filters or removes horizontal edges and retains only non-horizontal edge regions 802, the control module 34 converts 500 pixels of the original image 700 to a grayscale intensity value, performs Gaussian smoothing of the grayscale intensity values ​​to reduce noise, and then applies a filter, such as a Sobel filter, that identifies the regions where intensity values ​​change horizontally along a row of pixels or diagonally between pixels, while filtering out vertical intensity changes along a column of pixels (i.e., horizontal edges).Thereafter, the control module 34, 500 identifies the remaining regions 802 of pixels in which sufficient intensity changes in the horizontal or diagonal direction are present as non-horizontal edge regions of image data, which are used for later calibration and / or calibration verification.

[0063] Again referring to Fig. 6. After the non-horizontal edge regions of the image data have been identified, the data collection process 600 continues by selecting or otherwise identifying corresponding regions in the lidar data for association with the non-horizontal edge regions of the image data at 614. In this regard, the control module 34 selects or otherwise identifies 500 subsets of the lidar point cloud when depth or distance changes occur during horizontal movement along the horizontal reference plane of the lidar scan.In an exemplary embodiment, the lidar data point cloud includes thirty-two horizontal scan lines (or rows), with the control module 34, 500 selecting or otherwise identifying a point within each row whose depth discontinuity is greatest relative to neighboring points of that row, and then retaining these thirty-two points and their relative position within the point cloud and depth information as the regions in the lidar data for mapping to the non-horizontal edge regions of the image data. In this regard, limiting the number of points used for calibration reduces computation time and improves calibration accuracy by selecting points with larger depth discontinuities.

[0064] Fig. 9 shows a graphical representation of the uncalibrated lidar scan data 900 of an environment corresponding to a field of view of a camera and superimposed on a sample image 902 of the environment, wherein the shading or fill pattern of the lidar scan data 900 generally corresponds to the measured depth or distance from the lidar device. In this regard, Fig. 9 illustrates an uncalibrated relationship between the lidar scan data 900 and the example image 902, such that depth discontinuities along rows corresponding to the horizontal reference plane of the lidar device 504 are not necessarily aligned with corresponding features or elements in the image 902. The lidar scan data 900 includes a plurality of rows of points having different depths or distances relative to the associated lidar device 504. To identify non-horizontal edge regions for associating image data, the control module 34, 500 may analyze the rows of lidar data 900 and identify points where depths or distances change by more than a threshold upon horizontal movement across points within a row, and select or otherwise identify these points as a non-horizontal edge.

[0065] As described above, among the non-horizontal edge points within the array, the point with the greatest difference in depth or distance from its neighboring points in the array can be identified or selected as the point within that array with the greatest depth difference, which is retained for correlation with non-horizontal edge regions of image data. In this regard, among points that have substantially equal depth differences within the same array, the point with the minimum depth or distance from the lidar device (e.g., the point closest to the vehicle) can be selected and retained for later analysis, as edges closer to the vehicle are likely to be measured or observed with greater accuracy and precision than distant objects.For example, a point 910 corresponding to the left edge of the column may be selected as the point with the largest depth discontinuity within its respective row of the lidar scan data 900, while points 912, 914 corresponding to the right edge of the column may be selected as the points with the largest depth discontinuity within their respective rows of the lidar scan data 900.The substantially vertical left edge of the column identified within a captured image of the environment 902 may be correlated with point 910 (and possibly other points from the lower rows of the lidar data 900), and the substantially vertical right edge of the column identified within the captured image of the environment 902 may be correlated with points 912, 914 (and possibly other points from the upper rows of the lidar data 900) to determine transformation parameters for converting between points 910, 912, 914 within the lidar data 900 and the corresponding pixel positions for the respective edges of the column in the image data, as described herein.

[0066] Again referring to Fig. 6, after the data collection process 600 has identified the non-horizontal edge data subsets and the non-horizontal edge data subsets, the data collection process 600 stores or maintains the correlated non-horizontal edge data subsets in association with each other and the current vehicle at 616. In this regard, the control module 34, 500 stores in the data storage element 506 the pixel positions for the non-horizontal edge regions of the image (identified at 612) and the corresponding image data, the three-dimensional points (relative location and depth or distance) within the point cloud selected based on depth discontinuities along a horizontal scan line (identified at 614), and information characterizing or quantifying the current vehicle position (e.g., position coordinates, heading, altitude, and the like) and the associated time.

[0067] The data collection process 600 may be repeated to continuously capture or otherwise identify the steady-state vehicle condition of a vehicle that meets the calibration criteria, as well as to obtain corresponding data sets for calibrating on-board devices or verifying existing calibrations. For example, the data collection process 600 may be performed to initially obtain at least a minimum number of paired data sets prior to calibrating the relationship between a camera 502 and a lidar device 504. In one embodiment, after a vehicle 10, 400 is commissioned, the data collection process 600 is performed multiple times to obtain data sets for calibrating the relationship between a camera 502 and a lidar device 504 for subsequent vehicle operation.After calibration, the data collection process 600 may be repeated to obtain data sets under steady-state vehicle conditions and to verify the existing calibration substantially in real time. Subsequently, if the calibration is not verified, the data collection process 600 may be performed to obtain additional data sets to recalibrate the relationship between a camera 502 and a lidar device 504, as described in more detail below.

[0068] With reference to Fig. 10 and with continued reference to the Fig. 1-6, in exemplary embodiments, a calibration process 1000 is embedded in a control module 34, 500 in a vehicle 10, 400 for calibrating relationships between the cameras 502 and the lidar devices 504 in the vehicle 10, 400 using one or more data sets acquired according to the data collection process 600. That is, the calibration process 1000 is not necessarily limited to implementation in conjunction with the data collection process 600, and the order of operations is not limited to that in Fig. 10, but may, where appropriate, be performed in one or more different orders according to the present invention. Furthermore, one or more of the sequences described in the context of Fig. 10 can be omitted from a practical embodiment of the calibration process 1000, while still maintaining the intended overall functionality. In various embodiments, the calibration process 1000 can be scheduled for execution based on one or more predefined events and / or executed periodically during operation of the autonomous vehicle 10, 400 (e.g., to dynamically update the calibration with newer data sets).

[0069] The calibration process 1000 correlates or associates non-horizontal edge features between paired data sets at 1002 and constructs or generates a cost function that characterizes the correlations between non-horizontal edge features across paired data sets at 1004. In this regard, each non-horizontal edge region of image data of a paired data set is spatially linked to a corresponding non-horizontal edge region of image data of a paired data set with an equation that characterizes the conversion from the point cloud reference frame to the pixel reference frame (or pixel positions) subject to translation and rotation parameters for transforming or converting a point within the lidar point cloud into a relative pixel position and vice versa.

[0070] The calibration process 1000 then calculates or otherwise determines the values ​​for the conversion parameters that minimize the cost associated with the conversion at 1006. In other words, the calibration process 1000 minimizes the cost of converting between the camera and lidar reference frames across all paired datasets by determining values ​​for the conversion parameters that minimize the total cost related to the differences between a transformed position and the actual position of a correlated edge feature. For example, in one embodiment, the calibration process 1000 determines values ​​for six conversion parameters: three translational parameter values ​​(e.g., x-axis, y-axis, and z-axis), and three rotational values ​​(e.g.,Pitch, roll, and yaw) that minimize the cumulative differences between transformed pixel positions calculated based on the non-horizontal edge regions of the lidar data and the actual pixel positions of the non-horizontal edge regions of the image data corresponding to the respective non-horizontal edge regions of the lidar data.

[0071] The calibration process 1000 stores or maintains the resulting parameter values ​​to minimize conversion costs associated with the camera-lidar pairing at 1008. In this regard, the control module 34, 500 stores or maintains the translational and rotational parameter values ​​for translating from the point cloud reference frame of the lidar device 504 to the image pixel frame of the paired camera 502 in the data storage element 506 in conjunction with one or more identifiers for the camera 502 and / or the lidar device 504 for converting the lidar data corresponding to the camera field of view from the lidar device 504 into corresponding pixel positions, or vice versa. In this manner, depth or distance information can be associated with the pixel positions within the images captured by the camera 502, thereby projecting or otherwise assigning different depths or distances to different portions of the image.

[0072] Fig. 11 shows a graphical representation of calibrated lidar scan data 1100 of an environment corresponding to a camera field of view overlaid on an image 1102 of the environment captured by the camera. Using the calibrated transformation parameters, the lidar scan data 1100 can be converted into pixel positions so that depth discontinuities along rows corresponding to the horizontal reference plane of the lidar device 504 are more closely aligned with corresponding features or elements in the image 1102. Accordingly, pixel positions can be correlated with lidar points, and these pixel positions can be assigned corresponding depths or distances with increased accuracy or precision through calibration.

[0073] Again referring to Fig. 10, in one or more exemplary embodiments, the calibration process 1000 also otherwise stores or maintains the minimized transformation costs associated with the camera-lidar device pairing at 1010 to support the verification or validation of the calibration, as described below in the context of Fig. 12. In this regard, the transformation costs may be periodically analyzed or monitored substantially in real time during subsequent operation of the vehicle to detect or otherwise identify a potential loss of calibration or other need to recalibrate or analyze one or more of the on-board devices. For example, if it is subsequently determined that the transformation costs can be further minimized by a different combination of transformation parameter values, then the previously determined combination of transformation parameter values ​​at 1006 no longer provides the minimum transformation cost and therefore can no longer reflect a calibrated relationship.

[0074] With reference now to Fig. 12 and with further reference to the Fig. 1-5 illustrates a data flow diagram of various embodiments of a calibration verification process 1200 incorporated into a controller 34 in the control system 100 of Fig. 1, which includes the ADS 70 and the image processing module 500 from Fig. 5 according to the present invention. Here too, the sequence of operations within the method is not limited to the Fig. 12, but may be performed in one or more different sequences according to the present invention. In addition, one or more of the sequences described in the context of Fig.12 may be omitted from a practical embodiment of the verification process 1200, and the intended overall functionality may still be retained. In various embodiments, the verification process 1200 may be scheduled for execution based on one or more predefined events (e.g., when the vehicle 10, 400 exhibits a steady state that meets the calibration criteria) and / or executed continuously during operation of the autonomous vehicle 10, 400. For example, the verification process 1200 could be performed according to periodic time intervals (e.g., every 5 minutes), periodic distance intervals (e.g., every 5 miles), and / or the like.

[0075] The calibration verification process 1200 begins or otherwise initializes by obtaining an updated set of image data and lidar data from a paired camera 502 and lidar device 504 for which an existing calibration is to be analyzed at 1202. In exemplary embodiments, the calibration verification process 1200 invokes the data collection process 600 described above to obtain a correlated set of image and lidar data from the respective on-vehicle devices 502, 504 while the vehicle 10, 400 exhibits a steady state that also meets the applicable calibration criteria.Thus, as described above, the paired data sets used by the calibration verification process 1200 may be less susceptible to noise and may be obtained under substantially similar vehicle conditions than the paired data sets used for the previous calibration.

[0076] The calibration verification process 1200 also obtains a reference minimum value for the conversion cost associated with the calibrated conversion parameters of the camera-lidar pair being analyzed at 1202. In this regard, a control module 34, 500 may retrieve or otherwise obtain the stored minimum conversion cost value of the existing calibration of the paired camera 502 and lidar device 504 from the data storage element 506 at 12010. That is, in other embodiments, a reference transformation cost value may be calculated during each pass of the calibration verification process 1200.For example, the reference transformation cost value can be calculated using the existing transformation parameter values, using the previous paired data sets for the existing calibration and the updated paired data sets for verification, so that the reference value for the paired data set is used to verify the existing calibration.

[0077] In the exemplary embodiments, the calibration verification process 1200 continues by varying or otherwise adjusting one or more of the transformation parameter values ​​to minimize the transformation cost using the updated data set at 1206 and determining whether a transformation cost value less than the reference value can be achieved at 1208. In this regard, in a manner similar to that described above at 1206, the control module 34, 500 minimizes the cost of converting between the camera and lidar reference frames across one or more paired data sets that include the updated data set from 1202 by determining values ​​for the conversion parameters that minimize the total cost related to the differences between a transformed position and the actual position of the correlated non-horizontal edge features.If the minimized transformation cost value of the control module 34, 500 using the updated set is greater than or equal to the reference transformation cost value, the calibration verification process 1200 validates or otherwise verifies the existing calibration and terminates it.

[0078] If a transformation cost value less than the reference value can be achieved with the updated data set from 1202, the calibration verification process 1200 continues by comparing or otherwise analyzing the non-horizontal edge regions of the updated data set at 1210 to verify or otherwise confirm that none of the paired devices has an anomalous condition at 1212. In this regard, the calibration verification process 1200 distinguishes between a calibration loss or other anomalous condition with respect to the previously determined calibrated transformation parameter values ​​and a degradation or other anomalous condition with respect to one of the previously calibrated on-board devices 502, 504.

[0079] In an exemplary embodiment, the control module 34, 500 detects or otherwise determines an anomalous condition when a non-horizontal edge region identified within the updated image data is not present in the lidar point cloud data, or conversely, when a depth discontinuity across a set of lidar point cloud data does not have a corresponding non-horizontal edge region within the updated image data. In other words, the calibration verification process 1200 can ensure that the same number of non-horizontal edges are present in both the updated image data and the lidar data, and that each non-horizontal edge feature identified in one of the data sets has a corresponding non-horizontal edge feature in the other data set.If the non-horizontal edge regions of the updated image data match or otherwise match the non-horizontal edge regions of the updated lidar data, and vice versa, the calibration verification process 1200 determines the absence of an anomalous condition or otherwise verifies the function of the paired vehicle-mounted devices 502, 504.

[0080] Once it has been verified that no anomalous condition exists and the paired on-board devices 502, 504 are functioning properly, the calibration verification process 1200 proceeds with the recalibration of the on-board devices 502, 504 at 1214 in a manner similar to that described above. In one embodiment, the calibration verification process 1200 invokes or otherwise executes the calibration process 1000 to identify or otherwise determine updated calibrated transformation parameter values, thereby minimizing the cost of the conversion between the camera and lidar reference frame cumulatively across the paired datasets used for the previous calibration and the updated dataset.In some embodiments, the calibration verification process 1200 may update the stored transformation parameter values ​​by storing the transformation parameter values ​​resulting in the minimum transformation cost at 1206 in the data storage element 506 instead of the previous transformation parameter values. In other embodiments, the calibration verification process 1200 may invoke or otherwise perform the data collection process 600 to obtain additional new or updated paired data sets, which may then be used to recalibrate the relationship between the in-vehicle devices 502, 504 by invoking or executing the calibration process 1000 using these subsequently obtained paired data sets.By verifying the on-board devices 502, 504 prior to recalibrating the relationship, the calibration verification process 1200 ensures that the updated transformation parameter values ​​are likely to be more accurate or reliable than the previous transformation parameter values.

[0081] If the non-horizontal edge regions of the updated image data do not match the non-horizontal edge regions of the updated lidar data or the non-horizontal edge regions of the updated image data, and the non-horizontal edge regions of the updated lidar data otherwise do not show a match between the data sets, the calibration verification process 1200 initiates one or more corrective actions with respect to the device exhibiting the anomalous condition at 1216. In this regard, in exemplary embodiments, the control module 34, 500 detects or identifies the absence of a non-horizontal edge in one of the data sets indicating the corresponding source device 502, 504 for which the data has an anomalous condition.For example, if one or more non-horizontal edges with a depth discontinuity greater than a threshold are identified within the updated lidar data and corresponding non-horizontal edge regions are missing from the updated image data, the control module 34, 500 may determine an anomalous condition of the camera 502. Conversely, the control module 34, 500 may determine that the lidar device 504 has an anomalous condition based on one or more non-horizontal edges identified within the updated image data and missing from the updated lidar data.

[0082] In practice, the control module 34, 500 may utilize relationships with other on-board devices 502, 504 to ensure the proper functioning of the on-board devices 502, 504 that may be experiencing an abnormal condition. For example, when multiple cameras 502 are paired with a single lidar device 504, if one or more non-horizontal edges identified within the updated image data of one camera 502 are missing from the updated lidar data corresponding to that camera's field of view, and one or more non-horizontal edges identified within the updated image data of another camera 502 are also missing from the updated lidar data corresponding to that camera's field of view, the control module 34, 500 may determine an abnormal condition of the lidar device 504.Likewise, before determining a camera 502 has an anomalous condition, the control module 34, 500 may use updated image data from another camera 502 and lidar data corresponding to the non-horizontal edges in that updated image data to confirm that the lidar device 504 does not have an anomalous condition.

[0083] In one or more embodiments, the remedial action initiated by the calibration verification process 1200 includes temporarily deactivating the device 502, 504 exhibiting the anomalous condition. For example, if a lidar device 404, 504 exhibits an anomalous condition, the control module 34, 500 may disable or otherwise suspend the use of the ranging data output from that lidar device 404, 504 and utilize the ranging data output by another instance of the lidar device 404, 504 in the vehicle. Conversely, if a camera 402, 502 exhibits an anomalous condition, the control module 34, 500 may temporarily discontinue the use of image data from that camera 402, 502 for the purpose of object classification or the like.

[0084] The remedial action initiated by the calibration verification process 1200 may also include one or more user notifications. For example, in one embodiment, the control module 34, 500 may activate or otherwise illuminate one or more lights, indicators, or other elements on an instrument panel of the vehicle 10, 400 to notify a vehicle occupant of the abnormal condition. In some embodiments, an indicator on the instrument panel may also indicate which vehicle-mounted device 40, 502, 504 is suspected of having an abnormal condition and requires inspection or maintenance.The control module 34, 500 may also log or otherwise store event information about the anomalous condition, such as the identification of the device experiencing the anomalous condition, the vehicle position at which the anomalous condition was detected, and other information characterizing the context or state of the vehicle 10, 400 when the anomalous condition occurred. In some embodiments, the control module 34, 500 communicates an indication of the device 40, 502, 504 experiencing an anomalous condition to a remote device 48, 52, 54, which in turn generates or otherwise provides a graphical indication of the device 40, 502, 504 experiencing an anomalous condition.

[0085] In response to a notification of an abnormal condition, a user may inspect the device 40, 502, 504 suspected of having an abnormal condition and take action with respect to the device 40, 502, 504 to restore normal operation of the device 40, 502, 504. For example, a user may clean a lens or sensor of the device 40, 502, 504, remove any obstructions to the line of sight of the device 40, 502, 504, adjust a housing of the device 40, 502, 504, adjust the manner in which the device 40, 502, 504 engages or is disposed on the vehicle 10, 400, adjust electrical connections or communication interfaces between the device 40, 502, 504, and the like. In still further embodiments, the user may replace the device 40, 502, 504 with another instance of the device 40, 502, 504.Once the anomalous condition is resolved, the notification may be cleared, and in response, the control module 34, 500 may subsequently reinitialize the data collection process 600 and the calibration process 1000 and resume autonomous operation of the vehicle 10, 400 with the device 40, 502, 504 that was previously suspected of having an anomalous condition.

[0086] The calibration verification process 1200 can determine which of the on-vehicle devices 40, 502, 504 are suspected of having an abnormal condition, thereby reducing downtime and other costs associated with maintenance, as well as improving the efficiency of such maintenance. For example, instead of requiring a user to access and / or inspect each of the multiple cameras 502 and the distance measuring device 504 that may be located in a particular vehicle 10, 400, maintenance can first be focused on the specific device 40, 502, 504 suspected of having an abnormal condition.Furthermore, because the calibration verification process 1200 distinguishes between the possible cause of the loss of calibration at 1212, the calibration verification process 1200 may help prevent a pair of the devices 40, 502, 504 from being recalibrated if one of the devices 40, 502, 504 is not functioning properly, thereby avoiding subsequent operation with a potentially anomalous device 40, 502, 504 and / or with potentially inaccurate transformation parameter values.

[0087] It is appreciated that the subject matter described herein enables more convenient and accurate calibration by isolating and correlating non-horizontal edges that are less susceptible to noise and other artifacts, so that fewer data sets (or data sets from a smaller number of different vehicle positions) may be required to achieve a desired calibration accuracy. For example, a single pair of image and lidar data may be used to calibrate a camera-lidar pair when the number of non-horizontal edges identified in the environment is greater than a threshold (or alternatively, when the magnitude of the depth or distance variations at the non-horizontal edges is greater than a threshold for at least a threshold number of edges).Furthermore, the use of vehicle odometry or attitude information enables real-time detection when the vehicle is stationary, obtaining updated datasets that are also less susceptible to noise due to the fact that the vehicle is stationary. Multiple camera and lidar pairs can also be calibrated (or recalibrated) or their respective calibrations verified essentially in parallel using datasets acquired while stationary.Although the subject matter described herein is primarily in the context of calibrating a camera image to lidar point cloud data, the subject matter is not necessarily limited to cameras or lidar and could be used in the context of any other pair or combination of devices of different imaging or surveying types to calibrate relationships between data sets from different, independently acting devices.

Claims

[1] Vehicle comprising: a first device on board the vehicle that provides first data; a second device on board the vehicle that provides second data; one or more sensors on board the vehicle; one or more actuators on board the vehicle; and a controller that, through a processor, detects a steady state based on the output of the one or more sensors, obtains a first set of the first data from the first device during the steady state, filters out the horizontal edge regions from the first set, resulting in a filtered set of the first data, obtains a second set of the second data during the steady state, determines one or more transformation parameter values based on a relationship between the second set and the filtered set, and then autonomously operates the one or more actuators in the vehicle in a manner influenced by the one or more transformation parameter values. [2] The vehicle of claim 1, wherein the controller autonomously operates the one or more actuators on board the vehicle in a manner influenced by the one or more transformation parameter values: Obtaining an updated set of initial data; Obtaining an updated set of the second data; Assigning one or more attributes to the updated set of first data based at least in part on the updated set of second data and one or more transformation parameter values; Recognizing an object based at least in part on one or more attributes; and Determining a command for the one or more actuators in a manner influenced by the object. [3] Vehicle according to claim 2, wherein: the first device comprises a camera that captures images of a field of view; the second device comprises a distance measuring device that scans an environment around the vehicle; the first set comprises a first image of the images; the updated set of first data comprises a second image of the images acquired after the first image; the second set comprises a second set of the distance measurement data corresponding to the field of view during the steady state; and the updated set of second data comprises an updated set of range measurement data corresponding to the field of view temporally associated with the second image; and the one or more attributes comprise one or more depths associated with the second image based at least in part on the updated set of range measurement data and the one or more transformation parameter values. [4] The vehicle of claim 3, wherein the controller filters the horizontal edge regions from the first set by identifying non-horizontal edge regions in the first image and determines the one or more transformation parameter values by correlating the non-horizontal edge regions in the first image with corresponding non-horizontal edge regions in the second set of second data. [5] The vehicle of claim 4, wherein the one or more transformation parameter values minimize the cost associated with converting the non-horizontal edge regions in the second set of second data from a reference frame associated with the ranging device to the positions associated with the non-horizontal edge regions in the first image in a reference frame associated with the camera. [6] Vehicle according to claim 1, wherein: the first device comprises a camera that captures images of a field of view; the second device comprises a distance measuring device that scans an environment around the vehicle to provide distance measurement data corresponding to the environment; the first set comprises a first image of the images; and the second set comprises a subset of the distance measurement data corresponding to the field of view. [7] The vehicle of claim 6, wherein the controller converts an updated set of the range measurement data into a reference frame of the camera using one or more transformation parameter values. [8] The vehicle of claim 1, wherein the controller filters the horizontal edge regions from the first set by identifying non-horizontal edge regions in the first set and determines the one or more transformation parameter values by correlating the non-horizontal edge regions in the first data with corresponding non-horizontal edge regions in the second set of the second data, wherein the one or more transformation parameter values minimize the cost associated with converting the non-horizontal edge regions in the second set of the second data to the positions associated with the non-horizontal edge regions in the first set. [9] A method for controlling a vehicle, the method comprising: detecting a steady state of the vehicle via a control module on board the vehicle based at least in part on the output of a sensor system on board the vehicle; in response to the steady state: Obtaining, via the control module, a first image from an imaging device on board the vehicle during the steady state; and Obtaining, via the control module, a first set of range measurement data corresponding to a field of view of the imaging device from a range measurement device onboard the vehicle during the steady state; Filtering, via the control module, horizontal edge regions from the first image, resulting in filtered image data; Determining one or more transformation parameter values for converting between a reference frame associated with the imaging device and a reference frame associated with the distance measuring device based at least in part on the filtered image data and the first set of distance measurement data; Obtaining, via the control module, a second image from the imaging device; Obtaining, via the control module, a second set of ranging data corresponding to the field of view of the imaging device by a time period associated with the second image from the ranging device; Assigning, via the control module, one or more attributes to the second image based at least in part on the distance measurement data and one or more transformation parameter values; and autonomously operating one or more actuators on board the vehicle in a manner influenced by one or more attributes assigned to the second image. [10] The method of claim 9, wherein determining one or more transformation parameter values comprises: Correlating non-horizontal edges in the filtered image data with non-horizontal edges in the first set of range measurement data; and Minimizing the costs associated with converting the non-horizontal edges in the first set of range measurement data from the reference frame associated with the range measurement device to the positions in the reference frame associated with the imaging device that correspond to the correlated non-horizontal edges in the filtered image data.

Citation Information

Patent Citations

  • device and method for calibrating a camera

    DE102016009327A1

  • METHOD AND FUSION SYSTEM FOR DETERMINING THE VELOCITY OF A TARGET

    DE102016117710A1

  • Calibration verification methods for autonomous vehicle operations

    US20190012808A1