SYSTEM AND METHOD FOR ALIGNING A CAMERA ON THE GROUND FOR A VEHICLE

Abstract

A method for performing camera-ground alignment for a camera system in a vehicle. The method includes determining whether release conditions have occurred and estimating the position of a vanishing point in a source image. Ground lines are selected based on the source image. Lane line detection is performed based on clustering the ground lines to determine lane lines in the source image. At least one of the vehicle's pitch, yaw, or roll elements is estimated from the source image. A cost function based on the pitch, yaw, and roll estimates is minimized to obtain optimal pitch, yaw, and roll values, as well as lane lines, from the source image. The source images are refined using a sliding window.The alignment results are transferred to a downstream application, or it is determined whether the vehicle's camera system is misaligned.

Description

INTRODUCTION

[0001] Vehicles are an integral part of everyday life. Specialized cameras, microcontrollers, laser technologies, and sensors can be used in a wide variety of applications within a vehicle. Cameras, microcontrollers, and sensors can be used to enhance automated structures that provide customers with state-of-the-art experiences and services, for example, for tasks such as body inspection, camera vision, information display, security, autonomous control, and more. Automotive vision systems can also be used to support vehicle control. DESCRIPTION

[0002] Described herein is a method for performing camera-to-ground alignment for a camera system in a vehicle. The method includes determining whether a predetermined set of clearance conditions has occurred along a roadway and estimating a position of a vanishing point in a source image containing the roadway. Ground lines are selected along the roadway based on a source image. Lane line detection is performed based on clustering the ground lines to determine lane lines in the source image. At least one of pitch, yaw, or roll of the vehicle is estimated from the source image. A cost function based on the pitch, yaw, and roll estimates is minimized to obtain an optimal pitch value, an optimal yaw value, and an optimal roll value, as well as lane lines from the source image.A sliding window-based refinement is performed on the source images. Alignment results based on the sliding window refinement are passed to a downstream application, or it is determined whether the vehicle's camera system is misaligned based on the sliding window refinement.

[0003] Another aspect of the disclosure may be that the source image is captured by at least one optical sensor on the vehicle.

[0004] Another aspect of the disclosure may be that the predetermined set of enabling conditions includes a velocity along a first axis greater than a predetermined value, a velocity along a second axis less than a second predetermined value, and an acceleration along the first axis within a predetermined range.

[0005] Another aspect of the disclosure may be that the predetermined set of release conditions includes a steering angle that is less than a predetermined value and that the distances between keyframes are greater than a predetermined distance value.

[0006] Another aspect of the disclosure may be that the vanishing point in the source image is estimated based on the detection of a plurality of detected line segments in the source image having a convergence point corresponding to the vanishing point.

[0007] Another aspect of the disclosure may include recalculating the vanishing point based on the plurality of ground lines along the roadway.

[0008] Another aspect of the disclosure may be that the plurality of ground lines are selected by detecting line segments in the source image, detecting a horizon line in the source image, and determining a lane mask for the source image.

[0009] Another aspect of the disclosure may be that the majority of ground lines are selected by eliminating a group of detected line segments in the source image that appear above the horizon line.

[0010] Another aspect of the disclosure may be that the baselines are selected by selecting a set of detected line segments that are adjacent to lane markings in the lane mask.

[0011] Another aspect of the disclosure may be that the detection of lane lines is performed based on clustering, in which the detected line segments are grouped into individual sets based on the distance that each detected line segment in each of the individual sets has from an optimal lane line passing through the vanishing point.

[0012] Another aspect of the disclosure may be that the pitch and yaw of the vehicle are estimated from the source image by comparing the location of the plurality of lane lines relative to the vanishing point.

[0013] Another aspect of the disclosure may be that the roll of the vehicle is estimated from the source image when the lane lines include three separate lane lines.

[0014] Another aspect of the disclosure may be that a roll angle of the vehicle is estimated from the source image when the lane lines include at least two lanes with a certain width.

[0015] Included herein is a non-transitory, computer-readable storage medium containing programmed instructions that, when executed by a processor, are suitable for performing a method. The method includes determining whether a predetermined set of clearance conditions has occurred along a roadway and estimating a position of a vanishing point in a source image containing the roadway. Ground lines are selected along the roadway based on a source image. Lane line detection is performed based on clustering the ground lines to determine lane lines in the source image. At least one of pitch, yaw, or roll of the vehicle is estimated from the source image.A cost function based on the pitch, yaw, and roll estimates is minimized to obtain an optimal pitch value, an optimal yaw value, an optimal roll value, and lane lines from the source image. Sliding-window-based refinement is performed on a large number of source images. Alignment results based on sliding-window refinement are passed to a downstream application, or it is determined whether the vehicle's camera system is misaligned based on sliding-window refinement.

[0016] A vehicle system is described herein. The system includes at least one optical sensor configured to capture a plurality of images and a controller in communication with the at least one optical sensor. The controller is configured to determine whether a predetermined set of clearance conditions has occurred along a roadway, estimate a location of a vanishing point in a source image including the roadway, and select ground lines along the roadway based on a source image. The controller is also configured to perform lane line detection based on clustering the ground lines to determine lane lines in the source image and estimate at least one of pitch, yaw, or roll from the source image.The controller is also configured to minimize a cost function based on estimates of pitch, yaw, and roll to obtain an optimal pitch value, an optimal yaw value, an optimal roll value, and lane lines from the source image. The controller is further configured to perform sliding-window-based refinement of images and send alignment results based on the sliding-window-based refinement to a downstream application or determine if a camera system on the vehicle is misaligned based on the sliding-window-based refinement. BRIEF DESCRIPTION OF THE CHARACTERS Fig. 1 is a schematic representation of an example motor vehicle. Fig. 2 shows an example of a method for aligning a sensor on the motor vehicle of Fig. 1. Fig. Figure 3 shows a graphical representation of part of the process of Fig. 2. Fig. Figure 4 illustrates the clustering of detected line segments to determine lane lines as part of the method of Fig. 2. Fig. 5 shows lane lines that converge at a common vanishing point. Fig. 6 shows lane lines with a certain width between adjacent lane lines. Fig. 7 shows an extended explanation of a block from the method of Fig. 2.

[0017] The present disclosure may be modified or embodied in alternative forms, representative embodiments of which are shown in the drawings and described in detail below. The present disclosure is not limited to the disclosed embodiments. Rather, the present disclosure is intended to cover alternatives falling within the scope of the disclosure as defined by the appended claims. DETAILED DESCRIPTION

[0018] Those of ordinary skill in the art will recognize that terms such as "top," "bottom," "upward," "downward," "left," "right," etc., are used descriptively for the figures and do not represent limitations on the scope of the disclosure as defined by the appended claims. Furthermore, the teachings herein may be described in terms of functional and / or logical block components and / or various processing steps. It should be understood that such block components may include a number of hardware, software, and / or firmware components configured to perform the specified functions.

[0019] With reference to the figures, in which like numerals indicate like parts throughout the figures, like reference numbers referring to like components, Fig. 1 is a schematic view of a motor vehicle 10 positioned with respect to a roadway, e.g., a lane 12. As shown in Fig. 1, the vehicle 10 includes a vehicle body 14, a first axle having a first set of wheels 16-1, 16-2, and a second axle having a second set of wheels 16-3, 16-4 (e.g., individual left and right wheels on each axle). Each of the road wheels 16-1, 16-2, 16-3, 16-4 has tires configured to make notional contact with the vehicle track 12. Although two axles with respective road wheels 16-1, 16-2, 16-3, 16-4 are expressly shown, nothing precludes the motor vehicle 10 from having additional axles.

[0020] As in Fig. 1, a vehicle suspension system connects the vehicle body 14 to the respective wheel sets 16-1, 16-2, 16-3, 16-4 to maintain contact between the wheels and the road surface 12 and to maintain the handling of the motor vehicle 10. The motor vehicle 10 additionally includes a drivetrain 20 having one or more power sources 20A, which may be an internal combustion engine (ICE), an electric motor, or a combination of such devices, configured to transmit drive torque to the road wheels 16-1, 16-2 and / or the road wheels 16-3, 16-4. The motor vehicle 10 also employs vehicle operation or control systems, including devices such as one or more steering actuators 22 (e.g.,an electric power steering system) configured to steer the road wheels 16-1, 16-2, a steering angle (θ), an accelerator device 23 for controlling the power output of the energy source(s) 20A, a brake switch or braking device 24 for decelerating the rotation of the road wheels 16-1 and 16-2 (e.g., via individual friction brakes located on the respective road wheels), etc.

[0021] As in Fig. 1, the motor vehicle 10 includes at least one sensor 25A and an electronic control unit 26 that cooperate to control, guide, and maneuver the vehicle 10 at least partially in an autonomous mode in certain situations. As such, the vehicle 10 may be referred to as an autonomous vehicle. To enable efficient and reliable autonomous vehicle control, the electronic control unit 26 may be in operative communication with the steering actuator(s) 22 embodied as an electric power steering system, the accelerator device 23, and the braking device 24. The sensors 25A of the motor vehicle 10 are capable of scanning the lane 12 and monitoring a surrounding geographical area as well as traffic conditions in the vicinity of the motor vehicle 10.

[0022] The sensors 25A of the vehicle 10 may include, among other things, at least one LiDAR (Light Detection and Ranging) sensor, a radar, and camera systems, e.g., optical sensors, arranged around the vehicle 10 to detect the boundary indicators, e.g., the boundary conditions, of the vehicle lane 12. The nature of the sensors 25A, their position on the vehicle 10, and their functionality for detecting and / or sensing the boundary indicators of the vehicle lane 12 and for monitoring the surrounding geographic area and traffic conditions will be understood by those skilled in the art and are therefore not described in detail here.

[0023] The electronic control unit 26 communicates with the sensors 25A of the vehicle 10 to receive their respective measurement data related to the detection or sensing of the vehicle lane 12 and the monitoring of the surrounding geographical area and traffic conditions. The electronic control unit 26 may alternatively be referred to as a control module, control unit, control device, control unit of the vehicle 10, computer, etc. The electronic control unit 26 may include a computer and / or processor 28, as well as software, hardware, memory, algorithms, connections (e.g., to sensors 25A), etc., for managing and controlling the operation of the vehicle 10. As such, a method described below and generally in Fig. 2, may be embodied as a program or algorithm that may partially operate on the electronic control unit 26. It should be appreciated that the electronic control unit 26 may include a device capable of analyzing data from the sensors 25A, comparing data, making the necessary decisions to control the operation of the vehicle 10, and performing the necessary tasks to control the operation of the vehicle 10.

[0024] The electronic control unit 26 may be embodied as one or more digital computers or host machines, each including one or more processors 28, read-only memory (ROM), random access memory (RAM), electrically programmable read-only memory (EPROM), optical drives, magnetic drives, etc., a high-speed clock, analog-to-digital (A / D) circuitry, digital-to-analog (D / A) circuitry, and input / output (I / O) circuitry, I / O devices and communication interfaces, as well as signal conditioning and buffer electronics. The computer-readable memory may comprise a non-volatile / tangible medium involved in providing data or computer-readable instructions. The memory may be non-volatile or volatile. Non-volatile media may include, for example, optical or magnetic disks and other persistent storage.An example of volatile memory is dynamic random-access memory (DRAM), which can be used as main memory. Other examples of memory include flexible disks, hard disks, magnetic tape or other magnetic media, CD-ROMs, DVDs, and / or other optical media, as well as other possible storage devices such as flash memory.

[0025] The electronic control unit 26 includes a tangible, non-transferable memory 30 in which computer-executable instructions, including one or more algorithms, for controlling the operation of the motor vehicle 10 are recorded. The algorithm(s) in question may, in particular, include an algorithm configured to monitor the location of the motor vehicle 10 and determine the course of the vehicle relative to a mapped vehicle trajectory on a particular road course, which is described in detail below.

[0026] The motor vehicle 10 also includes a vehicle navigation system 34, which may be part of the integrated vehicle control system or an additional device used to determine the direction of travel in the vehicle. The vehicle navigation system 34 is also operatively connected to a global positioning system (GPS) 36 utilizing a satellite in Earth orbit. The vehicle navigation system 34, in conjunction with the GPS 36 and the aforementioned sensors 25A, may be used to automate the vehicle 10. The electronic control unit 26 communicates with the GPS 36 via the vehicle navigation system 34. The vehicle navigation system 34 uses a satellite navigation device (not shown) to receive its position data from the GPS 36, which is then correlated with the vehicle's position relative to the surrounding geographic area.Based on this information, if directions to a specific waypoint are needed, the route to that destination can be mapped and calculated. Current terrain and / or traffic information can be used to adjust the route. The current position of a vehicle 10 can be calculated using dead reckoning, using a previously determined position and extrapolating that position based on given or estimated speeds over elapsed time and heading using discrete checkpoints.

[0027] The electronic control unit is generally configured, ie programmed, to determine the localization 38 (current position in the XY plane, as shown in Fig. 1), determines or identifies the speed, acceleration, yaw rate, and intended path 40 and heading 42 of the motor vehicle 10 within the lane 12. The location 38, intended path 40, and heading 42 of the motor vehicle 10 may be determined via the navigation system 34 receiving data from the GPS 36, while the speed, acceleration (including longitudinal and lateral acceleration), and yaw rate may be determined from the vehicle sensors 25A. Alternatively, the electronic control unit 26 may utilize other systems or sensing sources remote from the vehicle 10, such as a camera, to determine the location 38 of the vehicle relative to the lane 12.

[0028] As previously mentioned, the motor vehicle 10 may be configured to operate in an autonomous mode controlled by the electronic control unit 26 to transport an occupant 62. In such a mode, the electronic control unit 26 may further receive data from the vehicle sensors 25A to guide the vehicle along the desired path, for example, by regulating the steering actuator 22. The electronic control unit 26 may additionally be programmed to detect and monitor the steering angle (θ) of the steering actuator(s) 22 along the desired path 40, for example, during a negotiated turn. In particular, the electronic control unit 26 may be programmed to determine the steering angle (θ) by receiving and processing data signals from a steering position sensor 44 (in Fig. 1) which is connected to the steering actuator(s) 22, the accelerator device 23 and the brake device 24.

[0029] Fig. Figure 2 shows a flowchart of the method 100 for performing a ground alignment of the camera system. The method 100 begins in block 102 by determining whether at least one enabling condition or a predetermined set of enabling conditions is met. The enabling conditions may include at least one of the following conditions: a norm of a vehicle speed along the x-axis of the ground that is greater than a predetermined speed (||v x || > v u ), a norm of the vehicle speed along the y-axis of the ground that is less than a given speed (||v y || < v l ), the acceleration along the x-axis of the vehicle is within a given range (a v < ||a x|| < a u ), the steering angle is smaller than a specified value (||θ||< θ u ), or a distance between keyframes that is greater than a predetermined value (||v||Δt > w u ). Release conditions may also include vehicle exit points being closed or performing a temporal comparison between images for lane masks to ensure a straight lane. Release conditions may also include rain, poor light, snow, fog, or a spare tire. Furthermore, in one example, each of the above release conditions may need to be met to proceed to block 104.

[0030] In block 104, the method 100 performs an initial vanishing point detection and selection, as shown in the Fig. shown. Fig. shows a graphical representation of the source image 200 moving from block 102 through blocks 104, 106, and 108. The initial vanishing point detection is re-estimated or refined in block 106, as explained in more detail below. In block 102, a source image 200 is obtained for analysis, as shown in the Fig. The analysis includes performing line segment detection on the source image 200 in block 104 to obtain a line-segmented image 202 having a plurality of detected line segments 204 identified from the source image 200, wherein the detected line segments are superimposed on the source image 200. In block 104, the detected line segments 204 are used to determine the position of an initial vanishing point 206 in an initial vanishing point detection image 208. In particular, the initial vanishing point 206 in the initial vanishing point detection image 208 is located near an endpoint of the detected line segments 204. EQ. ​​1 indicates the uncertainty of the vanishing point. cov(pv)=Jpvcov([α,β,γ,t3]T)JpvT In EQ. 1 above, p v and J pv can be found in EQ. 2 and EQ. 3. pv=λcgH v Jpv=∂pv∂(α,β,γ,t3)

[0031] Also, cov([α, β, γ, t3] T) in EQ. 1 above is provided by the manufacturing alignment of the camera system that forms at least a portion of the sensors 25A. Using the above information, a vanishing point is selected by determining a point that satisfies EQ. 4 below. pvTcov(pv)pv≤F−1(1−η / 2) In the above EQS., cgH=K[r1r2t], a vanishing direction v = [1 0 0] T , t a translation from the ground to the camera, t i is the i-th element of t, F -1 (x) is the inverse cumulative probability function of the χ2 distribution with 2 degrees of freedom, K is a camera-specific matrix, α is the roll angle, β is the pitch angle, γ is the yaw angle and r i is the i-column of R(α, β, y) with R(α, β, y) shown in EQ. 5 below. Rz(γ)Ry(β)Rx(α)=[cos γ−sin γ0sin γcos γ0001][cos β0sin β010−sinβ0cos β][1000cos α−sin α0sin αcos α]

[0032] In block 108, the selection of the floor line takes place. The selection of the floor line takes place as in the Fig. shown, using the initial vanishing point 206 from block 104, a horizon line 210 shown in the horizon line image 212 of Fig. is shown, and a lane mask 214, which is shown in the mask image 216 of block 104, and lane segmentation, which occurs in block 110. Non-ground-level line segments, which are represented by the detected line segments 204 of the image 202, are removed from the initial vanishing point detection image 208 using the horizon line 210 from the image 212 to generate a ground line image 220 with ground lines 218. The horizon line l h is determined by the following two points in the image (r ij is the element of the i-th row and j-th column of R(α, β, y)). If r 31 ≠ 0 and r 32 ≠ 0, p̌1 and p̌2 are provided by the EQS. 6 and 7 below. p⌣1=K[r11r31r21r311] p⌣2=K[r12r32r22r321]

[0033] If r 31 = 0 and r 32 ≠ 0, p̌1 and p̌2 are provided by EQS. 8 and 9 below. p⌣1=K[r12r33r22r321] p⌣1=K[r12r33r22r321]

[0034] If r 31 ≠ 0 and r 32 = 0, p̌1 and p̌2 are provided by EQS. 10 and 11 below. p⌣1=K[r11r31r21r311] p⌣1=K[r11r31r21r311]

[0035] In addition, if r 31 and r 32 both are zero, the camera system cannot observe the ground 12. Line segments whose endpoints p satisfy l hp̌ > 0 are selected to redetermine a refined vanishing point 222 in the refined image 224 from the ground line segments in block 106. Selection of the ground line segment (selection of the line segment l that is close to the lane mask) as shown in EQ. 12 below, where m is a pixel considered to be a lane, M is a lane pixel, and ε is a predetermined threshold variable. maxp∈l,m∈M‖p−m‖<ε

[0036] In block 112, the line segments are divided into different groups, whereby the detected line segments 204 in the same group belong to the same lane line. l. As in Fig. As shown in Figure 4, a first group of line segments 204-1 is grouped around line l1, a second group of line segments 204-2 is arranged around line l2, and a third group of line segments 204-3 is grouped around line l3, such that line segments 204 are connected to the corresponding line l. Line segments 204 that are outliers are eliminated as noise.

[0037] For lane line detection, a lane line is generated for each of the line segments that are in the same group, as in Fig. 5. For each line in the group L = U l that has the same designation, the optimal lane line should l+∗ through the vanishing point p v re-estimated from the baseline segments with l⌣+∗pv=0 and the smallest distance to the line segments in group L. The optimal lane line is determined using EQ. 13 below for line segments with the same designations. argminl+ ∑p∈l, l∈L‖l⌣+Tp‖

[0038] In block 114, the pitch (β) and yaw (γ) of the vehicle 10 are estimated. For EQ. 5 above and the escape direction v = [1 0 0] T The pitch (β) and yaw (γ) can be taken from EQS. 14 and 15, where R is the rotation matrix from the ground to the camera, K is the camera's own matrix, and λ is a scaling factor. pv=λKRv K−1[pxpy 1]=λ[cosβ cinγcosβ cinγ −sinβ]

[0039] In block 116, the method 100 then determines whether enough lanes have been detected. If enough lanes have been detected, the method 100 proceeds to block 120 to perform an estimation of the roll (α) angle estimate. If not enough lanes have been detected, the method 100 proceeds to block 118. In block 118, the method 100 determines whether the pitch and yaw angle estimates are reliable. If the pitch and yaw estimates are determined to be reliable, the method 100 returns to block 102 to determine whether the release conditions are met. If the pitch and yaw estimates are reliable, the method 100 proceeds to block 124 to perform a sliding window-based optimization to further refine the pitch and yaw estimates, as explained below.

[0040] As the method proceeds from block 116 to block 120, method 100 performs roll angle (α) estimation. Method 100 estimates the roll angle (α) from source image 200 using the lane lines detected as described above. In one example, EQ. 16 below becomes the development of EQ. 17. cgH=K[r1 r2 t] li'= cgHTli

[0041] In EQ. 16 above, r i the column i of R, t is a translation vector between ground and camera, and W is a lane width. Using EQ. 17, block 120 can use EQ. 18 to determine the roll angle (α) from a single source image 200 when three separate lane lines are present, as in Fig. 5, or EQ 19, when only two-lane lines with a given lane width between adjacent lane lines are detected, as in Fig. 6 shown. ‖⌊l1'⌋×

[100] −⌊l2'⌋×

[100] ‖−‖⌊l2'⌋×

[100] −⌊l3'⌋×

[100] ‖=0 |‖⌊l1'⌋×

[100] −⌊l2'⌋×

[100] ‖−W|=0

[0042] From block 120, the method 100 proceeds to block 122 to determine at least one alignment parameter from a single one of the source images 200. To obtain the lane lines in vehicle coordinates, the method 100 minimizes the cost equation in EQ. 20 below to obtain initial estimates of (α, β, γ) to determine optimal α k , β k , and γ k and lane lines in a single image. If the method identifies 100 lane lines that run parallel to the vehicle's direction of travel 20 ( Fig. 5), the method applies 100 EQ. 20 below. If the method identifies 100 lane lines with an equal lane width ( Fig. 6), the method 100 may use either EQS 21 or 22 below, where ω0 is a weighting factor and W is the lane width. fa(α,β,γ,i)=liTcgH

[100] fw(α,β,γ,i)=∑i‖⌊cgHTli⌋×

[100] −⌊cgHTli+1⌋×

[100] ‖−‖⌊cgHTli+1⌋×

[100] −⌊cgHTli+1⌋×

[100] ‖ fw(α,β,γ,i)=∑i‖⌊cgHTli⌋×

[100] −⌊cgHTli+1⌋×

[100] −W‖

[0043] From block 122, the method 100 proceeds to block 124 to perform a sliding window-based optimization for multiple images, e.g., for multiple source images. In a first example, the roll, pitch, and yaw parameters are refined in the sliding window by minimizing EQ. 23 below. fs(α,β,γ)=12∑kfl(α,β,γ)+ω1tr(RT(α,β,γ)R(αk,βk,γk)−1)

[0044] In a second example, the minimization or optimization for roll, pitch, yaw and ground camera center height (t3) and replace f w (α, β, γ, i) in f l (α, β, γ, i) by Eq. 24 below, where cgH=K[r1 r2 t] taking into account t3 is unknown and W is the road width. fw(α,β,γ,i)=∑i‖⌊cgHTli⌋×

[100] −⌊cgHTli+1⌋×

[100] −W‖

[0045] In addition, the following updates will be made: X k+1 = X k - (J T J + λdiag(J T J)) -1 J T f s , X k = [α, β, γ] T if case 1 otherwise [α,β,γ,t3]T,and J=∂f / ∂Xk for the second case. In the above equations, λ is a damping factor, ω1 is a weighting factor, and t3 is the third element of t (height between ground and camera center). The road width W can be obtained from a map or another active sensor such as LiDAR with sensors 25A and 1 is a 3x3 identity matrix.

[0046] In block 126, the method 100 detects a misalignment, refines the alignment parameters, and updates a coordination transformation matrix (CTM) for use by downstream applications. The CTM stores alignment results to characterize a transformation from one coordinate system to another coordinate system, e.g., a camera coordinate system to a vehicle coordinate system. In one example, the downstream applications may include at least one of the following: perception-based applications, low velocity maneuver (LVM) applications, such as automated parking or reversing, or viewing applications. Fig. shows another example of the implementation of the steps that take place in block 126.

[0047] In one example, detection of alignment errors by parallel alignment with an additional alignment approach, e.g., feature-based alignment, may be performed by using the alignment parameters determined from the sliding window-based optimization in block 124. Additional alignment approaches are described in commonly owned, commonly filed U.S. patent applications Ser. No. 17 / 651,407, U.S. Patent App. 17 / 651,405, U.S. Patent App. 17 / 651,406, and U.S. Patent No. 8,373,763, the disclosures of which are incorporated herein by reference in their entirety. The alignment by parallelism and the additional alignment approach are then compared to a prior community trademark in block 302. Starting at block 302, the method 100 determines whether the comparison to at least one of the above approaches is greater than a predetermined threshold in block 304.If the difference between at least one of the above approaches is less than the predefined threshold, the method 100 proceeds to block 306, updates the alignment, and publishes the CTM. This information can then be used by downstream applications.

[0048] If the difference between the approximations and the Community trademark is greater than the predetermined threshold, the method 100 proceeds to block 308 within block 126, as in Fig.7. In block 308, a temporal comparison is performed to determine the misalignment. If no misalignment is detected, the method 100 returns to block 302. If misalignment is detected in block 310, the method 100 proceeds to block 312 and determines whether the misalignment or alignment should be transmitted. If the misalignment or alignment should be transmitted, the method 100 may set a misalignment flag. If the misalignment should not be transmitted, the method 100 proceeds to block 314 and determines that the method 100 should pause and exit. If the method 100 should pause and exit, the method 100 clears the variables and reports the status. If not, the method 100 proceeds to block 302.

[0049] The terms "a" and "an" do not imply a quantitative limitation, but denote the presence of at least one of the mentioned elements. The term "or" means "and / or" unless the context clearly indicates otherwise. Whenever "an aspect" is mentioned throughout the description, this means that a particular element (e.g., a feature, structure, step, or property) described in connection with the aspect is included in at least one of the aspects described therein and may or may not be present in other aspects. Furthermore, the described elements in the different aspects may be combined as appropriate.

[0050] When an element, such as a layer, film, region, or substrate, is described as being "on" another element, it may be directly on top of the other element, or there may be intervening elements. Conversely, when an element is described as being "directly on" another element, there are no intervening elements.

[0051] Unless otherwise specified herein, examination standards shall be deemed to be the most recent standards in force on the filing date of this application or, if priority is claimed, the filing date of the earliest priority application in which the examination standard appears.

[0052] Unless otherwise defined, technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0053] Although the above disclosure has been described with reference to exemplary embodiments, those skilled in the art will understand that various changes may be made and equivalent elements may be substituted without departing from the scope of the disclosure. Furthermore, many modifications may be made to adapt a particular situation or material to the teachings of the disclosure without violating the scope of the disclosure. Therefore, the present disclosure is not intended to be limited to the specific embodiments, but is intended to include embodiments that fall within the scope of the disclosure. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 17 / 651,407

[0047] US 17 / 651,405

[0047] US 17 / 651,406

[0047] US 8,373,763

[0047]

Claims

[1] A method of performing a camera-to-ground alignment for a camera system on a vehicle, the method comprising: Determining whether a predetermined set of clearance conditions has occurred along a roadway; Estimating the position of a vanishing point in a source image containing the roadway; Selecting a plurality of ground lines along the roadway based on the source image; performing lane line detection based on clustering the plurality of ground lines to determine a plurality of lane lines in the source image; Estimating the pitch, yaw and / or roll of the vehicle from the source image; Minimizing a cost function based on estimates of pitch, yaw, and roll to obtain an optimal pitch value, an optimal yaw value, an optimal roll value, and lane lines from the source image; performing a sliding window-based refinement on a plurality of source images; and Transfer results based on sliding window refinement to a downstream application or determine whether the camera system on the vehicle is misaligned based on sliding window refinement. [2] The method of claim 1, wherein the source image is acquired by at least one optical sensor on the vehicle. [3] The method of claim 1, wherein the predetermined set of enabling conditions includes a velocity along a first axis greater than a predetermined value, a velocity along a second axis less than a second predetermined value, and an acceleration along the first axis within a predetermined range. [4] The method of claim 3, wherein the predetermined set of release conditions includes a steering angle less than a predetermined value and distances between keyframes greater than a predetermined distance value. [5] The method of claim 1, wherein the estimation of the vanishing point in the source image is based on the detection of a plurality of detected line segments in the source image having a convergence point corresponding to the vanishing point. [6] The method of claim 5, including recalculating the vanishing point based on the plurality of ground lines along the roadway. [7] The method of claim 1, wherein selecting the plurality of ground lines comprises detecting a plurality of detected line segments in the source image, detecting a horizon line in the source image, and determining a lane mask for the source image. [8] The method of claim 7, wherein selecting the plurality of ground lines includes eliminating a set of the plurality of detected line segments in the source image that appear above the horizon line. [9] The method of claim 8, wherein selecting the plurality of baselines comprises selecting a set of the plurality of detected line segments adjacent to pavement markings in the pavement mask. [10] The method of claim 1, wherein performing lane line detection based on clustering comprises grouping a plurality of detected line segments into individual sets based on a distance that each detected line segment in each of the individual sets is from an optimal lane line passing through the vanishing point.

Images