System and method for performing camera-to-ground alignment for vehicle

Abstract

A method of performing camera-to-ground alignment for a camera system onboard a vehicle. The method includes determining whether an enabling condition has occurred and estimating a location of a vanishing point in the source image. A ground line is selected based on the source image. Lane line detection is performed based on the aggregation of ground lines to determine lane lines in the source image. At least one of pitch, yaw, or roll of the vehicle is estimated from the source image. A cost function based on estimates of pitch, yaw, and roll is minimized to obtain an optimal pitch value, an optimal yaw value, an optimal roll value, and a lane line from the source image. A sliding window-based refinement is performed on a source image. The alignment result is broadcasted to a downstream application, or it is determined whether a camera system on the vehicle is misaligned.

Description

[0001] introduce

[0002] Vehicles are a necessity in everyday life. Specialized cameras, microcontrollers, laser technology and sensors can be used for many different applications in vehicles. Cameras, microcontrollers and sensors can be used to enhance the automation structure, which provides customers with state-of-the-art experiences and services, for example in tasks such as body control, camera vision, information display, safety, autonomous control, etc. Vehicle vision systems can also be used to assist in vehicle control. Summary of the invention

[0003] A method for performing camera-to-ground alignment for a camera system on a vehicle is disclosed herein. The method includes determining whether a predetermined set of enabling conditions have occurred along a road and estimating the location of a vanishing point in a source image including the road. Ground lines are selected along the road based on the source image. Lane line detection is performed based on a clustering of the ground lines to determine lane lines in the source image. At least one of pitch, yaw, or roll of the vehicle is estimated from the source image. A cost function based on the estimation of pitch, yaw, and roll is minimized to obtain an optimal pitch value, an optimal yaw value, and an optimal roll value and lane lines from the source image. A sliding window-based refinement is performed on the source image. The alignment result based on the sliding window refinement is broadcast to a downstream application, or a determination is made based on the sliding window-based refinement whether the camera system on the vehicle is misaligned.

[0004] Another aspect of the present disclosure may be wherein the source image is captured by at least one optical sensor on a vehicle.

[0005] Another aspect of the present disclosure may be wherein the predetermined set of enabling conditions includes a velocity along the first axis being greater than a predetermined value, a velocity along the second axis being less than a second predetermined value, and an acceleration along the first axis being within a predetermined range.

[0006] Another aspect of the present disclosure may be wherein the predetermined set of activation conditions includes a steering angle being less than a predetermined value and a distance between key frames being greater than a predetermined distance value.

[0007] Another aspect of the present disclosure may be wherein a vanishing point in the source image is estimated based on detecting a plurality of detected line segments in the source image having convergence points corresponding to the vanishing point.

[0008] Another aspect of the present disclosure may include re-estimating the vanishing point based on the plurality of ground lines along the road.

[0009] Another aspect of the disclosure may be wherein the plurality of ground lines are selected by detecting line segments detected in the source image, detecting a horizon in the source image, and determining a lane mask for the source image.

[0010] Another aspect of the present disclosure may be wherein the plurality of ground lines are selected by eliminating a set of detected line segments that appear above a horizon line in the source image.

[0011] Another aspect of the disclosure may be wherein the ground lines are selected by selecting a set of detected line segments adjacent to lane markings in a lane mask.

[0012] Another aspect of the present disclosure may be wherein lane line detection is performed based on aggregation by grouping the detected line segments into individual groups based on the distance of each detected line segment in each individual group from an optimal lane line passing through the vanishing point.

[0013] Another aspect of the present disclosure may be wherein the pitch and yaw of the vehicle are estimated from the source image by comparing the positions of the plurality of lane lines with the vanishing point.

[0014] Another aspect of the present disclosure may be wherein a roll of a vehicle is estimated from the source image when the lane marking includes three separate lane markings.

[0015] Another aspect of the present disclosure may be wherein estimating the roll angle of the vehicle from the source image occurs when the lane lines include at least two lane lines of a given width.

[0016] Disclosed herein is a non-transitory computer-readable storage medium containing programming instructions, which, when executed by a processor, are operable to perform a method. The method includes determining whether a predetermined set of enabling conditions have occurred along a road and estimating the location of a vanishing point in a source image including the road. Selecting ground lines along the road based on the source image. Performing lane line detection based on a cluster of the ground lines to determine lane lines in the source image. Estimate at least one of the pitch, yaw, or roll of the vehicle from the source image. Minimize a cost function based on the estimation of pitch, yaw, and roll to obtain an optimal pitch value, an optimal yaw value, an optimal roll value, and a lane line from the source image. Perform sliding window-based refinement on multiple source images. Broadcasting the alignment results based on the sliding window refinement to downstream applications, or determining whether a camera system on the vehicle is misaligned based on the sliding window-based refinement.

[0017] A vehicle system is disclosed herein. The system includes: at least one optical sensor configured to capture multiple images; and a controller in communication with the at least one optical sensor. The controller is configured to: determine whether a predetermined set of enabling conditions have occurred along a road; estimate the location of a vanishing point in a source image including a road; and select ground lines along the road based on the source image. The controller is also configured to: perform lane line detection based on the aggregation of the ground lines to determine lane lines in the source image; and estimate at least one of the pitch, yaw, or roll of the vehicle from the source image. The controller is also configured to: minimize a cost function based on the estimation of pitch, yaw, and roll to obtain an optimal pitch value, an optimal yaw value, an optimal roll value, and a lane line from the source image. The controller is also configured to: perform sliding window-based refinement on the image and broadcast the alignment result based on the sliding window refinement to downstream applications, or determine whether the camera system on the vehicle is misaligned based on the sliding window-based refinement.

[0018] A first aspect of the present disclosure provides a method for performing camera-ground alignment on a camera system on a vehicle, the method comprising:

[0019] determining whether a predetermined set of enabling conditions have occurred along the road;

[0020] estimating a location of a vanishing point in a source image including the road;

[0021] selecting a plurality of ground lines along the road based on the source image;

[0022] performing lane line detection based on the aggregation of the plurality of ground lines to determine a plurality of lane lines in the source image;

[0023] estimating at least one of pitch, yaw or roll of the vehicle from the source images;

[0024] minimizing a cost function based on the estimation of pitch, yaw and roll to obtain an optimal pitch value, an optimal yaw value, an optimal roll value and a lane line from the source image;

[0025] performing sliding window based refinement on the plurality of source images; and

[0026] The results of the sliding window refinement are broadcasted to downstream applications, or a determination is made as to whether a camera system on the vehicle is misaligned based on the sliding window based refinement.

[0027] The method according to the first aspect of the present disclosure, wherein the source image is captured by at least one optical sensor on a vehicle.

[0028] According to the method of the first aspect of the present disclosure, the predetermined set of enabling conditions includes: the speed along the first axis is greater than a predetermined value, the speed along the second axis is less than a second predetermined value, and the acceleration along the first axis is within a predetermined range.

[0029] According to the method of the first aspect of the present disclosure, the predetermined set of starting conditions includes: the steering angle is less than a predetermined value and the distance between key frames is greater than a predetermined distance value.

[0030] The method according to the first aspect of the present disclosure, wherein the vanishing point in the source image is estimated based on detecting a plurality of detected line segments having convergence points corresponding to the vanishing point in the source image.

[0031] The method according to the first aspect of the present disclosure comprises: re-estimating the vanishing point based on the plurality of ground lines along the road.

[0032] According to the method of the first aspect of the present disclosure, selecting the plurality of ground lines comprises: detecting a plurality of detected line segments in the source image, detecting a horizon in the source image, and determining a lane mask for the source image.

[0033] The method according to the first aspect of the present disclosure, wherein selecting the plurality of ground lines comprises: eliminating a group of the plurality of detected line segments appearing above the horizon in the source image.

[0034] The method according to the first aspect of the present disclosure, wherein selecting the plurality of ground lines comprises: selecting a group of the plurality of detected line segments adjacent to lane markings in a lane mask.

[0035] According to the method of the first aspect of the present disclosure, performing lane line detection based on aggregation comprises: grouping multiple detected line segments into individual groups based on the distance of each detected line segment in each individual group from the optimal lane line passing through the vanishing point.

[0036] The method according to the first aspect of the present disclosure, wherein the pitch and yaw of the vehicle are estimated from the source image based on comparing the positions of the plurality of lane lines with the vanishing point.

[0037] The method according to the first aspect of the present disclosure, wherein the roll of the vehicle is estimated from the source image when the plurality of lane markings include three separate lane markings.

[0038] According to the method of the first aspect of the present disclosure, estimating the roll angle of the vehicle from the source image occurs when the plurality of lane lines include at least two lane lines of a given width.

[0039] A second aspect of the present disclosure provides a non-transitory computer-readable storage medium containing programming instructions, which, when executed by a processor, are operable to perform a method, the method comprising:

[0040] determining whether a predetermined set of enabling conditions have occurred along the road;

[0041] estimating a location of a vanishing point in a source image including the road;

[0042] selecting a plurality of ground lines along a road based on the source image;

[0043] performing lane line detection based on the aggregation of the plurality of ground lines to determine a plurality of lane lines in the source image;

[0044] estimating at least one of pitch, yaw or roll of the vehicle from the source images;

[0045] Minimizing a cost function based on the estimation of pitch, yaw and roll to obtain an optimal pitch value, an optimal yaw value, and an optimal roll value, a lane line from the source image;

[0046] performing sliding window based refinement on the plurality of source images; and

[0047] The results of the sliding window refinement are broadcasted to downstream applications, or a determination is made based on the sliding window based refinement whether a camera system on the vehicle is misaligned.

[0048] The non-transitory computer-readable storage medium according to the second aspect of the present disclosure, wherein the source image is captured by at least one optical sensor on a vehicle.

[0049] According to the non-transitory computer-readable storage medium described in the second aspect of the present disclosure, at least one enabling condition includes: the speed along the first axis is greater than a predetermined value, the speed along the second axis is less than a second predetermined value, the acceleration along the first axis is within a predetermined range, the steering angle is less than a predetermined value, and the distance between key frames is greater than a predetermined distance value.

[0050] The non-transitory computer-readable storage medium according to the second aspect of the present disclosure, wherein the vanishing point in the source image is estimated based on detecting a plurality of detected line segments having convergence points corresponding to the vanishing point in the source image.

[0051] The non-transitory computer-readable storage medium according to the second aspect of the present disclosure includes: re-estimating the vanishing point based on the plurality of ground lines along the road.

[0052] According to the non-transitory computer-readable storage medium of the second aspect of the present disclosure, selecting the plurality of ground lines comprises:

[0053] detecting a plurality of detected line segments in the source image, detecting a horizon in the source image, and determining a lane mask for the source image;

[0054] eliminating a set of the plurality of detected line segments that appear above the horizon in the source image; and

[0055] A group of the plurality of detected line segments that are adjacent to lane markings in the lane mask is selected.

[0056] A third aspect of the present disclosure provides a vehicle system, comprising:

[0057] at least one optical sensor configured to capture a plurality of images;

[0058] a controller in communication with the at least one optical sensor, wherein the controller is configured to:

[0059] determining whether a predetermined set of enabling conditions have occurred along the road;

[0060] estimating a location of a vanishing point in a source image including the road;

[0061] selecting a plurality of ground lines along the road based on the source image;

[0062] performing lane line detection based on the aggregation of the plurality of ground lines to determine a plurality of lane lines in the source image;

[0063] estimating at least one of pitch, yaw or roll of the vehicle from the source images;

[0064] minimizing a cost function based on the estimation of pitch, yaw and roll to obtain an optimal pitch value, an optimal yaw value, an optimal roll value and a lane line from the source image;

[0065] performing sliding window based refinement on the plurality of images; and

[0066] The results of the sliding window refinement are broadcasted to downstream applications, or a determination is made based on the sliding window based refinement whether a camera system on a vehicle is misaligned. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 is a schematic illustration of an example motor vehicle.

[0068] Figure 2 Alignment diagram Figure 1 An example method for a sensor on a motor vehicle.

[0069] Figure 3 Pictured Figure 2A graphical representation of a portion of the method.

[0070] Figure 4 The diagram shows Figure 2 A portion of the method is used to determine the aggregation of detected line segments of lane lines.

[0071] Figure 5 Illustration of lane lines converging to a common vanishing point.

[0072] Figure 6 Lane markings having a given width between adjacent lane markings are illustrated.

[0073] Figure 7 The diagram shows Figure 2 An extended explanation of the method in the box.

[0074] The present disclosure may be modified or implemented in alternative forms, wherein representative embodiments are shown in the drawings and described in detail below. The present disclosure is not limited to the disclosed embodiments. On the contrary, the present disclosure is intended to cover alternatives that fall within the scope of the present disclosure defined by the appended claims.

[0075] Detailed description

[0076] Those of ordinary skill in the art will recognize that terms such as "above," "below," "upward," "downward," "top," "bottom," "left," "right," and the like are used descriptively with respect to the drawings and are not intended to limit the scope of the present disclosure as defined by the appended claims. Furthermore, the teachings may be described herein in terms of functional and / or logical block components and / or various processing steps. It should be recognized that such block components may include a plurality of hardware, software, and / or firmware components configured to perform the specified functions.

[0077] Referring to the drawings, wherein like reference numerals designate like parts throughout the drawings, wherein like reference numerals designate like parts, Figure 1 A schematic diagram of a motor vehicle 10 is shown positioned relative to a road, such as a vehicle lane 12. Figure 1 As shown in FIG. 1 , the vehicle 10 includes a body 14, a first axle having a first set of wheels 16-1, 16-2, and a second axle having a second set of wheels 16-3, 16-4 (such as individual left and right wheels on each axle). Each of the wheels 16-1, 16-2, 16-3, 16-4 employs a tire that is configured to provide frictional contact with the vehicle lane 12. Although two axles and corresponding wheels 16-1, 16-2, 16-3, 16-4 are specifically shown, it is not excluded that the motor vehicle 10 has additional axles.

[0078] like Figure 1As shown in FIG. 1 , the vehicle suspension system operatively connects the vehicle body 14 to each set of wheels 16-1, 16-2, 16-3, 16-4 for maintaining contact between the wheels and the vehicle lane 12 and maintaining the handling of the motor vehicle 10. The motor vehicle 10 also includes a drive system 20 having one or more power sources 20A, which may be an internal combustion engine (ICE), an electric motor, or a combination of these devices, and the power source 20A is configured to transmit drive torque to the wheels 16-1, 16-2 and / or the wheels 16-3, 16-4. The motor vehicle 10 also employs a vehicle operating or control system, which includes: devices such as one or more steering actuators 22 (e.g., an electric power steering unit) configured to steer the wheels 16-1, 16-2 at a steering angle (θ); an accelerator device 23 for controlling the power output of (one or more) power sources 20A; a brake switch or device 24 for slowing the rotation of the wheels 16-1 and 16-2 (such as via individual friction brakes located at the corresponding wheels), etc.

[0079] like Figure 1 As shown in FIG. 1 , the motor vehicle 10 includes at least one sensor 25A and an electronic controller 26 that cooperate during certain circumstances to at least partially control, guide, and steer the vehicle 10 in an autonomous mode. Therefore, the vehicle 10 may be referred to as an autonomous vehicle. To achieve efficient and reliable autonomous vehicle control, the electronic controller 26 may be in operative communication with a steering actuator (or actuators) 22, an accelerator device 23, and a brake device 24 configured as an electric power steering unit. The sensor 25A of the motor vehicle 10 is operable to sense the vehicle lane 12 and monitor the surrounding geographic area and traffic conditions near the motor vehicle 10.

[0080] The sensors 25A of the vehicle 10 may include, but are not limited to, at least one of: a light detection and ranging (LiDAR) sensor, a radar, and a camera system (such as an optical sensor) located around the vehicle 10 to detect boundary indicators, such as edge conditions, of the vehicle lane 12. The types of sensors 25A, their locations on the vehicle 10, and their operation for detecting and / or sensing boundary indicators of the vehicle lane 12 and monitoring the surrounding geographic area and traffic conditions are understood by those skilled in the art and are therefore not described in detail herein.

[0081] The electronic controller 26 is configured to communicate with the sensors 25A of the vehicle 10 to receive their respective sensed data related to the detection or sensing of the vehicle lane 12 and the monitoring of the surrounding geographic area and traffic conditions. The electronic controller 26 may alternatively be referred to as a control module, a control unit, a controller, a vehicle 10 controller, a computer, etc. The electronic controller 26 may include a computer and / or processor 28 and include software, hardware, memory, algorithms, connections (such as connections to the sensors 25A), etc. for managing and controlling the operation of the vehicle 10. Therefore, the electronic controller 26 described below and described in detail below may be referred to as a control module, a control unit, a controller, a vehicle 10 controller, a computer, etc. Figure 2 The method generally represented in FIG. 1 may be embodied as part of a program or algorithm operable on the electronic controller 26. It should be understood that the electronic controller 26 may include devices capable of analyzing data from the sensors 25A, comparing the data, making decisions necessary to control the operation of the vehicle 10, and performing tasks necessary to control the operation of the vehicle 10.

[0082] The electronic controller 26 may be embodied as one or more digital computers or mainframes, each having one or more processors 28, read-only memory (ROM), random access memory (RAM), electrically programmable read-only memory (EPROM), optical drives, magnetic drives, etc., high-speed clocks, analog-to-digital (A / D) circuits, digital-to-analog (D / A) circuits, and input / output (I / O) circuits, I / O devices, and communication interfaces, as well as signal conditioning and buffering electronics. Computer-readable memory may include non-temporary / tangible media that participate in providing data or computer-readable instructions. The memory may be non-volatile or volatile. Non-volatile media may include, for example, optical or magnetic disks and other persistent memories. Example volatile media may include dynamic random access memory (DRAM), which may constitute main memory. Other examples of embodiments of memory include floppy disks, hard disks, tapes or other magnetic media, CD-ROMs, DVDs and / or other optical media, and other possible memory devices, such as flash memory.

[0083] The electronic controller 26 includes a tangible, non-transitory memory 30 having recorded thereon computer executable instructions including one or more algorithms for regulating the operation of the motor vehicle 10. The subject algorithm(s) may specifically include an algorithm configured to monitor the positioning of the motor vehicle 10 and determine the vehicle's heading relative to a mapped vehicle trajectory on a particular road route, as will be described in detail below.

[0084] The motor vehicle 10 also includes a vehicle navigation system 34, which can be part of the integrated vehicle controls or an additional device for finding directions in the vehicle. The vehicle navigation system 34 is also operably connected to a global positioning system (GPS) 36 using earth orbit satellites. The vehicle navigation system 34 connected to the GPS 36 and the above-mentioned sensor 25A can be used for the automation of the vehicle 10. The electronic controller 26 communicates with the GPS 36 via the vehicle navigation system 34. The vehicle navigation system 34 receives its position data from the GPS 36 using a satellite navigation device (not shown), and then associates the position data with the position of the vehicle relative to the surrounding geographic area. Based on such information, when directions to a specific waypoint are needed, a route to such a destination can be mapped and calculated. Dynamic terrain and / or traffic information can be used to adjust the route. The current position of the vehicle 10 can be calculated via dead reckoning-by using a previously determined position and advancing the position based on the speed given or estimated by discrete control points on the time and route passed.

[0085] The electronic controller 26 is typically configured (ie, programmed) to determine or identify the position 38 (current position in the XY plane, Figure 1 ), speed, acceleration, yaw rate, and intended path 40, and heading 42. Position 38, intended path 40, and heading 42 of motor vehicle 10 may be determined via navigation system 34 receiving data from GPS 36, while speed, acceleration (including longitudinal and lateral g), and yaw rate may be determined by vehicle sensors 25A. Alternatively, electronic controller 26 may determine vehicle position 38 relative to vehicle lane 12 using other systems or detection sources (e.g., cameras) remotely disposed relative to vehicle 10.

[0086] As described above, the motor vehicle 10 can be configured to operate in an autonomous mode directed by the electronic controller 26 to transport the occupants 62. In this mode, the electronic controller 26 can also obtain data from the vehicle sensors 25A to guide the vehicle along the desired path 40, such as via adjusting the steering actuator 22. The electronic controller 26 can be additionally programmed to detect and monitor the steering angle (θ) of the steering actuator(s) 22 along the desired path 40, such as during a negotiated turn. Specifically, the electronic controller 26 can be programmed to receive and process data from the steering position sensor 44 ( Figure 1 The steering position sensor 44 communicates with the steering actuator(s) 22, the accelerator device 23, and the brake device 24 to determine the steering angle (θ) based on data signals from the steering position sensor 44 (shown in FIG. 1 ).

[0087] like Figure 2A flow chart of a method 100 for performing alignment of a camera system with the ground is illustrated. The method 100 begins at block 102 by determining whether at least one enabling condition or a predetermined set of enabling conditions is satisfied. The enabling condition may include at least one of the following: a vehicle velocity norm along the ground x-axis is greater than a predetermined velocity (||v x ||>v u ), the vehicle velocity norm along the ground y-axis is less than the predetermined velocity (||v y ||<v l ), the acceleration along the ground x-axis is within the predetermined range (a v <||a x ||<a u ), the steering angle is less than the predetermined value (||θ||<θ u ) or the distance between key frames is greater than a predetermined value (||v||Δt>w u ). In addition, the enabling conditions may include a vehicle exit point being closed or a time comparison between lane mask images to ensure straight lane driving. The enabling conditions may also be non-rainy conditions, low light, snow, fog, or a spare tire. In addition, in one example, each of the above enabling conditions may need to be met in order to proceed to box 104.

[0088] At block 104, method 100 performs initial vanishing point detection and selection, such as Figure 2-3 as shown in . Figure 3 A graphical representation of a source image 200 is provided moving from block 102 through blocks 104, 106, and 108. The initial vanishing point detection will be re-estimated or refined at block 106, as discussed in more detail below. At block 102, a source image 200 is obtained for analysis, such as Figure 2-3 The analysis includes: performing line segment detection on the source image 200 at block 104 to obtain a line segment image 202 with detected line segments overlaid on the source image 200, the line segment image 202 having a plurality of detected line segments 204 identified from the source image 200. Block 104 uses the detected line segments 204 to determine the position of an initial vanishing point 206 in an initial vanishing point detection image 208. Specifically, the initial vanishing point 206 is located in the initial vanishing point detection image 208 at a position adjacent to the end point of the detected line segment 204. The following equation 1 provides the uncertainty of the vanishing point.

[0089]

[0090] In Equation 1 above, p v and are provided in Equation 2 and Equation 3, respectively.

[0091]

[0092]

[0093] Moreover, cov([α, β, γ, t3] T ) is provided by the manufacturing alignment of a camera system forming at least a portion of the sensor 25A. Using the above information, the vanishing point is selected by determining a point that satisfies equation 4 below.

[0094]

[0095] In the above equation, Disappearing direction v = [1 0 0] T , t is the ground to camera translation, t i is the i-th element of t, F -1 (x) is the inverse cumulative probability function of the χ2 distribution with 2 degrees of freedom, K is the camera intrinsic matrix, α is the roll angle, β is the pitch angle, γ is the yaw angle, and r i is the i-th column of R(α, β, γ), where R(α, β, γ) is represented in Equation 5 below.

[0096]

[0097] At block 108, ground line selection occurs. Ground line selection is as follows Figure 2-3 By using the initial vanishing point 206 from block 104, Figure 3 The lane segmentation occurs at frame 110 by removing the non-ground line segments shown by the detected line segments 204 of image 202 from the initial vanishing point detection image 208 using the horizon 210 from image 212. h Determined by the following two points in the image (r ij is the element in the i-th row and j-th column of R(α,β,γ).

[0098] If r 31 ≠0 and r 32 ≠0, then and is provided by equations 6 and 7 below.

[0099]

[0100]

[0101] If r 31 = 0 and r 32 ≠0, then and is provided by equations 8 and 9 below.

[0102]

[0103]

[0104] If r 31 ≠0 and r 32 =0, then and It is provided by equations 10 and 11 below.

[0105]

[0106]

[0107] In addition, if r 31 and r 32 If both are equal to 0, the camera system cannot observe the ground 12. Select The refined vanishing point 222 in the refined image 224 is re-estimated at block 106 given the ground line segment by using the line segment with the endpoint p of the ground line. The ground line segment is selected as shown in Equation 12 below (a line segment l close to the lane mask is selected), where m is a pixel considered to be a lane, M is a set of lane pixels, and ε is a predetermined threshold variable.

[0108]

[0109] At block 112, line segments are clustered into different groups, wherein the detected line segments 204 in the same group belong to the same lane line l. Figure 4 As shown in FIG. 1 , the first group of line segments 204-1 are clustered around line l1, the second group of line segments 204-2 are clustered around line l2, and the third group of line segments 204-3 are clustered around line l3, so that the line segments 204 are associated with the corresponding line l. The line segments 204 as outliers are eliminated as noise.

[0110] For lane detection, such as Figure 5 As shown in , lane lines are generated for each line segment in the same group. For each line with the same label in the set L = ∪l, the best lane line Should be passed from having The ground line segment re-estimates the vanishing point p v And the distance with the line segments in the set L is the smallest. Next, equation 13 is used to obtain the optimal lane line for the line segments with the same label.

[0111]

[0112] At box 114, the pitch (β) and yaw (γ) of the vehicle 10 are estimated. Given equation 5 above, and the vanishing direction v = [1 0 0] T , pitch (β) and yaw (γ) are obtained from equations 14 and 15 below, where R is the rotation matrix from the ground to the camera, K is the camera intrinsic matrix, and λ is the scaling factor.

[0113] p v =λKRv Equation 14

[0114]

[0115] At box 116, the method 100 then determines whether enough lanes have been detected. If enough lanes have been detected, the method 100 proceeds to box 120 to perform roll (α) angle estimation. If not enough lanes have been detected, the method 100 proceeds to box 118. At box 118, the method 100 determines whether the pitch and yaw estimates are reliable. If it is determined that the pitch and yaw estimates are reliable, the method 100 returns to box 102 to determine whether the enabling conditions are met. If the pitch and yaw estimates are reliable, the method 100 proceeds to box 124 to perform a sliding window-based optimization to further refine the pitch and yaw estimates, as will be discussed further below.

[0116] When the method proceeds from box 116 to box 120, the method 100 performs roll (α) angle estimation. The method 100 estimates the roll (α) angle from the source image 200 through the lane lines that have been detected as described above. In one example, the following Equation 16 leads to the development of Equation 17.

[0117]

[0118]

[0119] In Equation 16 above, r i is the ith column of R, t is the ground-to-camera translation vector, and W is the lane width. In the case with Equation 17, if there are three separate lane lines, such as Figure 5 As shown in , block 120 may determine the roll (α) angle from a single source image 200 using Equation 18, or if only dual lane lines with a given lane width between adjacent lane lines are detected, as Figure 6 As shown in , equation 19 is used.

[0120]

[0121]

[0122] From block 120, the method 100 proceeds to block 122 to determine at least one alignment parameter from a single source image 200. To obtain lane lines in vehicle coordinates, the method 100 minimizes the cost equation for obtaining an initial estimate of (α, β, γ) in equation 20 below, thereby obtaining the optimal α in a single image. k , β k , and γ k When the method 100 identifies a lane line parallel to the direction of travel of the vehicle 20 ( Figure 5 ), the method 100 utilizes the following equation 20. When the method 100 identifies lane lines with equal lane widths ( Figure 6 ), method 100 may utilize either of Equations 21 and 22 below, where ω0 is a weight factor and W is the lane width.

[0123]

[0124]

[0125]

[0126] From block 122, method 100 proceeds to block 124 to perform a sliding window based optimization on a plurality of images, such as a plurality of source images. In a first example, roll, pitch, and yaw parameters are refined in a sliding window by minimizing Equation 23 below.

[0127]

[0128] In a second example, minimization or optimization may occur for roll, pitch, yaw, and ground to camera center height (t3), and replacing f with the following equation 24: l f in (α, β, γ, i) w (α, β, γ, i), where Consider t3 is unknown and W is the lane width.

[0129]

[0130] In addition, update as follows: X k+1 =X k -(J T J+λdiag(J T J)) -1 J T f s , for case 1, X k =[α, β, γ] T , otherwise for the second case it is equal to [α, β, γ, t3] T and In the above equation, λ is the damping factor, ω1 is the weighting factor, and t3 is the third element of t (ground to camera center height). The lane width W can be obtained from a map or another active sensor (such as LiDAR with sensor 25A), and 1 is the 3x3 identity matrix.

[0131] At block 126, the method 100 detects misalignment, matures alignment parameters, and updates a coordinated transformation matrix (CTM) for use by downstream applications. The CTM stores alignment results for characterizing a transformation from one coordinate system to another, such as a camera coordinate system to a vehicle coordinate system. In one example, the downstream application may include at least one of a perception-based application, a low-speed maneuvering (LVM) application (such as automated parking or exiting), or a viewing application. Figure 7 Additional example implementations of the steps occurring at block 126 are provided.

[0132] In one example, by using the alignment parameters determined from the sliding window-based optimization at box 124, misalignment detection can be performed by alignment via parallelism and an additional alignment method (such as feature-based alignment). The additional alignment method is described in detail in co-owned, co-pending U.S. patent application serial number 17 / 651,407, U.S. patent application 17 / 651,405, U.S. patent application 17 / 651,406, and U.S. Patent No. 8,373,763, the disclosures of which are incorporated herein by reference in their entirety. Then, at box 302, the alignment via parallelism and the additional alignment method are compared with the previous CTM. From box 302, the method 100 determines at box 304 whether the comparison with at least one of the above methods is greater than a predetermined threshold. If the difference between at least one of the above methods is less than the predetermined threshold, the method 100 proceeds to box 306 and updates the alignment and publishes the CTM. The downstream application can then use this information.

[0133] If the difference between the method and the CTM is greater than a predetermined threshold, the method 100 proceeds to block 308 within block 126, as shown in FIG. Figure 7 . In block 308, a time comparison occurs to determine misalignment. If misalignment is not determined, the method 100 returns to block 302. If misalignment is determined in block 310, the method 100 proceeds to block 312 and determines whether misalignment or alignment should be broadcast. If misalignment or alignment should be broadcast, the method 100 may set a misalignment flag. If misalignment should not be broadcast, the method 100 proceeds to block 314 and determines that the method 100 should stop and exit. If the method 100 should stop and exit, the method 100 clears the variables and reports the status. If not, the method 100 proceeds to block 302.

[0134] The terms "one" and "an" do not indicate a quantitative limitation, but rather indicate the presence of at least one of the items mentioned. The term "or" means "and / or", unless the context clearly indicates otherwise. References to "an aspect" throughout the specification mean that a particular element (e.g., a feature, structure, step, or characteristic) described in conjunction with that aspect is included in at least one aspect described herein, and may or may not be present in other aspects. In addition, it should be understood that the described elements may be combined in a suitable manner in each aspect.

[0135] When an element such as a layer, film, region, or substrate is referred to as being "on" another element, it can be directly on the other element or intervening elements may also be present. In contrast, when an element is referred to as being "directly on" another element, there are no intervening elements present.

[0136] Unless otherwise specified herein, the test standard is the most recent standard in effect as of the filing date of this application (or, if priority is claimed, the filing date of the earliest priority application in which the test standard appears).

[0137] Unless defined otherwise, technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0138] Although the above disclosure has been described with reference to exemplary embodiments, it will be appreciated by those skilled in the art that various changes may be made and that elements thereof may be substituted with equivalents without departing from the scope thereof. In addition, many modifications may be made to adapt specific circumstances or materials to the teachings of the disclosure without departing from the scope of the disclosure. Therefore, the disclosure is not intended to be limited to the specific embodiments disclosed, but will include embodiments falling within its scope.

Claims

1. A method for performing camera-to-ground alignment on a camera system on a vehicle, the method comprising: determining whether a predetermined set of enabling conditions have occurred along the road; estimating a location of a vanishing point in a source image including the road; selecting a plurality of ground lines along the road based on the source image; performing lane line detection based on the aggregation of the plurality of ground lines to determine a plurality of lane lines in the source image; estimating at least one of pitch, yaw or roll of the vehicle from the source images; minimizing a cost function based on the estimation of pitch, yaw and roll to obtain an optimal pitch value, an optimal yaw value, an optimal roll value and a lane line from the source image; performing sliding window based refinement on the plurality of source images; as well as The results of the sliding window refinement are broadcasted to downstream applications, or a determination is made as to whether a camera system on the vehicle is misaligned based on the sliding window based refinement. 2 . The method of claim 1 , wherein the source image is captured by at least one optical sensor on a vehicle.

3. The method of claim 1, wherein the predetermined set of enabling conditions comprises: The speed along the first axis is greater than a predetermined value, the speed along the second axis is less than a second predetermined value, and the acceleration along the first axis is within a predetermined range.

4. The method according to claim 3, wherein the predetermined set of start conditions comprises: The steering angle is less than a predetermined value and the distance between the key frames is greater than a predetermined distance value. 5 . The method of claim 1 , wherein a vanishing point in the source image is estimated based on detecting a plurality of detected line segments in the source image having convergence points corresponding to the vanishing point.

6. The method according to claim 5, comprising: The vanishing point is re-estimated based on the plurality of ground lines along the road.

7. The method of claim 1 , wherein selecting the plurality of ground lines comprises: A plurality of detected line segments are detected in the source image, a horizon is detected in the source image, and a lane mask is determined for the source image.

8. The method of claim 7, wherein selecting the plurality of ground lines comprises: A group of the plurality of detected line segments that appear above the horizon in the source image is eliminated.

9. The method of claim 8, wherein selecting the plurality of ground lines comprises: A group of the plurality of detected line segments that are adjacent to lane markings in the lane mask is selected.

10. The method of claim 1, wherein performing lane line detection based on aggregation comprises: The plurality of detected line segments are grouped into individual groups based on a distance of each detected line segment in each individual group from an optimal lane line passing through the vanishing point.

Images