Monitoring system for a space

Abstract

A monitoring system for a space is disclosed. A vehicle control system including the monitoring system for the space includes on-board cameras that capture images from which a plurality of three-dimensional (3D) points are recovered. A left ground plane normal vector is determined for a left image, a center ground plane normal vector is determined for a front image, and a right ground plane normal vector is determined for a right image. A first angular difference between the left ground plane normal vector and the center ground plane normal vector is determined, and a second angular difference between the right ground plane normal vector and the center ground plane normal vector is determined. An uneven ground surface is determined based on one of the first angular difference or the second angular difference, and an alignment compensation factor for the left camera or the right camera is determined. An overhead view image is determined based on the alignment compensation factor.

Description

Technical Field

[0001] The present invention relates to a vehicle comprising a spatial monitoring system for detecting uneven road surfaces and, based thereon, dynamically adjusting or otherwise compensating for camera alignment and / or vehicle control. Background Technology

[0002] The vehicle may include onboard cameras for monitoring the environment near the vehicle during operation to operate advanced driver assistance systems (ADAS) and / or autonomous vehicle functions. Proper alignment of one or more onboard cameras with respect to a reference point (such as the ground) is essential for the operation of bird's-eye view imaging systems, lane sensing, autonomous vehicle control, etc. The presence of uneven road surfaces can degrade the performance of spatial monitoring systems and autonomous vehicle control systems due to their impact on camera-to-ground alignment.

[0003] Therefore, a method, system, and device are needed to detect uneven road surfaces and dynamically adjust or otherwise compensate for camera alignment in response. Summary of the Invention

[0004] The concept described herein provides a vehicle that includes a spatial monitoring system for detecting uneven road surfaces and, based on this, dynamically adjusting or otherwise compensating for camera alignment and / or vehicle control.

[0005] One aspect of this disclosure includes a vehicle having a space monitoring system comprising a plurality of on-vehicle cameras in communication with a controller. The plurality of on-vehicle cameras include a front camera arranged to capture a front image of a forward field of view (FOV), a rear camera arranged to capture a rear image of a rearward FOV, a left camera arranged to capture a left image of a left-facing FOV, and a right camera arranged to capture a right image of a right-facing FOV. The controller includes a set of instructions executable to: simultaneously capture a front image from the front camera, a rear image from the rear camera, a left image from the left camera, and a right image from the right camera; and recover a plurality of three-dimensional (3D) points from the front and rear images. Based on the plurality of 3D points from one of the front or rear images, a left ground plane normal vector is determined for the left image, a central ground plane normal vector is determined from the front image, and a right ground plane normal vector is determined from the right image. A first angular difference between the left ground plane normal vector and the central ground plane normal vector is determined, and a second angular difference between the right ground plane normal vector and the central ground plane normal vector is determined. An uneven surface is determined based on either a first angular difference or a second angular difference, and an alignment compensation factor for either the left or right camera is determined based on this uneven surface. A bird's-eye view image is generated based on the alignment compensation factor.

[0006] Another aspect of this disclosure includes an autonomous vehicle control system capable of autonomously controlling one of the steering, acceleration, or braking systems based on bird's-eye view images.

[0007] Another aspect of this disclosure includes the fact that the front image, rear image, left image, and right image are 2D fisheye images.

[0008] Another aspect of this disclosure includes an instruction set executable to recover the plurality of three-dimensional (3D) points from a 2D fisheye image using structural frommotion analysis.

[0009] Another aspect of this disclosure includes an instruction set executable to determine ground plane normal vectors for the left, front, and right images.

[0010] Another aspect of this disclosure includes an instruction set executable to use feature detection and matching routines to find feature matching pairs in an overlapping region between the front and left images.

[0011] Another aspect of this disclosure includes instruction set executable to determine a fundamental matrix and a rotation transformation matrix based on feature matching pairs, wherein the rotation transformation matrix includes an alignment compensation factor.

[0012] Another aspect of this disclosure includes an instruction set executable to: determine the left camera ground plane normal vector based on the rotation transformation matrix and the center camera normal matrix; and align the left camera to the ground using the left camera ground plane normal vector.

[0013] Another aspect of this disclosure includes an instruction set executable to generate bird's-eye view images based on the alignment of the left camera to the ground.

[0014] Another aspect of this disclosure includes an instruction set executable to: determine a forward motion vector, a left motion vector, a ground plane normal vector from the left region, and an original ground plane normal vector; and determine a rotation transformation matrix that minimizes the loss based on the relationship between the forward motion vector, the left motion vector, the ground plane normal vector from the left region, and the original ground plane normal vector.

[0015] Another aspect of this disclosure includes an instruction set executable to generate a bird's-eye view image based on a front image, a left image, a right image, a rear image, and an alignment compensation factor.

[0016] Another aspect of this disclosure includes a vehicle having a space monitoring system with multiple on-vehicle cameras communicating with a controller. The multiple on-vehicle cameras include a front camera arranged to capture a forward image of the forward field of view (FOV) and a left camera arranged to capture a left image of the left FOV. The controller includes a set of instructions executable to simultaneously capture the front and left images and recover multiple three-dimensional (3D) points from the left image. A left ground plane normal vector is determined for the left image based on the near region of the multiple 3D points from the left camera, and a distance to the left ground plane normal vector is determined for each of the 3D points. The presence of a left-side upper curb (or "roadside curb") is determined based on the distance to the left ground plane normal vector for each of the 3D points.

[0017] Another aspect of this disclosure includes a vehicle with an autonomous vehicle control system that is capable of autonomously controlling one of the steering, acceleration, or braking systems based on the presence of a left-side curb.

[0018] Another aspect of this disclosure includes a vehicle having a space monitoring system comprising a Light Detection and Ranging (LiDAR) device and a controller, the LiDAR device being arranged to capture data representing the forward field of view (FOV), rearward FOV, leftward FOV, and rightward FOV. The controller includes a set of instructions executable to: capture multiple images from the LiDAR device; and determine a left image, a front image, and a right image based on the multiple images from the LiDAR device. Based on the multiple images from the LiDAR device, determine a left ground plane normal vector for the left image, a center ground plane normal vector from the front image, and a right ground plane normal vector from the right image. Determine a first angular difference between the left ground plane normal vector and the center ground plane normal vector, and determine a second angular difference between the right ground plane normal vector and the center ground plane normal vector. Detect an uneven ground surface based on one of the first or second angular differences, and determine an alignment compensation factor based on the uneven ground surface. Determine a bird's-eye view image based on the alignment compensation factor and the uneven ground surface.

[0019] The present invention also discloses the following technical solutions:

[0020] Option 1. A vehicle comprising:

[0021] A space monitoring system having multiple on-vehicle cameras that communicate with a controller, the multiple on-vehicle cameras including a front camera arranged to capture a front image of the forward field of view (FOV), a rear camera arranged to capture a rear image of the rearward FOV, a left camera arranged to capture a left image of the leftward FOV, and a right camera arranged to capture a right image of the rightward FOV.

[0022] The controller includes an instruction set capable of executing:

[0023] Simultaneously capture the front image from the front camera, the rear image from the rear camera, the left image from the left camera, and the right image from the right camera;

[0024] Recover multiple three-dimensional (3D) points from the front and back images;

[0025] Based on the plurality of 3D points from one of the previous image or the subsequent image, a left ground plane normal vector is determined for the left image, a center ground plane normal vector is determined from the previous image, and a right ground plane normal vector is determined from the right image;

[0026] Determine the first angular difference between the left ground plane normal vector and the central ground plane normal vector;

[0027] Determine the second angle difference between the right ground plane normal vector and the center ground plane normal vector;

[0028] Uneven ground surfaces are detected based on either the first angle difference or the second angle difference;

[0029] The alignment compensation factor for one of the left or right cameras is determined based on the uneven terrain surface; and

[0030] A bird's-eye view image is generated based on the alignment compensation factor.

[0031] Option 2. The vehicle according to claim 1, further comprising:

[0032] An autonomous vehicle control system is capable of autonomously controlling one of the steering, acceleration, or braking systems.

[0033] The autonomous vehicle control system controls one of the steering system, the acceleration system, or the braking system based on the bird's-eye view image.

[0034] Option 3. The vehicle according to claim 1, wherein the front image, the rear image, the left image, and the right image comprise 2D fisheye images.

[0035] Option 4. The vehicle of claim 3, wherein the instruction set is executable to recover the plurality of 3D points from the 2D fisheye image using motion structure analysis.

[0036] Option 5. The vehicle according to claim 1, wherein the instruction set is executable to determine the ground plane normal vectors of the left image, the front image, and the right image.

[0037] Option 6. The vehicle of claim 1, wherein the set of instructions capable of executing to determine the alignment compensation factor of the left camera based on the uneven ground surface includes a set of instructions capable of performing the following operations:

[0038] Feature detection and matching routines are used to find feature matching pairs in the overlapping region between the front image and the left image.

[0039] Option 7. The vehicle of claim 6, further comprising a set of instructions capable of executing a base matrix and a rotation transformation matrix based on the feature matching pairs, wherein the rotation transformation matrix includes the alignment compensation factor.

[0040] Option 8. The vehicle of claim 7, further comprising a set of instructions capable of performing the following operations:

[0041] The left camera's ground plane normal vector is determined based on the rotation transformation matrix and the center camera's normal matrix; and

[0042] The left camera is aligned with the ground using the ground plane normal vector of the left camera.

[0043] Option 9. The vehicle of claim 8, further comprising a set of instructions capable of executing to generate the bird's-eye view image based on the alignment of the left camera to the ground.

[0044] Option 10. The vehicle of claim 7, wherein the instruction set capable of executing to determine the rotation transformation matrix includes an instruction set capable of performing the following operations:

[0045] Determine the forward motion vector, the left motion vector, the ground plane normal vector from the left-directing FOV, and the original ground plane normal vector; and

[0046] The rotation transformation matrix that minimizes the loss is determined based on the relationship between the forward motion vector, the left motion vector, the ground plane normal vector from the left FOV, and the original ground plane normal vector.

[0047] Solution 11. The vehicle of claim 1, further comprising a set of instructions capable of executing to generate the bird's-eye view image based on the front image, the left image, the right image, the rear image, and the alignment compensation factor.

[0048] Option 12. A vehicle comprising:

[0049] A space monitoring system having an on-vehicle camera that communicates with a controller, the on-vehicle camera being one of a front camera arranged to capture a front image of the forward field of view (FOV), a left camera arranged to capture a left image of the left FOV, a right camera arranged to capture a right image of the right FOV, or a rear camera arranged to capture a rear image of the rear FOV.

[0050] The controller includes an instruction set capable of executing:

[0051] Images are captured from the onboard camera;

[0052] Recover multiple three-dimensional (3D) points from the image;

[0053] The ground plane normal vector is determined for the image based on the near region of the plurality of 3D points from the left camera;

[0054] For each of the 3D points, determine the distance to the normal vector of the ground plane; and

[0055] The presence of curb stones in the corresponding FOV is detected based on the distance to the left ground plane normal vector for each of the 3D points.

[0056] Option 13. The vehicle according to claim 12, further comprising:

[0057] The vehicle includes an autonomous vehicle control system, which is capable of autonomously controlling one of the steering system, acceleration system, or braking system.

[0058] The autonomous vehicle control system controls one of the steering system, the acceleration system, or the braking system based on the presence of the curb in the FOV.

[0059] Option 14. The vehicle of claim 12, wherein the image comprises a 2D fisheye image.

[0060] Option 15. The vehicle of claim 14, wherein the instruction set is capable of: recovering the plurality of three-dimensional (3D) points from the 2D fisheye image of the image using motion structure analysis.

[0061] Option 16. The vehicle of claim 12, wherein the instruction set is executable to determine the ground plane normal vector of the image.

[0062] Option 17. A vehicle comprising:

[0063] A space monitoring system comprising a light detection and ranging (LiDAR) device and a controller, the LiDAR device being arranged to capture data representing the forward field of view (FOV), rearward field of view (FOV), leftward field of view (FOV), and rightward field of view (FOV);

[0064] The controller includes an instruction set capable of executing:

[0065] Multiple images were captured from the LiDAR device;

[0066] The left image, front image, and right image are determined based on the multiple images from the LiDAR device;

[0067] Based on the multiple images from the LiDAR device, a left ground plane normal vector is determined for the left image, a center ground plane normal vector is determined from the front image, and a right ground plane normal vector is determined from the right image;

[0068] Determine the first angular difference between the left ground plane normal vector and the central ground plane normal vector;

[0069] Determine the second angle difference between the right ground plane normal vector and the center ground plane normal vector;

[0070] Uneven ground surfaces are detected based on either the first angle difference or the second angle difference;

[0071] The alignment compensation factor is determined based on the uneven ground surface; and

[0072] A bird's-eye view image is generated based on the alignment compensation factor and the uneven ground surface.

[0073] Option 18. The vehicle according to claim 17, further comprising:

[0074] The vehicle includes an autonomous vehicle control system, which is capable of autonomously controlling one of the steering system, acceleration system, or braking system.

[0075] The autonomous vehicle control system controls one of the steering system, the acceleration system, or the braking system based on the bird's-eye view image.

[0076] Option 19. The vehicle of claim 17, wherein the instruction set is executable to determine the ground plane normal vectors of the left image, the front image, and the right image.

[0077] Option 20. The vehicle of claim 17, further comprising a set of instructions capable of performing the following operations:

[0078] The left ground plane normal vector is determined for the left image based on the near region of the plurality of 3D points from the left camera;

[0079] For each of the 3D points, determine the distance to the normal vector of the left ground plane; and

[0080] The presence of a curb in the left-hand FOV is determined based on the distance to the left ground plane normal vector for each of the 3D points.

[0081] The above features and advantages of this teaching, as well as other features and advantages, will readily become apparent from the following detailed description of some of the best modes and other embodiments for carrying out this teaching as defined in the appended claims when understood in conjunction with the accompanying drawings. Attached Figure Description

[0082] One or more embodiments will now be described by way of example with reference to the accompanying drawings, wherein:

[0083] Figure 1 A vehicle according to this disclosure is schematically shown, which includes a space monitoring system and an autonomous vehicle control system.

[0084] Figure 2 The illustrations depict, in a painterly manner, a top view and an associated bird's-eye view of a vehicle positioned on a ground surface according to the present disclosure.

[0085] Figure 3 The illustration schematically depicts a method according to the present disclosure for generating an area bird's-eye view image surrounding a vehicle in order to control its operation.

[0086] Figure 4 The method for recovering 3D points from each of the plurality of cameras according to the present disclosure is illustrated schematically.

[0087] Figure 5-1 The painting depicts a raw fisheye image captured by the front camera of a vehicle operating on the road, according to this disclosure.

[0088] Figure 5-2 The diagram illustrates, in graphical form, the relationship between the present disclosure and... Figure 5-1 The raw fisheye image captured by the vehicle's front camera is associated with a multiplicity of points.

[0089] Figure 5-3 The painting depicts a second image in the form of a dedistorted image according to the present disclosure, the second image having... Figure 5-1 Multiple vector points were extracted from continuous original fisheye images and used to depict the ground plane.

[0090] Figure 5-4The ground plane indicated in the xyz plane according to this disclosure is shown graphically, the ground plane from... Figure 5-1 Extracted from the original fisheye image.

[0091] Figure 6 The illustration schematically depicts details relating to the execution of a routine according to this disclosure, which is used to determine the left, front, right, and rear ground plane normal vectors, as well as the associated angles and the angular differences between them.

[0092] Figure 7 The original fisheye image captured by a front camera of a vehicle operating on a road surface, as described in the present disclosure, is depicted in a painterly manner, including a front image capture, a left image capture, and a right image capture identifiable thereon.

[0093] Figure 8 The illustration schematically depicts details relating to a first embodiment of an alignment compensation routine for one of the left or right cameras based on the presence of uneven ground, according to the present disclosure.

[0094] Figure 9 The illustration schematically depicts details relating to a second embodiment of an alignment compensation routine for one of the left or right cameras based on the presence of uneven ground, according to the present disclosure.

[0095] Figure 10 The illustration schematically depicts details related to the process for generating bird's-eye view images of uneven terrain, according to the present disclosure.

[0096] Figure 11 The illustration schematically depicts details of a process according to the present disclosure for determining left, front, and right ground plane normal vectors and associated angles, as well as angle differences, using information from LiDAR point clouds.

[0097] Figure 12 The illustration schematically depicts details related to a curb detection algorithm according to this disclosure for detecting the presence of curbs in a field of view (FOV).

[0098] It should be understood that the accompanying drawings are not necessarily drawn to scale and present slightly simplified representations of the various preferred features of the present disclosure as disclosed herein, including, for example, specific dimensions, orientations, positions, and shapes. Details associated with such features will be determined in part by the specific intended application and environment of use. Detailed Implementation

[0099] As described and illustrated herein, the components of the disclosed embodiments can be arranged and designed in a variety of different configurations. Therefore, the following detailed description is not intended to limit the scope of this disclosure as claimed, but only represents possible embodiments thereof. Furthermore, while numerous specific details are set forth in the following description to provide a thorough understanding of the embodiments disclosed herein, some embodiments may be practiced without some of these details. Moreover, for clarity, certain technical materials understood in the relevant art have not been described in detail to avoid unnecessarily obscuring this disclosure. Additionally, as illustrated and described herein, this disclosure may be practiced without elements not specifically disclosed herein.

[0100] Referring to the accompanying drawings, similar reference numerals throughout several figures correspond to similar or analogous parts, consistent with the embodiments disclosed herein. Figure 1 The illustration shows a top view of vehicle 10, which is positioned on ground surface 50 and has a space monitoring system 40 illustrating the concept described herein. In one embodiment, vehicle 10 also includes an autonomous vehicle control system 20. In one embodiment, vehicle 10 may include a four-wheeled passenger vehicle with steerable front wheels and fixed rear wheels. By way of non-limiting example, vehicle 10 may include a passenger vehicle, a light or heavy-duty truck, a multi-purpose vehicle, an agricultural vehicle, an industrial / warehouse vehicle, or a recreational off-road vehicle.

[0101] The space monitoring system 40 and the space monitoring controller 140 may include a controller that communicates with a plurality of space sensors 41 to monitor the field of view near the vehicle 10 and generate digital representations of these fields of view including nearby remote objects.

[0102] The space monitoring controller 140 can evaluate input from the space sensor 41 to determine the linear range, relative speed, and trajectory of the vehicle 10 with respect to each nearby remote object.

[0103] The space sensor 41 is located at various positions on the vehicle 10 and includes a front camera 42 capable of observing a forward field of view (FOV) 52, a rear camera 44 capable of observing a rearward FOV 54, a left camera 46 capable of observing a leftward FOV 56, and a right camera 48 capable of observing a rightward FOV 58. The front camera 42, rear camera 44, left camera 46, and right camera 48 are capable of capturing and pixelating 2D images of their respective FOVs. The forward FOV 52, rearward FOV 54, leftward FOV 56, and rightward FOV 58 overlap. The front camera 42, rear camera 44, left camera 46, and right camera 48 can utilize a fisheye lens to maximize the magnification of their respective FOV range. The space sensor 41 may further include a radar sensor and / or a LiDAR device 43, although this disclosure is not so limited.

[0104] The placement of the aforementioned spatial sensor 41 allows the spatial monitoring controller 140 to monitor traffic flow, including nearby vehicles, other objects around vehicle 10, and ground surface 50. Data generated by the spatial monitoring controller 140 can be used by a lane marking detection processor (not shown) to estimate the road. The spatial sensor 41 of the spatial monitoring system 40 may further include object localization sensing devices, including ranging sensors such as FM-CW (Frequency Modulated Continuous Wave) radar, pulse and FSK (Frequency Shift Keying) radar and LiDAR (Light Detection and Ranging) devices, and ultrasonic devices that rely on effects such as Doppler effect measurements to locate objects ahead. Possible object localization devices include charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) video image sensors, and other cameras / video image processors that utilize digital photography methods to 'observe' objects ahead, including one or more nearby vehicles. Such sensing systems are used to detect and locate objects in automotive applications and can be used with systems including, for example, adaptive cruise control, autonomous braking, autonomous steering, and side object detection.

[0105] The space sensor 41 associated with the space monitoring system 40 is preferably positioned in a relatively unobstructed location within the vehicle 10 to monitor the spatial environment. As used herein, the spatial environment includes all external elements, including: fixed objects such as signs, poles, trees, houses, shops, bridges, etc.; and moving or movable objects such as pedestrians and other vehicles. Overlapping the coverage areas of the space sensor 41 creates opportunities for sensor data fusion.

[0106] The autonomous vehicle control system 20 includes an onboard control system, such as an advanced driver assistance system (ADAS), capable of providing a certain level of driving automation. The terms 'driver' and 'operator' describe the person responsible for guiding the operation of the vehicle, whether actively participating in controlling one or more vehicle functions or guiding the operation of an autonomous vehicle. Driving automation can include a series of dynamic driving and vehicle operations. Driving automation can include a level of automatic control or intervention related to a single vehicle function, such as steering, acceleration, and / or braking, where the driver continuously has overall control of the vehicle. Driving automation can include a level of automatic control or intervention related to the simultaneous control of multiple vehicle functions, such as steering, acceleration, and / or braking, where the driver continuously has overall control of the vehicle. Driving automation can include simultaneous automatic control of vehicle driving functions, including steering, acceleration, and braking, where the driver relinquishes control of the vehicle for a period of time during the journey. Driving automation can include simultaneous automatic control of vehicle driving functions, including steering, acceleration, and braking, where the driver relinquishes control of the vehicle throughout the journey. Driving automation includes hardware and controllers configured to monitor the spatial environment in various driving modes to perform various driving tasks during dynamic operation. By way of non-limiting examples, driving automation may include cruise control, adaptive cruise control, lane change warning, intervention and control, automatic parking, acceleration, braking, etc.

[0107] The vehicle systems, subsystems, and controllers associated with the autonomous vehicle control system 20 are implemented to perform one or more operations associated with autonomous vehicle functions, including, by way of non-limiting example, adaptive cruise control (ACC) operation, lane guidance and lane keeping operation, lane changing operation, steering assist operation, object avoidance operation, parking assist operation, vehicle braking operation, vehicle speed and acceleration operation, and vehicle lateral movement operation, for example, as part of lane guidance, lane keeping, and lane changing operations. By way of non-limiting example, the vehicle systems and associated controllers of the autonomous vehicle control system 20 may include a drivetrain 32 and a drivetrain controller (PCM) 132 operatively connected to one or more of a steering system 34, a braking system 36, and a chassis system 38.

[0108] Each of the vehicle system and associated controllers may further include one or more subsystems and one or more associated controllers. For ease of description, these subsystems and controllers are shown as discrete elements. The foregoing classification of subsystems is provided to describe one embodiment and is illustrative. Other configurations may be considered within the scope of this disclosure. It should be understood that the functions described and performed by the discrete elements may be performed using one or more means, which may include algorithm code, calibration, hardware, application-specific integrated circuits (ASICs), and / or non-vehicle or cloud-based computing systems.

[0109] PCM 132 communicates with and is operatively connected to drivetrain 32, and executes control routines to control the operation of the engine and / or other torque machinery, transmission, and powertrain (none of which are shown) to transmit traction torque to the vehicle wheels in response to driver input, external conditions, and vehicle operating conditions. PCM 132 is shown as a single controller but may include multiple controller units that operate to control various powertrain actuators, including the engine, transmission, torque machinery, wheel motors, and other elements of drivetrain 32. By way of non-limiting example, drivetrain 32 may include an internal combustion engine and transmission (with associated engine controller and transmission controller). Furthermore, the internal combustion engine may include multiple discrete subsystems and individual controllers, including, for example, electronic throttle units and controllers, fuel injectors and controllers, etc. Drivetrain 32 may also consist of an electric motor / generator (with associated power inverter module and inverter controller). The control routines of PCM 132 may also include an adaptive cruise control system (ACC) that controls vehicle speed, acceleration, and braking in response to driver input and / or autonomous vehicle control input.

[0110] VCM 136 communicates with and is operatively connected to multiple vehicle operating systems, and executes control routines to control their operation. The vehicle operating systems may include braking, stability control, and steering, which may be controlled by actuators associated with braking system 36, chassis system 38, and steering system 34, respectively, and these actuators are controlled by VCM 136. VCM 136 is shown as a single controller, but may include multiple controller devices that operate to monitor the system and control various vehicle actuators.

[0111] Steering system 34 is configured to control the lateral movement of the vehicle. Steering system 34 may include an electric power steering (EPS) system coupled to an active front steering system to enhance or replace operator input via the steering wheel by controlling the steering angle of the steerable wheels of vehicle 10 during autonomous maneuvers, such as lane-changing maneuvers. An exemplary active front steering system allows the vehicle driver to perform primary steering operations, including enhanced steering wheel angle control to achieve a desired steering angle and / or vehicle yaw angle. Alternatively or additionally, the active front steering system may provide full autonomous control over the vehicle's steering functions. It should be understood that the systems described herein are adapted, with modifications, to vehicle steering control systems (such as electric power steering, four / rear-wheel steering systems) and direct yaw control systems that control the traction of each wheel to generate yaw motion.

[0112] The braking system 36 is configured to control vehicle braking and includes wheel braking devices such as disc brake elements, calipers, master cylinders, and brake actuators (e.g., pedals). Wheel speed sensors monitor the speed of each wheel, and the brake controller can be mechanized to include anti-lock braking functionality.

[0113] Chassis system 38 preferably includes multiple onboard sensing systems and devices for monitoring vehicle operation to determine the vehicle's motion state, and in one embodiment includes multiple devices for dynamically controlling the vehicle suspension. The vehicle motion state preferably includes, for example, vehicle speed, steering angle of the steerable front wheels, and yaw rate. The onboard sensing systems and devices include inertial sensors such as rate gyroscopes and accelerometers. Chassis system 38 estimates the vehicle's motion state, such as longitudinal speed, yaw rate, and lateral speed, and estimates the lateral offset and heading angle of vehicle 10. The measured yaw rate is combined with the steering angle measurement to estimate the vehicle's lateral speed. The longitudinal speed can be determined based on signal inputs from wheel speed sensors arranged to monitor each of the front and rear wheels. Signals associated with the vehicle's motion state can be communicated to and monitored by other vehicle control systems for vehicle control and operation.

[0114] The term "controller" and related terms (such as control module, module, control, control unit, processor, and similar terms) refer to one or more combinations of: (multiple) application-specific integrated circuits (ASICs), (multiple) electronic circuits, (multiple) central processing units (e.g., (multiple) microprocessors), and associated (multiple) non-transitory memory components (read-only, programmable read-only, random access, hard disk drives, etc.) in the form of memory and storage devices. Non-transitory memory components are capable of storing machine-readable instructions in the following forms: one or more software or firmware programs or routines, (multiple) combinational logic circuits, (multiple) input / output circuits and devices, signal conditioning and buffering circuits, and other components accessible by one or more processors to provide the described functions. The (multiple) input / output circuits and devices include analog-to-digital converters and related devices that monitor inputs from sensors, wherein such inputs are monitored at a preset sampling frequency or in response to a triggering event. Software, firmware, program, instructions, control routines, code, algorithms, and similar terms mean a controller-executable instruction set, including calibration and lookup tables. Each controller executes multiple control routines to provide the desired functionality. Routines may be executed at periodic intervals (e.g., every 100 microseconds) during ongoing operation. Alternatively, routines may be executed in response to the occurrence of a triggering event. The term 'model' refers to the associated calibration based on a processor or processor-executable code and the physical presence of a simulating device or physical process. The terms 'dynamic' and 'dynamically' describe steps or processes executed in real time and characterized by monitoring or otherwise determining the state of parameters and regularly or periodically updating the state of parameters during the execution of a routine or between iterations of routine execution. The terms 'calibration,' 'calibrated,' and related terms refer to the result or process of comparing actual or standard measurements associated with a device with sensed or observed measurements or command positions. Calibration as described herein can be simplified to a storable parameter table, multiple executable equations, or another suitable form. Communication between controllers and between controllers, actuators, and / or sensors can be achieved using direct wired point-to-point links, networked communication bus links, wireless links, or other suitable communication links. Communication includes the exchange of data signals in suitable forms, including, for example, electrical signals via a conductive medium, electromagnetic signals via air, optical signals via optical waveguides, etc. Data signals can include discrete, analog, or digitized analog signals representing inputs from sensors, actuator commands, and communication between controllers. The term "signal" refers to a physically identifiable indicator that conveys information and can be a suitable waveform (e.g., electrical, optical, magnetic, mechanical, or electromagnetic), such as DC, AC, sine waves, triangle waves, square waves, vibrations, etc., capable of traveling through a medium.A parameter is defined as a measurable quantity that represents the physical properties of a device or other component that can be identified using one or more sensors and / or physical models. Parameters can have discrete values, such as "1" or "0", or can vary infinitely in value.

[0115] The concepts described herein provide a method, system, and / or device comprising algorithms for enhancing online camera-to-ground alignment and bird's-eye view imaging, the algorithms being referenced from... Figure 1 The described embodiments of the vehicle 10 and the space monitoring system 40 are performed. This includes, in one embodiment, dynamically detecting uneven ground surfaces using image regions from a front camera 42 and a rear camera 44 and calculating ground orientation based thereon. Uneven ground surface detection is accomplished by using angular differences to determine alignment, generating a bird's-eye view image and improving redundant lane sensing, wherein camera-to-ground alignment correction for the side cameras is performed using ground orientation from the front / rear cameras.

[0116] The concepts described herein provide an embodiment of using a spatial monitoring system 40 to generate bird's-eye view images surrounding the vehicle 10, as referenced in [reference]. Figures 3 to 12 As described, the space monitoring system 40 includes a set of instructions executed in one or more controllers. These instructions include: simultaneously capturing images from a front camera, a rear camera, a left camera, and a right camera (denoted as forward FOV 52, rearward FOV 54, leftward FOV 56, and rightward FOV 58); and recovering multiple three-dimensional (3D) points from the front and rear cameras. Based on the multiple 3D points from the front camera, a left ground plane normal vector, a central ground plane normal vector, and a right ground plane normal vector are determined from the central image. A first angular difference between the left ground plane normal vector and the central ground plane normal vector is determined, and a second angular difference between the right ground plane normal vector and the central ground plane normal vector is determined. An uneven ground surface is determined based on either the first or second angular difference, and an alignment compensation factor for either the left or right camera is determined based on the uneven ground surface. A bird's-eye view image is generated based on the alignment compensation factor and the uneven ground surface. In one embodiment, the operation of the autonomous vehicle control system 20 is controlled based on one or more of a bird's-eye view image, an alignment compensation factor, and an uneven ground surface.

[0117] Now for reference Figure 2 Continue to refer to this. Figure 1The described vehicle 10 depicts multiple digital images, including a forward 2D fisheye image 212 from camera 42, a rearward 2D fisheye image 222 from camera 44, a leftward 2D fisheye image 232 from camera 46, and a rightward 2D fisheye image 242 from camera 48, from which a bird's-eye (BV) image 200 of the vehicle 10 is derived. The BV image 200 is created by reconstructing 3D points for each of the aforementioned fisheye images, which were simultaneously captured from the space sensor 41. The BV image 200 includes a first image 210 from the forward 2D fisheye image 212 from the forward FOV 52, a second image 220 from the rearward 2D fisheye image 222 from the rearward FOV 54, a third image 230 from the leftward 2D fisheye image 232 from the leftward FOV 56, and a fourth image 240 from the rightward 2D fisheye image 242 from the rightward FOV 58. It also depicts a forward normal vector 214 orthogonal to the ground surface 250 in forward FOV 52, a rear normal vector 224 orthogonal to the ground surface 250 in backward FOV 54, a left normal vector 234 orthogonal to the ground surface 250 in left FOV 56, and a right normal vector 244 orthogonal to the ground surface 250 in right FOV 58.

[0118] Figure 3 The diagram schematically illustrates the use of [method / technology] to generate bounding boxes. Figure 1 The image shows a bird's-eye view of an embodiment of vehicle 10 and details related to a method, system, and apparatus (method) 300 for controlling its operation in response. Method 300 is simplified as a set of algorithmic practices, illustrated as a collection of blocks in a logic flowchart representing a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, blocks represent computer instructions that, when executed by one or more processors, perform the described operations. For ease of illustration and clarity, reference is made to... Figure 1 The method is described using the vehicle 10 and space monitoring system 40 shown in the figure. Table 1 is provided as a key point, in which the numbered blocks and their corresponding functions are explained below.

[0119] Table 1

[0120] piece Block content S301 Recover 3D points from each image S302 Calculate the ground plane normal vectors and altitudes for the left, front, and right ground plane normal vectors, as well as the angle differences. S303 Evaluation angle difference S304 Report on leveling the ground S305 Report uneven ground; compensate for camera alignment. S306 Generate bird's-eye view images of uneven terrain S307 Controlling autonomous vehicle control systems S308 End iteration

[0121] During the operation of vehicle 10, method 300 may be performed as follows. The steps of method 300 may be performed in a suitable order, and are not limited to the reference. Figure 3 The order described. As used herein, the term “1” indicates an affirmative answer or “yes”, and the term “0” indicates a negative answer or “no”.

[0122] During vehicle operation, step S301 includes: using structure-of-motion (SfM) photogrammetry for the front and rear cameras to recover 3D points for each of the original 2D fisheye images from the plurality of cameras. This references Figure 4 Let me describe it in detail.

[0123] In step S302, the left, front, and right ground plane normal vectors and their associated angles are calculated, along with the angular differences between them. A first angular difference is determined between the left and front ground plane normal vectors, and a second angular difference is determined between the right and front ground plane normal vectors. This references... Figure 6 Let me describe it in detail.

[0124] In step S303, the first angle difference between the left ground plane normal vector and the front ground plane normal vector is compared with a threshold angle difference, and the second angle difference between the right ground plane normal vector and the front ground plane normal vector is compared with the threshold angle difference.

[0125] When the first angle difference is less than the threshold angle distance and the second angle difference is less than the threshold angle distance (0), the method determines that the ground surface has minimum unevenness or no unevenness (i.e., a flat ground), and the iteration ends (S304).

[0126] When the first angle difference is greater than the threshold angle distance or the second angle difference is greater than the threshold angle distance (1), the method continues to S305.

[0127] At step S305, the occurrence of uneven ground is reported, and alignment compensation is generated for the left camera 46 and / or the right camera 48. This references... Figure 8 and Figure 9 Let me describe it in detail.

[0128] In step S306, a bird's-eye view image of the uneven terrain is generated. This references... Figure 12 Let me describe it in detail.

[0129] At step S307, vehicle operation is controlled based on one or more of the bird's-eye view image, alignment compensation factor, and uneven ground surface, specifically including the operation of the autonomous vehicle control system 20. The autonomous vehicle control system 20 is capable of autonomously controlling one or more of the steering system, acceleration system, and braking system, and does so based on the bird's-eye view image. This iteration of method 300 ends (308).

[0130] Figure 4 The illustration shows the relationship with Figure 3Details regarding the execution of step S301, which includes: using structure-of-motion (SfM) photogrammetry for both the front and rear cameras to recover 3D points for each of the 2D images from the plurality of cameras. Step S301 is illustrated as a collection of blocks in a logic flowchart representing a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, blocks represent computer instructions that, when executed by one or more processors, perform the described operations. For convenience and clarity of illustration, refer to... Figure 1 The method is described using the vehicle 10 and space monitoring system 40 shown in the figure. Table 2 is provided as a key point, in which the numbered blocks and their corresponding functions are explained below.

[0131] Table 2

[0132] piece Block content S401 Load FE image frames, features from the previous frame S402 Perform distortion removal and feature detection S403 Perform feature matching S404 Calculate the fundamental matrix and recover the R / T S405 Use triangulation to reconstruct 3D points

[0133] During the operation of vehicle 10, the execution of detailed algorithmic elements related to step S301 is performed iteratively with each successive image capture.

[0134] During vehicle operation, at step 401, image frames from the plurality of cameras and associated features from the previous frame are loaded. The image frames from the plurality of cameras are fisheye images. Features include, for example, color, texture, and other relevant elements from the image.

[0135] Feature detection algorithms (e.g., the Harris corner detector) are used to deduplicate the image and detect features (S402). The Harris corner detector is a corner detection operator that can be used in computer vision algorithms to extract corners and infer image features. Corners are features of an image (also known as points of interest) that are invariant to translation, rotation, or lighting.

[0136] The results of the corner detection operator are subjected to feature matching, such as optical flow feature matching (S403). The fundamental matrix is ​​calculated to recover rotation and translation (R / T) features in the form of rotation matrices and translation vectors (S404). Triangulation is used to recover 3D points in the image (S405). These steps are elements of an embodiment of the Structure-of-Motion (SfM) photogrammetry technique.

[0137] The result of each iteration (i.e., steps S401-S405) is the arrangement of 3D points of the ground plane in the depicted image from the image. The arrangement of 3D points of the ground plane in the depicted image from the image is provided as input. Figure 3 Step 302.

[0138] Figure 5-1The original fisheye image 510 from an embodiment of the front camera 42 of the vehicle 10 operating on the road surface 50 is illustrated in a painterly manner, including multiple focus points indicated by element 501. Figure 5-2 The multipoint in the xyz plane, indicated by element 501, is illustrated graphically. Multipoint 501 represents a 3D point in the camera coordinate system and corresponds to a point on the ground.

[0139] Figure 5-3 The image 520 is illustrated in a painterly style from an embodiment of the front camera 42 of a vehicle 10 operating on road surface 50. Figure 5-1 The distorted image is derived from the fisheye image 510. Image 520 includes multi-vector points, indicated by feature 503, and represents 3D points extracted from consecutive raw images 510 from the same camera, depicting the image obtained using... Figure 4 The steps determine the ground plane 504 in the image. Figure 5-4 The indicated ground plane 504 in the xyz plane is shown graphically.

[0140] Figure 6 The illustration schematically depicts the details related to the execution of step S302, which involves calculating the left, front, right, and rear ground plane normal vectors, their associated angles, and the angular differences between them. A first angular difference is determined between the left and front ground plane normal vectors, and a second angular difference is determined between the right and front ground plane normal vectors. This reference... Figure 6 This will be described in detail below. Step S302 is illustrated as a collection of blocks in a logic flowchart, which represents a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, blocks represent computer instructions that, when executed by one or more processors, perform the described operations. For ease of illustration and clarity, refer to... Figure 1 The method is described using the vehicle 10 and space monitoring system 40 shown in the figure. Table 3 is provided as a key point, in which the numbered blocks and their corresponding functions are explained below.

[0141] Table 3

[0142] piece Block content S601 Read feature points in the left, front, and right regions. S602 Select sample 3D points to fit the ground plane. S603 Calculate the angle difference S604 The angular difference < threshold S605 Calculate the distance from all 3D points to the plane. S606 The number of iterations is less than the threshold S607 Calculate the L2 norm S608 All left, front, and right areas have been processed. S609 Calculate the angle difference between regions

[0143] During the operation of vehicle 10, the execution of detailed algorithmic elements related to step S302 is carried out as follows with each successive image capture.

[0144] In step S601, feature points, corresponding 3D points, and reference ground vectors are read from different regions. Samples of 3D points for fitting the ground plane using singular value decomposition (SVD) are randomly selected for each of the left, front, and right regions (S602). The angular difference between the ground plane normal vector and the reference vector is calculated (S603), and this angular difference is compared with a threshold angle (S604). When the angular distance is greater than the threshold angle (S604) (0), the iteration jumps to step S602 and / or step S608.

[0145] When the angular distance is less than the threshold angle (S604) (1), calculate the distance to that point for all 3D points. If the distance is less than the distance of the normal vector, then maintain the normal vector (S605).

[0146] Maintain and evaluate the iteration count (S606). When the number of iterations is less than a threshold (S606) (1), the routine returns to step S602 to perform another iteration. When the number of iterations is greater than the threshold (S606) (0), the L2 norm of the normal vector is calculated as the ground-to-camera height for a specific region (S607). Repeat these steps for each of the left, front, and right regions (S608). When a normal vector is determined for each of the left, front, and right regions, the angular differences between the normal vectors of the left, front, and right regions are calculated (S609), and these angular differences are provided as input. Figure 3 Step 303 is used for evaluation.

[0147] Figure 7 The original fisheye image 710 from an embodiment of the front camera 42 of a vehicle 10 operating on road surface 750 is illustrated in a painterly manner, showing a front image 712 and a left image 732 identified thereon. The corresponding front normal vector 714 and left normal vector 734 are indicated. An angle difference 735 is identified, representing the angle difference between the front normal vector 714 and the left normal vector 734, which is determined by step S302 and reference steps S601 to S609 (as referenced). Figure 6 The relevant algorithms described herein are used to determine this.

[0148] Figure 8 Details related to an alignment compensation routine, an implementation of step S305 in the first embodiment, are schematically illustrated, wherein alignment compensation is generated for the left camera 46 and / or the right camera 48 based on the presence of uneven terrain. This routine is illustrated as a collection of blocks in a logic flowchart representing a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, blocks represent computer instructions that, when executed by one or more processors, perform the described operations. For ease of illustration and clarity, reference is made to... Figure 1The method is described using the vehicle 10 and space monitoring system 40 shown in the figure. Table 4 is provided as a key point, in which the numbered blocks and their corresponding functions are explained below.

[0149] Table 4

[0150] piece Block content S801 <![CDATA[Read the ground plane normal vector from the front area (Norm CF ).]]> S802 Read the front image and the left (or right) image. S803 Feature detection and matching algorithms are used to find matching pairs in overlapping regions. S804 Calculate the fundamental matrix and rotation matrix. S805 <![CDATA[Calculate the ground plane normal vector of the left camera (Norm L ).]]> S806 <![CDATA[Use Norm L to complete the alignment of the left camera to the ground]]>

[0151] During the operation of vehicle 10, the execution of the first embodiment of the detailed algorithmic elements related to the alignment compensation in step 305 is carried out with each successive image capture as follows.

[0152] At step S801, the ground plane normal vector (Norm) from the front region is input. CF The algorithm then reads the front image and the left (or right) image at step 802. It attempts to use feature detection and matching routines to find feature matching pairs in the overlapping region between the front and left (or right) images for right / front analysis (S803). The fundamental matrix and rotation transformation matrix R are then determined. FL (S804), and based on the rotation transformation matrix R FL and the front camera normal matrix (i.e., Norm) CF To determine the left camera's ground plane normal vector Norm L (S805). The left camera ground plane normal vector Norm is used. L To align the left camera with the ground (S806), this routine can also be executed to find the ground plane normal vector Norm for the right camera. R To align the right camera to the ground. Method 300 uses the alignment of the left (or right) camera to the ground to generate a bird's-eye view image of an uneven terrain, as shown in the reference. Figure 3 The steps described in S306. It should be understood that the ground plane normal vector (Norm) from the rear region can be used. CR (replacing the ground plane normal vector from the front region (Norm)) CF The alignment compensation routine is performed in one of the left (or right) images (step S305).

[0153] Figure 9 Details relating to the execution of a second embodiment of the alignment compensation routine (step S305) are schematically illustrated, which is used to generate alignment compensation for the left camera 46 or the right camera 48. The alignment compensation routine is illustrated as a collection of blocks in a logic flowchart representing a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, blocks represent computer instructions that, when executed by one or more processors, perform the described operations. For ease of illustration and clarity, reference is made to... Figure 1The method is described using the vehicle 10 and space monitoring system 40 shown in the figure. Table 5 is provided as a key point, in which the numbered blocks and their corresponding functions are explained below.

[0154] Table 5

[0155] piece Block content S901 <![CDATA[Read the ground plane normal vector Norm from the front region CF ; Read Norm LF ; Read the front motion vector T F <!-- 13 -->]]> S902 <![CDATA[Read the original normal vector Norm’ from the left camera L ; Read the post-motion vector T R > S903 <![CDATA[Optimize R FL to minimize the loss function]]> S904 Determine the left camera normal vector S905 <![CDATA[Use Norm L to complete the alignment of the left camera to the ground]]>

[0156] During the operation of vehicle 10, the execution of the second embodiment of the detailed algorithmic elements related to the alignment compensation in step 305 is carried out as follows with each successive image capture.

[0157] In step S901, the ground plane normal vector (Norm) from the front region is... CF ) and the ground plane normal vector from the left region (Norm LF Together with the previous motion vector (T) F The original ground plane normal vector (Norm') from the left region is provided as input. At step S902, the original ground plane normal vector from the left region is input. LF ) and left motion vector (T) L ). Perform an optimization algorithm (e.g., Adams' algorithm) to find the rotation transformation matrix R. FL The optimal value of satisfies the following loss function or relation:

[0158] [1]

[0159] in:

[0160] L is the loss function, which is minimized; and

[0161] Indicates a scalar relationship.

[0162] The rotation transformation matrix R can be determined using the geometric relationship of least squares according to the following relationship. FL :

[0163] .

[0164] Rotation transformation matrix R FL This acts as an alignment compensation factor to achieve alignment between the left camera 46 and the ground 50. In step S904, the left camera ground plane normal vector Norm can be determined as follows: L :

[0165] .

[0166] The left camera ground plane normal vector Norm can be used. L This determines the alignment of the left camera (46) to the ground (50) (S905). It should be understood that this routine can also be executed to find the ground plane normal vector Norm of the right camera.R The right camera 48 is aligned with the ground 50. Method 300 uses the alignment of the left (or right) camera with the ground to generate a bird's-eye view image of an uneven terrain, as shown in the reference. Figure 3 The steps described in step S306.

[0167] Figure 10 This schematically illustrates the generation of bird's-eye view images on uneven terrain (as shown by...). Figure 3 The details are as described in step S306. Figure 10 The bird's-eye view image generation routine is illustrated as a collection of blocks in a logic flowchart, which represents a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, blocks represent computer instructions that, when executed by one or more processors, perform the described operations. For ease of illustration and clarity, refer to... Figure 1 The method is described using the vehicle 10 and space monitoring system 40 shown in the figure. Table 6 is provided as a key point, in which the numbered blocks and their corresponding functions are explained below.

[0168] Table 6

[0169] piece Block content S1001 Read the ground plane normal vectors of the L, R, F, and R cameras. S1002 For each camera's field of view (FOV), the altitude in world coordinates is determined using the ground plane equation. S1003 Use a fisheye model to project 3D points onto 2D pixels. S1004 Distortion removal of 2D images and 2D points is performed using a fisheye model. S1005 Use dedistorted 2D points and desired pixels at the bird's-eye view location to utilize perspective transformation to convert the dedistorted image into a bird's-eye view image. S1006 Images from four cameras are overlaid to generate a holistic bird's-eye view image.

[0170] During the operation of vehicle 10, the execution of algorithmic elements related to step 306, which is used to generate a bird's-eye view image, is carried out as follows with each successive image capture.

[0171] In step S1001, the ground plane normal vectors of the left (L), right (R), front (F), and rear (R) cameras are read. In step S1002, for the four corner points of each FOV from the left (L), right (R), front (F), and rear (R) cameras, the ground plane equation is used to determine the altitude in world coordinates.

[0172] The 3D points are projected onto the 2D pixels using a fisheye model (S1003), and the 2D image and 2D points are dedistorted using the fisheye model (S1004).

[0173] The distorted 2D points and desired pixels are used in the bird's-eye view image to convert the distorted image into a bird's-eye view image using perspective transformation (S1005), and the four camera images are overlaid to generate an overall bird's-eye view image (S1006).

[0174] Figure 11 Details related to an alternative embodiment of S302 (for a vehicle configuration employing LiDAR device 43) are schematically illustrated, wherein information from the front and rear cameras is replaced with LiDAR point clouds to determine the left, front, and right ground plane normal vectors and associated angles, as well as the angle differences.

[0175] The routines used with LiDAR device 43 to determine the left, front, and right ground plane normal vectors and associated angles, as well as the angle differences, are illustrated as a set of blocks in a logic flowchart representing a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, blocks represent computer instructions that, when executed by one or more processors, perform the described operations. For ease of illustration and clarity, refer to... Figure 1 The method is described using the vehicle 10 and space monitoring system 40 shown in the figure. Table 7 is provided as a key point, in which the numbered blocks and their corresponding functions are explained below.

[0176] Table 7

[0177] piece Block content S1101 Read LiDAR point clouds and transform them into world coordinates S1102 Aggregating N LiDAR frames S1103 Select the left front, front center, and right front areas. S1104 PCA (SVD) is applied to the point cloud of each region. S1105 The third eigenvector is selected as the normal vector of each region. S1106 Calculate the distance from each point to the ground plane. S1107 Average distance less than threshold S1108 Use the angle difference equation to output the normal vector and the angle difference. S1109 quit

[0178] During the operation of vehicle 10, the execution of algorithmic elements related to using LiDAR device 43 to determine the left, front, and right ground plane normal vectors and related angles, as well as the angle differences, is performed as follows:

[0179] Initially, at step S1101, the LiDAR point cloud is read and transformed into world (xyz) coordinates, where N LiDAR frames are aggregated to account for the sparsity of the LiDAR points (S1102). Spatial filters are applied to the aggregated LiDAR frames to select the left-front, front-middle, and right-front regions (S1103). Principal component analysis (PCA) and singular value decomposition (SVD) are employed to reduce the high-dimensional dataset provided by the point cloud for each region (S1104), which involves generating multiple feature vectors.

[0180] The third feature vector is selected as the normal vector of each of the left front, front center, and right front regions (S1105), and the distance from each point of the normal vector to the ground plane is calculated (S1106). The average distance is calculated and compared with a threshold (S1107). When the average distance is greater than the threshold (S1107) (0), the process is repeated at step S1101. When the average distance is less than the threshold (S1107) (1), the process uses the angle difference equation to output the normal vector and the angle difference (S1108), and this iteration ends (S1109). In this way, a LiDAR device can be used to determine the left, front, and right ground plane normal vectors and associated angles, as well as the angle difference, where this information is supplied to Figure 3 The subsequent steps S303 and subsequent steps are to determine the existence of uneven ground and generate a bird's-eye view image.

[0181] Figure 12The illustration schematically depicts details related to the curb detection routine 1200 used to detect curbs in a field of view (FOV). Curbs are a specific type of uneven road where inaccurate alignment results may occur due to the presence of two different ground planes.

[0182] The curbstone detection routine 1200 is illustrated as a collection of blocks in a logic flowchart, which represents a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, blocks represent computer instructions that, when executed by one or more processors, perform the described operations. For ease of illustration and clarity, refer to [reference needed]. Figure 1 The method is described using the vehicle 10 and space monitoring system 40 shown in the figure. Table 8 is provided as a key point, in which the numbered blocks and their corresponding functions are explained below.

[0183] Table 8

[0184] piece Block content S1201 For the left and right cameras, estimate 3D points. S1202 N frames that aggregate 3D points S1203 Calculate the ground plane normal vector in the near region S1204 Calculate the distance from each point to the normal vector of the ground plane. S1205 Identifying outliers S1206 The ratio is greater than the threshold for the time period. S1207 Roadside stone detected

[0185] During the operation of vehicle 10, the execution of algorithmic elements related to curb detection routine 1200 is performed as follows.

[0186] Initially, at step S1201, S301 is performed on data from the left and right cameras to estimate 3D points, where N quantities of 3D points are aggregated in frames (S1202). The ground plane normal vector is determined using vector information associated with the near region (e.g., within a calibrated distance of 10 from the vehicle) and taking into account the estimated ground height of the respective camera (S1203), and the distance to the ground plane from each point is calculated (S1204). Outliers are identified as those points with a distance greater than a first threshold ratio and a ground height greater than a second threshold (S1205), and are evaluated by comparison (S1206), wherein a curb is detected when the evaluation of the outlier indicates that the ground height is greater than a minimum threshold (Th2) for at least a certain number of consecutive image frames within a time range (Th4) (Th3) (S1207). In this way, the detection of the following is completed: uneven road surface, misalignment of the left or right camera, curb detection, and alignment of the bird's-eye view image.

[0187] Onboard cameras are subject to dynamically changing internal and external factors, such as uneven road surfaces, which can affect the operation of onboard systems that depend on camera images. The concepts described herein provide a method, system, and / or device capable of: capturing a front image from a front camera, a rear image from a rear camera, a left image from a left camera, and a right image from a right camera; recovering multiple three-dimensional (3D) points from the front and rear images; determining a left ground plane normal vector for the left image, a central ground plane normal vector from the front image, and a right ground plane normal vector from the right image based on the multiple 3D points from one of the front or rear images; determining a first angular difference between the left ground plane normal vector and the central ground plane normal vector; determining a second angular difference between the right ground plane normal vector and the central ground plane normal vector; detecting an uneven ground surface based on one of the first or second angular differences; and determining an alignment compensation factor for one of the left or right cameras based on the uneven ground surface. A bird's-eye view image is generated based on the alignment compensation factor, and vehicle operation can be controlled based on this image. Therefore, the claimed embodiments represent an improvement in the technical field.

[0188] Flowcharts and block diagrams within flowcharts illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a portion of a module, segment, or code, including one or more executable instructions for implementing the specified logical function(s). It will also be noted that each block illustrated in the block diagrams and / or flowcharts, and combinations of blocks illustrated in the block diagrams and / or flowcharts, may be implemented by a system based on dedicated hardware or a combination of dedicated hardware and computer instructions that performs the specified function or action. These computer program instructions may also be stored in a computer-readable medium that can instruct a controller or other programmable data processing device to function in a particular manner, causing the instructions stored in the computer-readable medium to produce an article of art including the instructions to implement the function / action specified in one or more blocks of the flowcharts and / or block diagrams.

[0189] As used herein, the term “system” may refer to one or a combination of mechanical and electrical hardware, sensors, controllers, application-specific integrated circuits (ASICs), combinational logic circuits, software, firmware, and / or other components arranged to provide the described functions.

[0190] The use of ordinal numbers such as first, second, and third does not necessarily imply a sense of hierarchy, but may simply distinguish multiple instances of an action or structure.

[0191] The detailed description and accompanying drawings support and describe this teaching, but the scope of this teaching is defined solely by the claims. While some of the best modes and other embodiments for carrying out this teaching have been described in detail, various alternative designs and embodiments exist for practicing this teaching as defined in the appended claims.

Claims

1. A vehicle comprising: a spatial monitoring system having a plurality of on-board cameras in communication with a controller, the plurality of on-board cameras including a front camera arranged to capture a front image of a front-facing field of view, a rear camera arranged to capture a rear image of a rear-facing field of view, a left camera arranged to capture a left image of a left-facing field of view, and a right camera arranged to capture a right image of a right-facing field of view; the controller including a set of instructions executable to: simultaneously capture the front image from the front camera, the rear image from the rear camera, the left image from the left camera, and the right image from the right camera; recover a plurality of three-dimensional points from the front image and the rear image; determine a left ground plane normal vector for the left image, a center ground plane normal vector from the front image, and a right ground plane normal vector from the right image based on the plurality of three-dimensional points from one of the front image or the rear image; determine a first angular difference between the left ground plane normal vector and the center ground plane normal vector; determine a second angular difference between the right ground plane normal vector and the center ground plane normal vector; detect an uneven ground surface based on one of the first angular difference or the second angular difference; determine an alignment compensation factor for one of the left camera or the right camera based on the uneven ground surface; and generate an overhead view image based on the alignment compensation factor.

2. The vehicle of claim 1, further comprising: an autonomous vehicle control system capable of autonomously controlling one of a steering system, an acceleration system, or a braking system; wherein the autonomous vehicle control system controls one of the steering system, the acceleration system, or the braking system based on the overhead view image. the front image, the rear image, the left image, and the right image comprise two-dimensional fisheye images.

3. The vehicle of claim 1, wherein, the set of instructions is executable to recover the plurality of three-dimensional points from the two-dimensional fisheye images employing motion structure analysis.

4. The vehicle of claim 3, wherein, the set of instructions is executable to determine ground plane normal vectors for the left image, the front image, and the right image.

5. The vehicle of claim 1, wherein, the set of instructions executable to determine the alignment compensation factor for the left camera based on the uneven ground surface comprises a set of instructions executable to:

6. The vehicle of claim 1, wherein, find feature match pairs in an overlapping region between the front image and the left image using a feature detection and matching routine. the rotation transformation matrix comprises the alignment compensation factor.

7. The vehicle of claim 6, further comprising a set of instructions executable to determine an essential matrix and a rotation transformation matrix based on the feature match pairs, wherein, 8. The vehicle of claim 7, further comprising a set of instructions executable to: determine a left camera ground plane normal vector based on the rotation transformation matrix and a front camera normal matrix; and align the left camera to the ground employing the left camera ground plane normal vector.

9. The vehicle of claim 8, further comprising a set of instructions executable to generate the overhead view image based on the alignment of the left camera to the ground. the set of instructions executable to determine the rotation transformation matrix comprises a set of instructions executable to:

10. The vehicle of claim 7, wherein, ​ determining a front motion vector, a left motion vector, a ground plane normal vector from the left field of view, and an original ground plane normal vector; and determining a rotation transformation matrix that minimizes a loss based on a relationship between the front motion vector, the left motion vector, the ground plane normal vector from the left field of view, and the original ground plane normal vector.

11. The vehicle of claim 1, further comprising a set of instructions executable to generate the bird's eye view image based on the front image, the left image, the right image, the rear image, and the alignment compensation factor.

12. A vehicle, comprising: a spatial monitoring system having an on-board camera in communication with a controller, the on-board camera being one of a front camera arranged to capture a front image of a front field of view, a left camera arranged to capture a left image of a left field of view, a right camera arranged to capture a right image of a right field of view, or a rear camera arranged to capture a rear image of a rear field of view; the controller comprising a set of instructions executable to: capture an image from the on-board camera; recover a plurality of three-dimensional points from the image; determine a ground plane normal vector for the image based on a near region of the plurality of three-dimensional points from the on-board camera; determine a distance to the ground plane normal vector for each of the three-dimensional points; and detect a presence of a curb in a respective field of view based on the distance to the ground plane normal vector for each of the three-dimensional points.

13. The vehicle of claim 12, further comprising: the vehicle comprising an autonomous vehicle control system, the autonomous vehicle control system being capable of autonomously controlling one of a steering system, an acceleration system, or a braking system; wherein the autonomous vehicle control system controls one of the steering system, the acceleration system, or the braking system based on the presence of the curb in the field of view.

14. The vehicle of claim 12, wherein, the image comprises a two-dimensional fisheye image.

15. The vehicle of claim 14, wherein, the set of instructions are executable to recover the plurality of three-dimensional points from the two-dimensional fisheye image of the image using motion structure analysis.

16. The vehicle of claim 12, wherein, the set of instructions are executable to determine a ground plane normal vector for the image.

17. A vehicle, comprising: a spatial monitoring system comprising a light detection and ranging device arranged to capture data representative of a front field of view, a rear field of view, a left field of view, and a right field of view, and a controller; the controller comprising a set of instructions executable to: capture a plurality of images from the light detection and ranging device; determine a left image, a front image, and a right image based on the plurality of images from the light detection and ranging device; determine a left ground plane normal vector for the left image, a center ground plane normal vector from the front image, and a right ground plane normal vector from the right image based on the plurality of images from the light detection and ranging device; determine a first angular difference between the left ground plane normal vector and the center ground plane normal vector; determine a second angular difference between the right ground plane normal vector and the center ground plane normal vector; detect an uneven ground surface based on one of the first angle difference or the second angle difference; determine an alignment compensation factor based on the uneven ground surface; and generate an overhead view image based on the alignment compensation factor and the uneven ground surface.

18. The vehicle of claim 17, further comprising: the vehicle includes an autonomous vehicle control system capable of autonomously controlling one of a steering system, an acceleration system, or a braking system; wherein the autonomous vehicle control system controls one of the steering system, the acceleration system, or the braking system based on the overhead view image.

19. The vehicle of claim 17, wherein, the instruction set is executable to determine a ground plane normal vector for the left image, the front image, and the right image.

20. The vehicle of claim 17, wherein, the spatial monitoring system further has an on-board camera in communication with the controller, the on-board camera including a left camera arranged to capture a left image of a left field of view, the vehicle further includes an instruction set executable to: determine a left ground plane normal vector for the left image based on a near region of a plurality of three-dimensional points from the left camera; determine a distance to the left ground plane normal vector for each of the three-dimensional points; and determine a presence of a curb in the left field of view based on the distance to the left ground plane normal vector for each of the three-dimensional points.

Images