Environmental perception method based on binocular and look-around fisheye cameras and driving system applying method

Abstract

The invention discloses an environment perception method based on a binocular camera and an all-round fisheye camera, and the method comprises the steps: collecting the three-dimensional information of an image, carrying out the stereo vision obstacle detection, generating a disparity map, obtaining the three-dimensional information through the position deviation of a corresponding point after the comparison with a reference condition, carrying out the target detection based on deep learning, and carrying out the semantic segmentation. The method comprises the following steps: detecting a drivable area, inputting a sensing result into a decision planning module for track planning, judging whether each path point on a track is occupied at a time point or not, adjusting a plan to avoid risks, and inputting the adjusted track into a vehicle control module. And the control instruction is input into a vehicle platform drive-by-wire module to drive an original vehicle execution mechanism to complete execution of an acceleration / steering angle instruction. According to the method, the binocular camera is used for collecting images to calculate the depth information of the target pixel points, the distance information of the obstacle is obtained, in other words, depth information measurement and calculation are completed on the premise that obstacle target recognition is not completed, and the intelligent driving function of driving and parking is achieved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to an environment perception method based on binocular and surround-view fisheye cameras and a driving system using the method. Background Art

[0002] At present, intelligent driving technology is booming, and environmental perception is the top priority of intelligent driving technology. Multi-sensor fusion has become the development trend of autonomous driving. By integrating the perception results of multiple cameras and radars, vehicles can more accurately identify the surrounding environment.

[0003] Environmental perception typically uses sensors such as ultrasonic radar, surround-view cameras, and millimeter-wave radar to detect obstacles and lane information around the vehicle. For example, ultrasonic radar, while used to sense the location and distance of obstacles, cannot identify the specific form of the obstacle, which can interfere with obstacle recognition. Furthermore, when multiple ultrasonic radars operate simultaneously, co-frequency interference may occur, and the propagation delay of the ultrasonic signal can lead to recognition delays.

[0004] In order to improve the stability and reliability of the system, manufacturers use a large number of expensive sensors and controllers to implement intelligent driving functions, which makes the product manufacturing and user use costs extremely high.

[0005] With the development of technology, manufacturers have improved the technical solutions: 1. Use lidar, millimeter-wave radar, and cameras to complete environmental perception by fusing data from multiple cameras. In this solution, sensors and lidar are expensive and have obvious cost disadvantages; 2. Use cameras and millimeter-wave radars to complete environmental perception. Among them, three cameras with different optical parameters are used in front of the vehicle to identify the environment. This solution does not require the use of expensive lidar. However, when three cameras with different optical parameters are used to detect the front environment, deep learning or traditional computer vision methods are needed to complete target recognition of obstacles and further estimate the obstacle distance. For obstacles that have not been trained or cannot be identified, the distance cannot be estimated. Summary of the Invention

[0006] In order to reduce product costs and improve the accuracy of environmental detection, the present invention provides an environmental perception method based on binocular and surround-view fisheye cameras and a driving system using the method. A pair of lenses with the same optical parameters are used to form a binocular camera to realize point cloud data processing of the front environmental information. Four surround-view fisheye cameras are used to achieve 360° perception around the vehicle, avoiding the use of lidar and millimeter-wave radar.

[0007] The present invention provides an environment perception method based on binocular and surround-view fisheye cameras, comprising the following steps:

[0008] S1. Collect three-dimensional information of the image:

[0009] A binocular stereo camera is constructed using a pair of cameras located on the same horizontal plane with a fixed baseline distance and identical optical performance parameters. The two cameras simultaneously capture image information and perform image preprocessing in the camera's ISP chip. Depth information is obtained based on the pixel difference between the left and right cameras' images of corresponding points in the same image.

[0010] S2. Stereoscopic obstacle detection based on image information collected by binocular stereo cameras:

[0011] A pixel on the image captured by the left camera is selected as the left pixel, and an area with this pixel as the center is selected as the area to be matched. A matching area with high similarity is found on the right camera. The pixel with the most similar features in this matching area is the matching point of the left pixel. The disparity between the left pixel and its matching point is estimated using a local optimization function.

[0012] S3. Extract local features of corners, edges, and planes of objects in the images captured by the left and right cameras, or extract global features of complex shapes and overall image structures, perform feature matching on the images captured by the left and right cameras, and generate a disparity map based on the matching results. Each pixel value in the disparity map represents the horizontal pixel difference between the left and right images at that point.

[0013] The larger the parallax value, the closer the object is to the camera; the smaller the parallax value, the farther the object is from the camera;

[0014] S4. Obtain three-dimensional information using the position deviation after comparing the corresponding points with the reference situation:

[0015] Obtain the parallax of corresponding points in images of the same object taken at the same time but at different angles to restore the object's three-dimensional information for stereoscopic obstacle detection.

[0016] S5. Perform deep learning-based object detection on the image:

[0017] Use a common, trained deep learning neural network to detect trained dynamic obstacle targets using images from the left and right cameras of a binocular stereo camera and multiple fisheye cameras.

[0018] S6. Perform semantic segmentation on the image to complete the detection of the drivable area:

[0019] The image is divided into multiple regions based on meaning. Each region represents a different category, such as people, cars, and signs. Each category is divided into dynamic targets and static targets based on whether they are movable.

[0020] Give a classification result for each pixel in the image, and finally output a dense prediction map;

[0021] The dense prediction map contains the perception results of the image: edge information and semantic information of the original image;

[0022] This prediction preserves the edge information and semantic information of the original image to a great extent, which is used for scene understanding and decision planning of subsequent system algorithms;

[0023] S7, input the perception results to the decision-making and planning module for trajectory planning. The trajectory generated by the decision-making and planning module is a series of path points;

[0024] Each waypoint is assigned a waypoint timestamp and waypoint speed. The waypoint timestamp is based on the camera exposure timestamp as the initial time. The estimated time of the vehicle at the waypoint indicates the estimated time of the vehicle's arrival at the spatial point. The waypoint speed is based on the speed at the time the camera took the photo as the initial speed. The estimated speed of the vehicle at the waypoint indicates the speed at which the vehicle will arrive at the spatial point.

[0025] S8, the prediction module is a submodule of the decision-making and planning module. The prediction module determines whether each path point on the trajectory is occupied at a certain point in time:

[0026] Because moving obstacles may temporarily block part of the road, each waypoint in the trajectory has a timestamp. We can combine the timestamp with the output of the prediction module to ensure that each waypoint on the trajectory is not occupied when we plan to pass it. We use the following method:

[0027] a) Taking the position, direction, and speed of a dynamic target at the past N time points (Tn, Tn-1, Tn-2) as input, the Kalman filter method is used to predict the position and speed of the dynamic target at the time point Tn+1;

[0028] b) For each path point on the trajectory at time t_i, determine the precise predicted state (position, speed, etc.) of the surrounding dynamic targets (such as the vehicle in front):

[0029] The prediction module outputs a series of future state points (trajectory) of the target object, each of which has its own timestamp {t_pred1, State1}, {t_pred2, State2}, ...;

[0030] At time t_i, the decision planning module finds the state point with the timestamp closest to t_i from the predicted target trajectory as the accurate predicted state State_target(t_i);

[0031] c) The decision-making and planning module combines timestamps to perform spatial and temporal consistency collision checks, accurately assessing collision risks and adjusting the trajectory:

[0032] At time point t_i, the vehicle plan is located at Pi, and the dynamic target prediction is at State_target(t_i).position;

[0033] Calculate whether there is overlap between the area occupied by the vehicle (considering the vehicle outline / occupancy) and the area predicted to be occupied by the dynamic target at time t_i. If there is a collision risk (distance is too small or overlap), the decision-making planning module adjusts the speed plan, the spatial path, or both to avoid the risk;

[0034] S9. The decision-making and planning module inputs the adjusted trajectory to the vehicle control module. The vehicle control module outputs the acceleration command and steering angle command for vehicle control based on the vehicle dynamics parameters. Finally, the control command is input to the vehicle platform wire control module. The vehicle platform wire control module integrates a communication protocol that can interact with the MEB platform architecture. By calling this communication interface, the control command can normally drive the original vehicle actuator to complete the execution of the acceleration / steering angle command.

[0035] Preferably, the waypoint timestamp and waypoint speed are calculated as follows:

[0036] a) The decision-making and planning module calculates the time required for the vehicle to reach each waypoint from its current position based on factors such as target speed, acceleration limit, comfort requirements, traffic regulations (speed limits), and interactions between preceding vehicles and obstacles. It calculates waypoint timestamps using the camera exposure timestamp as the initial time and the waypoint speed using the speed at the time of the camera image as the initial speed.

[0037] b) The decision-making and planning module calculates the path points along the vehicle's travel direction at preset time intervals Δt (e.g., 0.1s, 0.2s);

[0038] This interval determines the density of waypoints and the amount of computation.

[0039] Preferably, interpolation is performed between two adjacent state points of the predicted trajectory to obtain the precise predicted state State_target(t_i) of the target at time t_i. The more interpolations are performed, the smoother and more accurate the predicted trajectory will be.

[0040] Preferably, the interpolation method used is: linear interpolation, kinematic model interpolation, etc.

[0041] Preferably, the decision planning module adjusts the speed planning by changing the time stamp of the t_i moment or the subsequent path point.

[0042] Preferably, the decision-making planning module adjusts the spatial path by changing the vehicle's planned position Pi.

[0043] By binding planned path points to precise future timestamps and combining them with the output of the prediction module at the same time point, the system achieves unified evaluation and collaborative planning of the vehicle's own future trajectory and the dynamic predicted trajectory of the surrounding environment in a spatiotemporal coordinate system, significantly improving driving safety and comfort in complex dynamic scenarios.

[0044] The present invention also provides a driving system that uses an environmental perception method based on binocular and surround-view fisheye cameras: a binocular stereo camera composed of two cameras with the same optical parameters is arranged on the front side of the vehicle, such as the top of the front windshield. The two cameras have the same field of view and resolution, are set on the same horizontal plane, and have a fixed baseline distance, and are used to complete the perception of environmental information in front of the vehicle; a surround-view fisheye camera is arranged on the front, back, left and right sides of the vehicle body to complete the perception of environmental information around the vehicle.

[0045] The ISP chip in the binocular stereo camera is used to complete the corresponding image data preprocessing;

[0046] The image data of the binocular stereo camera and the surround-view fisheye camera are transmitted to the central domain controller.

[0047] The central domain controller obtains depth information based on the pixel difference between the left and right cameras' corresponding points in an image, and performs stereoscopic obstacle detection:

[0048] A pixel on the image captured by the left camera is selected as the left pixel, and an area with this pixel as the center is selected as the area to be matched. A matching area with high similarity is found on the right camera. The pixel with the most similar features in this matching area is the matching point of the left pixel. The disparity between the left pixel and its matching point is estimated using a local optimization function.

[0049] The central domain controller extracts local features of corners, edges, and planes of objects in the images captured by the left and right cameras, or extracts global features of complex shapes and overall image structure. It performs feature matching on the images captured by the left and right cameras and generates a disparity map based on the matching results. Each pixel value in the disparity map represents the horizontal pixel difference between the left and right images at that point.

[0050] The larger the parallax value, the closer the object is to the camera; the smaller the parallax value, the farther the object is from the camera;

[0051] The central domain controller uses the position deviation after comparing the corresponding points with the reference situation to obtain the parallax of the corresponding points in the pictures of the same object taken at the same time but at different angles to restore the three-dimensional information of the object and detect stereoscopic obstacles.

[0052] The central domain controller performs semantic segmentation on the image and completes the detection of the drivable area:

[0053] The image is divided into multiple regions based on meaning. Each region represents a different category, such as people, cars, and signs. Each category is divided into dynamic targets and static targets based on whether it is movable and its type characteristics. The images are obtained from the left and right cameras of a binocular stereo camera and multiple fisheye cameras.

[0054] The central domain controller uses a common, trained deep learning neural network to detect dynamic obstacles.

[0055] The central domain controller gives a classification result for each pixel in the image and ultimately outputs a dense prediction map. The goal of dense prediction is to predict each pixel in the input image and assign a category label.

[0056] The dense prediction map contains pixel-level type recognition results. The dense prediction map is a prediction map that is the same size as the input image or scaled proportionally. The recognition value of each pixel is an integer representing the category to which it belongs.

[0057] This prediction preserves the edge information and semantic information of the original image to a great extent, which is used for scene understanding and decision planning of subsequent system algorithms;

[0058] The central domain controller inputs the perception results to the decision-making and planning module for trajectory planning. The trajectory generated by the decision-making and planning module is a series of path points.

[0059] The central domain controller assigns a waypoint timestamp and speed to each waypoint. The waypoint timestamp uses the camera exposure timestamp as the initial time. The estimated time of the vehicle at that waypoint indicates the estimated time of the vehicle's arrival at that spatial point. The waypoint speed uses the speed at the time the camera took the photo as the initial speed. The estimated speed of the vehicle at that waypoint indicates the speed at which the vehicle will arrive at that spatial point.

[0060] The prediction module is a submodule of the decision-making and planning module. The prediction module determines whether each path point on the trajectory is occupied at a certain point in time:

[0061] Because moving obstacles may temporarily block part of the road, each waypoint in the trajectory has a timestamp. We can combine the timestamp with the output of the prediction module to ensure that each waypoint on the trajectory is not occupied when we plan to pass it. We use the following method:

[0062] a) Taking the position, direction, and speed of a dynamic target at the past N time points (Tn, Tn-1, Tn-2) as input, the Kalman filter method is used to predict the position and speed of the dynamic target at the time point Tn+1;

[0063] b) For each path point on the trajectory at time t_i, determine the precise predicted state (position, speed, etc.) of the surrounding dynamic targets (such as the vehicle in front):

[0064] The prediction module outputs a series of future state points (trajectory) of the target object, each of which has its own timestamp {t_pred1, State1}, {t_pred2, State2}, ...;

[0065] At time t_i, the decision planning module finds the state point with the timestamp closest to t_i from the predicted target trajectory as the accurate predicted state State_target(t_i);

[0066] The decision-making and planning module combines timestamps to perform spatial and temporal consistency collision checks, accurately assesses collision risks, and adjusts the trajectory:

[0067] At time point t_i, the vehicle plan is located at Pi, and the dynamic target prediction is at State_target(t_i).position;

[0068] Calculate whether there is overlap between the area occupied by the vehicle (considering the vehicle outline / occupancy) and the area predicted to be occupied by the dynamic target at time t_i. If a collision risk is predicted (the distance is too small or there is overlap), the decision planning module adjusts the speed plan, the spatial path, or both to avoid the risk.

[0069] The decision-making and planning module inputs the adjusted trajectory to the vehicle control module. The vehicle control module outputs acceleration and steering angle commands based on vehicle dynamics parameters. Finally, the control commands are input to the vehicle platform's drive-by-wire module, which integrates a communication protocol interface that can interact with the MEB platform architecture. By calling this communication protocol interface, the control commands can be executed by the vehicle actuator to complete the acceleration / steering angle commands.

[0070] The wire control module completes the wire control adaptation for the vehicle platform and communicates with the vehicle actuator through the MEB vehicle architecture protocol;

[0071] Vehicle actuators include: drive motor, steering gear, and brakes, which provide correct control response and feedback.

[0072] Ultimately, intelligent driving / parking functions will be realized, including adaptive cruise control, lane centering, lever lane change, automatic parking and other L2+ intelligent driving functions.

[0073] Preferably, the system can be equipped with millimeter-wave radar and ultrasonic radar to increase the robustness of the system in strong light / glare scenes, and the additional sensors serve as a backup.

[0074] The present invention uses two cameras with fixed optical center distances and identical optical parameters to indirectly calculate the depth information of the pixels viewed by the two cameras through parallax. This allows the depth information of the target pixel to be calculated, and thus the distance information of the target obstacle to be obtained. This means that depth information can be calculated even without completing obstacle recognition.

[0075] The present invention realizes intelligent driving functions of driving and parking through a set of low-cost sensor systems, including adaptive cruise control, lane centering, active lane change, navigation assistance, parking distance warning, panoramic imaging, automatic parking and other driving / parking intelligent driving functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] To better understand the above and other objects, features, advantages, and functions of the present invention, reference may be made to the embodiments shown in the accompanying drawings. Like reference numerals in the accompanying drawings refer to like components. Those skilled in the art should understand that the accompanying drawings are intended to schematically illustrate preferred embodiments of the present invention and have no limiting effect on the scope of the present invention. The components in the drawings are not drawn to scale.

[0077] Figure 1 This is a flow chart of an environment perception method based on binocular and surround-view fisheye cameras of the present invention.

[0078] Figure 2 The present invention is an architecture diagram of a driving system that uses an environment perception method based on binocular and surround-view fisheye cameras.

[0079] Figure 3 The present invention is a schematic diagram of the installation structure of a driving system based on an environment perception method of binocular and surround-view fisheye cameras.

[0080] Figure 4 Schematic diagram of the components and functions of a driving system based on an environment perception method of binocular and surround-view fisheye cameras of the present invention DETAILED DESCRIPTION

[0081] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0082] As used herein, the term "including" and its variations represent open inclusion, i.e., "including but not limited to." Unless otherwise stated, the term "or" means "and / or." The term "based on" means "based at least in part on." The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment." The term "another embodiment" means "at least one additional embodiment." The terms "first," "second," etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0083] In order to at least partially solve one or more of the above problems and other potential problems, the embodiments of the present disclosure propose an environment perception method based on binocular and surround-view fisheye cameras, such as Figure 1 As shown, the following steps are included:

[0084] S1. Collect image information:

[0085] A binocular stereo camera is composed of a pair of cameras located on the same horizontal plane with a fixed baseline distance and identical optical performance parameters. The two cameras of the binocular stereo camera simultaneously and synchronously collect image information and complete the corresponding image information preprocessing in the ISP chip in the camera; the preprocessing includes binocular synchronous distortion correction, etc.

[0086] S2. Based on the image information collected by the binocular stereo camera, the depth information is obtained by the pixel difference of the corresponding points in an image, and stereoscopic visual obstacle detection is performed:

[0087] S21, selecting a pixel point on the image captured by the left camera as the left pixel point, selecting a range of areas with the pixel point as the center as the area to be matched, searching for a matching area with high similarity on the image captured by the right camera, and selecting the pixel point with the most similar features in the matching area as the matching point of the left pixel point, and estimating the disparity between the left pixel point and its matching point using a local optimization function;

[0088] S22, extracting local features of corners, edges, and planes of objects in the images captured by the left and right cameras, or extracting global features of complex shapes and overall image structures, performing feature matching on the images captured by the left and right cameras, and generating a disparity map based on the matching results, where each pixel value in the disparity map represents the horizontal pixel difference between the left and right images at that point;

[0089] The larger the parallax value, the closer the object is to the camera; the smaller the parallax value, the farther the object is from the camera;

[0090] S23. Obtain three-dimensional information using the positional deviation after comparing the corresponding point with a reference condition (the reference condition is the three-dimensional information of the obstacle determined based on the parallax between two corresponding points in the left and right cameras):

[0091] The disparity of corresponding points in images of the same object taken at different angles at the same time is obtained to restore the three-dimensional information of the object for detecting stereoscopic visual obstacles. How to use disparity to restore the three-dimensional information of an object belongs to the existing technology and does not need to be explained in detail.

[0092] S3. Perform semantic segmentation on the image to complete the detection of the drivable area:

[0093] The image is divided into multiple regions based on meaning. Each region represents a different category, such as people, cars, and signs. Each category is divided into dynamic targets and static targets based on whether it is movable and its type characteristics. For example, a sign is a static target because it is immovable, while a person is a dynamic target because he is movable. The images are obtained from the left and right cameras of a binocular stereo camera and multiple fisheye cameras.

[0094] S4. Perform deep learning-based object detection on the image:

[0095] Use common, trained deep learning neural networks to detect dynamic targets;

[0096] S5. For each pixel in the image, a classification result is given according to the dynamic target and the static target, and finally a dense prediction map is output; the goal of dense prediction is to predict each pixel in the input image and assign a category label;

[0097] The dense prediction map is a prediction map of the same size as the input image or scaled proportionally, containing pixel-level type recognition results. The recognition value / semantic information of each pixel is an integer representing the category to which it belongs, for example: 0 = background, 1 = person, 2 = car, 3 = tree, etc.;

[0098] This prediction preserves the edge information and semantic information of the original image to a great extent, which is used for scene understanding and decision planning of subsequent system algorithms;

[0099] S6. Input the perception results to the decision-making and planning module for trajectory planning. The trajectory generated by the decision-making and planning module is a series of path points.

[0100] Each waypoint is assigned a waypoint timestamp and waypoint speed. The waypoint timestamp is based on the camera exposure timestamp as the initial time. The estimated time of the vehicle at the waypoint indicates the estimated time of the vehicle's arrival at the spatial point. The waypoint speed is based on the speed at the time the camera took the photo as the initial speed. The estimated speed of the vehicle at the waypoint indicates the speed at which the vehicle will arrive at the spatial point.

[0101] Waypoint timestamps and waypoint speeds are calculated as follows:

[0102] a) The decision-making and planning module calculates the time required for the vehicle to reach each waypoint from its current position based on factors such as the target speed, acceleration limit, comfort requirements, traffic regulations (e.g., speed limits), and interactions between preceding vehicles and obstacles. The waypoint timestamps are calculated using the camera exposure timestamp as the initial time, and the waypoint speeds are calculated using the speed at the time the camera took the photo as the initial speed.

[0103] b) The decision-making and planning module calculates the path points along the vehicle's travel direction at preset time intervals Δt (e.g., 0.1s, 0.2s);

[0104] This interval determines the density of waypoints and the amount of computation.

[0105] S7, the prediction module is a submodule of the decision-making and planning module. The prediction module determines whether each path point on the trajectory is occupied at a certain point in time:

[0106] Because moving obstacles may temporarily block part of the road, each waypoint in the trajectory has a timestamp. We can combine the timestamp with the output of the prediction module to ensure that each waypoint on the trajectory is not occupied when we plan to pass through it. We use the following method to calculate and judge:

[0107] a) Taking the position, direction, and speed of a dynamic target at the past N time points (Tn, Tn-1, Tn-2) as input, the Kalman filter method is used to predict the position and speed of the dynamic target at the time point Tn+1;

[0108] b) For each path point on the trajectory at time t_i, determine the precise predicted state (position, speed, etc.) of the surrounding dynamic targets (such as the vehicle in front):

[0109] The prediction module outputs a series of future state points (trajectory) of the target object, each of which has its own timestamp {t_pred1, State1}, {t_pred2, State2}, ...;

[0110] At time t_i, the decision planning module finds the state point with the timestamp closest to t_i from the predicted target trajectory as the accurate predicted state State_target(t_i);

[0111] Interpolate between two adjacent state points of the predicted trajectory to obtain the precise predicted state State_target(t_i) of the target at time t_i; the greater the difference, the smoother and more accurate the predicted trajectory.

[0112] Interpolation methods used include: linear interpolation, kinematic model interpolation, etc.

[0113] c) The decision-making and planning module combines timestamps to perform spatial and temporal consistency collision checks, accurately assessing collision risks and adjusting the trajectory:

[0114] At time point t_i, the vehicle plan is located at Pi, and the dynamic target prediction is at State_target(t_i).position;

[0115] Calculate whether there is overlap between the area occupied by the vehicle (considering the vehicle outline / occupancy) and the area predicted to be occupied by the dynamic target at time t_i. If there is a collision risk (distance is too small or overlap), the decision-making planning module adjusts the speed plan, the spatial path, or both to avoid the risk;

[0116] The decision-making planning module adjusts the speed planning by changing the timestamp of t_i or subsequent path points. The timestamp refers to the unit time, such as 1s, 1.2s, and 1.4s. The timestamp cannot be changed, but the position and speed of the path points can be changed.

[0117] Preferably, the decision-making planning module adjusts the spatial path by changing the vehicle's planned position Pi.

[0118] By binding planned path points to precise future timestamps and combining them with the output of the prediction module at the same time point, the system achieves unified evaluation and collaborative planning of the vehicle's own future trajectory and the dynamic predicted trajectory of the surrounding environment in a spatiotemporal coordinate system, significantly improving driving safety and comfort in complex dynamic scenarios.

[0119] S8. The decision-making and planning module inputs the adjusted trajectory to the vehicle control module. The vehicle control module outputs the acceleration command and steering angle command for vehicle control based on the vehicle dynamics parameters. Finally, the control command is input to the vehicle platform wire control module. The vehicle platform wire control module integrates a communication protocol that can interact with the MEB platform architecture. By calling this communication interface, the control command can normally drive the original vehicle actuator to complete the execution of the acceleration / steering angle command.

[0120] The method described in this invention uses two cameras with identical optical parameters and fixed optical center distance to indirectly calculate the depth information of the pixels viewed by the two cameras through parallax. This allows the depth information of the target pixel to be calculated, thereby obtaining the distance information of the target obstacle. In other words, depth information is calculated without completing obstacle recognition.

[0121] The present invention also provides a driving system using an environment perception method based on binocular and surround-view fisheye cameras, see Figure 2 The system architecture diagram shown in the figure, Figure 3 The installation structure diagram shown in the figure is as follows: Figure 4 The system function diagram is shown.

[0122] The system consists of a binocular stereo camera consisting of two cameras with the same optical parameters, which are arranged on the front side of the vehicle, such as the top of the front windshield. The two cameras have the same field of view and resolution, are set on the same horizontal plane, and have a fixed baseline distance. They are used to complete the perception of the environmental information in front of the vehicle; a surround-view fisheye camera is arranged on the front, back, left and right sides of the vehicle body to complete the perception of the environmental information around the vehicle.

[0123] The ISP chip in the binocular stereo camera is used to complete the corresponding image data preprocessing;

[0124] The image data of the binocular stereo camera and the surround-view fisheye camera are transmitted to the central domain controller.

[0125] The central domain controller obtains depth information based on the pixel difference between the left and right cameras' corresponding points in an image, and performs stereoscopic obstacle detection:

[0126] A pixel on the image captured by the left camera is selected as the left pixel, and an area with this pixel as the center is selected as the area to be matched. A matching area with high similarity is found on the right camera. The pixel with the most similar features in this matching area is the matching point of the left pixel. The disparity between the left pixel and its matching point is estimated using a local optimization function.

[0127] The central domain controller extracts local features of corners, edges, and planes of objects in the images captured by the left and right cameras, or extracts global features of complex shapes and overall image structure. It performs feature matching on the images captured by the left and right cameras and generates a disparity map based on the matching results. Each pixel value in the disparity map represents the horizontal pixel difference between the left and right images at that point.

[0128] The larger the parallax value, the closer the object is to the camera; the smaller the parallax value, the farther the object is from the camera;

[0129] The central domain controller uses the position deviation after comparing the corresponding points with the reference situation to obtain the parallax of the corresponding points in the pictures of the same object taken at the same time but at different angles to restore the three-dimensional information of the object and detect stereoscopic obstacles.

[0130] The central domain controller performs semantic segmentation on the image and completes the detection of the drivable area:

[0131] The image is divided into multiple regions based on meaning. Each region represents a different category, such as people, cars, and signs. Each category is divided into dynamic targets and static targets based on whether they are movable. The images are obtained from the left and right cameras of a binocular stereo camera, as well as images from multiple fisheye cameras.

[0132] The central domain controller uses a common, trained deep learning neural network to detect dynamic obstacles.

[0133] The central domain controller generates a classification result for each pixel in the image and ultimately outputs a dense prediction map. The goal of dense prediction is to predict each pixel in the input image and assign a category label.

[0134] The dense prediction map is a prediction map of the same size as the input image or scaled proportionally, containing pixel-level type recognition results. The recognition value / semantic information of each pixel is an integer representing the category to which it belongs, for example: 0 = background, 1 = person, 2 = car, 3 = tree, etc.;

[0135] This prediction preserves the edge information and semantic information of the original image to a great extent, which is used for scene understanding and decision planning of subsequent system algorithms;

[0136] The central domain controller inputs the perception results to the decision-making and planning module for trajectory planning. The trajectory generated by the decision-making and planning module is a series of path points.

[0137] The central domain controller assigns a waypoint timestamp and speed to each waypoint. The waypoint timestamp uses the camera exposure timestamp as the initial time. The estimated time of the vehicle at that waypoint indicates the estimated time of the vehicle's arrival at that spatial point. The waypoint speed uses the speed at the time the camera took the photo as the initial speed. The estimated speed of the vehicle at that waypoint indicates the speed at which the vehicle will arrive at that spatial point.

[0138] The prediction module is a submodule of the decision-making and planning module. The prediction module determines whether each path point on the trajectory is occupied at a certain point in time:

[0139] Because moving obstacles may temporarily block part of the road, each waypoint in the trajectory has a timestamp. We can combine the timestamp with the output of the prediction module to ensure that each waypoint on the trajectory is not occupied when we plan to pass it. We use the following method:

[0140] a) Taking the position, direction, and speed of a dynamic target at the past N time points (Tn, Tn-1, Tn-2) as input, the Kalman filter method is used to predict the position and speed of the dynamic target at the time point Tn+1;

[0141] b) For each path point on the trajectory at time t_i, determine the precise predicted state (position, speed, etc.) of the surrounding dynamic targets (such as the vehicle in front):

[0142] The prediction module outputs a series of future state points (trajectory) of the target object, each of which has its own timestamp {t_pred1, State1}, {t_pred2, State2}, ...;

[0143] At time t_i, the decision planning module finds the state point with the timestamp closest to t_i from the predicted target trajectory as the accurate predicted state State_target(t_i);

[0144] The decision-making and planning module combines timestamps to perform spatial and temporal consistency collision checks, accurately assesses collision risks, and adjusts the trajectory:

[0145] At time point t_i, the vehicle plan is located at Pi, and the dynamic target prediction is at State_target(t_i).position;

[0146] Calculate whether there is overlap between the area occupied by the vehicle (considering the vehicle outline / occupancy) and the area predicted to be occupied by the dynamic target at time t_i. If a collision risk is predicted (the distance is too small or there is overlap), the decision planning module adjusts the speed plan, the spatial path, or both to avoid the risk.

[0147] The decision-making and planning module inputs the adjusted trajectory to the vehicle control module. The vehicle control module outputs acceleration and steering angle commands based on vehicle dynamics parameters. Finally, the control commands are input to the vehicle platform's drive-by-wire module, which integrates a communication protocol interface that can interact with the MEB platform architecture. By calling this communication protocol interface, the control commands can be executed by the vehicle actuator to complete the acceleration / steering angle commands.

[0148] The wire control module completes the wire control adaptation for the vehicle platform and communicates with the vehicle actuator through the MEB vehicle architecture protocol;

[0149] Vehicle actuators such as drive motors, steering gears, and brakes provide correct control response and feedback.

[0150] Ultimately, intelligent driving / parking functions will be realized, including adaptive cruise control, lane centering, lever lane change, automatic parking and other L2+ intelligent driving functions.

[0151] like Figure 2 As shown, the system can be equipped with millimeter-wave radar and ultrasonic radar to increase the robustness of the system in strong light / glare scenarios, and the additional sensors serve as a backup.

[0152] The system described in the present invention realizes intelligent driving functions for driving and parking through a set of low-cost sensor systems, including adaptive cruise control, lane centering, active lane change, navigation assistance, parking distance warning, panoramic imaging, automatic parking and other intelligent driving / parking functions.

[0153] The system can be applied to all models of Volkswagen's MEB platform, can perform vehicle control, and is scalable. By changing the software module in the wire-controlled controller, it can support more platform architectures and control the vehicle's lateral and longitudinal acceleration, longitudinal deceleration, and left and right turn signals; through binocular stereo vision combined with the visual images of the camera around the vehicle, it can complete the perception of static and dynamic obstacles in the environment.

Claims

1. An environmental perception method based on binocular and surround-view fisheye cameras, characterized in that: The steps include: S1, collecting three-dimensional information of the image and preprocessing the image; S2, performing stereoscopic visual obstacle detection based on image information collected by the binocular stereo camera; S3, performing feature matching on the images collected by the left and right cameras of the binocular stereo camera, and generating a disparity map based on the matching results; S4. Obtain three-dimensional information using the position deviation after comparing the corresponding points with the reference situation: Obtain the parallax of corresponding points in images of the same object taken at the same time but at different angles to restore the object's three-dimensional information for stereoscopic obstacle detection. S5. Perform dynamic obstacle target detection on the image based on deep learning neural network training; S6. Divide the image into multiple regions according to their meaning, and classify each category into dynamic targets and static targets according to whether they are movable. Give a classification result for each pixel in the image and output a dense prediction map. S7, input the perception results to the decision planning module for trajectory planning, and assign a waypoint timestamp and waypoint speed to each waypoint; S8, the prediction module is a submodule of the decision-making and planning module. The prediction module determines whether each path point on the trajectory is occupied at a certain point in time, assesses the collision risk, and adjusts the trajectory: S9. The decision-making and planning module inputs the adjusted trajectory to the vehicle control module, modifies the acceleration command and the steering angle command, and enables the platform wire control module to drive the original vehicle actuator to complete the execution of the acceleration / steering angle command.

2. The method for environmental perception based on binocular and surround-view fisheye cameras according to claim 1, characterized in that: Each path point in the trajectory of step S7 has a timestamp. The following method is used to determine whether each path point on the trajectory is occupied: a) Taking the position, orientation, and velocity information of the dynamic target at the past N time points (Tn, Tn-1, Tn-2) as input, the Kalman filter method is used to predict the position and velocity of the dynamic target at the time point Tn+1; b) For each path point on the trajectory at time t_i, determine the precise predicted state of the dynamic targets around it: The prediction module outputs a series of future state points of the target object as a trajectory, each of which has its own timestamp {t_pred1, State1}, {t_pred2, State2}, ...; At time t_i, the decision planning module finds the state point with the timestamp closest to t_i from the predicted target trajectory as the accurate predicted state State_target(t_i); c) The decision-making and planning module combines timestamps to perform collision checks, assess collision risks, and adjust trajectories.

3. The method for environmental perception based on binocular and surround-view fisheye cameras according to claim 1, characterized in that: The waypoint timestamp and waypoint speed are calculated as follows: a) The decision-making and planning module calculates the time required for the vehicle to reach each waypoint from its current position based on the target speed, acceleration limit, comfort requirements, traffic regulations, and the interaction between the preceding vehicle and obstacles. It calculates the waypoint timestamp using the camera exposure timestamp as the initial time and the waypoint velocity using the camera's speed at the time of the photo as the initial velocity. b) The decision-making and planning module calculates the path points along the vehicle's travel direction at preset time intervals Δt; This interval determines the density of waypoints and the amount of computation.

4. The method for environmental perception based on binocular and surround-view fisheye cameras according to claim 1, characterized in that: The preprocessing includes binocular synchronization and distortion correction.

5. The method for environmental perception based on binocular and surround-view fisheye cameras according to claim 1, characterized in that: Interpolate between two adjacent state points of the trajectory to obtain the accurate predicted state State_target(t_i) of the target at time t_i.

6. The method for environmental perception based on binocular and surround-view fisheye cameras according to claim 5, characterized in that: The interpolation method used is: linear interpolation and kinematic model interpolation.

7. The method for environmental perception based on binocular and surround-view fisheye cameras according to claim 1, characterized in that: The decision planning module adjusts the speed planning by changing the time stamp of the t_i moment or the subsequent path point.

8. The method for environmental perception based on binocular and surround-view fisheye cameras according to claim 7, characterized in that: The decision-making planning module adjusts the spatial path by changing the vehicle's planned position Pi.

9. A driving system using an environment perception method based on binocular and surround-view fisheye cameras, characterized in that: It includes two cameras with the same parameters to form a binocular stereo camera, which is used to complete the perception of the environment in front of the vehicle; a surround-view fisheye camera is arranged on the front, back, left and right sides of the vehicle body to complete the perception of the environment around the vehicle; The image data of the binocular stereo camera and the surround-view fisheye camera are transmitted to the central domain controller; The central domain controller obtains depth information based on the pixel difference between the left and right cameras' images of corresponding points, performs stereoscopic obstacle detection, and performs feature matching on the images to generate a disparity map. Each pixel value in the disparity map represents the horizontal pixel difference between the left and right images at that point. The central domain controller uses the position deviation of the corresponding points after comparing them with the reference situation to obtain the parallax of the corresponding points in the pictures of the same object taken at different angles at the same time to detect stereoscopic obstacles. The central domain controller performs semantic segmentation on the image, detects the drivable area, and uses a common, trained deep learning neural network to detect dynamic obstacles. It generates a classification result for each pixel in the image and ultimately outputs a dense prediction map to the decision-making and planning module for trajectory planning. The trajectory generated by the decision-making and planning module is a series of path points. The prediction module is a submodule of the decision-making and planning module. The prediction module determines whether each path point on the trajectory is occupied at a certain point in time and adjusts the trajectory. The decision-making and planning module inputs the adjusted trajectory to the vehicle control module, modifies the acceleration command and steering angle command, and enables the wire control module to control the vehicle actuator to complete the acceleration / steering angle command.

10. The driving system using the environment perception method based on binocular and surround-view fisheye cameras according to claim 9, characterized in that: The prediction module adjusts the trajectory in the following ways: a) Taking the position, orientation, and velocity information of the dynamic target at the past N time points (Tn, Tn-1, Tn-2) as input, the Kalman filter method is used to predict the position and velocity of the dynamic target at the time point Tn+1; b) For each path point on the trajectory at time t_i, determine the precise predicted state of the dynamic targets around it: The prediction module outputs a series of future state points of the target object as a trajectory, each of which has its own timestamp {t_pred1, State1}, {t_pred2, State2}, ...; At time t_i, the decision planning module finds the state point with the timestamp closest to t_i from the predicted target trajectory as the accurate predicted state State_target(t_i); The decision-making and planning module combines timestamps to perform spatial and temporal consistency collision checks, accurately assesses collision risks, and adjusts the trajectory: At time point t_i, the vehicle plan is located at Pi, and the dynamic target prediction is at State_target(t_i).position; Calculate whether there is overlap between the area occupied by the vehicle and the area predicted to be occupied by the dynamic target at time t_i. If the distance is too small or there is a risk of collision due to the overlapping prediction, adjust the speed planning or the spatial path, or both.

11. The driving system using an environment perception method based on binocular and surround-view fisheye cameras according to claim 9, characterized in that: The vehicle actuator includes: a drive motor, a steering gear, and a brake.

12. The driving system using the environment perception method based on binocular and surround-view fisheye cameras as claimed in claim 9, characterized in that: include: Millimeter-wave radar and / or ultrasonic radar to acquire environmental data.

Images