Multi-view cooperation based blind area perception and control method, device, equipment, system and vehicle

Abstract

The application discloses a blind area sensing and control method, device, equipment, system and vehicle based on multi-view cooperation, which comprises the following steps: receiving first environment image information from a surround-view camera group; receiving a steering signal, a steering wheel rotation angle and a vehicle speed from a vehicle chassis system; sending an opening signal to a corresponding side of a rear oblique camera according to the steering signal and receiving second environment image information of the corresponding side of the rear oblique camera; adjusting a user interested region in the second environment image information according to the steering wheel rotation angle and the vehicle speed; constructing a top view of a vehicle surrounding environment according to the first environment image information and image information in the user interested region, and determining target obstacle information according to the top view; calculating a collision time of the target obstacle and the vehicle according to the target obstacle information and a driving track of the vehicle; sending an alarm signal when the collision time is within a preset warning range; and sending an active intervention signal when the collision time is within a preset active intervention range. The application is helpful to improve safety in complex and changeable road scenes.

Description

Technical Field

[0001] This invention relates to the technical field of intelligent driving, and more specifically, to a method, device, equipment, system, and vehicle based on multi-view collaborative blind spot perception and control. Background Technology

[0002] With the rapid development of automotive active safety technologies and autonomous driving systems, a vehicle's environmental perception capabilities have a crucial impact on driver assistance and autonomous driving functions. However, in complex and ever-changing road scenarios, existing vehicle environmental perception systems still face numerous technical bottlenecks, resulting in insufficient reliability of decision-making by vehicle driver assistance systems in such conditions. For example, when existing vehicles make sharp turns, their forward-facing cameras, due to their limited field of view, cannot effectively capture dynamic information from blind spots to the sides and rear. They need to rely on rearview mirrors or short-range millimeter-wave radar, but these systems often lack sufficient resolution, making it easy to miss vulnerable road users. This can cause the driver assistance system to fail in complex turning scenarios and may even lead to side collisions. At T-junctions with obstructed views, such as green belts or buildings, existing millimeter-wave radars, due to their limited angular resolution, struggle to distinguish between stationary obstacles and laterally moving vehicles and cannot accurately predict the trajectory of oncoming vehicles, leading to an increased misjudgment rate in driver assistance systems. When making U-turns on asymmetrical road topologies, existing vehicles need to perceive the asymmetrical road structure in real time, such as the location of gaps in the central median and changes in the curvature of the oncoming lane. However, existing driver assistance systems rely on pre-stored information from high-precision maps for decision-making, making it difficult to cope with dynamically changing road conditions, such as temporary construction barriers. This can result in trajectory planning delays when making unprotected U-turns or insufficient information about the corresponding side and rearward directions when the driver is actively driving, increasing driving risks. Summary of the Invention

[0003] To address at least one aspect of the above problems, the present invention provides a method, apparatus, device, system, and vehicle for blind spot perception and control based on multi-view collaboration.

[0004] Firstly, this application provides a blind spot perception and control method based on multi-view collaboration, comprising the following steps: receiving vehicle speed in real time from the vehicle chassis system; when the vehicle speed is lower than a preset speed threshold, receiving first environmental image information located in front of, behind, to the left, and to the right of the vehicle from a surround-view camera group in real time; calculating the current lane information based on the first environmental image information located in front of, behind, to the left, and to the right of the vehicle using deep learning algorithms and SLAM algorithms; receiving steering signals and steering wheel angle in real time from the vehicle chassis system, and adjusting the steering signals according to the steering signals of the rear-view camera group. An activation signal is sent to activate the corresponding oblique rearview camera in the oblique rearview camera group; real-time second environmental image information located obliquely behind the vehicle is received from the corresponding oblique rearview camera in the oblique rearview camera group; based on the steering wheel angle and vehicle speed, and using a vehicle dynamics model, the vehicle's trajectory and target lane information are predicted; based on the steering wheel angle and the vehicle dynamics model, the angle between the vehicle and the target lane is calculated, and the user's region of interest in the second environmental image information is adjusted based on the angle between the vehicle and the target lane and the steering wheel angle, and then based on an image recognition algorithm... The system extracts image information from the user's region of interest. Based on the first environmental image information located in front of, behind, to the left and right of the vehicle, and the image information within the user's region of interest, it constructs a top-down view of the vehicle's surrounding environment using the BEV (Bird's Eye View) algorithm and a multi-view perception fusion algorithm, and marks the current lane information and the target lane information in this top-down view. Based on the top-down view of the vehicle's surrounding environment, it determines the target obstacle information using a deep learning algorithm. The target obstacle information includes the number of target obstacles, the semantic type of each target obstacle, the position of each target obstacle, and the outline of each target obstacle. Based on the target obstacle information in consecutive image frames, it predicts the trajectory of several target obstacles. Based on the predicted trajectory of several target obstacles and the predicted trajectory of the vehicle, it calculates the collision time between the target obstacle and the vehicle. When the collision time between the target obstacle and the vehicle is within a preset warning range, it sends an alarm signal to the in-vehicle infotainment system to trigger an alarm. When the collision time between the target obstacle and the vehicle is within a preset active intervention range, it sends an active intervention signal to the power steering system to trigger active intervention.

[0005] Preferably, the method further includes the following steps: based on the first environmental image information located in front of the vehicle, extracting the drivable area of ​​the road ahead using a deep learning algorithm; when the shape of the drivable area of ​​the road ahead is T-shaped, identifying the target obstacle information located in the transverse lane based on the first environmental image information located in front of the vehicle using a deep learning algorithm; calculating the collision time between the target obstacle in the transverse lane and the vehicle based on the target obstacle information in the transverse lane in consecutive image frames and the vehicle's driving trajectory; when the collision time between the target obstacle in the transverse lane and the vehicle is within a preset warning range, sending an alarm signal to the in-vehicle infotainment system to trigger an alarm; when the collision time between the target obstacle in the transverse lane and the vehicle is within a preset active intervention range, sending an active intervention signal to the power assist system to trigger active intervention.

[0006] Preferably, the method further includes the following steps: determining the actual running trajectories of several target obstacles based on target obstacle information in consecutive image frames; marking the target obstacle as a static obstacle when its actual running trajectory has a fixed spatial relationship with the actual running trajectory of the vehicle; marking the target obstacle as a dynamic obstacle when its actual running trajectory has an independent motion relationship with the actual running trajectory of the vehicle; forming a first safe area about the static obstacle by extending a buffer zone outward from the edge of the static obstacle's outline based on the spatial occupancy characteristics of the static obstacle's outline, wherein the width of the buffer zone is determined according to the height of the static obstacle; calculating the distance between the first safe areas of adjacent static obstacles; setting the area between adjacent first safe areas about the static obstacle as a second safe area when the distance is less than a preset distance threshold, wherein the preset distance threshold is determined according to the width of the vehicle; and constructing a first driving area by removing the first and second safe areas from the top view of the vehicle's surrounding environment. A safe driving zone is defined, and its boundaries are smoothed. Based on the actual trajectory of the dynamic obstacle, the possible location distribution of the dynamic obstacle at multiple future moments is predicted, forming a risk cone-shaped area that spreads over time. The cone angle of the risk cone-shaped area is determined according to the semantic type of the dynamic obstacle. Based on the braking capacity range of the vehicle and the risk cone-shaped area, the non-overlapping area between the two is defined as a second driving safe zone, and its boundaries are smoothed. The first and second driving safe zones are spatially superimposed. When the first and second driving safe zones overlap, the overlapping area is marked as an absolutely safe driving zone in the top view of the vehicle's surrounding environment, and the edge of the absolutely safe driving zone is optimized for transition. The marking information is sent to the in-vehicle infotainment system. When the first and second driving safe zones do not overlap, the second driving safe zone is marked in the top view of the vehicle's surrounding environment, and the marking information and alarm signal are sent to the in-vehicle infotainment system.

[0007] Preferably, the method for adjusting the user's region of interest is as follows: determining the size of the user's region of interest based on the steering wheel angle; determining the position of the user's region of interest based on the angle between the vehicle and the target lane, so that the user's region of interest is located in the center area within the angle range between the vehicle and the target lane.

[0008] Preferably, the method for constructing a top-down view of the vehicle's surrounding environment comprises: converting the first environmental image information located in front of, behind, to the left, and to the right of the vehicle into first environmental image information from a bird's-eye view in the coordinate system of the surround-view camera group, respectively, based on the BEV algorithm; according to the relative positions of the surround-view camera group coordinate system and the vehicle coordinate system, converting the first environmental image information from the bird's-eye view in the coordinate system of the surround-view camera group located in front of, behind, to the left, and to the right of the vehicle into first environmental image information from a bird's-eye view in the vehicle coordinate system, respectively; and converting the image information within the user's area of ​​interest into a rear-view perspective based on the BEV algorithm. Image information of the user's region of interest from a bird's-eye view in the camera group coordinate system; based on the relative positions of the rear-view camera group coordinate system and the vehicle coordinate system, the image information of the user's region of interest from a bird's-eye view in the rear-view camera group coordinate system is converted into the image information of the user's region of interest from a bird's-eye view in the vehicle coordinate system; image calibration and image stitching are performed on the first environmental image information from the bird's-eye view in the vehicle coordinate system located in front of, behind, to the left and right of the vehicle and the image information of the user's region of interest from the bird's-eye view in the vehicle coordinate system to form a top view of the vehicle's surrounding environment in the vehicle coordinate system.

[0009] Preferably, the alarm methods of the in-vehicle infotainment system include highlighting the risk area of ​​the target lane on the screen, using dynamic arrow prompts on the screen to adjust the steering angle or delay steering, sound warnings, and voice reminders.

[0010] Preferably, the active intervention methods of the power steering system include correcting the steering wheel, automatically applying light braking to reduce speed, and limiting the steering angle.

[0011] Secondly, this application provides a blind spot perception and control device based on multi-view collaboration. The device includes: a first processing module configured to receive vehicle speed in real time from a vehicle chassis system; when the vehicle speed is below a preset speed threshold, receiving first environmental image information located in front of, behind, to the left, and to the right of the vehicle from a surround-view camera group in real time; and calculating current lane information based on the first environmental image information located in front of, behind, to the left, and to the right of the vehicle using deep learning and SLAM algorithms; and a second processing module configured to receive steering input in real time from the vehicle chassis system. The system receives signals and steering wheel angles, and sends an activation signal to the oblique rearview camera group based on the steering signal, causing the corresponding oblique rearview camera in the oblique rearview camera group to turn on; it receives real-time second environmental image information located obliquely behind the vehicle from the corresponding oblique rearview camera in the oblique rearview camera group; based on the steering wheel angle and vehicle speed, it predicts the vehicle's driving trajectory and the target lane information based on the vehicle dynamics model; based on the steering wheel angle and the vehicle dynamics model, it calculates the angle between the vehicle and the target lane, and adjusts the second environmental image information based on the angle between the vehicle and the target lane and the steering wheel angle. The system first identifies the user's region of interest (ROI) in the environmental image information and extracts image information from the ROI based on image recognition algorithms. The third processing module is configured to construct a top-down view of the vehicle's surrounding environment based on the first environmental image information located in front of, behind, to the left, and to the right of the vehicle, and the image information within the ROI, using BEV algorithms and multi-view perception fusion algorithms. The current lane information and the target lane information are marked in this top-down view. Based on the top-down view of the vehicle's surrounding environment, target obstacle information is determined using deep learning algorithms. The trajectories of several target obstacles are predicted based on the target obstacle information in consecutive image frames. The collision time between the target obstacle and the vehicle is calculated based on the predicted trajectories of the target obstacles and the predicted driving trajectory of the vehicle. A warning module is configured to send an alarm signal to the in-vehicle infotainment system when the collision time between the target obstacle and the vehicle is within a preset warning range, causing the in-vehicle infotainment system to issue an alarm. An active intervention module is configured to send an active intervention signal to the power steering system when the collision time between the target obstacle and the vehicle is within a preset active intervention range, causing the power steering system to actively intervene.

[0012] Thirdly, this application provides a blind spot perception and control device based on multi-view collaboration. The device includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the blind spot perception and control method based on multi-view collaboration described above.

[0013] Fourthly, this application provides a blind spot perception and control system based on multi-view collaboration, including a surround-view camera group, a rear-view camera group, and the aforementioned blind spot perception and control device based on multi-view collaboration. The surround-view camera group includes four surround-view cameras, respectively fixedly installed at the center of the front bumper, the center of the rear bumper, below the left rearview mirror, and below the right rearview mirror of the vehicle, for acquiring first environmental images of the front, rear, left, and right sides of the vehicle, and transmitting these first environmental images to the blind spot perception and control device based on multi-view collaboration. The rear-view camera group includes two rear-view cameras, which are fisheye cameras, respectively fixedly installed at the front left fender and the front right fender of the vehicle, for acquiring second environmental images of the left and right rear sides of the vehicle, and transmitting these second environmental images to the blind spot perception and control device based on multi-view collaboration.

[0014] Fifthly, this application provides a vehicle including the aforementioned blind spot perception and control system based on multi-view collaboration.

[0015] The present invention provides a method, apparatus, device, system, and vehicle for blind spot perception and control based on multi-view collaboration, which has the following beneficial effects:

[0016] (1) The vehicle's surrounding environment is perceived and its position is detected by the surround-view camera group, enabling lane positioning of the vehicle in low-speed scenarios; the rear-view angle of the vehicle is detected by the left and right rear-view angles of the fisheye oblique cameras located on both sides of the vehicle, and the user's area of ​​interest in the oblique rear view is dynamically adjusted according to the steering wheel angle and vehicle speed, so that it can not only monitor the rear of the current lane, but also cover the target lane after turning, effectively capturing the dynamic information of the side and rear blind spots. The image information detected by the surround-view camera group and the image information detected by the oblique rear-view camera group are then fused to achieve collaborative perception and positioning of the rear view, providing the driver with a more reliable basis for steering decisions, which helps to improve safety in scenarios such as large-angle turns and U-turns.

[0017] (2) When driving at low speed to a T-junction, the vehicle uses a surround-view camera located in front of the vehicle to perceive obstacles in the transverse lanes of the T-junction, thereby enabling blind spot assistance and improving vehicle safety in complex intersections and other scenarios.

[0018] (3) By distinguishing between static and dynamic obstacles, the first and second safe driving zones are determined based on the static and dynamic obstacles respectively. The absolutely safe driving zone or the safe driving zone that can avoid dynamic obstacles is then determined based on the first and second safe driving zones. The safe driving zones are then displayed in the in-vehicle infotainment system according to their type, which further improves the assisted driving capability and helps to further improve the driving safety of the vehicle in urban roads, complex intersections, and unprotected left turns. Furthermore, when determining the first safe driving zone, the method of eliminating the safe zone related to static obstacles is adopted, while when determining the second safe driving zone, the method of predicting the risk cone area of ​​the dynamic obstacle is adopted, and then the determination is made based on the braking capacity range and risk cone area of ​​the vehicle. This helps to reduce computing power while ensuring the accuracy of the safe driving zone judgment. Attached Figure Description

[0019] To better understand the above and other objects, features, advantages, and functions of the present invention, reference can be made to the embodiments shown in the accompanying drawings. The same reference numerals in the drawings refer to the same parts. Those skilled in the art should understand that the drawings are intended to schematically illustrate preferred embodiments of the invention and do not limit the scope of the invention in any way; the parts in the drawings are not drawn to scale.

[0020] Figure 1 A flowchart of a blind spot perception and control method based on multi-view collaboration according to an embodiment of the present invention is shown;

[0021] Figure 2 A block diagram of a blind spot perception and control device based on multi-view collaboration according to an embodiment of the present invention is shown;

[0022] Figure 3 A schematic diagram of the camera distribution in a blind spot perception and control method system based on multi-view collaboration according to an embodiment of the present invention is shown.

[0023] Explanation of reference numerals in the attached figures:

[0024] 1. Surround view camera; 2. Oblique rear view camera; 31. First processing module; 32. Second processing module; 33. Third processing module; 34. Early warning module; 35. Active intervention module. Detailed Implementation

[0025] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0026] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0027] To at least partially address one or more of the aforementioned problems and other potential issues, embodiments of this disclosure propose a blind spot perception and control method based on multi-view collaboration, such as... Figure 1 As shown, it includes the following steps:

[0028] The vehicle speed is received in real time from the vehicle chassis system. When the vehicle speed is lower than a preset speed threshold (e.g., when the vehicle is at low speed, for example, the preset speed threshold is set to 15 km / h), the first environmental image information located in front of, behind, to the left, and to the right of the vehicle is received in real time from the surround-view camera group. Based on the first environmental image information located in front of, behind, to the left, and to the right of the vehicle, the current lane information is calculated based on deep learning algorithms and SLAM (Simultaneous Mapping and Localization) algorithms. Specifically, based on the first environmental image information located in front of, behind, to the left, and to the right of the vehicle, the lane lines around the vehicle are detected and lane line features are extracted using deep learning algorithms. Based on the SLAM algorithm, the current lane information is calculated by associating lane line feature points from historical frames. The lane information includes the lane line shape, lane line type, and the angle and position of the vehicle relative to the lane line.

[0029] The system receives steering signals and steering wheel angles in real time from the vehicle chassis system, and sends an activation signal to the oblique rearview camera group according to the steering signal, so that the oblique rearview camera 2 on the corresponding side of the oblique rearview camera group is activated. For example, when the steering signal is to the left, the oblique rearview camera 2 on the left side of the vehicle is activated. The system also receives second environmental image information located diagonally behind the vehicle in real time from the oblique rearview camera 2 on the corresponding side of the oblique rearview camera group. Since the oblique rearview camera 2 in this application is a fisheye oblique camera, its field of view ranges from 0 to 190°. Based on the steering wheel angle and vehicle speed, and using a vehicle dynamics model, the vehicle's trajectory and the target lane information are predicted. Based on the steering wheel angle and the vehicle dynamics model, the angle between the vehicle and the target lane is calculated. Then, based on the angle between the vehicle and the target lane and the steering wheel angle, the region of interest (ROI) in the second environmental image information is adjusted. Image information within the ROI is extracted using an image recognition algorithm. Specifically, the ROI adjustment method is as follows: the size of the ROI is determined based on the steering wheel angle; the larger the steering wheel angle, the larger the ROI. The position of the ROI is determined based on the angle between the vehicle and the target lane, ensuring that the ROI is located in the center of the angle range between the vehicle and the target lane, thus enabling it to monitor not only the area behind the current lane but also the target lane after the turn.

[0030] Based on the first environmental image information located in front of, behind, to the left, and to the right of the vehicle, and the image information within the user's region of interest, a top-down view of the vehicle's surrounding environment is constructed using the BEV algorithm and a multi-view perception fusion algorithm. The current lane information and the target lane information are then marked in this top-down view. Specifically, the method for constructing the top-down view of the vehicle's surrounding environment is as follows: the first environmental image information located in front of, behind, to the left, and to the right of the vehicle is converted into bird's-eye view first environmental image information in the coordinate system of the surround-view camera group based on the BEV algorithm; according to the relative position of the surround-view camera group coordinate system and the vehicle coordinate system, the bird's-eye view first environmental image information located in the coordinate system of the surround-view camera group in front of, behind, to the left, and to the right of the vehicle is correspondingly converted into bird's-eye view first environmental image information in the vehicle coordinate system; and the image information within the user's region of interest is converted into a rear-view image using the BEV algorithm. Image information of the user's region of interest from a bird's-eye view in the camera group coordinate system; based on the relative positions of the rear-view camera group coordinate system and the vehicle coordinate system, the image information of the user's region of interest from the bird's-eye view in the rear-view camera group coordinate system is converted into the image information of the user's region of interest from the bird's-eye view in the vehicle coordinate system; image calibration and image stitching are performed on the first environmental image information of the bird's-eye view in the vehicle coordinate system located in front of, behind, to the left and right of the vehicle and the image information of the user's region of interest from the bird's-eye view in the vehicle coordinate system to form a top view of the vehicle's surrounding environment in the vehicle coordinate system.

[0031] Based on a top-down view of the vehicle's surroundings, and using a deep learning algorithm, target obstacle information is determined. This information includes the number of obstacles, the semantic type of each obstacle, its location, and its outline. The trajectories of several obstacles are predicted based on the obstacle information in consecutive image frames. The collision times between the obstacles and the vehicle are then calculated based on both the predicted trajectories and the vehicle's predicted trajectory.

[0032] When the collision time between the target obstacle and the vehicle is within a preset warning range, indicating a risk of collision (e.g., the preset warning range is set to less than 10 seconds), an alarm signal will be sent to the in-vehicle infotainment system, triggering an alarm. Specifically, the infotainment system's alarm methods include highlighting the risk area of ​​the target lane on the screen, using dynamic arrows on the screen to prompt adjustments to the steering angle or delaying steering, providing audible warnings, and giving voice reminders. When the collision time between the target obstacle and the vehicle is within a preset active intervention range, indicating a high risk of collision (e.g., the preset active intervention range is set to less than 3 seconds), an active intervention signal will be sent to the power steering system, triggering active intervention. The power steering system's active intervention methods include correcting the steering wheel, automatically applying light braking to reduce speed, and limiting the steering angle.

[0033] In a preferred embodiment, the method further includes the following steps: based on the first environmental image information located in front of the vehicle, extracting the drivable area of ​​the road ahead using a deep learning algorithm; when the shape of the drivable area of ​​the road ahead is T-shaped, identifying the target obstacle information located in the transverse lane based on the first environmental image information located in front of the vehicle using a deep learning algorithm; calculating the collision time between the target obstacle in the transverse lane and the vehicle based on the target obstacle information in the transverse lane in consecutive image frames and the vehicle's driving trajectory; when the collision time between the target obstacle in the transverse lane and the vehicle is within a preset warning range, sending an alarm signal to the in-vehicle infotainment system to trigger an alarm; when the collision time between the target obstacle in the transverse lane and the vehicle is within a preset active intervention range, sending an active intervention signal to the power assist system to trigger active intervention.

[0034] In a preferred embodiment, the method further includes the following steps: determining the actual running trajectories of several target obstacles based on target obstacle information in consecutive image frames; marking the target obstacle as a static obstacle when its actual running trajectory has a fixed spatial relationship with the actual running trajectory of the vehicle; marking the target obstacle as a dynamic obstacle when its actual running trajectory has an independent motion relationship with the actual running trajectory of the vehicle; expanding a buffer zone outward from the edge of the static obstacle contour to form a first safe area for the static obstacle based on the spatial occupancy characteristics of the static obstacle contour, wherein the width of the buffer zone is determined according to the height of the static obstacle, specifically, the higher the height of the static obstacle, the wider its buffer zone; calculating the distance between the first safe areas of adjacent static obstacles; setting the area between adjacent first safe areas of static obstacles as a second safe area when the distance is less than a preset distance threshold, wherein the preset distance threshold is determined according to the width of the vehicle; constructing a first driving safe area by removing the first and second safe areas from the top view of the vehicle's surrounding environment, and smoothing its boundaries to avoid path jitter; predicting the future running trajectory of dynamic obstacles based on their actual running trajectories. The potential location distribution at multiple times forms a risk cone region that spreads over time. The cone angle of this risk cone region is determined based on the semantic type of the dynamic obstacle. Specifically, vehicles, pedestrians, and animals use different cone angles, which can be achieved by pre-setting the corresponding cone angles according to the semantic type of the dynamic obstacle. A second safe driving zone is defined as the area where the vehicle's braking capacity range and the risk cone region do not intersect, and its boundaries are smoothed. The first and second safe driving zones are spatially superimposed. When the first and second safe driving zones intersect, [the following is omitted as the text is incomplete and requires further context]. The overlapping area is marked as an absolutely safe driving zone in the top view of the vehicle's surroundings, and the edges of the absolutely safe driving zone are optimized for transition. This marking information is then sent to the in-vehicle infotainment system. For example, the absolutely safe driving zone can be marked in green in the in-vehicle infotainment system. When there is no overlap between the first and second safe driving zones, the second safe driving zone is marked in the top view of the vehicle's surroundings, and this marking information and an alarm signal are sent to the in-vehicle infotainment system. For example, the second safe driving zone can be marked in yellow in the in-vehicle infotainment system, accompanied by a voice prompt suggesting braking risks.

[0035] This application also provides a blind spot perception and control device based on multi-view collaboration, such as Figure 2As shown, the device includes: a first processing module 31, configured to receive vehicle speed in real time from the vehicle chassis system; when the vehicle speed is lower than a preset speed threshold, receive first environmental image information located in front of, behind, to the left and right of the vehicle from the surround-view camera group in real time; and calculate the current lane information based on the first environmental image information located in front of, behind, to the left and right of the vehicle using deep learning algorithms and SLAM algorithms. In a preferred embodiment, the first processing module 31 is further configured to extract the drivable area of ​​the road ahead based on the first environmental image information located in front of the vehicle using deep learning algorithms; when the shape of the drivable area of ​​the road ahead is T-shaped, identify the target obstacle information located in the transverse lane based on the first environmental image information located in front of the vehicle using deep learning algorithms; and calculate the collision time between the target obstacle located in the transverse lane and the vehicle based on the target obstacle information located in the transverse lane in consecutive image frames and the vehicle's driving trajectory.

[0036] The second processing module 32 is configured to receive steering signals and steering wheel angles in real time from the vehicle chassis system, and send an activation signal to the oblique rearview camera group according to the steering signal, so that the oblique rearview camera 2 on the corresponding side of the oblique rearview camera group is activated; receive second environmental image information located obliquely behind the vehicle in real time from the oblique rearview camera 2 on the corresponding side of the oblique rearview camera group; predict the driving trajectory of the vehicle and the target lane information based on the vehicle dynamics model according to the steering wheel angle and vehicle speed; calculate the angle between the vehicle and the target lane according to the steering wheel angle and the vehicle dynamics model; adjust the user interest region in the second environmental image information according to the angle between the vehicle and the target lane and the steering wheel angle; and extract the image information in the user interest region based on the image recognition algorithm.

[0037] The third processing module 33 is configured to construct a top-down view of the vehicle's surrounding environment based on the first environmental image information located in front of, behind, to the left and right of the vehicle and the image information within the user's region of interest, using the BEV algorithm and multi-view perception fusion algorithm, and to annotate the current lane information and the target lane information to be reached in this top-down view; determine the target obstacle information based on the top-down view of the vehicle's surrounding environment using a deep learning algorithm; predict the running trajectory of several target obstacles based on the target obstacle information in consecutive image frames; and calculate the collision time between the target obstacle and the vehicle based on the predicted running trajectory of several target obstacles and the predicted driving trajectory of the vehicle.

[0038] The warning module 34 is configured to send an alarm signal to the in-vehicle infotainment system when the collision time between the target obstacle and the vehicle is within a preset warning range, so that the in-vehicle infotainment system will issue an alarm; it is also configured to send an alarm signal to the in-vehicle infotainment system when the collision time between the target obstacle located in the transverse lane and the vehicle is within a preset warning range, so that the in-vehicle infotainment system will issue an alarm.

[0039] The active intervention module 35 is configured to send an active intervention signal to the power assist system to enable the power assist system to actively intervene when the collision time between the target obstacle and the vehicle is within a preset active intervention range; it is also configured to send an active intervention signal to the power assist system to enable the power assist system to actively intervene when the collision time between the target obstacle located in the transverse lane and the vehicle is within a preset active intervention range.

[0040] This application also provides a blind spot perception and control device based on multi-view collaboration, the device including a memory and a processor, the memory storing a computer program, which, when executed by the processor, implements the blind spot perception and control method based on multi-view collaboration described above.

[0041] This application also provides a blind spot perception and control system based on multi-view collaboration, such as... Figure 3 As shown, the system includes a surround-view camera group, a rear-view camera group, and the aforementioned blind spot perception and control device based on multi-view collaboration. The surround-view camera group comprises four surround-view cameras 1, which are fixedly installed at the center of the front bumper, the center of the rear bumper, below the left rearview mirror, and below the right rearview mirror, respectively. These cameras are used to acquire first environmental images of the front, rear, left, and right sides of the vehicle, and transmit these first environmental images to the blind spot perception and control device based on multi-view collaboration. The rear-view camera group comprises two rear-view cameras 2, which are fisheye cameras with a shooting angle range of 0–190°. These cameras are fixedly installed at the front left fender and front right fender, respectively. These cameras are used to acquire second environmental images of the left and right rear sides of the vehicle, respectively, and transmit these second environmental images to the blind spot perception and control device based on multi-view collaboration.

[0042] This application also provides a vehicle including the aforementioned multi-view collaborative blind spot perception and control system, wherein the multi-view collaborative blind spot perception and control system is communicatively connected to the vehicle chassis system, in-vehicle infotainment system and power steering system.

[0043] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand this document.

Claims

1. A blind spot perception and control method based on multi-view collaboration, characterized in that: Includes the following steps: The vehicle speed is received in real time from the vehicle chassis system. When the vehicle speed is lower than the preset vehicle speed threshold, the first environmental image information located in front of, behind, to the left and to the right of the vehicle is received in real time from the surround view camera group. Based on the first environmental image information located in front of, behind, to the left and to the right of the vehicle, the current lane information is calculated using deep learning and SLAM algorithms. The vehicle chassis system receives steering signals and steering wheel angles in real time, and sends an activation signal to the oblique rearview camera group according to the steering signals, so that the oblique rearview camera (2) on the corresponding side of the oblique rearview camera group is activated. Real-time reception of second environmental image information located diagonally behind the vehicle from the diagonal rearview camera (2) on the corresponding side of the diagonal rearview camera group; Based on the steering wheel angle and vehicle speed, and using a vehicle dynamics model, the vehicle's trajectory and the target lane information are predicted. Based on the steering wheel angle and the vehicle dynamics model, the angle between the vehicle and the target lane is calculated. Based on the angle between the vehicle and the target lane and the steering wheel angle, the user's region of interest in the second environmental image information is adjusted, and the image information within the user's region of interest is extracted based on the image recognition algorithm. Based on the first environmental image information located in front of, behind, to the left and right of the vehicle and the image information in the user's area of ​​interest, a top-down view of the vehicle's surrounding environment is constructed based on the BEV algorithm and the multi-view perception fusion algorithm, and the current lane information and the target lane information to be reached are marked in this top-down view. Based on a top-down view of the vehicle's surroundings, and using a deep learning algorithm, target obstacle information is determined. This information includes the number of target obstacles, the semantic type of each obstacle, the location of each obstacle, and the outline of each obstacle. Predict the trajectory of several target obstacles based on the target obstacle information in consecutive image frames; calculate the collision time between the target obstacle and the vehicle based on the predicted trajectory of the target obstacle and the predicted trajectory of the vehicle. When the collision time between the target obstacle and the vehicle is within the preset warning range, an alarm signal is sent to the in-vehicle infotainment system, causing the in-vehicle infotainment system to issue an alarm. When the collision time between the target obstacle and the vehicle is within the preset active intervention range, an active intervention signal is sent to the power assist system to enable the power assist system to actively intervene.

2. The blind spot perception and control method based on multi-view collaboration according to claim 1, characterized in that: It also includes the following steps: Based on the first environmental image information located in front of the vehicle, the drivable area of ​​the road ahead is extracted using a deep learning algorithm; When the drivable area ahead is T-shaped, the target obstacle information in the transverse lane is identified based on the first environmental image information in front of the vehicle and a deep learning algorithm. The collision time between the target obstacle in the transverse lane and the vehicle is calculated based on the target obstacle information in the transverse lane in consecutive image frames and the vehicle's driving trajectory. When the collision time between the vehicle and a target obstacle located in the transverse lane is within the preset warning range, an alarm signal is sent to the in-vehicle infotainment system, causing the in-vehicle infotainment system to issue an alarm. When the collision time between the target obstacle located in the transverse lane and the vehicle is within the preset active intervention range, an active intervention signal is sent to the power assist system to enable the power assist system to actively intervene.

3. A blind spot perception and control method based on multi-view collaboration according to claim 1 or 2, characterized in that: It also includes the following steps: Based on the target obstacle information in consecutive image frames, determine the actual running trajectory of several target obstacles; When the actual trajectory of a target obstacle has a fixed spatial relationship with the actual trajectory of the vehicle, the target obstacle is marked as a static obstacle; When the actual trajectory of a target obstacle is independent of the actual trajectory of the vehicle, the target obstacle is marked as a dynamic obstacle; Based on the spatial occupancy characteristics of the static obstacle outline, a buffer zone is extended outward from the edge of the static obstacle outline to form a first safe zone about the static obstacle, wherein the width of the buffer zone is determined according to the height of the static obstacle; the distance between the first safe zones of adjacent static obstacles is calculated; when the distance is less than a preset distance threshold, the area between the adjacent first safe zones about the static obstacle is set as a second safe zone, wherein the preset distance threshold is determined according to the width of the vehicle; the continuous area after removing the first and second safe zones in the top view of the vehicle's surrounding environment constitutes the first driving safe zone, and its boundary is smoothed. Based on the actual trajectory of the dynamic obstacle, the possible location distribution of the dynamic obstacle at multiple future moments is predicted, forming a risk cone region that spreads over time. The cone angle of the risk cone region is determined according to the semantic type of the dynamic obstacle. Based on the braking capacity range of the vehicle and the risk cone region, the area where the two do not intersect is set as the second driving safety zone, and its boundary is smoothed. The first safe driving zone and the second safe driving zone are spatially superimposed; when the first safe driving zone and the second safe driving zone have an intersection, the intersection is marked as an absolutely safe driving zone in the top view of the vehicle's surrounding environment, the edge of the absolutely safe driving zone is optimized for transition, and the marking information is sent to the in-vehicle infotainment system. When the first safe driving zone and the second safe driving zone do not overlap, the second safe driving zone is marked in the top view of the vehicle's surroundings, and the marking information and alarm signal are sent to the in-vehicle infotainment system.

4. The blind spot perception and control method based on multi-view collaboration according to claim 1, characterized in that: The method for adjusting the user's area of ​​interest is as follows: The size of the area of ​​interest for the user is determined based on the steering wheel angle. Based on the angle between the vehicle and the target lane, determine the location of the user's region of interest, ensuring that the region of interest is located in the center of the area within the angle range between the vehicle and the target lane.

5. The blind spot perception and control method based on multi-view collaboration according to claim 1, characterized in that: The method for constructing a top-down view of the vehicle's surrounding environment is as follows: The first environmental image information located in front of, behind, to the left and to the right of the vehicle is converted into first environmental image information from a bird's-eye view in the coordinate system of the surround-view camera group based on the BEV algorithm. Based on the relative positions of the surround view camera group coordinate system and the vehicle coordinate system, the first environmental image information of the bird's-eye view under the coordinate system of the surround view camera group located in front of, behind, to the left and to the right of the vehicle is converted into the first environmental image information of the bird's-eye view under the vehicle coordinate system. The image information within the user's region of interest is converted into image information within the user's region of interest from a bird's-eye view in the coordinate system of the oblique rear-view camera group based on the BEV algorithm; Based on the relative positions of the oblique rearview camera group coordinate system and the vehicle coordinate system, the image information of the user's region of interest from the bird's-eye view in the oblique rearview camera group coordinate system is converted into the image information of the user's region of interest from the bird's-eye view in the vehicle coordinate system. Image calibration and image stitching are performed on the first environmental image information from the bird's-eye view in the vehicle coordinate system located in front of, behind, to the left and right of the vehicle, and the image information of the user's region of interest in the bird's-eye view in the vehicle coordinate system to form a top view of the vehicle's surrounding environment in the vehicle coordinate system.

6. The blind spot perception and control method based on multi-view collaboration according to claim 1, characterized in that: The alarm methods of the in-vehicle infotainment system include highlighting the risk area of ​​the target lane on the screen, using dynamic arrow prompts on the screen to adjust the steering angle or delay steering, sound warnings, and voice reminders; The active intervention methods of the power steering system include correcting the steering wheel, automatically applying light braking to reduce speed, and limiting the steering angle.

7. A blind spot perception and control device based on multi-view collaboration, characterized in that: The device includes: The first processing module (31) is configured to receive vehicle speed in real time from the vehicle chassis system. When the vehicle speed is lower than the preset vehicle speed threshold, it receives first environmental image information located in front of, behind, to the left and to the right of the vehicle from the surround view camera group in real time. Based on the first environmental image information located in front of, behind, to the left and to the right of the vehicle, it calculates the current lane information based on deep learning algorithm and SLAM algorithm. The second processing module (32) is configured to receive steering signals and steering wheel angles from the vehicle chassis system in real time, and send an activation signal to the oblique rearview camera group according to the steering signal, so that the oblique rearview camera (2) on the corresponding side of the oblique rearview camera group is activated; receive second environmental image information located obliquely behind the vehicle in real time from the oblique rearview camera (2) on the corresponding side of the oblique rearview camera group; predict the driving trajectory of the vehicle and the target lane information based on the vehicle dynamics model according to the steering wheel angle and vehicle speed; calculate the angle between the vehicle and the target lane according to the steering wheel angle and the vehicle dynamics model, and adjust the user interest region in the second environmental image information according to the angle between the vehicle and the target lane and the steering wheel angle, and extract the image information in the user interest region based on the image recognition algorithm; The third processing module (33) is configured to construct a top-down view of the vehicle's surrounding environment based on the first environmental image information located in front of, behind, to the left and right of the vehicle and the image information in the user's area of ​​interest, using the BEV algorithm and multi-view perception fusion algorithm, and to mark the current lane information and the target lane information to be reached in this top-down view; to determine the target obstacle information based on the top-down view of the vehicle's surrounding environment using a deep learning algorithm; to predict the running trajectory of several target obstacles based on the target obstacle information in consecutive image frames; and to calculate the collision time between the target obstacle and the vehicle based on the predicted running trajectory of several target obstacles and the predicted driving trajectory of the vehicle. The warning module (34) is configured to send an alarm signal to the in-vehicle infotainment system when the collision time between the target obstacle and the vehicle is within the preset warning range, so that the in-vehicle infotainment system can issue an alarm. The active intervention module (35) is configured to send an active intervention signal to the power assist system when the collision time between the target obstacle and the vehicle is within the preset active intervention range, so that the power assist system can actively intervene.

8. A blind spot sensing and control device based on multi-view collaboration, characterized in that: The device includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements a blind spot perception and control method based on multi-view collaboration as described in any one of claims 1 to 6.

9. A blind spot perception and control system based on multi-view collaboration, characterized in that: It includes a surround-view camera group, a rear-view camera group, and the blind spot perception and control device based on multi-view collaboration as described in claim 8, wherein, The surround view camera group includes four surround view cameras (1), which are fixedly installed at the middle position of the front bumper, the middle position of the rear bumper, the lower position of the left rearview mirror, and the lower position of the right rearview mirror of the vehicle, respectively, to collect the first environmental images of the front, rear, left and right of the vehicle, and respectively transmit the first environmental images to the blind spot perception and control device based on multi-view collaboration. The oblique rearview camera group includes two oblique rearview cameras (2). The oblique rearview cameras (2) are fisheye cameras, which are fixedly installed on the front left fender and the front right fender of the vehicle, respectively. They are used to collect second environmental images of the left and right rear sides of the vehicle, respectively, and transmit the second environmental images to the blind spot perception and control device based on multi-view collaboration.

10. A vehicle, characterized in that: Includes the blind spot perception and control system based on multi-view collaboration as described in claim 9.

Images