Blind area sensing and control method, device, equipment, system and vehicle based on multi-view collaboration

Abstract

The invention discloses a blind area sensing and control method, device, equipment and system based on multi-view collaboration and a vehicle. The method comprises the following steps: receiving first environment image information from an all-round camera group; a steering signal, a steering wheel angle and a vehicle speed are received from a vehicle chassis system, a starting signal is sent to the oblique rearview camera on the corresponding side according to the steering signal, and second environment image information of the oblique rearview camera is received; adjusting a user interested area in the second environment image information according to the steering wheel angle and the vehicle speed; constructing a top view of the surrounding environment of the vehicle according to the first environment image information and the image information in the region of interest of the user, and determining target obstacle information according to the top view; calculating the collision time of the target obstacle and the vehicle according to the target obstacle information and the driving track of the vehicle; when the collision time is within a preset early warning range, an alarm signal is sent; and when the collision time is within a preset active intervention range, sending an active intervention signal. And the safety in a complex and changeable road scene can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving, in particular to a blind area perception and control method, device, equipment, system and vehicle based on multi-view cooperation. BACKGROUND

[0002] With the rapid development of active safety technology and automatic driving system of automobile, the environmental perception capability of vehicle has a crucial influence on the auxiliary driving and automatic driving function. However, in the complex and changeable road scene, the existing environmental perception system of vehicle still has many technical bottlenecks, which leads to the insufficient decision reliability of the auxiliary driving system of vehicle in the complex and changeable road scene. For example, when the existing vehicle makes a large angle turn, the front-view camera cannot effectively capture the dynamic information of the side rear blind area due to the limited view angle, and needs to rely on the rearview mirror or short-range millimeter wave radar, but the resolution of the rearview mirror or short-range millimeter wave radar is insufficient, which is easy to miss the weak road users, resulting in the failure of the auxiliary driving system in the complex turning scene, and even may cause the risk of side collision; the existing vehicle at the T-shaped intersection with the view obstruction such as green belt or building obstruction, the millimeter wave radar is difficult to distinguish the static obstacle and the transverse motion vehicle due to the limited angle resolution, and cannot accurately predict the motion trajectory of the oncoming vehicle, resulting in the increase of the misjudgment rate of the auxiliary driving system; the existing vehicle needs to perceive the asymmetric road structure such as the gap position of the central separation belt and the curvature change of the opposite lane in real time when making U-turn on the asymmetric road topology, but the existing auxiliary driving system relies on the pre-stored information of high-precision map for decision, which is difficult to cope with the dynamic changes of road conditions such as temporary construction fence, resulting in the delay of trajectory planning of vehicle when making U-turn without protection or the lack of information of corresponding side oblique rear when the driver drives actively, which increases the driving risk. SUMMARY

[0003] To solve at least one aspect of the above problem, the present application provides a blind area perception and control method, device, equipment, system and vehicle based on multi-view cooperation.

[0004] In a first aspect, the application provides a blind area perception and control method based on multi-view cooperation, comprising the following steps: receiving vehicle speed from the vehicle chassis system in real time; when the vehicle speed is lower than a preset vehicle speed threshold, receiving first environmental image information located in front, rear, left and right of the vehicle from the surround-view camera group in real time; calculating current lane information based on a deep learning algorithm and a SLAM algorithm according to the first environmental image information located in front, rear, left and right of the vehicle; receiving a steering signal and a steering wheel angle from the vehicle chassis system in real time, and sending an opening signal to the rear-sloping-view camera group according to the steering signal, so that the corresponding rear-sloping-view camera of the rear-sloping-view camera group is turned on; receiving second environmental image information located in the rear-sloping direction of the vehicle from the corresponding rear-sloping-view camera of the rear-sloping-view camera group in real time; predicting the driving trajectory of the vehicle and the target lane information to be reached based on a vehicle dynamics model according to the steering wheel angle and the vehicle speed; calculating the included angle between the vehicle and the target lane based on the vehicle dynamics model according to the steering wheel angle, and adjusting the user's region of interest in the second environmental image information according to the included angle between the vehicle and the target lane and the steering wheel angle, and extracting the image information in the user's region of interest based on an image recognition algorithm; constructing a top view of the vehicle's surrounding environment based on a BEV (bird's eye view) algorithm and a multi-view perception fusion algorithm according to the first environmental image information located in front, rear, left and right of the vehicle and the image information in the user's region of interest, and marking the current lane information and the target lane information to be reached in the top view; determining target obstacle information based on a deep learning algorithm according to the top view of the vehicle's surrounding environment, the target obstacle information including the number of target obstacles, the semantic type of each target obstacle, the position of each target obstacle, and the contour of each target obstacle; predicting the running trajectories of a plurality of target obstacles according to the target obstacle information in the continuous image frames; calculating the collision time of the target obstacles and the vehicle according to the predicted running trajectories of the target obstacles and the predicted driving trajectory of the vehicle; when the collision time of the target obstacles and the vehicle is within a preset warning range, sending an alarm signal to the vehicle information entertainment system to make the vehicle information entertainment system alarm; when the collision time of the target obstacles and the vehicle is within a preset active intervention range, sending an active intervention signal to the power-assisted system to make the power-assisted system actively intervene.

[0005] Preferably, the method further comprises the following steps: extracting a drivable area existing in the front road according to the first environmental image information located in front of the vehicle based on a deep learning algorithm; when the shape of the drivable area existing in the front road is T-shaped, identifying target obstacle information located in the lateral lane according to the first environmental image information located in front of the vehicle based on the deep learning algorithm; calculating a collision time of the target obstacle located in the lateral lane with the vehicle according to the target obstacle information located in the lateral lane in the continuous image frames and the driving track of the vehicle; when the collision time of the target obstacle located in the lateral lane with the vehicle is located in a preset warning range, sending a warning signal to the vehicle information entertainment system to make the vehicle information entertainment system give a warning; and when the collision time of the target obstacle located in the lateral lane with the vehicle is located in a preset active intervention range, sending an active intervention signal to the power-assisted system to make the power-assisted system give an active intervention.

[0006] Preferably, the method further comprises the following steps: determining actual running trajectories of the target obstacles according to the target obstacle information in the continuous image frames; marking the target obstacle as a static obstacle when the actual running trajectory of the target obstacle presents a fixed spatial relationship with respect to the actual running trajectory of the host vehicle; marking the target obstacle as a dynamic obstacle when the actual running trajectory of the target obstacle presents an independent motion relationship with respect to the actual running trajectory of the host vehicle; expanding a buffer zone outward at the edge of the static obstacle contour to form a first safety region with respect to the static obstacle according to the spatial occupancy characteristics of the static obstacle contour, wherein the width of the buffer zone is determined according to the height of the static obstacle; calculating the spacing of the first safety regions of adjacent static obstacles; setting the area between the first safety regions with respect to the static obstacles as a second safety region when the spacing is less than a preset spacing threshold, wherein the preset spacing threshold is determined according to the width of the host vehicle; constructing a first driving safety region from the continuous area after the first safety regions and the second safety regions are removed in the overhead view of the vehicle surrounding environment, and smoothing the boundaries thereof; predicting the position distribution of the dynamic obstacle at future time instants according to the actual running trajectory of the dynamic obstacle to form a risk cone-shaped region that diffuses over time, wherein the cone angle of the risk cone-shaped region is determined according to the semantic type of the dynamic obstacle; setting a region with no intersection between the braking capability range of the host vehicle and the risk cone-shaped region as a second driving safety region, and smoothing the boundaries thereof; spatially superimposing the first driving safety region and the second driving safety region; marking the intersection region as an absolute safe driving region in the overhead view of the vehicle surrounding environment when the first driving safety region and the second driving safety region have an intersection region, smoothing the edges of the absolute safe driving region, and sending the marking information to the vehicle information entertainment system; and marking the second driving safety region in the overhead view of the vehicle surrounding environment when the first driving safety region and the second driving safety region have no intersection region, sending the marking information and an alarm signal to the vehicle information entertainment system.

[0007] Preferably, the adjustment method of the user interest region is: determining the size of the user interest region according to the size of the steering wheel turning angle; and determining the position of the user interest region according to the included angle between the host vehicle and the target lane, so that the user interest region is located at the center region within the included angle range of the host vehicle and the target lane.

[0008] Preferably, the method for constructing the overhead view of the vehicle surrounding environment is: the first environment image information located in front, rear, left and right of the vehicle is respectively converted into the first environment image information of the bird's eye view under the surround camera group coordinate system based on the BEV algorithm; according to the relative position of the surround camera group coordinate system and the vehicle coordinate system, the first environment image information of the bird's eye view under the surround camera group coordinate system located in front, rear, left and right of the vehicle is respectively converted into the first environment image information of the bird's eye view under the vehicle coordinate system; the image information in the user interested region is converted into the image information in the user interested region of the bird's eye view under the rearview camera group coordinate system based on the BEV algorithm; according to the relative position of the rearview camera group coordinate system and the vehicle coordinate system, the image information in the user interested region of the bird's eye view under the rearview camera group coordinate system is converted into the image information in the user interested region of the bird's eye view under the vehicle coordinate system; the first environment image information of the bird's eye view under the vehicle coordinate system located in front, rear, left and right of the vehicle and the image information in the user interested region of the bird's eye view under the vehicle coordinate system are image calibrated and spliced to form the overhead view of the vehicle surrounding environment under the vehicle coordinate system.

[0009] Preferably, the alarm mode of the in-vehicle information entertainment system includes highlighting the risk area of the target lane on the screen, using a dynamic arrow on the screen to prompt the adjustment of the steering angle or delayed steering, sound warning, voice reminder.

[0010] Preferably, the active intervention mode of the power assistance system includes correcting the steering wheel, automatically lightly braking to slow down, and limiting the steering angle.

[0011] In the second aspect, the present application provides a blind spot perception and control device based on multi-perspective collaboration, the device comprising: a first processing module, configured to receive the vehicle speed in real time from the vehicle chassis system, and when the vehicle speed is lower than a preset speed threshold, receive the first environmental image information in front of, behind, to the left and to the right of the vehicle from the surround view camera group in real time; calculate the current lane information based on the deep learning algorithm and the SLAM algorithm according to the first environmental image information in front of, behind, to the left and to the right of the vehicle; a second processing module, configured to receive the steering information in real time from the vehicle chassis system signal and steering wheel angle, and sends a start signal to the oblique rearview camera group according to the turn signal, so that the oblique rearview camera on the corresponding side in the oblique rearview camera group is turned on; receives the second environment image information located obliquely behind the vehicle from the oblique rearview camera on the corresponding side in the oblique rearview camera group in real time; predicts the driving trajectory of the vehicle and the target lane to be reached based on the vehicle dynamics model according to the steering wheel angle and vehicle speed; calculates the angle between the vehicle and the target lane based on the vehicle dynamics model according to the steering wheel angle, and adjusts the second environment image information based on the angle between the vehicle and the target lane and the steering wheel angle. a user's area of ​​interest in the environmental image information, and extracting image information within the user's area of ​​interest based on an image recognition algorithm; a third processing module configured to construct a bird's-eye view of the vehicle's surrounding environment based on the first environmental image information located in front, rear, left, and right of the vehicle and the image information within the user's area of ​​interest, based on the BEV algorithm and the multi-view perception fusion algorithm, and annotate the current lane information and the target lane information to be reached in the bird's-eye view; determine target obstacle information based on the bird's-eye view of the vehicle's surrounding environment based on a deep learning algorithm; predict the running trajectories of multiple target obstacles based on the target obstacle information in consecutive image frames; and calculate the collision time between the target obstacles and the vehicle based on the predicted running trajectories of the multiple target obstacles and the predicted driving trajectory of the vehicle; a warning module configured to send an alarm signal to the in-vehicle infotainment system when the collision time between the target obstacle and the vehicle is within a preset warning range, causing the in-vehicle infotainment system to issue an alarm; and an active intervention module configured to send an active intervention signal to the power assist system when the collision time between the target obstacle and the vehicle is within a preset active intervention range, causing the power assist system to actively intervene.

[0012] In the third aspect, the present application provides a blind spot perception and control device based on multi-perspective collaboration, the device including a memory and a processor, the memory storing a computer program, and when the computer program is executed by the processor, implementing any of the above-mentioned blind spot perception and control methods based on multi-perspective collaboration.

[0013] In the fourth aspect, the present application provides a blind spot perception and control system based on multi-perspective collaboration, including a surround-view camera group, an oblique rearview camera group and the above-mentioned blind spot perception and control device based on multi-perspective collaboration, wherein the surround-view camera group includes four surround-view cameras, which are respectively fixedly installed at the middle position of the front bumper, the middle position of the rear bumper, the bottom of the left rearview mirror, and the bottom of the right rearview mirror of the vehicle, for respectively collecting first environmental images in front, rear, left and right of the vehicle, and transmitting the first environmental images to the blind spot perception and control device based on multi-perspective collaboration; the oblique rearview camera group includes two oblique rearview cameras, which are fisheye cameras, respectively fixedly installed at the front left fender and the front right fender of the vehicle, for respectively collecting second environmental images at the oblique rear of the left side of the vehicle and the oblique rear of the right side of the vehicle, and transmitting the second environmental images to the blind spot perception and control device based on multi-perspective collaboration.

[0014] In a fifth aspect, the present application provides a vehicle, including the above-mentioned blind spot perception and control system based on multi-perspective collaboration.

[0015] The present invention provides a method, device, equipment, system, and vehicle for blind spot perception and control based on multi-view collaboration, which has the following beneficial effects:

[0016] (1) The surround-view camera group is used to perceive the vehicle's surrounding environment and detect its position, thereby realizing lane positioning of the vehicle in low-speed scenarios; the left and right rear view angles of the fisheye oblique camera located on both sides of the vehicle are used to detect the oblique rear view range of the vehicle, and the user's area of ​​interest of the oblique rear view is dynamically adjusted according to the steering wheel angle and vehicle speed, so that it not only monitors the rear of the current lane, but also covers the target lane after turning, and can effectively capture the dynamic information of the side and rear blind spots. The image information detected by the surround-view camera group and the image information detected by the oblique rear view camera group are then integrated to realize collaborative perception and positioning of the rear view, providing the driver with a more reliable basis for steering decision-making, which helps to improve safety in scenarios such as large-angle steering and U-turns.

[0017] (2) When driving at a low speed to a T-junction, the surround-view camera located in front of the vehicle is used to perceive obstacles on the lateral lanes of the T-junction, thereby implementing front blind spot assistance behavior, which helps improve the safety of the vehicle in scenarios such as complex intersections.

[0018] (3) By distinguishing the target obstacles into static obstacles and dynamic obstacles, determining the first driving safety area and the second driving safety area according to the static obstacles and the dynamic obstacles respectively, then determining the absolute safety driving area or the safety driving area that can avoid the dynamic obstacles according to the first driving safety area and the second driving safety area, and displaying in the vehicle information entertainment system according to the type of the safety driving area, the auxiliary driving capability is further improved, which is helpful to further improve the driving safety of the vehicle in the scenes of urban roads, complex intersections, unprotected left turns and the like; and when determining the first driving safety area, the safety area related to the static obstacles is removed, while when determining the second driving safety area, the risk cone area of the dynamic obstacle is predicted according to the dynamic obstacle, and then the determination is made according to the braking ability range of the vehicle and the risk cone area, which is helpful to reduce the computing power under the condition of ensuring the accuracy of the safety driving area determination. BRIEF DESCRIPTION OF DRAWINGS

[0019] For better understanding of the above and other objects, features, advantages and functions of the present application, reference can be made to the embodiments shown in the drawings. The same reference signs in the drawings refer to the same components. It should be understood by those skilled in the art that the drawings are intended to illustrate the preferred embodiments of the present application schematically, and have no limiting effect on the scope of the present application, and the components in the drawings are not drawn to scale.

[0020] Figure 1 A flow chart of a blind area perception and control method based on multi-view cooperation according to an embodiment of the present application is shown;

[0021] Figure 2 A block diagram of a blind area perception and control device based on multi-view cooperation according to an embodiment of the present application is shown;

[0022] Figure 3 A distribution schematic diagram of a camera of a blind area perception and control method system based on multi-view cooperation according to an embodiment of the present application is shown.

[0023] Explanation of reference signs:

[0024] 1, surround view camera; 2, rear view camera; 31, first processing module; 32, second processing module; 33, third processing module; 34, early warning module; 35, active intervention module. DETAILED DESCRIPTION

[0025] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0026] As used herein, the term "including" and its variations represent open inclusion, i.e., "including but not limited to." Unless otherwise stated, the term "or" means "and / or." The term "based on" means "based at least in part on." The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment." The term "another embodiment" means "at least one additional embodiment." The terms "first," "second," etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0027] In order to at least partially solve one or more of the above problems and other potential problems, the embodiments of the present disclosure propose a blind spot perception and control method based on multi-view collaboration, such as Figure 1 As shown, the following steps are included:

[0028] The vehicle speed is received in real time from the vehicle chassis system. When the vehicle speed is below a preset speed threshold (for example, 15 km / h), i.e., when the vehicle is at a low speed, the system receives real-time first-stage environmental image information from the front, rear, left, and right sides of the vehicle from the surround-view camera group. Based on this first-stage environmental image information, the system calculates the current lane information using a deep learning algorithm and a SLAM (Simultaneous Mapping and Localization) algorithm. Specifically, based on the first-stage environmental image information, the system detects lane lines around the vehicle and extracts lane line features using a deep learning algorithm. The SLAM algorithm then calculates the current lane information by associating lane line feature points from historical frames. The lane information includes lane line shape, lane line type, and the vehicle's angle and position relative to the lane line.

[0029] The steering signal and the steering wheel angle are received in real time from the vehicle chassis system, and an opening signal is sent to the rear-view camera group according to the steering signal, so that the corresponding side rear-view camera 2 in the rear-view camera group is opened, for example, when the steering signal is to the left, the rear-view camera 2 located on the left side of the vehicle is opened; the second environment image information located in the rear of the vehicle is received in real time from the corresponding side rear-view camera 2 in the rear-view camera group, and since the rear-view camera 2 in the application adopts a fisheye type oblique camera, the angle of view range is 0-190°. According to the steering wheel angle and the vehicle speed, the driving track and the target lane information to be reached of the vehicle are predicted based on the vehicle dynamics model; according to the steering wheel angle, the angle between the vehicle and the target lane is calculated based on the vehicle dynamics model, and according to the angle between the vehicle and the target lane and the steering wheel angle, the user interested region in the second environment image information is adjusted, and the image information in the user interested region is extracted based on the image recognition algorithm, specifically, the adjustment method of the user interested region is: according to the size of the steering wheel angle, the size of the user interested region is determined, the larger the steering wheel angle, the larger the user interested region; according to the angle between the vehicle and the target lane, the user interested region position is determined, so that the user interested region is located at the center region in the angle range between the vehicle and the target lane, so that it not only monitors the rear of the current lane, but also covers the target lane after turning.

[0030] According to the first environment image information located in front, rear, left and right of the vehicle and the image information in the user interested region, a top view of the vehicle surrounding environment is constructed based on the BEV algorithm and the multi-view perception fusion algorithm, and the current lane information and the target lane information to be reached are marked in the top view. Specifically, the method for constructing the top view of the vehicle surrounding environment is: the first environment image information located in front, rear, left and right of the vehicle is converted into the bird's eye view first environment image information under the surround-view camera group coordinate system based on the BEV algorithm; according to the relative position of the surround-view camera group coordinate system and the vehicle coordinate system, the bird's eye view first environment image information under the surround-view camera group coordinate system located in front, rear, left and right of the vehicle is correspondingly converted into the bird's eye view first environment image information under the vehicle coordinate system; the image information in the user interested region is converted into the bird's eye view image information in the user interested region under the rear-view camera group coordinate system based on the BEV algorithm; according to the relative position of the rear-view camera group coordinate system and the vehicle coordinate system, the bird's eye view image information in the user interested region under the rear-view camera group coordinate system is converted into the bird's eye view image information in the user interested region under the vehicle coordinate system; the bird's eye view first environment image information under the vehicle coordinate system located in front, rear, left and right of the vehicle and the bird's eye view image information in the user interested region under the vehicle coordinate system are image calibrated and image spliced to form the top view of the vehicle surrounding environment under the vehicle coordinate system.

[0031] According to the overhead view of the vehicle surrounding environment, target obstacle information is determined based on a deep learning algorithm, the target obstacle information including the number of target obstacles, the semantic type of each target obstacle, the position of each target obstacle, and the contour of each target obstacle. Running trajectories of the target obstacles are predicted according to the target obstacle information in consecutive image frames; and the collision time of each target obstacle with the host vehicle is calculated according to the predicted running trajectory of the target obstacle and the predicted running trajectory of the host vehicle.

[0032] When the collision time of the target obstacle with the host vehicle is within a preset warning range, i.e., the target obstacle has a collision risk with the host vehicle, for example, the preset warning range is set to be less than 10s, an alarm signal is sent to the vehicle information entertainment system, so that the vehicle information entertainment system alarms, specifically, the alarm mode of the vehicle information entertainment system includes highlighting the risk area of the target lane on the screen, using a dynamic arrow to prompt the adjustment of the steering angle or the delay of the steering on the screen, sound warning, and voice reminder. When the collision time of the target obstacle with the host vehicle is within a preset active intervention range, i.e., the target obstacle has a high collision risk with the host vehicle, for example, the preset active intervention range is set to be less than 3s, an active intervention signal is sent to the power-assisted system, so that the power-assisted system actively intervenes, and the active intervention mode of the power-assisted system includes correcting the steering wheel, automatically applying light braking to slow down, and limiting the steering angle.

[0033] In the preferred embodiment, the method further comprises the following steps: according to the first environmental image information located in front of the host vehicle, target obstacle information located in the lateral lane is identified based on a deep learning algorithm when the shape of the drivable area existing in the front road is T-shaped; the collision time of the target obstacle located in the lateral lane with the host vehicle is calculated according to the target obstacle information located in the lateral lane in the consecutive image frames and the running trajectory of the host vehicle; when the collision time of the target obstacle located in the lateral lane with the host vehicle is within a preset warning range, an alarm signal is sent to the vehicle information entertainment system, so that the vehicle information entertainment system alarms; and when the collision time of the target obstacle located in the lateral lane with the host vehicle is within a preset active intervention range, an active intervention signal is sent to the power-assisted system, so that the power-assisted system actively intervenes.

[0034] In a preferred embodiment, the method of the present application further includes the following steps: determining the actual running trajectories of several target obstacles based on target obstacle information in continuous image frames; marking the target obstacle as a static obstacle when the actual running trajectory of the target obstacle presents a fixed spatial relationship with the actual running trajectory of the vehicle; marking the target obstacle as a dynamic obstacle when the actual running trajectory of the target obstacle presents an independent motion relationship with the actual running trajectory of the vehicle; expanding a buffer zone outward at the edge of the static obstacle outline based on the spatial occupation characteristics of the static obstacle outline to form a first safety zone for the static obstacle, wherein the width of the buffer zone is determined according to the height of the static obstacle. Specifically, the higher the height of the static obstacle, the wider the buffer zone; calculating the spacing between the first safety zones of adjacent static obstacles; when the spacing is less than a preset spacing threshold, setting the area between the adjacent first safety zones for the static obstacle as a second safety zone, wherein the preset spacing threshold is determined according to the width of the vehicle; excluding the first safety zone and the second safety zone in the top view of the vehicle's surrounding environment, the continuous area constitutes the first driving safety zone, and smoothing its boundaries to avoid path jitter; predicting the dynamic obstacle's future location based on the actual running trajectory of the dynamic obstacle The possible position distribution at multiple moments forms a risk cone area that spreads over time, wherein the cone angle of the risk cone area is determined according to the semantic type of the dynamic obstacle. Specifically, different cone angles are used for vehicles, pedestrians, and animals. The corresponding cone angle can be set in advance according to the semantic type of the dynamic obstacle; according to the braking capability range of the vehicle and the risk cone area, the non-intersection area between the two is determined to be set as the second driving safety area, and its boundary is smoothed; the first driving safety area and the second driving safety area are spatially overlapped; when there is an intersection area between the first driving safety area and the second driving safety area, The intersection area is marked as an absolutely safe driving area in the overhead view of the vehicle's surrounding environment, and the edge of the absolutely safe driving area is transition-optimized, and the marking information is sent to the in-vehicle infotainment system. For example, the absolutely safe driving area can be marked green in the in-vehicle infotainment system; when there is no intersection area between the first driving safety area and the second driving safety area, the second driving safety area is marked in the overhead view of the vehicle's surrounding environment, and the marking information and an alarm signal are sent to the in-vehicle infotainment system. For example, the second driving safety area can be marked yellow in the in-vehicle infotainment system, accompanied by a voice prompt suggesting braking risks.

[0035] This application also provides a blind spot perception and control device based on multi-view collaboration, such as Figure 2As shown, the device comprises: a first processing module 31 configured to receive vehicle speed from the vehicle chassis system in real time, receive first environmental image information located in front, rear, left and right of the vehicle from the surround-view camera group in real time when the vehicle speed is lower than the preset vehicle speed threshold, and calculate the current lane information based on the deep learning algorithm and the SLAM algorithm according to the first environmental image information located in front, rear, left and right of the vehicle. In the preferred embodiment, the first processing module 31 is further configured to extract the drivable area existing in the front road based on the deep learning algorithm according to the first environmental image information located in front of the vehicle; when the shape of the drivable area existing in the front road is T-shaped, the target obstacle information located in the lateral lane is recognized based on the deep learning algorithm according to the first environmental image information located in front of the vehicle; and the collision time between the target obstacle located in the lateral lane and the vehicle is calculated according to the target obstacle information located in the lateral lane in the continuous image frames and the driving trajectory of the vehicle.

[0036] A second processing module 32 is configured to receive the steering signal and the steering wheel angle from the vehicle chassis system in real time, send an opening signal to the rear-sloping camera group according to the steering signal so that the corresponding rear-sloping camera 2 of the rear-sloping camera group is turned on, receive second environmental image information located in the rear-sloping side of the vehicle from the corresponding rear-sloping camera 2 of the rear-sloping camera group in real time, predict the driving trajectory of the vehicle and the target lane information to be reached based on the vehicle dynamics model according to the steering wheel angle and the vehicle speed, calculate the included angle between the vehicle and the target lane based on the vehicle dynamics model according to the steering wheel angle, adjust the region of interest in the second environmental image information according to the included angle between the vehicle and the target lane and the steering wheel angle, and extract the image information in the region of interest based on the image recognition algorithm.

[0037] A third processing module 33 is configured to construct a bird's-eye view of the vehicle surrounding environment based on the BEV algorithm and the multi-view perception fusion algorithm according to the first environmental image information located in front, rear, left and right of the vehicle and the image information in the region of interest, and mark the current lane information and the target lane information to be reached in the bird's-eye view; determine the target obstacle information based on the deep learning algorithm according to the bird's-eye view of the vehicle surrounding environment; predict the running trajectories of a plurality of target obstacles according to the target obstacle information in the continuous image frames; and calculate the collision time between the target obstacles and the vehicle according to the predicted running trajectories of the plurality of target obstacles and the predicted driving trajectory of the vehicle.

[0038] The early warning module 34 is configured to send an alarm signal to the in-vehicle infotainment system when the collision time between the target obstacle and the vehicle is within a preset early warning range, so that the in-vehicle infotainment system issues an alarm; and is also configured to send an alarm signal to the in-vehicle infotainment system when the collision time between the target obstacle in the lateral lane and the vehicle is within a preset early warning range, so that the in-vehicle infotainment system issues an alarm.

[0039] The active intervention module 35 is configured to send an active intervention signal to the power assist system so that the power assist system performs active intervention when the collision time between the target obstacle and the vehicle is within a preset active intervention range; and is also configured to send an active intervention signal to the power assist system so that the power assist system performs active intervention when the collision time between the target obstacle in the lateral lane and the vehicle is within a preset active intervention range.

[0040] The present application also provides a blind spot perception and control device based on multi-perspective collaboration, the device including a memory and a processor, the memory storing a computer program, and implementing any of the above-mentioned blind spot perception and control methods based on multi-perspective collaboration when the computer program is executed by the processor.

[0041] This application also provides a blind spot perception and control system based on multi-view collaboration, such as Figure 3 As shown, it includes a surround-view camera group, an oblique rearview camera group and the above-mentioned blind spot perception and control device based on multi-perspective collaboration, wherein the surround-view camera group includes four surround-view cameras 1, which are fixedly installed at the middle position of the front bumper, the middle position of the rear bumper, the bottom of the left rearview mirror, and the bottom of the right rearview mirror of the vehicle, respectively, for respectively collecting first environmental images in front, behind, left and right of the vehicle, and respectively transmitting the first environmental images to the blind spot perception and control device based on multi-perspective collaboration; the oblique rearview camera group includes two oblique rearview cameras 2, which adopt fisheye cameras with a shooting angle range of 0 to 190°. They are fixedly installed at the front left fender and the front right fender of the vehicle, respectively, for respectively collecting second environmental images at the oblique rear left and right sides of the vehicle, and respectively transmitting the second environmental images to the blind spot perception and control device based on multi-perspective collaboration.

[0042] The present application also provides a vehicle, including the above-mentioned blind spot perception and control system based on multi-perspective collaboration, and the blind spot perception and control system based on multi-perspective collaboration is communicatively connected to the vehicle chassis system, on-board infotainment system and power steering system.

[0043] Having described various embodiments of the disclosure above, the descriptions are not exhaustive and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art. The selection of terms to be used in the description is intended to best explain the principles of the embodiments, practical application, or technical improvement over the prior art, or to enable other skilled persons in the art to understand the present document.

Claims

1. A blind spot perception and control method based on multi-view collaboration, characterized by: The following steps are involved: receiving a vehicle speed in real time from a vehicle chassis system, and when the vehicle speed is lower than a preset speed threshold, receiving first environmental image information in real time from a surround-view camera group in front of, behind, to the left, and to the right of the vehicle; Based on the first environment image information located in front, behind, left and right of the vehicle, the current lane information is calculated based on the deep learning algorithm and the SLAM algorithm; receiving a turn signal and a steering wheel angle in real time from a vehicle chassis system, and sending a start signal to the oblique rearview camera group according to the turn signal, so that the oblique rearview camera (2) on the corresponding side of the oblique rearview camera group is turned on; receiving in real time second environment image information located obliquely behind the vehicle from the oblique rearview camera (2) on the corresponding side of the oblique rearview camera group; According to the steering wheel angle and vehicle speed, based on the vehicle dynamics model, the vehicle's driving trajectory and the target lane to be reached are predicted; Calculating the angle between the vehicle and the target lane based on the steering wheel angle and the vehicle dynamics model, adjusting the user's region of interest in the second environment image information based on the angle between the vehicle and the target lane and the steering wheel angle, and extracting image information within the user's region of interest based on an image recognition algorithm; Based on the first environmental image information located in front of, behind, to the left and to the right of the vehicle and the image information in the user's area of ​​interest, a bird's-eye view of the vehicle's surrounding environment is constructed based on the BEV algorithm and multi-view perception fusion algorithm, and the current lane information and the target lane information are annotated in this bird's-eye view; Based on a bird's-eye view of the vehicle's surroundings and a deep learning algorithm, target obstacle information is determined. The target obstacle information includes the number of target obstacles, the semantic type of each target obstacle, the location of each target obstacle, and the outline of each target obstacle. Predicting the trajectories of several target obstacles based on target obstacle information in continuous image frames; calculating the collision time between the target obstacles and the vehicle based on the predicted trajectories of the several target obstacles and the predicted driving trajectory of the vehicle; When the collision time between the target obstacle and the vehicle is within the preset warning range, an alarm signal is sent to the in-vehicle infotainment system, causing the in-vehicle infotainment system to sound an alarm; When the collision time between the target obstacle and the vehicle is within the preset active intervention range, an active intervention signal is sent to the power assist system, causing the power assist system to perform active intervention.

2. The blind spot perception and control method based on multi-view collaboration according to claim 1, characterized in that: The following steps are also included: Extracting a drivable area on the road ahead based on a deep learning algorithm based on first environmental image information in front of the vehicle; When the drivable area on the road ahead is T-shaped, the target obstacle in the lateral lane is identified based on the first environmental image information in front of the vehicle using a deep learning algorithm; Calculate the collision time between the target obstacle in the lateral lane and the vehicle based on the target obstacle information in the lateral lane in the continuous image frames and the vehicle's driving trajectory; When the collision time between the target obstacle in the lateral lane and the vehicle is within the preset warning range, an alarm signal is sent to the in-vehicle infotainment system, causing the in-vehicle infotainment system to sound an alarm; When the collision time between the target obstacle in the lateral lane and the vehicle is within the preset active intervention range, an active intervention signal is sent to the power assist system, causing the power assist system to perform active intervention.

3. The blind spot perception and control method based on multi-view collaboration according to claim 1 or 2, characterized in that: The following steps are also included: Determine the actual running trajectories of several target obstacles based on target obstacle information in continuous image frames; When the actual running track of the target obstacle presents a fixed spatial relationship with the actual running track of the vehicle, the target obstacle is marked as a static obstacle; When the actual running trajectory of the target obstacle shows an independent motion relationship with the actual running trajectory of the vehicle, the target obstacle is marked as a dynamic obstacle; Based on the spatial occupancy characteristics of the static obstacle outline, a buffer zone is expanded outward from the edge of the static obstacle outline to form a first safety zone around the static obstacle, where the width of the buffer zone is determined by the height of the static obstacle. The spacing between adjacent first safety zones of the static obstacle is calculated. When the spacing is less than a preset spacing threshold, the area between adjacent first safety zones around the static obstacle is set as a second safety zone, where the preset spacing threshold is determined by the width of the vehicle. In the top view of the vehicle's surroundings, the continuous area after excluding the first and second safety zones constitutes the first driving safety zone, and its boundaries are smoothed. Based on the actual trajectory of a dynamic obstacle, the possible location distribution of the dynamic obstacle at multiple moments in the future is predicted, forming a risk cone area that spreads over time. The cone angle of the risk cone area is determined according to the semantic type of the dynamic obstacle. Based on the braking capability range of the vehicle and the risk cone area, the area where the two do not intersect is set as the second driving safety zone, and its boundary is smoothed. spatially overlapping the first driving safety zone and the second driving safety zone; when there is an intersection between the first driving safety zone and the second driving safety zone, marking the intersection as an absolutely safe driving zone in a top view of the vehicle's surroundings, performing transition optimization on the edge of the absolutely safe driving zone, and transmitting the marking information to the in-vehicle infotainment system; When there is no intersection between the first driving safety area and the second driving safety area, the second driving safety area is marked in the top view of the vehicle surroundings, and the marking information and an alarm signal are sent to the in-vehicle infotainment system.

4. The blind spot perception and control method based on multi-view collaboration according to claim 1, characterized in that: The method for adjusting the user's area of ​​interest is: Determine the size of the user's area of ​​interest based on the size of the steering wheel angle; The position of the user's area of ​​interest is determined according to the angle between the vehicle and the target lane, so that the user's area of ​​interest is located in the center area within the angle range between the vehicle and the target lane.

5. The blind spot perception and control method based on multi-view collaboration according to claim 1, characterized in that: The method for constructing a top view of the vehicle's surrounding environment is: The first environment image information located in front of, behind, to the left and to the right of the vehicle is converted into the first environment image information from a bird's-eye view in the coordinate system of the surround view camera group based on the BEV algorithm; According to the relative positions of the surround-view camera group coordinate system and the vehicle coordinate system, the first environment image information of the bird's-eye view in the surround-view camera group coordinate system located in front of, behind, to the left of, and to the right of the vehicle is converted into the first environment image information of the bird's-eye view in the vehicle coordinate system; Convert the image information in the user's area of ​​interest into the image information in the user's area of ​​interest from a bird's-eye view in the coordinate system of the oblique rearview camera group based on the BEV algorithm; According to the relative positions of the oblique rearview camera group coordinate system and the vehicle coordinate system, image information within the user's area of ​​interest from a bird's-eye view in the oblique rearview camera group coordinate system is converted into image information within the user's area of ​​interest from a bird's-eye view in the vehicle coordinate system; The first environmental image information from a bird's-eye view in the vehicle coordinate system located in front, behind, left and right of the vehicle and the image information within the user's area of ​​interest from a bird's-eye view in the vehicle coordinate system are calibrated and stitched to form a bird's-eye view of the vehicle's surrounding environment in the vehicle coordinate system.

6. The blind spot perception and control method based on multi-view collaboration according to claim 1, characterized in that: The warning methods of the in-vehicle infotainment system include highlighting the risk area of ​​the target lane on the screen, using dynamic arrow prompts on the screen to adjust the steering angle or delay steering, sound warnings, and voice reminders; The active intervention methods of the power-assist system include correcting the steering wheel, automatically braking and slowing down, and limiting the steering angle.

7. A blind spot perception and control device based on multi-view collaboration, characterized by: The device comprises: A first processing module (31) is configured to receive a vehicle speed in real time from a vehicle chassis system, and when the vehicle speed is lower than a preset vehicle speed threshold, receive first environmental image information in real time from a surround view camera group located in front of, behind, to the left of, and to the right of the vehicle; and calculate current lane information based on a deep learning algorithm and a SLAM algorithm based on the first environmental image information located in front of, behind, to the left of, and to the right of the vehicle; The second processing module (32) is configured to receive a turn signal and a steering wheel angle from the vehicle chassis system in real time, and send a start signal to the oblique rearview camera group according to the turn signal, so that the oblique rearview camera (2) on the corresponding side of the oblique rearview camera group is turned on; receive second environmental image information located obliquely behind the vehicle from the oblique rearview camera (2) on the corresponding side of the oblique rearview camera group in real time; predict the vehicle's driving trajectory and target lane information to be reached based on the steering wheel angle and vehicle speed based on the vehicle dynamics model; calculate the angle between the vehicle and the target lane based on the steering wheel angle and the vehicle dynamics model, and adjust the user's area of ​​interest in the second environmental image information according to the angle between the vehicle and the target lane and the steering wheel angle, and extract image information in the user's area of ​​interest based on the image recognition algorithm; The third processing module (33) is configured to construct a bird's-eye view of the vehicle's surrounding environment based on the first environment image information located in front of, behind, to the left and to the right of the vehicle and the image information in the user's area of ​​interest, based on the BEV algorithm and the multi-view perception fusion algorithm, and mark the current lane information and the target lane information to be reached in the bird's-eye view; determine the target obstacle information based on the bird's-eye view of the vehicle's surrounding environment based on the deep learning algorithm; predict the running trajectories of several target obstacles based on the target obstacle information in the continuous image frames; and calculate the collision time between the target obstacles and the vehicle based on the predicted running trajectories of the several target obstacles and the predicted driving trajectory of the vehicle; An early warning module (34) is configured to send an alarm signal to the in-vehicle infotainment system when the collision time between the target obstacle and the vehicle is within a preset early warning range, so that the in-vehicle infotainment system issues an alarm; The active intervention module (35) is configured to send an active intervention signal to the power assist system when the collision time between the target obstacle and the vehicle is within a preset active intervention range, so that the power assist system performs active intervention.

8. A blind spot perception and control device based on multi-view collaboration, characterized by: The device includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, a blind spot perception and control method based on multi-perspective collaboration according to any one of claims 1 to 6 is implemented.

9. A blind spot perception and control system based on multi-view collaboration, characterized by: It includes a surround view camera group, an oblique rear view camera group and the blind spot perception and control device based on multi-view collaboration as described in claim 8, wherein: The surround-view camera group includes four surround-view cameras (1), which are respectively fixedly mounted at the middle position of the front bumper, the middle position of the rear bumper, below the left rearview mirror, and below the right rearview mirror of the vehicle, and are used to respectively collect first environmental images in front of, behind, to the left, and to the right of the vehicle, and respectively transmit the first environmental images to a blind spot perception and control device based on multi-view collaboration; The oblique rearview camera group comprises two oblique rearview cameras (2), wherein the oblique rearview cameras (2) are fisheye cameras, and are fixedly mounted on the front left fender and the front right fender of the vehicle, respectively, for respectively collecting second environmental images of the left oblique rear of the vehicle and the right oblique rear of the vehicle, and respectively transmitting the second environmental images to a blind spot perception and control device based on multi-view collaboration.

10. A vehicle, characterized in that: Including the blind spot perception and control system based on multi-perspective collaboration as described in claim 9.

Images