Auxiliary driving method and device for automobile in backlight scene and automobile

Abstract

The invention relates to the technical field of automobile auxiliary driving, and provides an auxiliary driving method and device for an automobile in a backlight scene and the automobile. The method comprises the steps of collecting image data in the driving direction of an automobile, recognizing the image data to obtain a light scene where the automobile is located currently, collecting point cloud data in front of the automobile by using a vehicle-mounted laser radar when the automobile is in a backlight scene, and calculating the position of the automobile according to the motion state and the point cloud data of the automobile. Determining whether the three-dimensional scene model of the driving direction of the automobile and the automobile have a collision risk in a backlight scene or not; and if the automobile has the collision risk, displaying an image of a target object having the collision risk with the automobile in the three-dimensional scene model on target display equipment, and executing corresponding preset alarm measures. The visual field condition in front of the automobile is displayed to the driver in a visual mode after the driver encounters the backlight, the obstacle that the driver cannot normally recognize the road condition in front of the automobile due to the fact that the visual field is blocked in the backlight scene is overcome, and the automobile driving safety is enhanced.

Description

Technical Field

[0001] The present application relates to the field of automobile assisted driving technology, and in particular to an assisted driving method, device and automobile in a backlit scene. Background Art

[0002] Good vision is the key to safe driving, but in actual driving, we often encounter situations that affect vision. For example, on a sunny day, direct sunlight may cause strong glare, affecting the driver's and the camera's vision; at night, the high beams of oncoming vehicles or strong front lighting can also make it difficult for the driver to see the road clearly. These factors all pose hidden dangers to driving safety. Therefore, how to overcome the obstacle of the driver's obstructed vision in backlight scenes that affects safe driving is a technical problem that needs to be solved at present. Summary of the invention

[0003] In view of this, the embodiments of the present application provide an assisted driving method, device and automobile in a backlit scene to solve the technical problem that needs to be solved currently, that is, how to overcome the obstacle of the driver's obstructed field of vision in a backlit scene that affects safe driving.

[0004] According to a first aspect of an embodiment of the present application, a method for assisted driving of an automobile in a backlighting scene is provided, comprising: collecting image data of the direction in which the automobile is traveling, and identifying the image data to obtain a light scene in which the automobile is currently located, the light scene including a backlighting scene; when the light environment in which the automobile is currently located is a backlighting scene, collecting point cloud data in front of the automobile using an on-board laser radar, and determining a three-dimensional scene model of the direction in which the automobile is traveling and whether there is a risk of collision with the automobile in the backlighting scene based on the movement state of the automobile and the point cloud data; when it is determined that there is a risk of collision with the automobile, displaying an image of a target object in the three-dimensional scene model that is at risk of collision with the automobile on a target display device, and executing corresponding preset alarm measures.

[0005] According to a second aspect of an embodiment of the present application, there is provided an auxiliary driving device for an automobile in a backlighting scene, comprising: a scene recognition module, configured to collect image data of the direction in which the automobile is traveling, and to recognize the image data to obtain the light scene in which the automobile is currently located, wherein the light scene includes a backlighting scene; a risk determination module, configured to, when the light environment in which the automobile is currently located is a backlighting scene, collect point cloud data in front of the automobile using an on-board laser radar, and determine a three-dimensional scene model of the direction in which the automobile is traveling and the collision risk of the automobile in the backlighting scene according to the motion state of the automobile and the point cloud data; and an auxiliary driving module, configured to, when it is determined that the automobile is at risk of collision, display an image of a target object in the three-dimensional scene model that is at risk of collision with the automobile on a target display device, and execute corresponding preset alarm measures.

[0006] According to a third aspect of an embodiment of the present application, a car is provided, comprising a camera, a laser radar, a target display device and a control device, wherein the control device comprises an intelligent driving domain controller and a cockpit domain controller, the intelligent driving domain controller is respectively connected to the camera and the laser radar, the cockpit domain controller is respectively connected to the intelligent driving domain controller and the target display device, the intelligent driving domain controller and the cockpit domain controller each comprise at least a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above method when executing the computer program.

[0007] Compared with the prior art, the beneficial effects of the embodiments of the present application are as follows: the above-mentioned assisted driving method of the automobile in the backlight scene determines whether the automobile is currently in the backlight scene by collecting and identifying image data of the direction of travel of the automobile, and then uses the lidar to obtain the front point cloud data, constructs a three-dimensional scene model and evaluates the collision risk. If there is a risk, the image of the relevant target object is displayed on the display device and alarm measures are executed, so as to show the driver the field of vision in front of the car after encountering backlight in a visual manner, thereby overcoming the obstacle that the driver's field of vision is blocked and he cannot normally identify the road conditions in front of the car in the backlight scene, thereby enhancing the driving safety of the car in the backlight scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0009] Figure 1 It is a flow chart of an assisted driving method for a car in a backlit scene provided by an embodiment of the present application;

[0010] Figure 2 It is a flow chart of another method for assisting driving of an automobile in a backlit scene provided by an embodiment of the present application;

[0011] Figure 3 It is a structural schematic diagram of an auxiliary driving device for a car in a backlighting scene provided by an embodiment of the present application;

[0012] Figure 4 It is a partial structural schematic diagram of a car provided in an embodiment of the present application;

[0013] Figure 5 It is a structural diagram of an intelligent driving domain controller or a cockpit domain controller provided in an embodiment of the present application. DETAILED DESCRIPTION

[0014] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0015] Good vision is the key to safe driving. During driving, strong light will affect the vision, such as direct sunlight on a sunny day and high beams of oncoming headlights at night. Strong glare makes it difficult for the driver to see the road ahead, increasing safety risks.

[0016] In response to the above technical problems, an embodiment of the present application provides an assisted driving method for a car in a backlit scene. By identifying the backlit scene and using a lidar to generate a three-dimensional scene model, the collision risk is assessed in real time, and alarm measures are executed when the risk is detected, thereby improving driving safety in the backlit scene.

[0017] Furthermore, an embodiment of the present application provides an assisted driving method for a car in a backlit scene, which determines whether the car is currently in a backlit scene by collecting and identifying image data of the car's driving direction, then uses a lidar to obtain the point cloud data in front, constructs a three-dimensional scene model and evaluates the collision risk, and finally presents the image of the potential dangerous object on the display device, and implements preset alarm measures, thereby enhancing the driving safety of the car in the backlit scene.

[0018] See also Figure 1 In a first embodiment of the present application, a method for assisting driving of a car in a backlit scene is provided, comprising the steps of:

[0019] S101, collecting image data of the direction in which the vehicle is traveling, and identifying the image data to obtain the light scene in which the vehicle is currently located, wherein the light scene includes a backlight scene;

[0020] S102, when the current light environment of the vehicle is a backlit scene, using the on-board laser radar to collect point cloud data in front of the vehicle, and determining a three-dimensional scene model of the vehicle's driving direction and whether there is a collision risk of the vehicle in the backlit scene according to the vehicle's motion state and the point cloud data;

[0021] S103, when it is determined that the car has a collision risk, an image of a target object in the three-dimensional scene model that has a collision risk with the car is displayed on a target display device, and corresponding preset alarm measures are executed.

[0022] According to the technical solution provided in the embodiment of the present application, by collecting and identifying image data of the direction of travel of the car, it is determined whether the car is currently in a backlit scene, and then the front point cloud data is obtained by using a lidar, a three-dimensional scene model is constructed and the collision risk is evaluated. If there is a risk, the image of the relevant target object is displayed on the display device and alarm measures are executed, and the driver is shown the field of vision in front of the car after encountering backlight in a visual manner, thereby overcoming the obstacle that the driver's field of vision is blocked and the road conditions in front of the car are unable to normally identify the road conditions in front of the car in the backlit scene, thereby enhancing the driving safety of the car in the backlit scene.

[0023] In step S101, a vehicle-mounted camera is provided on the vehicle, and the vehicle-mounted camera can be used to capture image data in front of the vehicle, and the image data can be a picture or a video. When the image data is a picture, the vehicle-mounted camera can be controlled to periodically capture the road conditions in front of the vehicle to obtain a corresponding picture, and the picture can be identified to obtain the light scene in which the vehicle is currently located. When the image data is a video, one or more frames of images can be extracted from the video as a target image, and the light scene in which the vehicle is currently located can be obtained by identifying the target image. For example, the vehicle has an automatic driving function, and uses a vehicle-mounted camera to collect a video of the road conditions in front of the vehicle to perceive the surrounding environment of the vehicle. At this time, at least one frame of image can be selected from the video as a target image to identify the target image and obtain the light scene in which the vehicle is currently located.

[0024] Cars may encounter backlighting scenarios both during the day and at night. Backlighting scenarios for cars during the day include, but are not limited to, direct sunlight and strong reflected light. Direct sunlight occurs in the early morning or evening when the sun is at a low angle. The sunlight may directly shine in the direction of the driver or the onboard camera, resulting in obstructed vision or overexposure of the image. This is a typical daytime backlighting. Strong reflected light occurs in the daytime environment. Some reflective objects (such as slippery roads, glass, metal buildings, etc.) may also produce strong reflected light, resulting in backlighting scenarios. Backlighting scenarios for cars at night include, but are not limited to oncoming headlights, street lights, or other strong light sources. Oncoming headlights are when driving at night. The strong headlights of the opposite vehicle will produce a similar backlighting effect, affecting the driver's and the onboard camera's line of sight. Street lights or other strong light sources are street lights, billboards, building lights, etc. at night. They may directly shine in front of the car, causing a short-term backlighting effect and interfering with the driver's driving vision. It can be seen that backlight scenes do not only occur at night. Direct sunlight or strong reflected light during the day may also interfere with the driver's vision. How to effectively identify backlight scenes of the car during the day and at night is the basis for ensuring safe driving of the car.

[0025] In some optional embodiments, in step S101, image data of the direction of travel of the car is collected, and the image data is identified to obtain the light scene in which the car is currently located, including: identifying the driving environment of the car, the driving environment including the daytime driving environment and the nighttime driving environment; when the driving environment is the daytime driving environment, detecting the change in light intensity in the driving direction, and judging whether the change in light intensity exceeds a set threshold: if it exceeds the set threshold, it is judged that the car is in a backlight scene; if it does not exceed the set threshold, it is judged that the car is not in a backlight scene; when the driving environment is the night driving environment, acquiring image data in front of the car, and detecting whether the image data has local overexposure; if there is local overexposure, it is judged that the car is in a backlight scene; if there is no local overexposure, it is judged that the car is not in a backlight scene.

[0026] There is not only one way to identify the driving environment of a car. For example, an ambient light sensor can be used to identify the driving environment of a car. Specifically, the ambient light sensor can detect the intensity of the surrounding light in real time, and determine whether the current driving environment is daytime or nighttime based on the light brightness value. When the ambient light intensity is lower than a certain set threshold, the system can determine that it is a nighttime driving environment; when the light intensity is higher than the threshold, the system determines that it is a daytime driving environment. For another example, the on-board time information can also be used to identify the driving environment of a car. Specifically, the car can use the built-in clock to obtain the current time, and determine the sunrise and sunset times in combination with the geographical location. When the time is before sunrise or after sunset, the system can determine that the current driving environment is nighttime; between sunrise and sunset, it is determined to be a daytime driving environment.

[0027] When the driving environment is a daytime driving environment, the ambient light sensor can be used to detect the change in light intensity in front of the car. If the light intensity change ratio exceeds the set threshold, it is determined that the car is in a backlight scene. If the light intensity change does not exceed the set threshold, it is determined that the car is in a non-backlight scene.

[0028] It should be noted that backlight scene recognition is different from driving environment judgment. The judgment of daytime driving environment and nighttime driving environment usually depends on the light intensity. If the ambient light is strong (high light intensity), the system can infer that it is a daytime driving environment; if the light intensity is weak, it is inferred that it is a nighttime driving environment. The recognition of backlight scenes pays more attention to sudden changes in light.

[0029] For example, in an actual application scenario, a vehicle is driving on a highway in the early morning. There is no direct sunlight in front, and the lighting is relatively stable. But after the vehicle turns a corner, it suddenly faces the rising sun, and the light changes drastically. Then the working process of the car identifying the backlight scene in the daytime driving environment is as follows: First, the ambient light sensor in front of the vehicle continuously collects light intensity. At first, the system detects that the current light intensity is 8000 lux, and the ambient light is soft; then, as the vehicle enters the corner, the vehicle suddenly faces the sun, and the sensor detects that the light intensity increases rapidly from 8000 lux to 45000 lux in a short period of time, that is, a significant light intensity change occurs. The system sets a threshold for light intensity change. For example, if the light intensity changes by more than 20000 lux in a short period of time (such as within a few seconds), it is judged as a backlight scene. Through the monitoring of light intensity changes, it can be seen that the light intensity rises from 8000 lux to 45000 lux, and the change value is 37000 lux, which exceeds the threshold set by the system (20000 lux). Then, because the light intensity changes rapidly and exceeds the threshold, the system determines that the vehicle is currently in a backlighting scene, which may cause the driver's vision to be obstructed or the sensor recognition to fail.

[0030] In the case where the driving environment is a night driving environment, the on-board camera can be used to obtain the image in front of the car. In a normal night environment, the image brightness is relatively low and the entire picture is relatively uniform. However, if there is an oncoming vehicle and a strong light is used, the system will detect that the brightness of certain areas (the position of the oncoming vehicle's headlights) in the image has increased significantly. This phenomenon is called "local overexposure." The light scene is determined by detecting whether there is local overexposure in the image. If there is local overexposure, it is determined that the car is in a backlit scene; if there is no local overexposure, it is determined that the car is not in a backlit scene.

[0031] For example, in an actual application scenario, a car is driving at night, and there is a narrow country road ahead. A car is coming in the distance, turning on the strong high beam. At this time, the vehicle system needs to identify the backlit scene to prevent the strong light from interfering with the driver's vision and the vehicle's perception system. Then the working process of identifying the backlit scene at night is: first, the vehicle-mounted camera is used to continuously collect image data of the road ahead, where the camera can be selected as a night vision camera with HDR (abbreviation of High Dynamic Range Imaging, translated as high dynamic range imaging) function. The camera has a resolution of 1920x1080 pixels and has a high dynamic range (HDR) so that it can process details of bright and dark areas at the same time. Then, in the image data, the brightness of each pixel is represented by 8 bits, that is, the brightness value ranges from 0 (black) to 255 (full white). The system analyzes the distribution of brightness values ​​in the image frame by frame, focusing on monitoring whether there are obvious abnormal brightness areas in the road ahead. Among them, the system sets a threshold for local overexposure for the night driving environment. For example, when more than 30% of the pixel brightness values ​​in a certain area of ​​the image (such as a local area with a size of 100x100 pixels) are greater than 220, the area is judged to be overexposed. Here, the threshold 220 is selected based on the fact that in night scenes, the brightness value of normal street lights or taillights of the front car will not be higher than 220, while high beams or other strong light sources will cause the local pixel brightness to be close to 255. Then, the system analyzes the image data captured by the camera frame by frame to determine the local overexposure. For example, in a certain frame of the image, the headlight area of ​​the oncoming vehicle (occupying about 150x150 pixels of the image) is identified as a local highlight area. In this area, the system detects that more than 40% of the pixel brightness values ​​reach or exceed 230, which far exceeds the set overexposure threshold (220), indicating that there is a strong light source in this area, resulting in local overexposure. Finally, based on the overexposure analysis results, the system determines that there is an obvious backlight scene ahead, which may be the direct strong light of the oncoming vehicle. Because the local overexposed area occupies a certain proportion of the image (such as 5% or more of the entire image), and the overexposed pixels account for more than 40% of the total pixels in the area, the system confirms that backlighting affects driving safety.

[0032] It can be seen that according to the technical solution provided by the embodiment of the present application, by distinguishing the two driving environments of daytime and nighttime, and detecting the light intensity and image overexposure respectively, it is determined whether the car is in a backlit scene. In the daytime, the backlight is determined by monitoring the change in light intensity; at night, the backlight is identified by the partial overexposure of the image, thereby realizing accurate identification of backlit scenes under different lighting conditions and improving driving safety.

[0033] Since the field of vision in daytime driving environment is better than that in nighttime driving environment, and the backlight scene in daytime can be overcome by operating the sun visor or wearing sunglasses, therefore, in practical applications, only the backlight scene in nighttime driving environment can be identified.

[0034] In some embodiments, in the above step S101, image data of the direction of vehicle travel is collected, and the image data is identified to obtain the light scene in which the vehicle is currently located, including: detecting whether the vehicle is in a target driving environment, the target driving environment includes a night driving environment; when the target driving environment is a night driving environment, obtaining image data in front of the vehicle; inputting the image data into a pre-trained backlighting scene classification model, and the backlighting scene classification model outputs the light scene in which the vehicle is currently located, the light scene includes a backlighting scene; or, detecting whether the image data has a local overexposure phenomenon, if there is a local overexposure phenomenon, determining that the vehicle is in a backlighting scene; if there is no local overexposure phenomenon, determining that the vehicle is not in a backlighting scene.

[0035] The difference from the above optional embodiments is that, in addition to identifying the night backlight scenes by detecting whether the image data has local overexposure, the present embodiment can also use machine learning to pre-train a backlight scene classification model, and use the backlight scene classification model to automatically identify the captured image data to obtain the corresponding light scene recognition results, thereby being able to accurately judge complex light scenes such as backlighting and improve driving safety.

[0036] In this embodiment, a pre-trained backlight scene classification model is used to determine whether the current light scene of the car is backlight. There is no unique way to train the backlight scene classification model.

[0037] For example, the collected image data is analyzed by a deep learning model, and different lighting scenes (such as normal light, low light, and backlight) are classified using a pre-trained scene classification neural network. The network can be trained based on a large amount of backlight scene data to obtain a backlight scene classification model, and the backlight scene classification model is deployed on the car to achieve efficient recognition of backlight scenes.

[0038] It can be seen that according to the technical solution provided in the embodiment of the present application, by acquiring image data in front of the car and using a pre-trained classification model, it is automatically identified whether it is in a backlit scene, providing the vehicle with accurate light scene judgment, especially in a backlit environment. It helps the car to visualize the road conditions ahead and identify collision risks, thereby improving driving safety.

[0039] When the light scene the car is currently in is a backlit scene, the driver cannot see the road structure and obstacles behind the strong light in such a scene, which poses a great safety hazard. Therefore, the embodiment of the present application uses a lidar to collect three-dimensional point cloud data in front of the car for target recognition and scene reconstruction, and also identifies the collision risk of the target, so as to help the driver obtain the field of vision behind the strong backlight in front and increase the safety of the driving process.

[0040] In some optional embodiments, in step S102, a three-dimensional scene model of the car's driving direction and whether there is a collision risk for the car in a backlit scene are determined based on the car's motion state and point cloud data, including: generating a three-dimensional scene model based on the point cloud data, the three-dimensional scene model including the geometric state and motion state of objects around the car; determining the collision time between the car and the surrounding objects in the three-dimensional scene model based on the car's motion state and the geometric state and motion state of objects around the car; if the collision time is less than a safety threshold, determining that the car is at risk of collision in the backlit scene; if the collision time is greater than or equal to the safety threshold, determining that the car is not at risk of collision in the backlit scene.

[0041] Specifically, this embodiment uses the on-board laser radar to acquire point cloud data of the surrounding environment to construct a three-dimensional scene model that includes the geometric state and motion state of objects around the car, and combines the motion state of the car with the object information in the three-dimensional scene to calculate the collision time between the car and the surrounding objects. The calculated collision time is then compared with a preset safety threshold. If the collision time is less than the safety threshold, it is determined that there is a collision risk; otherwise, it is determined that there is no collision risk.

[0042] Specifically, the motion state of the car includes but is not limited to dynamic parameters such as the speed, acceleration, and driving direction of the car. These parameters can be used to predict the position and motion behavior of the car in the future.

[0043] Specifically, the geometric state of objects around the car includes but is not limited to the shape, size and relative position of objects around the car, such as the size of pedestrians, the shape and position of other vehicles, etc. This information helps to establish the shape and distribution of objects in the three-dimensional scene model.

[0044] Specifically, the motion state of objects around the car includes but is not limited to the speed, acceleration and motion direction of the surrounding objects, such as the speed of pedestrians, the speed and direction of other vehicles, etc. These dynamic information are used to determine the possibility and time of collision with the car.

[0045] Specifically, the safety threshold may be a fixed threshold pre-set based on experience, or may be a new threshold obtained by adjusting an initially set fixed threshold based on a recognized light scene, and the embodiments of the present application are not limited to this.

[0046] For example, in an application scenario, assuming that the safety threshold is 2s, the vehicle is equipped with a laser radar (LiDAR), which collects point cloud data in real time to obtain the coordinates of surrounding objects. For example, a pedestrian is located at (10,0,0) meters. Next, a three-dimensional scene model is generated using the collected point cloud data. Assume that the model includes a pedestrian (geometric state) and a stationary vehicle in front (geometric state). The pedestrian's speed is 1m / s and the vehicle's speed is 0m / s. Assume that the vehicle's current speed is 15m / s and it is moving forward. The distance between the pedestrian and the vehicle is calculated to be 10 meters. And the collision time is calculated based on the relative speed:

[0047]

[0048] Here it is assumed that the safety threshold is set to 2s, and the calculated collision time of 0.67s is compared with 2s. Since 0.67s is less than 2s, the system determines that there is a collision risk and issues a warning to prompt the driver to take evasive measures, so that the driver can be informed of the potential collision risk in time under backlight conditions, thereby improving driving safety and avoiding accidents.

[0049] It can be seen that according to the technical solution provided in the embodiment of the present application, a three-dimensional scene model is generated through point cloud data to describe the geometry and motion state of objects around the car. Then, the collision time with the surrounding objects is calculated in combination with the motion state of the car, and compared with the set safety threshold. If the collision time is less than the safety threshold, the system determines that there is a collision risk, otherwise it is considered safe, which effectively improves the safety in backlit scenes and can identify potential risks in real time, thereby enhancing the intelligence level of the driving assistance system.

[0050] In addition to calculating the collision time between the car and the target object in front to determine the collision risk of the car, the relative distance between the car and the target object in front within a preset time in the future can also be calculated to determine the collision risk of the car.

[0051] In some optional embodiments, in step S102, the three-dimensional scene model of the car's driving direction and whether the car has a collision risk in a backlit scene are determined based on the car's motion state and point cloud data, including: determining the obstacle in front of the car, as well as the geometric state and motion state of the obstacle based on the point cloud data; determining whether the relative distance between the car and the obstacle within a preset time period in the future is less than a set safety threshold based on the car's motion state and the geometric state and motion state of the obstacle; if it is less than the safety distance threshold, it is determined that the car has a collision risk; if it is greater than or equal to the safety distance threshold, it is determined that the car does not have a collision risk.

[0052] Specifically, this embodiment uses point cloud data to identify obstacles in front of the car and their geometry and motion state, and combines the car's motion state to predict the relative distance in the future. If the distance is less than the set safety threshold, it is determined that there is a collision risk; otherwise, it is determined to be safe.

[0053] Specifically, the motion state of the car includes but is not limited to dynamic parameters such as the speed, acceleration, and driving direction of the car. These parameters can be used to predict the position and motion behavior of the car in the future.

[0054] Specifically, the geometric state of obstacles includes but is not limited to the shape, size and relative position of objects around the car, such as the size of pedestrians, the shape and position of other vehicles, etc. This information helps to establish the shape and distribution of objects in the three-dimensional scene model.

[0055] Specifically, the geometric state of the obstacle includes but is not limited to the speed, acceleration and direction of movement of the surrounding objects, such as the speed of pedestrians, the speed and direction of other vehicles, etc. These dynamic information are used to determine the possibility and time of collision with the car.

[0056] Specifically, the safety distance threshold may be a fixed threshold pre-set based on experience, or may be a new threshold obtained by adjusting the initially set fixed threshold based on the identified light scene, and the embodiments of the present application are not limited to this.

[0057] For example, in an application scenario, assuming that the safety distance threshold is 5 meters, point cloud data is used to identify the stationary vehicle in front, determine its position and size, and if the current distance between the car and the obstacle is 20 meters, and the obstacle is stationary, the future preset time is 2 seconds. Then calculate the future relative distance according to the following formula: future distance = current distance - (car speed × preset time) = 20 meters - (15 meters / second × 2 seconds) = -10 meters. Then, compare the calculated future distance (-10 meters) with the safety distance threshold (5 meters). Since -10 meters is less than 5 meters, the system determines that there is a collision risk. It can be seen that this embodiment can monitor and predict potential collision risks in real time, ensure timely response in a dynamic environment, and enhance driving safety and driver confidence.

[0058] It is worth mentioning that in the above embodiment, the obstacles in front of the car are determined by using the collected point cloud data. In actual applications, the specific implementation method of identifying obstacles in front of the car by using point cloud data is not unique.

[0059] In some optional embodiments, obstacles in front of the car are determined based on point cloud data, including: inputting the point cloud data into a pre-trained object classification model, and the object classification model outputs a corresponding classification result, and the classification result includes the obstacle in front of the car.

[0060] Specifically, the object classification model is obtained through machine learning. In practical applications, deep learning algorithms, clustering analysis or target detection algorithms can be used to train and learn to obtain the object classification model.

[0061] In an application scenario, taking the use of a deep learning algorithm to learn an object classification model as an example, data collection is first performed, that is, point cloud data around the vehicle is obtained through a laser radar (LiDAR) or other sensor. These point cloud data contain information about various objects in the surrounding environment. Then data preprocessing is performed, including but not limited to filtering and denoising the point cloud data, removing unnecessary points, and retaining objects related to vehicle driving. Then model training is performed, using a labeled point cloud data set to train a convolutional neural network (generally abbreviated as CNN) model. The model learns to recognize objects of different categories, such as pedestrians, vehicles, trees, etc. Finally, the trained object classification model is arranged on the car. During the driving process of the car, the point cloud data collected in real time is input into the trained object classification model to perform obstacle recognition, and the object classification model outputs the type and position of the obstacle. Through the technical solution of this embodiment, the car can effectively identify obstacles ahead, thereby improving driving safety.

[0062] When it is determined that a car is at risk of collision in a backlit scene, a three-dimensional scene model constructed based on point cloud data can be displayed on the target display device, or a target object in the three-dimensional scene model that will have a collision risk with the car can be displayed on the target display device, so that the user can intuitively obtain the road conditions behind the backlit scene through the target display device.

[0063] It is worth mentioning that the three-dimensional scene model displayed on the target display device can be a simplified object outline image or a rendered three-dimensional real scene image, which is not limited in this embodiment of the present application.

[0064] In addition, since it takes a certain amount of processing time to display the three-dimensional scene model on the target display device, when there is a risk of collision between the car and the obstacle ahead, even if the user can see the three-dimensional scene model on the target display device, due to factors such as data processing delays and obstructed vision, the user may not have time to perform risk avoidance operations or make operational errors. Therefore, the user can be reminded by controlling the car to execute preset alarm measures, or the car can be assisted in control to avoid a collision.

[0065] In some optional embodiments, in step S103, corresponding preset alarm measures are executed, including executing at least one of the following measures: issuing a sound alarm prompt or a light alarm prompt; highlighting the image of the target object on the target display device; and activating an emergency braking system or an emergency steering assist system.

[0066] Specifically, when it is determined that the car is at risk of collision in a backlit scene, an audible alarm or a light alarm can be issued. For example, in an autonomous vehicle, when the sensor detects that a pedestrian suddenly enters the vehicle's path, the system will sound an alarm through the horn and activate the warning light in front to alert the driver and pedestrian.

[0067] When it is determined that the car has a collision risk in a backlit scene, the image of the target object can also be highlighted on the target display device. For example, a three-dimensional scene model is displayed on the target display device, and the target object in the three-dimensional scene model that has a collision risk with the car is highlighted on the target display device to provide a visual warning and help the driver respond in time.

[0068] It should be noted that the target display device displaying the three-dimensional scene model can be a physical screen such as an instrument screen or a central control screen on a car, or it can be a virtual screen formed by projection, such as a head-up display, and the embodiments of the present application do not limit this.

[0069] When it is determined that the car is at risk of collision in a backlit scene, the emergency braking system or emergency steering assistance system can also be activated. For example, when the system recognizes that the car is at risk of collision, the automatic emergency braking system is activated to automatically brake the car to prevent a collision. At the same time, the system can also activate the steering assistance function to control the car to automatically steer so that the car automatically avoids obstacles.

[0070] It can be seen that this embodiment significantly improves the safety of the vehicle by executing at least one of the above preset alarm measures, and provides timely feedback and response when the car is at risk of collision, thereby improving the safety of the car driving in backlight scenarios.

[0071] In addition, when point cloud data is used to identify whether a car is at risk of collision in a backlit scene, if the car is not at risk of collision, corresponding preset measures can be executed to ensure the safe driving of the car.

[0072] In some optional embodiments, in step S102, after determining the three-dimensional scene model of the vehicle's driving direction and whether the vehicle has a collision risk in a backlit scene, it also includes: if it is determined that the vehicle has no collision risk, displaying the three-dimensional scene model on a target display device.

[0073] Specifically, although no collision risk of the car is detected in the backlight scenario, the driver's field of vision is still obstructed. The driver cannot see the road structure and obstacles behind the strong light in such a scenario, which poses a great safety hazard. Therefore, this embodiment displays the constructed three-dimensional scene model on the target display device, that is, visualizes the road conditions behind the backlight to the driver, to help the driver obtain the field of vision behind the strong backlight ahead, thereby increasing the safety of the driving process.

[0074] See also Figure 2 In a second embodiment of the present application, another method for assisting driving of a car in a backlit scene is provided, comprising the steps of:

[0075] S201, detecting whether the vehicle has turned on the auxiliary driving function in the backlight scene: if it is turned on, executing step S202; if it is not turned on, continuing to execute step S201;

[0076] S202, collecting image data of the direction of the car's travel, and identifying the image data to determine whether the light scene the car is currently in is a backlight scene: if so, executing step S203, if not, returning to executing step S202;

[0077] S203, using the on-board laser radar to collect point cloud data in front of the car, and determining a three-dimensional scene model of the car's driving direction and whether there is a collision risk of the car in a backlit scene according to the car's motion state and the point cloud data;

[0078] S204, when it is determined that there is no collision risk with the car, display the three-dimensional scene model on the target display device, and return to step S202;

[0079] S205, when it is determined that the car has a collision risk, an image of a target object in the three-dimensional scene model that has a collision risk with the car is displayed on the target display device, and corresponding preset alarm measures are executed, and the execution returns to step S202.

[0080] According to the technical solution provided in the embodiment of the present application, by collecting and identifying image data of the direction of travel of the car, it is determined whether the car is currently in a backlit scene, and then the front point cloud data is obtained by using a lidar, a three-dimensional scene model is constructed and the collision risk is evaluated. If there is a risk, the image of the relevant target object is displayed on the display device and alarm measures are executed, and the driver is shown the field of vision in front of the car after encountering backlight in a visual manner, thereby overcoming the obstacle that the driver's field of vision is blocked and the road conditions in front of the car are unable to normally identify the road conditions in front of the car in the backlit scene, thereby enhancing the driving safety of the car in the backlit scene.

[0081] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0082] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0083] See also Figure 3 In the third embodiment of the present application, a driving assistance device for a car in a backlit scene is provided, such as Figure 3 As shown, the auxiliary driving device of the car in the backlight scene includes:

[0084] The scene recognition module 301 is configured to collect image data of the driving direction of the car and recognize the image data to obtain the light scene where the car is currently located, and the light scene includes a backlight scene;

[0085] The risk determination module 302 is configured to collect point cloud data in front of the car using the on-board laser radar when the light environment currently in which the car is located is a backlit scene, and determine the three-dimensional scene model of the car's driving direction and whether there is a collision risk of the car in the backlit scene according to the movement state of the car and the point cloud data;

[0086] The auxiliary driving module 303 is configured to display the image of the target object in the three-dimensional scene model that has the risk of collision with the car on the target display device when it is determined that the car has the risk of collision, and execute corresponding preset alarm measures.

[0087] According to the technical solution provided in the embodiment of the present application, by collecting and identifying image data of the direction of travel of the car, it is determined whether the car is currently in a backlit scene, and then the front point cloud data is obtained by using a lidar, a three-dimensional scene model is constructed and the collision risk is evaluated. If there is a risk, the image of the relevant target object is displayed on the display device and alarm measures are executed, and the driver is shown the field of vision in front of the car after encountering backlight in a visual manner, thereby overcoming the obstacle that the driver's field of vision is blocked and the road conditions in front of the car are unable to normally identify the road conditions in front of the car in the backlit scene, thereby enhancing the driving safety of the car in the backlit scene.

[0088] In some optional embodiments, the scene recognition module 301 is configured to identify the driving environment of the car, which includes a daytime driving environment and a nighttime driving environment; when the driving environment is a daytime driving environment, the light intensity change in the driving direction is detected, and it is determined whether the light intensity change exceeds a set threshold: if it exceeds the set threshold, it is determined that the car is in a backlight scene; if it does not exceed the set threshold, it is determined that the car is not in a backlight scene; when the driving environment is a nighttime driving environment, image data in front of the car is obtained, and the image data is detected whether there is local overexposure. If there is local overexposure, it is determined that the car is in a backlight scene; if there is no local overexposure, it is determined that the car is not in a backlight scene.

[0089] In some optional embodiments, the scene recognition module 301 is configured to detect whether the car is in a target driving environment, which includes a night driving environment; when the driving environment is a night driving environment, obtain image data in front of the car; input the image data into a pre-trained backlighting scene classification model, and the backlighting scene classification model outputs the light scene in which the car is currently located, which includes a backlighting scene; or, detect whether the image data has a local overexposure phenomenon, and if so, determine that the car is in a backlighting scene; if not, determine that the car is not in a backlighting scene.

[0090] In some optional embodiments, the risk determination module 302 is configured to generate a three-dimensional scene model based on point cloud data, where the three-dimensional scene model includes the geometric state and motion state of objects around the car; determine the collision time between the car and the surrounding objects in the three-dimensional scene model based on the motion state of the car and the geometric state and motion state of the objects around the car; if the collision time is less than a safety threshold, it is determined that the car has a collision risk in a backlit scene; if the collision time is greater than or equal to the safety threshold, it is determined that the car has no collision risk in a backlit scene.

[0091] In some optional embodiments, the risk determination module 302 is configured to determine the obstacle in front of the car, as well as the geometric state and motion state of the obstacle based on the point cloud data; determine whether the relative distance between the car and the obstacle is less than a set safety threshold within a preset time period in the future based on the motion state of the car and the geometric state and motion state of the obstacle; if it is less than the safety distance threshold, it is determined that the car is at risk of collision; if it is greater than or equal to the safety distance threshold, it is determined that the car is not at risk of collision.

[0092] In some optional embodiments, the risk determination module 302 is configured to input the point cloud data into a pre-trained object classification model, and the object classification model outputs a corresponding classification result, which includes obstacles in front of the car.

[0093] In some optional embodiments, the auxiliary driving module 303 is further configured to display a three-dimensional scene model on a target display device when it is determined that there is no collision risk with the automobile.

[0094] In some optional embodiments, the above-mentioned assisted driving module 303 is also configured to execute corresponding preset alarm measures, which include executing at least one of the following measures: issuing a sound alarm prompt or a light alarm prompt; highlighting the image of the target object on the target display device; activating the emergency braking system or the emergency steering assist system.

[0095] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0096] See also Figure 4 and Figure 5 In the fourth embodiment of the present application, a car is also provided, including a camera, a laser radar, a target display device and a control device, the control device includes an intelligent driving domain controller and a cockpit domain controller, the intelligent driving domain controller is respectively connected to the camera and the laser radar, and the cockpit domain controller is respectively connected to the intelligent driving domain controller and the target display device. Among them, the intelligent driving domain controller and the cockpit domain controller each include at least: a processor 601, a memory 602, and a computer program 603 stored in the memory 602 and executable on the processor 601. When the processor 601 executes the computer program 603, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 601 executes the computer program 603, the functions of each module in the above-mentioned device embodiments are implemented.

[0097] The laser radar provides three-dimensional point cloud data to the intelligent driving domain controller, and the camera provides video stream data to the intelligent driving domain controller. Optionally, in actual work, ground information such as vehicle speed can also be provided to the intelligent driving domain controller to determine the current driving conditions.

[0098] The intelligent driving domain controller will perform scene recognition based on visual data. The scene recognition result mainly determines whether it is a strong backlight scene at night. In this scenario, it will start to identify the target and reconstruct the scene based on the original 3D point cloud data of the lidar, and also identify the collision risk of the target. The final processing results, including 3D scene data and dangerous target data, will be sent to the cockpit domain controller.

[0099] After receiving the relevant processing data sent by the intelligent driving domain controller, the cockpit domain controller will perform visualization processing of the scene in front and send the visualization results to the central control screen / HUD for display. If there is a high risk of collision with a target, the alarm mechanism will be triggered, highlighting the relevant target on the central control screen / HUD, and also providing a sound alarm.

[0100] It is worth mentioning that the intelligent driving domain controller and the cockpit domain controller include but are not limited to the processor 601 and the memory 602. Those skilled in the art can understand that Figure 5 The electronic device 6 is merely an example and does not limit the electronic device 6 . The electronic device 6 may include more or fewer components than those shown in the figure, or different components.

[0101] The processor 601 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0102] The memory 602 may be an internal storage unit of the electronic device 6, for example, a hard disk or memory of the electronic device 6. The memory 602 may also be an external storage device of the electronic device 6, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 6. The memory 602 may also include both an internal storage unit of the electronic device 6 and an external storage device. The memory 602 is used to store computer programs and other programs and data required by the electronic device.

[0103] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0104] If the integrated module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium (e.g., a computer-readable storage medium). Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. Computer-readable storage media may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0105] Target display devices include but are not limited to central control screens, head-up displays, etc.

[0106] In an application scenario, the steps of the method for realizing assisted driving of a car in a backlit scene provided by this embodiment are as follows: first, check whether the function is turned on. If it is not turned on, the process ends directly. After the function is turned on, the intelligent driving domain controller will first receive the camera data and perform image-based scene recognition. If no strong backlit scene at night is detected, the function jumps directly to the end. When it is detected that the current scene is a strong backlit scene at night, the lidar will be turned on and the current lidar point cloud data will be collected, and the 3D scene will be reconstructed based on the point cloud data, and the reconstruction result will be sent to the downstream cockpit domain controller.

[0107] Next, the intelligent driving domain controller will perform a collision risk detection between the vehicle and the target in front based on the real-time vehicle motion information and lidar target information received. If there is a high risk (the level of collision risk can be characterized by the collision time TTC (abbreviation of Time-To-Collision)), the target-related attributes will also be sent to the downstream cockpit domain controller.

[0108] Then, the cockpit domain controller receives the reconstructed 3D scene data and dangerous target attributes, and first determines whether there is a dangerous target sent. If not, it only needs to control the corresponding central control screen / HUD for visual display. If there is a dangerous target, it is also necessary to highlight the target in the visualization and give information prompts, and issue a voice alarm prompt message at the same time.

[0109] Finally, after the vehicle passes through the strong backlight scene, the relevant alarms and visual displays are cancelled and the process ends.

[0110] It can be seen that this embodiment targets strong backlighting scenes at night. In such scenes, the driver cannot see the road structure and obstacles behind the strong light, which poses a great safety hazard. When it is determined that the car is at risk of collision, the three-dimensional scene model constructed by the point cloud data is visualized to help the driver obtain the field of vision behind the strong backlight in front, thereby increasing the safety of the driving process. In addition, when it is determined that the car is at risk of collision, the corresponding auxiliary driving function can be turned on to assist in controlling the car to avoid a collision, which is very practical for safe driving.

[0111] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for assisting driving of a car in a backlit scene, characterized in that: include: Collecting image data of the direction in which the vehicle is traveling, and identifying the image data to obtain a light scene in which the vehicle is currently located, wherein the light scene includes a backlight scene; When the light scene currently in which the car is located is a backlit scene, the on-board laser radar is used to collect point cloud data in front of the car, and the three-dimensional scene model of the car's driving direction and whether there is a collision risk of the car in the backlit scene are determined according to the car's motion state and the point cloud data; When it is determined that the automobile is at risk of collision, an image of a target object in the three-dimensional scene model that is at risk of collision with the automobile is displayed on a target display device, and corresponding preset alarm measures are executed.

2. The method according to claim 1, characterized in that: After determining the three-dimensional scene model of the vehicle's driving direction and whether the vehicle has a collision risk in the backlit scene, the method further includes: When it is determined that there is no collision risk for the automobile, the three-dimensional scene model is displayed on the target display device.

3. The method according to claim 1, characterized in that Execute corresponding preset alarm measures, including executing at least one of the following measures: Send out sound alarm or light alarm; highlighting an image of a target object on a target display device; Activate the emergency brake system or emergency steering assist.

4. The method according to claim 1, characterized in that Collect image data of the direction in which the car is traveling, and identify the image data to obtain the light scene in which the car is currently located, including: Identify the driving environment of the car, the driving environment includes a daytime driving environment and a nighttime driving environment; In the case where the driving environment is a daytime driving environment, detecting the change in light intensity in the driving direction, and determining whether the light intensity change exceeds a set threshold: if it exceeds the set threshold, determining that the vehicle is in a backlit scene; if it does not exceed the set threshold, determining that the vehicle is not in a backlit scene; When the driving environment is a night driving environment, image data in front of the car is obtained, and it is detected whether the image data has local overexposure. If there is local overexposure, it is determined that the car is in a backlit scene; if there is no local overexposure, it is determined that the car is not in a backlit scene.

5. The method according to claim 1, characterized in that Collect image data of the direction in which the car is traveling, and identify the image data to obtain the light scene in which the car is currently located, including: Detecting whether the vehicle is in a target driving environment, wherein the target driving environment includes a night driving environment; When the target driving environment is a night driving environment, acquiring image data in front of the vehicle; Inputting the image data into a pre-trained backlighting scene classification model, and having the backlighting scene classification model output a light scene in which the vehicle is currently located, wherein the light scene includes a backlighting scene; Alternatively, it is detected whether the image data has a local overexposure phenomenon. If the local overexposure phenomenon exists, it is determined that the car is in a backlit scene; if the local overexposure phenomenon does not exist, it is determined that the car is not in a backlit scene.

6. The method according to any one of claims 1 to 5, characterized in that Determining, according to the motion state of the car and the point cloud data, a three-dimensional scene model of the car's driving direction and whether the car has a collision risk in the backlight scene, including: Generate a three-dimensional scene model according to the point cloud data, wherein the three-dimensional scene model includes the geometric state and motion state of objects around the car; Determine the collision time between the car and surrounding objects in the three-dimensional scene model according to the motion state of the car and the geometric state and motion state of objects around the car; If the collision time is less than the safety threshold, it is determined that the vehicle has a collision risk in the backlighting scenario; If the collision time is greater than or equal to the safety threshold, it is determined that the vehicle does not have a collision risk in the backlighting scenario.

7. The method according to any one of claims 1 to 5, characterized in that Determining, according to the motion state of the car and the point cloud data, a three-dimensional scene model of the car's driving direction and whether the car has a collision risk in the backlight scene, including: Determine the obstacle in front of the vehicle, as well as the geometric state and motion state of the obstacle according to the point cloud data; Determine whether the relative distance between the vehicle and the obstacle is less than a set safety threshold within a preset time period in the future according to the vehicle's motion state and the obstacle's geometric state and motion state; If it is less than the safety distance threshold, it is determined that the car is at risk of collision; If it is greater than or equal to the safety distance threshold, it is determined that the car does not have a collision risk.

8. The method according to claim 7, characterized in that According to the point cloud data, obstacles in front of the car are determined, including: The point cloud data is input into a pre-trained object classification model, and the object classification model outputs a corresponding classification result, wherein the classification result includes obstacles in front of the car.

9. An auxiliary driving device for a car in a backlit scene, characterized in that: include: A scene recognition module is configured to collect image data of the direction in which the vehicle is traveling, and recognize the image data to obtain a light scene in which the vehicle is currently located, wherein the light scene includes a backlight scene; The risk determination module is configured to, when the light environment currently located in the vehicle is a backlit scene, collect point cloud data in front of the vehicle using the on-board laser radar, and determine the three-dimensional scene model of the vehicle's driving direction and whether the vehicle has a collision risk in the backlit scene according to the vehicle's motion state and the point cloud data; The auxiliary driving module is configured to display the image of the target object in the three-dimensional scene model that has the risk of collision with the car on the target display device when it is determined that the car has the risk of collision, and execute corresponding preset alarm measures.

10. An automobile, characterized in that: It includes a camera, a laser radar, a target display device and a control device, the control device includes an intelligent driving domain controller and a cockpit domain controller, the intelligent driving domain controller is connected to the camera and the laser radar respectively, and the cockpit domain controller is connected to the intelligent driving domain controller and the target display device respectively. Among them, the intelligent driving domain controller and the cockpit domain controller each include at least a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any one of claims 1 to 8 when executing the computer program.

Images