Method, device and equipment for sensing environment around vehicle and storage medium

Abstract

The invention provides a method, device and equipment for sensing the environment around a vehicle and a storage medium. The method for sensing the environment around the vehicle comprises the steps that image information around the vehicle is acquired; target feature recognition is carried out on the image information, and point cloud features of all target objects are generated; determining three-dimensional position information of each target feature in a vehicle coordinate system according to the point cloud feature of each target object; and according to the three-dimensional position information of each target feature under the vehicle coordinate system, converting to a top view plane of the vehicle body, and generating environment information around the vehicle at a bird's-eye view angle. The method for sensing the environment around the vehicle is easy to implement, the requirement for the sensor and the hardware requirement for algorithm implementation are low, high-cost schemes such as a high-precision map and cloud computing are not needed, the price is low, and the coverage degree is high.

Description

Technical Field

[0001] The present application relates to the technical field of environmental perception around a vehicle, and in particular to a method, device, equipment and storage medium for environmental perception around a vehicle. Background Art

[0002] Today, expectations for the intelligence of automobile functions are getting higher and higher. Intelligent driving in different scenarios, such as automatic parking and intelligent pick-up in parking lots, and L4 high-level intelligent driving on roads are the main directions for future vehicle development. In order to achieve high-level autonomous driving functions, vehicles first need to rely on sensors to collect surrounding road conditions information in order to provide data support for subsequent downstream algorithms such as regulation and control. If the vehicle itself can achieve intelligent driving functions, then this function must have a judgment ability that is not inferior to that of human drivers to ensure driving safety. Human drivers mainly judge the relationship between the vehicle and surrounding obstacles through vision and hearing. Human drivers make reasonable driving operation decisions by consciously analyzing various driving situations and planning driving behaviors. In order to enable autonomous vehicles to have judgment abilities that are not inferior to human drivers, the perception function of the vehicle should make comprehensive use of various sensor data on the vehicle body as much as possible to maximize the sensitivity and accuracy of the vehicle's ability to perceive information. However, auxiliary devices such as the need to ensure the mutual position relationship of each sensor greatly increase the cost of the multi-sensor system solution. Summary of the invention

[0003] The embodiments of the present application provide a method, device, equipment and storage medium for sensing the environment around a vehicle to solve the problems existing in the related technologies. The technical solutions are as follows:

[0004] In a first aspect, an embodiment of the present application provides a method for sensing an environment around a vehicle, comprising:

[0005] Acquire image information around the vehicle;

[0006] Perform target feature recognition on image information to generate point cloud features of each target object;

[0007] According to the point cloud features of each target object, determine the three-dimensional position information of each target feature in the vehicle coordinate system;

[0008] The three-dimensional position information of each target feature in the vehicle coordinate system is converted to a top-view plane of the vehicle body to generate an environment map of the vehicle surroundings from a bird's-eye view.

[0009] In one embodiment, performing target feature recognition on image information to generate point cloud features of each target object includes:

[0010] The image information is converted and input into a deep convolutional neural network for target feature recognition to determine the feature information of each target;

[0011] According to the information of each target feature, the point cloud features of each target object are generated.

[0012] In one embodiment, generating point cloud features of each target object according to information of each target feature includes:

[0013] The feature information of each target is segmented to generate the point cloud features of each target object. The point cloud features of each target object are the class point cloud results of each target feature in three-dimensional space.

[0014] In one embodiment, determining the three-dimensional position information of each target feature in the vehicle coordinate system according to the point cloud features of each target object includes:

[0015] According to the point cloud features of each target object, a first feature and a second feature are determined, wherein the first feature is a target feature that can be clearly identified and defined, and the first feature is surrounded by a minimum cubic frame box, and the second feature is a target feature that cannot be clearly identified and defined;

[0016] Filling the corresponding cubic frame box surrounding each first feature to obtain the corresponding dense point cloud of each first feature;

[0017] The dense point clouds corresponding to the first features and the point cloud-like results of the second features are detected through the point cloud model to determine the three-dimensional position information of each target feature in the vehicle coordinate system.

[0018] In one embodiment, the method further comprises:

[0019] Determine the confidence level of each target feature according to the three-dimensional position information of each target feature in the vehicle coordinate system;

[0020] The bird's-eye view is divided into N target presence confidence grids in length and width, and the confidence of each target feature is compared with the specified threshold. When the confidence of the target feature is higher than the specified threshold, the confidence of the target feature is marked in the bird's-eye view of the environment around the vehicle.

[0021] In one embodiment, the method further comprises:

[0022] Acquire reference coordinate system data and environmental image data of the top view of the vehicle during driving;

[0023] Searching for a target object within a specified range from the vehicle body according to the environmental image data to obtain target object information;

[0024] According to the target information and the reference coordinate system data, the position of the target in the reference coordinate system is obtained;

[0025] Determine a distance error between the position of the target object in the reference coordinate system and the position of the target object in the environment map around the vehicle from a bird's eye view, based on the position of the target object in the reference coordinate system and the environment map around the vehicle from a bird's eye view;

[0026] When the distance error is greater than the specified distance error, the corresponding number and the position of the target object in the environment map around the vehicle from a bird's-eye view are recorded.

[0027] In one embodiment, obtaining image information around the vehicle includes:

[0028] The surround view camera responds to the synchronization signal to collect images and obtain image information of the vehicle's surroundings with a timestamp. The synchronization signal is used to attach a timestamp and trigger all surround view cameras to expose simultaneously.

[0029] In a second aspect, an embodiment of the present application provides an environment sensing device around a vehicle, comprising:

[0030] A first acquisition module, used to acquire image information around the vehicle;

[0031] The first generation module is used to perform target feature recognition on the image information and generate point cloud features of each target object;

[0032] A first determination module is used to determine the three-dimensional position information of each target feature in the vehicle coordinate system according to the point cloud features of each target object;

[0033] The second generation module is used to convert the three-dimensional position information of each target feature in the vehicle coordinate system into a top view plane of the vehicle body, and generate an environment map around the vehicle from a bird's-eye view.

[0034] In a third aspect, an embodiment of the present application provides an electronic device, the device comprising: a memory and a processor. The memory and the processor communicate with each other through an internal connection path, the memory is used to store instructions, the processor is used to execute the instructions stored in the memory, and when the processor executes the instructions stored in the memory, the processor executes the method in any one of the above-mentioned embodiments.

[0035] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer, the method in any one of the above-mentioned embodiments is executed.

[0036] The advantages or beneficial effects of the above technical solution include at least:

[0037] The technical solution of the disclosed embodiment obtains image information around the vehicle; identifies target features of the image information to generate point cloud features of each target object; determines the three-dimensional position information of each target feature in the vehicle coordinate system according to the point cloud features of each target object; converts the three-dimensional position information of each target feature in the vehicle coordinate system to the top view plane of the vehicle body to generate the environmental information around the vehicle from a bird's-eye view. The environmental perception method around the vehicle of this embodiment is simple to implement, has low requirements for sensors and hardware requirements for algorithm implementation, and does not need to rely on high-precision maps, cloud computing and other high-cost solutions. It is low-priced and has a high degree of coverage. Moreover, the technical solution of this embodiment can achieve high-precision 3D detection of surrounding obstacles at low speeds, and generate road condition signs that are easy for drivers to understand, so that drivers can understand the decision-making judgment of the vehicle, and remind the driver to take over in time to avoid accidents when there are abnormal situations that are prone to danger. A solution that can be operated locally in the vehicle is adopted, so that the vehicle itself can maintain the judgment function of surrounding obstacles in areas with poor communication signals. It can also operate in areas with imperfect communication facilities such as suburbs and wilderness. In addition, through the understanding of the environment based on vision, high-precision positioning of the vehicle can be achieved. By simply modifying the vehicle and the algorithm, achieving a closed data loop can further improve the effectiveness and computational efficiency of the algorithm.

[0038] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present application will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in the present application and should not be regarded as limiting the scope of the present application.

[0040] Figure 1 A flow chart of a method for sensing the environment around a vehicle according to an embodiment of the present application;

[0041] Figure 2 This is a structural block diagram of a method for sensing the environment around a vehicle according to an embodiment of the present application;

[0042] Figure 3 A flow chart of a method for sensing the environment around a vehicle according to another embodiment of the present application;

[0043] Figure 4 A structural block diagram of a vehicle surrounding environment perception device according to an embodiment of the present invention is shown;

[0044] Figure 5 A structural block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0045] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0046] In related technologies, today's expectations for the intelligence of automobile functions are getting higher and higher. Intelligent driving in different scenarios, such as automatic parking and intelligent pick-up in parking lots, and L4 high-level intelligent driving on roads are the main directions of future vehicle development. In order to achieve high-level autonomous driving functions, vehicles first need to rely on sensors to collect surrounding road conditions information in order to provide data support for subsequent downstream algorithms such as regulation and control. If the vehicle itself can achieve intelligent driving functions, then this function must have a judgment ability that is not inferior to that of human drivers to ensure driving safety. Human drivers mainly judge the relationship between the vehicle and surrounding obstacles through vision and hearing. Human drivers make reasonable driving operation decisions by consciously analyzing various driving situations with their brains to plan driving behaviors. In order to enable autonomous vehicles to have judgment abilities that are not inferior to human drivers, the perception function of the vehicle should make comprehensive use of various sensor data on the vehicle body as much as possible to maximize the sensitivity and accuracy of the vehicle's ability to perceive information.

[0047] The main sensors of a vehicle are cameras and radars. Cameras at different positions on the vehicle body can obtain road images of different ranges outside the vehicle body. The information obtained by the camera not only includes the type and location information of obstacles in the environment, but also includes semantic information that determines driving behavior, such as road signs, speed limit signs, lane virtual and real lines, and traffic lights (information that human drivers can obtain from some visual features). In addition, visual semantic information such as the turn signal of the front vehicle, road damage and bumps, water accumulation and ice, weather (day and night, rainy and foggy), and other conditions related to driving behavior should also be perceived to enable the vehicle to respond to different road and environmental conditions and make different driving behavior judgments. For example, when the front vehicle changes lanes or the rear vehicle requests to overtake, the vehicle should cooperate with other vehicles to avoid appropriately. In bad road conditions or rainy days, nights, and other conditions that are not conducive to driving itself and other vehicles, it should slow down and drive slowly, although vehicle sensors can enhance certain effects, such as night vision and fog, through algorithm improvements. However, perception capability not only includes ensuring the driving behavior of the vehicle itself, but also the interaction with surrounding vehicles (traffic participants) must be considered. The improved perception capability of the vehicle under unfavorable driving conditions not only enables the vehicle itself to drive in a safe area, but also provides driving convenience for other vehicles through appropriate driving behavior. This not only ensures the safety of the vehicle itself, but also does not cause hidden dangers to other vehicles.

[0048] ADAS (Advanced Driving Assistance System) developed based on vehicle-mounted cameras can already achieve auxiliary functions such as following the vehicle in front and body surround image. How to reuse the existing software and hardware facilities of the vehicle to continue to expand the vehicle's perception capabilities is also an aspect that the vehicle must consider in order to further realize advanced functions and reduce manufacturing costs. Using the surround view camera installed on the vehicle body, it is possible to obtain vehicle information in front, behind and on both sides of the vehicle at the same time. When driving, when avoiding the vehicle in front, it is necessary not only to perceive the distance of the vehicle in front, but also to perceive whether the rear vehicle will not be able to avoid it in time and cause a dangerous rear-end collision if the vehicle brakes suddenly. In this way, when driving, it is necessary to comprehensively consider as many of the above situations as possible to decide whether the vehicle should follow the vehicle or avoid it. In addition, the vehicle conditions behind and on both sides of the vehicle should also be included in the vehicle's judgment logic at the same time, similar to the actions of the rear vehicle overtaking, and dangerous vehicles such as trucks with untied cargo should be avoided as much as possible. When driving, try to avoid paralleling with other vehicles for a long time or sandwiched between two vehicles. However, cameras also have their disadvantages as sensors. For example, the original image data of wide-angle cameras contains distortion and cannot directly produce image results that are easy for human drivers to understand. Although the image corrected by internal and external parameters can remove the influence of image distortion, the image corrected by internal and external parameters will reduce the field of view, which is not conducive to the maximum perception ability of the sensor. In addition, a single camera cannot directly provide reliable target distance information. Humans can know the distance and position relationship of each object in the vision through the parallax relationship between the two eyes and the processing of the brain. However, for autonomous driving vehicles, the ideal solution is to use only one camera for a specific field of view to obtain complete scene information through technical means, processing and calculation. This not only reduces the requirements for system equipment and thus reduces costs, but also the stability and complexity of the ranging system using a single camera is not as high as that of a system composed of multiple cameras, which is conducive to ensuring the stable and normal operation of the ranging system in various scenarios. Although there are corresponding solutions for depth measurement through binocular or multi-camera cameras, the increase in the number of cameras not only requires matching more complex algorithms to affect the accuracy and speed of system recognition, but also requires auxiliary devices such as ensuring the mutual position relationship of each sensor, which greatly increases the cost of the multi-sensor system solution.

[0049] In addition, human judgment of surrounding road conditions is obtained by comprehensively utilizing all sensory information. For example, at a busy intersection, although small targets such as bicycles and pedestrians do not appear in the field of vision due to the obstruction of surrounding vehicles, human drivers can predict the movement trajectory of these disappeared targets through memory, and thus slow down in advance in places where pedestrians are likely to appear to prevent pedestrians from suddenly appearing in the field of vision and not being able to brake in time, thus causing traffic accidents. When encountering an ambulance or police car with a sirens, you should actively avoid it. These are simple questions of common sense for humans, but they are key issues in how the perception function can further provide vehicles with driving decisions. How to make vehicles reach the same level of understanding as humans requires vehicles to be able to run complex algorithms to achieve this. In high-speed multi-lane scenarios, the above problem can be expanded to the situation where the target may disappear due to obstruction due to the blocked line of sight of peripheral vehicles. When driving at high speed, the driver's decision on vehicle driving should not be to slow down only when danger is detected at the current moment. This may easily lead to rear-end collisions with the following vehicles. It is also required that the vehicle must identify the drivable area in real time during driving to reserve driving operations for surrounding vehicles. Driving decisions should not cause the vehicle to change lanes and speeds frequently during normal driving. When there are vehicles nearby, the vehicle should first send a signal through the vehicle signal lights to determine whether the other party has noticed the vehicle or there is no danger of an accident before changing lanes or overtaking and other driving actions. This places high demands on the vehicle's perception ability when driving at high speeds. On the other hand, consider the situation where the vehicle is driving slowly. In areas where pedestrians may pass by, the vehicle should be able to determine which locations and intersections need to slow down, because the probability of pedestrians appearing at these intersections is relatively high. Although stopping at each gap to wait for confirmation of safety is a solution, the vehicle should also ensure driving speed and ensure operating efficiency to ensure the driving experience. These requirements have led to increasing attention to issues such as how to comprehensively use the vehicle's existing sensors so that the vehicle can judge road conditions as well as human drivers.

[0050] The radars used in vehicles are mainly laser radar and millimeter wave radar. Laser radar is generally installed on the roof of the vehicle, and the vehicle can obtain 360° information about the targets around the vehicle. Laser radar has a long detection range, strong reflection signals, high accuracy, and dense point clouds. It is very suitable for sensing the distance of surrounding objects, and can obtain information such as the movement speed of objects based on the changes in multi-frame point clouds. However, the recognition results of laser radar are not like cameras that can recognize the texture and semantic information of objects such as traffic lights and traffic signs. Therefore, only using laser radar signals for perception cannot cope with the complex road conditions of actual traffic roads. Moreover, laser radar, like cameras, will also have the problem of targets disappearing from the field of view due to occlusion. Compared with laser radar, millimeter wave radar has weak echo signals, fewer point clouds, and a short detection distance. Although a single millimeter wave radar can only detect targets at a certain angle, millimeter wave radars are low in cost and can be installed on the vehicle to increase the angle range of vehicle perception. Since the LiDAR is usually installed on the top of the vehicle, the sensing range of the area directly below the LiDAR (door side) is limited. For example, if the vehicle is driving on the edge of a green belt, if the vegetation in the green belt is lower than the detection range of the LiDAR, the target in the green belt will not be detected. Therefore, a combination of LiDAR and millimeter-wave radar is generally used to achieve the requirement of vehicle distance perception of surrounding target obstacles under various road conditions.

[0051] In addition, information such as inertial sensors and inter-frame changes of video streams can also be provided as auxiliary information to the driving decision-making end. For example, when driving at high speed and low speed, the vehicle's perception distance range should be set to different modes. When the vehicle is driving at high speed, the warning range of the sensor perception should be larger than that at low speed. At low speed, the vehicle should be based on the current driving task, such as automatic parking or driving in complex traffic areas such as mixed vehicles and pedestrians. The vehicle's distance accuracy to surrounding obstacles is required to be more accurate to prevent accidents caused by scratches or collisions. With the complexity of traffic road construction, overpasses, tunnels, underground parking lots and other scenes require vehicles to be able to use their own perception results to review and correct error information that cannot be solved by traditional methods. For example, the accuracy error of position sensors such as GPS may reach several meters when the signal is poor and does not meet the accuracy requirements of vehicle automatic driving. The vehicle can compensate for the poor accuracy of GPS signals to a certain extent by inferring the current position from the surrounding environment through vision; when the road surface is damaged and iced, the vehicle should be able to identify and avoid areas that are not conducive to driving when driving, and update the optimal driving route under the current environment. On the contrary, if the vehicle can calculate its own location information through perception, it will be very helpful for identifying the destination and realizing more complex vehicle driving functions. Whether to use high-precision maps is also an option for vehicle autonomous driving. The benefit of using sensor information data to obtain calculation results in the vehicle's own processor is that the vehicle can be freed from dependence on online high-precision maps. Although the use of high-precision maps can allow vehicles to have more accurate data input on the current vehicle's road conditions and even future vehicle conditions, the cost of maintaining high-precision maps is huge, and the update of road conditions cannot include all situations of vehicle driving, such as accidents ahead, temporary roadblocks on the road, and large foreign objects on the road that are not conducive to vehicle passage (such as fallen trees). Therefore, in the end, the vehicle still needs to have a certain degree of intelligence to achieve autonomous driving under various road conditions. In addition to getting rid of the dependence on pre-defined high-precision maps, if the vehicle is driving in suburban wilderness and other areas, the degree of intelligence required for the vehicle will be higher, because the degree of perfection of the signs in these places is not as good as that in cities, there is no unified standard for the division and judgment of driving areas, and the accuracy of GPS signals and map information is more unreliable than in urban areas. Autonomous driving in these areas requires the vehicle to have perfect perception capabilities, so that the vehicle can perceive and judge the drivable areas in the absence of sign guidance.How to enable the vehicle to obtain human-like capabilities to comprehensively judge the speed of surrounding vehicles and the movement status of the vehicle and other vehicles in the constantly updated current data, and further judge whether to follow the front vehicle or overtake the front vehicle; and in complex road conditions, such as intersections with pedestrians, pedestrians are easily blocked by vehicles and temporarily do not appear in the field of vision around the vehicle, but the vehicle can infer the movement trajectory of pedestrians who are not in the field of vision by adding a memory function, and slow down at the corresponding intersection to avoid collision. This is the key issue that the present invention focuses on.

[0052] Figure 1 A flow chart of a method for sensing the environment around a vehicle according to an embodiment of the present application is shown. Figure 1 and Figure 2 As shown, a method for sensing the environment around a vehicle may include:

[0053] S110: Acquire image information around the vehicle;

[0054] S120: performing target feature recognition on the image information to generate point cloud features of each target object;

[0055] S130: Determine the three-dimensional position information of each target feature in the vehicle coordinate system according to the point cloud features of each target object;

[0056] S140: Convert the three-dimensional position information of each target feature in the vehicle coordinate system to a top view plane of the vehicle body to generate an environment map of the vehicle surroundings from a bird's-eye view.

[0057] In this embodiment, the environmental perception method around the vehicle of this embodiment can be implemented through the cloud and / or the vehicle system. If the cloud service is used, the data can be transmitted to the cloud, and after the cloud server executes the above method, the result is fed back to the vehicle, and then displayed through the vehicle display screen or projection screen, etc., to form a three-dimensional vehicle environment information from a bird's-eye view, so that the driver can adjust the driving in time. The environmental perception method around the vehicle of this embodiment can also be implemented in a mobile terminal, such as a mobile phone, a tablet or a laptop computer. The mobile terminal obtains image information around the vehicle, and can obtain image information around the vehicle from the vehicle by connecting to the vehicle, or by using a map of third-party data, etc., to obtain information around the vehicle.

[0058] The technical solution of the disclosed embodiment obtains image information around the vehicle; identifies target features of the image information to generate point cloud features of each target object; determines the three-dimensional position information of each target feature in the vehicle coordinate system according to the point cloud features of each target object; converts the three-dimensional position information of each target feature in the vehicle coordinate system to the top view plane of the vehicle body to generate the environmental information around the vehicle from a bird's-eye view. The environmental perception method around the vehicle of this embodiment is simple to implement, has low requirements for sensors and hardware requirements for algorithm implementation, and does not need to rely on high-precision maps, cloud computing and other high-cost solutions. It is low-priced and has a high degree of coverage. Moreover, the technical solution of this embodiment can achieve high-precision 3D detection of surrounding obstacles at low speeds, and generate road condition signs that are easy for drivers to understand, so that drivers can understand the decision-making judgment of the vehicle, and remind the driver to take over in time to avoid accidents when there are abnormal situations that are prone to danger. A solution that can be operated locally in the vehicle is adopted, so that the vehicle itself can maintain the judgment function of surrounding obstacles in areas with poor communication signals. It can also operate in areas with imperfect communication facilities such as suburbs and wilderness. In addition, through the understanding of the environment based on vision, high-precision positioning of the vehicle can be achieved. By simply modifying the vehicle and the algorithm, achieving a closed data loop can further improve the effectiveness and computational efficiency of the algorithm.

[0059] In step S110 , image information of the vehicle's surroundings is acquired.

[0060] In this embodiment, the image information of the vehicle's surroundings can be obtained through multiple cameras of the vehicle, and the image information can be obtained frame by frame through the video information collected by the camera. The camera can be a surround view camera of the vehicle.

[0061] Because the on-board surround view camera not only contains a unique field of view for each camera, but also a shared field of view between adjacent cameras. If the images cannot be guaranteed to be taken at the same time, then as the vehicle travels farther and the target moves, even if the cameras detect the same target, they cannot obtain the accurate value of its location during labeling or reasoning, thereby introducing errors in the target distance. In order to improve the accuracy and reliability of the results, the surround view cameras involved in the imaging process can be hard synchronized in time. For example, the vehicle computer transmits a synchronization signal to ensure that the cameras are exposed at the same time. The images collected from each camera are also marked with the time information of the synchronization signal emitted by the vehicle computer, ensuring that the image information around the vehicle is collected at the same time. This ensures the accuracy of the image information around the vehicle.

[0062] In step S120, target feature recognition is performed on the image information to generate point cloud features of each target object.

[0063] In this embodiment, the image information of the environment around the vehicle is input into a deep convolutional neural network for target feature recognition, the target feature information in each camera image is extracted, and the target feature information in each camera image is segmented by the deep neural network, instead of performing depth inference on each pixel to reduce the amount of calculation, thereby obtaining a point cloud-like result for each target feature in three-dimensional space, that is, generating point cloud features for each target object.

[0064] In step S130 , the three-dimensional position information of each target feature in the vehicle coordinate system is determined according to the point cloud features of each target object.

[0065] In this embodiment, the first feature and the second feature are determined according to the point cloud features of each target object, the first feature is a target feature that can be clearly identified and defined, the first feature is surrounded by a minimum cubic frame box, and the second feature is a target feature that cannot be clearly identified and defined; the corresponding cubic frame boxes surrounding each first feature are filled to obtain the corresponding dense point clouds of each first feature; the corresponding dense point clouds of each first feature and the point cloud-like results of each second feature are detected through a point cloud model to generate the point cloud features of each target object.

[0066] Among them, the first feature that can be clearly defined, usually obstacles such as vehicles or roadblocks, etc., which can be directly identified and defined by the deep convolutional neural network, is represented by the smallest cubic box that can surround the first feature. The cubic boxes in the three-dimensional space obtained by converting each first feature are filled with dense point clouds, and the dense point clouds and the point cloud results of the second features that cannot be clearly identified and defined are used together through the detection head of the point cloud model for target detection, such as through pointpillars to perform target detection on dense point clouds and other point cloud results of the second feature element class that cannot be clearly defined at the identified location, and generate point cloud features for each target object. The second feature is usually obtained by reasoning that cannot be directly identified by the deep convolutional neural network.

[0067] The three-dimensional position information of each target feature in the vehicle coordinate system is the three-dimensional position positioning of each target feature in the vehicle coordinate system. The target feature may be an adjacent vehicle or a roadside obstacle, etc.

[0068] In step S140, the three-dimensional position information of each target feature in the vehicle coordinate system is converted to a top view plane of the vehicle body to generate an environment map of the vehicle surroundings from a bird's eye view.

[0069] In this embodiment, the three-dimensional position information of each target feature in the vehicle coordinate system is converted to a two-dimensional top view plane of the vehicle body to generate a two-dimensional bird's-eye view of the vehicle surrounding environment, that is, a top view of the vehicle surrounding environment.

[0070] In order to realize the obstacle detection around the vehicle from the perspective of BEV, the industry has already used multiple sensors in combination, such as the combination of radar and camera, and the comprehensive use of multiple surround cameras. Because radar is more accurate than camera in detecting the distance of the target, it is more advantageous to use the point cloud result of radar for target detection. In the plane image obtained by the camera, it is actually possible to rely on a large amount of feature information such as the grounding point of the target and the road in the image, the overlap of distant and near scenes, the size of the object in the image, and the proportional relationship between the object and the shadow to provide conditions for inferring the distance of the target obstacle. In addition, the camera perspective can know the offset position of the target relative to the optical axis through the internal and external parameters through inverse transformation. If the specific distance of each feature relative to the optical axis can be known, then the pixel coordinates of the camera can be converted to the three-dimensional space of the vehicle coordinate system and the result can be represented by the point cloud. The environment perception method around the vehicle of this embodiment obtains point cloud features similar to radar scanning results by calculating the pixel depth corresponding to each feature in the two-dimensional image obtained by the surround camera, and finally estimates the three-dimensional position of the target feature in the vehicle coordinate system to realize the positioning of the target object (obstacle). Then the obstacle positioning result is unified into the top view plane of the vehicle body through coordinate position conversion. This method only uses cameras and therefore has a cost advantage over radar. In order to achieve real-time computing capabilities on the board, target features are segmented in a neural network and a minimum-volume three-dimensional bounding box is inferred for the features to replace the huge amount of computation required for pixel-by-pixel depth prediction.

[0071] Because the vehicle environment perception method of this embodiment uses a surround camera to identify obstacles in low-speed scenarios, the specific usage scenario limits the obstacles around the driving vehicle to not move at high speeds, causing significant changes in the conditions around the vehicle. The requirements for the recognition accuracy of the results of the vehicle environment perception method of this embodiment are more important than coping with severe disturbances in the external environment for functions such as picking up and parking in low-speed scenarios. The core of the implementation of the vehicle environment perception method of this embodiment is to build a complete and sufficient data set to improve the computing performance of the algorithm, rather than reducing the computing speed by using complex special models with many parameters and steps.

[0072] In some embodiments, it is considered whether the relevant parameters of the vehicle sensor are accurate and reliable, and if the performance parameters of the vehicle sensor change in actual use, the measurement error caused by the change should be kept within the set error tolerance value. First, it is assumed that the experimental vehicle requires the best performance of the sensor in order to produce an environment that is conducive to model calculation to obtain the most accurate results. Therefore, it should be ensured that the camera has been calibrated, the camera has external and internal parameters consistent with the design value, the vehicle has not collided, and there is no performance aging, signal delay and other interference between the electronic circuits. The camera collects images according to the set frame number and resolution (such as 60 frames, 1080p). Since the surround view camera generally uses a large-angle fisheye camera to obtain the maximum field of view, the image of the target object in the field of view will be radially distorted and laterally distorted. After the parameters of the camera have been determined, the perspective transformation of the internal and external parameter coefficients can first generate a bird's-eye view of the vehicle panoramic system, and the target coordinates at a close distance around the vehicle are converted from image coordinates to panoramic coordinates as a reference coordinate system for nearby obstacle targets. Since this reference coordinate system only uses traditional coordinate transformation calculations, it can only obtain the precise coordinates of objects within 1m around the vehicle from the surround image under the premise of a flat ground. As the distance increases and the flatness of the ground deteriorates, the image in the reference coordinate system will also become blurred and stretched, causing the panoramic bird's-eye view to lose its distance reference function in areas far away from the vehicle body.

[0073] The reference coordinate system of the bird's-eye view in the panoramic algorithm and the bird's-eye view representing the result in the inference process of the deep convolutional neural network model are used to search for the distance error of the target recognition result in the reference coordinate system and the environment map around the vehicle generated by the deep convolutional neural network model inference in the bird's-eye view. If the distance error exceeds the designed accuracy error, then all the targets existing in the environment image at that moment and the number of target objects whose distance error exceeds the specified value and the position inference value of the target object are recorded. If the inference value of the target object produces a result that affects the safe driving process of the vehicle, it is necessary to specially mark that the model does not meet the actual usability standard. For example, if the warning range around the vehicle is set to 20cm, if the inferred distance of the target object is outside the warning range, but the same target object is less than 20cm away from the vehicle body in the reference coordinate system, then it is believed that the current perception ability of the deep convolutional neural network cannot guarantee the safe driving of the vehicle, so it cannot be used directly. It is necessary to continue to improve the adaptability of the deep convolutional neural network in various driving scenarios by learning and correcting the parameters of the deep convolutional neural network.

[0074] In some embodiments, for the recognition results of a single camera in the image taken by the surround camera, a more complex model structure is not used to perform unified target recognition. Therefore, in the evaluation process, before filling the cubic frame box of the first feature in the converted three-dimensional space into a dense point cloud, it is also necessary to compare whether there are obvious differences in the feature results recognized by each adjacent camera. If the three-dimensional frame box of the same target in the adjacent camera inferenced by the deep convolutional neural network is obviously two objects in the two-dimensional vehicle top view, since the various parameters of the camera are ensured to be normal, the cause of the problem can be located to be the insufficient reasoning accuracy of the deep convolutional neural network. At this time, the model also needs to be marked as not meeting the actual usability standards because the model has obvious defects in long-distance target recognition. For example, there is a stationary car in front of the left side of the vehicle. The position relationship between the car and the center point of the vehicle is (2m, -45°) inferred by the deep convolutional neural network in the front camera, while in the camera, the position relationship of the car is (1.9m, -30°). If the set angle error is 5° and the distance error is 10%, then although the deep convolutional neural network meets the requirements in terms of distance error, the angle error exceeds the allowable value for the object speculation ability, resulting in the deep convolutional neural network failing to meet the actual usability standards.

[0075] In actual use, it is found that the deep convolutional neural network model often meets the usability standard, but the error exceeds the design value. This is because compared with the training set annotated target information collected manually in advance, there are many factors that affect the result of deep convolutional neural network model reasoning: for example, the weather is sunny or rainy, which causes the reflection of the car body or lane lines and other auxiliary information to be difficult to identify; the lighting is daytime or night, which causes the reflection of the car body, and whether the shadow exists or not, which causes the grounding point of the target to be accurately judged; the road signs are unclear, the road surface is damaged, and the water and ice accumulate, which causes the vehicle to be unable to understand the surrounding road conditions; and there may be roadblocks on the road, load-bearing columns in underground garages, fallen trees, garbage and other foreign objects, which cause the deep convolutional neural network model to be unable to accurately understand the relationship between related obstacles, thereby affecting the recognition accuracy of the results. Therefore, testing the deep convolutional neural network model in the display world is equivalent to verifying the deep convolutional neural network model in an open data set. Finally, the main factor affecting the performance of the deep convolutional neural network model is that the training data can meet the driving conditions of the vehicle as much as possible, and the model should not cause accidents such as collisions when identifying obstacles at close range. When the above conditions are met, the deep convolutional neural network model can improve the true value of the deep convolutional neural network model by transmitting back the data of the scene moment with large error value and manually updating the true value of the scene analysis result. When the vehicle can meet the error exceeding the set value, it can remind the driver to take over the driving. At the same time, under the test results that do not fail to meet the usability standard in the pre-set test driving scene, the performance of the deep convolutional neural network model is evaluated by setting the deep convolutional neural network model evaluation criteria, such as calculating the number of targets with error values ​​exceeding the threshold and the total number of targets in the current scene.

[0076] Before evaluating the deep convolutional neural network model, it is necessary to train the deep convolutional neural network model based on existing data, and the data set used for training needs to ensure the comprehensiveness and completeness of the data. The source of the data can be the surround view data actually collected by the test vehicle, or the data collected through simulation, or the surround view data obtained by perspective conversion, rotation, splicing, etc. of the existing data. In addition, a sound evaluation mechanism needs to be established for data annotation. For example, for pedestrians, they can be classified and labeled according to their gender, age, walking status, walking speed, whether they are avoiding vehicles, etc., and of course, more detailed classification labels can be added later. For vehicles, they can be labeled according to the size, speed, running status, whether the signal lights are flashing, whether they are driving within the lane markings, etc., and more detailed classification labels can be added later according to the actual needs of the model. In particular, for some complex situations, more complex annotation methods can be set. For example, for a trailer truck, the front and cargo box can be labeled separately, and the size, speed, and driving direction of the front and car body can be labeled, and the attributes of the car body can be distinguished according to the goods carried. Through this operation method, the model can increase the recognition ability of some difficult-to-recognize special-shaped cars and modified cars, thereby improving the accuracy of the three-dimensional positioning results generated by different obstacles. In addition, for rain and snow, poor lighting at night, and unclear vision caused by direct rear lights, more samples of edge cases are needed to mark and process the scene. At the same time, it is also possible to consider adding night vision, reducing light intensity, and eliminating rain stains. The post-processing algorithm can enhance the model's ability to handle vehicle driving in extreme scenes. For abnormalities such as deep pits, ice, damage, and garbage on the road, the deep convolutional neural network model can also be improved by adding annotations to improve its understanding of different road conditions. In addition, the sensor's internal parameters, external parameters, vehicle model, sensor's own identification, positioning sensor signal, time signal, timing information of each photo, and the length, width, height, and center point position information of the three-dimensional box of the annotation result can all be added to the annotation file of the dataset as annotation information. In addition, since this dataset is aimed at the specific scenario of identifying surrounding vehicle obstacles in low-speed scenes, key frames are selected for annotation. For example, only 2 frames are extracted for annotation in every 10 frames of images, and the remaining 8 frames are manually annotated based on the results of the 2 frames. The semi-automatic reasoning annotation by the deep convolutional neural network model script can reduce the annotation cost.

[0077] In actual use, the update and maintenance of the data set also need to be considered. On the one hand, road information is constantly developing and changing with the municipal construction. In recent years, underground parking lots in large shopping malls have gradually become popular, making the intelligent pick-up service based on underground parking lots attract attention. The markings, signs, cordons and other markings in underground parking lots are different from the road scenes. In the future, new driving scenes may also emerge with the development of cities and vehicles. In addition, when evaluating the deep convolutional neural network model itself, it will be found that the prediction results and the results of the inverse conversion of the perspective are large. Consider saving the road conditions that are temporarily difficult for the deep convolutional neural network model to handle in actual operation, and then use the latest data to update the existing deep convolutional neural network model to improve the vehicle's perception ability. For example, when a vehicle is driving in a mountainous area, a tunnel is a common scene. The vehicle drives at a low speed in the tunnel according to the marking line for perception. Based on the perception and recognition of the road signs and the lighting, curbs and other facilities in the tunnel, it is found that the detection accuracy error at close range of the vehicle body is large. At this time, the vehicle will collect data on this section of the road and transmit the data back to the data center, and then hand over the data section to manual identification and annotation, and then use the updated data set to train and update the model. At this time, the deep convolutional neural network model has the ability to recognize this tunnel in the mountainous area. Ideally, the difference between other tunnels in the mountainous area and this tunnel is within the generalization ability of the deep convolutional neural network model, so the deep convolutional neural network model can realize the perception of obstacles around the vehicle at low speed in other tunnel scenes. If the target information and driving scene information annotated in the data set are not sufficient to keep the error value within the set safety range, or if there is a situation that affects driving safety, the driver needs to take over the driving of the vehicle immediately and transmit the scene data back to the data center to update the deep convolutional neural network model. From this, we can extend the consideration of adding the data interaction function between the vehicle and the data center on top of the deep convolutional neural network model recognition method of this scheme. On the one hand, the data collected by the vehicle and the deep convolutional neural network model trained by the data center can form a closed loop, and the deep convolutional neural network model can be extensively tested to ensure the perception adaptability of the deep convolutional neural network model in different driving scenarios; on the other hand, targeted data enhancement can also be performed for various driving scenarios. For example, if the driving area has been identified as an underground parking lot, then the recognition of environmental features should be less likely to appear in the high-speed signs of the elevated bridge, the curbs on the high-speed lanes, and large trucks, and more attention should be paid to the detection of pedestrians and family cars, especially whether the surrounding vehicles are already parked or driven and need to be avoided. At this time, the data should be enhanced to reduce the number of image stitching operations for targets that do not appear in the corresponding scene. For example, reduce the number of elevated curbs and road traffic lights, and increase the number of image stitching and simulation results of pedestrians in different states around the vehicle.

[0078] In addition, in order to improve the efficiency of calculation from the perspective of algorithm design, for the input of image information, by pyramid upsampling the surround image according to 1 / 4, 1 / 16, and 1 / 64 to form a group of images of different sizes, the network's perception ability for targets of different scales can be increased. By performing regression analysis on the results, such as using a linear regression network, the convergence speed of the results can be increased and the training efficiency can be improved. The recognition results of the target feature results are not limited to the box, key point connection, and dense / sparse point cloud representation, which can be obtained by simple modification of the embodiment.

[0079] like Figure 2 As shown, in one embodiment, performing target feature recognition on image information to generate point cloud features of each target object includes:

[0080] S210: image information conversion is input into a deep convolutional neural network for target feature recognition to determine feature information of each target;

[0081] S220: Generate point cloud features of each target object based on information of each target feature.

[0082] In this embodiment, the deep convolutional neural network is a trained model. By converting image information and inputting it into the deep convolutional neural network for target feature recognition, the trained deep convolutional neural network recognizes each image information and can determine the target features in each image. The target features can be nearby vehicles or obstacles such as fences around vehicles.

[0083] The target features in the information of each target feature are segmented through a deep convolutional neural network to generate point cloud features of each target object.

[0084] In one embodiment, generating point cloud features of each target object according to information of each target feature includes:

[0085] The feature information of each target is segmented to generate the point cloud features of each target object. The point cloud features of each target object are the class point cloud results of each target feature in three-dimensional space.

[0086] In this embodiment, the target features in the information of each target feature are segmented by a deep convolutional neural network, instead of performing depth inference on each pixel to reduce the amount of calculation, and a point cloud-like result for each feature in three-dimensional space is obtained. This embodiment uses only a camera to obtain the functions of a radar, which has a cost advantage over radar. In order to achieve real-time computing capabilities on the board side, the target features are segmented in the neural network, and the three-dimensional bounding box of the minimum volume is inferred for the features to replace the huge amount of calculation required for depth prediction pixel by pixel. The requirements for sensors and hardware for algorithm implementation are relatively low, and there is no need to rely on high-precision maps, cloud computing and other high-cost solutions. It is low in price and has a high degree of coverage.

[0087] In one embodiment, determining the three-dimensional position information of each target feature in the vehicle coordinate system according to the point cloud features of each target object includes:

[0088] According to the point cloud features of each target object, a first feature and a second feature are determined, wherein the first feature is a target feature that can be clearly identified and defined, and the first feature is surrounded by a minimum cubic frame box, and the second feature is a target feature that cannot be clearly identified and defined;

[0089] Filling the corresponding cubic frame box surrounding each first feature to obtain the corresponding dense point cloud of each first feature;

[0090] The dense point clouds corresponding to the first features and the point cloud-like results of the second features are detected through the point cloud model to determine the three-dimensional position information of each target feature in the vehicle coordinate system.

[0091] In this embodiment, the first feature that can be clearly defined, usually obstacles such as vehicles or roadblocks, etc., which can be directly identified and defined by a deep convolutional neural network, is represented by the smallest cubic box that can surround the first feature, and the cubic boxes in the three-dimensional space obtained by converting each first feature are filled with dense point clouds, and the dense point clouds and the point cloud results of the second features that cannot be clearly identified and defined are used together through the detection head of the point cloud model for target detection, such as through pointpillars to perform target detection on dense point clouds and other point cloud results of the second feature class that cannot be clearly defined at the identified place, and generate point cloud features of each target object. The second feature is usually obtained by reasoning that cannot be directly identified by a deep convolutional neural network. Among them, the second feature is a target feature that cannot be clearly identified and defined, such as the invisible part of an obstacle blocked in the camera's field of view; a 3D position point that is not correctly matched when the pixel feature is converted to a point cloud feature; a smear-like point cloud result introduced by spatial fusion and temporal fusion.

[0092] In this embodiment, by segmenting the target features in a deep convolutional neural network, a cube box in a three-dimensional space with the smallest volume is used as the first feature inference to replace the huge amount of calculation required for depth prediction pixel by pixel, which can save a lot of computing power and reduce the computing power requirements for the vehicle.

[0093] like Figure 3 As shown, in one embodiment, the method further includes:

[0094] S310: Determine the confidence level of each target feature according to the three-dimensional position information of each target feature in the vehicle coordinate system;

[0095] S320: Divide the bird's-eye view into N target presence confidence grids in length and width, compare the confidence of each target feature with a specified threshold, and if the confidence of the target feature is higher than the specified threshold, mark the confidence of the target feature in the bird's-eye view of the environment around the vehicle.

[0096] In this embodiment, the positioning information of the three-dimensional position of each target feature and the confidence corresponding to the positioning information can be determined according to the three-dimensional position information of each target feature in the vehicle coordinate system. For the confidence, when the image information is recognized by a deep convolutional neural network, the confidence of each target feature can be obtained at the same time. The top view of the two-dimensional bird's-eye view space is evenly divided into H*W=N target existence confidence grids in length and width, and the points where the confidence of each target feature is higher than the specified threshold are marked on the vehicle top view to deal with the situation where some targets are suddenly lost in the field of vision due to occlusion.

[0097] In one embodiment, the method further comprises:

[0098] Acquire reference coordinate system data and environmental image data of the top view of the vehicle during driving;

[0099] Searching for a target object within a specified range from the vehicle body according to the environmental image data to obtain target object information;

[0100] According to the target information and the reference coordinate system data, the position of the target in the reference coordinate system is obtained;

[0101] Determine a distance error between the position of the target object in the reference coordinate system and the position of the target object in the environment map around the vehicle from a bird's eye view, based on the position of the target object in the reference coordinate system and the environment map around the vehicle from a bird's eye view;

[0102] When the distance error is greater than the specified distance error, the corresponding number and the position of the target object in the environment map around the vehicle from a bird's-eye view are recorded.

[0103] In this embodiment, the reference coordinate system data and environmental image data of the top view of the vehicle during driving can be obtained through the surround view camera of the vehicle, and the environmental map of the vehicle surrounding the bird's-eye view that represents the result during the reasoning process of the deep convolutional neural network model. According to the environmental image data, the target object within the specified range of the vehicle body is searched to obtain the target object information; according to the target object information and the reference coordinate system data, the position of the target object in the reference coordinate system is obtained; and the distance error between the position of the target object in the reference coordinate system and the position of the target object in the environmental map of the vehicle surrounding the bird's-eye view is determined. If the distance error of the target object exceeds the designed accuracy error, the number of targets and the target position reasoning value of all targets in the environmental image at that moment whose distance errors exceed the specified value are recorded. If the reasoning value of the target object produces a result that affects the safe driving process of the vehicle, it is necessary to specially mark the deep convolutional neural network model as not meeting the actual usability standard. For example, if the warning range around the vehicle is set to 20 cm, if the inferred target distance is outside the warning range but the target is less than 20 cm away from the vehicle body in the reference coordinate system, then it is considered that the current perception ability of the deep convolutional neural network model cannot guarantee the safe driving of the vehicle, so it cannot be used directly. It is necessary to learn and correct the parameters of the deep convolutional neural network model to continue to improve the adaptability of the deep convolutional neural network model in various driving scenarios.

[0104] In one embodiment, obtaining image information around the vehicle includes:

[0105] The surround view camera responds to the synchronization signal to collect images and obtain image information of the vehicle's surroundings with a timestamp. The synchronization signal is used to attach a timestamp and trigger all surround view cameras to expose simultaneously.

[0106] In this embodiment, because the on-board surround view camera not only contains a unique field of view area for each camera, but also a shared field of view area between adjacent cameras, if the images cannot be guaranteed to be taken at the same time in terms of time, then as the distance traveled by the vehicle and the movement of the target object, even if the cameras detect the same target, they cannot obtain the accurate value of its location during labeling or reasoning, thereby introducing errors in the target distance. In order to improve the accuracy and reliability of the results, the surround view cameras involved in the imaging process can be hard synchronized in time. For example, the vehicle computer transmits a synchronization signal to ensure that the cameras are exposed at the same time, and the images collected by each camera are marked with the time information of the synchronization signal emitted by the vehicle computer, ensuring that the image information around the vehicle obtained is data collected at the same time. This ensures the accuracy of the image information around the vehicle.

[0107] Figure 4FIG. 2 shows a structural block diagram of a vehicle environment perception device according to an embodiment of the present invention. Figure 4 As shown, in a second aspect, an embodiment of the present application provides an environment sensing device around a vehicle, which may include:

[0108] A first acquisition module 410 is used to acquire image information around the vehicle;

[0109] The first generating module 420 is used to perform target feature recognition on the image information and generate point cloud features of each target object;

[0110] A first determination module 430 is used to determine the three-dimensional position information of each target feature in the vehicle coordinate system according to the point cloud features of each target object;

[0111] The second generating module 440 is used to convert the three-dimensional position information of each target feature in the vehicle coordinate system into a top view plane of the vehicle body, and generate an environment map of the vehicle surroundings from a bird's-eye view.

[0112] The technical solution of the disclosed embodiment obtains image information around the vehicle; identifies target features of the image information to generate point cloud features of each target object; determines the three-dimensional position information of each target feature in the vehicle coordinate system according to the point cloud features of each target object; converts the three-dimensional position information of each target feature in the vehicle coordinate system to the top view plane of the vehicle body to generate the environmental information around the vehicle from a bird's-eye view. The environmental perception method around the vehicle of this embodiment is simple to implement, has low requirements for sensors and hardware requirements for algorithm implementation, and does not need to rely on high-precision maps, cloud computing and other high-cost solutions. It is low-priced and has a high degree of coverage. Moreover, the technical solution of this embodiment can achieve high-precision 3D detection of surrounding obstacles at low speeds, and generate road condition signs that are easy for drivers to understand, so that drivers can understand the decision-making judgment of the vehicle, and remind the driver to take over in time to avoid accidents when there are abnormal situations that are prone to danger. A solution that can be operated locally in the vehicle is adopted, so that the vehicle itself can maintain the judgment function of surrounding obstacles in areas with poor communication signals. It can also operate in areas with imperfect communication facilities such as suburbs and wilderness. In addition, through the understanding of the environment based on vision, high-precision positioning of the vehicle can be achieved. By simply modifying the vehicle and the algorithm, achieving a closed data loop can further improve the effectiveness and computational efficiency of the algorithm.

[0113] In one embodiment, performing target feature recognition on image information to generate point cloud features of each target object includes:

[0114] The image information is converted and input into a deep convolutional neural network for target feature recognition to determine the feature information of each target;

[0115] According to the information of each target feature, the point cloud features of each target object are generated.

[0116] In one embodiment, generating point cloud features of each target object according to information of each target feature includes:

[0117] The feature information of each target is segmented to generate the point cloud features of each target object. The point cloud features of each target object are the class point cloud results of each target feature in three-dimensional space.

[0118] In one embodiment, determining the three-dimensional position information of each target feature in the vehicle coordinate system according to the point cloud features of each target object includes:

[0119] According to the point cloud features of each target object, a first feature and a second feature are determined, wherein the first feature is a target feature that can be clearly identified and defined, and the first feature is surrounded by a minimum cubic frame box, and the second feature is a target feature that cannot be clearly identified and defined;

[0120] Filling the corresponding cubic frame box surrounding each first feature to obtain the corresponding dense point cloud of each first feature;

[0121] The dense point clouds corresponding to the first features and the point cloud-like results of the second features are detected through the point cloud model to determine the three-dimensional position information of each target feature in the vehicle coordinate system.

[0122] In one embodiment, the method further comprises:

[0123] Determine the confidence level of each target feature according to the three-dimensional position information of each target feature in the vehicle coordinate system;

[0124] The bird's-eye view is divided into N target presence confidence grids in length and width, and the confidence of each target feature is compared with the specified threshold. When the confidence of the target feature is higher than the specified threshold, the confidence of the target feature is marked in the bird's-eye view of the environment around the vehicle.

[0125] In one embodiment, the method further comprises:

[0126] Acquire reference coordinate system data and environmental image data of the top view of the vehicle during driving;

[0127] Searching for a target object within a specified range from the vehicle body according to the environmental image data to obtain target object information;

[0128] According to the target information and the reference coordinate system data, the position of the target in the reference coordinate system is obtained;

[0129] Determine a distance error between the position of the target object in the reference coordinate system and the position of the target object in the environment map around the vehicle from a bird's eye view, based on the position of the target object in the reference coordinate system and the environment map around the vehicle from a bird's eye view;

[0130] When the distance error is greater than the specified distance error, the corresponding number and the position of the target object in the environment map around the vehicle from a bird's-eye view are recorded.

[0131] In one embodiment, obtaining image information around the vehicle includes:

[0132] The surround view camera responds to the synchronization signal to collect images and obtain image information of the vehicle's surroundings with a timestamp. The synchronization signal is used to attach a timestamp and trigger all surround view cameras to expose simultaneously.

[0133] The functions of the modules in the devices in the embodiments of the present invention can be found in the corresponding descriptions of the above methods, which will not be described in detail here.

[0134] Figure 5 FIG. 2 shows a structural block diagram of an electronic device according to an embodiment of the present invention. Figure 5 As shown, the electronic device includes: a memory 910 and a processor 920, wherein the memory 910 stores a computer program that can be run on the processor 920. When the processor 920 executes the computer program, the method for sensing the environment around the vehicle in the above embodiment is implemented. The number of the memory 910 and the processor 920 can be one or more.

[0135] The electronic device / terminal / server also includes:

[0136] The communication interface 930 is used to communicate with external devices and perform data exchange transmission.

[0137] If the memory 910, the processor 920 and the communication interface 930 are implemented independently, the memory 910, the processor 920 and the communication interface 930 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0138] Optionally, in a specific implementation, if the memory 910, the processor 920 and the communication interface 930 are integrated on a chip, the memory 910, the processor 920 and the communication interface 930 can communicate with each other through an internal interface.

[0139] An embodiment of the present invention provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.

[0140] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0141] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0142] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processing (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the advanced RISC machines (ARM) architecture.

[0143] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct rambus RAM (DR RAM).

[0144] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0145] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0146] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0147] Any process or method description in the flow chart or otherwise described herein can be understood to represent a module, fragment or portion of a code including one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0148] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0149] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0150] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0151] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for sensing the environment around a vehicle, characterized in that: include: Acquire image information around the vehicle; Performing target feature recognition on the image information to generate point cloud features of each target object; According to the point cloud features of each target object, determine the three-dimensional position information of each target feature in the vehicle coordinate system; The three-dimensional position information of each target feature in the vehicle coordinate system is converted to a top-view plane of the vehicle body to generate an environment map of the vehicle surroundings from a bird's-eye view.

2. The method for sensing the environment around a vehicle according to claim 1, characterized in that: The performing target feature recognition on the image information to generate point cloud features of each target object comprises: The image information is converted and input into a deep convolutional neural network for target feature recognition to determine the feature information of each target; According to the information of each target feature, the point cloud features of each target object are generated.

3. The method for sensing the environment around a vehicle according to claim 2, characterized in that: Generating point cloud features of each target object according to information of each target feature includes: The feature information of each target is segmented to generate a point cloud feature of each target object, where the point cloud feature of each target object is a point cloud-like result of each target feature in three-dimensional space.

4. The method for sensing the environment around a vehicle according to claim 2, characterized in that: Determining the three-dimensional position information of each target feature in the vehicle coordinate system according to the point cloud features of each target object includes: Determine a first feature and a second feature according to the point cloud features of each target object, wherein the first feature is a target feature that can be clearly identified and defined, and the first feature is surrounded by a minimum cubic frame box, and the second feature is a target feature that cannot be clearly identified and defined; Filling the corresponding cubic frame box surrounding each first feature to obtain the corresponding dense point cloud of each first feature; The dense point clouds corresponding to the first features and the point cloud-like results of the second features are detected through the point cloud model to determine the three-dimensional position information of each target feature in the vehicle coordinate system.

5. The method for sensing the environment around a vehicle according to claim 1, characterized in that: The method further comprises: Determine the confidence level of each target feature according to the three-dimensional position information of each target feature in the vehicle coordinate system; The bird's-eye view is divided into N target presence confidence grids in length and width, and the confidence of each target feature is compared with the specified threshold. When the confidence of the target feature is higher than the specified threshold, the confidence of the target feature is marked in the bird's-eye view of the environment around the vehicle.

6. The method for sensing the environment around a vehicle according to claim 1, characterized in that: The method further comprises: Acquire reference coordinate system data and environmental image data of the top view of the vehicle during driving; Searching for a target object within a specified range from the vehicle body according to the environmental image data to obtain target object information; Obtaining the position of the target object in the reference coordinate system according to the target object information and the reference coordinate system data; Determining a distance error between the position of the target object in the reference coordinate system and the position of the target object in the environment map around the vehicle from a bird's-eye view according to the position of the target object in the reference coordinate system and the environment map around the vehicle from a bird's-eye view; When the distance error is greater than the specified distance error, the corresponding number and the position of the target object in the environment map around the vehicle from a bird's-eye view are recorded.

7. The method for sensing the environment around a vehicle according to claim 1, characterized in that: The obtaining of image information around the vehicle includes: The surround view camera responds to the synchronization signal to collect images and obtain image information of the vehicle's surroundings with a timestamp. The synchronization signal is used to attach a timestamp and trigger all surround view cameras to expose simultaneously.

8. A device for sensing the environment around a vehicle, characterized in that: include: A first acquisition module, used to acquire image information around the vehicle; A first generating module is used to perform target feature recognition on the image information to generate point cloud features of each target object; A first determination module is used to determine the three-dimensional position information of each target feature in the vehicle coordinate system according to the point cloud features of each target object; The second generation module is used to convert the three-dimensional position information of each target feature in the vehicle coordinate system into a top view plane of the vehicle body, and generate an environment map around the vehicle from a bird's-eye view.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images