Travelable area detection method and device and storage medium

Abstract

The invention relates to a drivable area detection method and device and a storage medium, and the method comprises the steps: building an initial grid map at an initial moment, and enabling the grid map to comprise a drivable area and a non-drivable area; and at each moment after the initial moment, dynamically updating the grid map by using the vehicle environment image or using the obstacle distance information to obtain the grid map at the current moment. If the grid map is dynamically updated by using the vehicle environment image at the previous moment, dynamically updating the grid map at the previous moment by using the obstacle distance information at the current moment; and if the grid map is dynamically updated by using the obstacle distance information at the previous moment, dynamically updating the grid map at the previous moment by using the vehicle environment image at the current moment. According to the invention, the method can effectively reduce the shielding problem, also has the advantage of low cost, can obtain the accurate boundary information of the driving region, and can carry out the modeling of a dynamic obstacle.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a drivable area detection method and device, and a storage medium. Background Art

[0002] An excellent autonomous driving system helps reduce labor costs, improve travel safety, and reduce accident rates. Drivable area detection can help the autonomous driving system perceive the surrounding environment, thereby completing functions such as obstacle avoidance and path planning. It is an indispensable part of the autonomous driving system.

[0003] The current drivable area detection methods mainly include pure vision-based methods, lidar-based methods, and ultrasonic sensor-based methods. Most of the above methods use a single sensor and ignore historical information. The pure vision method cannot handle occlusion situations, the lidar method is costly, and the ultrasonic sensor method cannot obtain accurate drivable area boundary information. Moreover, the above methods are only applicable to static scenes and cannot model dynamic obstacles. Therefore, the current drivable area detection methods need to be improved to better balance detection effect and cost. Summary of the invention

[0004] The purpose of this application is to propose a drivable area detection method and its device, storage medium, which can effectively reduce the occlusion problem while having the advantage of low cost, can obtain accurate drivable area boundary information, and can model dynamic obstacles.

[0005] To achieve the above object, an embodiment of the present application provides a drivable area detection method, the method comprising:

[0006] Establishing an initial grid map at an initial moment, wherein the grid map includes a drivable area and a non-drivable area;

[0007] At each moment after the initial moment, the grid map is dynamically updated using the vehicle environment image collected by the vehicle camera or the obstacle distance information collected by the vehicle ultrasonic sensor to obtain the grid map of the current moment;

[0008] Among them, if the grid map was dynamically updated using the vehicle environment image collected by the vehicle camera at the previous moment, the grid map of the previous moment is dynamically updated using the obstacle distance information collected by the vehicle ultrasonic sensor at the current moment; if the grid map was dynamically updated using the obstacle distance information collected by the vehicle ultrasonic sensor at the previous moment, the grid map of the previous moment is dynamically updated using the vehicle environment image collected by the vehicle camera at the current moment.

[0009] In some solutions, the grid map is dynamically updated at each moment after the initial moment to obtain the grid map at the current moment, specifically including:

[0010] When determining to dynamically update the grid map of the previous moment using the vehicle environment image collected by the vehicle camera, if the vehicle environment image of the current moment cannot be obtained, the obstacle distance information of the current moment is obtained, and the grid map of the previous moment is dynamically updated according to the obstacle distance information of the current moment;

[0011] When determining to dynamically update the grid map of the previous moment using the obstacle distance information at the current moment collected by the vehicle ultrasonic sensor, if the obstacle distance information at the current moment cannot be obtained, the vehicle environment image at the current moment is obtained, and the grid map of the previous moment is dynamically updated based on the vehicle environment image at the current moment.

[0012] In the following scheme, the vehicle environment image collected by the vehicle camera is used to update the grid map at the previous moment, including:

[0013] Obtaining a vehicle environment image at the current moment, obtaining a visual drivable area probability map according to the vehicle environment image at the current moment, converting the visual drivable area probability map to a world coordinate system and rasterizing it to obtain a visual grid map and a visual mask at the current moment, and updating a grid map at the previous moment according to the visual grid map and the visual mask at the current moment;

[0014] The obstacle distance information collected by the vehicle ultrasonic sensor is used to update the grid map at the previous moment, including:

[0015] Obtain obstacle distance information at the current moment, obtain an ultrasonic drivable area probability map based on the obstacle distance information at the current moment, convert the ultrasonic drivable area probability map to the world coordinate system and rasterize it to obtain the ultrasonic grid map and ultrasonic mask at the current moment, and update the grid map at the previous moment based on the ultrasonic grid map and ultrasonic mask at the current moment.

[0016] In the following scheme, the visual grid map includes a plurality of grids and a probability value of each grid being a drivable area, and the visual mask includes a plurality of grids and a feature value of whether each grid is detected by a vehicle camera;

[0017] The updating of the grid map at the previous moment according to the visual grid map and the visual mask at the current moment specifically includes:

[0018] Determine the visual grid map area detected by the vehicle camera at the current moment according to the visual mask;

[0019] Calculating the likelihood probability and the forward probability of each grid according to the hidden Markov model and the probability value of each grid in the visual grid map area being a drivable area;

[0020] Calculating the occupied probability and unoccupied probability of each grid according to the likelihood probability and forward probability of each grid;

[0021] Determine whether each grid is a passable area according to the occupied probability and unoccupied probability of each grid, and obtain a partial grid map corresponding to the visual grid map area;

[0022] The partial grid map is merged with the grid map at the previous moment to obtain the grid map at the current moment.

[0023] In the following scheme, the ultrasonic grid map includes a plurality of grids and a probability value of each grid being a drivable area, and the ultrasonic mask includes a plurality of grids and a characteristic value of whether each grid is detected by a vehicle ultrasonic sensor;

[0024] The updating of the grid map at the previous moment according to the ultrasonic grid map and the ultrasonic mask at the current moment specifically includes:

[0025] Determine the ultrasonic grid map area detected by the vehicle ultrasonic sensor at the current moment according to the ultrasonic mask;

[0026] Calculate the likelihood probability and forward probability of each grid according to the hidden Markov model and the probability value of each grid in the ultrasonic grid map area being a drivable area;

[0027] Calculating the occupied probability and unoccupied probability of each grid according to the likelihood probability and forward probability of each grid;

[0028] Determine whether each grid is a passable area according to the occupied probability and unoccupied probability of each grid, and obtain a partial grid map corresponding to the ultrasonic grid map area;

[0029] The partial grid map is merged with the grid map at the previous moment to obtain the grid map at the current moment.

[0030] The present application also provides a drivable area detection device, the device comprising:

[0031] An initial mapping module, used to establish an initial grid map at an initial moment, wherein the grid map includes a drivable area and a non-drivable area;

[0032] A dynamic update module is used to dynamically update the grid map at each moment after the initial moment using the vehicle environment image collected by the vehicle camera or the obstacle distance information collected by the vehicle ultrasonic sensor to obtain the grid map at the current moment; wherein, if the grid map was dynamically updated using the vehicle environment image collected by the vehicle camera at the previous moment, then the obstacle distance information collected by the vehicle ultrasonic sensor is used to dynamically update the grid map at the previous moment at the current moment; if the grid map was dynamically updated using the obstacle distance information collected by the vehicle ultrasonic sensor at the previous moment, then the vehicle environment image collected by the vehicle camera is used to dynamically update the grid map at the previous moment at the current moment.

[0033] In the following scheme, the dynamic update module includes:

[0034] A visual update submodule is used to obtain a vehicle environment image at the current moment, obtain a visual drivable area probability map based on the vehicle environment image at the current moment, convert the visual drivable area probability map into a world coordinate system and rasterize it to obtain a visual grid map and a visual mask at the current moment, and update a grid map at the previous moment based on the visual grid map and the visual mask at the current moment;

[0035] The ultrasonic update submodule is used to obtain the obstacle distance information at the current moment, obtain the ultrasonic drivable area probability map based on the obstacle distance information at the current moment, convert the ultrasonic drivable area probability map to the world coordinate system and rasterize it to obtain the ultrasonic grid map and ultrasonic mask at the current moment, and update the grid map at the previous moment based on the ultrasonic grid map and ultrasonic mask at the current moment.

[0036] In the following scheme, the visual grid map includes a plurality of grids and a probability value of each grid being a drivable area, and the visual mask includes a plurality of grids and a feature value of whether each grid is detected by a vehicle camera;

[0037] Wherein, the visual update submodule is specifically used for:

[0038] Determine the visual grid map area detected by the vehicle camera at the current moment according to the visual mask;

[0039] Calculating the likelihood probability and the forward probability of each grid according to the hidden Markov model and the probability value of each grid in the visual grid map area being a drivable area;

[0040] Calculating the occupied probability and unoccupied probability of each grid according to the likelihood probability and forward probability of each grid;

[0041] Determine whether each grid is a passable area according to the occupied probability and unoccupied probability of each grid, and obtain a partial grid map corresponding to the visual grid map area;

[0042] The partial grid map is merged with the grid map at the previous moment to obtain the grid map at the current moment.

[0043] In the following scheme, the ultrasonic grid map includes a plurality of grids and a probability value of each grid being a drivable area, and the ultrasonic mask includes a plurality of grids and a characteristic value of whether each grid is detected by a vehicle ultrasonic sensor;

[0044] Wherein, the ultrasonic updating submodule is specifically used for:

[0045] Determine the ultrasonic grid map area detected by the vehicle ultrasonic sensor at the current moment according to the ultrasonic mask;

[0046] Calculate the likelihood probability and forward probability of each grid according to the hidden Markov model and the probability value of each grid in the ultrasonic grid map area being a drivable area;

[0047] Calculating the occupied probability and unoccupied probability of each grid according to the likelihood probability and forward probability of each grid;

[0048] Determine whether each grid is a passable area according to the occupied probability and unoccupied probability of each grid, and obtain a partial grid map corresponding to the ultrasonic grid map area;

[0049] The partial grid map is merged with the grid map at the previous moment to obtain the grid map at the current moment.

[0050] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the drivable area detection method as described above is implemented.

[0051] The embodiment of the present application provides a drivable area detection method and its device, and storage medium. An initial grid map is established at an initial moment, and the grid map includes a drivable area and a non-drivable area. At each moment after the initial moment, the vehicle environment image collected by the vehicle camera and the obstacle distance information collected by the vehicle ultrasonic sensor are alternately used to dynamically update the grid map to obtain the grid map of the current moment, so as to achieve modeling of dynamic obstacles. Among them, the dynamic update of the grid map using the vehicle environment image collected by the vehicle camera can obtain accurate drivable area boundary information, and the dynamic update of the grid map using the obstacle distance information collected by the vehicle ultrasonic sensor can effectively reduce the occlusion problem. Among them, since the surrounding environment of the vehicle does not change much between two adjacent moments, in order to reduce the requirements for computing resources and adapt to different sensor frequencies, the embodiment of the present application only uses one type of sensor information (vehicle environment image or obstacle distance information) to dynamically update the grid map at each moment, which has high computing efficiency and meets the high real-time requirements of autonomous driving. It can not only ensure the detection effect of the drivable area, but also will not affect the realization of other vehicle functions. Moreover, the vehicle camera and the vehicle ultrasonic sensor are the original configurations of most vehicle models, so there is no additional cost. Compared with laser radar, the cost is relatively low.

[0052] Details and advantages not described in detail in the embodiments of the present application are described in detail in the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 This is a flow chart of a drivable area detection method in one embodiment of the present application.

[0055] Figure 2 This is a schematic diagram of an ultrasonic sensor probability map in one embodiment of the present application.

[0056] Figure 3 The present invention is a flowchart of a drivable area detection method in a specific embodiment of the present application.

[0057] Figure 4 This is a rendering of the drivable area detection result in a specific embodiment of the present application.

[0058] Figure 5 This is a framework diagram of a drivable area detection device in one embodiment of the present application. DETAILED DESCRIPTION

[0059] The detailed description of the drawings is intended as an illustration of the current embodiment of the present application, and is not intended to represent the only form in which the present application can be implemented. It should be understood that the same or equivalent functions can be accomplished by different embodiments intended to be included in the spirit and scope of the present application.

[0060] See also Figure 1 An embodiment of the present application provides a drivable area detection method, comprising the following steps:

[0061] Step S10, establishing an initial grid map at an initial moment, wherein the grid map includes a drivable area and a non-drivable area.

[0062] Specifically, the range of the front and rear [x1, x2] and the left and right [y1, y2] of the vehicle is selected, and the dynamic grid map Map(T0) is established with a resolution of res. The default parameter values ​​are: x1 = -50m, x2 = 50m, y1 = -50m, y2 = 50m, res = 0.05m; Step S10 considers two states of the grid: occupied and unoccupied, and initializes the dynamic grid map Map(T0), wherein Map(T0) contains two channels: channel C0 and channel C1, which are expressed as Map(T0) C0 and Map(T0) C1 , represent the probabilities of being occupied and unoccupied, respectively.

[0063] Step S20, at each moment after the initial moment, dynamically updating the grid map using the vehicle environment image collected by the vehicle camera or the obstacle distance information collected by the vehicle ultrasonic sensor to obtain the grid map of the current moment;

[0064] Among them, if the grid map was dynamically updated using the vehicle environment image collected by the vehicle camera at the previous moment, the grid map of the previous moment is dynamically updated using the obstacle distance information collected by the vehicle ultrasonic sensor at the current moment; if the grid map was dynamically updated using the obstacle distance information collected by the vehicle ultrasonic sensor at the previous moment, the grid map of the previous moment is dynamically updated using the vehicle environment image collected by the vehicle camera at the current moment.

[0065] The method of this embodiment establishes an initial grid map at the initial moment, and then at each moment, the vehicle environment image collected by the vehicle camera and the obstacle distance information collected by the vehicle ultrasonic sensor are alternately used to dynamically update the grid map to obtain the grid map of the current moment, so as to achieve modeling of dynamic obstacles; wherein, dynamically updating the grid map using the vehicle environment image collected by the vehicle camera can obtain accurate boundary information of the drivable area, and dynamically updating the grid map using the obstacle distance information collected by the vehicle ultrasonic sensor can effectively reduce the occlusion problem; it should be noted that since the environment around the vehicle does not change much between two adjacent moments, in order to reduce the requirements for computing resources and adapt to different sensor frequencies, this embodiment only uses one type of sensor information (vehicle environment image or obstacle distance information) to dynamically update the grid map at each moment, which has high computing efficiency and meets the high real-time requirements of autonomous driving, and can ensure the detection effect of the drivable area without affecting the realization of other vehicle functions, and the vehicle camera and the vehicle ultrasonic sensor are the original configurations of most vehicle models, so there is no additional cost, and the cost is relatively low compared to the lidar.

[0066] In the following embodiments, step S20 specifically includes:

[0067] When determining to dynamically update the grid map of the previous moment using the vehicle environment image collected by the vehicle camera, if the vehicle environment image of the current moment cannot be obtained, the obstacle distance information of the current moment is obtained, and the grid map of the previous moment is dynamically updated according to the obstacle distance information of the current moment;

[0068] When determining to dynamically update the grid map of the previous moment using the obstacle distance information at the current moment collected by the vehicle ultrasonic sensor, if the obstacle distance information at the current moment cannot be obtained, the vehicle environment image at the current moment is obtained, and the grid map of the previous moment is dynamically updated based on the vehicle environment image at the current moment.

[0069] Specifically, this embodiment provides two grid map update methods, namely, using the vehicle environment image collected by the vehicle camera to dynamically update the grid map of the previous moment, and using the obstacle distance information at the current moment collected by the vehicle ultrasonic sensor to dynamically update the grid map of the previous moment. The two methods are used alternately and also serve as backup for each other to avoid the situation where one of the sensor information cannot be obtained, so as to continue to ensure the detection and update of the drivable area, thereby improving the safety and reliability of autonomous driving.

[0070] In the following embodiments, the step S20 uses the vehicle environment image collected by the vehicle camera to update the grid map at the previous moment, including:

[0071] The vehicle environment image at the current moment is obtained, and a visual drivable area probability map is obtained according to the vehicle environment image at the current moment. The visual drivable area probability map is converted to the world coordinate system and rasterized to obtain the visual grid map and visual mask at the current moment, and the grid map at the previous moment is updated according to the visual grid map and visual mask at the current moment.

[0072] Specifically, the vehicle is equipped with cameras in the front, rear, left, and right directions. The vehicle environment images collected by multiple cameras are processed using semantic segmentation algorithms to obtain the corresponding visual drivable area probability map, which is projected into the vehicle coordinate system for splicing and then converted to the world coordinate system to obtain a 360-degree visual probability map. The probability value of each grid in the visual probability map is expressed as p_cam(T i ), T i represents the i-th moment, where the semantic segmentation algorithm used is, for example, DDRNet, where the grid p_cam(T i ) is closer to 1, the grid is more likely to be a drivable area. The grid where the vehicle is located is a drivable area by default, and its p_cam(T i ) is p_ego, the default value is 0.95;

[0073] Each grid of the visual mask can be represented by Mask_Cam(T i ) indicates that according to the grid at T i Whether the grid is detected by the vehicle camera at the moment is 0 or 1, where 0 indicates that the grid is not detected and 1 indicates that the grid is detected.

[0074] Among them, T i Each grid in the moment visual grid map can be mapped using Map_Cam(T i ) to represent, including Map_Cam(T i ) C0 and Map_Cam(T corresponding to C1 channel i ) C1 , can be calculated by formula (1)~(2);

[0075] Map_Cam(T i ) C0 = 1-p_cam(T i ) (1)

[0076] Map_Cam(T i ) C1 = p_cam(T i ) (2)

[0077] It can be understood that according to the visual grid map and visual mask at the current moment, the obstacle information of the latest detected area can be obtained, and then the drivable area and non-formable area of ​​the latest detected area can be obtained, and merged with the grid map at the previous moment to update the grid map at the previous moment; it should be noted that the visual grid map and the visual mask are the same size, and the visual mask can be used to shield the undetected range, so that the dynamic update of the grid map only focuses on the currently detected range.

[0078] The step S20 uses the vehicle environment image collected by the vehicle ultrasonic sensor to update the grid map at the previous moment, including:

[0079] Obtain obstacle distance information at the current moment, obtain an ultrasonic drivable area probability map based on the obstacle distance information at the current moment, convert the ultrasonic drivable area probability map to the world coordinate system and rasterize it to obtain the ultrasonic grid map and ultrasonic mask at the current moment, and update the grid map at the previous moment based on the ultrasonic grid map and ultrasonic mask at the current moment.

[0080] Specifically, in T i+1 moment, create a T around the body i+1 At time t, the ultrasonic drivable area probability map and ultrasonic mask are initialized, where the probability value of each grid in the ultrasonic drivable area probability map being a drivable area can be expressed as p_uss(T i+1 ), the characteristic value of each grid in the ultrasonic mask is expressed by Mask_USS(T i+1 ) is used to represent p_uss(T i+1 ) is initialized with a probability of 0.5, Mask_USS(T i+1 ) is initialized to 0. i+1 ), with the center of each ultrasonic sensor as the center and the detection distance as the radius, an arc with a width of D is created within its detection angle, where the center and the arc are considered to be a drivable area with a probability of p1_uss, the arc is considered to be a non-drivable area with a probability of p0_uss, and the area outside the arc is considered to be an unknown area with a probability value that remains unchanged; at the detected position, Mask_USS(T i+1 ) is 1, and the default values ​​of the parameters are: D=5, p1_uss=0.9, p0_uss=0.1. If a position on the probability map is scanned by multiple ultrasonic sensors, its probability is the average of the multiple probabilities. Among them, the vehicle's own position is the drivable area by default, and the probability value is p_ego, with a default value of 0.95.

[0081] Each grid of the ultrasonic mask can be represented by Mask_USS(T i+1 ) indicates that according to the grid at Ti+1 Whether the grid is detected by the vehicle ultrasonic sensor at a certain moment is 0 or 1, where 0 indicates that the grid is not detected and 1 indicates that the grid is detected.

[0082] Among them, T i+1 Each grid in the moment visual grid map can be mapped using Map_USS(T i+1 ) to indicate that it includes Map_USS (T i+1 ) C0 and Map_USS(T i+1 ) C1 , can be calculated by formula (3)~(4);

[0083] Ultrasonic grid map Map_USS(T i+1 ) can be calculated using equations (3) to (4).

[0084] Map_USS(T i+1 ) C0 = 1-p_uss(T i+1 ) (3)

[0085] Map_USS(T i+1 ) C1 = p_uss(T i+1 ) (4)

[0086] It can be understood that according to the ultrasonic grid map and ultrasonic mask at the current moment, the obstacle information of the latest detected area can be obtained, and then the drivable area and non-drivable area of ​​the latest detected area can be obtained, and then merged with the grid map at the previous moment to update the grid map at the previous moment; it should be noted that the ultrasonic grid map and the ultrasonic mask are the same size, and the ultrasonic mask can be used to shield the undetected range, so that the dynamic update of the grid map only focuses on the currently detected range.

[0087] In the following embodiments, the visual grid map includes a plurality of grids and a probability value p_cam (T i ), the visual mask includes a plurality of grids and a characteristic value Mask_Cam (T i ), eigenvalue Mask_Cam(T i ) is 0 or 1;

[0088] The updating of the grid map at the previous moment according to the visual grid map and the visual mask at the current moment specifically includes:

[0089] Step S211, determining the visual grid map area detected by the vehicle camera at the current moment according to the visual mask.

[0090] Step S212, calculating the likelihood probability and forward probability of each grid according to the hidden Markov model and the probability value that each grid in the visual grid map area is a drivable area;

[0091] Specifically, the hidden Markov model includes 4 parameters, initial probability start_prob, state transfer matrix trans_mat, state mean mean and state variance covar. In this embodiment, their initial values ​​are: start_prob = [0.5, 0.5], trans_mat = [[0.9, 0.1], [0.2, 0.8]], mean = [0.9, 0.1], covar = [0.1, 0.1]; the initialization of the hidden Markov model is performed in step S10;

[0092] This embodiment calculates the likelihood probability L_cam(T i );

[0093] L_cam(T i ) Cj =-0.5*((p_cam(T i ) Cj -mean[j]) / covar[j] 0.5 )+log(2π)+2log(covar[j] 0.5 ) (5)

[0095] Wherein, j = 0 or 1;

[0096] This embodiment calculates the forward probability F_cam(T i );

[0097] F_cam(T i ) Cj =logΣ k (exp(Map(T i-1 ) Ck +trans_mat[k,j]))+L_cam(T i ) Cj (6)

[0098] Where j, k = 0 or 1;

[0099] Step S213, calculating the occupied probability and unoccupied probability of each grid according to the likelihood probability and forward probability of each grid;

[0100] Specifically, this embodiment calculates the occupied probability and unoccupied probability of the grid according to formula (7).

[0101] Map(T i )Cj = exp(F_cam(T i ) Cj -logΣ k (exp(F_cam(T i ) Cj )) (7)

[0102] Where j, k = 0 or 1.

[0103] Step S214, determining whether each grid is a passable area according to the occupied probability and unoccupied probability of each grid, and obtaining a partial grid map corresponding to the visual grid map area;

[0104] Specifically, by comparing the dynamic grid map Map (T i ) is used to binarize the values ​​of channel C0 and channel C1. If the value of channel C0 is Map(T i )C0 is greater than the value of channel C1Map(T i )C1, the grid is considered to be a non-drivable area. On the contrary, if the value of channel C0 Map(T i )C0 is less than the value of channel C1Map(T i )C1, the grid is considered to be a drivable area.

[0105] Step S215, fusing the partial grid map with the grid map at the previous moment to obtain the grid map at the current moment;

[0106] Specifically, the historical grid map portion of the area that is not detected at the current moment is superimposed with the partial grid map obtained in step S214 to obtain the grid map at the current moment.

[0107] In the following embodiments, the ultrasonic grid map includes a plurality of grids and a probability value p_uss (T i+1 ), the ultrasonic mask includes a plurality of grids and a characteristic value Map_USS (T i+1 );

[0108] The updating of the grid map at the previous moment according to the ultrasonic grid map and the ultrasonic mask at the current moment specifically includes:

[0109] Step S221, determining the ultrasonic grid map area detected by the vehicle ultrasonic sensor at the current moment according to the ultrasonic mask.

[0110] Step S222, calculating the likelihood probability and forward probability of each grid according to the hidden Markov model and the probability value that each grid in the ultrasonic grid map area is a drivable area;

[0111] Specifically, the hidden Markov model includes 4 parameters, initial probability start_prob, state transfer matrix trans_mat, state mean mean and state variance covar. In this embodiment, their initial values ​​are: start_prob = [0.5, 0.5], trans_mat = [[0.9, 0.1], [0.2, 0.8]], mean = [0.9, 0.1], covar = [0.1, 0.1]; the initialization of the hidden Markov model is performed in step S10;

[0112] This embodiment calculates the likelihood probability L_uss(T i );

[0113] L_uss(T i ) Cj =-0.5*((p_uss(T i ) Cj -mean[j]) / covar[j] 0.5 )+log(2π)+2log(covar[j] 0.5 ) (8)

[0115] Wherein, j = 0 or 1;

[0116] This embodiment calculates the forward probability F_uss(T i );

[0117] F_uss(T i ) Cj =logΣ k (exp(Map(T i-1 ) Ck +trans_mat[k,j]))+L_uss(T i ) Cj (9)

[0118] Where j, k = 0 or 1;

[0119] Step S223, calculating the occupied probability and unoccupied probability of each grid according to the likelihood probability and forward probability of each grid;

[0120] Specifically, this embodiment calculates the occupied probability and unoccupied probability of the grid according to formula (10).

[0121] Map(T i )Cj = exp(F_uss(T i ) Cj -logΣ k (exp(F_uss(T i ) Cj )) (10)

[0122] Where j, k = 0 or 1.

[0123] Step S224, determining whether each grid is a passable area according to the occupied probability and unoccupied probability of each grid, and obtaining a partial grid map corresponding to the ultrasonic grid map area;

[0124] Specifically, by comparing the dynamic grid map Map (T i ) is used to binarize the values ​​of channel C0 and channel C1. If the value of channel C0 is Map(T i )C0 is greater than the value of channel C1Map(T i )C1, the grid is considered to be a non-drivable area. On the contrary, if the value of channel C0 Map(T i )C0 is less than the value of channel C1Map(T i )C1, the grid is considered to be a drivable area.

[0125] Step S225, merging the partial grid map with the grid map at the previous moment to obtain the grid map at the current moment;

[0126] Specifically, the historical grid map portion of the area that is not detected at the current moment is superimposed with the partial grid map obtained in step S214 to obtain the grid map at the current moment.

[0127] Figure 3 is a flow chart of a drivable area detection method in a specific embodiment of the present application, such as Figure 3 As shown in the figure, firstly, a dynamic grid map is established within a preset range around the vehicle, and the dynamic grid map Map(T0) at time T0 is initialized; the camera is used to collect T i The visual drivable area probability map is obtained by semantic segmentation, and the visual drivable area probability map is projected into the vehicle coordinate system, spliced ​​and rasterized to obtain a 360° visual grid map around the vehicle from a bird's eye view. The visual grid map is converted to the world coordinate system to obtain T i Visual grid map Map_Cam(T i ) and visual mask Mask_Cam(T i ); Use T iVisual grid map Map_Cam(T i ) and visual mask Mask_Cam(T i ), update T according to the hidden Markov model i-1 Dynamic grid map Map(T i-1 ), and get T i Dynamic grid map Map(T i ), perform binarization, and obtain T i The drivable area after fusion at all times; using ultrasonic sensors to collect T i+1 The distance information at that moment is created based on the ultrasonic sensor position, parameters and measured distance information. i+1 The probability map of the ultrasonic drivable area at time T is rasterized and converted to the world coordinate system to obtain i+1 Ultrasonic grid map Map_USS(T i+1 ) and ultrasonic mask Mask_USS(T i+1 ); Use T i+1 Ultrasonic grid map Map_USS(T i+1 ) and ultrasonic mask Mask_USS(T i+1 ), update T according to the hidden Markov model i Dynamic grid map Map(T i ), and get T i+1 Dynamic grid map Map(T i+1 ), perform binarization, and obtain T i+1 The drivable area after fusion at all times. The drivable area detection effect diagram is as follows Figure 4 As shown, Figure 4 (a) is the detection result of the drivable area from a bird's eye view. The blue curve and the red curve are the boundaries of the drivable area without and with the probability map, respectively. Figure 4 (b) is the drivable area without using the probability map for fusion. The white area represents the drivable area and the black area represents the non-drivable area. Figure 4 (c) is the drivable area fused using the probability map, where white represents the drivable area, black represents the non-drivable area, and gray represents the unknown area.

[0128] In summary, the method of this embodiment designs a drivable area detection method based on a multi-sensor fusion based on a grid map, which has the following advantages:

[0129] (1) It uses a low-cost camera and ultrasonic sensor fusion solution, eliminating the need for lidar and high-precision maps, reducing usage and maintenance costs and expanding application scenarios;

[0130] (2) Based on the dynamic grid map, the hidden Markov model is used to model the grid state in the scene, which is applicable to both dynamic and static scenes;

[0131] (3) A probability-based method is used to fuse the detection probabilities of the camera and ultrasonic sensor, and the camera and ultrasonic sensor data of different frequencies can be processed, minimizing information loss;

[0132] (4) The present invention combines the drivable area detection results of the current frame with the historical results, which reduces the impact of occlusion and improves the detection accuracy. Figure 4 As shown in (c), the drivable area detection based on the present invention can determine whether the blocked area is a drivable area based on historical information.

[0133] See also Figure 5 Another embodiment of the present application further provides a drivable area detection device, the device comprising:

[0134] An initial mapping module, used to establish an initial grid map at an initial moment, wherein the grid map includes a drivable area and a non-drivable area;

[0135] A dynamic update module is used to dynamically update the grid map at each moment after the initial moment using the vehicle environment image collected by the vehicle camera or the obstacle distance information collected by the vehicle ultrasonic sensor to obtain the grid map at the current moment; wherein, if the grid map was dynamically updated using the vehicle environment image collected by the vehicle camera at the previous moment, then the obstacle distance information collected by the vehicle ultrasonic sensor is used to dynamically update the grid map at the previous moment at the current moment; if the grid map was dynamically updated using the obstacle distance information collected by the vehicle ultrasonic sensor at the previous moment, then the vehicle environment image collected by the vehicle camera is used to dynamically update the grid map at the previous moment at the current moment.

[0136] In the following embodiments, the dynamic update module includes:

[0137] A visual update submodule is used to obtain a vehicle environment image at the current moment, obtain a visual drivable area probability map based on the vehicle environment image at the current moment, convert the visual drivable area probability map into a world coordinate system and rasterize it to obtain a visual grid map and a visual mask at the current moment, and update a grid map at the previous moment based on the visual grid map and the visual mask at the current moment;

[0138] The ultrasonic update submodule is used to obtain the obstacle distance information at the current moment, obtain the ultrasonic drivable area probability map based on the obstacle distance information at the current moment, convert the ultrasonic drivable area probability map to the world coordinate system and rasterize it to obtain the ultrasonic grid map and ultrasonic mask at the current moment, and update the grid map at the previous moment based on the ultrasonic grid map and ultrasonic mask at the current moment.

[0139] In the following embodiments, the visual grid map includes a plurality of grids and a probability value of each grid being a drivable area, and the visual mask includes a plurality of grids and a feature value of whether each grid is detected by a vehicle camera;

[0140] Wherein, the visual update submodule is specifically used for:

[0141] Determine the visual grid map area detected by the vehicle camera at the current moment according to the visual mask;

[0142] Calculating the likelihood probability and the forward probability of each grid according to the hidden Markov model and the probability value of each grid in the visual grid map area being a drivable area;

[0143] Calculating the occupied probability and unoccupied probability of each grid according to the likelihood probability and forward probability of each grid;

[0144] Determine whether each grid is a passable area according to the occupied probability and unoccupied probability of each grid, and obtain a partial grid map corresponding to the visual grid map area;

[0145] The partial grid map is merged with the grid map at the previous moment to obtain the grid map at the current moment.

[0146] In the following embodiments, the ultrasonic grid map includes a plurality of grids and a probability value of each grid being a drivable area, and the ultrasonic mask includes a plurality of grids and a characteristic value of whether each grid is detected by a vehicle camera;

[0147] Wherein, the ultrasonic updating submodule is specifically used for:

[0148] Determine the ultrasonic grid map area detected by the vehicle ultrasonic sensor at the current moment according to the ultrasonic mask;

[0149] Calculate the likelihood probability and forward probability of each grid according to the hidden Markov model and the probability value of each grid in the ultrasonic grid map area being a drivable area;

[0150] Calculating the occupied probability and unoccupied probability of each grid according to the likelihood probability and forward probability of each grid;

[0151] Determine whether each grid is a passable area according to the occupied probability and unoccupied probability of each grid, and obtain a partial grid map corresponding to the ultrasonic grid map area;

[0152] The partial grid map is merged with the grid map at the previous moment to obtain the grid map at the current moment.

[0153] It should be noted that the drivable area detection device of the present embodiment corresponds to the drivable area detection method of the above-mentioned embodiment. Therefore, the parts of the drivable area detection device of the present embodiment that are not described in detail can be obtained by referring to the contents of the drivable area detection method of the above-mentioned embodiment, that is, the specific steps recorded in the drivable area detection method of the above-mentioned embodiment can be understood as the functions that can be realized in the drivable area detection device of the present embodiment, and will not be repeated here.

[0154] Furthermore, if the drivable area detection device of the above embodiment is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0155] Another embodiment of the present application further proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the drivable area detection method as described in the above embodiment is implemented.

[0156] Specifically, the computer-readable storage medium may include: any entity or recording medium that can carry the computer program instructions, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0157] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for detecting a drivable area, characterized in that: The method comprises: Establishing an initial grid map at an initial moment, wherein the grid map includes a drivable area and a non-drivable area; At each moment after the initial moment, the grid map is dynamically updated using the vehicle environment image collected by the vehicle camera or the obstacle distance information collected by the vehicle ultrasonic sensor to obtain the grid map of the current moment; Among them, if the grid map was dynamically updated using the vehicle environment image collected by the vehicle camera at the previous moment, the grid map of the previous moment is dynamically updated using the obstacle distance information collected by the vehicle ultrasonic sensor at the current moment; if the grid map was dynamically updated using the obstacle distance information collected by the vehicle ultrasonic sensor at the previous moment, the grid map of the previous moment is dynamically updated using the vehicle environment image collected by the vehicle camera at the current moment.

2. The method according to claim 1, characterized in that The grid map is dynamically updated at each moment after the initial moment to obtain the grid map at the current moment, specifically including: When determining to dynamically update the grid map of the previous moment using the vehicle environment image collected by the vehicle camera, if the vehicle environment image of the current moment cannot be obtained, the obstacle distance information of the current moment is obtained, and the grid map of the previous moment is dynamically updated according to the obstacle distance information of the current moment; When determining to dynamically update the grid map of the previous moment using the obstacle distance information at the current moment collected by the vehicle ultrasonic sensor, if the obstacle distance information at the current moment cannot be obtained, the vehicle environment image at the current moment is obtained, and the grid map of the previous moment is dynamically updated based on the vehicle environment image at the current moment.

3. The method according to claim 1, characterized in that The vehicle environment image collected by the vehicle camera is used to update the grid map at the previous moment, including: Obtaining a vehicle environment image at the current moment, obtaining a visual drivable area probability map according to the vehicle environment image at the current moment, converting the visual drivable area probability map to a world coordinate system and rasterizing it to obtain a visual grid map and a visual mask at the current moment, and updating a grid map at the previous moment according to the visual grid map and the visual mask at the current moment; The vehicle environment image collected by the vehicle ultrasonic sensor is used to update the grid map at the previous moment, including: Obtain obstacle distance information at the current moment, obtain an ultrasonic drivable area probability map based on the obstacle distance information at the current moment, convert the ultrasonic drivable area probability map to the world coordinate system and rasterize it to obtain the ultrasonic grid map and ultrasonic mask at the current moment, and update the grid map at the previous moment based on the ultrasonic grid map and ultrasonic mask at the current moment.

4. The method according to claim 3, characterized in that: The visual grid map includes a plurality of grids and a probability value of each grid being a drivable area, and the visual mask includes a plurality of grids and a feature value of whether each grid is detected by a vehicle camera; The updating of the grid map at the previous moment according to the visual grid map and the visual mask at the current moment specifically includes: Determine the visual grid map area detected by the vehicle camera at the current moment according to the visual mask; Calculating the likelihood probability and the forward probability of each grid according to the hidden Markov model and the probability value of each grid in the visual grid map area being a drivable area; Calculating the occupied probability and unoccupied probability of each grid according to the likelihood probability and forward probability of each grid; Determine whether each grid is a passable area according to the occupied probability and unoccupied probability of each grid, and obtain a partial grid map corresponding to the visual grid map area; The partial grid map is merged with the grid map at the previous moment to obtain the grid map at the current moment.

5. The method according to claim 3, characterized in that: The ultrasonic grid map includes a plurality of grids and a probability value of each grid being a drivable area, and the ultrasonic mask includes a plurality of grids and a characteristic value of whether each grid is detected by a vehicle ultrasonic sensor; The updating of the grid map at the previous moment according to the ultrasonic grid map and the ultrasonic mask at the current moment specifically includes: Determine the ultrasonic grid map area detected by the vehicle ultrasonic sensor at the current moment according to the ultrasonic mask; Calculate the likelihood probability and forward probability of each grid according to the hidden Markov model and the probability value of each grid in the ultrasonic grid map area being a drivable area; Calculating the occupied probability and unoccupied probability of each grid according to the likelihood probability and forward probability of each grid; Determine whether each grid is a passable area according to the occupied probability and unoccupied probability of each grid, and obtain a partial grid map corresponding to the ultrasonic grid map area; The partial grid map is merged with the grid map at the previous moment to obtain the grid map at the current moment.

6. A drivable area detection device, characterized in that: The device comprises: An initial mapping module, used to establish an initial grid map at an initial moment, wherein the grid map includes a drivable area and a non-drivable area; A dynamic update module is used to dynamically update the grid map at each moment after the initial moment using the vehicle environment image collected by the vehicle camera or the obstacle distance information collected by the vehicle ultrasonic sensor to obtain the grid map at the current moment; wherein, if the grid map was dynamically updated using the vehicle environment image collected by the vehicle camera at the previous moment, then the obstacle distance information collected by the vehicle ultrasonic sensor is used to dynamically update the grid map at the previous moment at the current moment; if the grid map was dynamically updated using the obstacle distance information collected by the vehicle ultrasonic sensor at the previous moment, then the vehicle environment image collected by the vehicle camera is used to dynamically update the grid map at the previous moment at the current moment.

7. The device according to claim 6, characterized in that The dynamic update module comprises: A visual update submodule is used to obtain a vehicle environment image at the current moment, obtain a visual drivable area probability map based on the vehicle environment image at the current moment, convert the visual drivable area probability map into a world coordinate system and rasterize it to obtain a visual grid map and a visual mask at the current moment, and update a grid map at the previous moment based on the visual grid map and the visual mask at the current moment; The ultrasonic update submodule is used to obtain the obstacle distance information at the current moment, obtain the ultrasonic drivable area probability map based on the obstacle distance information at the current moment, convert the ultrasonic drivable area probability map to the world coordinate system and rasterize it to obtain the ultrasonic grid map and ultrasonic mask at the current moment, and update the grid map at the previous moment based on the ultrasonic grid map and ultrasonic mask at the current moment.

8. The device according to claim 7, characterized in that The visual grid map includes a plurality of grids and a probability value of each grid being a drivable area, and the visual mask includes a plurality of grids and a feature value of whether each grid is detected by a vehicle camera; Wherein, the visual update submodule is specifically used for: Determine the visual grid map area detected by the vehicle camera at the current moment according to the visual mask; Calculating the likelihood probability and the forward probability of each grid according to the hidden Markov model and the probability value of each grid in the visual grid map area being a drivable area; Calculating the occupied probability and unoccupied probability of each grid according to the likelihood probability and forward probability of each grid; Determine whether each grid is a passable area according to the occupied probability and unoccupied probability of each grid, and obtain a partial grid map corresponding to the visual grid map area; The partial grid map is merged with the grid map at the previous moment to obtain the grid map at the current moment.

9. The device according to claim 7, characterized in that The ultrasonic grid map includes a plurality of grids and a probability value of each grid being a drivable area, and the ultrasonic mask includes a plurality of grids and a characteristic value of whether each grid is detected by a vehicle ultrasonic sensor; Wherein, the ultrasonic updating submodule is specifically used for: Determine the ultrasonic grid map area detected by the vehicle ultrasonic sensor at the current moment according to the ultrasonic mask; Calculate the likelihood probability and forward probability of each grid according to the hidden Markov model and the probability value of each grid in the ultrasonic grid map area being a drivable area; Calculating the occupied probability and unoccupied probability of each grid according to the likelihood probability and forward probability of each grid; Determine whether each grid is a passable area according to the occupied probability and unoccupied probability of each grid, and obtain a partial grid map corresponding to the ultrasonic grid map area; The partial grid map is merged with the grid map at the previous moment to obtain the grid map at the current moment.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the drivable area detection method according to any one of claims 1 to 5 is implemented.

Images