A travelable area detection method, device, electronic equipment, and storage medium

Abstract

The application discloses a drivable area detection method, device, electronic equipment and storage medium. The method comprises the following steps: acquiring perception information of a traffic environment, wherein the perception information comprises a three-dimensional laser point cloud perceived by a roadside laser radar and a two-dimensional image perceived by a roadside camera; determining an initial ground point cloud and a lane isolation point cloud in the traffic environment according to the three-dimensional laser point cloud; determining a lane edge line in the traffic environment according to the two-dimensional image; identifying whether the initial ground point cloud contains a non-ground point cloud according to the lane edge line and the lane isolation point cloud; and acquiring a drivable area in the traffic environment according to the identification result. The technical scheme of the application enables the roadside device to quickly and accurately detect the drivable area of a vehicle in the traffic environment.

Description

Technical Field

[0001] This application relates to the field of automated inspection technology, and in particular to a method, apparatus, electronic device, and storage medium for detecting drivable areas. Background Technology

[0002] Methods for detecting drivable areas of vehicles on roads are generally divided into two types: single-sensor detection and multi-sensor fusion detection.

[0003] Most single-sensor detection methods rely on vehicle-mounted cameras or LiDAR. However, single-sensor detection of drivable road areas has some unavoidable drawbacks: for example, vehicle-mounted cameras cannot adapt to complex weather and scene changes, are easily affected by weather variations, and have unstable robustness; and when LiDAR detects road edges, if the height difference between the road area and the non-road area is not significant, the LiDAR detection method is very prone to errors.

[0004] Regarding multi-sensor fusion detection, the most commonly used method is the fusion of vehicle-mounted cameras and LiDAR. The road image captured by the vehicle-mounted camera is segmented into a drivable area, and the LiDAR point cloud is projected onto the road image based on the calibration relationship between the vehicle-mounted camera and the LiDAR. Based on the drivable area segmentation result of the road image, the non-road area in the LiDAR point cloud is identified.

[0005] Research has found that current multi-sensor fusion detection technologies typically detect drivable areas from the vehicle's perspective. However, in many scenarios, drivable area detection needs to be performed from the roadside perspective. From the roadside perspective, the perception range of roadside equipment is much larger than that of the vehicle, and the traffic scenarios are more complex. For example, intersections usually include multiple lanes, various curbs, lane separation facilities, etc. Some lanes may have no-entry zones such as yellow grid lines and guide lines. In such complex traffic environments, it is difficult to segment accurately drivable areas from the images perceived by roadside cameras. Summary of the Invention

[0006] In view of the above-mentioned problems existing in the prior art, this application provides a drivable area detection method, device, electronic device, and storage medium, so that roadside equipment can quickly and accurately detect the drivable area of ​​vehicles on the road.

[0007] The embodiments of this application adopt the following technical solutions:

[0008] In a first aspect, embodiments of this application provide a method for detecting drivable areas, the method comprising:

[0009] Acquire traffic environment perception information, including three-dimensional laser point clouds perceived by roadside lidar and two-dimensional images perceived by roadside cameras;

[0010] The initial ground point cloud and lane separation point cloud in the traffic environment are determined based on the three-dimensional laser point cloud;

[0011] The lane edge lines in the traffic environment are determined based on the two-dimensional image;

[0012] Based on the lane edge line and the lane isolation point cloud, identify whether the initial ground point cloud contains non-ground point clouds;

[0013] The drivable area in the traffic environment is obtained based on the identification results.

[0014] Optionally, identifying whether the initial ground point cloud contains non-ground point clouds based on the lane edge line and the lane isolation point cloud includes:

[0015] The target region is obtained by dividing the two-dimensional image into regions using the lane edge lines;

[0016] Based on the calibration parameters between the roadside lidar and the roadside camera, the first mapping region of the initial ground point cloud in the two-dimensional image and the second mapping region of the lane isolation point cloud in the two-dimensional image are obtained.

[0017] The type of the lane edge line is determined based on the positional relationship between the lane edge line and the second mapping area, and the non-drivable area in the target area is determined based on the type of the lane edge line corresponding to the target area.

[0018] The initial ground point cloud is identified as containing non-ground point clouds based on the positional relationship between the non-drivable area and the first mapped area.

[0019] Optionally, determining the type of the lane edge line based on the positional relationship between the lane edge line and the second mapping area includes:

[0020] Determine the lane edge line located to the left of the second mapping area, and in order of increasing relative distance from the second mapping area, determine the adjacent lane edge lines located to the left of the second mapping area as the right lane edge line and the left lane edge line in sequence;

[0021] Determine the lane edge line located to the right of the second mapping area, and follow the direction of the second mapping area.

[0022] Based on the relative distance from near to far, the adjacent lane edge lines located on the right side of the second mapping area are sequentially designated as the left lane edge line and the right lane edge line.

[0023] Optionally, the type of lane edge line includes left lane edge line and right lane edge line, and determining the non-drivable area in the target area according to the type of lane edge line corresponding to the target area includes:

[0024] When the left edge of the target area corresponds to the right lane edge line, and / or the right edge of the target area 0 corresponds to the left lane edge line, the target area is determined to be a non-drivable area.

[0025] Optionally, identifying whether the initial ground point cloud contains non-ground point clouds based on the positional relationship between the non-drivable area and the first mapped area includes:

[0026] When the first mapping area does not overlap with the non-driving area, it is determined that the initial ground point cloud does not contain non-ground point clouds.

[0027] 5. When the first mapping area overlaps with the non-drivable area, it is determined that the initial ground point cloud contains non-ground point clouds, and the point cloud corresponding to the overlapping area is determined as the non-ground point cloud in the initial ground point cloud.

[0028] Optionally, when determining lane edge lines in the traffic environment based on the two-dimensional image, the method further includes:

[0029] 0. Based on the two-dimensional image, a preset road traffic sign line is determined in the traffic environment. The preset road traffic sign line is used to indicate the no-entry zone for vehicles on the road surface.

[0030] The vehicle restricted area in the two-dimensional image is obtained based on the preset road traffic signs;

[0031] After obtaining the drivable area in the traffic environment based on the recognition results, the method further includes:

[0032] 5. Based on the calibration parameters between the roadside lidar and the roadside camera, obtain the third mapping region of the ground point cloud in the two-dimensional image, wherein the ground point cloud is the final ground point cloud determined from the initial ground point cloud based on the recognition result;

[0033] Determine the final drivable area in the third mapping region, excluding the vehicle restricted area, and obtain the drivable area in the traffic environment based on the point cloud corresponding to the final drivable area.

[0034] Optionally, the method further includes:

[0035] A training dataset is constructed, which includes training samples and labeled data. The training samples are obtained from historical two-dimensional images perceived by the roadside camera, and the labeled data includes the position information of lane edge lines and road traffic marking lines in the training samples.

[0036] A detection model is constructed, comprising an encoder, a DLA feature extraction network, and a decoder. The encoder is used to determine the proximity affinity domain of lane edge lines and / or preset road traffic signs in the training samples based on the labeled data, and to encode the training samples based on the proximity affinity domain. The DLA feature extraction network is used for feature extraction, and the decoder is used to decode the features based on the labeled data, and to predict lane edge lines and / or preset road traffic signs in the training samples based on the decoding results.

[0037] The detection model is trained based on the training samples and the labeled data of the training samples.

[0038] The trained detection model is used to detect lane edge lines and preset road traffic markings in the two-dimensional image.

[0039] Secondly, embodiments of this application also provide a drivable area detection device, the device comprising:

[0040] The perception information acquisition unit is used to acquire perception information of the traffic environment, including three-dimensional laser point clouds perceived by roadside lidar and two-dimensional images perceived by roadside cameras.

[0041] A point cloud processing unit is used to determine the initial ground point cloud and lane isolation point cloud in the traffic environment based on the three-dimensional laser point cloud.

[0042] An image processing unit is configured to determine lane edge lines in the traffic environment based on the two-dimensional image;

[0043] A non-ground point cloud recognition unit is used to identify whether the initial ground point cloud contains non-ground point clouds based on the lane edge line and the lane isolation point cloud.

[0044] A drivable area detection unit is used to obtain drivable areas in the traffic environment based on the identification results.

[0045] Thirdly, embodiments of this application also provide an electronic device, including:

[0046] Processor; and

[0047] A memory is configured to store computer-executable instructions, which, when executed, cause the processor to perform a drivable area detection method.

[0048] Fourthly, embodiments of this application also provide a computer-readable storage medium storing one or more programs that, when executed by an electronic device including multiple applications, cause the electronic device to perform a drivable area detection method.

[0049] The at least one technical solution adopted in this application embodiment can achieve the following beneficial effects: This application embodiment first senses a three-dimensional laser point cloud and a two-dimensional image using a roadside lidar and a roadside camera; then, it detects the initial ground point cloud and lane isolation point cloud from the three-dimensional laser point cloud, and detects the lane edge line from the two-dimensional image; then, it combines the lane edge line with the lane isolation point cloud to identify whether there are non-ground point clouds in the initial ground point cloud; finally, it detects the drivable area in the traffic environment based on the identification result. This application embodiment, based on the lane edge line in the two-dimensional image and the lane isolation point cloud in the three-dimensional laser point cloud, can accurately identify whether there are non-ground point clouds in the initial ground point cloud, enabling the roadside equipment to quickly detect the true ground point cloud in the three-dimensional laser point cloud, obtain the drivable area of ​​the traffic environment based on the ground point cloud, and obtain highly accurate detection results. Attached Figure Description

[0050] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0051] Figure 1 This is a flowchart illustrating a drivable area detection method in an embodiment of this application;

[0052] Figure 2 This is a schematic diagram illustrating the mapping effect of an initial ground point cloud onto a two-dimensional image, as shown in an embodiment of this application.

[0053] Figure 3 This is a schematic diagram illustrating the distribution of relevant target objects in a two-dimensional image in an embodiment of this application;

[0054] Figure 4 This is a schematic diagram of the structure of a drivable area detection device shown in an embodiment of this application;

[0055] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0057] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0058] The execution entity of the drivable area detection method provided in this application embodiment can be a roadside device (such as a roadside camera or roadside computing device), a roadside server, or a cloud control platform; the execution entity of the drivable area detection method in this application embodiment can also be software or hardware.

[0059] Please refer to Figure 1 , Figure 1 Taking roadside equipment as an example, this application provides a method for detecting drivable areas. For instance... Figure 1 As shown, a drivable area detection method provided in this application embodiment may include the following steps S110 to S150:

[0060] Step S110: Obtain traffic environment perception information, including three-dimensional laser point clouds perceived by roadside lidar and two-dimensional images perceived by roadside cameras.

[0061] Once the roadside camera and roadside lidar are installed on the roadside poles and their individual and joint calibrations are completed, they can be used to perceive the traffic environment, obtaining 3D laser point clouds and 2D images. Each scan point in the 3D laser point cloud is represented by four-dimensional features: the x, y, and z coordinates of the object in the lidar coordinate system, and the reflectivity intensity. The 2D image is a grayscale image, which can be viewed as a W*H matrix, where the elements range from 0 to 255, with W representing the width and H representing the height. Of course, the 2D image can also be a color image, such as an RGB color image.

[0062] It should be noted that the traffic environment in this embodiment includes lane separation facilities, which can be... Figure 2 The green isolation strip shown can also be a isolation net, isolation pole, or isolation curb with a certain height. In this way, in the three-dimensional laser point cloud sensed by the roadside lidar, the point cloud cluster of lane isolation facilities and the point cloud cluster of the road surface have a significant height difference, so that the lane isolation point cloud can be detected from the three-dimensional laser point cloud.

[0063] Step S120: Determine the initial ground point cloud and lane isolation point cloud in the traffic environment based on the three-dimensional laser point cloud.

[0064] This application embodiment uses a ground detection model to perform ground detection on 3D laser point clouds. Historical 3D laser point clouds from roadside lidar can be used as point cloud training samples. These point cloud training samples are used to train a convolutional neural network to obtain a ground detection model. Alternatively, a laser point cloud ground detection algorithm can be used to implement the ground detection model.

[0065] Because lane separation facilities in a traffic environment have a significant height difference from the ground, the initial ground point cloud and lane separation point cloud in a 3D laser point cloud can be detected using a ground detection model. However, the traffic environment also includes non-ground areas where the height difference from the ground is not significant, such as... Figure 2 The right-hand curb area is a non-driving area. Because the height difference between this area and the ground is small, the ground detection model will misidentify this area as the initial ground point cloud, so the detected initial ground point cloud may include non-ground point clouds corresponding to the non-driving area.

[0066] To address this issue, this application embodiment combines lane edge lines in a two-dimensional image and lane isolation point clouds in a three-dimensional laser point cloud to perform secondary detection on the initial ground point cloud in the three-dimensional laser point cloud, and determines the ground point cloud that truly belongs to the ground based on the secondary detection results.

[0067] Step S130: Determine the lane edge lines in the traffic environment based on the two-dimensional image.

[0068] Lane edge lines refer to the edges of lanes, typically located at the outermost edge of the road. A certain lateral clearance is maintained between the lane edge line and the lane divider. In this embodiment, the lane edge lines include a left lane edge line and a right lane edge line. The left lane edge line is the lane edge line located to the right of the lane divider and closest to it, while the right lane edge line is the lane edge line located to the left of the lane divider and closest to it. Figure 3 L1, L3, and L5 are the left lane edge lines, and L2, L4, and L6 are the right lane edge lines.

[0069] In this embodiment, the execution order of steps S120 and S130 is not limited. Step S120 can be executed first, followed by step S130, or step S130 can be executed first, followed by step S120. Of course, steps S120 and S130 can also be executed simultaneously.

[0070] Step S140: Identify whether the initial ground point cloud contains ground point cloud based on the lane edge line and the lane isolation point cloud.

[0071] Research has found that the computational complexity and generalization ability of drivable area detection algorithms for 2D images are related to the complexity of the traffic environment. The more complex the traffic environment corresponding to the 2D image, the higher the computational complexity of the drivable area detection algorithm and the worse the generalization ability. For example, in complex traffic environments such as intersections, the detection efficiency and accuracy of drivable area detection algorithms are both poor.

[0072] Compared to drivable area detection algorithms, lane edge detection algorithms are less correlated with the complexity of the traffic environment. Therefore, using lane edge detection algorithms to detect two-dimensional images has significant advantages in terms of detection efficiency and accuracy.

[0073] Based on this, embodiments of this application detect lane edge lines in a traffic environment from a two-dimensional image, detect lane isolation point clouds and initial ground point clouds from a three-dimensional laser point cloud, identify non-drivable areas in a two-dimensional image by combining lane edge lines and lane isolation point clouds, determine whether the initial ground point cloud contains non-ground point clouds based on the non-drivable areas in the two-dimensional image, and obtain the ground point cloud that truly belongs to the ground in the three-dimensional laser point cloud according to the identification results.

[0074] Step S150: Obtain the drivable area in the traffic environment based on the recognition result.

[0075] Based on the above steps, it can be seen that the initial ground point cloud detected from the three-dimensional laser point cloud may include non-ground point clouds that do not belong to the ground. In this embodiment of the application, the initial ground point cloud is detected by combining the lane edge line and lane isolation point cloud, so that the ground point cloud that truly belongs to the ground can be obtained. In this way, the boundary of the drivable area in the traffic environment can be accurately obtained based on the point cloud range of the ground point cloud.

[0076] like Figure 1 As illustrated in the drivable area detection method, this embodiment first uses roadside lidar and roadside cameras to perceive 3D lidar point clouds and 2D images. Then, it detects the initial ground point cloud and lane separation point cloud from the 3D lidar point cloud, and lane edge lines from the 2D image. Next, it combines the lane edge lines with the lane separation point cloud to identify non-ground point clouds and ground point clouds in the initial ground point cloud. Finally, it detects the drivable area in the traffic environment based on the identified ground point cloud. This embodiment, based on the lane edge lines in the 2D image and the lane separation point cloud in the 3D lidar point cloud, can accurately identify whether there are non-ground point clouds in the initial ground point cloud. This allows roadside equipment to quickly detect the ground point clouds that truly belong to the 3D lidar point cloud, and obtain the drivable area of ​​the traffic environment based on the ground point cloud, resulting in highly accurate detection results.

[0077] Studies have found that road markings such as guide lines are installed on special road sections. Guide lines are mainly white V-shaped lines or diagonal lines, designed according to the terrain of intersections, indicating that vehicles must travel along the prescribed route and must not cross or drive over the lines. Figure 3 The two V-shaped guide lines at the ends of the greenbelt are no-entry zones for vehicles.

[0078] In some embodiments of this application, when determining lane edge lines in the traffic environment based on the two-dimensional image, the method further includes: (This is in response to situations where vehicle-restricted areas exist on the road surface.)

[0079] Based on the two-dimensional image, preset road traffic signs are determined in the traffic environment. The preset road traffic signs are used to indicate no-entry zones for vehicles on the road surface.

[0080] The vehicle restricted area in the two-dimensional image is obtained based on the preset road traffic signs;

[0081] Accordingly, after obtaining the drivable area in the traffic environment based on the recognition result, the method further includes:

[0082] Based on the calibration parameters between the roadside lidar and the roadside camera, the third mapping region of the ground point cloud in the two-dimensional image is obtained, and the ground point cloud is the final ground point cloud determined from the initial ground point cloud based on the recognition result;

[0083] Determine the final drivable area in the third mapping region, excluding the vehicle restricted area, and obtain the drivable area in the traffic environment based on the point cloud corresponding to the final drivable area.

[0084] This embodiment detects whether there are vehicle-restricted areas on the road surface based on two-dimensional images. When a vehicle-restricted area exists on the road surface, it can be removed from the drivable area to obtain the final drivable area, thereby improving the accuracy of the drivable area detection results.

[0085] This embodiment illustrates a specific implementation method for adjusting detection results based on vehicle restricted areas. It should be understood that other methods can also be used. For example, the ground point cloud can be projected onto a world coordinate system based on the calibration parameters of the roadside lidar to obtain a first point cloud image corresponding to the ground point cloud. Similarly, the vehicle restricted area can be projected onto a world coordinate system based on the calibration parameters of the roadside camera to obtain a second point cloud image corresponding to the vehicle restricted area. The type of each point in the first point cloud image can be determined based on the second point cloud image. For instance, the latitude and longitude information of each point can determine whether each point in the first point cloud image belongs to a vehicle restricted area. This yields the point cloud of points in the first point cloud image that do not belong to a vehicle restricted area, and the drivable area in the traffic environment can be obtained based on this point cloud.

[0086] In some embodiments of this application, Figure 1 The drivable area detection method also includes:

[0087] A training dataset is constructed, which includes training samples and labeled data. The training samples are obtained from historical two-dimensional images perceived by the roadside camera. For example, historical two-dimensional images perceived by the roadside camera under different weather conditions, different day and night times, and different seasons are used as training samples to improve the generalization ability of the detection model. The labeled data includes the position information of lane edge lines and road traffic marking lines in the training samples.

[0088] A detection model is constructed, comprising an encoder, a DLA (Deep Layer Aggregation) feature extraction network, and a decoder. The encoder is used to determine the neighborhood affinity domain of lane edge lines and / or preset road traffic signs in the training samples based on the labeled data, and to globally encode the training samples based on the neighborhood affinity domain. The DLA feature extraction network is used for feature extraction, and the decoder is used to decode the features based on the labeled data, and to predict lane edge lines and / or preset road traffic signs in the training samples based on the decoding results.

[0089] The detection model is trained based on the training samples and the labeled data of the training samples;

[0090] The trained detection model is used to detect lane edge lines and preset road traffic markings in the two-dimensional image.

[0091] In this embodiment, the detection model constructed by the encoder-feature extraction network-decoder enables the detection model to simultaneously detect multiple road traffic markings. Therefore, this embodiment does not require building a separate detection model for each type of road traffic marking, which improves the detection efficiency of roadside equipment in drivable areas. Of course, in some alternative embodiments, a detection model can be built separately for each type of road traffic marking.

[0092] In some embodiments of this application, identifying whether the initial ground point cloud contains non-ground point clouds based on the lane edge line and lane isolation point cloud includes:

[0093] First, the two-dimensional image is divided into regions using the lane edge lines to obtain the target region. For example... Figure 3 As shown, lane edge lines L1 to L6 divide the two-dimensional image into 7 target regions, which are assumed to be s1 to s7 from left to right.

[0094] Then, based on the calibration parameters between the roadside lidar and the roadside camera, the first mapping region of the initial ground point cloud in the two-dimensional image is obtained (e.g., Figure 2 The thick black solid line area in the middle is the first mapping area, and the lane isolation point cloud in the second mapping area Sa, Sb in the two-dimensional image.

[0095] Next, the type of the lane edge line is determined based on its positional relationship with the second mapping area, and the non-drivable area in the target area is determined based on the type of the lane edge line corresponding to the target area; thereby, it is possible to determine... Figure 3 The target areas s1, s3, s5 and s7 are non-driving areas.

[0096] Finally, based on the positional relationship between the non-drivable area and the first mapped area, the non-ground point cloud and ground point cloud in the initial ground point cloud are identified.

[0097] This application embodiment illustrates a specific implementation of the aforementioned step S140. Of course, it should be understood that step S140 can also be implemented in other ways, and this application embodiment does not limit this. For example, the initial ground point cloud and lane isolation point cloud can be transformed to a world coordinate system, such as the WGS-84 coordinate system, based on the calibration parameters of the roadside lidar. By normalizing the height features of the initial ground point cloud and lane isolation point cloud, a third point cloud image can be obtained. For ease of description, the area of ​​the initial ground point cloud in the third point cloud image is denoted as the initial ground area, and the area of ​​the lane isolation point cloud in the third point cloud image is denoted as the lane isolation area. Then, based on the calibration parameters of the roadside camera, the pixels of the lane edge line are projected onto the WGS-84 coordinate system to obtain the projection points corresponding to each pixel of the lane edge line. These projection points form a virtual point cloud of the lane edge line. The virtual point cloud is mapped onto the third point cloud image to obtain the position of the lane edge line in the third point cloud image. The lane edge line in the third point cloud image divides the initial ground area into multiple sub-regions. The type of lane edge line can be determined based on the positional relationship between the lane edge line and the lane isolation area in the third point cloud image. Based on the type of lane edge line corresponding to each sub-region, it can be determined whether the sub-region is a non-drivable area. In this way, it can be detected whether the initial ground area in the third point cloud image contains a non-drivable area. If it contains a non-drivable area, it is determined that the initial ground point cloud contains non-ground point clouds. If it does not contain a non-drivable area, it is determined that the initial ground point cloud does not contain non-ground point clouds.

[0098] In some possible implementations of this embodiment, determining the type of the lane edge line based on the positional relationship between the lane edge line and the second mapping area includes:

[0099] Determine the lane edge line located to the left of the second mapping area, and in order of increasing relative distance from the second mapping area, determine the adjacent lane edge lines located to the left of the second mapping area as the right lane edge line and the left lane edge line in sequence;

[0100] Determine the lane edge line located to the right of the second mapping area, and in order of increasing relative distance from the second mapping area, determine the adjacent lane edge lines located to the right of the second mapping area as the left lane edge line and the right lane edge line.

[0101] Combination Figure 3For the second mapping area Sb, the lane edge lines on the left side of the second mapping area Sb include lane edge lines L1 to L4. Lane edge lines L4 and L2 are the right lane edge lines, and lane edge lines L3 and L1 are the left lane edge lines. The lane edge lines on the right side of the second mapping area Sb include lane edge lines L5 to L6. Lane edge line L5 is the left lane edge line, and lane edge line L6 is the right lane edge line.

[0102] It should be noted that in practical applications, some roads use curb objects to construct isolation zones between two roads. For example, in certain traffic environments, this might be achieved by... Figure 3 A green median strip Sa is formed on the right side of the roadside. At this time, only the lane isolation point cloud of the green median strip Sa can be detected from the 3D laser point cloud, while the "green median strip Sa" will be misidentified as the initial ground point cloud. At this time, the type determination method based on the embodiment of this application can accurately determine the type of the lane edge lines on both sides of the "green median strip Sa", and thus accurately determine that the mapped area corresponding to the "green median strip Sa" is a non-drivable area.

[0103] In some other possible implementations of this embodiment, determining the non-drivable area within the target area based on the type of lane edge line corresponding to the target area includes:

[0104] When the left edge of the target area corresponds to the right lane edge line, and / or the right edge of the target area corresponds to the left lane edge line, the target area is determined to be a non-drivable area.

[0105] Combination Figure 3 Of the seven target areas shown, target area s1 is a non-drivable area because the lane edge line corresponding to the right edge of the target area is the left lane edge line L1. Target area s3 is also a non-drivable area because the lane edge line corresponding to the right edge of the target area is the left lane edge line L3, and the lane edge line corresponding to the left edge of the target area is the right lane edge line L2. Similarly, target area s5 is a non-drivable area because the lane edge line corresponding to the right edge of the target area is the left lane edge line L5, and the lane edge line corresponding to the left edge of the target area is the right lane edge line L4. Finally, target area s7 is a non-drivable area because the lane edge line corresponding to the left edge of the target area is the right lane edge line L7.

[0106] After determining the non-drivable region in the two-dimensional image according to the above embodiments, in some embodiments of this application, the positional relationship between the non-drivable region and the first mapped region is used to identify whether the initial ground point cloud contains non-ground point clouds. Specifically:

[0107] When the first mapped area does not overlap with the non-drivable area, it is determined that the initial ground point cloud does not contain non-ground point clouds. At this time, the initial ground point cloud is the final ground point cloud, and the boundary of the drivable area in the traffic environment can be detected based on the initial ground point cloud.

[0108] When the first mapped area overlaps with the non-drivable area, it is determined that the initial ground point cloud contains non-ground point clouds. The point cloud corresponding to the overlapping area is determined as the non-ground point cloud in the initial ground point cloud. At this time, the point cloud corresponding to the area of ​​the first mapped area other than the overlapping area can be determined as the final ground point cloud. The boundary of the drivable area in the traffic environment is detected based on the final ground point cloud.

[0109] This application embodiment also provides a drivable area detection device 400, such as Figure 4 The diagram shows a schematic representation of a drivable area detection device according to an embodiment of this application. The device 400 is applied to roadside equipment and includes: a perception information acquisition unit 410, a point cloud processing unit 420, an image processing unit 430, a non-ground point cloud recognition unit 440, and a drivable area detection unit 450, wherein:

[0110] The perception information acquisition unit 410 is used to acquire perception information of the traffic environment, including three-dimensional laser point clouds perceived by roadside lidar and two-dimensional images perceived by roadside cameras.

[0111] Point cloud processing unit 420 is used to determine the initial ground point cloud and lane isolation point cloud in the traffic environment based on the three-dimensional laser point cloud;

[0112] Image processing unit 430 is used to determine lane edge lines in the traffic environment based on the two-dimensional image;

[0113] The non-ground point cloud recognition unit 440 is used to identify whether the initial ground point cloud contains non-ground point clouds based on the lane edge line and the lane isolation point cloud.

[0114] The drivable area detection unit 450 is used to obtain the drivable area in the traffic environment based on the recognition result.

[0115] In one embodiment of this application, the non-ground point cloud recognition unit 440 includes: an image segmentation module, a mapping module, a first calculation module, and a second calculation module;

[0116] An image segmentation module is used to segment the two-dimensional image into regions based on the lane edge lines to obtain the target region;

[0117] The mapping module is used to obtain the first mapping region of the initial ground point cloud in the two-dimensional image and the second mapping region of the lane isolation point cloud in the two-dimensional image based on the calibration parameters between the roadside lidar and the roadside camera.

[0118] The first calculation module is used to determine the type of the lane edge line based on the positional relationship between the lane edge line and the second mapping area, and to determine the non-drivable area in the target area based on the type of the lane edge line corresponding to the target area;

[0119] The second calculation module is used to identify whether the initial ground point cloud contains non-ground point clouds based on the positional relationship between the non-drivable area and the first mapped area.

[0120] In one embodiment of this application, a first calculation module is used to determine the lane edge lines located on the left side of the second mapping area, and to sequentially determine the adjacent lane edge lines located on the left side of the second mapping area as the right lane edge line and the left lane edge line in order of their relative distance from the second mapping area from near to far; and to determine the lane edge lines located on the right side of the second mapping area, and to sequentially determine the adjacent lane edge lines located on the right side of the second mapping area as the left lane edge line and the right lane edge line in order of their relative distance from the second mapping area from near to far.

[0121] In one embodiment of this application, the type of lane edge line includes a left lane edge line and a right lane edge line. The first calculation module is further configured to determine that the target area is a non-drivable area when the left edge of the target area corresponds to the right lane edge line and / or the right edge of the target area corresponds to the left lane edge line.

[0122] In one embodiment of this application, the second calculation module is configured to determine that the initial ground point cloud does not contain non-ground point clouds when the first mapped area and the non-drivable area do not overlap; and to determine that the initial ground point cloud contains non-ground point clouds when the first mapped area and the non-drivable area overlap, and to determine the point cloud corresponding to the overlapping area as the non-ground point cloud in the initial ground point cloud.

[0123] In one embodiment of this application, the image processing unit 430 is further configured to determine a preset road traffic sign line in the traffic environment based on the two-dimensional image, the preset road traffic sign line being used to indicate a vehicle no-entry area on the road surface; and to obtain the vehicle no-entry area in the two-dimensional image based on the preset road traffic sign line.

[0124] Correspondingly, the drivable area detection unit 450 is also used to obtain a third mapping region of the ground point cloud in the two-dimensional image based on the calibration parameters between the roadside lidar and the roadside camera, wherein the ground point cloud is the final ground point cloud determined from the initial ground point cloud based on the recognition result; determine the final drivable area in the third mapping region other than the vehicle restricted area, and obtain the drivable area in the traffic environment based on the point cloud corresponding to the final drivable area.

[0125] In one embodiment of this application, the drivable area detection device 400 further includes a preprocessing unit for constructing a training dataset, the training dataset including training samples and labeled data. The training samples are obtained from historical two-dimensional images perceived by the roadside camera, and the labeled data includes the positional information of lane edge lines and preset road traffic signs in the training samples. The device also constructs a detection model, which includes an encoder, a DLA feature extraction network, and a decoder. The encoder determines the proximity affinity domains of lane edge lines and / or preset road traffic signs in the training samples based on the labeled data, and encodes the training samples based on the proximity affinity domains. The DLA feature extraction network extracts features, and the decoder decodes the features based on the labeled data and predicts lane edge lines and / or preset road traffic signs in the training samples based on the decoding results. Finally, the device trains the detection model based on the training samples and the labeled data of the training samples.

[0126] Correspondingly, the image processing unit 430 is also used to perform lane edge line detection and preset road traffic sign detection on the two-dimensional image using the trained detection model.

[0127] It is understood that the aforementioned drivable area detection device can implement all the steps of the drivable area detection method provided in the foregoing embodiments, and the relevant explanations regarding the drivable area detection method apply to drivable areas.

[0128] The driving area detection device will not be described in detail here.

[0129] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 5 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory.

[0130] The memory may include RAM, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for its operations.

[0131] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0132] Memory is used to store programs. Specifically, the program may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0133] The processor reads the corresponding computer program from non-volatile memory into memory and then runs it, forming a drivable area detection device at the logical level. The processor executes the program stored in memory and is specifically used to perform the following operations: 5. Acquire perception information of the traffic environment, including three-dimensional laser point clouds perceived by roadside lidar and two-dimensional images perceived by roadside cameras;

[0134] The initial ground point cloud and lane separation point cloud in the traffic environment are determined based on the three-dimensional laser point cloud;

[0135] The lane edge lines in the traffic environment are determined based on the two-dimensional image;

[0136] Based on the lane edge line and the lane isolation point cloud, identify whether the initial ground point cloud contains non-ground point clouds;

[0137] The drivable area in the traffic environment is obtained based on the identification results.

[0138] The above is as stated in this application. Figure 1The method executed by the drivable area detection device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in the memory, and the processor reads the information from the memory and, in conjunction with its hardware, completes the steps of the aforementioned drivable area detection method.

[0139] The electronic device can also perform Figure 1 The method for executing the drivable area detection device, and the implementation of the drivable area detection device in... Figure 1 The functions of the embodiments shown are not described in detail here.

[0140] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by an electronic device including multiple applications, enable the electronic device to perform... Figure 1 The method for detecting the drivable area shown in the embodiment will not be described again in this application.

[0141] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and the blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions.

[0143] Or a combination of boxes. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which are executable by the processor of the computer or other programmable data processing device, produce instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0144] 5. These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0145] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment, causing a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0146] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0147] 5. Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0148] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0149] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0150] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0151] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for detecting drivable areas, characterized in that, The method includes: Acquire traffic environment perception information, including three-dimensional laser point clouds perceived by roadside lidar and two-dimensional images perceived by roadside cameras; The initial ground point cloud and lane separation point cloud in the traffic environment are determined based on the three-dimensional laser point cloud; The lane edge lines in the traffic environment are determined based on the two-dimensional image; Based on the lane edge line and the lane isolation point cloud, identify whether the initial ground point cloud contains non-ground point clouds; The drivable area in the traffic environment is obtained based on the identification results; The step of identifying whether the initial ground point cloud contains non-ground point clouds based on the lane edge line and the lane isolation point cloud includes: The target region is obtained by dividing the two-dimensional image into regions using the lane edge lines; Based on the calibration parameters between the roadside lidar and the roadside camera, the first mapping region of the initial ground point cloud in the two-dimensional image and the second mapping region of the lane isolation point cloud in the two-dimensional image are obtained. The type of the lane edge line is determined based on the positional relationship between the lane edge line and the second mapping area, and the non-drivable area in the target area is determined based on the type of the lane edge line corresponding to the target area. The initial ground point cloud is identified as containing non-ground point clouds based on the positional relationship between the non-drivable area and the first mapped area.

2. The drivable area detection method as described in claim 1, characterized in that, Determining the type of the lane edge line based on the positional relationship between the lane edge line and the second mapping area includes: Determine the lane edge line located to the left of the second mapping area, and in order of increasing relative distance from the second mapping area, determine the adjacent lane edge lines located to the left of the second mapping area as the right lane edge line and the left lane edge line in sequence; Determine the lane edge line located to the right of the second mapping area, and in order of increasing relative distance from the second mapping area, determine the adjacent lane edge lines located to the right of the second mapping area as the left lane edge line and the right lane edge line.

3. The drivable area detection method as described in claim 1, characterized in that, The types of lane edge lines include left lane edge lines and right lane edge lines. Determining the non-drivable area within the target area based on the type of lane edge lines corresponding to the target area includes: When the left edge of the target area corresponds to the right lane edge line, and / or the right edge of the target area corresponds to the left lane edge line, the target area is determined to be a non-drivable area.

4. The drivable area detection method as described in claim 1, characterized in that, The step of identifying whether the initial ground point cloud contains non-ground point clouds based on the positional relationship between the non-drivable area and the first mapped area includes: When the first mapping area does not overlap with the non-driving area, it is determined that the initial ground point cloud does not contain non-ground point clouds. When the first mapped area overlaps with the non-drivable area, it is determined that the initial ground point cloud contains non-ground point clouds, and the point cloud corresponding to the overlapping area is determined as the non-ground point cloud in the initial ground point cloud.

5. The drivable area detection method as described in claim 1, characterized in that, When determining lane edge lines in the traffic environment based on the two-dimensional image, the method further includes: Based on the two-dimensional image, preset road traffic signs are determined in the traffic environment. The preset road traffic signs are used to indicate no-entry zones for vehicles on the road. The vehicle restricted area in the two-dimensional image is obtained based on the preset road traffic signs; After obtaining the drivable area in the traffic environment based on the recognition results, the method further includes: Based on the calibration parameters between the roadside lidar and the roadside camera, the third mapping region of the ground point cloud in the two-dimensional image is obtained, and the ground point cloud is the final ground point cloud determined from the initial ground point cloud based on the recognition result; Determine the final drivable area in the third mapping region, excluding the vehicle restricted area, and obtain the drivable area in the traffic environment based on the point cloud corresponding to the final drivable area.

6. The drivable area detection method as described in claim 5, characterized in that, The method further includes: A training dataset is constructed, which includes training samples and labeled data. The training samples are obtained from historical two-dimensional images perceived by the roadside camera, and the labeled data includes the position information of lane edge lines and road traffic marking lines in the training samples. A detection model is constructed, comprising an encoder, a DLA feature extraction network, and a decoder. The encoder is used to determine the proximity affinity domain of lane edge lines and / or preset road traffic signs in the training samples based on the labeled data, and to encode the training samples based on the proximity affinity domain. The DLA feature extraction network is used for feature extraction, and the decoder is used to decode the features based on the labeled data, and to predict lane edge lines and / or preset road traffic signs in the training samples based on the decoding results. The detection model is trained based on the training samples and the labeled data of the training samples. The trained detection model is used to detect lane edge lines and preset road traffic markings in the two-dimensional image.

7. A drivable area detection device, characterized in that, The device includes: The perception information acquisition unit is used to acquire perception information of the traffic environment, including three-dimensional laser point clouds perceived by roadside lidar and two-dimensional images perceived by roadside cameras. A point cloud processing unit is used to determine the initial ground point cloud and lane isolation point cloud in the traffic environment based on the three-dimensional laser point cloud. An image processing unit is configured to determine lane edge lines in the traffic environment based on the two-dimensional image; A non-ground point cloud recognition unit is used to identify whether the initial ground point cloud contains non-ground point clouds based on the lane edge line and the lane isolation point cloud. A drivable area detection unit is used to obtain drivable areas in the traffic environment based on the recognition results; The non-ground point cloud recognition unit is used for: The target region is obtained by dividing the two-dimensional image into regions using the lane edge lines; Based on the calibration parameters between the roadside lidar and the roadside camera, the first mapping region of the initial ground point cloud in the two-dimensional image and the second mapping region of the lane isolation point cloud in the two-dimensional image are obtained. The type of the lane edge line is determined based on the positional relationship between the lane edge line and the second mapping area, and the non-drivable area in the target area is determined based on the type of the lane edge line corresponding to the target area. The initial ground point cloud is identified as containing non-ground point clouds based on the positional relationship between the non-drivable area and the first mapped area.

8. An electronic device, comprising: processor; as well as A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the drivable area detection method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of applications, cause the electronic device to perform the drivable area detection method according to any one of claims 1 to 6.

Images