Method and device for determining drivable area, storage medium and electronic equipment

Abstract

The embodiment of the invention discloses a driving area determination method and device, a storage medium and electronic equipment, and the method comprises the steps: determining environment point cloud data and terminal attitude information; obtaining point cloud sequence information based on the environment point cloud data and the terminal attitude information; obtaining static dense point cloud data based on the environment point cloud data; obtaining dynamic dense point cloud data based on the dynamic point cloud data and the point cloud sequence information; and determining a drivable area based on the static dense point cloud data and the dynamic dense point cloud data. According to the embodiment of the invention, based on the dynamic dense point cloud data and the static dense point cloud data, compared with single-frame environment point cloud data, the static dense point cloud data and the dynamic dense point cloud data comprise more road surface information, and the credibility of the driving area determined based on the static dense point cloud data and the dynamic dense point cloud is higher; on the premise that manual marking is not needed, identification of the driving area with higher credibility is achieved.

Description

Technical Field

[0001] The present disclosure relates to the field of intelligent driving technology, and in particular to a method, device, storage medium and electronic device for determining a drivable area. Background Art

[0002] Road area estimation is a method used in intelligent driving scenarios to perceive the drivable and non-drivable areas in the driving scenario, and is used to assist mobile terminals in intelligent driving scenarios to better perceive and control. The perception of drivable areas, such as obstacle-free roads or flat fields; and the perceived curbs, general obstacles, dynamic objects and other areas are set as non-drivable areas. After the mobile terminal perceives the non-drivable area, it needs to perform avoidance control.

[0003] In the prior art, one way to annotate images using regional truth values ​​is manual annotation at the pixel level. This method mainly uses manual annotation, which requires a lot of manpower and needs to verify the accuracy of the manual annotation. Summary of the invention

[0004] In order to solve the above technical problems, the present disclosure is proposed. The embodiments of the present disclosure provide a method, device, storage medium and electronic device for determining a drivable area, so as to improve the labeling efficiency without relying on manpower.

[0005] According to one aspect of an embodiment of the present disclosure, a method for determining a drivable area is provided, comprising:

[0006] Determine environmental point cloud data and terminal posture information;

[0007] Based on the environmental point cloud data and the terminal posture information, obtaining point cloud sequence information;

[0008] Based on the environmental point cloud data, dynamic point cloud data and static point cloud data are obtained;

[0009] Based on the static point cloud data and the point cloud sequence information, obtaining static dense point cloud data;

[0010] Based on the dynamic point cloud data and the point cloud sequence information, dynamic dense point cloud data is obtained;

[0011] A drivable area is determined based on the static dense point cloud data and the dynamic dense point cloud data.

[0012] According to another aspect of an embodiment of the present disclosure, a device for determining a drivable area is provided, comprising:

[0013] Information collection module, used to determine environmental point cloud data and terminal posture information;

[0014] A point cloud sequence determination module, used to obtain point cloud sequence information based on the environmental point cloud data and the terminal posture information;

[0015] A point cloud segmentation module, used to obtain dynamic point cloud data and static point cloud data based on the environmental point cloud data;

[0016] A static point cloud densification module, used for obtaining static dense point cloud data based on the static point cloud data and the point cloud sequence information;

[0017] A dynamic point cloud densification module, used for obtaining dynamic dense point cloud data based on the dynamic point cloud data and the point cloud sequence information;

[0018] The area determination module is used to determine a drivable area based on the static dense point cloud data and the dynamic dense point cloud data.

[0019] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the method for determining a drivable area described in any of the above embodiments.

[0020] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, the electronic device comprising:

[0021] processor;

[0022] a memory for storing instructions executable by the processor;

[0023] The processor is used to read the executable instructions from the memory and execute the instructions to implement the method for determining the drivable area described in any one of the above embodiments.

[0024] Based on a method, device, storage medium and electronic device for determining a drivable area provided by the above-mentioned embodiments of the present disclosure, environmental point cloud data and terminal posture information are determined; point cloud sequence information is obtained based on the environmental point cloud data and the terminal posture information; dynamic point cloud data and static point cloud data are obtained based on the environmental point cloud data; static dense point cloud data are obtained based on the static point cloud data and the point cloud sequence information; dynamic dense point cloud data are obtained based on the dynamic point cloud data and the point cloud sequence information; and a drivable area is determined based on the static dense point cloud data and the dynamic dense point cloud data. This embodiment includes more road surface information in static dense point cloud data and dynamic dense point cloud data compared to single-frame environmental point cloud data, and the drivable area determined based on static dense point cloud data and dynamic dense point cloud has a higher degree of credibility; without the need for manual labeling, the identification of a drivable area with higher credibility is achieved.

[0025] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0027] Figure 1 is a flowchart of a method for determining a drivable area provided by an exemplary embodiment of the present disclosure;

[0028] Figure 2 It is a schematic diagram of a flow chart of determining a point cloud sequence in a method for determining a drivable area provided by an exemplary embodiment of the present disclosure;

[0029] Figure 3 It is a flowchart of point cloud segmentation in a method for determining a drivable area provided by an exemplary embodiment of the present disclosure;

[0030] Figure 4 is a schematic diagram of a flow chart of determining a drivable area in a drivable area determination method provided by an exemplary embodiment of the present disclosure;

[0031] Figure 5 is a structural schematic diagram of a drivable area determination device provided by an exemplary embodiment of the present disclosure;

[0032] Figure 6 is a structural schematic diagram of a drivable area determination device provided by another exemplary embodiment of the present disclosure;

[0033] Figure 7a is a structural schematic diagram of a point cloud segmentation module in a drivable area determination device provided by another exemplary embodiment of the present disclosure;

[0034] Figure 7b is a structural schematic diagram of a point cloud segmentation module in a drivable area determination device provided by another exemplary embodiment of the present disclosure;

[0035] Figure 7c is a structural schematic diagram of a point cloud densification module of a drivable area determination device provided by another exemplary embodiment of the present disclosure;

[0036] Figure 8 is a structural schematic diagram of a drivable area determination device provided by another exemplary embodiment of the present disclosure;

[0037] Fig. 9is a structural schematic diagram of a drivable area determination device provided by yet another exemplary embodiment of the present disclosure;

[0038] Fig.10 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] In order to explain the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure. It should be understood that the present disclosure is not limited to the example embodiments described herein.

[0040] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0041] Application Overview

[0042] In the process of realizing the present disclosure, the inventors found that most of the solutions for determining the true value of the executable area in the prior art are based on images to manually identify the executable area and the non-executable area, and mark them on the images. Due to the high pixel-level standard, the cost is high and the marking efficiency is low. At the same time, because it relies on manual operation, there is also the problem of labeling errors.

[0043] Exemplary Methods

[0044] Figure 1 FIG. 1 is a flow chart of a method for determining a drivable area provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as Figure 1 As shown, the following steps are included:

[0045] Step 102: Determine the environmental point cloud data and terminal posture information.

[0046] In this embodiment, environmental information and posture information can be collected within a preset time period by at least one sensor set on a mobile terminal (for example, a vehicle, an aircraft, a robot, etc.), to obtain at least one frame of environmental point cloud data and at least one frame of terminal posture information, and each frame of information corresponds to a time point within the preset time period. Optionally, point cloud data in the environment where the mobile terminal is located can be collected by a visual sensor (for example, a camera, a depth camera, etc.) set on the mobile terminal to obtain environmental point cloud data, and each point cloud in the environmental point cloud data includes three-dimensional coordinate information and color information; the terminal posture information can be obtained by a posture recognition device built into the mobile terminal (for example, an inertial measurement unit IMU, etc., which can directly obtain the position and posture information of the mobile terminal). Optionally, the terminal posture information may include but is not limited to: mobile terminal position information, mobile terminal direction information, etc.

[0047] Step 104: obtaining point cloud sequence information based on the environment point cloud data and the terminal posture information.

[0048] In one embodiment, the point cloud sequence information can be used to represent the position and posture changes of the mobile terminal within a preset time period. For example, the initial point cloud sequence information can be obtained based on at least one frame of terminal posture information, and then the initial point cloud sequence information can be updated in combination with at least one frame of environmental point cloud data.

[0049] Step 106: obtaining dynamic point cloud data and static point cloud data based on the environmental point cloud data.

[0050] In this embodiment, static point cloud data refers to point cloud data belonging to static objects in the environmental point cloud data; for example, a curb is identified, and the curb is static point cloud data in the environmental point cloud data; or, after dynamic point cloud data corresponding to a dynamic object in the environmental point cloud data is identified, the dynamic point cloud data is clipped and removed from the environmental point cloud data to obtain static point cloud data. For example, a dynamic object may be a pedestrian, a bicycle, etc.

[0051] Step 108: Obtain static dense point cloud data based on the static point cloud data and the point cloud sequence information.

[0052] In this embodiment, the distribution of point clouds in a single frame of environmental point cloud data is relatively sparse, and the description of objects in the environment can only be based on a small number of point clouds, which has a small amount of information and cannot accurately predict the environment. In this embodiment, static point cloud data and point cloud sequence information are combined to combine at least one frame of static point cloud data to obtain static dense point cloud data. Since static point cloud data within a preset time period are combined in the static dense point cloud data, more environmental information is included, and the accuracy of region recognition based on the static dense point cloud data is higher.

[0053] Step 110: obtaining dynamic dense point cloud data based on the dynamic point cloud data and the point cloud sequence information.

[0054] Optionally, similar to static point cloud data, a single-frame dynamic point cloud also has the problem of small amount of information. After dividing the point cloud data into dynamic point cloud data and static point cloud data, this embodiment can obtain dynamic dense point cloud data by combining at least one frame of dynamic point cloud data according to point cloud sequence information. Optionally, based on the dynamic point cloud data of a preset frame as a reference, at least one frame of dynamic point cloud data adjacent to the preset frame can be used as a scene flow with the dynamic point cloud data of the frame to realize the combination of multiple frames of dynamic point cloud data; or, based on the association between dynamic frames detected from multiple frames of environmental point cloud data, multiple frames of dynamic point cloud data can be combined.

[0055] Step 112: Determine a drivable area based on the static dense point cloud data and the dynamic dense point cloud data.

[0056] Optionally, the static dense point cloud data and dynamic dense point cloud data can be identified and classified through functions such as semantic classification, and the area in the point cloud data that belongs to a preset category (which can be set according to the scene, for example, road surface, lane line, etc.) is determined as a drivable area. Specifically, the static dense point cloud data and the dynamic dense point cloud data are identified and split through functions such as semantic classification to determine the true value of the drivable area, and the drivable area is determined based on the true value of the drivable area. By combining static dense point cloud data with dynamic dense point cloud data, the problem of inaccurate prediction caused by sparse information in single-frame environmental point cloud data is overcome. In other optional examples, a bird's-eye view can be obtained based on the dense point cloud, the true value of the drivable area can be marked in the bird's-eye view, and the drivable area can be determined based on the true value of the drivable area.

[0057] The above-mentioned embodiment of the present disclosure provides a method for determining a drivable area, which determines environmental point cloud data and terminal posture information; obtains point cloud sequence information based on the environmental point cloud data and the terminal posture information; obtains dynamic point cloud data and static point cloud data based on the environmental point cloud data; obtains static dense point cloud data based on the static point cloud data and the point cloud sequence information; obtains dynamic dense point cloud data based on the dynamic point cloud data and the point cloud sequence information; and determines a drivable area based on the static dense point cloud data and the dynamic dense point cloud data. This embodiment includes more road surface information in the static dense point cloud data and the dynamic dense point cloud compared to a single frame of environmental point cloud data based on the dynamic dense point cloud data and the static dense point cloud data, and the credibility of the drivable area determined based on the static dense point cloud data and the dynamic dense point cloud is higher; without the need for manual labeling, the identification of a drivable area with higher credibility is achieved.

[0058] like Figure 2 As shown in the above Figure 1 Based on the embodiment shown, the terminal posture information includes the location information of the mobile terminal and the direction information of the mobile terminal; step 104 may include the following steps:

[0059] Step 1041, determining the direction information and position information of each point cloud in the environmental point cloud data at the next moment according to the position information of the mobile terminal and the direction information of the mobile terminal corresponding to the environmental point cloud data.

[0060] Optionally, the location information and direction information of the mobile terminal at the current moment, combined with the speed of the mobile terminal, can determine the location information (time, direction and speed are known, and the distance moved in the corresponding direction is the time multiplied by the speed) and direction information (the direction does not change at consecutive moments) of the mobile terminal at the next moment. The direction information of the point cloud at the next moment is the opposite direction of the direction information of the mobile terminal at the next moment. The location information of the point cloud at the next moment is obtained by moving the corresponding distance from the location information at the current moment to the direction information of the point cloud at the current moment.

[0061] In some optional embodiments, the location information of the mobile terminal can be obtained based on the Global Navigation Satellite System (GNSS), and the obtained information can include real-time location and time information transmitted by satellite signals, which is used to calculate the geographical location information such as the longitude and latitude of the ground receiving device. The direction information of the mobile terminal can be obtained based on the inertial measurement unit (IMU) built into the mobile terminal. The inertial measurement unit is a device for measuring the motion state of an object. It integrates an accelerometer and a gyroscope, and can also include a magnetometer and a barometer to provide data about the acceleration, angular velocity, direction and altitude information of the object.

[0062] Step 1042, determining the path information of each point cloud in the environmental point cloud data according to the direction information and position information of each point cloud at the next moment.

[0063] In this embodiment, after determining the direction information and position information of each point cloud in a frame of environmental point cloud data at the next moment, the corresponding direction information can be searched at the corresponding position information in the next frame of environmental point cloud data based on the direction information and position information to obtain the corresponding point cloud, that is, to associate the point cloud at the previous moment with the point cloud at the next moment, and to associate one point cloud in all frames of environmental point cloud data in chronological order, so as to obtain the path information corresponding to the point cloud.

[0064] Step 1043, obtaining point cloud sequence information based on the path information of each point cloud in the environmental point cloud data.

[0065] In this embodiment, after obtaining the path information of each point cloud in the environmental point cloud data, the point cloud sequence information can be obtained by combination. Based on the point cloud sequence information, a more intuitive understanding of the inter-frame posture changes of the point cloud data can be achieved. Based on the inter-frame posture changes of the point cloud data, at least one frame of environmental data can be combined to obtain dense point cloud data.

[0066] In some optional embodiments, step 106 may include: performing state separation processing on the environmental point cloud data to obtain static point cloud data and dynamic point cloud data.

[0067] In this embodiment, the dynamic point cloud data and the static point cloud data in the environmental point cloud data can be determined by semantic recognition. Alternatively, after the dynamic object or the dynamic object is framed by the detection frame, the corresponding dynamic point cloud data and the static point cloud data are obtained; optionally, since the dynamic objects are usually less in the environment where the mobile terminal moves and have specific contours (for example, pedestrians, vehicles, etc.), the detection frame detection usually detects dynamic objects. Since the environmental point cloud data is obtained by combining the dynamic point cloud data and the static point cloud data, as long as one of them is obtained, the other can be obtained by removing it.

[0068] like Figure 3 As shown in the above Figure 1 Based on the illustrated embodiment, step 106 may include the following steps:

[0069] Step 1061, input the environmental point cloud data into a preset prediction model to obtain dynamic point cloud data.

[0070] Optionally, the preset prediction model may be a semantic prediction model for predicting point clouds, such as a lidar point cloud segmentation model (lidarseg). In this embodiment, semantic prediction is performed on each point cloud included in the environmental point cloud data based on the preset prediction model, and when the semantic prediction result is a preset result, the point cloud is determined to be a dynamic point cloud, and the dynamic point cloud data including all dynamic point clouds are removed from the environmental point cloud data to obtain static point cloud data.

[0071] Step 1062: Obtain static point cloud data based on the environmental point cloud data and the dynamic point cloud data.

[0072] This embodiment determines the semantics of each point cloud in the environmental point cloud data through semantic prediction. Since the scene includes a limited number of dynamic objects (for example, vehicles, pedestrians, etc.) and there are many static objects whose changes cannot be predicted, this embodiment presets multiple semantic results as dynamic objects, and determines the point clouds identified as these semantic results as dynamic point clouds. Compared with directly identifying static point cloud data, this embodiment has a higher recognition efficiency.

[0073] In some other optional embodiments, step 106 may also include:

[0074] a1. Use the preset detection model to determine the dynamic detection box included in the environmental point cloud data.

[0075] Optionally, the preset detection model can be a point cloud three-dimensional detection model to detect the point cloud data to obtain an object detection model of the detection frame. Each dynamic detection frame in this embodiment corresponds to a dynamic object (e.g., a vehicle, a pedestrian, etc.). Optionally, the dynamic detection frame is a bounding box for a dynamic object and can be any shape, for example, a rectangular frame, etc.

[0076] a2, based on the dynamic detection frame, determine the dynamic point cloud data from the environmental point cloud data.

[0077] Optionally, all point clouds included in the dynamic detection frame are selected from the environmental point cloud data based on the dynamic detection frame, and the point clouds can be determined as dynamic point cloud data in the environmental point cloud data.

[0078] a3. Determine static point cloud data based on environmental point cloud data and dynamic point cloud data.

[0079] In this embodiment, it can be understood that the environmental point cloud data is a combination of dynamic point cloud data and static point cloud data; when the dynamic point cloud data has been determined, all dynamic detection frames are removed from the environmental point cloud data, that is, the dynamic point cloud data in the environmental point cloud data is removed. In this embodiment, the dynamic objects in the environmental point cloud data are selected by the detection frame, and the selected dynamic point cloud data is removed to obtain the static point cloud data. Through the detection of the deep neural network, the speed and accuracy of obtaining the static point cloud are improved.

[0080] In some optional embodiments, step 108 may include:

[0081] b1, adjusting the terminal posture information based on the point cloud sequence information to obtain adjusted posture information.

[0082] In this embodiment, the posture information obtained based on the inertial measurement unit may have errors and is not very accurate. When the direction information and position information are determined based on the original posture information to establish a point cloud trajectory for each point cloud, when searching for the point cloud at the next time point for a point cloud, the point cloud may not be found based on the original direction information and position information. At this time, nearby point clouds are searched and the nearby point clouds are used as the next point cloud of the previous point cloud to form a point cloud trajectory, and point cloud sequence information is obtained based on all point cloud trajectories; the original direction information and position information are adjusted based on the point cloud sequence information to obtain more accurate posture information.

[0083] b2, based on the adjusted posture information, the static point cloud data is fused to obtain static dense point cloud data.

[0084] In this embodiment, an association relationship is established between each point cloud in at least one frame of static point cloud data based on the adjusted posture information, and a fusion and superposition is performed on at least one frame of static point cloud data based on the association relationship. Multiple frames of static point cloud information can be reflected in the same point cloud data. The point cloud data including multiple frames of static point cloud information is static dense point cloud data. Since the static dense point cloud data includes static point cloud data of all time points within a preset time period (including multiple frames of static point cloud data), the amount of information is a multiple of a single frame of environmental point cloud data. Therefore, the drivable area result determined based on the static dense point cloud is more accurate.

[0085] In some optional embodiments, step 110 may include:

[0086] c1, determine multiple scene flows based on the point cloud sequence information and the dynamic point cloud data group corresponding to the current frame dynamic point cloud data.

[0087] Among them, the dynamic point cloud data group includes the dynamic point cloud data of odd frames in the forward and backward directions centered on the dynamic point cloud data of the current frame. Scene flow (SF) is a 3D motion field defined for surface points in two consecutive scenes, which represents the spatial displacement of point clouds in the dynamic point cloud data between different frames. This embodiment takes the dynamic point cloud data of the current frame as a benchmark, and connects the dynamic point cloud data of other frames in the dynamic point cloud data group with the dynamic point cloud data of the current frame in a time series according to the point cloud sequence information to obtain at least one scene flow; based on at least one scene flow, the corresponding point cloud of each point cloud in each frame of dynamic point cloud data in other frames of dynamic point cloud data can be determined, and therefore, based on the scene flow, the dynamic point cloud data of multiple frames included in the dynamic point cloud data group can be fused and superimposed.

[0088] c2. Based on multiple scene flows, multiple frames of dynamic point cloud data included in the dynamic point cloud data group are fused to obtain dynamic dense point cloud data corresponding to the current frame of dynamic point cloud data.

[0089] In this embodiment, after determining multiple scene flows, multiple frames of dynamic point cloud data included in the dynamic point cloud data group are fused and superimposed. For example, the dynamic point cloud data group includes 5 frames of dynamic point cloud data: frame t-2, frame t-1, frame t, frame t+1, frame t+2, frame t is taken as the current frame dynamic point cloud data, frame t-2, frame t-1, frame t+1 and frame t+2 are respectively connected with frame t in time series to obtain 4 scene flows, and the 5 frames of dynamic point cloud data are fused to the current frame dynamic point cloud data (frame t) based on the scene flows, that is, the dynamic dense point cloud data corresponding to the current frame dynamic point cloud data is obtained.

[0090] In some optional embodiments, in addition to outputting the drivable area, the method provided by the present disclosure may also include:

[0091] d1, determining multiple scene flows based on the point cloud sequence information and the dynamic point cloud data group corresponding to the next frame of dynamic point cloud data.

[0092] Optionally, similar to the process of densifying dynamic point cloud data, this embodiment takes the next frame of dynamic point cloud data as a reference, and connects multiple frames of dynamic point cloud data adjacent to the next frame of dynamic point cloud data in a time series through point cloud sequence information to obtain multiple scene flows.

[0093] d2, based on multiple scene flows, multiple frames of dynamic point cloud data included in the dynamic point cloud data group are fused to obtain dynamic dense point cloud data corresponding to the next frame of dynamic point cloud data.

[0094] Similar to the above embodiment, multiple frames of dynamic point cloud data included in the dynamic point cloud data group corresponding to the next frame of dynamic point cloud data can be fused and superimposed based on the scene flow to obtain dynamic dense point cloud data corresponding to the next frame of dynamic point cloud data.

[0095] d3, based on the dynamic dense point cloud data corresponding to the current frame dynamic point cloud data and the dynamic dense point cloud data corresponding to the next frame dynamic point cloud data, a dense scene flow is obtained.

[0096] In this embodiment, it can be understood that after densification based on each frame of dynamic point cloud data, the dynamic dense point cloud data corresponding to each frame of dynamic point cloud data is obtained. For example, in this embodiment, the dynamic dense point cloud data of the current frame and the dynamic dense point cloud data of the next frame, and then the next frame of dynamic dense point cloud data is used as the dynamic dense point cloud data of the current frame, and the next frame of dynamic dense point cloud data corresponding to the dynamic dense point cloud data of the current frame is continued to be determined, and so on, multiple frames of dynamic dense point cloud data of consecutive frames can be obtained. At this time, the multiple frames of dynamic dense point cloud data are connected in time series to obtain the corresponding dense scene flow.

[0097] like Figure 4 As shown in the above Figure 1 Based on the illustrated embodiment, step 112 may include the following steps:

[0098] Step 1121, determining multiple frames of static dense point cloud data and dynamic dense point cloud data within a preset time period.

[0099] In this embodiment, multiple frames of static dense point cloud data corresponding to a preset time period are decomposed in chronological order to obtain multiple frames of static dense point cloud data; optionally, multiple frames of dynamic dense point cloud data corresponding to a preset time period are decomposed in chronological order to obtain multiple frames of dynamic dense point cloud data.

[0100] Step 1122: Obtain target point cloud data based on the static dense point cloud data and the dynamic dense point cloud data.

[0101] In this embodiment, when obtaining static point cloud data based on environmental point cloud data, the dynamic point cloud data in the environmental point cloud data is first determined. Therefore, the dynamic point cloud data corresponding to each frame of environmental point cloud data has been obtained. The obtaining method can be determined based on a preset prediction model or a preset detection model. When the dynamic dense point cloud data and the static dense point cloud data of each frame are known, the two are combined, and the obtained point cloud data realizes the embodiment of dense static point cloud and dense dynamic point cloud, so that the target point cloud data includes all the densified information within the preset time period.

[0102] Optionally, step 1122 includes: fusing static dense point cloud data with dynamic dense point cloud data to obtain target point cloud data.

[0103] Among them, the static dense point cloud data and the dynamic dense point cloud data correspond based on time.

[0104] In this embodiment, a correspondence is established between the static dense point cloud data and the dynamic dense point cloud data at the same moment through time, so as to obtain target point cloud data including static and dynamic dense point cloud data at each moment (corresponding to each frame) within a preset time period. The target point cloud data reflects the original environment point cloud data at the corresponding moment, and because it has been densified, it includes more information related to the frame in other frames.

[0105] Step 1123, determining a drivable area based on the semantic classification result corresponding to the target point cloud data.

[0106] In this embodiment, since each frame of target point cloud data includes densified dynamic and static dense point cloud data, that is, the information of dynamic and static objects included in the current moment is supplemented by combining with other frame environmental point clouds, so that the information of dynamic objects and static objects corresponding to each moment is more comprehensive. On this basis, the dynamic dense point cloud data is combined with the static dense point cloud data, which overcomes the problem of inaccurate prediction caused by sparse information in a single frame of environmental point cloud data.

[0107] Optionally, the method provided by the embodiment of the present disclosure may further include:

[0108] e1, performing discretization processing on the target point cloud data to obtain a plurality of voxels included in the target point cloud data.

[0109] In this embodiment, voxels are discretized units in three-dimensional space, and a plurality of voxels included in the target point cloud data can be determined through discretization processing.

[0110] e2, based on the dynamic dense point cloud data, determine the dynamic voxels in the target point cloud data corresponding to the dynamic point cloud; and / or, based on the scene flow and the target point cloud data, determine the voxel occupancy flow corresponding to the dynamic point cloud; and / or, based on the dense scene flow and the target point cloud data, determine the dense voxel occupancy flow corresponding to the dynamic point cloud.

[0111] The disclosed embodiment can reflect the occupation of dynamic objects in the environmental point cloud through dynamic voxels. Optionally, the 3D occupancy flow truth value (Occupancy flow truth value) of the single-frame target point cloud data can also be obtained based on the scene flow, wherein Occupancy is a three-dimensional representation, which divides the environment or a continuous object (corresponding to the multi-frame environmental point cloud data in this application) into a series of cubes (or voxels), and assigns a value to each voxel to indicate whether the voxel is occupied by an object (the part belongs to an object point or a non-object point); the 3D occupancy flow truth value in this embodiment represents the displacement of each voxel corresponding to the dynamic point cloud in the environmental point cloud data relative to the next frame, and optionally, the three-dimensional point cloud can also be reduced to BEV space (for example, directly setting the coordinate value of the z-axis to 0 to achieve dimensionality reduction to BEV space) to obtain the 2D occupancy flow truth value. Optionally, after obtaining the 3D occupancy flow true value and / or the 2D occupancy flow true value, a first preset network model (a network model that predicts the 3D occupancy flow true value or the 2D occupancy flow true value in the point cloud data) can be trained based on the 3D occupancy flow true value and / or the 2D occupancy flow true value to obtain the trained first preset network model.

[0112] Optionally, the disclosed embodiment can also obtain a densified 3D occupancy flow truth value (Occupancy flow truth value), that is, based on the dense scene flow, the displacement of the point cloud determined to be occupied by the dynamic object in the target point cloud data between adjacent frames is obtained; in addition, the densified 3D occupancy flow truth value can also be reduced to the BEV space to obtain a densified 2D occupancy flow truth value. Optionally, after obtaining the densified 3D occupancy flow truth value and / or the densified 2D occupancy flow truth value, a second preset network model (a network model that predicts the densified 3D occupancy flow truth value or the densified 2D occupancy flow truth value in the point cloud data) can be trained based on the densified 3D occupancy flow truth value and / or the densified 2D occupancy flow truth value to obtain the trained second preset network model.

[0113] The method for determining a drivable area provided by the above embodiment of the present disclosure reduces the manual annotation process, greatly reduces the annotation cost and improves the efficiency of generating true values. The drivable area determined by the method for determining a drivable area can be used as training annotation data for other deep neural network models.

[0114] Any method for determining a drivable area provided in the embodiments of the present disclosure may be executed by any appropriate device with data processing capabilities, including but not limited to: a terminal device and a server, etc. Alternatively, any method for determining a drivable area provided in the embodiments of the present disclosure may be executed by a processor, such as the processor executing any method for determining a drivable area mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.

[0115] Exemplary Devices

[0116] Figure 5 is a schematic diagram of the structure of a device for determining a drivable area provided by an exemplary embodiment of the present disclosure. Figure 5 As shown, the device provided in this embodiment includes:

[0117] The information collection module 51 is used to determine the environmental point cloud data and the terminal posture information.

[0118] The point cloud sequence determination module 52 is used to obtain point cloud sequence information based on the environmental point cloud data and the terminal posture information.

[0119] The point cloud segmentation module 53 is used to obtain dynamic point cloud data and static point cloud data based on the environmental point cloud data.

[0120] The static point cloud densification module 54 is used to obtain static dense point cloud data based on the static point cloud data and the point cloud sequence information.

[0121] The dynamic point cloud densification module 55 is used to obtain dynamic dense point cloud data based on the dynamic point cloud data and point cloud sequence information.

[0122] The area determination module 56 is used to determine a drivable area based on the static dense point cloud data and the dynamic dense point cloud data.

[0123] The above-mentioned embodiment of the present disclosure provides a device for determining a drivable area, which determines environmental point cloud data and terminal posture information; obtains point cloud sequence information based on the environmental point cloud data and the terminal posture information; obtains dynamic point cloud data and static point cloud data based on the environmental point cloud data; obtains static dense point cloud data based on the static point cloud data and the point cloud sequence information; obtains dynamic dense point cloud data based on the dynamic point cloud data and the point cloud sequence information; and determines a drivable area based on the static dense point cloud data and the dynamic dense point cloud data. This embodiment includes more road surface information in the static dense point cloud data and the dynamic dense point cloud compared to a single frame of environmental point cloud data based on the dynamic dense point cloud data and the static dense point cloud data, and the credibility of the drivable area determined based on the static dense point cloud data and the dynamic dense point cloud is higher; without the need for manual labeling, the identification of a drivable area with higher credibility is achieved.

[0124] Figure 6 FIG. 1 is a schematic diagram of a structure of a device for determining a drivable area provided by another exemplary embodiment of the present disclosure. Figure 6 As shown, the point cloud sequence determination module 52 includes:

[0125] The posture information determination unit 521 is used to determine the direction information and position information of each point cloud in the environmental point cloud data at the next moment according to the position information of the mobile terminal and the direction information of the mobile terminal corresponding to the environmental point cloud data.

[0126] The terminal posture information includes the location information of the mobile terminal and the direction information of the mobile terminal.

[0127] The path determination unit 522 is used to determine the path information of each point cloud in the environmental point cloud data according to the direction information and position information of each point cloud at the next moment.

[0128] The sequence determination unit 523 is used to obtain point cloud sequence information based on the path information of each point cloud in the environmental point cloud data.

[0129] Optionally, the point cloud segmentation module 53 is specifically used to perform state separation on the environmental point cloud data to obtain static point cloud data and dynamic point cloud data.

[0130] Figure 7a FIG. 1 is a schematic diagram of a structure of a device for determining a drivable area provided by another exemplary embodiment of the present disclosure. Figure 7a As shown, the point cloud segmentation module 53 includes:

[0131] The point cloud prediction unit 531 is used to input the environmental point cloud data into a preset prediction model to obtain dynamic point cloud data.

[0132] The first static point cloud unit 532 is used to obtain static point cloud data based on the environmental point cloud data and the dynamic point cloud data.

[0133] Figure 7b FIG. 1 is a schematic diagram of a structure of a device for determining a drivable area provided by another exemplary embodiment of the present disclosure. Figure 7b As shown, the point cloud segmentation module 53 includes:

[0134] The point cloud detection unit 533 is used to determine the dynamic detection box included in the environmental point cloud data by using a preset detection model.

[0135] The dynamic detection unit 534 is used to determine the dynamic point cloud data from the environmental point cloud data based on the dynamic detection frame.

[0136] The second static point cloud unit 535 is used to determine static point cloud data based on the environmental point cloud data and the dynamic point cloud data.

[0137] Figure 7c FIG. 1 is a schematic diagram of a structure of a device for determining a drivable area provided by another exemplary embodiment of the present disclosure. Figure 7c As shown, the static point cloud densification module 54 includes:

[0138] The posture adjustment unit 541 is used to adjust the terminal posture information based on the point cloud sequence information to obtain adjusted posture information.

[0139] The static point cloud fusion unit 542 is used to fuse the static point cloud data based on the adjusted posture information to obtain static dense point cloud data.

[0140] The dynamic point cloud densification module 55 includes:

[0141] The scene flow determination unit 551 is used to determine multiple scene flows based on the point cloud sequence information and the dynamic point cloud data group corresponding to the current frame dynamic point cloud data.

[0142] The dynamic point cloud data group includes the odd-numbered frame dynamic point cloud data centered on the current frame dynamic point cloud data.

[0143] The dynamic point cloud fusion unit 552 is used to fuse multiple frames of dynamic point cloud data included in the dynamic point cloud data group based on multiple scene streams to obtain dynamic dense point cloud data corresponding to the current frame of dynamic point cloud data.

[0144] Optionally, the dynamic point cloud densification module 55 is also used to determine multiple scene flows based on the point cloud sequence information and the dynamic point cloud data group corresponding to the next frame of dynamic point cloud data; fuse the multiple frames of dynamic point cloud data included in the dynamic point cloud data group based on the multiple scene flows to obtain the dynamic dense point cloud data corresponding to the next frame of dynamic point cloud data; and obtain the dense scene flow based on the dynamic dense point cloud data corresponding to the current frame of dynamic point cloud data and the dynamic dense point cloud data corresponding to the next frame of dynamic point cloud data.

[0145] Figure 8 FIG. 1 is a schematic diagram of a structure of a device for determining a drivable area provided by another exemplary embodiment of the present disclosure. Figure 8 As shown, the area determination module 56 includes:

[0146] The dense data acquisition unit 561 is used to determine multiple frames of static dense point cloud data and multiple frames of dynamic dense point cloud data within a preset time period.

[0147] The static-dynamic combining unit 562 is used to obtain target point cloud data based on the static dense point cloud data and the dynamic dense point cloud data.

[0148] The drivable area unit 563 is used to determine the drivable area based on the semantic classification result corresponding to the target point cloud data.

[0149] Optionally, the static-dynamic combining unit 562 is specifically used to fuse the static dense point cloud data with the dynamic dense point cloud data to obtain target point cloud data.

[0150] Optionally, the device provided in this embodiment may further include:

[0151] A discretization module 57 (not shown in the figure), used to perform discretization processing on the target point cloud data to obtain a plurality of voxels included in the target point cloud data;

[0152] The dynamic voxel module 58 (not shown in the figure) is used to determine the dynamic voxels corresponding to the dynamic point cloud in the target point cloud data based on the dynamic dense point cloud data; and / or, based on the scene flow and the target point cloud data, determine the voxel occupancy flow corresponding to the dynamic point cloud; and / or, based on the dense scene flow and the target point cloud data, determine the dense voxel occupancy flow corresponding to the dynamic point cloud.

[0153] Fig. 9 FIG. 1 is a schematic diagram of a structure of a device for determining a drivable area provided by another exemplary embodiment of the present disclosure. Fig. 9 As shown, the device provided in this embodiment includes:

[0154] In this embodiment, the information collection module 51 collects at least one frame of environmental point cloud data, at least one frame of GNSS data, and at least one frame of IMU data through at least one device set on the mobile terminal within a preset time period.

[0155] The posture information determination unit 521 performs a single reconstruction based on the position information of the mobile terminal and the direction information of the mobile terminal corresponding to each frame of the environmental point cloud data, and determines the direction information and position information of each point cloud in the environmental point cloud data at the next moment.

[0156] The path determination unit 522 determines the path information of each point cloud in the environmental point cloud data according to the direction information and position information of each point cloud at the next moment determined by the posture information determination unit 521 .

[0157] The sequence determination unit 523 obtains point cloud sequence information based on the path information of each point cloud in the environment point cloud data determined by the path determination unit 522 .

[0158] In this embodiment, the point cloud segmentation module 53 includes two branches, and the first branch includes:

[0159] The point cloud prediction unit 531 inputs the environmental point cloud data into a preset prediction model, determines the semantic information of each point cloud, and obtains dynamic point cloud data. The first static point cloud unit 532 obtains static point cloud data based on the environmental point cloud data and the dynamic point cloud data.

[0160] The second branch includes: the point cloud detection unit 533 uses a preset detection model to determine the dynamic detection frame included in the environmental point cloud data. The dynamic detection unit 534 determines the dynamic point cloud data from the environmental point cloud data based on the dynamic detection frame. The second static point cloud unit 535 determines the static point cloud data based on the environmental point cloud data and the dynamic point cloud data.

[0161] The posture adjustment unit 541 adjusts the terminal posture information based on the point cloud sequence information determined by the sequence determination unit 523 to obtain adjusted posture information.

[0162] The point cloud fusion unit 542 fuses the static point cloud data based on the adjusted posture information determined by the posture adjustment unit 541 to obtain static dense point cloud data.

[0163] The scene flow determination unit 551 determines a plurality of scene flows based on the point cloud sequence information determined by the sequence determination unit 523 and the dynamic point cloud data group corresponding to the dynamic point cloud data of the current frame.

[0164] The dynamic point cloud fusion unit 552 fuses the multiple frames of dynamic point cloud data included in the dynamic point cloud data group based on the multiple scene flows determined by the scene flow determination unit 551 to obtain dynamic dense point cloud data corresponding to the current frame of dynamic point cloud data.

[0165] The dense data acquisition unit 561 acquires the static dense point cloud data obtained by the point cloud fusion unit 542 and the dynamic dense point cloud data determined by the dynamic point cloud fusion unit 552 .

[0166] The dynamic-static combining unit 562 combines the dynamic dense point cloud data acquired by the dense data acquiring unit 561 with the static dense point cloud data to obtain target point cloud data.

[0167] The drivable area unit 563 determines the drivable area based on the semantic classification result corresponding to the target point cloud data obtained by the dynamic and static combining unit 562 .

[0168] The beneficial technical effects corresponding to the exemplary embodiment of the present device can be found in the corresponding beneficial technical effects of the above exemplary method section, which will not be repeated here.

[0169] Exemplary Electronic Devices

[0170] Fig.10 A structural diagram of an electronic device provided in an embodiment of the present disclosure includes at least one processor 11 and a memory 12.

[0171] The processor 11 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0172] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute one or more computer program instructions to implement the drivable area determination method and / or other desired functions of the various embodiments of the present disclosure described above.

[0173] In one example, the electronic device 10 may further include: an input device 13 and an output device 14, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0174] The input device 13 may also include, for example, a keyboard, a mouse, etc.

[0175] The output device 14 can output various information to the outside, and may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0176] Of course, to simplify, Fig.10 Only some of the components related to the present disclosure in the electronic device 10 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 10 may also include any other appropriate components.

[0177] Exemplary computer program products and computer-readable storage media

[0178] In addition to the above-mentioned methods and devices, embodiments of the present disclosure may also provide a computer program product, including computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the drivable area determination method of various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section.

[0179] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0180] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the drivable area determination method of various embodiments of the present disclosure described in the above "Exemplary Method" section.

[0181] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium is, for example, but not limited to, a system, device or device including electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0182] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and cannot be considered as necessary for each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, rather than limitation, and the above details do not limit the present disclosure to being implemented by adopting the above specific details.

[0183] Those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present disclosure claims and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A method for determining a drivable area, comprising: Determine environmental point cloud data and terminal posture information; Based on the environmental point cloud data and the terminal posture information, obtaining point cloud sequence information; Based on the environmental point cloud data, dynamic point cloud data and static point cloud data are obtained; Based on the static point cloud data and the point cloud sequence information, obtaining static dense point cloud data; Based on the dynamic point cloud data and the point cloud sequence information, dynamic dense point cloud data is obtained; A drivable area is determined based on the static dense point cloud data and the dynamic dense point cloud data.

2. The method according to claim 1, wherein obtaining dynamic point cloud data and static point cloud data based on the environmental point cloud data comprises: The environmental point cloud data is state-separated to obtain static point cloud data and dynamic point cloud data.

3. The method according to claim 2, wherein: The state separation of the environmental point cloud data to obtain static point cloud data and dynamic point cloud data includes: Inputting the environmental point cloud data into a preset prediction model to obtain the dynamic point cloud data; The static point cloud data is obtained based on the environmental point cloud data and the dynamic point cloud data.

4. The method according to claim 2, wherein: The state separation of the environmental point cloud data to obtain static point cloud data and dynamic point cloud data includes: Determine a dynamic detection frame included in the environmental point cloud data by using a preset detection model; Based on the dynamic detection frame, determining dynamic point cloud data from the environmental point cloud data; The static point cloud data is determined based on the environmental point cloud data and the dynamic point cloud data.

5. The method according to claim 1, wherein: The obtaining of the static dense point cloud data based on the static point cloud data and the point cloud sequence information comprises: Adjusting the terminal posture information based on the point cloud sequence information to obtain adjusted posture information; The static point cloud data is fused based on the adjusted posture information to obtain the static dense point cloud data.

6. The method according to claim 1, wherein: The terminal posture information includes the location information of the mobile terminal and the direction information of the mobile terminal; The step of obtaining point cloud sequence information based on the environment point cloud data and the terminal posture information includes: Determine the direction information and the position information of each point cloud in the environmental point cloud data at the next moment according to the position information of the mobile terminal and the direction information of the mobile terminal corresponding to the environmental point cloud data; Determining path information of each point cloud in the environmental point cloud data according to direction information and position information of each point cloud at the next moment; The point cloud sequence information is obtained based on the path information of each point cloud in the environmental point cloud data.

7. The method according to any one of claims 1 to 6, wherein: Based on the dynamic point cloud data and the point cloud sequence information, dynamic dense point cloud data is obtained, including: Determine multiple scene flows based on the point cloud sequence information and the dynamic point cloud data group corresponding to the current frame dynamic point cloud data; the dynamic point cloud data group includes odd-numbered frame dynamic point cloud data centered on the current frame dynamic point cloud data; Based on the multiple scene flows, multiple frames of dynamic point cloud data included in the dynamic point cloud data group are fused to obtain the dynamic dense point cloud data corresponding to the current frame of dynamic point cloud data.

8. The method according to claim 7, wherein: Also includes: Determine a plurality of scene flows based on the point cloud sequence information and a dynamic point cloud data group corresponding to a next frame of dynamic point cloud data; Based on the multiple scene streams, multiple frames of dynamic point cloud data included in the dynamic point cloud data group are merged to obtain the dynamic dense point cloud data corresponding to the next frame of dynamic point cloud data; A dense scene flow is obtained based on the dynamic dense point cloud data corresponding to the current frame dynamic point cloud data and the dynamic dense point cloud data corresponding to the next frame dynamic point cloud data.

9. The method according to any one of claims 1 to 8, wherein: The determining of a drivable area based on the static dense point cloud data and the dynamic dense point cloud data includes: Determine a plurality of frames of the static dense point cloud data and a plurality of frames of the dynamic dense point cloud data within a preset time period; Based on the static dense point cloud data and the dynamic dense point cloud data, obtaining target point cloud data; The drivable area is determined based on the semantic classification result corresponding to the target point cloud data.

10. The method according to claim 9, wherein: The obtaining target point cloud data based on the static dense point cloud data and the dynamic dense point cloud data comprises: The static dense point cloud data is fused with the dynamic dense point cloud data to obtain the target point cloud data.

11. The method according to claim 9, wherein: The method further comprises: Performing discretization processing on the target point cloud data to obtain a plurality of voxels included in the target point cloud data; Determine the dynamic voxels of the target point cloud data corresponding to the dynamic point cloud based on the dynamic dense point cloud data; and / or, determine the voxel occupancy flow corresponding to the dynamic point cloud based on the scene flow and the target point cloud data; and / or, determine the dense voxel occupancy flow corresponding to the dynamic point cloud based on the dense scene flow and the target point cloud data.

12. A device for determining a drivable area, comprising: Information collection module, used to determine environmental point cloud data and terminal posture information; A point cloud sequence determination module, used to obtain point cloud sequence information based on the environmental point cloud data and the terminal posture information; A point cloud segmentation module, used to obtain dynamic point cloud data and static point cloud data based on the environmental point cloud data; A static point cloud densification module, used for obtaining static dense point cloud data based on the static point cloud data and the point cloud sequence information; A dynamic point cloud densification module, used for obtaining dynamic dense point cloud data based on the dynamic point cloud data and the point cloud sequence information; The area determination module is used to determine a drivable area based on the static dense point cloud data and the dynamic dense point cloud data.

13. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the method for determining a drivable area according to any one of claims 1 to 11.

14. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the method for determining the drivable area described in any one of claims 1-11 above.

Images