Automatic driving semantic trafficability classification method and system based on point cloud data

Abstract

The invention provides a semantic trafficability classification method and a semantic trafficability classification system based on point cloud data. The semantic trafficability classification method comprises the following steps: acquiring multiple frames of point cloud data; rasterizing each frame of point cloud data according to the x-y coordinates in the two-dimensional plane to generate a columnar volume structure of each frame of point cloud data; extracting a plurality of column features of the columnar volume structure of each frame of point cloud data; correspondingly fusing the plurality of column features of the multi-frame point cloud data to obtain fused features; and carrying out semantic trafficability classification on the fused features to obtain a final trafficability map. According to the method, the environment trafficability information can be efficiently extracted from the point cloud data, the semantic trafficability map is generated, and the method is suitable for real-time environment perception of automatic driving and robot navigation. The method is low in calculation overhead, can efficiently solve the sparsity problem of the point cloud data, and is suitable for various tasks.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving and robot navigation, and in particular to a semantic accessibility classification method and system based on point cloud data. Background Art

[0002] Autonomous driving technology has been one of the research hotspots in the field of artificial intelligence in recent years. Its core goal is to achieve autonomous perception, decision-making and control of vehicles to ensure driving safety and improve traffic efficiency. With the development of artificial intelligence and sensor technology, especially the advancement of deep learning and lidar technology, environmental perception technology has gradually become an indispensable part of autonomous driving systems.

[0003] Environmental perception technology relies on external environmental data collected by sensors. The main sensors currently include cameras and lidar. Cameras can provide high-resolution visual information, but they are easily affected by external lighting and weather conditions and are difficult to work stably at night or in harsh environments. In contrast, lidar generates a three-dimensional point cloud by emitting lasers and receiving reflected signals. It can provide high-precision geometric and distance information and is not limited by lighting conditions. Therefore, lidar has been widely used in autonomous vehicles for navigation, obstacle detection, and path planning.

[0004] The 3D point cloud data generated by LiDAR provides accurate geometric information for driverless vehicles, which can help vehicles understand the surrounding environment. However, there are many challenges in point cloud data processing. Point cloud data is often sparse, especially in long-distance areas, where the density of data points is low, which affects the accuracy of environmental perception. In addition, point clouds lack semantic information such as color and texture, which makes point cloud-based environmental perception tasks more complicated.

[0005] Scene semantic understanding of 3D point clouds is an important component of unmanned driving systems. The purpose of scene semantic understanding is to classify each point in the point cloud data according to its semantic category, thereby providing vehicles with information about the traversability of the environment. This task has made some progress in urban scenes, but it is still difficult to apply it to complex terrain in the wild. The wild environment is complex and changeable, with undulating terrain and many obstacles such as cliffs, trees, and deep pits. Traditional point cloud processing methods based on geometric features are difficult to cope with these complex scenes.

[0006] When unmanned vehicles are performing autonomous navigation, accurate traversability classification is the key to ensuring safe driving of the vehicle. The traversability classification task requires the vehicle to be able to determine which terrain is passable and which is impassable. Existing research mainly focuses on structured urban scenes, while there are few studies on traversability classification in wild scenes. Due to the fuzzy semantic boundaries and complex scene changes in wild scenes, designing a traversability classification method that can efficiently process sparse point clouds and is applicable to wild scenes has become a major challenge facing unmanned driving technology. Summary of the invention

[0007] In view of the defects in the prior art, the object of the present invention is to provide a method and system for semantic accessibility classification of autonomous driving based on point cloud data.

[0008] According to one aspect of the present invention, a semantic accessibility classification method based on point cloud data is provided, comprising:

[0009] Get multi-frame point cloud data;

[0010] Performing rasterization processing on each frame of the point cloud data according to the xy coordinates in the two-dimensional plane to generate a columnar volume structure of each frame of the point cloud data;

[0011] Extracting a plurality of column features of the columnar volume structure of each frame of the point cloud data;

[0012] Correspondingly fusing the plurality of column features of the plurality of frames of the point cloud data to obtain a fused feature;

[0013] The fused features are subjected to semantic accessibility classification to obtain a final accessibility map.

[0014] Preferably, the step of acquiring multi-frame point cloud data includes:

[0015] Acquire multi-frame point cloud data of the surrounding environment of a vehicle or robot through a lidar sensor;

[0016] Each frame of point cloud data includes three-dimensional coordinates (x, y, z) and reflection intensity value r, denoted as Where N is the number of points in the point cloud.

[0017] Preferably, the rasterizing of each frame of the point cloud data according to the xy coordinates in the two-dimensional plane to generate a columnar volume structure of each frame of the point cloud data includes:

[0018] Dividing the point cloud data into uniform two-dimensional grids in the x and y directions;

[0019] Project each point cloud in the two-dimensional grid into a column, and record the number of point clouds in the jth column as Pj.

[0020] Set the number of point clouds in each column to M;

[0021] Zero-fill the columns with less than M point clouds, and randomly sample the columns with more than M point clouds to obtain a fixed-size point cloud set. All the point cloud sets of fixed size are the columnar volume structure of the point cloud data of this frame.

[0022] Preferably, the extracting of the plurality of column features of the columnar volume structure of each frame of the point cloud data comprises:

[0023] Each point cloud feature vector in each column is embedded into a high-dimensional feature:

[0024] f(x i ,y i ,z i ,r i )=ReLU(BN(W·(x i ,y i , z i ,r i ) T ))

[0025] Where W is the linear transformation matrix, BN is the batch normalization operation, and ReLU is the nonlinear activation function;

[0026] The point cloud features within the column are aggregated through the maximum pooling operation to obtain the column features;

[0027] P j =max i=1 M f(x i ,y i , z i ,r i );

[0028] 1 of them M represents the maximum pooling operation, and j represents the jth column.

[0029] Preferably, the step of correspondingly fusing the plurality of column features of the plurality of frames of the point cloud data to obtain a plurality of fused features comprises:

[0030] By transforming the matrix H k , align the column features of the k-frame point cloud data to the coordinate system of the current frame:

[0031]

[0032] The features of all aligned frames are weighted and fused through the attention mechanism to obtain the fused features:

[0033]

[0034] where f s and f c are the spatial and channel attention networks respectively, σ is the Softmax function; W s is the spatial attention weight, W c is the channel attention weight.

[0035] Preferably, the performing semantic accessibility classification on the fused features to obtain a final accessibility map includes:

[0036] The encoder is used to perform layer-by-layer convolution and downsampling to generate multi-layer feature maps, specifically:

[0037] Generate a 1 / 4 resolution feature map through a 64-channel dilated convolution block;

[0038] Generate a 1 / 8 resolution feature map through two 128-channel dilated convolution blocks;

[0039] Through multiple 256-channel dilated convolution blocks, a 320-channel feature map with 1 / 16 resolution is generated in the final layer;

[0040] The decoder is used to restore the feature resolution of the encoder by upsampling layer by layer, and generate the final feasibility map, which is specifically:

[0041] Applying 1×1 convolution to the feature map with 1 / 16 resolution and the feature map with 1 / 8 resolution to obtain a feature map with 128 channels;

[0042] By upsampling and splicing, the feature map with a resolution of 1 / 16 and the feature map with a resolution of 1 / 8 are merged to generate a new feature map M′1 / 8, where M′ represents the feature map newly obtained by upsampling and splicing;

[0043] Apply 3×3 convolution to the feature map M′1 / 8 to generate a 64-channel feature map, then upsample and concatenate it with the 1 / 4 resolution feature map to finally generate a new feature map M′1 / 4;

[0044] The M′1 / 4 is subjected to 3×3 convolution and 1×1 convolution to generate a 5-channel output as the final feasibility map.

[0045] Preferably, the dilated convolution block performs the following operations on the input feature map:

[0046] Extract multi-scale features of the input feature map through group convolution and different dilation rates;

[0047] Normalizing and activating the multi-scale features;

[0048] For the normalized and activated multi-scale features, a reduction rate of 1 / 4 is used to enhance the feature selection capability.

[0049] According to a second aspect of the present invention, there is provided a semantic accessibility classification system based on point cloud data, comprising:

[0050] Perception module: obtain multi-frame point cloud data;

[0051] Preprocessing module: rasterizing each frame of the point cloud data according to the xy coordinates in the two-dimensional plane to generate a columnar volume structure of each frame of the point cloud data;

[0052] Feature extraction module: extracting a plurality of column features of the columnar volume structure of each frame of the point cloud data;

[0053] Fusion module: fusing the plurality of column features of the plurality of frames of point cloud data accordingly to obtain fusion features;

[0054] Classification module: performs semantic accessibility classification on the fused features to obtain the final accessibility map.

[0055] According to a third aspect of the present invention, there is provided a terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute any one of the methods described, or to run the system described, when executing the program.

[0056] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to execute any of the methods described above, or to run the system described above.

[0057] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0058] The semantic accessibility classification method and system based on the fast attention framework in the embodiments of the present invention can efficiently extract the accessibility information of the environment from point cloud data and generate a semantic accessibility map, which is suitable for real-time environmental perception of autonomous driving and robot navigation.

[0059] The semantic accessibility classification method and system based on the fast attention framework in the embodiment of the present invention has low computational overhead. Compared with the traditional 3D convolution method, the embodiment of the present invention adopts a 2D convolution structure, which greatly reduces the computational complexity and is suitable for real-time system applications.

[0060] The semantic accessibility classification method and system based on the fast attention framework in the embodiment of the present invention can efficiently process sparse data. The sparsity problem of point cloud data is effectively processed through the spatiotemporal attention mechanism, which improves the accuracy of accessibility assessment of long-distance areas.

[0061] The semantic accessibility classification method and system based on the fast attention framework in the embodiment of the present invention have strong applicability, and can be applied to complex environment perception tasks in various autonomous driving and robot navigation scenarios, and have broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0063] Figure 1 Flow chart of a method for classifying semantic accessibility of autonomous driving based on point cloud data in one embodiment of the present invention;

[0064] Figure 2 A schematic diagram of a process for extracting column features in a preferred embodiment of the present invention;

[0065] Figure 3 A schematic diagram of a feature fusion process in a preferred embodiment of the present invention;

[0066] Figure 4 A process in a decoder in a preferred embodiment of the present invention;

[0067] Figure 5 The process in the dilated convolution block in a preferred embodiment of the present invention;

[0068] Figure 6 A schematic diagram of a semantic accessibility map in the form of a two-dimensional bird's-eye view in a preferred embodiment of the present invention;

[0069] Figure 7 Comparison of the classification effects of the single-frame model in a specific embodiment of the present invention and BEVNet-S on the SemanticKITTI dataset (above) and the RELLIS-3D dataset (below);

[0070] Figure 8 This is a comparison of the classification effects of the multi-frame model in a specific embodiment of the present invention and BEVNet-R on the SemanticKITTI dataset (above) and the RELLIS-3D dataset (below). DETAILED DESCRIPTION

[0071] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0072] like Figure 1 As shown, in one embodiment of the present invention, a method for classifying semantic traversability of autonomous driving based on point cloud data is provided, comprising:

[0073] S1, obtain multi-frame point cloud data;

[0074] S2, rasterizes each frame of point cloud data obtained in S1 according to the xy coordinates in the two-dimensional plane to generate a columnar volume structure of each frame of point cloud data;

[0075] S3, extracting multiple column features of the columnar volume structure of each frame of point cloud data obtained in S2;

[0076] S4, fusing the plurality of column features of the multi-frame point cloud data obtained in S3 accordingly to obtain a fused feature;

[0077] S5, performs semantic accessibility classification on the fused features obtained in S4 to obtain the final accessibility map.

[0078] The above-mentioned embodiments can efficiently extract the accessibility information of the environment from the point cloud data and generate a semantic accessibility map, which is suitable for real-time environment perception of autonomous driving and robot navigation.

[0079] In conventional applications, lidar sensors are usually configured on self-driving cars or robots to collect three-dimensional point cloud data of their surrounding environment. In a preferred embodiment of the present invention, the point cloud data is expressed by the three-dimensional spatial coordinates (x, y, z) of each point and the reflection intensity information (r). This expression enables the construction of a high-resolution three-dimensional model of the vehicle's environment. The collected point cloud data not only covers the distance information of surrounding objects, but also reveals their shape and position, so that the system can accurately identify obstacles and terrain features in the environment.

[0080] In some specific embodiments, the collected point cloud data is represented as: N is the number of points in the point cloud.

[0081] In order to optimize the subsequent computing efficiency and classification accuracy, the point cloud data needs to be preprocessed. Figure 1As shown in Figure 1, the preprocessing is as follows: rasterize the point cloud data in a two-dimensional plane (x, y direction), divide the environment into multiple fixed-size cylindrical volumes (cylinders), and each cylinder contains a set of point cloud data in the area. The number of point clouds in the jth column is denoted as p j .

[0082] Since the number of points in each column may be uneven, the fixed number of points is set to M. For columns with less than M points, a zero-filling strategy is used to maintain the consistency of the data format; for columns with more than M points, a random sampling method is used to avoid excessive computational burden. Finally, the number of points in each column is normalized to a fixed number. After normalization, the point cloud set of the jth column is represented as Where M is the predefined maximum number of points within a bar.

[0083] In some other embodiments, before the aforementioned rasterization process, a noise reduction process may be performed on the point cloud data to eliminate potential perceptual noise and ensure that the generated accessibility map has a higher accuracy.

[0084] In the aforementioned embodiment, by applying a two-dimensional convolution structure, the computational complexity is significantly reduced compared to the traditional three-dimensional convolution technology, making it more suitable for application in real-time systems.

[0085] After preprocessing, it is necessary to extract features from the point cloud data in each column. In a preferred embodiment of the present invention, a PointNet model is used to calculate the point cloud data in each column point by point. Figure 2 As shown in the figure, PointNet is a network structure that can effectively process irregular point cloud data. It extracts high-dimensional features through point-by-point calculations and supports aggregating information at local and global scales.

[0086] Specifically, the point cloud data in each cylinder generates high-dimensional features through a series of linear transformations and nonlinear activation operations. These features can represent the geometric information of each point in space and its relationship with surrounding points. In order to further improve the expressiveness of the features, all points in each cylinder are normalized and processed by activation functions to generate the global features of the cylinder. This global feature calculation method ensures that the feature information of each cylinder can be effectively aggregated, and the final global features will be used for subsequent classification.

[0087] In some specific implementations, the process of collecting the features of the point cloud data inside the column can be expressed as follows:

[0088] f(x i ,y i , z i ,r i )=ReLU(BN(W·(x i,y i , z i ,r i ) T ))

[0089] Where W is the linear transformation matrix and ReLU is the activation function. Perform the maximum pooling operation on the features within the cylinder to obtain the global feature representation of the cylinder:

[0090] P j =maxi=1 M f(x i ,y i , z i ,r i )

[0091] The above embodiments disclose methods for extracting global features of columnar structures. The key is to maximize the use of local characteristics of point cloud data and construct a representative global description inside the columnar structure through pooling operations. In these embodiments, the entire classification method can effectively cope with point cloud data of different resolutions, thereby enhancing its robustness.

[0092] After feature extraction, the sparse nature of the sparse point cloud has not changed completely. Therefore, in a preferred embodiment, a spatiotemporal attention mechanism is deliberately introduced to solve the sparsity problem of point cloud data and enhance the adaptability to environmental changes. Figure 3 As shown in the figure, the information of multiple frames of point cloud data is fused to provide a more comprehensive environmental perception in space and time. Specifically:

[0093] First, the multi-frame point cloud data is aligned according to the coordinate transformation. This alignment process ensures that each frame of data is processed in the same coordinate system, effectively avoiding feature deviation caused by vehicle movement or perspective changes.

[0094] Next, the aligned data is weighted and fused through the spatial attention mechanism and the channel attention mechanism. The spatial attention mechanism dynamically adjusts its weight according to the position changes of the column in different frames, thereby ensuring that the system can focus on those important areas. The channel attention mechanism weights various types of passability features according to the semantic information of the column features, so that the system can better distinguish different types of terrain information.

[0095] In some specific implementations, there are K frames of point cloud data, and the cylindrical features of each frame are The spatiotemporal feature alignment and fusion are performed through the following steps: Figure 3 As shown:

[0096] Coordinate alignment: using transformation matrix H k Align the features of each frame:

[0097]

[0098] Weighted fusion: using spatial attention weights W s and channel attention weight W c Weighted fusion of multi-frame features:

[0099]

[0100] Further, more specifically, in some other specific embodiments, a spatiotemporal attention mechanism is introduced to solve the sparsity problem of point cloud data and enhance the adaptability to environmental changes. The detailed steps are as follows: Figure 3 As shown:

[0101] F1 and F3 represent the features of other frames, and F2 represents the features of the current frame.

[0102] First, perform feature alignment, average pooling, and normalized softmax operations on F1 and F3 respectively;

[0103] Multiply the original F1, F3 and the normalized features element by element to obtain the fused features F1' and F3' respectively;

[0104] Perform average pooling and maximum pooling on F1'(F3'), and fuse and normalize the two pooling results;

[0105] Multiply the normalized result of F1'(F3') by F1'(F3') itself element by element;

[0106] Then, the element-by-element multiplication results of the two F1' and F3' are fused and finally fused with the current frame F2.

[0107] In the above embodiment, the spatiotemporal attention mechanism can dynamically adjust the attention weight according to the timestamp information of different frames, effectively deal with the sparsity problem of point cloud data, and improve the accuracy of the accessibility assessment of long-distance areas.

[0108] After feature extraction and fusion, the obtained column features are classified according to semantic accessibility. In a preferred embodiment of the present invention, Figure 3 As shown in the figure, the semantic accessibility classification adopts an encoder-decoder architecture, which can complete the feature map and generate a bird's-eye view (BEV) accessibility map. Its specific structure includes the following main parts:

[0109] Encoder: The encoder module consists of multiple dilated convolutional blocks with different dilation rate combinations (d1, d2) and strides, as shown in Table 1.

[0110]

[0111] The encoder generates multi-layer feature maps through layer-by-layer convolution and downsampling, as shown below:

[0112] It starts with a 64-channel dilated convolution block and a 96-channel dilated convolution block to generate a 1 / 4 resolution feature map.

[0113] Then, two 128-channel dilated convolution blocks are used to generate a 1 / 8 resolution feature map.

[0114] Finally, multiple 256-channel dilated convolution blocks are passed, and a 320-channel feature map with a resolution of 1 / 16 is generated in the final layer.

[0115] Through the encoder composed of the above multiple dilated convolution blocks, a total of 18 layers of output (1+1+2+1+1+1+4+6+1) can be obtained. Among them, the second layer outputs a feature map with a resolution of 1 / 4, the fifth layer outputs a feature map with a resolution of 1 / 8, and the final layer outputs a feature map with a resolution of 1 / 16.

[0116] Decoder: The decoder module restores the feature resolution through layer-by-layer upsampling operations and generates the final feasibility map. Figure 4 As shown, the specific steps include:

[0117] Apply 1×1 convolution on the M1 / 16 and M1 / 8 feature maps to obtain a 128-channel feature map. M1 represents the feature map obtained by the encoder.

[0118] The new feature map M′1 / 8 is generated by merging M1 / 16 and M1 / 8 through upsampling and accumulation operations. M′ represents the new feature map after merging.

[0119] A 3×3 convolution is applied on M′1 / 8 to generate a 64-channel feature map, which is then upsampled and concatenated with the M1 / 4 feature map to finally generate M′1 / 4.

[0120] M′1 / 4 is subjected to 3×3 convolution and 1×1 convolution to generate a 5-channel output as the final feasibility map.

[0121] In some specific embodiments, Figure 5 As shown in FIG. 1 , the structure of the dilated convolution block D block (left 1stride = 2 right 1stride = 1) is used in the encoder, which can be compared with Table 1 above. In the dilated convolution block, Figure 5 As shown in the figure, each dilated convolution block extracts multi-scale features through grouped convolution and different dilation rates. Each convolution layer is followed by batch normalization (BatchNorm) and ReLU activation function, and a Squeeze-and-Excitation network (compression and excitation module) is used to enhance feature selection capabilities with a reduction rate of 1 / 4.

[0122] The final output of semantic accessibility classification is the semantic category of each column, including free passage, low-cost passage, medium-cost passage, and impassable. Free passage areas represent terrain that vehicles can pass safely, usually marked in green; low-cost passage areas are suitable for cautious driving and are marked in yellow; medium-cost passage areas may have obstacles and are marked in orange; impassable areas include dangerous terrain such as rocks and deep pits and are marked in red. This color-based accessibility classification can help autonomous vehicles or robots to accurately navigate and select paths in complex environments.

[0123] Figure 6 The system generates a two-dimensional bird's-eye view of the passability map, marking the passability areas in the environment with different colors. This map can not only reflect the passability status of the vehicle's surrounding environment in real time, but also provide a reliable basis for subsequent path planning and decision-making.

[0124] Based on the same inventive concept, in another embodiment of the present invention, according to the second aspect of the present invention, a semantic accessibility classification system based on point cloud data is provided, comprising:

[0125] Perception module: obtain multi-frame point cloud data;

[0126] Preprocessing module: rasterizing each frame of point cloud data according to the xy coordinates in the two-dimensional plane to generate a columnar volume structure of each frame of point cloud data;

[0127] Feature extraction module: extracting multiple column features of the columnar volume structure of each frame of point cloud data;

[0128] Fusion module: fusing the plurality of column features of the multi-frame point cloud data accordingly to obtain fusion features;

[0129] Classification module: performs semantic accessibility classification on the fused features to obtain the final accessibility map.

[0130] The modules / units in the above examples of the present invention may specifically refer to the implementation techniques of the corresponding steps of the method for evaluating quality of a reference light field image without reference based on deep learning in the above embodiments, which will not be described in detail here.

[0131] In order to verify the feasibility and effectiveness of the semantic accessibility classification method and system for autonomous driving based on point cloud data in the above-mentioned embodiment, in a specific implementation of the present invention, extensive experiments were carried out based on two datasets, SemanticKITTI and RELLIS-3D, focusing on comparing the classification performance and real-time performance of the method of the embodiment of the present invention with BevNet in different scenarios.

[0132] Experimental conditions

[0133] Hardware configuration: The experiments were conducted on a computer with an NVIDIA RTX1080 GPU, 32GB memory, and an Intel i9 processor to ensure that the processing speed can meet the real-time requirements of the autonomous driving environment.

[0134] Comparison method: BevNet is used as the comparison baseline method to test its classification performance under single-frame input and multi-frame input conditions, and the processing speed is compared under the same hardware environment.

[0135] Evaluation indicators: Mean intersection over union (mIoU), average accuracy (mAcc) and processing speed (frames / second) are used as the main performance indicators and evaluated on two datasets respectively.

[0136] Experimental content

[0137] Urban scene test (SemanticKITTI dataset)

[0138] The SemanticKITTI dataset contains rich urban environment scene information, such as roads, pedestrian areas, buildings, etc., which is used to test the classification performance of the implemented method in a structured environment. This experiment focuses on the classification effect of distinguishing free passage, low-cost passage, medium-cost passage and impassable areas.

[0139] Field scene test (RELLIS-3D dataset)

[0140] The RELLIS-3D dataset provides unstructured scenes with rich wild environment features, including complex terrains such as mountains, grasslands, and shrubs. This dataset is used to verify the adaptability and robustness of the embodiment method in sparse point cloud data and complex environments.

[0141] Real-time performance and resource consumption

[0142] In order to verify the real-time performance of the embodiment of the present invention, the processing speed and memory usage of the system are tested at different frame rates, and compared with the processing speed of BevNet under the same conditions.

[0143] Quantitative analysis

[0144] Table 1 shows the comparative experimental results of the above embodiment method and BevNet on two data sets, specifically comparing the classification accuracy (mIoU and mAcc) and processing speed (fps).

[0145]

[0146] Qualitative analysis

[0147] 1. Urban scene test results

[0148] In the SemanticKITTI dataset, the method used in the embodiment of the present invention shows better performance than the BevNet-S method in distinguishing free passage, low-cost passage, and impassable areas. In particular, in the classification task of road and pedestrian areas, the two-dimensional bird's-eye view generated by the embodiment of the present invention is as follows: Figure 7 As shown. This figure shows the comparison of the classification effects of the single-frame model proposed in this paper and the BEVNet-S method on the SemanticKITTI dataset (upper part of the figure) and the RELLIS-3D dataset (lower part of the figure). The purpose is to evaluate the passability classification performance of the single-frame model in different complex scenarios. The figure uses color coding to indicate different levels of passability: green represents easy passability, yellow represents medium passability, and red represents impassability. Through comparative analysis, the classification effect and advantages of the single-frame model of an embodiment of the present invention in a variety of urban and wild environments are revealed.

[0149] 2. Field scene test results

[0150] On the RELLIS-3D dataset, the embodiments of the present invention demonstrate their significant advantages in identifying complex terrain and obstacles (such as bushes and slopes). Compared with BevNet-R, the method of the present invention is more accurate in predicting road contours in field scenes and can effectively detect obstacles in long-distance areas. Figure 8 The classification effect comparison between the multi-frame model proposed in this paper and BEVNet-R on the SemanticKITTI dataset (upper part of the figure) and the RELLIS-3D dataset (lower part of the figure) is demonstrated, aiming to evaluate the detection and classification performance of the multi-frame model for long-distance obstacles and complex terrain under different environmental conditions. The superposition effect of point cloud data and the actual driving path of the vehicle further confirms the traffic classification accuracy and environmental adaptability of the multi-frame model in the embodiment of the present invention in a variety of scenarios.

[0151] Based on the same inventive concept, in other embodiments of the present invention, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute the above-mentioned method or to run the above-mentioned system when executing the program.

[0152] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above-mentioned computer programs, computer instructions, etc. can be partitioned and stored in one or more memories.

[0153] The processor is used to execute the computer program stored in the memory to implement the various steps in the method involved in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0154] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0155] Based on the same inventive concept, in other embodiments of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, it can be used to execute the above method, or run the above system.

[0156] Among them, computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special-purpose computer. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a user device. Of course, the processor and the storage medium can also be present in a communication device as discrete components.

[0157] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0158] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0159] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0160] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0161] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A semantic accessibility classification method based on point cloud data, characterized in that: include: Get multi-frame point cloud data; Performing rasterization processing on each frame of the point cloud data according to the xy coordinates in the two-dimensional plane to generate a columnar volume structure of each frame of the point cloud data; Extracting a plurality of column features of the columnar volume structure of each frame of the point cloud data; Correspondingly fusing the plurality of column features of the plurality of frames of the point cloud data to obtain a fused feature; The fused features are subjected to semantic accessibility classification to obtain a final accessibility map.

2. The semantic accessibility classification method based on point cloud data according to claim 1, characterized in that: The step of acquiring multi-frame point cloud data comprises: Acquire multi-frame point cloud data of the surrounding environment of a vehicle or robot through a lidar sensor; Each frame of point cloud data includes three-dimensional coordinates (x, y, z) and reflection intensity value r, denoted as Where N is the number of points in the point cloud.

3. The semantic accessibility classification method based on point cloud data according to claim 1, characterized in that: The step of performing rasterization processing on each frame of the point cloud data according to the xy coordinates in the two-dimensional plane to generate a columnar volume structure of each frame of the point cloud data includes: Dividing the point cloud data into uniform two-dimensional grids in the x and y directions; Project each point cloud in the two-dimensional grid into a column, and record the number of point clouds in the j-th column as pj; Set the number of point clouds in each column to M; Zero-fill the columns with less than M point clouds, and randomly sample the columns with more than M point clouds to obtain a fixed-size point cloud set. All the point cloud sets of fixed size are the columnar volume structure of the point cloud data of this frame.

4. The semantic accessibility classification method based on point cloud data according to claim 1, characterized in that: The step of extracting a plurality of column features of the columnar volume structure of each frame of the point cloud data comprises: Each point cloud feature vector in each column is embedded into a high-dimensional feature: f(x i y i ,z i ,r i )=ReLU(BN(W·(x i y i ,z i ,r i ) T )) Where w is the linear transformation matrix, BN is the batch normalization operation, and ReLU is the nonlinear activation function; The point cloud features within the column are aggregated through the maximum pooling operation to obtain the column features; P j =maxi=1 M f(x i ,y i ,z i ,r i ); 1 of them M represents the maximum pooling operation, and j represents the jth column.

5. The semantic accessibility classification method based on point cloud data according to claim 1, characterized in that: The step of correspondingly fusing the plurality of column features of the plurality of frames of point cloud data to obtain a plurality of fused features comprises: By transforming the matrix H k , align the column features of the k-frame point cloud data to the coordinate system of the current frame: The features of all aligned frames are weighted and fused through the attention mechanism to obtain the fused features: where f s and f c are the spatial and channel attention networks respectively, σ is the Softmax function; W s is the spatial attention weight, W c is the channel attention weight.

6. The semantic accessibility classification method based on point cloud data according to claim 1, characterized in that: The performing semantic accessibility classification on the fusion features to obtain a final accessibility map includes: The encoder is used to perform layer-by-layer convolution and downsampling to generate multi-layer feature maps, specifically: Generate a 1 / 4 resolution feature map through a 64-channel dilated convolution block; Generate a 1 / 8 resolution feature map through two 128-channel dilated convolution blocks; Through multiple 256-channel dilated convolution blocks, a 320-channel feature map with 1 / 16 resolution is generated in the final layer; The decoder is used to restore the feature resolution of the encoder by upsampling layer by layer, and generate the final feasibility map, which is specifically: Applying 1×1 convolution to the feature map with 1 / 16 resolution and the feature map with 1 / 8 resolution to obtain a feature map with 128 channels; By upsampling and splicing, the feature map with a resolution of 1 / 16 and the feature map with a resolution of 1 / 8 are merged to generate a new feature map M′1 / 8, where M′ represents the feature map newly obtained by upsampling and splicing; Apply 3×3 convolution to the feature map M′1 / 8 to generate a 64-channel feature map, then upsample and concatenate it with the 1 / 4 resolution feature map to finally generate a new feature map M′1 / 4; The M′1 / 4 is subjected to 3×3 convolution and 1×1 convolution to generate a 5-channel output as the final feasibility map.

7. The semantic accessibility classification method based on point cloud data according to claim 6, characterized in that: The dilated convolution block performs the following operations on the input feature map: Extract multi-scale features of the input feature map through group convolution and different dilation rates; Normalizing and activating the multi-scale features; For the normalized and activated multi-scale features, a reduction rate of 1 / 4 is used to enhance the feature selection capability.

8. A semantic accessibility classification system based on point cloud data, characterized in that: include: Perception module: obtain multi-frame point cloud data; Preprocessing module: rasterizing each frame of the point cloud data according to the xy coordinates in the two-dimensional plane to generate a columnar volume structure of each frame of the point cloud data; Feature extraction module: extracting a plurality of column features of the columnar volume structure of each frame of the point cloud data; Fusion module: fusing the plurality of column features of the plurality of frames of point cloud data accordingly to obtain fusion features; Classification module: performs semantic accessibility classification on the fused features to obtain the final accessibility map.

9. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it can be used to execute the method described in any one of claims 1 to 7, or to run the system described in claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it can be used to execute the method described in any one of claims 1 to 7, or to run the system described in claim 8.

Images