A semantic segmentation method, system, electronic device and medium for autonomous driving

Abstract

The present invention is applicable to the fields of deep learning and autonomous driving technology, and provides a semantic segmentation method, system, electronic device and medium for autonomous driving, the method comprising: acquiring three-dimensional point cloud data, mapping it into a two-dimensional depth map including multiple channel data, and forming a sample data set according to the multiple channel data; constructing an initial model of a semantic segmentation network including a first model and a second model, using the sample data set to train the initial model of the semantic segmentation network to obtain a target model, wherein the second model comprises an encoder and a decoder; acquiring target three-dimensional point cloud data, mapping it into a target two-dimensional depth map, and inputting the target two-dimensional depth map into the target model to obtain a target semantic segmentation result; the method is used to solve the problem of low semantic segmentation performance of three-dimensional point clouds.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning and autonomous driving technology, and in particular to a semantic segmentation method, system, electronic device and medium for autonomous driving. Background Art

[0002] With the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role. Semantic segmentation is one of the important applications in the field of artificial intelligence, and is widely used in autonomous driving, video understanding, face recognition systems, smart hardware, etc. In the field of autonomous driving, accurate, robust, reliable, and real-time perception and understanding of the traffic environment can be achieved through accurate semantic segmentation of the traffic environment.

[0003] At present, most autonomous driving systems use multiple types of sensors with complementary characteristics, such as cameras, radars, etc. Since radar can provide distance measurement and high-precision 3D geometric information, it plays a vital role in semantic scene perception, especially in the key task of 3D point cloud semantic segmentation. However, the 3D point cloud data generated by radar is usually sparse, irregular, and disordered, resulting in low semantic segmentation performance of 3D point clouds. Summary of the invention

[0004] The present invention provides a semantic segmentation method, system, electronic device and medium for autonomous driving, so as to solve the problem of low semantic segmentation performance of three-dimensional point clouds in the prior art.

[0005] The semantic segmentation method for autonomous driving provided by the present invention includes:

[0006] Acquire three-dimensional point cloud data, map it into a two-dimensional depth map including multiple channel data, and form a sample data set according to the multiple channel data;

[0007] Constructing an initial model of a semantic segmentation network including a first model and a second model, training the initial model of the semantic segmentation network using the sample data set to obtain a target model, wherein the second model includes an encoder and a decoder;

[0008] The target three-dimensional point cloud data is obtained, mapped into a target two-dimensional depth map, and the target two-dimensional depth map is input into the target model to obtain a target semantic segmentation result.

[0009] Optionally, the first model includes a feature extraction model and a feature fusion model, and the adopting of the sample data set to train the semantic segmentation network initial model to obtain the target model includes:

[0010] The sample data set is input into the feature extraction model, and the features of each channel data in the sample data set are extracted respectively to obtain the channel features of each channel, wherein the feature extraction model includes at least two first components and one second component, the first component includes a first convolution module and a residual module, and the second component includes a second convolution module and a residual module.

[0011] Optionally, the adopting the sample data set to train the initial model of the semantic segmentation network to obtain a target model further includes:

[0012] The channel features of each channel are input into the feature fusion model, and feature fusion processing is performed on them to obtain fusion features, wherein the feature fusion model is a spatial channel attention module.

[0013] Optionally, the adopting the sample data set to train the initial model of the semantic segmentation network to obtain a target model includes:

[0014] The output result of the first model is obtained, and the output result is input into the encoder for a first convolution process to obtain a first processing result, wherein the encoder includes a third convolution module and a residual module.

[0015] Optionally, the decoder includes a converter module, and the using of the sample data set to train the initial model of the semantic segmentation network to obtain the target model further includes:

[0016] Obtaining an output result of the encoder, performing a first upsampling process on the output result, and obtaining a second processing result;

[0017] After the second processing result is input into the converter module, a second convolution processing is performed on it using a non-square sliding window to obtain a target semantic segmentation result. The converter module is constructed based on a multi-head attention submodule and a multi-layer perceptron submodule.

[0018] Optionally, the encoder further includes a sub-encoder and a segmentation head network, and the using the sample data set to train the initial model of the semantic segmentation network to obtain the target model also includes:

[0019] The processing result of the second convolution processing is obtained, and after a second upsampling is performed on it, it is sequentially input into the sub-encoder and the segmentation head network to obtain a target semantic segmentation result.

[0020] Optionally, the mathematical expression of the loss function L of the target model is:

[0021] L = λ1L1+λ2L2+λ3L3;

[0022]

[0023]

[0024]

[0025]

[0026]

[0027]

[0028]

[0029] Among them, L is the loss function of the semantic segmentation network, L1 is the first loss function, λ1 is the weight of the first loss function, L2 is the second loss function, λ2 is the weight of the second loss function, L3 is the third loss function, λ3 is the weight of the third loss function, C is the total number of object categories corresponding to the three-dimensional point cloud data, c is the category label, and f t is the median frequency of all categories, f c is the frequency of category c, W c is the category weight of category c, i is the pixel label, y c is the true value, is the predicted value, ΔJ nc is the Lovász expansion of the Jacobi index, m i (c) is the probability function of the i-th pixel of category c, is the predicted probability of the i-th pixel of category c, is the true probability value of the i-th pixel of category c, y pd is the predicted boundary map, y gt is the true value of category c, P C To predict the boundary map about the true value y of category c gt The precision rate, R C To predict the boundary map about the true value y of category c gt The recall rate is θ0, θ0 is the size of the sliding window, and pool() is the maximum pooling operation on the sliding window of size θ0.

[0030] The present invention also provides a semantic segmentation system for autonomous driving, comprising:

[0031] A mapping module, used to acquire three-dimensional point cloud data, map it into a two-dimensional depth map including multiple channel data, and form a sample data set according to the multiple channel data;

[0032] A target model building module is used to build an initial model of a semantic segmentation network including a first model and a second model, and train the initial model of the semantic segmentation network using the sample data set to obtain a target model, wherein the second model includes an encoder and a decoder;

[0033] The target result acquisition module is used to acquire target three-dimensional point cloud data, map it into a target two-dimensional depth map, and input the target two-dimensional depth map into the target model to obtain the target semantic segmentation result. The mapping module, the target model establishment module and the target result acquisition module are connected.

[0034] The present invention also provides an electronic device, comprising: a processor and a memory;

[0035] The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device performs the semantic segmentation method for autonomous driving.

[0036] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the semantic segmentation method for autonomous driving as described above is implemented.

[0037] As described above, the present invention provides a semantic segmentation method, system, electronic device and medium for autonomous driving, which have the following beneficial effects: first, by mapping the acquired three-dimensional point cloud data into a two-dimensional depth map including multiple channel data, and on this basis, building a semantic segmentation network model with superior performance, then acquiring the target point cloud data, and mapping the target point cloud data into a target two-dimensional depth map, and finally inputting the target two-dimensional depth map into the trained semantic segmentation network model to obtain the target semantic segmentation result; by mapping the three-dimensional point cloud data into a two-dimensional depth map, a spherical projection strategy is adopted to convert the three-dimensional point cloud segmentation problem into a two-dimensional depth map segmentation problem, thereby achieving the purpose of fully utilizing the excellent performance of the two-dimensional convolutional neural network to reduce the computational complexity and memory requirements, and also improving the three-dimensional semantic segmentation effect; solving the problem of low semantic segmentation performance of three-dimensional point clouds in the prior art, thereby achieving accurate and efficient understanding of three-dimensional scenes, and thus providing technical support for autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 is a flow chart of a semantic segmentation method for autonomous driving in an embodiment of the present invention;

[0040] Figure 2 is a structural diagram of a feature extraction model in an embodiment of the present invention;

[0041] Figure 3 is a schematic diagram of the structure of an encoder in an embodiment of the present invention;

[0042] Figure 4 is a schematic structural diagram of a converter module in an embodiment of the present invention;

[0043] Figure 5 is a structural schematic diagram of a semantic segmentation system for autonomous driving in an embodiment of the present invention;

[0044] Figure 6 Schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0046] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0047] In order to illustrate the technical solution of the present invention, a specific embodiment is provided below for illustration.

[0048] Figure 1 It is a flowchart of a semantic segmentation method for autonomous driving provided in one embodiment of the present invention.

[0049] like Figure 1 As shown, the semantic segmentation method for autonomous driving includes steps S110-S130:

[0050] S110, acquiring three-dimensional point cloud data, mapping it into a two-dimensional depth map including a plurality of channel data, and forming a sample data set according to the plurality of channel data;

[0051] S120, constructing an initial model of a semantic segmentation network including a first model and a second model, using the sample data set to train the initial model of the semantic segmentation network to obtain a target model, wherein the second model includes an encoder and a decoder;

[0052] S130, obtaining target three-dimensional point cloud data, mapping it into a target two-dimensional depth map, and inputting the target two-dimensional depth map into the target model to obtain a target semantic segmentation result.

[0053] In step S110 of this embodiment, the three-dimensional point cloud data may be the three-dimensional point cloud data scanned by the radar in the autonomous driving task. By mapping the three-dimensional point cloud data into a two-dimensional depth map, the three-dimensional point cloud segmentation problem is converted into a two-dimensional depth map segmentation problem using a spherical projection strategy, thereby making full use of the excellent performance of the two-dimensional convolutional neural network to reduce the computational complexity and memory requirements, thereby achieving accurate and efficient understanding of the three-dimensional scene and providing technical support for autonomous driving. The mathematical expression for mapping the three-dimensional point cloud data into a two-dimensional depth map is:

[0054]

[0055]

[0056]

[0057] Among them, the coordinates of the point cloud are (x, y, z), the coordinates of the two-dimensional depth map are (u, v), W is the height of the two-dimensional depth map, H is the width of the two-dimensional depth map, d is the depth of each point, and f up Indicates the upper limit of viewing angle, f down Indicates the lower limit of viewing angle;

[0058] The two-dimensional depth map includes multiple channel data, and the mathematical expression of the channel data m is:

[0059] m=(H,W,n);

[0060] Among them, n is one of x, y, z, d and r, and r is the reflection intensity information; in addition, n can also be one of xyz, d and r, and xyz is the coordinate of a point cloud data.

[0061] In step S120 of the present embodiment, the first model includes a feature extraction model and a feature fusion model, wherein the feature extraction model includes at least two first components and one second component, the first component includes a first convolution module and a residual module, the second component includes a second convolution module and a residual module, and the feature fusion model is a spatial channel attention module. The specific implementation method of using the sample data set to train the initial model of the semantic segmentation network and obtain the target model includes: inputting the sample data set into the feature extraction model, extracting the features of each channel data in the sample data set respectively, and obtaining the channel features of each channel; inputting the channel features of each channel into the feature fusion model, performing feature fusion processing on them, and obtaining fusion features.

[0062] In one embodiment, different channel data contained in the projected two-dimensional depth image, different channel data correspond to different modal information, the values ​​of these modal information follow different distributions, and they have different contributions to feature learning. Therefore, by extracting channel features separately and then fusing channel features, the feature space corresponding to each independent channel can be learned, so that the semantic segmentation network model established on this basis can understand the three-dimensional scene more accurately and efficiently. In the feature extraction model, the input end of the second component is connected to a first component, the output end of the second component is connected to another first component, and the output end of the feature extraction model is connected to the input end of the feature fusion model. The first convolution module can be connected in parallel with a plurality of convolution layers of different convolution kernels, and then connected in series with a 1*1 convolution layer. In the first component, the first convolution module is connected in parallel with the residual module. In the second component, the second convolution module is a hole convolution, and the second convolution module and the residual module are connected in parallel. Among them, the residual module is a structural feature with 1*1 convolution. The fusion model is a spatial channel attention module, which obtains channel features through independent channel data and then fuses the channel features to achieve the purpose of outputting contextual features from local to global with spatial fine-grained information.

[0063] In a specific embodiment, the structural diagram of the feature extraction model can refer to Figure 2 The feature extraction model consists of a first component 1, a second component 2 and a first component 3. The first component 1 includes a first convolution module 11 and a residual module 12. The first convolution module 11 includes a convolution layer 111 with a convolution kernel of 3*3, a convolution layer 112 with a convolution kernel of 5*5, a convolution layer 113 with a convolution kernel of 7*7 and a convolution layer 114 with a convolution kernel of 1*1; the second component 2 includes a second convolution module 21 and a residual module 22. The second convolution module 21 includes A convolution layer 211 with a convolution kernel of 3*3 and a dilated convolution 212 with a convolution kernel of 3*3, wherein the dilated convolution has a dilated ratio of 2; the first component 3 includes a first convolution module 31 and a residual module 32, the first convolution module 31 includes a convolution layer 311 with a convolution kernel of 3*3, a convolution layer 312 with a convolution kernel of 5*5, a convolution layer 313 with a convolution kernel of 7*7 and a convolution layer 314 with a convolution kernel of 1*1; wherein C represents cascade and + represents residual addition.

[0064] In step S130 of this embodiment, the specific implementation method of using the sample data set to train the semantic segmentation network initial model and obtain the target model also includes: obtaining the output result of the first model, inputting it into the encoder for first convolution processing, and obtaining the first processing result, and the encoder includes a third convolution module and a residual module. Specifically, please refer to the schematic diagram of the encoder structure Figure 3, the encoder 4 includes a third convolution module 41, a residual module 42 and a residual module 43, wherein the residual module has a 1*1 convolution structure. The third convolution module includes a convolution layer 411 with a convolution kernel of 3*3, a convolution layer 412 with a convolution kernel of 3*3 and a dilated convolution 413 with a convolution kernel of 3*3, wherein the dilated convolution has a dilated ratio of 2. Through this design, the encoder can make full use of multi-level, multi-scale and multi-channel information.

[0065] In one implementation, there is a problem of missing points in the two-dimensional depth map, which is defined as missing noise in this embodiment. The noise mainly comes from sensor limitations, mirror reflections, object self-occlusion, and spherical projection. These noises have a negative impact on the performance of the convolutional neural network model. In order to solve this problem, this embodiment chooses to use the context aggregation module in the SqueezeSegV2 method, and places it after the basic encoder module to construct a residual context aggregation module to enhance the robustness of the context feature to this noise.

[0066] In one embodiment, the decoder includes a converter module, and the converter module is constructed based on a multi-head attention submodule and a multi-layer perceptron submodule. The sample data set is used to train the initial model of the semantic segmentation network, and the implementation method of obtaining the target model includes: obtaining the output result of the encoder, performing a first upsampling process on it, and obtaining a second processing result; after inputting the second processing result into the converter module, a non-square sliding window is used to perform a second convolution process on it, and a target semantic segmentation result is obtained.

[0067] In a specific embodiment, the structural diagram of the converter module can be referred to as Figure 4 , where LN stands for LayerNormalization, MSA stands for Multi-head Self Attention, SMSA stands for Shifted window Multi-head Self Attention, and MLP stands for Multilayer Perceptron. l is the output of the multi-head self-attention module of the lth module, z l is the output of the multilayer perceptron module of the lth module; y l-1 is the output of the multi-head self-attention module of the l-1th module, z l-1 is the output of the multilayer perceptron module of the l-1th module; y l+1 is the output of the multi-head self-attention module of the l+1th module, z l+1 is the output of the multilayer perceptron module of the l+1th module; the formal expression of the converter module is:

[0068] z l+1 =MLP(LN(y l+1 ))+y l+1 .

[0069] In one embodiment, compared with ordinary two-dimensional color images, the width of the depth image is much larger than its height. This embodiment defines it as a size imbalance problem. In order to solve this problem, this embodiment uses a non-square sliding window instead of a square sliding window to adapt to the problem of uneven size of the depth image. In order to achieve better feature learning, each decoder adds four stacked converter modules after the upsampling layer. Converter modules with different resolutions are used to construct a hierarchical decoder structure for learning multi-scale information. Specifically, the implementation method of the balanced non-square conversion module includes: performing a 2-fold upsampling process on the first processing result (feature map) of the encoder input to the encoder, and then connecting it with the corresponding feature map output by the encoder to complete the fusion of multi-scale information. Finally, a linear layer is used to adjust the number of channels. The entire encoder stage consists of three balanced non-square converter modules of different scales.

[0070] In one embodiment, although the balanced non-square converter module has the ability to obtain superior performance, it is difficult to decode feature maps with greater resolution. Therefore, a convolution-based decoder module is added at the end of the decoder process to solve this problem. Specifically, the encoder also includes a sub-encoder and a segmentation head network, and the sample data set is used to train the initial model of the semantic segmentation network. The specific implementation method of obtaining the target model also includes: obtaining the processing result of the second convolution processing, performing a second upsampling on it, and then inputting it into the sub-encoder and the segmentation head network in sequence to obtain the target semantic segmentation result.

[0071] In step S130 of this embodiment, the target three-dimensional point cloud data can be the three-dimensional point cloud data scanned by the radar under the autonomous driving task. The target three-dimensional point cloud data is mapped to the target two-dimensional depth map in the same way as in step S110, and then the target two-dimensional depth map is input into the target model to obtain the target semantic segmentation result. By mapping the target three-dimensional point cloud data to the target two-dimensional depth map, the spherical projection strategy is used to convert the three-dimensional point cloud segmentation problem into a two-dimensional depth map segmentation problem, thereby achieving the purpose of fully utilizing the excellent performance of the two-dimensional convolutional neural network to reduce the computational complexity and memory requirements, and also improving the three-dimensional semantic segmentation effect; then the target two-dimensional depth map is input into the semantic segmentation network model to obtain the target semantic segmentation result, which solves the problem of low semantic segmentation performance of the three-dimensional point cloud in the prior art, thereby achieving accurate and efficient understanding of the three-dimensional scene, and thus providing technical support for autonomous driving.

[0072] In one embodiment, the loss function L of the semantic segmentation network model is mathematically expressed as:

[0073] L = λ1L1+λ2L2+λ3L3;

[0074]

[0075]

[0076]

[0077]

[0078]

[0079]

[0080]

[0081] Among them, L is the loss function of the semantic segmentation network, L1 is the first loss function, λ1 is the weight of the first loss function, L2 is the second loss function, λ2 is the weight of the second loss function, L3 is the third loss function, λ3 is the weight of the third loss function, C is the total number of object categories corresponding to the three-dimensional point cloud data, c is the category label, and f t is the median frequency of all categories, f c is the frequency of category c, W c is the category weight of category c, i is the pixel label, y c is the true value, is the predicted value, ΔJ nc is the Lovász expansion of the Jacobi index, m i (c) is the probability function of the i-th pixel of category c, is the predicted probability of the i-th pixel of category c, is the true probability value of the i-th pixel of category c, y pd is the predicted boundary map, y gt is the true value of category c, P C To predict the boundary map about the true value y of category c gt The precision rate, R C To predict the boundary map about the true value y of category c gt The recall rate is θ0, θ0 is the size of the sliding window, and pool() is the maximum pooling operation on the sliding window of size θ0.

[0082] The following is a specific example for explanation:

[0083] Example 1

[0084] (1) Dataset selection

[0085] In order to verify the performance of the semantic segmentation network model of this embodiment, the three-dimensional point cloud data of this embodiment comes from a large-scale SemanticKITTI dataset and a small-scale SemanticPOSS dataset. The SemanticKITTI dataset is constructed by providing dense point semantic annotation for the full 360-degree scan of the KITTI Odometry Benchmark. This dataset contains 21 sequences with a total of 43,000 scan data. Among them, the 21,000 scan data from sequences 00 to 10 are used for training, the 08th sequence is used for verification, and the 11th to 21st sequences are used for testing. The SemanticPOSS data is a small-scale dataset created by Peking University, which contains 2,988 complex scenes with high-quality dynamic targets. It follows the same data format specification as SemanticKITTI. The dataset has a total of 6 parts, of which the 2nd and 3rd parts are used for testing, and the rest are used for training.

[0086] (2) Performance Evaluation Metrics

[0087] In order to evaluate the performance of the semantic segmentation network and make a fair comparison with the mainstream methods, this embodiment uses the standard mean IOU as the performance metric, which is defined as follows:

[0088]

[0089] Among them, mIOU is the performance measurement indicator, N is the total number of object categories corresponding to the 3D point cloud data, c is the category label, TPc, FPc, FNc represent the number of true positive examples, false positive examples and true negative examples of category c.

[0090] (3) Implementation details

[0091] This example uses PyTorch to implement the TransRVNet framework. All models will be optimized using the stochastic gradient descent algorithm, with the momentum set to 9 and the weight decay rate set to 0.0001. During the training process, random rotation, random point dropout, and random change of X and Y values ​​will be used to enhance the data to avoid overfitting. For the above two specific data sets, the corresponding settings are as follows:

[0092] For the SemanticKITTI dataset, the model of this embodiment uses a training block of size 4 to train the model for 100 rounds, the initial learning rate is set to 0.005, and the learning rate decays by 0.01 after each round of training. The height of the projected depth map is set to 64 and the width is set to 1024. The sliding window size of the converter module in the model is set to [4,64]. For the K nearest neighbor post-processing stage, a window of size 7 is used to find the neighborhood, and the truncation is 2 meters.

[0093] For the SemanticPOSS dataset, this embodiment sets the initial learning rate to 0.0025, and uses the cosine simulated annealing strategy to dynamically adjust the learning rate. The training block size is 2, and a total of 50 rounds are trained. The height of the depth map is 64 and the width is 1600. The sliding window size of the converter is set to [4,100]. Since the SemanticPOSS database is relatively sparse, in the post-processing stage, the window size is set to 11 and the truncation is 5 meters.

[0094] (4) Results

[0095] Quantization results: Table 1 shows the quantization comparison results of TransRVNet and mainstream methods in the SemanticKITTI dataset. Among them, the mainstream methods are point-based methods and image-based methods. From the experimental results, the following conclusions can be drawn:

[0096] The method using a 64×2048 depth map as input obtains a better mIoU value for segmentation than using 64×1024. This is mainly because a larger depth map means that the network's learning ability to segment small-scale and fine-grained objects can be improved. In addition, large-scale input also means that more points are back-projected into the three-dimensional point cloud. The TransRVNet of this embodiment achieves the best performance, not only outperforming point-based methods, but also outperforming image-based methods. More notably, the performance of the TransRVNet method of this embodiment is far better than the image-based baseline method RangeNet++, with an mIoU score improvement of 12.7%, and even exceeds the performance of RangeNet++ when using a 64×2048 depth map as input, an improvement of 8.5%.

[0097] Table 2 shows the comparison results of the 02nd sequence of TransRVNet in this embodiment and the mainstream method on SemanticPOSS, and Table 3 shows the comparison results of the 03rd sequence of TransRVNet in this embodiment and the mainstream method on SemanticPOSS. From the table, it can be clearly seen that TransRVNet achieved the best performance in both test sequences. On sequence 02, it surpassed the previous mainstream methods RangeNet++ and MINet by 16.6% and 4.3% respectively. On sequence 03, it was improved by 11.6% and 6.9% over RandLA-Net and MinkNet respectively. The mIoU scores of all methods are relatively low because compared

[0098]

[0099] Compared with the SemanticKITTI dataset, SemanticPOSS is smaller in scale and has more sparse data.

[0100] Table 1 Quantization comparison results of TransRVNet in this embodiment and mainstream methods on the SemanticKITTI dataset

[0101]

[0102] Table 2 Comparison results of the 02nd sequence of TransRVNet in this embodiment and mainstream methods on SemanticPOSS

[0103]

[0104] Table 3 Comparison results of the 03rd sequence of TransRVNet in this embodiment and mainstream methods on SemanticPOSS

[0105] (5) Ablation experiment

[0106] In order to further understand the TransRVNet of this embodiment, this embodiment conducts an ablation experiment to discuss the various technologies of the network model, including the network structure design, the sliding window size, and the role of the first model and the second model, and their impact on the segmentation performance. All experiments are conducted on SemanticKITTI, where sequence 08 is selected as the test set and other sequences are used as training sets.

[0107] Network structure: In order to explore the performance of the convolutional neural network-converter structure used in this embodiment on the semantic segmentation task, this embodiment conducts the following comparative experiments. Based on the TransRVNet structure, two converter-convolutional neural network models are designed, in which all converter decoding modules are replaced with the convolution decoder modules used in this embodiment, and then high-performance sliding window converter modules and visual converter modules are used as encoders respectively. The specific parameter settings are as follows:

[0108] 1) The query dimension of each head in the multi-head self-attention mechanism is set to 32, and 4 expansion layers are integrated in each multi-layer perceptron. 2) For the sliding window converter module, the number of channels in the first stage is set to 96, and the number of converter modules in each encoder module is set to {2,2,6,2}. 3) For the visual converter, the number of channels in the first stage is set to 64, and the number of converter modules in each encoder module is set to {4,4,4,4}

[0109] Table 4 shows the comparison between the three converter-convolutional neural network structures and the convolutional neural network-converter structure of the present invention. From the perspective of accuracy, the convolutional neural network-balanced non-square converter structure of this embodiment achieves the best performance and is more suitable for radar semantic segmentation tasks.

[0110] Table 4. Impact of network structure on semantic segmentation task performance

[0111]

[0112] Influence of window size: As mentioned in the previous discussion, the non-square sliding window strategy designed in this embodiment is more suitable for processing depth images. As a crucial design of TransRVNet, it not only affects the segmentation performance, but also affects the parameters and computational complexity. Table 5 shows the performance comparison of using different window sizes. Obviously, 1) the semantic segmentation performance continues to improve as the sliding window increases, mainly because a large window means a larger receptive field, which can aggregate a wider range of contextual information. However, a large window brings high computational complexity and memory consumption. 2) Compared with square windows, non-square windows achieve more ideal performance. For example, the sizes of [4,8], [4,16], and [4,32] increase the mIoU scores of the corresponding square window [4,4] by 0.2%, 0.9%, and 1.2%, respectively. This further shows that non-square sliding windows are more suitable for processing depth images of uneven size, and are a more ideal selection strategy.

[0113] Table 5 Effect of sliding window size

[0114]

[0115] The influence of feature extraction model (multiple residual context learning module) and feature fusion model (residual context aggregation module). Table 6 gives the ablation experiment on the feature extraction model (multiple residual context learning module) and feature fusion model (residual context aggregation module). In this experiment, the two modules are added to the backbone network one by one to verify the effectiveness of each module. From the laboratory results, it can be seen that only adding the multi-residual context learning module and only adding the residual context aggregation module can improve the performance of the TransRVNet network to 57.2% and 56.6%, respectively, which are 2.2% and 1.6% higher than the backbone network, respectively. Adding both to the backbone network improves the overall performance by 3.1%. On the other hand, in order to further verify the superiority of the multi-residual context learning module, Table 7 gives the mIoU performance of several categories (such as pedestrians, tree trunks, trucks, buildings, fences, parks, etc.) after adding the module. Compared with the backbone network, the performance of the network model with the multi-residual context learning module has been significantly improved in all categories. This also further illustrates that the module's ability to capture multi-scale contextual information at the channel level and model inter-channel and intra-channel feature dependencies plays a crucial role in improving radar segmentation performance.

[0116] Table 6 Influence of feature extraction model and feature fusion model

[0117]

[0118] Table 7 Impact analysis of multiple residual context learning modules

[0119]

[0120] Based on the same inventive concept as the semantic segmentation method for autonomous driving, this embodiment also provides a semantic segmentation system. In this embodiment, the semantic segmentation system executes the semantic segmentation method described in any of the above embodiments, and the specific functions and technical effects can be referred to the above embodiments, which will not be repeated here.

[0121] Figure 5 A schematic diagram of the structure of a semantic segmentation system for autonomous driving provided by the present invention.

[0122] like Figure 5 As shown, the semantic segmentation system for autonomous driving includes: 51 a mapping module, 52 a target model building module, and 53 a target result acquisition module.

[0123] Wherein, the mapping module is used to obtain three-dimensional point cloud data, map it into a two-dimensional depth map including multiple channel data, and form a sample data set according to the multiple channel data;

[0124] A target model building module is used to build an initial model of a semantic segmentation network including a first model and a second model, and train the initial model of the semantic segmentation network using the sample data set to obtain a target model, wherein the second model includes an encoder and a decoder;

[0125] The target result acquisition module is used to acquire target three-dimensional point cloud data, map it into a target two-dimensional depth map, and input the target two-dimensional depth map into the target model to obtain the target semantic segmentation result.

[0126] This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, any one of the methods in this embodiment is implemented.

[0127] In one embodiment, see Figure 6 This embodiment also provides an electronic device 600, including a memory 601, a processor 602, and a computer program stored in the memory and executable on the processor, wherein the processor 602 implements the steps of the method described in any of the above embodiments when executing the computer program.

[0128] The computer-readable storage medium in this embodiment can be understood by ordinary technicians in this field: all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0129] The electronic device provided in this embodiment includes a processor, a memory, a transceiver and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication with each other. The memory is used to store computer programs, the communication interface is used to communicate, and the processor and the transceiver are used to run the computer program so that the electronic device executes each step of the above method.

[0130] In this embodiment, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0131] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0132] In the above embodiments, the description refers to "this embodiment", "an embodiment", "another embodiment", "in some exemplary embodiments" or "other embodiments" to indicate that the specific features, structures or characteristics described in conjunction with the embodiments are included in at least some embodiments, but not necessarily all embodiments. Multiple occurrences of "this embodiment", "an embodiment", "another embodiment" do not necessarily all refer to the same embodiment.

[0133] In the above-described embodiments, although the invention has been described in conjunction with specific embodiments of the invention, many substitutions, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art based on the foregoing description. For example, other storage structures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed. Embodiments of the invention are intended to encompass all such substitutions, modifications, and variations that fall within the broad scope of the appended claims.

[0134] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0135] The present invention can be used in many general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like.

[0136] The present invention may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0137] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A semantic segmentation method for autonomous driving, characterized in that: The semantic segmentation method for autonomous driving includes: Acquire three-dimensional point cloud data, map it into a two-dimensional depth map including multiple channel data, and form a sample data set according to the multiple channel data; Constructing an initial model of a semantic segmentation network including a first model and a second model, training the initial model of the semantic segmentation network using the sample data set to obtain a target model, wherein the second model includes an encoder and a decoder; Acquire target three-dimensional point cloud data, map it into a target two-dimensional depth map, and input the target two-dimensional depth map into the target model to obtain a target semantic segmentation result; The first model includes a feature extraction model and a feature fusion model. The method of using the sample data set to train the semantic segmentation network initial model to obtain a target model includes: inputting the sample data set into the feature extraction model, extracting the features of each channel data in the sample data set respectively, and obtaining the channel features of each channel, wherein the feature extraction model includes at least two first components and one second component, the first component includes a first convolution module and a residual module, and the second component includes a second convolution module and a residual module; inputting the channel features of each channel into the feature fusion model, performing feature fusion processing on the features, and obtaining fusion features, wherein the feature fusion model is a spatial channel attention module; obtaining the output result of the first model, inputting the output result into the encoder for first convolution processing, and obtaining a first processing result, wherein the encoder includes a third convolution module and a residual module; The decoder includes a converter module, the sample data set is used to train the initial model of the semantic segmentation network to obtain the target model, and further includes: obtaining the output result of the encoder, performing a first upsampling process on it, and obtaining a second processing result; after inputting the second processing result into the converter module, performing a second convolution process on it using a non-square sliding window to obtain a target semantic segmentation result, and the converter module is constructed based on a multi-head attention submodule and a multi-layer perceptron submodule; The encoder also includes a sub-encoder and a segmentation head network. The sample data set is used to train the initial model of the semantic segmentation network to obtain a target model. The method also includes: obtaining the processing result of the second convolution processing, performing a second upsampling on the result, and sequentially inputting the result into the sub-encoder and the segmentation head network to obtain a target semantic segmentation result.

2. The semantic segmentation method for autonomous driving according to claim 1, characterized in that: The mathematical expression of the loss function L of the target model is: L = λ1L1+λ2L2+λ3L3; Among them, L is the loss function of the semantic segmentation network, L1 is the first loss function, λ1 is the weight of the first loss function, L2 is the second loss function, λ2 is the weight of the second loss function, L3 is the third loss function, λ3 is the weight of the third loss function, C is the total number of object categories corresponding to the three-dimensional point cloud data, c is the category label, and f t is the median frequency of all categories, f c is the frequency of category c, W c is the category weight of category c, i is the pixel label, y c is the true value, is the predicted value, ΔJ nc is the Lovász expansion of the Jacobi index, m i (c) is the probability function of the i-th pixel of category c, is the predicted probability of the i-th pixel of category c, is the true probability value of the i-th pixel of category c, y pd is the predicted boundary map, y gt is the true value of category c, P C To predict the boundary map about the true value y of category c gt The precision rate, R C To predict the boundary map about the true value y of category c gt The recall rate is θ0, θ0 is the size of the sliding window, and pool() is the maximum pooling operation on the sliding window of size θ0.

3. A semantic segmentation system for autonomous driving, characterized in that: The semantic segmentation system for autonomous driving includes: A mapping module, used to acquire three-dimensional point cloud data, map it into a two-dimensional depth map including multiple channel data, and form a sample data set according to the multiple channel data; A target model establishment module is used to construct an initial model of a semantic segmentation network including a first model and a second model, and the sample data set is used to train the initial model of the semantic segmentation network to obtain a target model, wherein the second model includes an encoder and a decoder; the first model includes a feature extraction model and a feature fusion model, and the target model establishment module is specifically used to input the sample data set into the feature extraction model, respectively extract the features of each channel data in the sample data set, and obtain the channel features of each channel, wherein the feature extraction model includes at least two first components and one second component, the first component includes a first convolution module and a residual module, and the second component includes a second convolution module and a residual module; the target model establishment module is specifically used to input the channel features of each channel into the feature fusion model, perform feature fusion processing on it, and obtain fusion features, wherein the feature fusion model is a spatial channel Attention module; the target model building module is specifically used to obtain the output result of the first model, input it into the encoder for the first convolution processing, and obtain the first processing result. The encoder includes a third convolution module and a residual module; the decoder includes a converter module, and the target model building module is specifically used to obtain the output result of the encoder, perform a first upsampling processing on it, and obtain a second processing result; after the second processing result is input into the converter module, a non-square sliding window is used to perform a second convolution processing on it to obtain the target semantic segmentation result. The converter module is constructed based on a multi-head attention submodule and a multi-layer perceptron submodule; the encoder also includes a sub-encoder and a segmentation head network. The target model building module is specifically used to obtain the processing result of the second convolution processing, perform a second upsampling on it, and then sequentially input the sub-encoder and the segmentation head network to obtain the target semantic segmentation result; The target result acquisition module is used to acquire target three-dimensional point cloud data, map it into a target two-dimensional depth map, and input the target two-dimensional depth map into the target model to obtain the target semantic segmentation result. The mapping module, the target model establishment module and the target result acquisition module are connected.

4. An electronic device, characterized in that: including a processor, a memory and a communication bus; The communication bus is used to connect the processor and the memory; The processor is configured to execute the computer program stored in the memory to implement the method according to any one of claims 1 to 2.

5. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is used to make the computer execute the method according to any one of claims 1 to 2.

Images