Point cloud semantic segmentation method based on cross-modal Transformer

Abstract

A point cloud semantic segmentation method based on a cross-modal Transformer belongs to the field of semantic segmentation technology. The method focuses on guiding the dense visual information of the camera image into the point cloud semantic segmentation task to complete the point cloud semantic segmentation task. First, the three-dimensional point cloud is unified to the camera image coordinate system according to the coordinate system transformation relationship, and then the perspective projection is used to obtain the two-dimensional representation of the three-dimensional point cloud. Then, the multi-scale feature map is calculated and extracted, and then cross-modal attention fusion is performed. The camera image and the projected point cloud image are fused at the feature level. Finally, the image is upsampled and classified. The classification result is projected onto the three-dimensional point cloud according to the inverse projection transformation relationship to complete the point cloud semantic segmentation task. The Transformer self-attention mechanism is used to establish cross-modal feature dependencies. The feature information of the two modalities of image and point cloud is combined to enhance the feature expression ability of the model. A boundary loss function is designed to emphasize the boundaries of semantic objects, thereby improving the boundary accuracy of object segmentation.

Description

Technical Field

[0001] The present invention relates to the field of semantic segmentation technology, and in particular to a point cloud semantic segmentation method based on a cross-modal Transformer, which is suitable for application scenarios requiring the acquisition and perception of environmental information, such as autonomous driving or road monitoring. Background Art

[0002] With the rapid development of deep learning technology, the potential of computers is being tapped in an increasing number of fields. Scene understanding is a fundamental task for many applications, such as autonomous driving, autonomous robotics, and augmented reality. Scene understanding provides fine-grained environmental information for advanced applications such as self-driving cars. One of its key tasks is semantic segmentation, which assigns a category label to each data point in the input data. In recent years, with the upgrading of 3D acquisition equipment and the reduction in cost, more and more researchers have devoted their attention to the study of 3D scene semantic segmentation.

[0003] Point cloud data is a type of three-dimensional spatial data information about the external environment that is actively acquired by a LiDAR scanner by emitting laser light and receiving reflected laser light. It is characterized by sparsity, discreteness, and unstructured nature. In the field of point clouds, similar to developments in the field of images, traditional point cloud semantic extraction methods are designed using hand-crafted features for point cloud descriptors. These methods lack learnability, are poorly adaptable to specific tasks, and fail to meet practical needs. With the development of deep learning technology, many representative works have emerged in recent years. The point cloud classification network PointNet is a groundbreaking research achievement, and PointNet++ has become a model for subsequent work.

[0004] Cross-modal fusion is the process of integrating multiple types of data to enhance feature expression and improve model performance, especially to overcome the inherent defects caused by single-modal data. Specifically, point cloud data and camera image data have three-dimensional information such as spatial scale and depth, but lack object texture and color information; camera images contain rich texture and color information, but the images are easily affected by environmental factors and the image quality is easily damaged. Therefore, point cloud data and image data complement each other. Cross-modal information interaction and fusion can further improve the research level of three-dimensional scene understanding. However, due to the inherent data format differences and field gaps between point cloud and image data, how to effectively fuse these two types of data for three-dimensional scene semantic extraction tasks remains a challenging problem. Summary of the Invention

[0005] Technical problem: The purpose of the present invention is to overcome the problem of low degree of cross-modal fusion, effectively improve the cross-modal fusion capability, and provide a point cloud semantic segmentation method based on a cross-modal Transformer, which provides complementary information for point clouds, enriches the spatial distribution of implicit features, enhances the feature expression capability of the model, and improves the boundary accuracy of object segmentation with the help of the Transformer global dependency modeling mechanism.

[0006] Technical solution: To achieve the purpose of the present invention, the present invention provides a point cloud semantic segmentation method based on a cross-modal Transformer, comprising the following steps:

[0007] S1. 3D point cloud for a given scene and the camera color image of the corresponding field of view According to the rotation matrix R and translation matrix T, the three-dimensional point cloud The perspective projection transformation is converted to a two-dimensional plane, and the three-dimensional point cloud is transformed into a two-dimensional plane. The 4-dimensional data of each of the N points in the equation, plus the calculated distance A total of 5 dimensions of data, combined into a two-dimensional image N is the number of point clouds, H is the image height, and W is the image width;

[0008] S2, the projected two-dimensional image and camera color images After being processed based on the ResNet network, four scale feature maps are obtained respectively. and l is the scale index, I represents the two-dimensional image The feature map obtained by ResNet, S represents the camera color image Feature map obtained by ResNet;

[0009] S3. Get multi-scale feature map and After that, take the feature map of scale l=1 and Input into Transformer for feature cross-modal self-attention to obtain cross-modal image feature F1 C ;

[0010] S4: Cross-modal image feature F1 obtained in step S3 C , perform bilinear interpolation upsampling steps and CNN convolutional network processing to obtain the final feature map of scale l = 1 Where H is the image height and W is the image width;

[0011] S5. Similarly, the feature maps of scales l=2, 3, and 4 in step S2 are gradually processed through steps S3 and S4 to obtain the final feature map. Feature map group

[0012] S6. Obtain feature map group Finally, all feature maps are connected in dimension, and the probability body T is obtained through the convolution layer. The softmax operation is performed on the probability body along the dimension direction to normalize the probability to obtain the segmentation result map P0;

[0013] S7. Calculate the loss between the segmentation result map and the true value, and train the entire network according to the obtained loss value until the limited number of times is reached. The segmentation result map obtained at this time is the final segmentation map. The dimension value at the maximum probability is selected as the predicted category value, and the segmentation result of the three-dimensional point cloud is obtained according to the inverse perspective projection process.

[0014] In step S1, the three-dimensional point cloud The perspective projection transformation method into a two-dimensional image is as follows:

[0015] S11. Take out the given 3D point cloud The first three dimensions of the coordinate values ​​are x, y, and z. Taking one point p as an example, add the fourth dimension and perform homogeneous coordinate conversion to obtain the coordinates p = (x, y, z, 1) T ;

[0016] S12. Get the homogeneous coordinate p = (x, y, z, 1) T , according to the perspective projection transformation, the point cloud coordinate system coordinates are converted into the camera coordinate system coordinates. The perspective projection transformation formula is:

[0017]

[0018] in, is the camera intrinsic parameter matrix, Coordinate system transformation matrix, is the rotation correction matrix, and the camera coordinate system coordinates of point p are written as

[0019] S13, normalize the obtained camera coordinate system coordinates to obtain As the two-dimensional plane coordinates corresponding to point p in the three-dimensional point cloud; All points are processed in steps S11, S12, and S13;

[0020] S14, the original three-dimensional point cloud 4 channel information and distance information A total of 5 dimensional information is used as the dimensional information of the two-dimensional image obtained by perspective projection transformation, and finally a two-dimensional image is obtained. H is the image height and W is the image width.

[0021] In step S2, the four scale feature maps and The extraction method is as follows:

[0022] S21. Project the given point cloud into a two-dimensional image and camera color images They are input into the ResNet network for preprocessing, and CNN convolution layer is used to extract 4 scale feature maps based on the ResNet network. and Specifically: the image obtained by processing in stage S1 and The 4 scale feature maps obtained by the ResNet network are 8 feature maps in total. The final feature map is and From largest to smallest: and

[0023] In step S3, the feature cross-modal self-attention is calculated as follows:

[0024] S31, the feature map obtained in step S2 and Fusion scale l = 1 feature map and For example, input into Transformer for feature cross-modal self-attention calculation, the calculation steps are as follows:

[0025] (1) For the input image features and The number of channels is changed from 256 to C through linear embedding, and the initial input features are obtained, which are denoted as g I and g S , g I Indicated by The initial input features obtained, g S Indicated by The initial input features obtained;

[0026] (2) The obtained g I and g S Input to Transformer for processing: g I and g S First, after the normalization layer and self-attention calculation, it is combined with g I Add together to get Then get After being processed by the normalization layer and the multi-layer perceptron and added to itself, the cross-modal feature output F is finally obtained. l C , the formula is as follows:

[0027]

[0028]

[0029]

[0030] Among them, LN represents the normalization layer, linear represents the linear mapping layer, MLP represents the multi-layer perceptron, a, b, and c refer to the three input parameters, respectively, LN(g S )、LN(g I ) and LN(g I ); F1 C is the cross-modal image feature with scale l = 1; SA represents the self-attention calculation, and the calculation process of formula (4) is performed; C is the number of channels; φ represents the softmax function.

[0031] In step S4, the bilinear interpolation upsampling step and the CNN convolutional network processing method are as follows:

[0032] S41, cross-modal image feature F1 with scale l=1 C For example, for a given cross-modal image feature F1 C , after bilinear interpolation, the image feature size is increased to twice its original size, and then processed by the CNN convolutional network. The specific results of the network are as follows:

[0033] In the first convolutional layer, a convolution kernel of size 3×3 is defined, the stride is defined as 1, the padding value is 1, and a ReLU activation function and a batch normalization layer are added after the layer1 layer for normalization.

[0034] In the second convolutional layer, a convolution kernel of size 3×3 is defined, with a stride of 1, a padding value of 2, and a dilation rate of 2. A ReLU activation function and a batch normalization layer are added after the layer2 layer for normalization.

[0035] In the third convolutional layer, a convolution kernel of size 2×2 is defined, with a stride of 1, a padding value of 1, and a dilation rate of 2. A ReLU activation function and a batch normalization layer are added after the layer3 layer for normalization.

[0036] In the fourth convolutional layer, a convolution kernel of size 1×1 is defined, and a stride of 1 is defined. A ReLU activation function and a batch normalization layer are added after the layer4 layer for normalization.

[0037] S42: For the cross-modal image features with scale l=1, the processing of step S41 needs to be performed once to obtain the final feature map Wherein, H is the image height and W is the image width; cross-modal image features with scales l=2, l=3, and l=4 need to be processed by step S41 twice, three times, and four times respectively, in order to restore the cross-modal image features to the source image size.

[0038] In step S6, the method for generating the probability body T is as follows:

[0039] S61, obtain feature map group After that, all feature maps are connected in the dimension direction and then passed through two convolutional layers to obtain the probability volume T. The calculation formula of the probability volume T is as follows:

[0040] T=conv2(conv1(cat(F1 C' ,F2 C' ,F3 C' ,F4 C' ))) (5)

[0041] Among them, cat represents the connection operation, conv1 represents a 1×1 convolutional layer, and conv2 represents a 1×1 convolutional layer; is the final feature map with l=1, is the final feature map with l=2, is the final feature map with l=3, is the final feature map with l=4;

[0042] S62. Perform a softmax operation on the probability body T along the dimensional direction to normalize the probability and obtain the segmentation result map P0.

[0043] In step S7, the perspective projection process is used to obtain the segmentation result of the three-dimensional point cloud as follows:

[0044] S71. After obtaining the segmentation result image P0, calculate the loss between it and the true value, and train the entire network based on the obtained loss value. Iterate the network for 100 to 150 rounds. The loss function consists of two parts and is calculated as follows:

[0045] (1) Calculation using the cross entropy loss function:

[0046]

[0047] Among them, q xy and p xy Represent the true value of the pixel at the image coordinate (x, y) and the predicted probability in the segmentation result map P0, respectively. B is the batch size, H is the image height, and W is the image width.

[0048] (2) Calculation method of marginal loss function:

[0049]

[0050]

[0051] where w x,y is the boundary weight of the pixel at the image coordinate (x, y), c x,y and c x+i,j+i are the true semantic labels of the pixels at the image coordinates (x, y), It is an exclusive OR operation;

[0052] (3) Adding the above two together, the loss function is expressed as:

[0053]

[0054] Among them, L seg represents the cross entropy loss in formula (6), L w represents the boundary loss in formula (8);

[0055] S72. The segmentation result image P0 obtained after the training stops is the final segmentation image. After obtaining the segmentation image, the dimension value at the maximum probability is selected as the predicted category value. The category result corresponding to the three-dimensional point is obtained through the reverse transformation correspondence in step S12. All pixel points are reversely transformed to finally obtain the segmentation result of the source three-dimensional point cloud P0.

[0056] Beneficial effects: Due to the adoption of the above technical solution, compared with the deep segmentation network model with a single point cloud input in the prior art, the present invention introduces camera image information cross-modally, provides complementary information for the point cloud, and enriches the implicit feature space distribution; the Transformer self-attention mechanism is used to establish feature dependencies between cross-modalities, combining the feature information of the two modalities of image and point cloud to enhance the feature expression ability of the model. In addition, the present invention also designs a boundary loss function to emphasize the boundaries of semantic objects, improves the boundary accuracy of object segmentation, and solves the problems of traditional point cloud semantic extraction methods such as lack of learnability, weak adaptability to specific tasks, and difficulty in meeting actual needs. The main advantages compared with the prior art are as follows:

[0057] 1) Introducing low-cost camera image information into the point cloud semantic segmentation task alleviates the problem of single point cloud data's defects in environmental texture and color information, and expands and enriches the implicit feature space distribution.

[0058] 2) For 3D point cloud input, spherical projection or bird's-eye view projection is not used. Instead, perspective projection transformation is used to convert the 3D point cloud into a 2D image, avoiding the problems of information loss and deformation during the projection process.

[0059] 3) For the two modal input information, the Transformer self-attention mechanism is used to establish long-distance cross-modal feature dependencies on a global scale, effectively combining the feature information of the two modalities of image and point cloud, enriching the feature map information, and enhancing the feature expression ability of the model.

[0060] 4) In order to alleviate the boundary blur problem inherent in convolution operations, a boundary loss function is designed to emphasize the boundaries of semantic objects; combined with cross-entropy loss, it jointly guides the model to perform more comprehensive segmentation tasks, further improving the accuracy of semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 Schematic diagram of the network structure of the present invention. DETAILED DESCRIPTION

[0062] An embodiment of the present invention will be further described below with reference to the accompanying drawings:

[0063] like Figure 1 As shown in FIG, the point cloud semantic segmentation method based on the cross-modal Transformer of the present invention has the following specific steps:

[0064] S1. 3D point cloud for a given scene and the camera color image of the corresponding field of view According to the rotation matrix R and translation matrix T, the three-dimensional point cloud The perspective projection transformation is converted to a two-dimensional plane, and the three-dimensional point cloud is transformed into a two-dimensional plane. The 4-dimensional data of each of the N points in the equation, plus the calculated distance A total of 5 dimensions of data, combined into a two-dimensional image N is the number of point clouds, H is the image height, and W is the image width;

[0065] The three-dimensional point cloud The perspective projection transformation method into a two-dimensional image is as follows:

[0066] S11. Take out the given 3D point cloud The first three dimensions of the coordinate values ​​are x, y, and z. Taking one point p as an example, add the fourth dimension and perform homogeneous coordinate conversion to obtain the coordinates p = (x, y, z, 1) T ;

[0067] S12. Get the homogeneous coordinate p = (x, y, z, 1) T, according to the perspective projection transformation, the point cloud coordinate system coordinates are converted into the camera coordinate system coordinates. The perspective projection transformation formula is:

[0068]

[0069] in, is the camera intrinsic parameter matrix, Coordinate system transformation matrix, is the rotation correction matrix, and the camera coordinate system coordinates of point p are written as

[0070] S13, normalize the obtained camera coordinate system coordinates to obtain As the two-dimensional plane coordinates corresponding to point p in the three-dimensional point cloud; All points are processed in steps S11, S12, and S13;

[0071] S14, the original three-dimensional point cloud 4 channel information and distance information A total of 5 dimensional information is used as the dimensional information of the two-dimensional image obtained by perspective projection transformation, and finally a two-dimensional image is obtained. H is the image height and W is the image width.

[0072] S2, the projected two-dimensional image and camera color images After being processed based on the ResNet network, four scale feature maps are obtained respectively. and l is the scale index, I represents the two-dimensional image The feature map obtained by ResNet, S represents the camera color image Feature map obtained by ResNet;

[0073] The four scale feature maps and The extraction method is as follows:

[0074] S21. Project the given point cloud into a two-dimensional image and camera color images They are input into the ResNet network for preprocessing, and CNN convolution layer is used to extract 4 scale feature maps based on the ResNet network. and Specifically: the image obtained by processing in stage S1 and , 4 scale feature maps are obtained through the ResNet network, a total of 8 feature maps, and the final feature map and From largest to smallest: and

[0075] S3. Get multi-scale feature map and After that, take the feature map of scale l=1 and Input into Transformer for feature cross-modal self-attention to obtain cross-modal image feature F1 C ;

[0076] The calculation method of the feature cross-modal self-attention is as follows:

[0077] S31, the feature map obtained in step S2 and Fusion scale l = 1 feature map and For example, input into Transformer for feature cross-modal self-attention calculation, the calculation steps are as follows:

[0078] (1) For the input image features and The number of channels is changed from 256 to C through linear embedding, and the initial input features are obtained, which are denoted as g I and g S , g I Indicated by The initial input features obtained, g S Indicated by The initial input features obtained;

[0079] (2) The obtained g I and g S Input to Transformer for processing: g I and g S First, after the normalization layer and self-attention calculation, it is combined with g I Add together to get Then get After being processed by the normalization layer and the multi-layer perceptron and added to itself, the cross-modal feature output F is finally obtained. l C , the formula is as follows:

[0080]

[0081]

[0082]

[0083] Among them, LN represents the normalization layer, linear represents the linear mapping layer, MLP represents the multi-layer perceptron, a, b, and c refer to the three input parameters, respectively, LN(g S )、LN(g I ) and LN(g I ); F1 C is the cross-modal image feature with scale l = 1; SA represents the self-attention calculation, and the calculation process of formula (4) is performed; C is the number of channels; φ represents the softmax function.

[0084] S4: Cross-modal image feature F1 obtained in step S3 C , perform bilinear interpolation upsampling steps and CNN convolutional network processing to obtain the final feature map of scale l = 1 Where H is the image height and W is the image width;

[0085] The bilinear interpolation upsampling step and the processing method of the CNN convolutional network are as follows:

[0086] S41, cross-modal image feature F1 with scale l=1 C For example, for a given cross-modal image feature F1 C , after bilinear interpolation, the image feature size is increased to twice its original size, and then processed by the CNN convolutional network. The specific results of the network are as follows:

[0087] In the first convolutional layer, a convolution kernel of size 3×3 is defined, the stride is defined as 1, the padding value is 1, and a ReLU activation function and a batch normalization layer are added after the layer1 layer for normalization.

[0088] In the second convolutional layer, a convolution kernel of size 3×3 is defined, with a stride of 1, a padding value of 2, and a dilation rate of 2. A ReLU activation function and a batch normalization layer are added after the layer2 layer for normalization.

[0089] In the third convolutional layer, a convolution kernel of size 2×2 is defined, with a stride of 1, a padding value of 1, and a dilation rate of 2. A ReLU activation function and a batch normalization layer are added after the layer3 layer for normalization.

[0090] In the fourth convolutional layer, a convolution kernel of size 1×1 is defined, and a stride of 1 is defined. A ReLU activation function and a batch normalization layer are added after the layer4 layer for normalization.

[0091] S42: For the cross-modal image features with scale l=1, the processing of step S41 needs to be performed once to obtain the final feature map Wherein, H is the image height and W is the image width; cross-modal image features with scales l=2, l=3, and l=4 need to be processed by step S41 twice, three times, and four times respectively, in order to restore the cross-modal image features to the source image size.

[0092] S5. Similarly, the feature maps of scales l=2, 3, and 4 in step S2 are gradually processed through steps S3 and S4 to obtain the final feature map. Feature map group

[0093] S6. Obtain feature map group Finally, all feature maps are connected in dimension, and the probability body T is obtained through the convolution layer. The probability body is normalized by softmax operation along the dimension direction to obtain the segmentation result map P0;

[0094] The method for generating the probability body T is as follows:

[0095] S61, obtain feature map group After that, all feature maps are connected in the dimension direction and then passed through two convolutional layers to obtain the probability volume T. The calculation formula of the probability volume T is as follows:

[0096] T=conv2(conv1(cat(F1 C' ,F2 C' ,F3 C' ,F4 C' ))) (5)

[0097] Among them, cat represents the connection operation, conv1 represents a 1×1 convolutional layer, and conv2 represents a 1×1 convolutional layer; is the final feature map with l=1, is the final feature map with l=2, is the final feature map with l=3, is the final feature map with l=4;

[0098] S62. Perform a softmax operation on the probability body T along the dimensional direction to normalize the probability and obtain the segmentation result map P0.

[0099] S7. Calculate the loss between the segmentation result map and the true value, and train the entire network according to the obtained loss value until the limited number of times is reached. The segmentation result map obtained at this time is the final segmentation map. The dimension value at the maximum probability is selected as the predicted category value, and the segmentation result of the three-dimensional point cloud is obtained according to the inverse perspective projection process.

[0100] The method for obtaining the segmentation result of the three-dimensional point cloud through the perspective projection process is as follows:

[0101] S71. After obtaining the segmentation result image P0, calculate the loss between it and the true value, and train the entire network based on the obtained loss value. Iterate the network for 100 to 150 rounds. The loss function consists of two parts and is calculated as follows:

[0102] (1) Calculation using the cross entropy loss function:

[0103]

[0104] Among them, q xy and p xy Represent the true value of the pixel at the image coordinate (x, y) and the predicted probability in the segmentation result map P0, respectively. B is the batch size, H is the image height, and W is the image width.

[0105] (2) Calculation method of marginal loss function:

[0106]

[0107]

[0108] where w x,y is the boundary weight of the pixel at the image coordinate (x, y), c x,y and c x+i,j+i are the true semantic labels of the pixels at the image coordinates (x, y), It is an exclusive OR operation;

[0109] (3) Adding the above two together, the loss function is expressed as:

[0110] L=L seg +L w (9)

[0111] Among them, L seg represents the cross entropy loss in formula (6), L w represents the boundary loss in formula (8);

[0112] S72. The segmentation result image P0 obtained after the training stops is the final segmentation image. After obtaining the segmentation image, the dimension value at the maximum probability is selected as the predicted category value. The category result corresponding to the three-dimensional point is obtained through the reverse transformation correspondence in step S12. All pixel points are reversely transformed to finally obtain the segmentation result of the source three-dimensional point cloud P0.

Claims

1. A point cloud semantic segmentation method based on a cross-modal Transformer, characterized by: The method comprises the following steps: S1. 3D point cloud for a given scene and the camera color image of the corresponding field of view According to the rotation matrix R and translation matrix T, the three-dimensional point cloud The perspective projection transformation is converted to a two-dimensional plane, and the three-dimensional point cloud is transformed into a two-dimensional plane. The 4-dimensional data of each of the N points in the equation, plus the calculated distance A total of 5 dimensions of data, combined into a two-dimensional image N is the number of point clouds, x, y, z are the dimension coordinate values, H is the image height, and W is the image width; S2, the projected two-dimensional image and camera color images After being processed based on the ResNet network, four scale feature maps are obtained respectively. and l is the scale index, I represents the two-dimensional image The feature map obtained by ResNet, S represents the camera color image Feature map obtained by ResNet; S3. Get multi-scale feature map and After that, take the feature map of scale l=1 and Input into Transformer for feature cross-modal self-attention to obtain cross-modal image feature F1 C ; S4: Cross-modal image feature F1 obtained in step S3 C , perform bilinear interpolation upsampling steps and CNN convolutional network processing to obtain the final feature map of scale l = 1 Where H is the image height and W is the image width; S5. Similarly, the feature maps of scales l=2, 3, and 4 in step S2 are gradually processed through steps S3 and S4 to obtain the final feature map. Feature map group S6. Obtain feature map group Finally, all feature maps are connected in dimension, and the probability body T is obtained through the convolution layer. The softmax operation is performed on the probability body along the dimension direction to normalize the probability to obtain the segmentation result map P0; S7. Calculate the loss between the segmentation result map and the true value, and train the entire network according to the obtained loss value until the limited number of times is reached. The segmentation result map obtained at this time is the final segmentation map. The dimension value at the maximum probability is selected as the predicted category value, and the segmentation result of the three-dimensional point cloud is obtained according to the inverse perspective projection process.

2. The point cloud semantic segmentation method based on a cross-modal Transformer according to claim 1, characterized in that: In step S1, the three-dimensional point cloud The perspective projection transformation method into a two-dimensional image is as follows: S11. Take out the given 3D point cloud The first three dimensions of the coordinate values ​​are x, y, and z. Taking one point p as an example, add the fourth dimension and perform homogeneous coordinate conversion to obtain the coordinates p = (x, y, z, 1) Τ ; S12. Get the homogeneous coordinate p = (x, y, z, 1) Τ , according to the perspective projection transformation, the point cloud coordinate system coordinates are converted into the camera coordinate system coordinates. The perspective projection transformation formula is: in, is the camera intrinsic parameter matrix, Coordinate system transformation matrix, is the rotation correction matrix, and the camera coordinate system coordinates of point p are written as S13, normalize the obtained camera coordinate system coordinates to obtain As the two-dimensional plane coordinates corresponding to point p in the three-dimensional point cloud; All points are processed in steps S11, S12, and S13; S14, the original three-dimensional point cloud 4 channel information and distance information A total of 5 dimensional information is used as the dimensional information of the two-dimensional image obtained by perspective projection transformation, and finally a two-dimensional image is obtained. H is the image height and W is the image width.

3. The point cloud semantic segmentation method based on a cross-modal Transformer according to claim 1, characterized in that: In step S2, the four scale feature maps and The extraction method is as follows: S21. Project the given point cloud into a two-dimensional image and camera color images They are input into the ResNet network for preprocessing, and CNN convolution layer is used to extract 4 scale feature maps based on the ResNet network. and Specifically: the image obtained by processing in stage S1 and The 4 scale feature maps obtained by the ResNet network are 8 feature maps in total. The final feature map is and From largest to smallest: and 4. The point cloud semantic segmentation method based on a cross-modal Transformer according to claim 1, characterized in that: In step S3, the feature cross-modal self-attention is calculated as follows: S31, the feature map obtained in step S2 and Fusion scale l = 1 feature map and For example, input into Transformer for feature cross-modal self-attention calculation, the calculation steps are as follows: (1) For the input image features and The number of channels is changed from 256 to C through linear embedding. At this time, the initial input features are obtained and are recorded as g I and g S , g I Indicated by The initial input features obtained, g S Indicated by The initial input features obtained; (2) The obtained g I and g S Input to Transformer for processing: g I and g S First, after the normalization layer and self-attention calculation, it is combined with g I Add together to get Then get After being processed by the normalization layer and the multi-layer perceptron and added to itself, the cross-modal feature output F is finally obtained. l C , the formula is as follows: Among them, LN represents the normalization layer, linear represents the linear mapping layer, MLP represents the multi-layer perceptron, a, b, and c refer to the three input parameters, respectively, LN(g S )、LN(g I ) and LN(g I ); F1 C is the cross-modal image feature with scale l = 1; SA represents the self-attention calculation, and the calculation process of formula (4) is performed; C is the number of channels; φ represents the softmax function, and B is the batch size.

5. The point cloud semantic segmentation method based on a cross-modal Transformer according to claim 1, characterized in that: In step S4, the bilinear interpolation upsampling step and the CNN convolutional network processing method are as follows: S41, cross-modal image feature F1 with scale l=1 C For example, for a given cross-modal image feature F1 C , after bilinear interpolation, the image feature size is increased to twice its original size, and then processed by the CNN convolutional network. The specific results of the network are as follows: In the first convolutional layer, a convolution kernel of size 3×3 is defined, the stride is defined as 1, the padding value is 1, and a ReLU activation function and a batch normalization layer are added after the layer1 layer for normalization. In the second convolutional layer, a convolution kernel of size 3×3 is defined, with a stride of 1, a padding value of 2, and a dilation rate of 2. A ReLU activation function and a batch normalization layer are added after the layer2 layer for normalization. In the third convolutional layer, a convolution kernel of size 2×2 is defined, with a stride of 1, a padding value of 1, and a dilation rate of 2. A ReLU activation function and a batch normalization layer are added after the layer3 layer for normalization. In the fourth convolutional layer, a convolution kernel of size 1×1 is defined, and a stride of 1 is defined. A ReLU activation function and a batch normalization layer are added after the layer4 layer for normalization. S42: For the cross-modal image features with scale l=1, the processing of step S41 needs to be performed once to obtain the final feature map Wherein, H is the image height and W is the image width; cross-modal image features with scales l=2, l=3, and l=4 need to be processed by step S41 twice, three times, and four times respectively, in order to restore the cross-modal image features to the source image size.

6. The point cloud semantic segmentation method based on a cross-modal Transformer according to claim 1, characterized in that: In step S6, the method for generating the probability body T is as follows: S61, obtain feature map group After that, all feature maps are connected in the dimension direction and then passed through two convolutional layers to obtain the probability volume T. The calculation formula of the probability volume T is as follows: Among them, cat represents the connection operation, conv1 represents a 1×1 convolutional layer, and conv2 represents a 1×1 convolutional layer; is the final feature map with l=1, is the final feature map with l=2, is the final feature map with l=3, is the final feature map with l=4; S62. Perform a softmax operation on the probability body T along the dimensional direction to normalize the probability and obtain the segmentation result map P0.

7. The point cloud semantic segmentation method based on a cross-modal Transformer according to claim 1, characterized in that: In step S7, the perspective projection process is used to obtain the segmentation result of the three-dimensional point cloud as follows: S71. After obtaining the segmentation result image P0, calculate the loss between it and the true value, and train the entire network based on the obtained loss value. Iterate the network for 100 to 150 rounds. The loss function consists of two parts and is calculated as follows: (1) Calculation using the cross entropy loss function: Among them, q xy and p xy Represent the true value of the pixel at the image coordinate (x, y) and the predicted probability in the segmentation result map P0, respectively. B is the batch size, H is the image height, and W is the image width. (2) Calculation method of marginal loss function: where w x,y is the boundary weight of the pixel at the image coordinate (x, y), c x,y and c x+i,j+i are the true semantic labels of the pixels at the image coordinates (x, y), It is an exclusive OR operation; (3) Adding the above two together, the loss function is expressed as: L=L seg +L w (9) Among them, L seg represents the cross entropy loss in formula (6), L w represents the boundary loss in formula (8); S72. The segmentation result image P0 obtained after the training stops is the final segmentation image. After obtaining the segmentation image, the dimension value at the maximum probability is selected as the predicted category value. The category result corresponding to the three-dimensional point is obtained through the reverse transformation correspondence in step S12. All pixel points are reversely transformed to finally obtain the segmentation result of the source three-dimensional point cloud P0.

Images