Semantic segmentation method and device based on efficient multi-sensor fusion, equipment and medium

Abstract

The invention discloses a semantic segmentation method and device based on efficient multi-sensor fusion, equipment and a medium, and the method comprises the steps: collecting multi-modal data which comprises an RGB image and point cloud data; the multi-modal data are preprocessed respectively; inputting the preprocessed multi-modal data into a dual-network encoder, and respectively extracting image features and point cloud features; respectively inputting the image features and the point cloud features output by the dual-network encoder into a dual-network decoder, and respectively carrying out image feature recovery and point cloud feature recovery; inputting the decoded image features and the point cloud features into a multi-modal fusion module, and carrying out multi-modal data alignment and multi-modal data fusion; and inputting the aligned and fused multi-modal features into a predictor to carry out point cloud semantic classification prediction. Therefore, more accurate point cloud semantic prediction can be realized on the basis of ensuring multi-modal fusion lightweight based on a deep learning method, so that efficient multi-sensor fusion is achieved.

Description

Technical Field

[0001] The present invention relates to a semantic segmentation method, device, equipment and medium based on efficient multi-sensor fusion, and relates to the technical field of image and point cloud processing. Background Art

[0002] Point clouds have rich geometric information and can effectively represent the shape, size and spatial position of objects. Point cloud segmentation is the process of dividing point cloud data into different parts or objects, which is usually used to identify and extract objects of interest. In many applications, such as environmental perception of autonomous vehicles, robot grasping, 3D modeling of buildings, etc., accurate point cloud segmentation is the basis for efficient processing and decision-making. In autonomous driving systems, point cloud segmentation is a key step in environmental perception, which aims to divide the acquired point cloud data into different objects or regions to facilitate the understanding and analysis of the surrounding environment. Accurate point cloud segmentation is crucial for tasks such as obstacle detection, path planning and dynamic scene understanding. For example, autonomous vehicles need to identify pedestrians, vehicles, road boundaries and other important features in real time to ensure safe driving.

[0003] Traditional point cloud segmentation methods primarily include model-based segmentation methods and region growing algorithms. Model-based segmentation methods typically rely on known geometric models for segmentation, making them suitable for specific scenarios but less adaptable to new types of objects. Region growing algorithms, by selecting a seed point and expanding its neighborhood for segmentation, offer superior local feature extraction capabilities.

[0004] With advances in sensor technology, multi-sensor fusion has become an effective means of improving the accuracy and robustness of point cloud segmentation, overcoming the limitations of single sensors. The complementary information provided by different types of sensors can enhance the accuracy and reliability of environmental perception. Although point cloud segmentation methods based on multi-sensor fusion have achieved significant improvements in accuracy and robustness, they still face several challenges. Current point cloud and image multimodal fusion algorithms have limitations in efficiently fusing features. For example, these include effectively processing large-scale point cloud data, ensuring segmentation efficiency in real-time applications, and further improving the algorithm's adaptability to complex environments. Summary of the Invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, in response to the above-mentioned problems, the present invention aims to provide a semantic segmentation method, apparatus, device, and medium based on efficient multi-sensor fusion, which can achieve more accurate point cloud semantic prediction, thereby achieving efficient multi-sensor fusion.

[0006] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0007] In a first aspect, the present invention provides a semantic segmentation method based on efficient multi-sensor fusion, comprising:

[0008] Collect multimodal data, where the multimodal data includes RGB images and point cloud data;

[0009] Preprocess the multimodal data separately;

[0010] The preprocessed multimodal data is input into the dual network encoder to extract image features and point cloud features respectively;

[0011] The image features and point cloud features output by the dual network encoder are input into the dual network decoder respectively to perform image feature recovery and point cloud feature recovery respectively;

[0012] The decoded image features and point cloud features are input into the multimodal fusion module to perform multimodal data alignment and multimodal data fusion;

[0013] The aligned and fused multimodal features are input into the predictor for point cloud semantic classification prediction.

[0014] In some possible implementations, the multimodal data are preprocessed separately, including image preprocessing and / or point cloud data preprocessing, wherein:

[0015] Image preprocessing, including image cropping, brightness adjustment, contrast adjustment, saturation adjustment, hue adjustment, random horizontal flipping of images, and / or image data regularization;

[0016] Point cloud data preprocessing, including point cloud data rotation, flipping, scaling and / or noise addition.

[0017] In some possible implementations, the dual network encoder includes an image encoder and a point cloud encoder, wherein:

[0018] The image encoder is used to extract image features and includes several image branch encoders. The image branch encoder is provided with an image feature extraction module. The image feature extraction module is mainly composed of the ResNet backbone network, and the activation function adopts ReLU.

[0019] The point cloud encoder is used to extract point cloud features. The point cloud encoder includes several point cloud branch encoders. The point cloud branch encoder includes several point cloud feature extraction modules. The point cloud feature extraction module uses sparse convolution to extract point cloud features of the three-dimensional network. The SPVCNN based on the ResNet bottleneck structure is used as the encoder, and the activation function uses LeakyReLU.

[0020] In some possible implementations, the dual network decoder includes an image decoder and a point cloud decoder, wherein:

[0021] The image decoder includes several image multi-scale fusion modules: first, all feature maps are restored to the same resolution through upsampling; then they are merged through element-wise addition; finally, the fused features are passed through a classifier to obtain the semantic segmentation results of the two-dimensional network. During the image decoding process, linear interpolation modules and transposed convolution modules are used for upsampling. During each upsampling process, a convolution layer is introduced to extract finer detail features. Through continuous upsampling and convolution operations, the multi-scale features extracted by the encoder are effectively decoded and finally restored to the resolution of the original image.

[0022] The point cloud decoder includes several point cloud branch decoders and adopts the decoding structure of UNet: first, features of different scales are upsampled to the original resolution to recover point cloud features; then multi-scale features are spliced with the features of the encoder; then the spliced multi-scale features are classified in the channel direction, and finally the channel classification results are sent to the linear classifier, where the linear classifier is used to convert the point cloud features into the final classification results.

[0023] In some possible implementations, the multimodal feature fusion module includes a multimodal data alignment and enhancement module, wherein the implementation process of the multimodal data alignment and enhancement module is as follows:

[0024] First, the image and point cloud data are aligned, and the perspective projection method is used to calculate the point-to-pixel mapping. In the perspective projection process, the goal is to convert the point coordinates (x i ,y i ,z i ) is mapped to the corresponding pixel coordinates (u i ,v i ), the formula is as follows:

[0025]

[0026] Where P∈R 3×4 and T∈R 4×4 They are the internal parameter matrix and external parameter matrix of the camera respectively, and T is the original external matrix T r Add a fourth line 1 to the basis; after projection, the mapping from point to pixel is defined as:

[0027]

[0028] in, For value removal operation, M img is the corresponding image point;

[0029] Secondly, the self-attention mechanism is used to adjust information. In the self-attention mechanism, trainable external parameters are used as key vectors and value vectors. The key vector is used to determine the correlation between different input features, while the value vector carries the actual information content. The formula is as follows:

[0030]

[0031] Among them, EA represents the external attention mechanism, Q represents the query, EK and EV represent the external key and external value respectively, and d k is a scaling factor used to prevent gradient vanishing and improve the stability of the model.

[0032] In some possible implementations, the multimodal feature fusion module further includes a Transformer-based multimodal feature fusion module. The implementation process of the Transformer-based multimodal feature fusion module is as follows:

[0033] First, the image decoding features and the enhanced point cloud decoding features are taken as input, and the two-dimensional image is used as prior information. They are fused into a three-dimensional point cloud through matrix operations. That is, the image features and point cloud features are fused in a pixel-by-pixel manner. The fused feature map will be used as the query vector Query of the transformer.

[0034] Next, perform a maximum pooling operation on the spatial dimension of the image features;

[0035] Then, the multi-layer perceptron module is used to generate the key vector and value vector, and the formula is as follows:

[0036] K,V=σ(Max(F 2D ));

[0037] Among them, K, V represent the key vector and value vector, σ is the MLP and regularization operation, Max is the pooling operation, F 2D is the image feature;

[0038] Finally, through matrix operations between the query vector, key vector, and value vector, the fused feature representation is obtained, and its formula is as follows:

[0039]

[0040] Among them, F 2D 、F 3D and F 2D3D They represent image features, enhanced point cloud features and fusion features respectively. FFN is a feedforward network, which is mainly composed of MLP operations.

[0041] In some possible implementations, the predictor includes a knowledge distillation module and a point cloud semantic segmentation classifier. The knowledge distillation module is used to transfer feature knowledge, and the point cloud semantic segmentation classifier is used to perform semantic segmentation based on the point cloud features after knowledge distillation. The specific implementation process is as follows:

[0042] First, based on the knowledge distillation module, the fusion features output by the multimodal fusion module are applied to the decoding features of the point cloud branch for knowledge distillation, with the aim of transferring useful information in the fusion features to the point cloud branch.

[0043] Secondly, the point cloud features after knowledge distillation are input into the trained point cloud semantic segmentation classifier for point cloud semantic segmentation. The point cloud semantic segmentation classifier is composed of an MLP multi-layer perceptron and is trained through image features and point cloud features.

[0044] In a second aspect, the present invention provides a semantic segmentation device based on efficient multi-sensor fusion, comprising:

[0045] a data acquisition unit configured to acquire multimodal data, wherein the multimodal data includes RGB images and point cloud data;

[0046] a preprocessing unit, configured to preprocess the multimodal data separately;

[0047] The encoder unit is configured to input the preprocessed multimodal data into the dual network encoder to extract image features and point cloud features respectively;

[0048] The decoder unit is configured to input the image features and point cloud features output by the dual network encoder into the dual network decoder to perform image feature recovery and point cloud feature recovery respectively;

[0049] The fusion unit inputs the decoded image features and point cloud features into the multimodal fusion module to perform multimodal data alignment and multimodal data fusion;

[0050] The classification prediction unit is configured to input the aligned and fused multimodal features into the predictor to perform point cloud semantic classification prediction.

[0051] In a third aspect, the present invention also provides an electronic device comprising: at least one processor; and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor to enable the processor to execute the described method.

[0052] In a fourth aspect, the present invention further provides a computer-readable storage medium storing one or more programs, wherein the one or more programs include computer instructions, and the computer instructions are used to enable a computer to execute the method described above.

[0053] The present invention adopts the above technical solution, which has the following characteristics:

[0054] 1. The deep learning-based method of the present invention ensures lightweight multimodal fusion and can achieve more accurate point cloud semantic prediction, thereby achieving efficient multi-sensor fusion.

[0055] 2. The present invention not only achieves more accurate point cloud semantic segmentation prediction, but also takes real-time into consideration, ensuring that the algorithm can respond quickly when dealing with complex environments.

[0056] 3. Since existing point cloud and image multimodal fusion algorithms cannot efficiently fuse features, the present invention uses an efficient feature extraction and fusion mechanism to identify and segment various obstacles, pedestrians and other important elements in real time in dynamically changing autonomous driving scenarios.

[0057] In summary, the present invention can be widely used in autonomous driving scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. Throughout the drawings, the same reference numerals are used to denote the same components. In the drawings:

[0059] Figure 1 This is a flow chart of a semantic segmentation method based on efficient multi-sensor fusion according to an embodiment of the present invention;

[0060] Figure 2 This is a flowchart of point cloud and image data preprocessing according to an embodiment of the present invention;

[0061] Figure 3 This is a flowchart of a dual network encoder for point cloud and image according to an embodiment of the present invention;

[0062] Figure 4 This is a flowchart of a dual-network decoder for point cloud and image according to an embodiment of the present invention;

[0063] Figure 5 This is a multimodal fusion flow chart of an embodiment of the present invention;

[0064] Figure 6 A flowchart of a point cloud and image predictor according to an embodiment of the present invention;

[0065] Figure 7 FIG. 1 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] It should be understood that the terms used herein are for the purpose of describing specific example embodiments only and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "comprise", "include", "contain" and "have" are inclusive and therefore specify the presence of stated features, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring them to be performed in the specific order described or illustrated, unless the order of execution is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0067] Although the terms first, second, third, etc. can be used in the text to describe multiple elements, components, regions, layers and / or sections, these elements, components, regions, layers and / or sections should not be limited by these terms. These terms can only be used to distinguish an element, component, region, layer or section from another region, layer or section. Unless the context clearly indicates otherwise, terms such as "first", "second" and other numerical terms do not imply order or sequence when used in the text. Therefore, the first element, component, region, layer or section discussed below can be referred to as the second element, component, region, layer or section without departing from the teaching of the example embodiments.

[0068] For ease of description, spatially relative terms may be used herein to describe the relationship of one element or feature relative to another element or feature as shown in the figures, such as "inside," "outside," "inner side," "outer side," "lower," "upper," etc. Such spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures.

[0069] Developing efficient multi-sensor fusion point cloud segmentation methods based on deep learning is of great significance. First, it can significantly enhance environmental perception capabilities. By integrating data from multiple sensors such as lidar and RGB cameras, it overcomes the limitations of a single sensor, thereby achieving more accurate obstacle detection and scene understanding. Second, deep learning algorithms can handle complex and dynamic scenes, improving the robustness and adaptability of the system. This method also supports real-time data processing, enabling autonomous driving systems to quickly respond to environmental changes and ensure driving safety. In addition, the enhanced feature representation capability improves the generalization performance of the model, promotes technological innovation, and lays the foundation for the development of fields such as intelligent transportation and unmanned driving. Therefore, the development of such methods is crucial to promoting the popularization and application of intelligent systems. To address the problem that existing multimodal fusion methods in autonomous driving scenarios cannot efficiently fuse features, the present invention provides a semantic segmentation method, apparatus, device, and medium based on efficient multi-sensor fusion. The method comprises: collecting multimodal data, including RGB images and lidar point cloud data; preprocessing the multimodal data separately; inputting the preprocessed multimodal data into a 2D image feature encoder and a 3D point cloud feature encoder, respectively, to extract features such as color, texture, and edge information of the image, and depth features of the point cloud; inputting the image features and point cloud features output by dual-network encoding into a dual-network decoder, respectively, to perform image feature recovery and point cloud feature recovery; inputting the decoded multimodal features into a multimodal fusion module to perform multimodal data alignment and fusion; and inputting the fused multimodal features into a point cloud semantic segmentation classifier to perform point cloud semantic classification prediction. Therefore, the present invention can achieve higher-precision point cloud prediction and classification while ensuring lightweight multimodal feature fusion, thus realizing efficient multi-sensor fusion.

[0070] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0071] Example 1: In autonomous driving scenarios, traditional point cloud segmentation methods, including model-based segmentation, region growing, and clustering algorithms, perform well in specific scenarios but lack robustness in complex environments. While multi-sensor fusion methods based on deep learning have improved accuracy and robustness, they still face challenges in processing large-scale point cloud data, ensuring real-time segmentation efficiency, and improving adaptability to complex environments.

[0072] In order to solve the above problems, Figure 1As shown, the semantic segmentation method based on efficient multi-sensor fusion provided by this embodiment includes:

[0073] S1. Collect multimodal data, including RGB images and lidar point cloud data.

[0074] In this embodiment, an RGB camera is used to collect RGB image data, and a lidar sensor is used to collect lidar point cloud data.

[0075] S2. Preprocess the multimodal data. Data preprocessing can enhance the dataset samples and improve the model's generalization ability in unknown scenarios.

[0076] In this embodiment, data preprocessing is performed on the RGB image and the lidar point cloud data, specifically:

[0077] 1) Image preprocessing, including random position selection of the image to crop a fixed-shape image, random adjustment of image brightness, contrast, saturation and hue, random horizontal flipping of the image, and image data regularization.

[0078] 2) Point cloud data preprocessing, including small random rotation of point cloud data, point cloud flipping, point cloud scaling, and point cloud noise addition.

[0079] Furthermore, if Figure 2 As shown in Figure 1, image preprocessing is an important step in data enhancement, which aims to improve the generalization ability and robustness of the model through various techniques. Image preprocessing specifically includes the following:

[0080] Cropping: randomly select positions from the original image to crop an image of a fixed shape;

[0081] Brightness adjustment: Randomly adjust the image brightness to simulate scenes under different lighting conditions;

[0082] Contrast adjustment: randomly changing the contrast of an image to enhance the recognizability of objects in the image;

[0083] Saturation adjustment: Increase or decrease the intensity of colors by randomly adjusting the saturation of the image;

[0084] Hue adjustment: randomly change the hue of the image to make the image present different color styles;

[0085] Flip: Randomly flip the image horizontally to increase the diversity of samples;

[0086] Regularization: Regularize image data to ensure that the input data is within a certain range, usually by scaling pixel values to between 0 and 1. This process can accelerate the convergence of the model and improve training results.

[0087] Furthermore, if Figure 2 As shown in Figure 1, the preprocessing of point cloud data specifically includes the following:

[0088] Rotation: Performs a small random rotation on the point cloud data to simulate different viewing angles;

[0089] Flip: Flip the point cloud data, including flipping in the x, y, and z dimensions;

[0090] Scaling: Randomly scale the point cloud to simulate objects at different distances;

[0091] Noise: Add a certain amount of noise to the point cloud data to improve the robustness of the model.

[0092] S3. The preprocessed multimodal data are input into the dual network encoder to extract the image features such as color, texture, and edge information and the depth features of the point cloud.

[0093] In this embodiment, Figure 3 As shown, the dual network encoder includes a 2D image encoder and a 3D point cloud encoder.

[0094] Furthermore, the image encoder is used to extract image features, and includes several image branch encoders. The image branch encoder is provided with an image feature extraction module, which is composed of a ResNet backbone network and uses ReLU as the activation function. At the same time, this embodiment uses four downsamplings to generate multi-scale image features, wherein the first downsampling uses a maximum pooling module, and the other three downsamplings all use convolution downsampling with a step size of 2. Specifically, in the image encoding process, the present invention uses a structure similar to ResNet34: first, a 3×3 convolution layer is passed, which is responsible for extracting the basic information of the input features. Then, the feature map is downsampled by a max pooling operation, which aims to reduce the spatial dimension of the feature map while retaining the most significant features. The max pooling operation can effectively capture the main information in the feature map and enhance the model's attention to key information. Then it passes through 4 feature extraction modules, which respectively contain 3, 6, 4 and 3 ResNet basic modules. These ResNet basic modules effectively alleviate the vanishing gradient problem in deep networks through residual connections, ensuring efficient information flow, thereby accelerating model training and improving performance. Unlike ResNet34, the first module of the above ResNet feature extraction module does not use a downsampling operation with a stride of 2. The purpose of this design choice is to retain more detailed information to ensure that the spatial resolution of the input data can be fully utilized in the subsequent feature extraction process.

[0095] Furthermore, the point cloud encoder is used to extract point cloud features, including several point cloud branch encoders. The point cloud branch encoder includes several point cloud feature extraction modules. The point cloud branch encoder uses sparse convolution to extract point cloud features of the three-dimensional network. First, the point cloud data is mapped to the voxel dimension, and then the point cloud voxels are used as input for sparse convolution feature extraction. The convolution operation can ignore empty voxels. The present invention adopts SPVCNN based on the ResNet bottleneck structure as the encoder, uses LeakyReLU as the activation function, and the point cloud branch uses 4 sparse convolutions to generate multi-scale point cloud features. Specifically, in the point cloud encoding process, the present invention uses the SPVCNN module for feature extraction. The SPVCNN includes a voxel encoder, a point encoder and a fusion module. The specific process is:

[0096] First, the point cloud data needs to be mapped and converted into voxel format. This process is to transform the discrete point set in the point cloud into a regular three-dimensional grid structure to facilitate subsequent feature extraction and analysis. The formula is as follows:

[0097]

[0098] Where r is the voxel resolution of the corresponding layer, M voxel is the feature after voxelization, For value removal operation, x i 、y i 、z i is the coordinate of the point cloud, R is the dimension representation of the data, N is the number of point clouds, and i is the coordinate of the point cloud data.

[0099] Next, in the voxel encoder, the present invention uses SparseBasicBlock for feature extraction. This module is an efficient module specially designed to process sparse data and can effectively capture the potential feature information in the voxel.

[0100] Subsequently, the extracted voxel features are mapped back to the point cloud data. This mapping process ensures the association between the voxel features and the original point cloud, allowing subsequent feature extraction to fully utilize this information. In the point cloud encoder, the present invention uses a multi-layer perceptron (MLP) to extract point cloud features. MLP, through the combination of multiple fully connected layers, can capture the complex spatial relationships and feature structures in the point cloud, further enhancing the representation ability of the point cloud;

[0101] Finally, the features extracted by the point encoder are mapped back to voxels through the fusion module. At this point, the present invention uses sparse convolution features for fusion and pooling operations. Sparse convolution can effectively process sparse data while maintaining computational efficiency.

[0102] S4. The image features and point cloud features extracted by the dual network encoder are input into the dual network decoder for image feature recovery and point cloud feature recovery respectively.

[0103] In this embodiment, the image features and point cloud features extracted by the dual network encoder are input into the dual network decoder for image and point cloud feature recovery, such as Figure 4 As shown, the dual network decoder includes an image decoder and a point cloud decoder.

[0104] Furthermore, the image decoder includes several multi-scale image fusion modules, similar to the decoder in FCN. First, all feature maps need to be restored to the same resolution through upsampling, and then merged through element-wise addition. Finally, the fused features are passed through a classifier to obtain the semantic segmentation result of the two-dimensional network.

[0105] During the image decoding process, the present invention uses a linear interpolation module (UpsamplingNearest) and a transposed convolution module (Deconvolution) for upsampling. The combination of these two modules is intended to effectively improve the resolution of the image and ensure the integrity of the details. Among them, the linear interpolation module can quickly upsample, but it may cause the loss of image details. Therefore, the transposed convolution module is introduced to compensate for this loss to ensure that the image retains the original features as much as possible during the enlargement process; in addition, in each upsampling process, the present invention also introduces a 7×7 convolution layer to extract finer detail features. This convolution layer can capture the complex structure and texture information in the local area by means of a sliding window, thereby enhancing the detail performance of the image. Through continuous upsampling and convolution operations, the present invention can effectively decode the multi-scale features extracted by the encoder and ultimately restore the resolution of the original image.

[0106] Furthermore, the point cloud decoder includes several point cloud branch decoders, which adopt the decoding structure of UNet. It first upsamples features of different scales to the original resolution for point cloud feature recovery, then performs multi-scale feature splicing with the features of the encoder, and then classifies them in the channel direction, and finally sends them to the linear classifier.

[0107] During the point cloud decoding process, the point cloud branch decoder of the present invention adopts the decoding structure of UNet: the decoder first gradually expands the feature maps of different scales through a series of upsampling operations. After each upsampling, the corresponding low-level features from the encoder are combined to retain more detail information and context information; after the features are restored to the original resolution, the present invention classifies the restored features in the channel direction. The purpose of this step is to map each point cloud feature to the corresponding category label, thereby achieving accurate recognition of the object; finally, these classified features are sent to the linear classifier, which is responsible for converting the point cloud features into the final classification results. Through such a structural design, the point cloud branch decoder can effectively process and analyze point cloud data to achieve efficient semantic segmentation or object recognition tasks.

[0108] S5. Input the image features and point cloud features output by the decoder into the multimodal fusion module, and perform multimodal data alignment and multimodal data fusion operations in sequence.

[0109] In this embodiment, the image features and point cloud features output by the decoder are input into the multimodal fusion module, and multimodal data alignment and fusion operations are performed in sequence. Figure 5 As shown in Figure 3, the multimodal fusion module consists of two parts: a multimodal data alignment and enhancement module and a Transformer-based multimodal feature fusion module.

[0110] Multimodal data alignment and enhancement module: First, the image and point cloud data need to be aligned. For the relationship between the point cloud and the image, the present invention uses the perspective projection method to calculate the mapping from point to pixel. In the perspective projection process, the purpose is to transform the point cloud from the point coordinate p to the pixel. i =(x i ,y i ,z i )∈R 3 Projected to pixel coordinate p i =(u i ,v i )∈R 2 , the formula is as follows:

[0111]

[0112] Where P∈R 3×4 and T∈R 4×4 They are the internal parameter matrix and external parameter matrix of the camera respectively, and T is the original external matrix T r On this basis, add a fourth line with value 1.

[0113] After projection, the point-to-pixel mapping is defined as:

[0114]

[0115] in, For value removal operation, M img is the corresponding image point.

[0116] Since there are information differences between multimodal data, this difference may affect the subsequent point cloud prediction. Therefore, the main purpose of the multimodal data enhancement module is to narrow these gaps to achieve more effective information fusion. To this end, the present invention adopts a self-attention mechanism for information adjustment, which can flexibly focus on important features in different modal data. In the self-attention mechanism, trainable external parameters are used as key vectors (Key) and value vectors (Value). The key vector is used to determine the correlation between different input features, while the value vector carries the actual information content. This design enables the model to adaptively adjust the information flow between different modalities, so as to better capture important features. The formula is as follows:

[0117]

[0118] Where EA stands for external attention mechanism, Q stands for query, EK and EV stand for external key and external value respectively. k is a scaling factor used to prevent gradient vanishing and improve the stability of the model.

[0119] Furthermore, a multimodal feature fusion module based on Transformer: First, the image decoding features and the enhanced point cloud decoding features are taken as input, and the two-dimensional image is used as prior information to fuse them into a three-dimensional point cloud through matrix operations. Specifically, the present invention adopts a pixel-by-pixel addition method to fuse the image features with the point cloud features. Specifically, the same local points are added and summed by mapping the image features to the point cloud feature positions. The fused feature map will be used as the query vector (Query) of the transformer; then, the spatial dimension of the image features is subjected to a maximum pooling operation (Maxpooling) to reduce the number of parameters. Subsequently, a multi-layer perceptron (MLP) module is used to generate a key vector (Key) and a value vector (Value), and the formula is as follows:

[0120] K,V=σ(Max(F 2D ));

[0121] Among them, K, V represent the key vector (Key) and value vector (Value), σ is the MLP and regularization operation, Max is the pooling operation, F 2D is the image feature.

[0122] Since the computational complexity of the ordinary Transformer module is large, this paper uses a one-dimensional convolution module to process the key and value, thereby performing a convolution operation on the fused features. Finally, through matrix operations between the query, key, and value vectors, the fused feature representation is obtained, and its formula is as follows:

[0123]

[0124] Among them, F 2D 、F 3D and F 2D3D They represent image features, enhanced point cloud features, and fusion features respectively. FFN is a feedforward network, which is mainly composed of MLP operations.

[0125] S6. Input the image decoding features, point cloud decoding features and multimodal fusion features into the predictor respectively, perform knowledge transfer of the feature knowledge distillation module and semantic prediction of the point cloud semantic segmentation classifier in turn, and finally realize point cloud semantic segmentation prediction.

[0126] In this embodiment, the image decoding features, point cloud decoding features and multimodal fusion features are input into the predictor respectively, and the knowledge transfer of the feature knowledge distillation module and the semantic prediction of the point cloud semantic segmentation classifier are performed in sequence. Figure 6 As shown in Figure 2, the predictor includes a knowledge distillation module and a point cloud semantic segmentation classifier, where:

[0127] First, the fused features output by the multimodal fusion module are applied to the decoded features of the point cloud branch to perform knowledge distillation. This process aims to transfer useful information from the fused features to the point cloud branch to improve its representation ability and accuracy. The knowledge distillation loss function formula is as follows:

[0128]

[0129] Among them, x i represents the input data, p and q represent the predicted outputs of the two models, and L KL is the KL divergence loss function.

[0130] Next, the decoder output features of the image branch and the point cloud branch decoder output features are input into the point cloud semantic segmentation classifier, which is composed of an MLP multi-layer perceptron. The loss function is calculated between the output value of the point cloud semantic segmentation classifier and the true value of the label. The purpose of this step is to evaluate the performance of the model and improve feature learning by optimizing the loss function. The present invention uses the image branch loss function, the point cloud branch loss function and the multimodal feature loss function in the training process. The total loss function formula is as follows:

[0131] L=λ1L 2D +λ2L3D +λ3L KL ;

[0132] Among them, λ1, λ2, and λ3 are the weight coefficients of the image branch loss function, the point cloud branch loss function, and the multimodal feature loss function, respectively. 2D and L 3D Both use cross entropy and lovasz loss function, that is, L 2D =L c +L jc 、L 3D =L c +L jc , L KL The KL divergence loss function is used.

[0133] Among them, the loss function uses cross entropy and lovasz loss function:

[0134] The cross entropy loss function formula is as follows:

[0135]

[0136] Where log represents the natural logarithm with base e, M represents the number of categories, and N represents the number of samples. ic The value is 0 or 1. If the true value of sample i is equal to c, it takes 1, otherwise it takes 0. ic The model predicts the probability value.

[0137] The lovasz loss function formula is as follows:

[0138]

[0139] Among them, y * represents the true label, represents the predicted label, ∩ and ∪ represent the intersection and union operations respectively.

[0140] Finally, in the inference stage, only the point cloud branch is used for semantic segmentation prediction of each point cloud. Specifically, the output features of the point cloud branch decoder are input into the point cloud classifier for point cloud segmentation. This process can reduce the time consumption of the image branch network while further improving the prediction accuracy of the point cloud branch.

[0141] Example 2: The above-mentioned Example 1 provides a semantic segmentation method based on efficient multi-sensor fusion. Correspondingly, this embodiment provides a semantic segmentation device based on efficient multi-sensor fusion. The device provided in this embodiment can implement the semantic segmentation method based on efficient multi-sensor fusion of Example 1, and the device can be implemented by software, hardware, or a combination of software and hardware. For the convenience of description, this embodiment is described by dividing the functions into various units and describing them separately. Of course, the functions of each unit can be implemented in the same or multiple software and / or hardware during implementation. For example, the device may include integrated or separate functional modules or functional units to perform the corresponding steps in each method of Example 1. Since the device of this embodiment is basically similar to the method embodiment, the description process of this embodiment is relatively simple, and the relevant parts can refer to the partial description of Example 1. The embodiment of the semantic segmentation device based on efficient multi-sensor fusion provided by the present invention is only schematic.

[0142] Example 3: This example provides an electronic device corresponding to the semantic segmentation method based on efficient multi-sensor fusion provided in Example 1. The electronic device may be an electronic device for a client, such as a mobile phone, a laptop computer, a tablet computer, a desktop computer, etc., to execute the method of Example 1.

[0143] like Figure 7 As shown, the electronic device includes a processor, a memory, a communication interface and a bus. The processor, the memory and the communication interface are connected via the bus to complete communication between them. The memory stores a computer program that can be run on the processor. When the processor runs the computer program, it executes the method of embodiment 1. Its implementation principle and technical effect are similar to those of embodiment 1 and will not be repeated here. It can be understood by those skilled in the art that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computing device to which the solution of the present application is applied. The specific computing device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0144] In a preferred embodiment, the logic instructions in the above-mentioned memory can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), optical disk and other media that can store program code.

[0145] In a preferred embodiment, the processor may be a central processing unit (CPU), a digital signal processor (DSP), or other general-purpose processors of various types, which are not limited herein.

[0146] Embodiment 4: This embodiment provides a computer-readable storage medium storing one or more programs, wherein the one or more programs include computer instructions. When the computer instructions are executed by a computer, the computer executes the method provided in the above embodiment 1.

[0147] In a preferred embodiment, a computer-readable storage medium may be a tangible device that retains and stores instructions executed by the computer, such as, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof. The computer-readable storage medium stores computer program instructions that cause a computer to execute the method provided in the first embodiment.

[0148] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (apparatus), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0149] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0150] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0151] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In the description of this specification, the reference terms "a preferred embodiment", "further", "specifically", "in the present embodiment", etc. mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of this specification. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A semantic segmentation method based on efficient multi-sensor fusion, characterized in that: include: Collect multimodal data, where the multimodal data includes RGB images and point cloud data; Preprocess the multimodal data separately; The preprocessed multimodal data is input into the dual network encoder to extract image features and point cloud features respectively; The image features and point cloud features output by the dual network encoder are input into the dual network decoder respectively to perform image feature recovery and point cloud feature recovery respectively; The decoded image features and point cloud features are input into the multimodal fusion module to perform multimodal data alignment and multimodal data fusion; The aligned and fused multimodal features are input into the predictor for point cloud semantic classification prediction.

2. The semantic segmentation method based on efficient multi-sensor fusion according to claim 1, characterized in that The multimodal data are preprocessed separately, including image preprocessing and / or point cloud data preprocessing, wherein: Image preprocessing, including image cropping, brightness adjustment, contrast adjustment, saturation adjustment, hue adjustment, random horizontal flipping of images, and / or image data regularization; Point cloud data preprocessing, including point cloud data rotation, flipping, scaling and / or noise addition.

3. The semantic segmentation method based on efficient multi-sensor fusion according to claim 1, characterized in that The dual network encoder includes an image encoder and a point cloud encoder, where The image encoder is used to extract image features and includes several image branch encoders. The image branch encoder is provided with an image feature extraction module. The image feature extraction module is mainly composed of the ResNet backbone network, and the activation function adopts ReLU. The point cloud encoder is used to extract point cloud features. The point cloud encoder includes several point cloud branch encoders. The point cloud branch encoder includes several point cloud feature extraction modules. The point cloud feature extraction module uses sparse convolution to extract point cloud features of the three-dimensional network. The SPVCNN based on the ResNet bottleneck structure is used as the encoder, and the activation function uses LeakyReLU.

4. The semantic segmentation method based on efficient multi-sensor fusion according to claim 1, characterized in that The dual network decoder includes an image decoder and a point cloud decoder, where: The image decoder includes several image multi-scale fusion modules: first, all feature maps are restored to the same resolution through upsampling; then they are merged through element-wise addition; finally, the fused features are passed through a classifier to obtain the semantic segmentation results of the two-dimensional network. During the image decoding process, linear interpolation modules and transposed convolution modules are used for upsampling. During each upsampling process, a convolution layer is introduced to extract finer detail features. Through continuous upsampling and convolution operations, the multi-scale features extracted by the encoder are effectively decoded and finally restored to the resolution of the original image. The point cloud decoder includes several point cloud branch decoders and adopts the decoding structure of UNet: first, features of different scales are upsampled to the original resolution to recover point cloud features; then multi-scale features are spliced with the features of the encoder; then the spliced multi-scale features are classified in the channel direction, and finally the channel classification results are sent to the linear classifier, where the linear classifier is used to convert the point cloud features into the final classification results.

5. The semantic segmentation method based on efficient multi-sensor fusion according to claim 1, characterized in that The multimodal feature fusion module includes a multimodal data alignment and enhancement module. The implementation process of the multimodal data alignment and enhancement module is as follows: First, the image and point cloud data are aligned, and the perspective projection method is used to calculate the point-to-pixel mapping. In the perspective projection process, the goal is to convert the point coordinates (x i ,y i ,z i ) is mapped to the corresponding pixel coordinates (u i ,v i ), the formula is as follows: Where P∈R 3×4 and T∈R 4×4 They are the internal parameter matrix and external parameter matrix of the camera respectively, and T is the original external matrix T r Add a fourth line 1 to the basis; after projection, the mapping from point to pixel is defined as: in, For value removal operation, M img is the corresponding image point; Secondly, the self-attention mechanism is used to adjust information. In the self-attention mechanism, trainable external parameters are used as key vectors and value vectors. The key vector is used to determine the correlation between different input features, while the value vector carries the actual information content. The formula is as follows: Among them, EA represents the external attention mechanism, Q represents the query, EK and EV represent the external key and external value respectively, and d k is a scaling factor used to prevent gradient vanishing and improve the stability of the model.

6. The semantic segmentation method based on efficient multi-sensor fusion according to claim 5, characterized in that: The multimodal feature fusion module also includes a Transformer-based multimodal feature fusion module. The implementation process of the Transformer-based multimodal feature fusion module is as follows: First, the image decoding features and the enhanced point cloud decoding features are taken as input, and the two-dimensional image is used as prior information. They are fused into a three-dimensional point cloud through matrix operations. That is, the image features and point cloud features are fused in a pixel-by-pixel manner. The fused feature map will be used as the query vector Query of the transformer. Next, perform a maximum pooling operation on the spatial dimension of the image features; Then, the multi-layer perceptron module is used to generate the key vector and value vector, and the formula is as follows: K,V=σ(Max(F 2D )); Among them, K, V represent the key vector and value vector, σ is the MLP and regularization operation, Max is the pooling operation, F 2D is the image feature; Finally, through matrix operations between the query vector, key vector, and value vector, the fused feature representation is obtained, and its formula is as follows: Among them, F 2D 、F 3D and F 2D3D They represent image features, enhanced point cloud features and fusion features respectively. FFN is a feedforward network, which is mainly composed of MLP operations.

7. The semantic segmentation method based on efficient multi-sensor fusion according to claim 1, characterized in that The predictor includes a knowledge distillation module and a point cloud semantic segmentation classifier. The knowledge distillation module is used to transfer feature knowledge, and the point cloud semantic segmentation classifier is used to perform semantic segmentation based on the point cloud features after knowledge distillation. The specific implementation process is as follows: First, based on the knowledge distillation module, the fusion features output by the multimodal fusion module are applied to the decoding features of the point cloud branch for knowledge distillation, with the aim of transferring useful information in the fusion features to the point cloud branch. Secondly, the point cloud features after knowledge distillation are input into the trained point cloud semantic segmentation classifier for point cloud semantic segmentation. The point cloud semantic segmentation classifier is composed of an MLP multi-layer perceptron and is trained through image features and point cloud features.

8. A semantic segmentation device based on efficient multi-sensor fusion, characterized in that: include: a data acquisition unit configured to acquire multimodal data, wherein the multimodal data includes RGB images and point cloud data; a preprocessing unit, configured to preprocess the multimodal data separately; The encoder unit is configured to input the preprocessed multimodal data into the dual network encoder to extract image features and point cloud features respectively; The decoder unit is configured to input the image features and point cloud features output by the dual network encoder into the dual network decoder to perform image feature recovery and point cloud feature recovery respectively; The fusion unit inputs the decoded image features and point cloud features into the multimodal fusion module to perform multimodal data alignment and multimodal data fusion; The classification prediction unit is configured to input the aligned and fused multimodal features into the predictor to perform point cloud semantic classification prediction.

9. An electronic device, characterized in that: include: at least one processor; and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor to enable the processor to perform the method according to any one of claims 1-7.

10. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include computer instructions for causing a computer to execute the method according to any one of claims 1 to 7.

Images