A multi-source sensing method applicable to unstructured scenarios of engineering machinery

Abstract

This invention provides a multi-source perception method suitable for unstructured scenarios in engineering machinery, belonging to the field of environmental perception technology for engineering machinery. First, LiDAR point cloud data is preprocessed to generate a 2D perspective view aligned with the RGB image, providing a foundation for subsequent feature extraction. The image branch utilizes an improved deep learning network for semantic segmentation, while the point cloud branch extracts spatial depth features through an optimized network structure. The two branches complement each other through an FFAM module, significantly enhancing the ability to understand complex scenes. Furthermore, a refinement module is introduced, employing advanced 3D sparse convolution technology to optimize the back-projected point cloud features, effectively solving the problem of blurred target boundary segmentation.

Description

Technical Field

[0001] This invention relates to the field of environmental perception technology for engineering machinery, and specifically to a multi-source perception method applicable to unstructured scenarios of engineering machinery. Background Technology

[0002] With the rapid development of industrial automation and intelligence, construction machinery plays an increasingly important role in modern construction. The intelligent upgrading of construction machinery can not only improve operational efficiency but also significantly enhance operational safety and accuracy. However, construction machinery typically operates in complex, unstructured environments, such as mines and construction sites. These environments are characterized by complex terrain, varied obstacle shapes, and unstable lighting conditions, posing significant challenges to the autonomous operation of construction machinery.

[0003] In the intelligentization process of construction machinery, environmental perception technology is a core component for achieving autonomous operation. Environmental perception systems need to be able to identify and understand various information in the work environment in real time and accurately, including passable areas, obstacle locations, and work targets. However, existing environmental perception technologies still face many challenges in unstructured scenarios. On the one hand, traditional perception systems are mostly based on single sensors, such as cameras or LiDAR, which have significant limitations in complex environments. For example, cameras struggle to accurately acquire information in low-light or strong light reflection conditions, while LiDAR, although providing depth information, offers sparse data lacking texture detail. On the other hand, while multi-sensor fusion technology can compensate for the shortcomings of single sensors, effectively fusing data from different sensors and leveraging their complementary advantages remains a problem that urgently needs to be solved in practical applications.

[0004] Furthermore, the operating scenarios of construction machinery differ significantly from those of traditional autonomous driving. Construction machinery needs to perform complex tasks in dynamically changing environments, such as excavation, loading, and transportation. This requires the perception system to not only accurately identify static objects but also detect dynamic obstacles in real time and respond quickly. However, existing perception systems often suffer from delays or misjudgments when processing dynamic scenarios, making it difficult to meet the high-precision operation requirements of construction machinery.

[0005] In the field of multi-sensor fusion, although existing research has attempted to fuse image data and point cloud data, most current fusion methods suffer from problems such as low fusion accuracy and blurred target boundary segmentation. For example, simple data stitching or weighted averaging methods cannot fully utilize the advantages of both modalities, while deep learning-based fusion methods, although theoretically offering better performance, still face challenges in practical applications such as high model complexity and insufficient training data. Furthermore, most existing fusion methods are designed for structured scenarios, and their performance significantly degrades in unstructured scenarios, such as mines or construction sites.

[0006] In view of the above, this application is hereby submitted. Summary of the Invention

[0007] This invention provides a multi-source sensing method applicable to unstructured scenarios in engineering machinery, which can at least partially improve the above-mentioned problems.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A multi-source sensing method applicable to unstructured scenarios in engineering machinery includes:

[0010] The original LiDAR point cloud data and RGB image are acquired, and the original LiDAR point cloud data is preprocessed through the perspective projection module to construct a 2D point cloud perspective view that is geometrically aligned with the RGB image.

[0011] The image branching module is used to perform feature extraction and semantic segmentation on the RGB image to generate a semantic segmentation prediction map with the same spatial resolution as the RGB image.

[0012] Based on the semantic segmentation prediction map, a point cloud branch network is used to perform spatial depth feature extraction and cross-modal fusion processing on the 2D point cloud perspective view to obtain a 2D point cloud feature map.

[0013] The 2D point cloud feature map is optimized using a refinement module to obtain the refined multi-source perception results of the unstructured scene.

[0014] In summary, the multi-source perception method applicable to unstructured scenarios in engineering machinery employs a unique dual-branch network architecture, combining a Feature Fusion Attention (FFAM) module and refined network design. First, LiDAR point cloud data is preprocessed to generate a 2D perspective view aligned with the RGB image, providing a foundation for subsequent feature extraction and fusion. The image branch utilizes a deep learning network for semantic segmentation, while the point cloud branch extracts spatial depth features through an improved network structure. Through the FFAM module, the system achieves complementary fusion of image and point cloud features, fully leveraging the advantages of both data modalities and enhancing the ability to understand complex scenes.

[0015] In addition, a refined point cloud processing module was introduced, employing advanced 3D sparse convolution technology to optimize point cloud features, effectively solving the problem of blurred target boundary segmentation. This module not only enhances the spatial feature representation capability of point clouds but also provides more accurate perception results for the autonomous operation of engineering machinery.

[0016] This method demonstrates significant innovation and practicality in several aspects. It can operate stably in complex environments (such as insufficient lighting or occlusion), providing reliable perception support for the autonomous operation of construction machinery. The design of this method considers the needs of practical application scenarios, possessing high efficiency, robustness, and flexibility, and can meet the application needs of different construction machinery in diverse unstructured scenarios. Furthermore, the refined network design of this method optimizes computational efficiency, making it more feasible in practical applications.

[0017] Compared with existing technologies, the proposed multi-source perception method for unstructured scenarios of construction machinery has the following advantages: 1. Addressing the challenge of environmental perception for construction machinery, a multi-source perception-based unstructured scenario perception system for construction machinery is designed. Through a multi-sensor fusion architecture, the data coupling between modules is reasonable, and there is no methodological coupling, improving the overall integrity and stability of the system. This enables it to operate stably in unstructured scenarios, meeting the perception requirements for the movement and operation of construction machinery and realizing various key functions. 2. An innovative dual-branch network architecture bridged by a feature fusion module is constructed. The image branch uses ResNet34 as the backbone network, while the point cloud branch is built based on an improved SalsaNext, and perspective projection is used instead of spherical projection, thus preserving the geometric consistency between LiDAR perspective views and RGB images. Through the feature fusion module, effective fusion of multi-source features is achieved, fully leveraging the complementary advantages of image vision and point cloud spatial depth information, and improving the scene understanding ability of construction machinery in unstructured environments. 3. By introducing a refinement module, an overall network architecture from coarse to fine is constructed, which effectively improves the problem of blurred target boundary segmentation in unstructured scenarios, significantly enhances the spatial feature representation capability of point clouds, and makes the target boundary segmentation results more accurate and clear, providing reliable protection for the safe movement and precise operation of engineering machinery. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a multi-source sensing method applicable to unstructured scenarios in engineering machinery, provided in an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the image branching network framework provided in an embodiment of the present invention;

[0020] Figure 3This is a schematic diagram of the encoder network framework for point cloud branches provided in an embodiment of the present invention;

[0021] Figure 4 This is a schematic diagram of the decoder network framework for point cloud branches provided in an embodiment of the present invention;

[0022] Figure 5 This is a schematic diagram of the Feature Fusion Attention Module (FFAM) provided in an embodiment of the present invention;

[0023] Figure 6 This is a schematic diagram of the framework structure of the refinement module provided in an embodiment of the present invention;

[0024] Figure 7 This is a schematic diagram of the overall network flow of the multi-source sensing method applicable to unstructured scenarios of engineering machinery provided in this embodiment of the invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0026] refer to Figure 1 , Figure 7 As shown, the first embodiment of the present invention discloses a multi-source sensing method suitable for unstructured scenarios of engineering machinery. This method can be executed by a multi-source sensing device suitable for unstructured scenarios of engineering machinery (hereinafter referred to as the sensing device), specifically by one or more processors within the sensing device, to implement the following method:

[0027] S1. Acquire raw LiDAR point cloud data and RGB image, and preprocess the raw LiDAR point cloud data through the perspective projection module to construct a 2D point cloud perspective view that is geometrically aligned with the RGB image.

[0028] Specifically, step S1 further includes: acquiring raw LiDAR point cloud data. RGB images captured by an RGB camera The original LiDAR point cloud data is filtered, retaining only the forward-looking point cloud data. The original LiDAR point cloud data is 3D point cloud data. Each point They are respectively defined by Cartesian coordinates It consists of the reflectivity value r. For the set of real numbers, The number of point clouds, The height of the image. The width of the image. For the i-th point cloud, Let x be the x-coordinate of the i-th point cloud. Let be the y-coordinate of the i-th point cloud. Let be the z-axis coordinate of the i-th point cloud. Let be the reflectance value of the i-th point cloud;

[0029] Based on the preset extrinsic parameter matrix between multiple sensors, the filtered raw LiDAR point cloud data is projected onto a two-dimensional image coordinate system to obtain the projected image. , The number of channels in the projected point cloud, where the raw LiDAR point cloud data is extracted. Spatial coordinate information of each point The extracted features are concatenated in the fourth column to form a vector containing only 1s, expanded to four channels, and then the combined features are transposed to obtain the final vector. Each point cloud is ;

[0030] Obtain the projected point cloud in the camera coordinate system The i-th projected point cloud The image coordinates are discretized into pixel coordinates, thus obtaining the corresponding pixel coordinates of the point cloud in the projected image. , , ,in, This is the intrinsic parameter matrix of the camera. This is the rotation matrix for transforming from the LiDAR coordinate system to the camera coordinate system;

[0031] By aligning the front-view point cloud with the front-view RGB image through perspective projection, a 2D point cloud perspective view is obtained. As input data for the point cloud branch, the 2D perspective view Each pixel in It stores 5 channels of information, and the pixel positions to which the point cloud is projected store LiDAR features. , This represents the distance value for each point; for other pixel positions that are not projected, the value is stored as 0.

[0032] The projection process preserves five channels of geometric features of the point cloud, including depth, spatial coordinates, and reflection intensity, and establishes a correspondence between point cloud data and image space.

[0033] In this embodiment, raw LiDAR point cloud data and RGB images are acquired, and the raw LiDAR point cloud data is preprocessed using a perspective projection module to construct a 2D point cloud perspective view geometrically aligned with the RGB image. This process lays the foundation for subsequent multi-source data fusion, ensuring that point cloud data and image data can be effectively processed within the same geometric space.

[0034] Specifically, let's set A training sample provided for the dataset. It is each point P in the point cloud P. i The semantic tag set. Since the camera only acquires the front view image. In order to fuse the point cloud projection data with the image data, the point cloud is filtered, retaining only the front view point cloud ( The process involves acquiring raw 3D point cloud data from a LiDAR sensor and RGB images from an RGB camera. The raw LiDAR point cloud data is preprocessed, retaining only the forward-looking point cloud data to reduce data volume and focus on the forward-looking area relevant to the construction machinery operation. Each point in the raw LiDAR point cloud data consists of its Cartesian coordinates and reflectance value, providing rich information for subsequent feature extraction and fusion.

[0035] Next, based on the preset extrinsic parameter matrix between the multiple sensors, the filtered raw LiDAR point cloud data is projected into a two-dimensional image coordinate system. This projection process involves extracting the spatial coordinate information of the point cloud data and expanding it to four channels, with the fourth channel being an all-one vector to facilitate subsequent matrix operations. By transposing the point cloud data and combining it with the camera's intrinsic parameter matrix and the rotation matrix from LiDAR to the camera coordinate system, the projected point cloud in the camera coordinate system can be obtained. Furthermore, the image coordinates of the projected point cloud are discretized into pixel coordinates, thereby determining the position of each point cloud in the two-dimensional image space.

[0036] Through the aforementioned perspective projection operation, precise alignment of the front-view point cloud with the front-view RGB image was achieved, generating a 2D point cloud perspective view. This 2D point cloud perspective view serves as input data for the point cloud branch, with each pixel storing 5-channel geometric feature information, including point cloud depth, spatial coordinates, and reflection intensity. This multi-channel feature representation not only preserves the depth information of the point cloud but also integrates spatial location and reflection characteristics, providing a rich data foundation for subsequent feature extraction and fusion. Simultaneously, this alignment method establishes a direct correspondence between point cloud data and image space, enabling effective interaction and fusion of the two modalities in subsequent processing.

[0037] This step effectively solves the alignment problem in multi-source data fusion, providing a high-quality data foundation for subsequent feature extraction and fusion. This precise alignment method not only improves the efficiency of data fusion but also enhances the system's adaptability to complex unstructured scenarios, providing more accurate and reliable perception support for the autonomous operation of construction machinery.

[0038] Please see Figure 2S2, The image branching module is used to perform feature extraction and semantic segmentation processing on the RGB image to generate a semantic segmentation prediction map with the same spatial resolution as the RGB image;

[0039] Specifically, step S2 further includes: the image branch module adopts an encoder-decoder structure, takes the RGB image as the input of the image encoder of the image branch module, performs feature extraction, and obtains image features, wherein the backbone network of the image encoder is built based on ResNet34 and is used for 2D scene segmentation tasks;

[0040] The image decoder of the image branch module is used for image semantic segmentation tasks. It performs semantic segmentation processing on image features using the image decoder and uses U-Net's skip connection technology to fuse multi-scale features from the encoder and decoder to generate a semantic segmentation prediction map with the same spatial resolution as the RGB image.

[0041] The image encoder consists of one ordinary convolutional layer and four residual block groups. Based on a multi-scale feature extraction strategy, it improves the abstract semantics of features layer by layer. Specifically, the stride of the first convolutional layer of the original ResNet is modified to 1, the downsampling operation is removed, the full resolution of the input RGB image is preserved, and a 64-channel feature map is output through a convolutional layer with a 7×7 kernel. Combined with BN and ReLU activation functions, the feature map size is reduced to 1 / 2 of the original input after max pooling.

[0042] The residual block group is designed hierarchically, with the number of channels and resolution varying layer by layer with network depth. The Layer 1 blockchain consists of 3 residual blocks ResBlock_1, each containing 2 3×3 convolutional layers. Output and input features are fused through skip connections, with the formula as follows: ,in, It is a nonlinear transformation consisting of two 3×3 convolutional layers. The feature map is the output after skip connections. Input feature map;

[0043] The Layer 2 blockchain consists of four residual blocks, ResBlock_2. The first residual block downsamples and expands the number of channels to 128 through a convolution with a stride of 2. Subsequent residual blocks maintain the same number of channels and continue feature extraction.

[0044] The Layer3 blockchain consists of 6 residual blocks ResBlock_3. The first residual block downsamples and expands the number of channels to 256 through a convolution with a stride of 2. Subsequent residual blocks continue feature extraction with the same number of channels and introduce a dropout layer for regularization.

[0045] The Layer4 blockchain consists of three residual blocks, ResBlock_4. The first residual block downsamples and expands the number of channels to 256 through a convolution with a stride of 2. Subsequent residual blocks continue feature extraction while maintaining the same number of channels, and a dropout layer is introduced for regularization.

[0046] During training, the image encoder uses ImageNet pre-trained weights to initialize its parameters.

[0047] The image decoder adopts a symmetrical upsampling structure, which gradually restores the spatial resolution of the feature map through four decoding blocks. Each upsampling module includes feature convolution, LeakyReLU activation function, BN and bilinear interpolation upsampling.

[0048] In this process, 3×3 convolution is used for feature convolution to fix the features to 16 channels. The upsampling module uses bilinear interpolation to double the size of the feature map. After processing by the first three decoding blocks, it follows feature fusion by concatenating the upsampled features with the features of the corresponding layer of the encoder along the channel dimension. The formula is as follows: , This is the output feature map after concatenation. This is the feature map after upsampling. For splicing operations;

[0049] At the end of the image encoder, a 3×3 convolution is used to map the feature channel dimension to the number of semantic categories, and the Softmax function is used to calculate the category probability distribution of each pixel.

[0050] In this embodiment, the image branch module adopts an encoder-decoder structure. The encoder is based on ResNet34 but has been specifically improved for scene segmentation tasks, optimized specifically for 2D scene segmentation. Using an RGB image as input, features are first extracted through a standard convolutional layer with a 7×7 kernel, outputting a 64-channel feature map. Batch normalization (BN) and ReLU activation functions are then used for non-linear transformation. To preserve the full resolution of the input image, the stride of the first convolutional layer in the original ResNet is modified to 1, and downsampling is removed. After max pooling, the feature map size is reduced to half the original input size. This process not only extracts low-level image features but also lays the foundation for subsequent feature extraction. Furthermore, high-level semantic features are extracted. To improve the model's generalization ability, the encoder uses ImageNet pre-trained weights to initialize parameters. The image encoder provides the image feature foundation for multi-source feature fusion on the point cloud branch.

[0051] The encoder is further composed of four residual block groups, which employ a hierarchical design. The number of channels and resolution vary layer by layer with increasing network depth to achieve multi-scale feature extraction. Layer 1 consists of three residual blocks (ResBlock_1), each containing two 3×3 convolutional layers. Skip connections are used to fuse output and input features, enhancing feature expressiveness. Layer 2 consists of four residual blocks (ResBlock_2). The first residual block downsamples and expands the number of channels to 128 through a convolution with a stride of 2. Subsequent residual blocks maintain the same number of channels for feature extraction. Layers 3 and 4 consist of six and three residual blocks, respectively, further expanding the number of channels to 256. Dropout layers are introduced for regularization to prevent overfitting. During training, the image encoder uses ImageNet pre-trained weights to initialize parameters, which not only accelerates training but also improves the model's generalization ability.

[0052] The image decoder employs a symmetrical upsampling structure, progressively restoring the spatial resolution of the feature map through four decoding blocks. Each upsampling module includes feature convolution, LeakyReLU activation, Batch Normalization (BN), and bilinear interpolation upsampling. The feature convolution uses a 3×3 convolution to fix the features to 16 channels. The upsampling module utilizes bilinear interpolation to double the feature map size. After processing by the first three decoding blocks, feature fusion is performed, concatenating the upsampled features with the features of the corresponding layer in the encoder along the channel dimension, further enhancing the expressive power of the features. At the end of the image encoder, a 3×3 convolution maps the feature channel dimension to the number of semantic categories, and the Softmax function is used to calculate the category probability distribution of each pixel, thereby generating a semantic segmentation prediction map with the same spatial resolution as the input RGB image.

[0053] It should be noted that the semantic segmentation map of the image obtained by image branching is the result of the 2D image segmentation task. This result does not play a role in the point cloud branching process. The purpose of image segmentation is to enable the perception system to perform functions including 2D image segmentation and 3D point cloud segmentation.

[0054] The beneficial effects of this step are that, through a carefully designed encoder-decoder structure, the deep feature extraction capabilities of ResNet34 and the skip connection technology of U-Net are fully utilized to achieve efficient semantic segmentation of RGB images. Simultaneously, the use of pre-trained weights and the application of regularization techniques improve the model's training efficiency and generalization ability. Furthermore, the generated semantic segmentation prediction map provides accurate image semantic information for subsequent multi-source data fusion, significantly improving the overall system's perceptual accuracy and robustness.

[0055] Please see Figure 3 , Figure 4 S3, Based on the image feature map extracted by the image branch, the point cloud branch network is used to perform spatial depth feature extraction and cross-modal fusion processing on the 2D point cloud perspective view to obtain the 2D point cloud feature map;

[0056] Specifically, step S3 further includes: the model architecture of the point cloud branch network is based on the improved SalsaNext network design and adopts an asymmetric encoder-decoder framework.

[0057] The 2D point cloud perspective view is used as the input to the point cloud branch network to extract spatial depth features;

[0058] By using point cloud perspective projection and introducing a feature fusion attention module to fuse point cloud and image features, a 2D point cloud feature map is generated.

[0059] The point cloud encoder of the point cloud branch network includes 3 context encoding modules, 5 residual blocks, 4 bridging feature fusion modules and 1 holed spatial pyramid pooling module, and the point cloud decoder of the point cloud branch network includes 4 upsampling blocks.

[0060] The 2D point cloud perspective view is used as the input of the point cloud branch network to extract spatial depth features. Through point cloud perspective projection, and by introducing a feature fusion attention module to fuse point cloud and image features, a 2D point cloud feature map is generated. Specifically, the point cloud encoder of the point cloud branch network adopts a multi-scale feature extraction strategy. The context encoding module extracts features from the input 2D perspective view through three consecutive steps. Each context encoding module contains three convolutional operations: 1×1 convolution to adjust channel dimensions, 3×3 standard convolution and dilated convolution to extract local features, and fusion of local and global context information through skip connections.

[0061] The process sequentially passes through four extraction units consisting of one residual block, one average pooling layer, and one bridging feature fusion module, as well as one residual block. The residual block adopts a main branch and skip branch structure. The main branch contains three sequential convolutional layers: a standard 3×3 convolution, a 3×3 dilated convolution with a dilation rate of 2, and a 2×2 dilated convolution with a dilation rate of 2.

[0062] The three convolutional features in the channel are concatenated in dimension, and then a 1×1 convolution is used for feature extraction. The skip branch uses a 1×1 convolution to align the channel dimensions.

[0063] We introduce voided spatial pyramid pooling and use convolution with different voiding rates and global average pooling to capture multi-scale contextual information;

[0064] The point cloud decoder of the point cloud branch network uses four upsampling blocks to gradually restore the spatial resolution of the perspective view by extracting spatial depth features. Each upsampling block reduces the number of channels by four times based on pixel recombination operation, thereby increasing the resolution by two times.

[0065] The upsampled features are fused with the corresponding features of the encoder through skip connections, while maintaining the transmission of geometric details in the perspective view. Inside the upsampled block, a multi-branch structure similar to that of the incomplete block in the point cloud encoder is adopted.

[0066] After feature splicing and fusion, features are extracted sequentially through three serial convolutional layers: a 3×3 standard convolution, a 3×3 convolution with a dilation rate of 2, and a 2×2 convolution with a dilation rate of 2. The outputs of the three convolutional layers are spliced ​​together and then subjected to a 1×1 convolution to obtain the final output of the upsampled block.

[0067] A dropout layer is introduced, and a 1×1 convolution operation is used to map the feature channel dimension to the number of semantic categories. The class probability distribution of each pixel is calculated by the Softmax function to generate the scene segmentation result of the perspective view, and a 2D point cloud feature map is obtained.

[0068] In this embodiment, a cloud branch network is designed based on the characteristics of 2D perspective images. The cloud branch network is based on an improved SalsaNext network design and adopts an asymmetric encoder-decoder framework. This framework design enables the network to efficiently perform feature fusion and resolution restoration while extracting the spatial depth features of point cloud data.

[0069] First, a 2D point cloud perspective view is used as input to the cloud branch network. The point cloud encoder consists of three context encoding modules, five residual blocks, four bridging feature fusion modules, and one dilated spatial pyramid pooling (ASPP) module. The context encoding modules continuously extract features from the input 2D perspective view. Each context encoding module contains three convolutional operations: a 1×1 convolution to adjust the channel dimensions, and a 3×3 standard convolution and dilated convolution to extract local features. Skip connections are used to fuse local and global contextual information. This multi-scale feature extraction strategy effectively captures detailed information and contextual relationships in the point cloud data.

[0070] Next, the input data sequentially passes through four extraction units, each consisting of one residual block, one average pooling layer, and one bridging feature fusion module, as well as one residual block. The residual block employs a main branch and skip branch structure. The main branch contains three sequential convolutional layers: a standard 3×3 convolution, a 3×3 dilated convolution with a dilation rate of 2, and a 2×2 dilated convolution with a dilation rate of 2. The three convolutional features are concatenated along the channel dimension and then extracted using a 1×1 convolution. The skip branch uses a 1×1 convolution for channel dimension alignment. This residual block design not only enhances the expressive power of the features but also preserves the detailed information of the input data through skip connections.

[0071] To further capture multi-scale contextual information, the network introduces Spatial Pyramid Pooling with Diffusion (ASPP). ASPP uses convolutions with different dilation rates and global average pooling, which can effectively expand the receptive field and capture a wider range of contextual information. This is of great significance for target recognition and segmentation in complex scenes.

[0072] The point cloud encoder comprises five ResBlocks, with the number of feature channels increasing sequentially (64-128-256-256-256). After the first four ResBlocks, an average pooling layer (AvgPool) is applied for downsampling, followed by an FFAM module to enhance feature representation. Finally, the point cloud encoder introduces ASPP, which uses convolutions with different dilation rates (6, 12, 18) and global average pooling to capture multi-scale contextual information and expand the receptive field.

[0073] After feature extraction, the decoder part of the cloud branch network uses four upsampling blocks to progressively restore the spatial resolution of the perspective view. Combined with skip connections, it preserves detail information, ultimately outputting a 2D point cloud feature map after point cloud branch processing. Specifically, each upsampling block reduces the number of channels by a factor of four and increases the resolution by a factor of two based on pixel recombination. The upsampled features are fused with the features of the corresponding layer in the encoder through skip connections, preserving the geometric details in the perspective view. Inside the upsampling block, a multi-branch structure similar to that in the residual blocks of the point cloud encoder is used to further enhance the expressive power of the features.

[0074] After feature concatenation and fusion, features are extracted sequentially through three serial convolutional layers: a standard 3×3 convolution, a 3×3 convolution with a dilation rate of 2, and a 2×2 convolution with a dilation rate of 2. The outputs of the three convolutional layers are concatenated and then subjected to a 1×1 convolution to obtain the final output of the upsampled block. To prevent overfitting, a dropout layer is introduced after feature fusion and convolution operations. Finally, a 1×1 convolution operation is used to map the feature channel dimension to the number of semantic categories, and the class probability distribution of each pixel is calculated using the Softmax function to generate the perspective scene segmentation result, resulting in a 2D point cloud feature map.

[0075] The beneficial effects of this step are that, through the improved SalsaNext network architecture and a carefully designed encoder-decoder framework, efficient spatial depth feature extraction and cross-modal fusion with image features are achieved from 2D point cloud perspective views. The introduction of the feature fusion attention module allows point cloud features to fully utilize image semantic information, significantly improving the perception accuracy of complex unstructured scenes. Furthermore, the use of hollow spatial pyramid pooling and multi-scale feature extraction strategies further enhances the network's ability to capture multi-scale contextual information, enabling the system to maintain good robustness and adaptability under different environmental conditions. The resulting 2D point cloud feature map provides accurate environmental perception information for the autonomous operation of engineering machinery, offering strong support for subsequent decision-making and control.

[0076] Please see Figure 5 Preferably, the point cloud is fused with image features by point cloud perspective projection and a feature fusion attention module is introduced to generate a 2D point cloud feature map. Specifically, this includes: using a bridging feature fusion module to align and fuse the image semantic features output by each layer in the image encoder with the point cloud geometric features output by the corresponding layer in the point cloud encoder.

[0077] Let the l-th layer image feature map from the camera branch be... , Let l be the number of channels in the l-th layer of the camera branch. The height of the l-th layer image feature map of the camera branch. The width of the l-th layer image feature map of the camera branch;

[0078] Let the feature map of the l-th layer from the LiDAR branch be... , This represents the number of channels in the l-th layer of the LiDAR branch. The height of the l-th layer feature map in the LiDAR branch. The width of the feature map of the l-th layer of the LiDAR branch;

[0079] The bridging feature fusion module adopts a bridging dual-branch structure, using visual appearance information as auxiliary information for point cloud features, and obtaining a feature map after passing through the bridging feature fusion module. The calculation formula is as follows:

[0080] ,

[0081] ,

[0082] ,

[0083] in, This is an operation for splicing at the channel dimension. For convolution operations with a kernel of (1,1), This is a convolution operation with a kernel of (3,3). For the Sigmoid function, For adaptive average pooling layer, For the Softmax function, To perform element-wise multiplication. To extract the l-th layer image feature map from the camera branch and the feature map of layer l from the LiDAR branch The fused feature map is obtained by concatenating the data along the channel dimension and then performing a convolution operation. To fuse feature maps The feature map obtained after processing by the spatial attention mechanism.

[0084] In this embodiment, the bridging feature fusion module is introduced before the point cloud perspective projection and feature fusion attention module processing to enhance the scene understanding perception capability. The multimodal features of the two branch encoders are effectively fused using FFAM, and the fused features serve as input to subsequent point cloud branch encoding layers. FFAM, by combining spatial and channel attention mechanisms, adaptively adjusts the weights of different modal features based on the characteristics of the input image and point cloud data, thereby achieving complementary learning of point cloud data with the advantage of image information.

[0085] In this stage, the image encoder and point cloud encoder extract features from their respective modalities. The image encoder, based on the ResNet34 architecture, outputs multi-scale semantic features of the image, while the point cloud encoder, based on an improved SalsaNext network, outputs geometric features of the point cloud. These features undergo multiple convolutional and pooling operations in their respective encoders to extract rich semantic and geometric information, but modal differences exist between them, requiring effective alignment and fusion.

[0086] To achieve this goal, the bridging feature fusion module adopts a bridging dual-branch structure. Specifically, for the image semantic feature map output from layer l of the image encoder and the point cloud geometric feature map output from layer l of the point cloud encoder, the bridging feature fusion module performs feature alignment and fusion through the following steps, so that the feature map output by FFAM has more comprehensive information and optimizes the input feature information of the point cloud branch.

[0087] First, the semantic feature map of the image and the geometric feature map of the point cloud are concatenated along the channel dimension to obtain a comprehensive feature map. This operation, achieved through channel concatenation, integrates the feature information of two modalities into a single feature map, providing a foundation for subsequent fusion operations. The concatenated feature map is then subjected to feature extraction using a convolution operation with a kernel of (1,1). This convolution operation performs channel compression and feature transformation on the concatenated feature map, enabling effective interaction and fusion of features from different modalities along the same channel dimension. Subsequently, a convolution operation with a kernel of (3,3) is used to further extract features. This operation captures local feature information and enhances the spatial expressiveness of the feature map. To introduce nonlinear characteristics and further enhance the expressiveness of the features, the feature map is transformed element-wise using the Sigmoid function, mapping each element in the feature map to the range (0,1), which facilitates subsequent feature weighting operations. Simultaneously, an adaptive average pooling layer is used to perform global pooling on the feature map, resulting in a global feature vector. This vector captures global information in the feature map, providing global contextual information for subsequent feature fusion. Finally, the global feature vector is normalized using the Softmax function to obtain a normalized weight vector. This weight vector can weight features of different modalities based on global information, thereby achieving adaptive feature fusion. Through element-wise multiplication, the weighted feature map is fused with the original point cloud geometric feature map to obtain the final fused feature map. The feature fusion method of the FFAM module facilitates the fusion of complementary information from two inputs, utilizes image semantic information to emphasize important regions in the point cloud features, and promotes the effective utilization of features in subsequent processing within the point cloud branch.

[0088] This process not only achieves effective alignment and fusion of image semantic features and point cloud geometric features, but also enhances the flexibility and adaptability of feature fusion by introducing global contextual information and an adaptive weighting mechanism. This bridging feature fusion module design enables the system to effectively interact and complement features from different modalities, significantly improving the system's perception accuracy and robustness. In this way, the system can better understand and process various information in complex unstructured scenes, providing more reliable perception support for the autonomous operation of engineering machinery.

[0089] Please see Figure 6 S4, the 2D point cloud feature map is optimized using a refinement module to obtain the refined multi-source perception results of the unstructured scene.

[0090] Specifically, step S4 further includes: the refinement module adopts the SPVCNN architecture, and backprojects the 2D feature map of the last layer of the cloud branch network back to the original 3D space according to the perspective projection mapping index to obtain the initial point cloud features.

[0091] The point-by-point feature voxels in the Cartesian coordinate system are converted into initial 3D sparse voxels. The features processed by the 3D sparse convolution branch and the point-by-point MLP branch are fused to extract local features and capture effective spatial geometric information from the sparse point cloud. This results in a refined multi-source perception result of the unstructured scene, which can make up for the information loss caused by the dimensionality reduction of the point cloud data.

[0092] In a dual-branch network, the point cloud branch is trained solely on 2D perspective images as input. The resulting 2D segmentation is mapped to point clouds to obtain semantic labels for each point cloud in 3D space. However, the conversion from 3D point clouds to a 2D plane inevitably leads to dimensionality reduction and decreased feature resolution, resulting in the loss of spatial information in the point clouds. To address this issue, a thinning module is introduced, using a point-voxel hybrid representation to improve the efficiency and accuracy of feature extraction.

[0093] In this embodiment, after backprojecting the features obtained from point cloud branching to 3D space, a thinning module based on sparse convolution is used for boundary optimization. The 2D point cloud feature map is backprojected back to the original 3D point cloud space, and then the thinning module based on 3D sparse convolution refines the result after the point cloud branching backprojection to enhance the spatial representation information of the point cloud and optimize the initial segmentation result. Finally, the refined 3D point cloud perception result is output. The final multi-source perception result for the unstructured scene includes: network processing of data from the overlapping areas of the camera and LiDAR forward-looking regions, enabling the engineering machinery perception system to accurately perceive passable ground, scene obstacles, and work areas in the forward-looking region.

[0094] Specifically, step S4 is mainly responsible for optimizing the 2D point cloud feature map to generate a refined multi-source perception result for the unstructured scene. This process, by introducing a refinement module, significantly improves the spatial geometric information representation capability of the point cloud features, compensating for the information loss caused by dimensionality reduction of the point cloud data, thus providing more accurate and reliable perception support for the autonomous operation of engineering machinery. The refinement module adopts the SPVCNN (Sparse Voxel CNN) architecture. The core idea of ​​this architecture is to transform the processing of point cloud data from traditional dense representation to sparse representation, thereby fully utilizing the three-dimensional spatial structure information of the point cloud while maintaining computational efficiency. The workflow of the refinement module is as follows:

[0095] First, the 2D feature map output from the last layer of the cloud branch network (point cloud processing network) is back-projected back into the original 3D space according to the mapping index of perspective projection. This process is achieved through inverse perspective projection, which maps each pixel in the 2D feature map back to its corresponding 3D point cloud position, thereby restoring the 3D structural information of the point cloud. The back-projection operation not only preserves the semantic information in the 2D feature map, but also provides a foundation for subsequent 3D processing.

[0096] Next, the point-by-point features in Cartesian coordinates will be voxelized into initial 3D sparse voxels. Voxelization is a process of discretizing point cloud data into a 3D mesh, where each mesh cell (voxel) contains information about one or more points in the point cloud. Through voxelization, the point cloud data is converted into a sparse 3D tensor, which not only reduces data redundancy but also makes subsequent 3D convolution operations more efficient.

[0097] The refinement module further integrates features processed by the 3D sparse convolution branch and the point-by-point MLP (Multilayer Perceptron) branch to capture effective spatial geometric information from the sparse point cloud to the greatest extent possible. This effectively compensates for the information loss caused by dimensionality reduction of the point cloud data and alleviates the problem of blurred segmentation of target boundaries. The 3D sparse convolution branch extracts local features through sparse convolution operations, effectively capturing the spatial geometric information of the point cloud. Sparse convolution operations have high computational efficiency on sparse data and can fully utilize the three-dimensional structure of the point cloud. The point-by-point MLP branch processes the features of each point cloud point independently, further enhancing the expressive power of the features. By fusing the features of these two branches, the refinement module can capture effective spatial geometric information from the sparse point cloud, thereby generating refined multi-source perception results for the unstructured scene.

[0098] Based on this step, the introduction of the SPVCNN architecture significantly enhances the spatial geometric information representation capability of point cloud features. Backprojection and voxelization not only restore the 3D structure of the point cloud but also provide efficient data representation for subsequent 3D convolutional operations. The fusion of sparse convolutional branches and point-by-point MLP branches further enhances the feature representation capability, enabling the system to better capture local and global geometric information in the point cloud. The final refined multi-source perception results not only compensate for the information loss caused by dimensionality reduction of the point cloud data but also significantly improve the system's perception accuracy and robustness, providing more precise environmental perception information for the autonomous operation of engineering machinery.

[0099] Specifically, in this embodiment, to effectively utilize RGB visual information and the spatial depth information of point clouds, a multi-source perception method for unstructured scenes in engineering machinery proposes an unstructured scene perception system based on multi-source perception, which consists of three main modules: a perspective projection module, a dual-branch network based on feature fusion and attention bridging, and a point cloud refinement module. First, the perspective projection module projects the 3D point cloud onto the camera coordinate system to obtain a 2D perspective view of the point cloud. The projection process preserves the five-channel geometric features of the point cloud—depth (D), spatial coordinates (XYZ), and reflection intensity (I)—and establishes a correspondence between the point cloud data and the image space. Then, the image is processed by the camera branch to obtain image-level segmentation results, and the 2D perspective view of the point cloud is processed by the point cloud branch. The multimodal features of the two branch encoders are effectively fused through FFAM, allowing the point cloud branch to fully utilize the visual advantages of the image to assist in the complementary learning of the point cloud data. Finally, to address the issue of blurred target point cloud boundary segmentation in unstructured scenes and improve the segmentation accuracy of point cloud branches, a refinement module to enhance the spatial features of the point cloud is introduced. The overall network is designed with a coarse-to-fine network architecture to enhance the spatial representation information of the LiDAR point cloud and optimize the initial point cloud segmentation results. The refinement module first backprojects the 2D perspective feature map back into the original 3D point cloud space, then extracts local features through sparse convolution, and finally outputs the refined point cloud perception result. This concludes the introduction of a multi-source perception system suitable for unstructured scenarios in engineering machinery.

[0100] In summary, a multi-source perception method suitable for unstructured scenarios in engineering machinery constructs a coarse-to-fine segmentation network architecture for such scenarios. Through cascaded processing, it effectively fuses image semantics and point cloud geometric features and addresses the issue of blurred target boundary segmentation, thereby improving the perception accuracy of unstructured scenarios. Furthermore, FFAM is used to effectively fuse image semantic information and point cloud geometric information. The introduced refinement module designs the network into a coarse-to-fine structure, thus improving the problem of blurred target point cloud boundary segmentation in unstructured scenarios and enhancing the perception accuracy of point cloud branches.

[0101] In summary, the multi-source perception method applicable to unstructured scenarios for construction machinery aims to significantly improve the perception accuracy and robustness of construction machinery in complex environments by fusing image visual information and point cloud depth information. It provides an efficient and reliable solution for the intelligent operation of construction machinery, possessing significant technical value and broad application prospects. Through innovative multimodal data fusion technology and refined feature processing procedures, the system significantly enhances the perception capabilities and operational efficiency of construction machinery in complex environments, laying a solid foundation for the future development of intelligent construction machinery.

[0102] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A multi-source sensing method suitable for unstructured scenarios in engineering machinery, characterized in that, include: The raw LiDAR point cloud data and RGB image are acquired, and the raw LiDAR point cloud data is preprocessed using a perspective projection module to construct a 2D point cloud perspective view geometrically aligned with the RGB image. Specifically: Acquire raw LiDAR point cloud data RGB images captured by an RGB camera The original LiDAR point cloud data is filtered, retaining only the forward-looking point cloud data. The original LiDAR point cloud data is 3D point cloud data. Each point They are respectively defined by Cartesian coordinates It consists of the reflectivity value r. For the set of real numbers, The number of point clouds, The height of the image. The width of the image. For the i-th point cloud, Let x be the x-coordinate of the i-th point cloud. Let be the y-coordinate of the i-th point cloud. Let be the z-axis coordinate of the i-th point cloud. Let be the reflectance value of the i-th point cloud; Based on the preset extrinsic parameter matrix between multiple sensors, the filtered raw LiDAR point cloud data is projected onto a two-dimensional image coordinate system to obtain the projected image. , The number of channels in the projected point cloud, where the raw LiDAR point cloud data is extracted. Spatial coordinate information of each point The extracted features are concatenated in the fourth column to form a vector containing only 1s, expanded to four channels, and then the combined features are transposed to obtain the final vector. Each point cloud is ; Obtain the projected point cloud in the camera coordinate system The i-th projected point cloud The image coordinates are discretized into pixel coordinates, thus obtaining the corresponding pixel coordinates of the point cloud in the projected image. , , ,in, This is the intrinsic parameter matrix of the camera. This is the rotation matrix for transforming from the LiDAR coordinate system to the camera coordinate system; By aligning the front-view point cloud with the front-view RGB image through perspective projection, a 2D point cloud perspective view is obtained. As input data for the point cloud branch, the 2D perspective view Each pixel in It stores 5 channels of information, and the pixel positions to which the point cloud is projected store LiDAR features. , This represents the distance value for each point; for other pixel positions that are not projected, the value is stored as 0. Among them, the five-channel geometric features of the point cloud, including depth, spatial coordinates and reflection intensity, were preserved during projection, and the correspondence between point cloud data and image space was established. The image branching module is used to perform feature extraction and semantic segmentation on the RGB image to generate a semantic segmentation prediction map with the same spatial resolution as the RGB image. Based on the image feature map extracted from the image branch, a point cloud branch network is used to perform spatial depth feature extraction and cross-modal fusion processing on the 2D point cloud perspective view to obtain a 2D point cloud feature map, specifically: The model architecture of the point cloud branch network is based on the improved SalsaNext network design and adopts an asymmetric encoder-decoder framework. The 2D point cloud perspective view is used as the input to the point cloud branch network to extract spatial depth features; By using point cloud perspective projection and introducing a feature fusion attention module to fuse point cloud and image features, a 2D point cloud feature map is generated. The point cloud encoder of the point cloud branch network includes 3 context encoding modules, 5 residual blocks, 4 bridging feature fusion modules and 1 holed spatial pyramid pooling module, and the point cloud decoder of the point cloud branch network includes 4 upsampling blocks. The 2D point cloud feature map is optimized using a refinement module to obtain the refined multi-source perception results of the unstructured scene.

2. The multi-source sensing method applicable to unstructured scenarios of engineering machinery according to claim 1, characterized in that, The image branching module is used to perform feature extraction and semantic segmentation on the RGB image to generate a semantic segmentation prediction map with the same spatial resolution as the RGB image. Specifically: The image branch module adopts an encoder-decoder structure, taking the RGB image as the input of the image encoder of the image branch module to extract features and obtain image features. The backbone network of the image encoder is built based on ResNet34 and is used for 2D scene segmentation tasks. The image decoder of the image branch module is used for image semantic segmentation tasks. It performs semantic segmentation processing on image features using the image decoder and uses U-Net's skip connection technology to fuse multi-scale features from the encoder and decoder to generate a semantic segmentation prediction map with the same spatial resolution as the RGB image.

3. The multi-source sensing method applicable to unstructured scenarios of engineering machinery according to claim 2, characterized in that, The image encoder consists of one ordinary convolutional layer and four residual block groups. Based on a multi-scale feature extraction strategy, it improves the abstract semantics of features layer by layer. Specifically, the stride of the first convolutional layer of the original ResNet is modified to 1, the downsampling operation is removed, the full resolution of the input RGB image is preserved, and a 64-channel feature map is output through a convolutional layer with a 7×7 kernel. Combined with BN and ReLU activation functions, the feature map size is reduced to 1 / 2 of the original input after max pooling. The residual block group is designed hierarchically, with the number of channels and resolution varying layer by layer with network depth. The Layer 1 blockchain consists of 3 residual blocks ResBlock_1, each containing 2 3×3 convolutional layers. Output and input features are fused through skip connections, with the formula as follows: ,in, It is a nonlinear transformation consisting of two 3×3 convolutional layers. The feature map is the output after skip connections. Input feature map; The Layer 2 blockchain consists of four residual blocks, ResBlock_2. The first residual block downsamples and expands the number of channels to 128 through a convolution with a stride of 2. Subsequent residual blocks maintain the same number of channels and continue feature extraction. The Layer3 blockchain consists of 6 residual blocks ResBlock_3. The first residual block downsamples and expands the number of channels to 256 through a convolution with a stride of 2. Subsequent residual blocks continue feature extraction with the same number of channels and introduce a dropout layer for regularization. The Layer4 blockchain consists of three residual blocks, ResBlock_4. The first residual block downsamples and expands the number of channels to 256 through a convolution with a stride of 2. Subsequent residual blocks continue feature extraction while maintaining the same number of channels, and a dropout layer is introduced for regularization. During training, the image encoder uses ImageNet pre-trained weights to initialize its parameters.

4. The multi-source sensing method applicable to unstructured scenarios of engineering machinery according to claim 3, characterized in that, The image decoder adopts a symmetrical upsampling structure, which gradually restores the spatial resolution of the feature map through four decoding blocks. Each upsampling module includes feature convolution, LeakyReLU activation function, BN and bilinear interpolation upsampling. In this process, 3×3 convolution is used for feature convolution to fix the features to 16 channels. The upsampling module uses bilinear interpolation to double the size of the feature map. After processing by the first three decoding blocks, it follows feature fusion by concatenating the upsampled features with the features of the corresponding layer of the encoder along the channel dimension. The formula is as follows: , This is the output feature map after concatenation. This is the feature map after upsampling. For splicing operations; At the end of the image encoder, a 3×3 convolution is used to map the feature channel dimension to the number of semantic categories, and the Softmax function is used to calculate the category probability distribution of each pixel.

5. The multi-source sensing method applicable to unstructured scenarios of engineering machinery according to claim 1, characterized in that, The 2D point cloud perspective view is used as input to the point cloud branch network for spatial depth feature extraction. Point cloud perspective projection is then performed, and a feature fusion attention module is introduced to fuse point cloud and image features to generate a 2D point cloud feature map. Specifically: The point cloud encoder of the point cloud branch network adopts a multi-scale feature extraction strategy. The context encoding module extracts three consecutive features from the input 2D perspective view. Each context encoding module contains three convolution operations: 1×1 convolution to adjust the channel dimension, 3×3 standard convolution and dilated convolution to extract local features, and fusion of local and global context information through skip connections. The process sequentially passes through four extraction units consisting of one residual block, one average pooling layer, and one bridging feature fusion module, as well as one residual block. The residual block adopts a main branch and skip branch structure. The main branch contains three sequential convolutional layers: a standard 3×3 convolution, a 3×3 dilated convolution with a dilation rate of 2, and a 2×2 dilated convolution with a dilation rate of 2. The three convolutional features in the channel are concatenated in dimension, and then a 1×1 convolution is used for feature extraction. The skip branch uses a 1×1 convolution to align the channel dimensions. We introduce voided spatial pyramid pooling and use convolution with different voiding rates and global average pooling to capture multi-scale contextual information; The point cloud decoder of the point cloud branch network uses four upsampling blocks to gradually restore the spatial resolution of the perspective view by extracting spatial depth features. Each upsampling block reduces the number of channels by four times based on pixel recombination operation, thereby increasing the resolution by two times. The upsampled features are fused with the corresponding features of the encoder through skip connections, while maintaining the transmission of geometric details in the perspective view. Inside the upsampled block, a multi-branch structure similar to that in the residual block of the point cloud encoder is adopted. After feature splicing and fusion, features are extracted sequentially through three serial convolutional layers: a 3×3 standard convolution, a 3×3 convolution with a dilation rate of 2, and a 2×2 convolution with a dilation rate of 2. The outputs of the three convolutional layers are spliced ​​together and then subjected to a 1×1 convolution to obtain the final output of the upsampled block. A dropout layer is introduced, and a 1×1 convolution operation is used to map the feature channel dimension to the number of semantic categories. The class probability distribution of each pixel is calculated by the Softmax function to generate the scene segmentation result of the perspective view, and a 2D point cloud feature map is obtained.

6. The multi-source sensing method applicable to unstructured scenarios of engineering machinery according to claim 5, characterized in that, By using point cloud perspective projection and introducing a feature fusion attention module to fuse point cloud and image features to generate a 2D point cloud feature map, the specific steps include: using a bridging feature fusion module to align and fuse the image semantic features output from each layer of the image encoder with the point cloud geometric features output from the corresponding layer of the point cloud encoder. Let the l-th layer image feature map from the camera branch be... , Let l be the number of channels in the l-th layer of the camera branch. The height of the l-th layer image feature map of the camera branch. The width of the l-th layer image feature map of the camera branch; Let the feature map of the l-th layer from the LiDAR branch be... , This represents the number of channels in the l-th layer of the LiDAR branch. The height of the l-th layer feature map in the LiDAR branch. The width of the feature map of the l-th layer of the LiDAR branch; The bridging feature fusion module adopts a bridging dual-branch structure, using visual appearance information as auxiliary information for point cloud features, and obtaining a feature map after passing through the bridging feature fusion module. The calculation formula is as follows: ， ， ， in, This is an operation for splicing at the channel dimension. For convolution operations with a kernel of (1,1), This is a convolution operation with a kernel of (3,3). For the Sigmoid function, For adaptive average pooling layer, For the Softmax function, To perform element-wise multiplication. To extract the l-th layer image feature map from the camera branch and the feature map of layer l from the LiDAR branch The fused feature map is obtained by concatenating the data along the channel dimension and then performing a convolution operation. To fuse feature maps The feature map obtained after processing by the spatial attention mechanism.

7. The multi-source sensing method applicable to unstructured scenarios of engineering machinery according to claim 5, characterized in that, The 2D point cloud feature map is optimized using a refinement module to obtain a refined multi-source perception result for the unstructured scene, specifically: The refinement module adopts the SPVCNN architecture, which backprojects the 2D feature map of the last layer of the point cloud branch network back to the original 3D space according to the perspective projection mapping index to obtain the initial point cloud features. The point-by-point feature voxels in the Cartesian coordinate system are converted into initial 3D sparse voxels. The features processed by the 3D sparse convolution branch and the point-by-point MLP branch are fused to extract local features and capture effective spatial geometric information from the sparse point cloud. This results in a refined multi-source perception result of the unstructured scene, which can make up for the information loss caused by the dimensionality reduction of the point cloud data.

Images