Semantic precise segmentation method and system based on multi-mode and multi-stage deep fusion

Abstract

The invention provides a semantic precise segmentation method and system based on multi-mode multi-stage deep fusion, and relates to the technical field of intelligent driving automobile environment awareness, and the method comprises the steps: receiving point cloud data uploaded by a vehicle-mounted laser radar and first image data uploaded by a vehicle-mounted camera; projecting the point cloud data to the corresponding first image data through perspective projection to obtain second image data; fusing the second image data and the first image data to obtain fusion features, and fusing the fusion features with the second image data and the first image data to obtain a first semantic segmentation result and a second semantic segmentation result; and according to the first semantic segmentation result, the second semantic segmentation result and a difference value between the first semantic segmentation result and the second semantic segmentation result, calculating to obtain a final semantic segmentation result. According to the method, through multi-level and cross-modal feature interaction and fusion, the accuracy of semantic segmentation in a complex scene is effectively improved.

Description

Technical Field

[0001] This application relates to the field of environmental perception technology for intelligent driving vehicles, and in particular to a semantically accurate segmentation method and system based on multimodal and multi-stage deep fusion. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a core technology for autonomous vehicles to achieve environmental perception, autonomous navigation, and path planning, and its performance directly determines the vehicle's safety and reliability. Semantic segmentation technology, as a key support for scene understanding in SLAM systems, can provide semantic labels and fine-grained information such as target categories and spatial distribution in traffic scenes. By incorporating semantic constraints into vehicle position and attitude estimation, the odometry accuracy of SLAM systems can be effectively improved, thereby optimizing their environmental modeling accuracy and perception performance.

[0003] Based on the type of input information, current mainstream semantic segmentation methods can be mainly divided into three categories: image-based semantic segmentation, LiDAR point cloud-based semantic segmentation, and multimodal fusion-based semantic segmentation methods. In dynamic traffic scenarios involving real-time interaction among multiple traffic participants, the performance of single-modal semantic segmentation based on images or LiDAR point clouds is difficult to guarantee. This is because image-based methods are susceptible to illumination effects and lack depth information, making it difficult to effectively distinguish objects with similar colors and textures, such as blue walls and the sky. LiDAR point cloud-based methods have relatively sparse information and lack texture details, especially in distant areas, making it difficult to distinguish small objects such as tree trunks and utility poles. These errors in semantic labels will significantly affect odometry performance.

[0004] To overcome the limitations of single-modal perception, multimodal semantic segmentation methods that fuse images and LiDAR point clouds have become an important direction. The core logic is to use point cloud depth information to eliminate image geometric errors, while simultaneously filling sparse gaps in the point cloud with fine-grained image texture. However, existing multimodal schemes still have certain shortcomings. Some studies mainly extract semantic labels from point cloud information and combine semantic LiDAR features and visual features to estimate vehicle pose information, but this does not improve the accuracy of the semantic labels themselves. Other studies segment LiDAR point clouds and images separately, fusing the segmentation results to obtain higher-precision semantic labels, thereby optimizing the accuracy of vehicle pose estimation. However, these methods, which perform decision-level fusion at the output stage, fail to effectively address the modal differences between the geometric structure of point cloud data and the color and texture of images at the feature level. This leads to insufficient interaction and unbalanced contributions between the two modal features, limiting further improvements in semantic segmentation accuracy.

[0005] Therefore, how to effectively fuse point clouds and images at the deep feature level, overcome modal differences, and thus improve the accuracy and robustness of multimodal semantic segmentation results is a key problem that urgently needs to be solved in the fields of SLAM and even intelligent driving. Summary of the Invention

[0006] This application provides a semantic segmentation method and system based on multimodal and multi-stage deep fusion, aiming to solve the problems affecting semantic segmentation accuracy, such as dynamic interference caused by inconsistent scene data, insufficient feature robustness, and modal information imbalance. The technical solution is as follows: Firstly, a semantically accurate segmentation method based on multimodal, multi-stage deep fusion is provided, including: Receive point cloud data uploaded by the vehicle-mounted LiDAR and the first image data uploaded by the vehicle-mounted camera; The second image data is obtained by projecting the point cloud data onto the corresponding first image data through perspective projection, including: establishing a correspondence between the point cloud data and the first image data based on their spatiotemporal alignment; projecting the point cloud data onto the corresponding first image data to generate an initial projection image containing multiple test points; wherein, the test points are the pixel coordinates of the point cloud data in the first image data; for any test point in the initial projection image, constructing multiple feature blocks with different lengths, widths, and number of channels; extracting and fusing features from the multiple feature blocks to obtain the second image data. The second image data and the first image data are fused to obtain a fusion feature. The fusion feature is then fused with the second image data and the first image data respectively to obtain a first semantic segmentation result and a second semantic segmentation result. Based on the first semantic segmentation result, the second semantic segmentation result, and the difference between the two, the final semantic segmentation result is calculated; wherein, the calculation formula for the final semantic segmentation result is: , The first fusion result is obtained by extracting features from the first semantic segmentation result and the second semantic segmentation result respectively and then fusing the features. The second fusion result is obtained by extracting the neighborhood features corresponding to the features in the first semantic segmentation result and the second semantic segmentation result respectively and then fusing the neighborhood features. This represents the difference value between the features of the first semantic segmentation result and the second semantic segmentation result. (·) denotes the normalization function. This indicates an element-wise addition operation.

[0007] In one possible implementation, when constructing three feature blocks with lengths, widths, and channel numbers of 3×3×3, 5×5×3, and 9×9×3, features are extracted and fused from the 3×3×3 blocks and the extracted and fused feature blocks to obtain the second image data, including: The features in the 3×3×3 feature block are: , This represents a dilated convolution operation with a kernel size of 3×3 and a dilation rate of 1. Represents a 3×3×3 feature block; The features in the 5×5×3 feature block are: , This represents a dilated convolution operation with a kernel size of 3×3 and a dilation rate of 2. Represents a 5×5×3 feature block; The features in the 9×9×3 feature block are: , This represents a dilated convolution operation with a kernel size of 3×3 and a dilation rate of 4. Represents a 9×9×3 feature block; Integrating the features ,feature ,feature The obtained size is The second image data, , , These represent the width, height, and number of channels of the second image data, respectively.

[0008] In one possible implementation, fusing the second image data and the first image data yields a fused feature, including: Extract initial features from the second image data and the first image data at different levels; The initial features of the second image data at the same level are fused with the initial features of the first image data to obtain the first feature; Determine whether the level corresponding to the first feature is the last level; If so, then the first feature is taken as the fusion feature; If not, the first feature is input to the next level and concatenated with the initial features of the second image data and the initial features of the first image data in the next level to obtain a new first feature. The judgment step is then returned until the level corresponding to the new first feature is the last level. At this point, the new feature is output as a fused feature.

[0009] In one possible implementation, fusing the initial features of the second image data at the same level with the initial features of the first image data to obtain the first feature includes: The initial features of the second image data at the same level are concatenated with the initial features of the first image data to obtain the first intermediate feature; A multilayer perceptron is used to perform channel fusion and adjust the number of channels on the first intermediate feature to obtain the second intermediate feature; Calculate the weight of the second intermediate feature; The first feature is obtained by multiplying the second intermediate feature by the weight.

[0010] Secondly, a semantic precision segmentation system based on multimodal multi-stage deep fusion is provided. This system is used to execute the semantic precision segmentation method based on multimodal multi-stage deep fusion described above, including: The first processing module is used to receive point cloud data uploaded by the vehicle-mounted LiDAR; The second processing module is used to receive the first image data uploaded by the vehicle-mounted camera. An early feature fusion module, connected to both the first and second processing modules, is used to project the point cloud data onto corresponding first image data via perspective projection to obtain second image data. This includes: establishing a correspondence between the point cloud data and the first image data based on their spatiotemporal alignment; projecting the point cloud data onto its corresponding first image data to generate an initial projection image containing multiple test points; wherein each test point is a pixel coordinate corresponding to the point cloud data in the first image data; constructing multiple feature blocks with different lengths, widths, and channel numbers for any test point in the initial projection image; extracting and fusing features from the multiple feature blocks to obtain the second image data; and returning the second image data to the first processing module. An intermediate feature fusion module is connected to the first processing module and the second processing module respectively, and is used to fuse the second image data and the first image data to obtain fused features. The first processing module obtains a first semantic segmentation result based on the second image data and the fused features, and the second processing module obtains a second semantic segmentation result based on the first image data and the fused features. The late-stage feature fusion module, connected to both the first and second processing modules, is used to calculate the final semantic segmentation result based on the first semantic segmentation result, the second semantic segmentation result, and the difference between them; wherein the calculation formula for the final semantic segmentation result is: , The first fusion result is obtained by extracting features from the first semantic segmentation result and the second semantic segmentation result respectively and then fusing the features. The second fusion result is obtained by extracting the neighborhood features corresponding to the features in the first semantic segmentation result and the second semantic segmentation result respectively and then fusing the neighborhood features. This represents the difference value between the features of the first semantic segmentation result and the second semantic segmentation result. (·) denotes the normalization function. This represents an element-wise addition operation; Both the first processing module and the second processing module use an encoder-decoder architecture as their main framework, and the encoder and decoder portions of the main framework each contain four sub-processing modules; wherein: The encoder section comprises four sub-processing modules: a first encoding sub-module, a second encoding sub-module, a third encoding sub-module, and a fourth encoding sub-module, which are connected in series. The first encoding sub-module consists of a convolutional layer, a batch normalization layer, and a linear rectified function connected in series. The second, third, and fourth encoding sub-modules each consist of two residual network blocks and a max pooling layer connected in series. The decoder section comprises four sub-processing modules: a first decoding sub-module, a second decoding sub-module, a third decoding sub-module, and a fourth decoding sub-module, which are connected in series. The first decoding sub-module, the second decoding sub-module, and the third decoding sub-module are each composed of two residual network blocks and one transposed convolutional layer connected in series, while the fourth decoding sub-module is composed of one convolutional layer and one Softmax activation function connected in series.

[0011] Using the above technical solution, the semantic accurate segmentation method and system based on multimodal multi-stage deep fusion provided in this application embodiment firstly receives point cloud data by a first processing module and receives first image data by a second processing module, thereby realizing the input of multimodal data. Secondly, the early feature fusion module maps the point cloud data to the plane where the corresponding first image data is located through perspective projection, forming second image data that simultaneously contains visual texture and three-dimensional spatial distribution information. Then, the first image data and the second image data are input into the main framework composed of an encoder and decoder, and after multi-stage hierarchical processing, their respective preliminary semantic segmentation results are generated. At the same time, in the multi-level processing stage of the encoder and decoder, the intermediate feature fusion module performs cross-modal fusion of the initial features of the first image data and the initial features of the second image data at different levels to form fused features; then, each processing module further combines its own features with the cross-modal fused features, so that the semantic segmentation results in each processing module can fuse the information of another modality, realizing feature complementarity and enhancement. Finally, the late feature fusion module integrates the semantic segmentation results from the two processing modules and combines the differences between them to perform result fusion and optimization, ultimately outputting accurate and robust semantic segmentation results, which effectively improves the accuracy of semantic segmentation in complex scenarios. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. In the drawings: Figure 1 This is a structural diagram of a semantically accurate segmentation system based on multimodal and multi-stage deep fusion according to an embodiment of this application; Figure 2 This is a flowchart of the semantic accurate segmentation method based on multimodal and multi-stage deep fusion according to an embodiment of this application; Figure 3 This is an example diagram illustrating the generation of second image data in an embodiment of the method of this application; Figure 4 This is an example diagram of generating fusion features in the method embodiments of this application; Figure 5 This is an example diagram illustrating the generation of the final semantic segmentation result in the method embodiments of this application; Figure 6 This is a diagram showing the results of a segmentation test on a static object in an embodiment of the method of this application; Figure 7 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0013] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0014] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the term "comprising" and its variations should be interpreted as open-ended terms meaning "including but not limited to."

[0015] To achieve effective fusion of point clouds and images at the deep feature level, this application proposes a semantically accurate segmentation system based on multimodal, multi-stage deep fusion, such as... Figure 1 As shown, the system includes a first processing module, a second processing module, an early-phase fusion block (EPFB), a mid-phase fusion block (MPFB), and a late-phase fusion block (LPFB).

[0016] Both the first and second processing modules use an encoder-decoder architecture as their main framework, with each component containing four sub-processing modules. For clarity, the first processing module will be used as an example below: The encoder section of the first processing module comprises four sub-processing modules, namely the first encoding sub-module, the second encoding sub-module, the third encoding sub-module, and the fourth encoding sub-module, which are sequentially connected in series. Figure 1 The first encoding submodule is represented by four boxes from left to right. The first encoding submodule consists of one convolutional layer, one batch normalization layer, and one linear rectified function connected in series. The second, third, and fourth encoding submodules each consist of two residual network blocks connected in series and one max-pooling layer. The decoder section of the first processing module has four sub-processing modules, namely the first decoding submodule, the second decoding submodule, the third decoding submodule, and the fourth decoding submodule, which are connected in series. Figure 1Excluding the four sub-processing modules of the encoder section, the four sub-modules are represented by four boxes from left to right. The first, second, and third decoding sub-modules are each composed of two residual network blocks and one transposed convolutional layer connected in series, while the fourth decoding sub-module is composed of one convolutional layer and one Softmax activation function connected in series.

[0017] It should be noted that since the second processing module has the same structure as the first processing module, its structure will not be described in detail here. The difference is that the first processing module is mainly used to receive and process point cloud data uploaded by the vehicle-mounted LiDAR, while the second processing module is mainly used to receive and process the first image data uploaded by the vehicle-mounted camera.

[0018] The early feature fusion module is connected to the first processing module and the second processing module respectively. It is used to project point cloud data onto the corresponding first image data through perspective projection, thereby fusing visual texture into point cloud data, making up for the lack of information in a single modality, and then returning the fused image obtained at this time as the second image data to the first processing module.

[0019] The intermediate feature fusion module is also connected to the first processing module and the second processing module respectively. It is used to fuse the second image data and the first image data to obtain fused features, so that the first processing module can obtain the first semantic segmentation result based on the second image data and the fused features, and the second processing module can obtain the second semantic segmentation result based on the first image data and the fused features. This realizes efficient cross-dimensional fusion between point cloud data and corresponding image data, which facilitates the subsequent mining of the association information of multimodal features.

[0020] The late feature fusion module is also connected to the first processing module and the second processing module respectively. It is used to calculate the final semantic segmentation result based on the first semantic segmentation result, the second semantic segmentation result and the difference between the two. That is, the late feature fusion module integrates multiple semantic segmentation results to obtain the final semantic segmentation result, which effectively improves the semantic segmentation accuracy in scenarios where single modality is prone to failure, such as sparse geometric structure and poor lighting conditions, and enhances the adaptability of the system of this application to complex scenarios.

[0021] The system described in this application is primarily used to improve the accuracy of semantic segmentation in complex scenes through multi-level, cross-modal feature interaction and fusion. However, before performing semantic segmentation using this system, it needs to be trained first. The specific process is as follows: training is performed on the KITTI dataset, which contains different complex urban scenes and provides 11 annotated data sequences {00-10}. In training and testing, sequences {00, 01, 02, 04, 05, 06, 07, 09, 10} are used for training, while sequence 08, which contains more moving objects than other sequences, is used for testing. During training, the Adam (Adaptive Moment Estimation) optimizer is used with an initial learning rate of 0.001, and training is conducted for 150 epochs. The training loss function is... The calculation formula is as follows: ; ; ; ; in, , , These represent the point cloud data, the first image data, and the actual semantic labels after the point cloud data and the first image data are fused, respectively. , , These represent the semantic labels predicted from the point cloud data, the first image data, and the final semantic segmentation result after the point cloud data and the first image data are fused. Indicates the result of semantic segmentation calculation The loss function, The overall category, including the first semantic segmentation result, the second semantic segmentation result, and the final semantic segmentation result, can be considered as follows: When, function The calculation is based on the loss of the first semantic segmentation result; when When, function The calculation is based on the loss of the second semantic segmentation result; when When, function The calculation is based on the loss of the final semantic segmentation result, i.e. This represents the loss of the first semantic segmentation result. The loss represents the result of the second semantic segmentation. This represents the loss of the final semantic segmentation result.

[0022] This embodiment uses a loss function The minimum output value is taken as the training target. The parameter configuration of each module in the system is continuously optimized, so that the trained system is output after the training target is achieved.

[0023] To further illustrate how the system achieves high-precision semantic segmentation results by fusing point cloud data and image data, based on the trained system, this application provides a semantic precision segmentation method based on multimodal, multi-stage deep fusion, such as... Figure 2 As shown, the method mainly includes steps S10 to S40.

[0024] Step S10: Receive point cloud data uploaded by the vehicle-mounted LiDAR and the first image data uploaded by the vehicle-mounted camera.

[0025] Point cloud data is a set of three-dimensional spatial information acquired by an onboard LiDAR through emitting laser beams and receiving their reflected signals. It includes the three-dimensional coordinates of each point in the environment and their reflection intensity. The first image data is a two-dimensional pixel array acquired by an onboard camera through optical imaging, containing information such as the texture and color of the environment. The point cloud data and the first image data have a strict spatiotemporal alignment relationship, being synchronously acquired by the same vehicle at the same time from a matching perspective, thus ensuring the consistency and fusion of three-dimensional geometric information and two-dimensional visual information in spatiotemporal space. After the onboard LiDAR acquires the point cloud data, it is uploaded to the first processing module, while the onboard camera generates the first image data and uploads it to the second processing module. That is, the system of this application acquires both the point cloud data and the first image data.

[0026] Step S20: Project the point cloud data onto the corresponding first image data using perspective projection to obtain the second image data.

[0027] Because point cloud data exhibits typical sparsity, while the first image data is pixel-dense, there are significant differences between the two in terms of data density and representation dimension. Directly fusing the two types of data can easily lead to the loss of detailed information such as texture, color, and pixel-level semantic relationships in the image, resulting in a decrease in the accuracy of semantic perception. To address this, this application includes an early feature fusion module, which effectively fuses the point cloud data and the first image data. The specific fusion process is shown in steps S21 to S25.

[0028] Step S21: Based on the spatiotemporal alignment relationship between point cloud data and first image data, establish the correspondence between the two; Step S22: Project the point cloud data onto the corresponding first image data to generate an initial projection image containing multiple test points; wherein, the test points are the pixel coordinates of the point cloud data in the first image data. Step S23: For any point to be tested in the initial projection image, construct multiple feature blocks with different lengths, widths, and number of channels; Step S24: Extract and fuse features from multiple feature blocks to obtain the second image data.

[0029] Specifically, point cloud data and first image data acquired simultaneously from the same viewpoint are correlated, and the point cloud data is projected onto the corresponding first image data using perspective projection. The formula for calculating perspective projection is: ; in, This represents the intrinsic parameter matrix of the vehicle-mounted camera. This represents the extrinsic calibration matrix of the vehicle-mounted camera. , , This represents the coordinates of any point in the point cloud data. , Indicates the location ( , , The pixel coordinates of the point to be measured.

[0030] like Figure 3 As shown, in this embodiment, for any point to be measured in the initial projected image, three feature blocks with lengths of 3×3×3, widths of 5×5×3, and number of channels of 9×9×3 are constructed respectively, denoted as . , , and feature blocks , , The dimensions can be expressed as: , , .

[0031] Then, the feature blocks are processed through dilated convolution operations. , , Perform feature extraction, such as: Extract feature blocks The features in the middle are obtained , Represents the convolution kernel Size 3×3, void ratio A dilated convolution operation with a value of 1; Extract feature blocks The features in the middle are obtained , Represents the convolution kernel Size 3×3, void ratio A dilated convolution operation of 2; Extract feature blocks The features in the middle are obtained , Represents the convolution kernel Size 3×3, void ratio A dilated convolution operation of 4.

[0032] Then, the features corresponding to each extracted feature block are fused to obtain the second image data. Specifically, the features are... , , This additional information is fused into the point cloud data to obtain a size of... The second image data, , , These represent the width, height, and number of channels of the second image data, respectively. It should be noted that the number of channels in the second image data is 7 because the point cloud data contains points (…). , , The four-dimensional feature information, including reflectance r, is combined with the three extracted features. , , There are a total of 7 channels.

[0033] Therefore, it can be seen that the early feature fusion module can effectively establish the association between point cloud data and the corresponding first image data. After mapping the point cloud data to the first image data, multiple features are extracted, and the extracted features are then fused back into the point cloud data to obtain the second image data. This fusion process supplements the visual information dimensions of the point cloud data and reduces the degree of information loss during the fusion process between the point cloud data and the first image data.

[0034] Step S30: Fuse the second image data and the first image data to obtain fusion features, and fuse the fusion features with the second image data and the first image data respectively to obtain the first semantic segmentation result and the second semantic segmentation result.

[0035] First, the intermediate feature fusion module fuses the second image data and the first image data to obtain fused features. The specific fusion process is shown in steps S31 to S335: Step S31: Extract initial features from the second image data and the first image data at different levels; Step S32: The initial features of the second image data at the same level are fused with the initial features of the first image data to obtain the first feature; Step S33: Determine whether the level corresponding to the first feature is the last level; Step S34: If yes, then the first feature is used as the fusion feature; If not, in step S35, the first feature is input into the next level and concatenated with the initial features of the second image data and the initial features of the first image data in the next level to obtain a new first feature, and then the process returns to step S33.

[0036] like Figure 1 For example, the second image data and the first image data are respectively in size , , , These represent different levels, and the features extracted from the second and first image data at these levels are called initial features, i.e. , , , , , , , Both represent initial features, while and , and , and , and This represents a pair of initial features at the same level.

[0037] like Figure 4 As shown, for a pair of initial features at the same level, the pair of initial features are first concatenated to obtain the first intermediate feature. , Then, a multilayer perceptron is used to process the first intermediate features. By performing channel fusion and adjusting the number of channels, a second intermediate feature is obtained. Then, for the second intermediate feature Convolution kernels are performed sequentially. Size 3×3, void ratio Dilated convolution operation and convolution kernel with a value of 1 Size 3×3, void ratio A dilated convolution operation of 2 is performed to obtain the second intermediate feature. weight Finally, the second intermediate feature... and weight Element-by-element multiplication yields the first characteristic.

[0038] For the obtained first feature, determine whether the level corresponding to the first feature is the last level. If so, use the first feature as the fusion feature, such as... Figure 4 In the example, The corresponding level is the last level, so it will be... As a fusion feature, and the fusion feature The process returns to the first processing module and the second processing module respectively. Otherwise, if the level corresponding to the first feature is not the last level, the first feature is input into the next level and concatenated with a pair of initial features in the next level to obtain a new first feature, such as... Figure 4 In the example, If none of the corresponding levels are the last level, they will all be used as inputs to their respective next level and concatenated with a pair of initial features in the next level to obtain a new first feature. This process continues until the level corresponding to the new first feature is the last level, at which point the new feature is output as a fused feature.

[0039] Therefore, we can see that the fusion feature It is obtained by continuously fusing the first and second image data through multiple levels, ensuring the quality of the resulting fusion features. It encompasses more correlation information from multimodal data such as point cloud data and image data. In this embodiment, based on the obtained fusion features... The intermediate feature fusion module then returns the features to the first processing module and the second processing module, respectively.

[0040] The fourth decoding submodule in the first processing module uses the output and fusion features of the third decoding submodule. As a common input, a fused second image data of size w×h×6 is obtained. This fused second image data is then processed to obtain the first semantic segmentation result. Specifically, a convolution operation is performed on the w×h×6 fused second image data, and then the Softmax activation function is applied to it to obtain the first semantic segmentation result. The first semantic segmentation result is then used... express.

[0041] Similarly, the fourth decoding submodule in the second processing module uses the output and fused features of the third decoding submodule. As a common input, a fused first image data of size w×h×6 is obtained. This fused first image data is then processed to obtain a second semantic segmentation result. Specifically, a convolution operation is performed on the w×h×6 fused first image data, and then the Softmax activation function is applied to it to obtain the second semantic segmentation result. The second semantic segmentation result is then used... express.

[0042] Step S40: Calculate the final semantic segmentation result based on the first semantic segmentation result, the second semantic segmentation result, and the difference between the two.

[0043] The late-stage feature fusion module obtains the first semantic segmentation result from the first processing module. The second semantic segmentation result is obtained from the second processing module. Then combine the results of the first semantic segmentation Second semantic segmentation results The difference between the two values ​​yields the final semantic segmentation result. The specific calculation process is shown in steps S41 to S43: Step S41: Extract features from the first semantic segmentation result and the second semantic segmentation result respectively to obtain the first semantic feature and the second semantic feature; Step S42: Fuse the first semantic feature and the second semantic feature to obtain the first fusion result; Step S43: Fuse the neighborhood features of the first semantic feature and the neighborhood features of the second semantic feature to obtain the second fusion result; Step S44: Calculate the difference between the first semantic feature and the second semantic feature; Step S45: Calculate the final semantic segmentation result based on the first fusion result, the second fusion result, and the gap value.

[0044] like Figure 5 As shown, assuming the first semantic segmentation result Each single feature in the first semantic feature is a first semantic feature. The neighborhood features of each point are Second semantic segmentation result Each single-point feature in the second semantic feature is a second semantic feature. The neighborhood features of each point are .

[0045] If directly addressing the first semantic feature Second semantic features The fusion process yielded the following first fusion result: ;in, represents a dilated convolution with a kernel size of 1×1 and a dilation rate of 1, and © represents splicing or concatenation.

[0046] If the first semantic feature neighborhood features Second semantic features neighborhood features The fusion was performed, and the second fusion result was obtained as follows: ;in, This represents a dilated convolution with a kernel size of 3×3 and a dilation rate of 2.

[0047] Then calculate the first semantic segmentation result. Second semantic segmentation results The difference value: .

[0048] Finally, based on the first fusion result Second fusion result and difference value The final semantic segmentation result is calculated using the following formula: ;in, (·) denotes the normalization function. This indicates an element-wise addition operation.

[0049] Therefore, the final semantic segmentation result integrates the first semantic segmentation result. Second semantic segmentation results And the difference between the two. This ensures the final semantic segmentation result obtained from the calculation. This improves the accuracy of the final semantic segmentation result. The accuracy.

[0050] In summary, the implementation principle of the semantic accurate segmentation method based on multimodal multi-stage deep fusion in this application embodiment is as follows: Multimodal inputs such as point cloud data and first image data are received; the point cloud data is mapped onto the plane containing the corresponding first image data through perspective projection, forming second image data that simultaneously contains visual texture and three-dimensional spatial distribution information. Subsequently, the first and second image data are input into the main framework composed of an encoder and decoder, and after multi-stage hierarchical processing, their respective preliminary semantic segmentation results are generated. Simultaneously, in the multi-level processing stage of the encoder and decoder, the initial features of the first and second image data are fused across different levels to form fused features; then, each processing module further combines its own features with the cross-modal fused features, so that the semantic segmentation result of each processing module can fuse information from another modality, achieving feature complementarity and enhancement. Finally, by integrating the semantic segmentation results corresponding to the two processing modules and combining the differences between them, the results are fused and optimized, ultimately outputting accurate and robust semantic segmentation results. As can be seen, the method of this application effectively improves the accuracy of semantic segmentation in complex scenarios through multi-level, cross-modal feature interaction and fusion.

[0051] To demonstrate the effectiveness of the semantic precision segmentation method based on multimodal and multi-stage deep fusion proposed in this application, this embodiment also conducted comparative experiments using the KITTI dataset, and verified the performance of the proposed method through quantitative indicators and visualization results.

[0052] Specifically, the test results for semantic segmentation of static objects in three typical scenarios on KITTI 08 are compared, and the comparison results are as follows: Figure 6 As shown, (a), (b), and (c) represent the true semantic labels of three typical scene images on KITTI 08, respectively; (d), (e), and (f) form a control group, representing the semantic labels of static objects in the three typical scene images, respectively; (g), (h), and (i) form another control group, representing the semantic labels of static objects in the three typical scene images, respectively; and (j), (k), and (l) represent the semantic segmentation results of static objects in the three typical scene images processed using the method of this application, respectively. The segmentation results are displayed in the form of perspective images, where purple represents roads, green represents tree trunks, orange represents utility poles, black represents traffic signs, and blue represents buildings.

[0053] By comparing (d) and (j), it can be seen that the method of this application can effectively achieve accurate segmentation and differentiation of objects with similar geometric structures. Specifically, in Figure (a), the building and the container have highly similar geometric shapes. Such similarity can interfere with the feature representation of point cloud data, leading to a decrease in model performance. To address the problem of difficulty in accurate segmentation due to similar object geometric structures, the method of this application uses a mid-term feature fusion module to fuse the interrelated features in point cloud data and image data using a multilayer perceptron (MLP), and uses convolution operations to calculate the weights of the fused features, thereby preserving reliable cross-modal features and improving segmentation accuracy in single-modal failure scenarios.

[0054] By comparing (e), (h), and (k), it can be seen that, in semantic segmentation, the recognition of small-scale or distant objects is often difficult due to insufficient feature capture. The method in this application effectively improves the segmentation accuracy through the perspective projection mechanism of the early feature fusion module. For example, for the segmentation of small-scale distant objects such as distant poles and trees in the rectangular frame of Figure (b), the method in this application enhances the feature extraction capability of fine targets by fusing visual information from multiple ranges around the projection position of point cloud data in the image data, thereby significantly improving the segmentation performance of small-scale and distant objects.

[0055] By comparing (i) and (l), it can be seen that the method of this application addresses the problem of decreased segmentation performance caused by high correlation but low confidence levels in predictions of point cloud data and image data at the same location. By employing a late-stage feature fusion module that draws on semantic consistency, the method comprehensively utilizes surrounding information from both point cloud data and image data to generate segmentation results, thereby improving segmentation accuracy. For example, in the segmentation of the tree trunk region in Figure (c), this method achieves accurate segmentation of the target by effectively utilizing surrounding information.

[0056] Based on the above test results, the results obtained by using the method of this application were quantified by crossover ratio (IoU), mean crossover ratio (mIoU), and computation time. The results are shown in Table 1.

[0057] Table 1

[0058] Table 1 shows the performance of the proposed method on the KITTI dataset validation set. The average intersection-union ratio (IUU) reaches 75.17%, with a processing time of only 28 milliseconds. In terms of specific categories, the segmentation accuracy for buildings and roads is 92.43% and 96.94%, respectively, both exceeding 90%. For slender, rod-shaped targets and trees with complex shapes, segmentation accuracy is achieved at 67.75% and 73.86%, respectively.

[0059] Therefore, by using the method of this application to fuse point cloud data and image data in multiple stages, static objects in the scene can be effectively identified and dynamic targets can be filtered out, which alleviates the problem of multimodal data inconsistency and promotes the application of lidar-visual semantic segmentation technology in odometry tasks.

[0060] It should be noted that the sequence numbers of the steps in the above embodiments do not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In practical applications, all the above possible implementation methods can be arbitrarily combined in a combined manner to form possible embodiments of this application, which will not be described in detail here.

[0061] Based on the same inventive concept, embodiments of this application also provide an electronic device, including a processor and a memory, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute a semantically accurate segmentation method based on multimodal multi-stage deep fusion of any of the above embodiments.

[0062] In an exemplary embodiment, an electronic device is provided, such as Figure 7 As shown, Figure 7 The illustrated electronic device 700 includes a processor 701 and a memory 703. The processor 701 and the memory 703 are connected, for example, via a bus 702. Optionally, the electronic device 700 may also include a transceiver 704. It should be noted that in practical applications, the transceiver 704 is not limited to one type, and the structure of this electronic device 700 does not constitute a limitation on the embodiments of this application.

[0063] Processor 701 may be a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 701 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0064] Bus 702 may include a pathway for transmitting information between the aforementioned components. Bus 702 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 702 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0065] The memory 703 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0066] The memory 703 stores computer program code that executes the scheme of this application, and its execution is controlled by the processor 701. The processor 701 executes the computer program code stored in the memory 703 to implement the content shown in the foregoing method embodiments.

[0067] Among them, electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0068] Based on the same inventive concept, this application also provides a storage medium storing a computer program, wherein the computer program is configured to execute a semantically accurate segmentation method based on multimodal multi-stage deep fusion of any of the above embodiments at runtime.

[0069] Those skilled in the art will clearly understand that the specific working process of the systems, devices, and modules described above can be referred to the corresponding process in the foregoing method embodiments. For the sake of brevity, it will not be repeated here.

[0070] Those skilled in the art will understand that the technical solution of this application, or all or part of it, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several program instructions to cause an electronic device (e.g., a personal computer, server, or network device) to execute all or part of the steps of the methods described in the embodiments of this application when running the program instructions. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0071] Alternatively, all or part of the steps of the foregoing method embodiments can be implemented by hardware (such as electronic devices like personal computers, servers, or network devices) associated with program instructions. The program instructions can be stored in a computer-readable storage medium. When the program instructions are executed by the processor of the electronic device, the electronic device executes all or part of the steps of the methods described in the embodiments of this application.

[0072] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that within the spirit and principles of this application, modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the corresponding technical solutions to leave the protection scope of this application.

Claims

1. A multi-modal multi-stage deep fusion based semantic precise segmentation method, characterized in that, The method comprises the following steps: receiving point cloud data uploaded by a vehicle-mounted laser radar and first image data uploaded by a vehicle-mounted camera; projecting the point cloud data into corresponding first image data by perspective projection to obtain second image data, comprising: establishing a correspondence between the point cloud data and the first image data based on a spatio-temporal alignment relationship therebetween; projecting the point cloud data into the first image data corresponding thereto to generate an initial projection image containing a plurality of test points; wherein the test points are pixel coordinates corresponding to the point cloud data in the first image data; for any test point in the initial projection image, a plurality of feature blocks with different lengths, widths and channel numbers are constructed respectively; features in the plurality of feature blocks are extracted and fused to obtain the second image data; fusing the second image data and the first image data to obtain fused features, and fusing the fused features with the second image data and the first image data respectively to obtain first and second semantic segmentation results; According to the first semantic segmentation result, the second semantic segmentation result and the gap value between the two, a final semantic segmentation result is calculated, wherein the calculation formula of the final semantic segmentation result is: , denotes a first fusion result, which is obtained by respectively extracting features in the first semantic segmentation result and the second semantic segmentation result and then performing feature fusion; denotes a second fusion result, which is obtained by respectively extracting neighborhood features corresponding to the features in the first semantic segmentation result and the second semantic segmentation result and then performing neighborhood feature fusion; denotes a gap value between the features in the first semantic segmentation result and the second semantic segmentation result, (·) denotes a normalization function, denotes an element-wise addition operation.

2. The method of claim 1, wherein, when three feature blocks with lengths, widths and channel numbers of 3x3x3, 5x5x3 and 9x9x3 are constructed, the extracting and fusing of the features in the plurality of feature blocks to obtain the second image data comprises: The features in the 3x3x3 feature block are: , denotes a dilated convolution operation with a kernel size of 3x3 and a dilation rate of 1, denotes a 3x3x3 feature block; The features in the 5x5x3 feature block are: , denotes a dilated convolution operation with a kernel size of 3x3 and a dilation rate of 2, denotes a 5x5x3 feature block; The features in the 9x9x3 feature block are: , denotes a dilated convolution operation with a kernel size of 3x3 and a dilation rate of 4, denotes a 9x9x3 feature block; Integrating the features ,feature ,feature The obtained size is The second image data, , , These represent the width, height, and number of channels of the second image data, respectively.

3. The method of claim 1, wherein, fusing the second image data and the first image data to obtain fused features, comprising: extracting initial features of the second image data and the first image data at different levels; fusing the initial features of the second image data at the same level with the initial features of the first image data to obtain first features; judging whether the level corresponding to the first features is the last level; if yes, the first features are taken as fused features; if no, the first features are input into the next level and spliced with the initial features of the second image data and the first image data at the next level to obtain new first features, and the judgment step is returned until the level corresponding to the new first features is the last level, and the new features are output as fused features.

4. The method of claim 3, wherein, the fusing of the initial features of the second image data at the same level with the initial features of the first image data to obtain first features comprises: splicing the initial features of the second image data at the same level with the initial features of the first image data to obtain first intermediate features; performing channel fusion and adjusting the number of channels on the first intermediate features by using a multi-layer perception to obtain second intermediate features; calculating the weight of the second intermediate features; multiplying the second intermediate features by the weight to obtain first features.

5. A multi-modal multi-stage deep fusion based semantic precise segmentation system for performing the method of any one of claims 1-4, characterized in that, The method comprises the following steps: a first processing module for receiving point cloud data uploaded by a vehicle-mounted laser radar; a second processing module for receiving first image data uploaded by a vehicle-mounted camera; An early feature fusion module is connected with the first processing module and the second processing module respectively, and is configured to project the point cloud data into the corresponding first image data by perspective projection to obtain second image data, including: establishing a corresponding relationship between the point cloud data and the first image data based on a space-time alignment relationship therebetween; projecting the point cloud data into the first image data corresponding thereto to generate an initial projection image containing a plurality of to-be-tested points; wherein the to-be-tested points are pixel coordinates corresponding to the point cloud data in the first image data; constructing a plurality of feature blocks different in length, width and channel number for any to-be-tested point in the initial projection image; extracting and fusing features in the plurality of feature blocks to obtain the second image data; and returning the second image data to the first processing module; A mid-term feature fusion module is connected with the first processing module and the second processing module respectively, and is configured to fuse the second image data and the first image data to obtain fused features, the first processing module obtains a first semantic segmentation result based on the second image data and the fused features, and the second processing module obtains a second semantic segmentation result based on the first image data and the fused features; The late feature fusion module is connected with the first processing module and the second processing module respectively, and is used for calculating a final semantic segmentation result according to the first semantic segmentation result, the second semantic segmentation result and a gap value between the first semantic segmentation result and the second semantic segmentation result; wherein a calculation formula of the final semantic segmentation result is: , The first fusion result is obtained by respectively extracting features in the first semantic segmentation result and the second semantic segmentation result and performing feature fusion. The second fusion result is obtained by respectively extracting corresponding neighborhood features of the features in the first semantic segmentation result and the second semantic segmentation result and performing neighborhood feature fusion. The gap value between the features of the first semantic segmentation result and the second semantic segmentation result is represented by The normalization function is represented by The element-wise addition operation is represented by The first processing module and the second processing module both take an encoder-decoder as a main framework, and the encoder part and the decoder part of the main framework both contain four sub-processing modules; wherein: The four sub-processing modules of the encoder part are a first encoding submodule, a second encoding submodule, a third encoding submodule and a fourth encoding submodule connected in sequence; wherein the first encoding submodule is composed of one convolutional layer, one batch normalization layer and one linear rectifier function connected in sequence, and the second encoding submodule, the third encoding submodule and the fourth encoding submodule are all composed of two residual network blocks and one max-pooling layer connected in sequence; The four sub-processing modules of the decoder part are a first decoding submodule, a second decoding submodule, a third decoding submodule and a fourth decoding submodule connected in sequence; wherein the first decoding submodule, the second decoding submodule and the third decoding submodule are all composed of two residual network blocks and one transpose convolutional layer connected in sequence, and the fourth decoding submodule is composed of one convolutional layer and one Softmax activation function connected in sequence.

Images