Multi-modal panoramic segmentation method based on traffic scene

Abstract

The invention discloses a multi-modal panoramic segmentation method based on a traffic scene, and relates to the technical field of image processing, and the method comprises the steps: obtaining original multi-modal perception data, executing modal synchronization enhancement processing, and obtaining multi-modal data with enhanced alignment; inputting the multi-modal data subjected to enhanced alignment into a three-dimensional coding network and a two-dimensional coding network for feature extraction to obtain voxel-level three-dimensional geometric features and image-level two-dimensional visual features, and executing cross-modal alignment and fusion to obtain multi-modal fusion features; and inputting the multi-modal fusion features into a dual-branch decoder based on a Transform architecture for processing to obtain a semantic category prediction result and an instance mask prediction result, and performing fusion processing to obtain a panoramic segmentation result image. According to the method, the technical problems that space and semantics are not aligned and the panoramic segmentation precision is limited due to independent enhancement of multi-modal data in the prior art are solved, and the technical effect of improving the precision and robustness of traffic scene panoramic segmentation is achieved.

Description

A Multimodal Panoramic Segmentation Method Based on Traffic Scenes Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a multimodal panoramic segmentation method based on traffic scenes. Background Technology

[0002] In multimodal traffic scene perception, LiDAR point cloud data and visible light image data are often fused to improve the completeness of environmental understanding. However, in actual data processing, the two modalities are often augmented separately, such as rotating and cropping images, and randomly sampling or perturbing point clouds. This independent data augmentation method easily disrupts the strict spatial correspondence between the two types of data, leading to positional misalignment and semantic inconsistencies in the feature representation of the same object in the point cloud and image. This misalignment problem directly restricts the effectiveness of multimodal feature fusion and limits the accuracy of panoramic segmentation in complex traffic scenes. Summary of the Invention

[0003] This application provides a multimodal panoramic segmentation method based on traffic scenarios, which addresses the technical problem of spatial and semantic misalignment caused by independent enhancement of multimodal data in existing technologies, thus limiting the accuracy of panoramic segmentation.

[0004] In view of the above problems, this application provides a multimodal panoramic segmentation method based on traffic scenes. The method includes:

[0005] Acquire simultaneous LiDAR point cloud data and multi-view visible light image data in the same traffic scene to obtain raw multimodal perception data; perform modal synchronization enhancement processing on the raw multimodal perception data to obtain enhanced aligned multimodal data; input the point cloud data in the enhanced aligned multimodal data into a 3D encoding network for feature extraction to obtain voxel-level 3D geometric features; input the image data in the enhanced aligned multimodal data into a 2D encoding network for feature extraction to obtain image-level 2D visual features; based on sensor calibration parameters, perform cross-modal alignment and fusion on the voxel-level 3D geometric features and the image-level 2D visual features to obtain multimodal fusion features; input the multimodal fusion features into a dual-branch decoder based on the Transformer architecture for processing to obtain semantic category prediction results for each 3D point and instance mask prediction results for each independent traffic target; fuse the semantic category prediction results and the instance mask prediction results to obtain a panoramic segmentation result map of the traffic scene.

[0006] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0007] This application acquires synchronously collected LiDAR point cloud data and multi-view visible light image data in the same traffic scene to obtain raw multimodal perception data; performs modal synchronization enhancement processing on the raw multimodal perception data to obtain enhanced aligned multimodal data; inputs the point cloud data in the enhanced aligned multimodal data into a 3D encoding network for feature extraction to obtain voxel-level 3D geometric features; inputs the image data in the enhanced aligned multimodal data into a 2D encoding network for feature extraction to obtain image-level 2D visual features; performs cross-modal alignment and fusion on the voxel-level 3D geometric features and the image-level 2D visual features based on sensor calibration parameters to obtain multimodal fusion features; inputs the multimodal fusion features into a dual-branch decoder based on the Transformer architecture for processing to obtain semantic category prediction results for each 3D point and instance mask prediction results for each independent traffic target; and performs fusion processing on the semantic category prediction results and the instance mask prediction results to obtain a panoramic segmentation result map of the traffic scene. This invention addresses the technical problem of spatial and semantic misalignment caused by independent enhancement of multimodal data in existing technologies, which limits the accuracy of panoramic segmentation. By combining modal synchronous enhancement and cross-modal feature alignment fusion, the invention achieves the technical effect of improving the accuracy and robustness of panoramic segmentation in traffic scenes. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 is a schematic flowchart of a multimodal panoramic segmentation method based on a traffic scene provided in an embodiment of this application;

[0010] Figure 2 is a schematic diagram of the process of obtaining original multimodal perception data by a multimodal panoramic segmentation method based on traffic scenes provided in an embodiment of this application. Detailed Implementation

[0011] This application provides a multimodal panoramic segmentation method based on traffic scenes. It addresses the technical problem that the spatial and semantic misalignment caused by independent enhancement of multimodal data in existing technologies limits the accuracy of panoramic segmentation. By combining modal synchronous enhancement and cross-modal feature alignment fusion, the method improves the accuracy and robustness of panoramic segmentation in traffic scenes.

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0013] It should be noted that any variation of the terms "comprising" and "having" is intended to cover non-exclusive inclusion, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or modules that are not explicitly listed or that are inherent to such processes, methods, products, or devices.

[0014] As shown in Figure 1, this application provides a multimodal panoramic segmentation method based on traffic scenes, the method comprising:

[0015] Step S100: Acquire lidar point cloud data and multi-view visible light image data synchronously collected under the same traffic scene to obtain raw multimodal perception data.

[0016] In this embodiment, point cloud data in a traffic scene is first continuously acquired using a LiDAR device. Simultaneously, corresponding image data is acquired using a multi-view visible light camera. The continuously acquired point cloud data forms a LiDAR point cloud frame sequence, and the continuously acquired image data forms a multi-view image frame sequence. The LiDAR point cloud frame sequence and the multi-view image frame sequence together constitute a timestamped raw data stream.

[0017] Then, based on the joint calibration parameters of the sensors, a spatial mapping relationship between the point cloud and images from various viewpoints is established. Finally, the original data stream is registered according to this spatial mapping relationship and timestamps to obtain the original multimodal sensing data.

[0018] Furthermore, as shown in Figure 2, the method provided in the application embodiment further includes: acquiring the original multimodal sensing data.

[0019] Simultaneously acquire lidar point cloud frame sequences and multi-view image frame sequences to obtain a raw data stream with timestamps; establish a spatial mapping relationship between point clouds and images from various views based on sensor joint calibration parameters; register the raw data stream according to the spatial mapping relationship and timestamps to obtain spatiotemporally aligned raw multimodal sensing data.

[0020] In this embodiment, a LiDAR device and a multi-view visible light camera deployed in the same traffic scene first conduct synchronous data acquisition. The LiDAR device continuously captures the three-dimensional spatial information of various targets in the traffic scene, forming a LiDAR point cloud frame sequence composed of a large number of discrete three-dimensional points, where each three-dimensional point is defined as... This includes core data such as Cartesian coordinates (xyz coordinates) in Euclidean space and reflection intensity. Multi-view visible light cameras simultaneously capture two-dimensional images of a traffic scene from different spatial locations, forming a multi-view image frame sequence. This sequence is defined as... Where H and W represent the height and width of the image, respectively. This represents the image from the k-th viewpoint. A unique timestamp is added to each set of acquired point cloud frames and image frames, serving as a unique identifier for the data acquisition time. This process yields the final raw data stream with timestamps.

[0021] Next, a spatial mapping relationship between the point cloud and images from various viewpoints is established based on the joint calibration parameters of the sensors. These joint calibration parameters include extrinsic parameters (including translation vectors and rotation matrices) between the LiDAR and the multi-view visible light camera, and intrinsic parameters (including focal length, principal point coordinates, and distortion coefficients) of the multi-view visible light camera. Based on the camera's imaging principle, the spatial position and attitude relationship between the LiDAR and the multi-view visible light camera is clarified through the extrinsic parameters, while the imaging characteristics of the multi-view visible light camera itself are defined through the intrinsic parameters. This is achieved using a mapping formula. This method enables the transformation of any 3D point coordinates in a LiDAR point cloud to the corresponding 2D pixel coordinates in images from various viewpoints, where K represents the intrinsic parameters of the multi-view visible light camera and T represents the intrinsic parameter transformation matrix. express transpose of a point express The three-dimensional coordinates of a point; This represents the two-dimensional coordinates mapped to the image plane. This process establishes a one-to-one spatial relationship between point cloud data and image data from various viewpoints, ultimately yielding the spatial mapping relationship between the point cloud and the images from each viewpoint.

[0022] Finally, the original data stream is registered based on the established spatial mapping relationship and timestamps. At the time registration level, by comparing the timestamps of point cloud frames and image frames, it is ensured that the two types of data correspond to the same acquisition time. If there is a difference in the sampling frequency between the LiDAR and the multi-view visible light camera, time alignment algorithms such as interpolation are used to adjust the data, achieving precise synchronization in the time dimension. At the spatial registration level, the previously established spatial mapping relationship is utilized, combined with... The mapping formula precisely matches each 3D point in the LiDAR point cloud with the corresponding pixel region in the multi-view image, so that the same physical target forms a spatial correspondence in the point cloud data and image data. After dual processing of time synchronization and spatial alignment, the original spatiotemporally aligned multimodal perception data is finally obtained.

[0023] Step S200: Perform modal synchronization enhancement processing on the original multimodal sensing data to obtain enhanced and aligned multimodal data.

[0024] Furthermore, in the method provided in the application embodiments, the modal synchronization enhancement processing further includes:

[0025] The point cloud data is divided into cylindrical voxel units to obtain multiple cylindrical voxel units; a target replacement unit is selected from the multiple cylindrical voxel units based on a preset three-dimensional binary mask; the target replacement unit is replaced with a corresponding voxel block from a preset scene database; the pixel region corresponding to the target replacement unit in the image is replaced with a corresponding image block from the preset scene database; the replaced cylindrical voxel units and the corresponding image blocks constitute the enhanced aligned multimodal data.

[0026] In this embodiment of the application, when performing modal synchronization enhancement processing on the original multimodal sensing data, the lidar point cloud data in the original multimodal sensing data is first divided into cylindrical voxels. Based on the polar coordinates of the activated radar point cloud, the discrete three-dimensional point cloud is converted into structured cylindrical voxels. Each cylindrical voxel unit contains a variable number of three-dimensional points, and finally multiple cylindrical voxel units are obtained.

[0027] Then based on the preset three-dimensional binary mask Filter the target replacement unit, where, Z and Z represent the radial, angular, and height axes along the cylindrical voxel, respectively; the mask value S=1 indicates the voxel unit to be replaced, and S=0 indicates that the original state is maintained. The target replacement unit that meets the replacement requirements is selected by mask matching.

[0028] Then, the target replacement unit is replaced with the corresponding voxel block in the preset scene database. The preset scene database stores diverse voxel resources with the same structure and dimensions as the original voxel unit under different traffic scenarios. The replacement process follows the spatial position correspondence to ensure that the replaced voxel block is seamlessly connected with the surrounding unreplaced units, maintaining the continuity of three-dimensional geometric information, and obtaining the replaced point cloud voxel data.

[0029] Subsequently, the 3D points within the target replacement unit are mapped to the image plane using a spatial mapping formula to determine the corresponding pixel region. To select image blocks semantically consistent with the replacement voxel block, voxel blocks and image blocks in the preset scene database are associated and labeled according to traffic scene semantics. Each voxel block is accompanied by a semantic label, such as vehicle, road, pedestrian, greenery, etc., and corresponding image blocks with the same semantic labels from the same acquisition scene are stored to ensure that the voxel block and the image block describe the same type of traffic target or scene region. During replacement, the semantic label of the replacement voxel block is first extracted, and then the corresponding image block with the same semantic label and spatial resolution matching the corresponding pixel region of the image is matched in the preset scene database. The pixel region is then replaced synchronously to obtain the replaced image data.

[0030] Finally, the replaced cylindrical pixel units and corresponding image blocks are integrated, and conventional data augmentation methods such as random rotation, flipping, and scaling are used to enhance data diversity. The alignment accuracy of the two is verified through spatial mapping relationship to ensure that the spatiotemporal position and semantic information are completely matched, thus forming enhanced and aligned multimodal data.

[0031] Step S300: Input the point cloud data in the enhanced aligned multimodal data into a three-dimensional coding network for feature extraction to obtain voxel-level three-dimensional geometric features.

[0032] In this embodiment of the application, the point cloud voxel data (represented as...) in the enhanced aligned multimodal data after cylindrical voxelization is... Where M represents the number of effective cylindrical elements, For each individual cylindrical voxel, a 3D encoding network is input for feature extraction. This network employs the Cylinder3D architecture, processing voxelized point cloud data through cylindrical partitioning and asymmetric 3D convolution. Internally, it includes multiple downsampling modules and a context aggregation module. The downsampling module sequentially performs 3D convolution, batch normalization, activation functions, and other operations to progressively compress the feature space dimension and extract high-level geometric features. The context aggregation module fuses multi-scale features, compensating for details lost during downsampling. During feature extraction, for each cylindrical voxel... The point cloud data is processed layer by layer, including its contained 3D point information, to capture the 3D spatial structure, distance relationships, and local topological features of the point cloud data. The final output dimension is... The voxel-level three-dimensional geometric features, where D is the feature dimension.

[0033] Step S400: Input the image data in the enhanced aligned multimodal data into a two-dimensional coding network for feature extraction to obtain image-level two-dimensional visual features.

[0034] In this embodiment of the application, image data in enhanced aligned multimodal data Feature extraction is performed using a two-dimensional encoding network, which adopts a ResNet-based architecture. The two-dimensional encoding network contains multiple convolutional modules, each consisting of a convolutional layer, a batch normalization layer, and an activation function. By progressively increasing the receptive field of the convolutional kernels and enhancing the number of feature channels, it extracts visual information from the image layer by layer, from low-level to high-level. Low-level features focus on basic visual elements such as edges, textures, and colors, while high-level features encompass abstract information such as target contours and semantic relationships.

[0035] Meanwhile, the two-dimensional coding network compresses spatial dimensions and enhances feature representation through downsampling operations, and retains key details by combining multi-scale feature fusion strategies, ultimately resulting in an output dimension of [missing value]. The image-level two-dimensional visual features are defined as follows: M represents the number of effective cylindrical pixels, K represents the K views of the image, H and W represent the height and width of the image, respectively, and D represents the feature dimension, which is consistent with the feature dimension output by the three-dimensional coding network.

[0036] Furthermore, the method provided in the application embodiments also includes:

[0037] The three-dimensional coding network adopts the Cylinder3D architecture and outputs multi-scale voxel-level three-dimensional geometric features; the two-dimensional coding network adopts a ResNet-based architecture and outputs multi-scale image-level two-dimensional visual features.

[0038] In this embodiment, the 3D coding network adopts the Cylinder3D architecture, taking the point cloud voxel data after cylindrical voxelization from the enhanced aligned multimodal data as input, and processing it layer by layer through its internal downsampling module and context aggregation module, outputting a dimension of Multiscale voxel-level three-dimensional geometric features.

[0039] The 2D coding network adopts a ResNet-based architecture, taking image data from enhanced aligned multimodal data as input. After processing by multiple internal convolutional modules and multi-scale feature fusion, the output dimension is... Multi-scale image-level two-dimensional visual features.

[0040] Step S500: Based on the sensor calibration parameters, perform cross-modal alignment and fusion on the voxel-level three-dimensional geometric features and the image-level two-dimensional visual features to obtain multimodal fusion features.

[0041] Furthermore, in the method provided in the application embodiments, the cross-modal alignment and fusion further includes:

[0042] Based on the camera's intrinsic and extrinsic parameters, the three-dimensional points within each voxel unit are projected onto a two-dimensional image plane to obtain their corresponding pixel region coordinates; the image-level two-dimensional visual features within the pixel region coordinates are aggregated and calculated to obtain two-dimensional aggregated features aligned with the voxel unit; the two-dimensional aggregated features are then stitched together with the corresponding voxel-level three-dimensional geometric features to obtain the multimodal fusion features.

[0043] In this embodiment, when performing cross-modal alignment and fusion of voxel-level three-dimensional geometric features and image-level two-dimensional visual features based on sensor calibration parameters, cross-modal alignment is first performed. Based on the camera intrinsic and extrinsic parameters in the sensor calibration parameters, all three-dimensional points in each voxel unit are projected onto the two-dimensional image plane one by one through the spatial mapping formula. Then, the rectangular pixel region enveloping the voxel unit is determined according to the distribution range of all projected points, thereby obtaining the pixel region coordinates corresponding to each voxel unit, and realizing the accurate alignment of voxel-level three-dimensional geometric features and image-level two-dimensional visual features in spatial dimension.

[0044] Next, two-dimensional aggregated feature extraction is performed. For the pixel region coordinates corresponding to each voxel unit, all two-dimensional visual features within that pixel region are extracted from the image-level two-dimensional visual features output by the two-dimensional coding network, and then processed using the formula... = Calculations are performed to obtain the two-dimensional aggregated feature that is precisely aligned with the i-th voxel unit. . in, Let be the number of three-dimensional points contained within the i-th voxel unit. For the image Two-dimensional visual features corresponding to coordinates.

[0045] Finally, cross-modal fusion is performed to aggregate the two-dimensional features corresponding to each voxel unit. The i-th voxel-level three-dimensional geometric feature corresponding to the multi-scale voxel-level three-dimensional geometric features output by the three-dimensional coding network. The feature concatenation operation Concat is used for fusion, that is... The fusion features of all voxel units are integrated to form a multimodal fusion feature with unified dimensions and both three-dimensional geometric information and two-dimensional visual information.

[0046] Step S600: Input the multimodal fusion features into a dual-branch decoder based on the Transformer architecture for processing to obtain the semantic category prediction result of each 3D point and the instance mask prediction result of each independent traffic target.

[0047] Furthermore, the method provided in the application embodiments also includes:

[0048] The dual-branch decoder includes a parallel semantic decoding branch and an instance decoding branch; the output of the semantic decoding branch is the semantic category prediction result, which classifies uncountable areas such as roads, sky, and greenery; the output of the instance decoding branch is the instance mask prediction result, which segments countable targets such as vehicles and pedestrians and assigns them different labels.

[0049] In this embodiment of the application, the dual-branch decoder includes a parallel semantic decoding branch and an instance decoding branch.

[0050] After the multimodal fusion features are fed into the shared feature encoding layer of the dual-branch decoder, this layer relies on the encoder module of the Transformer to perform global correlation modeling on the fusion features of all voxel units through self-attention calculation, weighted aggregate the feature information of all voxel units, and obtains enhanced global fusion features containing global semantic relationships, thus completing the high-order information extraction of the multimodal fusion features.

[0051] Subsequently, the enhanced global fusion features are simultaneously input into two parallel branches. The semantic decoding branch receives the global fusion features, first maps the features to a preset semantic category space through a fully connected layer, and then determines the semantic category corresponding to each voxel unit through a classification activation operation. Then, based on the attribution relationship between voxels and 3D points, combined with the association logic between voxels and point clouds corresponding to the spatial mapping formula, the semantic category result of the voxel is mapped to all the 3D points it contains, and the semantic category prediction result is output, realizing accurate classification of uncountable areas such as roads, sky, and green plants.

[0052] The instance decoding branch interacts with the global fusion features and the preset instance query vector. It learns the correlation between voxel units and instance query vectors by using the cross attention mechanism of Transformer. After activation operation, it outputs the instance attribution probability of each voxel unit. Then, it maps the probability to the corresponding 3D point to obtain the instance attribution label of each 3D point and outputs the instance mask prediction result. This completes the segmentation of countable targets such as vehicles and pedestrians and assigns independent labels to different targets.

[0053] Finally, the dual-branch decoder synchronously outputs the semantic category prediction results generated by the semantic decoding branch and the instance mask prediction results generated by the instance decoding branch.

[0054] Step S700: The semantic category prediction result and the instance mask prediction result are fused to obtain a panoramic segmentation result map of the traffic scene.

[0055] Furthermore, in the method provided in the application embodiments, the fusion processing of the semantic category prediction result and the instance mask prediction result further includes:

[0056] The non-background category segmentation regions in the instance mask prediction results are used, and the remaining regions are filled with the semantic category prediction results to generate the non-overlapping panoramic segmentation result map.

[0057] In this embodiment, when fusing the semantic category prediction result and the instance mask prediction result, the segmentation regions of non-background categories such as vehicles and pedestrians in the instance mask prediction result are first extracted, and the target identifiers and spatial location information corresponding to these regions are retained. Then, for the remaining areas in the scene not covered by the instance mask, the semantic category prediction result is used to fill in the remaining areas, supplementing the category information of uncountable regions such as roads, sky, and greenery. Finally, the two types of results are integrated to form a non-overlapping unified segmentation layer, resulting in a panoramic segmentation image of the traffic scene.

[0058] Furthermore, the method provided in the application embodiments also includes:

[0059] A panoramic segmentation loss function is constructed, comprising semantic segmentation branch loss and instance segmentation branch loss. The difference between the semantic category prediction result and the real semantic label, and the difference between the instance mask prediction result and the real instance label are calculated to obtain the total loss value. Based on the total loss value, the gradients of each parameter in the 3D encoding network, the 2D encoding network, and the dual-branch decoder are calculated using the backpropagation algorithm to obtain parameter gradient information. Using the parameter gradient information, the parameters of the 3D encoding network, the 2D encoding network, and the dual-branch decoder are iteratively updated using an optimization algorithm to obtain the parameter-updated 3D encoding network, the 2D encoding network, and the dual-branch decoder.

[0060] In this embodiment, a panoptic segmentation loss function is first constructed, comprising semantic segmentation branch loss and instance segmentation branch loss, specifically in the form of: ,in, and This represents the weighting coefficients, which sum to 1 and are preset by technical experts.

[0061] In the loss calculation stage, the semantic segmentation branch loss uses the cross-entropy loss function to calculate the difference between the semantic category prediction result and the true semantic label, as shown in the formula: Where C represents the number of semantic segmentation categories, This represents the probability that the predicted sample belongs to class c. This represents the true distribution of label categories, thereby constraining the classification accuracy of uncountable areas such as roads, sky, and greenery.

[0062] The instance segmentation branch loss consists of three parts: the bounding box classification loss, the bounding box location loss, and the mask segmentation loss, as shown in the formula: .in, , , This represents the weighting coefficient, which is preset by technical experts. This represents the classification loss function for the detection boxes. This represents the loss function for the detection box position. This represents the mask segmentation loss function. The bounding box classification loss is calculated using... Calculate the classification difference between the predicted bounding boxes and the ground truth labels. Here, N represents the total number of predicted bounding boxes. This indicates the probability that the bounding box predicts the target. The predicted bounding box represents the ground truth label, and the detection box position loss is first passed through... Calculate the intersection-union ratio (IUGR). The predicted bounding box is denoted as P, and the labeled bounding box as G.

[0063] Then pass The quantization position error is replaced by a binary cross-entropy loss for mask segmentation. The difference between the instance mask prediction result and the real instance label is calculated, where P is the predicted category and y is the actual category. The two types of losses are weighted and summed to obtain the total loss value of panoptic segmentation.

[0064] Based on the total loss value, the gradients of all learnable parameters in the 3D encoder network, 2D encoder network, and bi-branch decoder are calculated layer by layer using the backpropagation algorithm, generating corresponding parameter gradient information. Subsequently, optimization algorithms such as Adam are used to iteratively update the parameters of the above networks based on this gradient information. The process of forward calculation, loss calculation, gradient backpropagation, and parameter update is repeated until the total loss value converges, finally obtaining the parameter-updated 3D encoder network, 2D encoder network, and bi-branch decoder.

[0065] In summary, the embodiments of this application have at least the following technical effects:

[0066] This application acquires synchronously collected LiDAR point cloud data and multi-view visible light image data in the same traffic scene to obtain raw multimodal perception data; performs modal synchronization enhancement processing on the raw multimodal perception data to obtain enhanced aligned multimodal data; inputs the point cloud data in the enhanced aligned multimodal data into a 3D encoding network for feature extraction to obtain voxel-level 3D geometric features; inputs the image data in the enhanced aligned multimodal data into a 2D encoding network for feature extraction to obtain image-level 2D visual features; performs cross-modal alignment and fusion on the voxel-level 3D geometric features and the image-level 2D visual features based on sensor calibration parameters to obtain multimodal fusion features; inputs the multimodal fusion features into a dual-branch decoder based on the Transformer architecture for processing to obtain semantic category prediction results for each 3D point and instance mask prediction results for each independent traffic target; and performs fusion processing on the semantic category prediction results and the instance mask prediction results to obtain a panoramic segmentation result map of the traffic scene. This invention addresses the technical problem of spatial and semantic misalignment caused by independent enhancement of multimodal data in existing technologies, which limits the accuracy of panoramic segmentation. By combining modal synchronous enhancement and cross-modal feature alignment fusion, the invention achieves the technical effect of improving the accuracy and robustness of panoramic segmentation in traffic scenes.

[0067] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this specification. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are possible or may be advantageous.

[0068] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A multimodal panoramic segmentation method based on traffic scenes, characterized in that, The method includes: acquiring synchronously collected LiDAR point cloud data and multi-view visible light image data in the same traffic scene to obtain raw multimodal perception data; performing modal synchronization enhancement processing on the raw multimodal perception data to obtain enhanced and aligned multimodal data; inputting the point cloud data in the enhanced and aligned multimodal data into a 3D encoding network for feature extraction to obtain voxel-level 3D geometric features; inputting the image data in the enhanced and aligned multimodal data into a 2D encoding network for feature extraction to obtain image-level 2D visual features; performing cross-modal alignment and fusion on the voxel-level 3D geometric features and the image-level 2D visual features based on sensor calibration parameters to obtain multimodal fusion features; inputting the multimodal fusion features into a dual-branch decoder based on the Transformer architecture for processing to obtain semantic category prediction results for each 3D point and instance mask prediction results for each independent traffic target; and fusing the semantic category prediction results and the instance mask prediction results to obtain a panoramic segmentation result map of the traffic scene.

2. The multimodal panoramic segmentation method based on traffic scenes according to claim 1, characterized in that, The acquisition of raw multimodal sensing data includes: simultaneously acquiring lidar point cloud frame sequences and multi-view image frame sequences to obtain raw data streams with timestamps; establishing spatial mapping relationships between point clouds and images from various viewpoints based on sensor joint calibration parameters; and registering the raw data streams according to the spatial mapping relationships and timestamps to obtain spatiotemporally aligned raw multimodal sensing data.

3. The multimodal panoramic segmentation method based on traffic scenes according to claim 1, characterized in that, The modal synchronization enhancement process includes: dividing the point cloud data into cylindrical voxel units to obtain multiple cylindrical voxel units; selecting target replacement units from the multiple cylindrical voxel units based on a preset three-dimensional binary mask; replacing the target replacement units with corresponding voxel blocks from a preset scene database; replacing the pixel regions corresponding to the target replacement units in the image with corresponding image blocks from the preset scene database; and constructing the enhanced aligned multimodal data from the replaced cylindrical voxel units and the corresponding image blocks.

4. The multimodal panoramic segmentation method based on traffic scenes according to claim 1, characterized in that, The three-dimensional coding network adopts the Cylinder3D architecture and outputs multi-scale voxel-level three-dimensional geometric features; the two-dimensional coding network adopts a ResNet-based architecture and outputs multi-scale image-level two-dimensional visual features.

5. The multimodal panoramic segmentation method based on traffic scenes according to claim 1, characterized in that, The cross-modal alignment and fusion includes: projecting the three-dimensional points in each voxel unit onto a two-dimensional image plane according to the camera's intrinsic and extrinsic parameters to obtain its corresponding pixel region coordinates; performing aggregation calculation on the image-level two-dimensional visual features within the pixel region coordinates to obtain two-dimensional aggregated features aligned with the voxel unit; and concatenating the two-dimensional aggregated features with the corresponding voxel-level three-dimensional geometric features to obtain the multimodal fusion features.

6. The multimodal panoramic segmentation method based on traffic scenes according to claim 1, characterized in that, The dual-branch decoder includes a parallel semantic decoding branch and an instance decoding branch; The output of the semantic decoding branch is the semantic category prediction result, which classifies uncountable areas such as roads, sky, and greenery; the output of the instance decoding branch is the instance mask prediction result, which segments countable targets such as vehicles and pedestrians and assigns them different labels.

7. The multimodal panoramic segmentation method based on traffic scenes according to claim 1, characterized in that, The semantic category prediction result and the instance mask prediction result are fused together, including: using the segmentation regions of non-background categories in the instance mask prediction result, and filling the remaining regions with the semantic category prediction result to generate the panoramic segmentation result map.

8. The multimodal panoramic segmentation method based on traffic scenes according to claim 1, characterized in that, Also includes: A panoramic segmentation loss function is constructed, comprising semantic segmentation branch loss and instance segmentation branch loss. The difference between the semantic category prediction result and the real semantic label, and the difference between the instance mask prediction result and the real instance label are calculated to obtain the total loss value. Based on the total loss value, the gradients of each parameter in the 3D encoding network, the 2D encoding network, and the dual-branch decoder are calculated using the backpropagation algorithm to obtain parameter gradient information. Using the parameter gradient information, the parameters of the 3D encoding network, the 2D encoding network, and the dual-branch decoder are iteratively updated using an optimization algorithm to obtain the parameter-updated 3D encoding network, the 2D encoding network, and the dual-branch decoder.

Images