A multi-modal fusion road detection method, device, equipment and medium
By employing a multimodal fusion road detection method, which combines vehicle surround-view cameras and bird's-eye view segmentation networks with a diffusion model, the problems of low recognition accuracy and blurred boundaries in long-distance areas are solved, achieving high-precision detection of complex road surfaces and meeting the high-reliability perception requirements of intelligent driving.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- FAW JIEFANG AUTOMOTIVE CO
- Filing Date
- 2026-05-18
- Publication Date
- 2026-07-14
Smart Images
Figure CN122392019A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent driving environment perception technology, and in particular to a multimodal fusion road detection method, device, equipment and medium. Background Technology
[0002] In recent years, intelligent driving technology has developed rapidly. Environmental perception, as a core module of intelligent driving systems, directly determines the safety and reliability of autonomous driving. Among these, road detection and drivable area identification are key aspects of environmental perception. Existing technologies are mainly divided into three categories: detection methods based on 2D images, BEV (Bird's-Eye-View) segmentation methods based on multimodal fusion, and methods based on multi-task learning. Each method achieves road area identification and detection through different technical paths to adapt to the basic perception requirements of intelligent driving.
[0003] Existing road detection technologies still have many shortcomings, making it difficult to meet the high-precision and high-reliability perception requirements of intelligent driving: First, the detection accuracy at long distances is insufficient. Due to the small pixel ratio of distant targets, BEV projection errors, and insufficient utilization of contextual information, the detection accuracy in distant areas drops significantly. Second, boundary recognition is ambiguous. Existing methods have low classification accuracy for the boundaries between roads and adjacent areas such as shoulders and sidewalks, and cannot achieve accurate differentiation. Third, they have poor adaptability to complex road conditions. The detection effect is not ideal for non-standard road surfaces such as gravel roads, icy roads, and waterlogged roads. Fourth, they have a high dependence on multimodal reasoning. Some BEV segmentation methods rely on LiDAR data during the reasoning stage, which increases the cost and complexity of the system. Summary of the Invention
[0004] Based on this, the present invention provides a multimodal fusion road detection method, apparatus, equipment and medium to solve the problems of low accuracy in long-distance area recognition, blurred road boundary distinction and insufficient ability to adapt to and recognize various complex road surface scenarios in existing road detection technologies.
[0005] In a first aspect, embodiments of the present invention provide a multimodal fusion road detection method, comprising: Preprocessing is performed on the original multi-view images captured by the vehicle surround view camera to obtain preprocessed multi-view images. The multi-view images are then input into a pre-trained multimodal fusion bird's-eye view segmentation network to extract visual features at different scales layer by layer, resulting in multi-scale image feature maps. Based on preset camera intrinsic and extrinsic parameters, spatial transformation and feature aggregation processing are performed on multi-scale image feature maps from camera viewpoint to bird's-eye viewpoint to obtain a globally unified bird's-eye viewpoint feature representation. The multimodal fusion bird's-eye view segmentation network is invoked to perform pixel-level multi-class road surface semantic classification prediction on the bird's-eye view feature representation, resulting in an initial bird's-eye view road surface segmentation result containing multiple road surface types. The initial bird's-eye view road segmentation result is used as a conditional input to the pre-trained diffusion model for repair and optimization, resulting in a refined bird's-eye view road segmentation map. The detailed bird's-eye view road segmentation map is subjected to morphological denoising and temporal filtering post-processing operations to obtain the final road detection results.
[0006] Secondly, embodiments of the present invention also provide a multimodal fusion road detection device, comprising: The multimodal feature extraction module is used to perform preprocessing operations on the original multi-view images captured by the vehicle surround view camera to obtain preprocessed multi-view images. The multi-view images are then input into a pre-trained multimodal fusion bird's-eye view segmentation network to extract visual features at different scales layer by layer, resulting in multi-scale image feature maps. The perspective transformation module is used to perform spatial transformation and feature aggregation processing on multi-scale image feature maps from camera perspective to bird's-eye view based on preset camera intrinsic and extrinsic parameters, so as to obtain a globally unified bird's-eye view feature representation. The initial segmentation prediction module is used to call the multimodal fusion bird's-eye view segmentation network to perform pixel-level multi-class road surface semantic classification prediction on the bird's-eye view feature representation, so as to obtain the initial bird's-eye view road surface segmentation result containing multiple road surface types. The diffusion model optimization module is used to take the initial bird's-eye view road segmentation result as a conditional input to the pre-trained diffusion model, perform repair and optimization, and obtain a refined bird's-eye view road segmentation map. The post-processing module performs morphological denoising and temporal filtering on the refined bird's-eye view road segmentation map to obtain the final road detection result.
[0007] Thirdly, embodiments of the present invention also provide an electronic device, the electronic device comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform a multimodal fusion road detection method according to any embodiment of the present invention.
[0008] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer instructions, which are used to cause a processor to execute and implement a multimodal fusion road detection method according to any embodiment of the present invention.
[0009] Fifthly, embodiments of the present invention also provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements a multimodal fusion road detection method as described in any embodiment of the present invention.
[0010] This invention, through multi-scale feature extraction from multi-view images combined with bird's-eye view spatial transformation and feature aggregation, effectively enhances the ability to represent road features at long distances and addresses the problem of insufficient detection accuracy at distant locations. It achieves fine-grained division of road areas based on multi-category pixel-level semantic segmentation, effectively mitigating the blurring of boundaries between roads and adjacent areas. Furthermore, by combining diffusion model generative repair and post-processing optimization, it significantly improves the ability to identify and adapt to various complex road surfaces such as asphalt, sand, snow, and sidewalks. Overall, it enhances the accuracy of long-distance road detection, the precision of boundary segmentation, and the generalization ability across multiple road conditions, meeting the high-precision and reliable detection requirements of intelligent driving environment perception.
[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description
[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0013] Figure 1 This is a flowchart of a multimodal fusion road detection method according to Embodiment 1 of the present invention; Figure 2 This is a flowchart of another multimodal fusion road detection method provided in Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the structure of a multimodal fusion road detection device according to Embodiment 3 of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device that implements a multimodal fusion road detection method according to an embodiment of the present invention. Detailed Implementation
[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0016] Example 1 Figure 1 This is a flowchart of a multimodal fusion road detection method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations in intelligent driving scenarios where high-precision, long-distance, and boundary-accurate road detection is performed on various complex road surfaces surrounding a vehicle. This method can be executed by a multimodal fusion road detection device, which can be implemented in hardware and / or software and can be configured in an intelligent vehicle. Figure 1 As shown, the method includes: S110. Perform preprocessing operations on the original multi-view images captured by the vehicle surround view camera to obtain preprocessed multi-view images. Input the multi-view images into the pre-trained multimodal fusion bird's-eye view segmentation network to extract visual features at different scales layer by layer to obtain multi-scale image feature maps.
[0017] Surround-view cameras are deployed around the intelligent vehicle, simultaneously capturing multiple views of the surrounding environment. However, these raw images often suffer from lens distortion, lighting differences, and imaging noise, making them unsuitable for direct input into the network for feature calculation. Preprocessing is implemented to correct, regularize, and standardize the raw images, eliminating interference from hardware imaging and outputting image data that conforms to the network input specifications. The multimodal fusion bird's-eye view segmentation network is a dedicated perception network pre-trained offline, possessing multi-level feature extraction capabilities. Through layer-by-layer computation, it can capture shallow edge details and deep global structural information, forming multi-scale image feature maps.
[0018] S120. Based on preset camera intrinsic and extrinsic parameters, perform spatial transformation and feature aggregation processing on multi-scale image feature maps from camera viewpoint to bird's-eye viewpoint to obtain a globally unified bird's-eye viewpoint feature representation.
[0019] The camera's intrinsic and extrinsic parameters are fixed parameters obtained from prior calibration, defining the optical properties and mounting pose of each camera relative to the vehicle body, serving as the baseline for viewpoint coordinate mapping. Each surround view image represents a local perspective camera view, with independent coordinate systems and fragmented fields of view, failing to directly express the continuous road topology surrounding the vehicle. Spatial transformation is used to uniformly map the multi-scale feature maps from the perspective view to the bird's-eye view coordinate system, and then feature aggregation is used to stitch and fuse the dispersed features from multiple paths, eliminating blind spots and coordinate discrepancies.
[0020] S130. The multimodal fusion bird's-eye view segmentation network is invoked to perform pixel-level multi-class road surface semantic classification prediction on the bird's-eye view feature representation, so as to obtain an initial bird's-eye view road surface segmentation result containing multiple road surface types.
[0021] After obtaining a globally unified bird's-eye view feature representation, the pre-trained segmentation network is reused to perform pixel-level semantic reasoning. Each pixel unit in the feature map is individually classified into its corresponding road surface semantic category, distinguishing between different traversable and non-traversable areas. The initial bird's-eye view road segmentation result directly output by the network is generated based on only single-level feature reasoning, inevitably containing defects such as irregular edges, small-area noise, and local structural breaks, which constitutes a coarse-grained semantic segmentation output.
[0022] S140. The initial bird's-eye view road segmentation result is used as a conditional input to the pre-trained diffusion model for repair and optimization, resulting in a refined bird's-eye view road segmentation map.
[0023] The diffusion model possesses strong image generation and detail reconstruction capabilities, making it suitable for edge smoothing, structural completion, and detail regularization of semantic segmentation results. Using the initial bird's-eye view road segmentation result as a constraint input to the diffusion model, it can generatively repair and complete segmentation edges, broken lanes, and voids without altering the original pixel semantic categories. Compared to traditional filtering, which only provides simple noise reduction, the diffusion model can achieve structural detail reconstruction, thereby outputting a refined bird's-eye view road segmentation map with regular boundaries, topological continuity, and semantic accuracy.
[0024] S150. Perform morphological denoising and temporal filtering post-processing operations on the refined bird's-eye view road segmentation map to obtain the final road detection result.
[0025] Even after optimization using a diffusion model, single-frame segmentation images may still retain sporadic isolated noise points. Furthermore, continuous video streams from vehicles exhibit issues such as jumps and jitter in single-frame detection results. Morphological denoising removes isolated noise points, fills in small holes, and smooths region boundaries in the spatial dimension. Temporal filtering utilizes the correlation between consecutive frames to impose inter-frame constraints and smooth the flow over time, suppressing instantaneous abnormal fluctuations. Through the synergistic effect of these two post-processing stages, both regional regularity of a single frame and detection stability between consecutive frames can be ensured, resulting in a final road detection result that can be directly used for vehicle planning and perception decision-making.
[0026] This invention, through multi-scale feature extraction from multi-view images combined with bird's-eye view spatial transformation and feature aggregation, effectively enhances the ability to represent road features at long distances and addresses the problem of insufficient detection accuracy at distant locations. It achieves fine-grained division of road areas based on multi-category pixel-level semantic segmentation, effectively mitigating the blurring of boundaries between roads and adjacent areas. Furthermore, by combining diffusion model generative repair and post-processing optimization, it significantly improves the ability to identify and adapt to various complex road surfaces such as asphalt, sand, snow, and sidewalks. Overall, it enhances the accuracy of long-distance road detection, the precision of boundary segmentation, and the generalization ability across multiple road conditions, meeting the high-precision and reliable detection requirements of intelligent driving environment perception.
[0027] Optionally, preprocessing operations can be performed on the original multi-view images captured by the vehicle surround-view camera to obtain preprocessed multi-view images, which may include: By simultaneously acquiring images of the vehicle's surrounding environment through multi-angle surround-view cameras deployed around the vehicle, original images covering the entire area around the vehicle at the same time can be obtained. Based on pre-calibrated camera parameters, distortion correction is performed on the original images from multiple perspectives to obtain corrected images that eliminate lens distortion; The corrected image is subjected to brightness equalization, color gamut conversion, channel conversion and pixel normalization to obtain a standardized image with unified imaging style and data dimension. Standardized images are scaled down to obtain preprocessed multi-view images that meet network input specifications.
[0028] Surround-view cameras are mounted around the vehicle in a multi-directional layout. A synchronous acquisition mechanism ensures that the imaging time of each camera is consistent, avoiding parallax and scene shift caused by timing misalignment. This multi-directional hardware layout can completely cover the entire field of view around the vehicle, with no obvious blind spots, thus capturing the entire surrounding environment in one go. Surround-view cameras often use wide-angle or fisheye lenses, which naturally exhibit radial and tangential distortion. This can cause bending and deformation of road edges and lane contours, directly affecting the accuracy of subsequent feature extraction and perspective transformation. The camera parameters obtained through offline calibration can accurately characterize the lens optical distortion patterns. Based on these calibration parameters, pixel coordinates are reverse-mapped to correct the original image, restoring the true geometric shape of the scene at the imaging level.
[0029] Different orientations of panoramic cameras, influenced by installation angles and lighting directions, can easily lead to issues such as variations in image brightness, color gamut shifts, and inconsistent channel formats. Directly feeding these images into the network can cause feature extraction biases. Brightness equalization smooths out the brightness differences between images, color gamut and channel conversion unifies the image color space and data layout format, and pixel normalization maps pixel values to the numerical range suitable for the model. Since the resolution and aspect ratio of the original and corrected images differ, and the multimodal fusion bird's-eye view segmentation network has a fixed input size constraint, non-standardized images cannot directly participate in forward inference. By uniformly scaling all standardized images to the network's preset fixed resolution and aspect ratio, the network adapts to the model's input specifications while preserving effective scene features.
[0030] Furthermore, the multi-view images are input into a pre-trained multimodal fusion bird's-eye view segmentation network, and visual features at different scales are extracted layer by layer to obtain multi-scale image feature maps, which may include: The preprocessed multi-view images are input into the pre-trained multimodal fusion bird's-eye view segmentation network, which has a built-in Transformer structure bird's-eye view encoder for feature extraction and spatial mapping. The backbone network of the bird's-eye view encoder is used to extract multi-scale features from images from various viewpoints to obtain two-dimensional image features from each camera viewpoint. Multi-scale information fusion is performed on the features of each two-dimensional image to obtain a multi-scale image feature map containing multi-view texture and semantic information.
[0031] The core functional modules of the multimodal fusion bird's-eye view segmentation network include feature extraction, view transformation, and semantic segmentation. Among them, the Transformer-structured bird's-eye view encoder is the core component responsible for feature extraction and spatial mapping. It is built into the segmentation network, eliminating the need for external modules and simplifying the overall computation process. The Transformer structure has powerful global feature capture and multi-scale correlation capabilities. Compared with traditional convolutional structures, it is more suitable for handling global feature correlation and spatial mapping tasks of multi-view images, and can effectively explore semantic associations and spatial correspondences between images from different viewpoints.
[0032] The backbone network of the bird's-eye view encoder employs a multi-level feature extraction structure, enabling it to perform feature extraction layer by layer on the pre-processed image for each viewpoint. For example, shallow layers primarily capture fine-grained, small-scale features such as edges, lines, and textures, while deep layers focus on global, large-scale semantic features such as road structure, lane orientation, and the outlines of surrounding obstacles. Unlike single-scale feature extraction, multi-scale extraction can take into account both detailed and global features, preserving subtle structural information such as road edges and lane lines while capturing global information such as the overall road topology and multi-view correlations, thus avoiding subsequent semantic segmentation deviations due to incomplete feature extraction.
[0033] Two-dimensional image features from a single path can only reflect local scene information from the corresponding viewpoint, resulting in a limited field of view and an inability to fully represent the road surface features around the vehicle. If directly used for subsequent bird's-eye view transformation and semantic segmentation, it will lead to the loss of global features and insufficient segmentation accuracy. Multi-scale information fusion is not a simple feature stitching, but rather a deep fusion of two-dimensional image features from different viewpoints and scales based on the scale correlation and spatial correspondence of features from various viewpoints. This process eliminates redundant features, enhances effective features, and achieves complementarity and synergy of multi-viewpoint information.
[0034] Optionally, based on preset camera intrinsic and extrinsic parameters, spatial transformation and feature aggregation processing are performed on multi-scale image feature maps from camera viewpoint to bird's-eye viewpoint to obtain a globally unified bird's-eye viewpoint feature representation, which may include: Based on preset camera intrinsic and extrinsic parameters, a spatial mapping relationship is established between the image space of each camera viewpoint and the vehicle's bird's-eye view space. A deformable attention mechanism is used to adaptively sample the multi-scale image feature map, and the two-dimensional image features from each camera viewpoint are projected and transformed into a unified vehicle bird's-eye view space. In the vehicle's bird's-eye view space, the projection features corresponding to all camera viewpoints are fused and aggregated to integrate multi-directional spatial feature information and generate a unified bird's-eye view feature representation with global spatial position representation capabilities.
[0035] The camera's intrinsic and extrinsic parameters are pre-calibrated and fixed, accurately characterizing the optical properties of each surround-view camera and its mounting pose relative to the vehicle body. Based on these parameters, a fixed coordinate transformation relationship can be established between the two-dimensional perspective image space and the bird's-eye view space, clearly defining the positional correspondence rules of each pixel feature in both perspectives. The deformable attention mechanism possesses flexible adaptive sampling capabilities, dynamically adjusting the sampling position according to the actual distribution of image semantic features, not limited to a fixed grid sampling mode. The deformable attention mechanism is used to adaptively sample and select effective feature points from multi-scale feature maps, and then, according to the established spatial mapping relationship, the two-dimensional features from each camera's perspective are mapped and transformed point-by-point to the vehicle's bird's-eye view coordinate system.
[0036] After each camera's viewpoint is projected individually, its features remain scattered across the bird's-eye view, lacking correlation and integration, and cannot be directly used for global semantic discrimination. By performing cross-view fusion and aggregation of all projected features within a unified bird's-eye view, multi-directional feature information complementarity and redundant information suppression are achieved, filling in the feature content missing in the blind spots of a single camera's field of view.
[0037] Example 2 Figure 2 This is a flowchart of another multimodal fusion road detection method provided in Embodiment 2 of the present invention. This embodiment is a refinement based on Embodiment 1, specifically as follows: Figure 2 As shown, the method includes: S210. Perform preprocessing operations on the original multi-view images captured by the vehicle surround view camera to obtain preprocessed multi-view images. Input the multi-view images into the pre-trained multimodal fusion bird's-eye view segmentation network to extract visual features at different scales layer by layer to obtain multi-scale image feature maps.
[0038] S220. Based on preset camera intrinsic and extrinsic parameters, perform spatial transformation and feature aggregation processing on multi-scale image feature maps from camera viewpoint to bird's-eye viewpoint to obtain a globally unified bird's-eye viewpoint feature representation.
[0039] S230. Input the bird's-eye view feature representation into the segmentation head of the multimodal fusion bird's-eye view segmentation network, and use the built-in independent Transformer decoder and multi-scale convolutional layers to perform semantic parsing and feature refinement on the bird's-eye view feature representation, and output feature channels corresponding to the number of road surface categories.
[0040] The segmentation head is the core module in the multimodal fusion bird's-eye view segmentation network, specifically responsible for semantic classification output. It works in conjunction with the aforementioned bird's-eye view encoder to jointly complete feature extraction and semantic discrimination. Its built-in Transformer decoder has powerful global semantic association parsing capabilities, enabling it to deeply mine road surface semantic information in the bird's-eye view feature representation and clarify the feature associations between different road surface categories. The multi-scale convolutional layers are responsible for further refining the features, strengthening key features such as road surface edges and category boundaries, and compensating for the detailed information lost during the global parsing process.
[0041] The core function of this step is to transform the globally unified bird's-eye view feature representation from the abstract feature space to specific road surface category features. Through semantic parsing and detail refinement, the feature information is transformed into feature channels corresponding to the preset number of road surface categories, providing clear and accurate feature basis for subsequent pixel-level category determination. This is a key link connecting feature representation and category prediction.
[0042] S240. Perform probability normalization on the feature channels to obtain the probability of each pixel belonging to each type of road surface.
[0043] The feature channels output by the segmentation head can only reflect the feature response intensity of each pixel corresponding to different road surface categories, and cannot be directly used as the basis for category determination. Moreover, the numerical ranges of different feature channels are not uniform, which can easily lead to category determination bias. The probability normalization operation uses a specific algorithm to map the response value of each pixel on each feature channel to a probability interval of 0-1, so that the sum of the probabilities of all pixels corresponding to each type of road surface is 1, forming a standardized probability distribution of classification.
[0044] S250. Determine the corresponding road surface category based on the assignment probability, and generate an initial bird's-eye view road surface segmentation result containing multiple road surface types.
[0045] By combining the preset category determination rules, the probability of each pixel being assigned is analyzed. Usually, the road category with the highest probability of assignment is selected as the final category determination result for that pixel. By integrating the category determination results of all pixels, a complete bird's-eye view road segmentation mask map can be formed.
[0046] Since the segmentation result at this time has only undergone feature parsing, probability normalization and category determination, without any optimization processing, it inevitably has original defects such as pixel category misjudgment, irregular category boundaries and small area noise. Therefore, it is used as the initial bird's-eye view road segmentation result, that is, the coarse-grained segmentation result.
[0047] S260. The initial bird's-eye view road segmentation result is used as conditional information and input into the pre-trained diffusion model. The noise prediction is performed by combining the built-in denoising network of the diffusion model with the conditional information.
[0048] The initial bird's-eye view road segmentation results have inherent defects such as rough edges, local holes, discontinuous lanes, and sporadic missegments, which can be equivalently regarded as redundant noise and missing defects in the semantic structure of the image. The denoising network built into the diffusion model has the ability to learn the distribution of semantic image structure. Using the initial segmentation results as preconditional information, it can constrain the model to not deviate from the original road semantic framework. Relying on the guidance of the conditional information, the denoising network can accurately identify abnormal regions and structural defects in the segmentation image that do not conform to the road topology, and complete the feature prediction of inherent noise and structural defects in the image, providing a basis for judgment for subsequent rounds of denoising inference.
[0049] S270. Based on the preset sampling iteration mechanism, the denoising simulation is performed round by round with the initial bird's-eye view road segmentation results as constraints.
[0050] The refined repair of the diffusion model relies on multiple rounds of iterative deduction to gradually restore a reasonable semantic structure. The pre-defined sampling iteration mechanism specifies the operational logic and update rules of the deduction. Throughout the iterative process, the initial bird's-eye view road segmentation result is always used as a rigid constraint, ensuring that each round of deduction does not tamper with the road category attributes of the original pixels, and only corrects and completes edge contours, broken areas, and small holes within the established semantic range. By continuously correcting unreasonable segmentation areas through round-by-round denoising deduction, the continuity of road boundaries and topological integrity are gradually optimized, achieving a progressive optimization from coarse segmentation to a regular structure.
[0051] S280 adopts a lightweight sampling strategy to reduce the number of iteration steps, accelerates the reasoning process of denoising and inference, and generates and outputs a refined bird's-eye view road segmentation map after multiple rounds of denoising and inference convergence.
[0052] Conventional diffusion models involve numerous iterations and high inference latency, making it difficult to meet the computational power and latency requirements of real-time onboard detection in intelligent vehicles. A lightweight sampling strategy, by reasonably reducing unnecessary and redundant iterations, compresses the computational load without sacrificing repair accuracy, thus accelerating the inference process of the entire denoising and inference workflow. When multiple rounds of iterative inference reach convergence, the image's road surface boundaries, lane connectivity, and region segmentation regularity all tend to stabilize. At this point, the output segmentation map has undergone edge smoothing, defect completion, and noise removal, forming a refined bird's-eye view road surface segmentation map with continuous structure, accurate semantics, and regular boundaries, providing high-quality input for subsequent post-processing stages.
[0053] Optionally, before inputting the multi-view images into the pre-trained multimodal fusion bird's-eye view segmentation network, the following may also be included: A set of sample data consisting of multi-view images and LiDAR point cloud data is acquired simultaneously, and a set of annotation data uniquely corresponding to the current sample data set is obtained; the annotation data set includes two-dimensional segmentation annotation data corresponding to the multi-view images, and bird's-eye view segmentation annotation data corresponding to the LiDAR point cloud data; A set of mutually uniquely corresponding sample data and a set of labeled data are integrated into a set of training samples. For each set of training samples, a supervision signal corresponding to the current training sample set is generated using lidar point cloud data, two-dimensional segmentation labels and bird's-eye view segmentation labels. Input the multi-view images from the current training samples into the initial multimodal fusion bird's-eye view segmentation network to obtain the bird's-eye view segmentation prediction results; For each training sample, the loss value is calculated using the bird's-eye view segmentation prediction results, the supervision signal, and the preset loss calculation term; Backpropagation is performed based on the loss value to update the network weights and bias parameters of the initial multimodal fusion bird's-eye view segmentation network, and multiple sets of training samples are iteratively traversed for training. When the fluctuation range of the loss value after a preset number of consecutive iterations is less than a preset threshold, and the overlap between the bird's-eye view segmentation prediction result and the corresponding bird's-eye view segmentation label reaches a preset ratio, training is stopped, and the trained multimodal fusion bird's-eye view segmentation network is obtained. The initial multimodal fusion bird's-eye view segmentation network is a pre-built multimodal semantic segmentation basic network with an encoder and segmentation head structure, and configured with iteratively updatable network weights and bias parameters.
[0054] Synchronous acquisition of sample data is fundamental to network training. Multi-view images are acquired by vehicle surround-view cameras, while LiDAR point cloud data is simultaneously acquired by the vehicle-mounted LiDAR. Both acquisition times are kept consistent to ensure accurate data correspondence to the same vehicle's surrounding scene, avoiding sample mismatch caused by temporal discrepancies. The labeled data sets are pre-labeled offline and uniquely correspond to each set of sample data. Two-dimensional segmentation labeled data characterizes the road surface semantic category of each pixel in the multi-view images, while the bird's-eye view segmentation labeled data corresponds to the actual road surface segmentation from the bird's-eye view perspective reflected in the LiDAR point cloud data, providing a precise reference standard for network training.
[0055] The integration of a single set of sample data with its corresponding labeled data is intended to form independent training units, ensuring the integrity and independence of each set of training data and avoiding data interference between different samples. The generation of the supervision signal relies on the LiDAR point cloud data, 2D segmentation annotations, and bird's-eye view segmentation annotations within the sample data. These three elements work synergistically, utilizing the spatial depth information of the LiDAR point cloud to compensate for deficiencies in image features, and providing accurate semantic references through the 2D and bird's-eye view segmentation annotations. The generated supervision signal comprehensively reflects the true features and semantic distribution of the samples.
[0056] The initial multimodal fusion bird's-eye view segmentation network serves as the base network to be trained. It possesses basic capabilities in feature extraction, view transformation, and semantic segmentation. Its structure is consistent with the network used for road detection after training, except that the network weights and bias parameters have not been optimized. Multi-view images from the training samples are input into this initial network, which extracts features layer by layer according to the preset structure, performs view transformation and semantic classification, and finally outputs the bird's-eye view segmentation prediction result.
[0057] Loss calculation is a core prerequisite for updating network parameters. The pre-defined loss calculation terms cover various loss types, including segmentation loss and deep supervision loss, comprehensively measuring the deviation between the predicted results and the ground truth annotations. It focuses on both pixel-level semantic classification accuracy and the precision of feature extraction and viewpoint transformation. During the calculation, the supervision signal is used as the ground truth reference standard. The bird's-eye view segmentation prediction results are compared with the supervision signal, and the deviation between the two is quantified through the pre-defined loss calculation terms. The resulting loss value directly reflects the rationality of the current network parameters. That is, the larger the loss value, the greater the deviation between the predicted results and the reality, and the more the network parameters need further optimization; the smaller the loss value, the better the network training effect.
[0058] Backpropagation is the core mechanism for network parameter optimization. Based on the loss value calculated in the previous step, the backpropagation algorithm backtracks layer by layer, adjusting the weights and bias parameters of each layer to make the subsequent predictions more closely resemble the real situation reflected by the supervision signal. Iterating through multiple sets of training samples allows the network to fully learn the road features and semantic distributions in different scenarios, avoiding overfitting caused by a single sample and ensuring good generalization ability. Each iteration updates the parameters based on the loss value of the current sample. After multiple iterations, the network parameters gradually become more reasonable, and the prediction accuracy continuously improves.
[0059] When both conditions are met simultaneously, network training stops. At this point, the network has good feature extraction, viewpoint transformation, and semantic segmentation capabilities, which is the pre-trained multimodal fusion bird's-eye view segmentation network used for subsequent road detection. The initial multimodal fusion bird's-eye view segmentation network has two core structures: an encoder and a segmentation head. The encoder is responsible for multi-scale feature extraction and viewpoint transformation, while the segmentation head is responsible for semantic classification output. The two work together to complete the bird's-eye view segmentation prediction.
[0060] Example 3 Figure 3 This is a schematic diagram of a multimodal fusion road detection device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes: The multimodal feature extraction module 310 is used to perform preprocessing operations on the original multi-view images captured by the vehicle surround view camera to obtain preprocessed multi-view images. The multi-view images are then input into a pre-trained multimodal fusion bird's-eye view segmentation network to extract visual features at different scales layer by layer, thereby obtaining multi-scale image feature maps. The perspective transformation module 320 is used to perform spatial transformation and feature aggregation processing on multi-scale image feature maps from camera perspective to bird's-eye view based on preset camera intrinsic and extrinsic parameters, so as to obtain a globally unified bird's-eye view feature representation. The initial segmentation prediction module 330 is used to call the multimodal fusion bird's-eye view segmentation network to perform pixel-level multi-class road surface semantic classification prediction on the bird's-eye view feature representation, so as to obtain an initial bird's-eye view road surface segmentation result containing multiple road surface types. The diffusion model optimization module 340 is used to take the initial bird's-eye view road segmentation result as a conditional input to the pre-trained diffusion model, perform repair and optimization, and obtain a refined bird's-eye view road segmentation map. The post-processing module 350 is used to perform morphological denoising and temporal filtering post-processing operations on the refined bird's-eye view road segmentation map to obtain the final road detection result.
[0061] This invention, through multi-scale feature extraction from multi-view images combined with bird's-eye view spatial transformation and feature aggregation, effectively enhances the ability to represent road features at long distances and addresses the problem of insufficient detection accuracy at distant locations. It achieves fine-grained division of road areas based on multi-category pixel-level semantic segmentation, effectively mitigating the blurring of boundaries between roads and adjacent areas. Furthermore, by combining diffusion model generative repair and post-processing optimization, it significantly improves the ability to identify and adapt to various complex road surfaces such as asphalt, sand, snow, and sidewalks. Overall, it enhances the accuracy of long-distance road detection, the precision of boundary segmentation, and the generalization ability across multiple road conditions, meeting the high-precision and reliable detection requirements of intelligent driving environment perception.
[0062] Optionally, based on the above embodiments, the multimodal feature extraction module 310 may include: The multi-view raw image acquisition unit is used to simultaneously acquire images of the vehicle's surrounding environment through vehicle surround-view cameras deployed in multiple directions around the vehicle, thereby obtaining multi-view raw images covering the entire area around the vehicle at the same time. The distortion correction unit is used to perform distortion correction on the original multi-view images based on pre-calibrated camera parameters to obtain a corrected image that eliminates lens distortion. The image normalization unit is used to perform brightness equalization, color gamut conversion, channel conversion and pixel normalization on the corrected image to obtain a standardized image with unified imaging style and data dimension. The image scaling unit is used to perform uniform scaling on standardized images to obtain preprocessed multi-view images that meet the network input specifications.
[0063] Optionally, based on the above embodiments, the multimodal feature extraction module 310 may further include: The encoder input unit is used to input the preprocessed multi-view images into the pre-trained multimodal fusion bird's-eye view segmentation network, which is built into the Transformer structure bird's-eye view encoder for feature extraction and spatial mapping. The feature extraction unit is used to extract multi-scale features from images from each viewpoint through the backbone network mounted on the bird's-eye view encoder, so as to obtain two-dimensional image features from each camera viewpoint. The feature fusion unit is used to perform multi-scale information fusion on each two-dimensional image feature to obtain a multi-scale image feature map containing multi-view texture and semantic information.
[0064] Optionally, based on the above embodiments, the viewpoint conversion module 320 may include: The mapping relationship determination unit is used to establish a spatial mapping relationship between the image space of each camera view and the vehicle bird's-eye view space based on preset camera intrinsic and extrinsic parameters. The projection conversion unit is used to adaptively spatially sample the multi-scale image feature map using a deformable attention mechanism, and project and convert the two-dimensional image features from each camera viewpoint to a unified vehicle bird's-eye view space. The bird's-eye view feature representation generation unit is used to fuse and aggregate the projection features corresponding to all camera viewpoints in the vehicle's bird's-eye view space, integrate multi-directional spatial feature information, and generate a unified bird's-eye view feature representation with global spatial position representation capabilities.
[0065] Optionally, based on the above embodiments, the initial segmentation prediction module 330 may include: The feature channel establishment unit is used to input the bird's-eye view feature representation into the segmentation head of the multimodal fusion bird's-eye view segmentation network, and use the built-in independent Transformer decoder and multi-scale convolutional layer to perform semantic parsing and feature refinement on the bird's-eye view feature representation, and output feature channels corresponding to the number of road surface categories. The attribution probability generation unit is used to perform probability normalization operation on the feature channel to obtain the attribution probability of each pixel for each type of road surface. The initial road surface segmentation result generation unit is used to determine the corresponding road surface category based on the assignment probability and generate an initial bird's-eye view road surface segmentation result containing multiple road surface types.
[0066] Optionally, based on the above embodiments, the diffusion model optimization module 340 may include: The noise prediction unit is used to input the initial bird's-eye view road segmentation result as conditional information into the pre-trained diffusion model, and use the built-in denoising network of the diffusion model in combination with the conditional information to perform noise prediction. The denoising and derivation unit is used to perform denoising and derivation round by round based on the preset sampling and iteration mechanism and the initial bird's-eye view road segmentation results as constraints; The inference convergence unit is used to reduce the number of iteration steps by adopting a lightweight sampling strategy, thereby accelerating the inference process of denoising inference. After multiple rounds of denoising inference convergence, it generates and outputs a refined bird's-eye view road segmentation map.
[0067] Optionally, based on the above embodiments, it may further include: a network pre-training unit, used to simultaneously collect a set of sample data consisting of multi-view images and LiDAR point cloud data before inputting multi-view images into a pre-trained multimodal fusion bird's-eye view segmentation network, and obtain a set of labeled data uniquely corresponding to the current sample data set; the labeled data set includes two-dimensional segmentation labeled data corresponding to the multi-view images, and bird's-eye view segmentation labeled data corresponding to the LiDAR point cloud data; A set of mutually uniquely corresponding sample data and a set of labeled data are integrated into a set of training samples. For each set of training samples, a supervision signal corresponding to the current training sample set is generated using lidar point cloud data, two-dimensional segmentation labels and bird's-eye view segmentation labels. Input the multi-view images from the current training samples into the initial multimodal fusion bird's-eye view segmentation network to obtain the bird's-eye view segmentation prediction results; For each training sample, the loss value is calculated using the bird's-eye view segmentation prediction results, the supervision signal, and the preset loss calculation term; Backpropagation is performed based on the loss value to update the network weights and bias parameters of the initial multimodal fusion bird's-eye view segmentation network, and multiple sets of training samples are iteratively traversed for training. When the fluctuation range of the loss value after a preset number of consecutive iterations is less than a preset threshold, and the overlap between the bird's-eye view segmentation prediction result and the corresponding bird's-eye view segmentation label reaches a preset ratio, training is stopped, and the trained multimodal fusion bird's-eye view segmentation network is obtained. The initial multimodal fusion bird's-eye view segmentation network is a pre-built multimodal semantic segmentation basic network with an encoder and segmentation head structure, and configured with iteratively updatable network weights and bias parameters.
[0068] The multimodal fusion road detection device provided in this embodiment of the invention can execute the multimodal fusion road detection method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.
[0069] Example 4 Figure 4 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.
[0070] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0071] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0072] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as a multimodal fusion road detection method.
[0073] That is: perform preprocessing on the original multi-view images captured by the vehicle surround view camera to obtain preprocessed multi-view images, input the multi-view images into the pre-trained multimodal fusion bird's-eye view segmentation network, extract visual features at different scales layer by layer, and obtain multi-scale image feature maps. Based on preset camera intrinsic and extrinsic parameters, spatial transformation and feature aggregation processing are performed on multi-scale image feature maps from camera viewpoint to bird's-eye viewpoint to obtain a globally unified bird's-eye viewpoint feature representation. The multimodal fusion bird's-eye view segmentation network is invoked to perform pixel-level multi-class road surface semantic classification prediction on the bird's-eye view feature representation, resulting in an initial bird's-eye view road surface segmentation result containing multiple road surface types. The initial bird's-eye view road segmentation result is used as a conditional input to the pre-trained diffusion model for repair and optimization, resulting in a refined bird's-eye view road segmentation map. The detailed bird's-eye view road segmentation map is subjected to morphological denoising and temporal filtering post-processing operations to obtain the final road detection results.
[0074] In some embodiments, a multimodal fusion road detection method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the multimodal fusion road detection method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform a multimodal fusion road detection method by any other suitable means (e.g., by means of firmware).
[0075] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0076] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0077] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0078] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0079] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.
[0080] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.
[0081] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.
[0082] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A multimodal fusion road detection method, characterized in that, Applied to smart vehicles equipped with surround-view cameras, including: Preprocessing is performed on the original multi-view images captured by the vehicle surround view camera to obtain preprocessed multi-view images. The multi-view images are then input into a pre-trained multimodal fusion bird's-eye view segmentation network to extract visual features at different scales layer by layer, resulting in multi-scale image feature maps. Based on preset camera intrinsic and extrinsic parameters, spatial transformation and feature aggregation processing are performed on multi-scale image feature maps from camera viewpoint to bird's-eye viewpoint to obtain a globally unified bird's-eye viewpoint feature representation. The multimodal fusion bird's-eye view segmentation network is invoked to perform pixel-level multi-class road surface semantic classification prediction on the bird's-eye view feature representation, resulting in an initial bird's-eye view road surface segmentation result containing multiple road surface types. The initial bird's-eye view road segmentation result is used as a conditional input to the pre-trained diffusion model for repair and optimization, resulting in a refined bird's-eye view road segmentation map. The detailed bird's-eye view road segmentation map is subjected to morphological denoising and temporal filtering post-processing operations to obtain the final road detection results.
2. The method according to claim 1, characterized in that, Preprocessing is performed on the raw multi-view images captured by the vehicle surround-view camera to obtain preprocessed multi-view images, including: By simultaneously acquiring images of the vehicle's surrounding environment through multi-angle surround-view cameras deployed around the vehicle, original images covering the entire area around the vehicle at the same time can be obtained. Based on pre-calibrated camera parameters, distortion correction is performed on the original images from multiple perspectives to obtain corrected images that eliminate lens distortion; The corrected image is subjected to brightness equalization, color gamut conversion, channel conversion and pixel normalization to obtain a standardized image with unified imaging style and data dimension. Standardized images are scaled down to obtain preprocessed multi-view images that meet network input specifications.
3. The method according to claim 1, characterized in that, Multi-view images are input into a pre-trained multimodal fusion bird's-eye view segmentation network, and visual features at different scales are extracted layer by layer to obtain multi-scale image feature maps, including: The preprocessed multi-view images are input into the pre-trained multimodal fusion bird's-eye view segmentation network, which has a built-in Transformer structure bird's-eye view encoder for feature extraction and spatial mapping. The backbone network of the bird's-eye view encoder is used to extract multi-scale features from images from various viewpoints to obtain two-dimensional image features from each camera viewpoint. Multi-scale information fusion is performed on the features of each two-dimensional image to obtain a multi-scale image feature map containing multi-view texture and semantic information.
4. The method according to claim 1, characterized in that, Based on preset camera intrinsic and extrinsic parameters, spatial transformation and feature aggregation processing are performed on multi-scale image feature maps from camera viewpoint to bird's-eye viewpoint to obtain a globally unified bird's-eye viewpoint feature representation, including: Based on preset camera intrinsic and extrinsic parameters, a spatial mapping relationship is established between the image space of each camera viewpoint and the vehicle's bird's-eye view space. A deformable attention mechanism is used to adaptively sample the multi-scale image feature map, and the two-dimensional image features from each camera viewpoint are projected and transformed into a unified vehicle bird's-eye view space. In the vehicle's bird's-eye view space, the projection features corresponding to all camera viewpoints are fused and aggregated to integrate multi-directional spatial feature information and generate a unified bird's-eye view feature representation with global spatial position representation capabilities.
5. The method according to claim 1, characterized in that, The multimodal fusion bird's-eye view segmentation network is invoked to perform pixel-level multi-class road surface semantic classification prediction on the bird's-eye view feature representation, resulting in an initial bird's-eye view road surface segmentation result containing multiple road surface types, including: The bird's-eye view feature representation is input into the segmentation head of the multimodal fusion bird's-eye view segmentation network. Using the built-in independent Transformer decoder and multi-scale convolutional layers, the bird's-eye view feature representation is semantically parsed and its features are refined, and the feature channels corresponding to the number of road surface categories are output. Perform probability normalization on the feature channels to obtain the probability of each pixel belonging to each type of road surface; The road surface category is determined based on the probability of its classification, and an initial bird's-eye view road surface segmentation result containing multiple road surface types is generated.
6. The method according to claim 1, characterized in that, The initial bird's-eye view road segmentation results are used as conditional input to a pre-trained diffusion model for repair and optimization, resulting in a refined bird's-eye view road segmentation map, including: The initial bird's-eye view road segmentation result is used as conditional information and input into a pre-trained diffusion model. The built-in denoising network of the diffusion model is used in conjunction with the conditional information to predict noise. Based on the preset sampling iteration mechanism, the denoising derivation is performed round by round with the initial bird's-eye view road segmentation results as constraints; A lightweight sampling strategy is adopted to reduce the number of iteration steps and accelerate the inference process of denoising. After multiple rounds of denoising and inference convergence, a refined bird's-eye view road segmentation map is generated and output.
7. The method according to claim 1, characterized in that, Before inputting multi-view images into a pre-trained multimodal fusion bird's-eye view segmentation network, the following steps are also included: A set of sample data consisting of multi-view images and LiDAR point cloud data is acquired simultaneously, and a set of annotation data uniquely corresponding to the current sample data set is obtained; the annotation data set includes two-dimensional segmentation annotation data corresponding to the multi-view images, and bird's-eye view segmentation annotation data corresponding to the LiDAR point cloud data; A set of mutually uniquely corresponding sample data and a set of labeled data are integrated into a set of training samples. For each set of training samples, a supervision signal corresponding to the current training sample set is generated using lidar point cloud data, two-dimensional segmentation labels and bird's-eye view segmentation labels. Input the multi-view images from the current training samples into the initial multimodal fusion bird's-eye view segmentation network to obtain the bird's-eye view segmentation prediction results; For each training sample, the loss value is calculated using the bird's-eye view segmentation prediction results, the supervision signal, and the preset loss calculation term; Backpropagation is performed based on the loss value to update the network weights and bias parameters of the initial multimodal fusion bird's-eye view segmentation network, and training is performed iteratively by traversing multiple sets of training samples. When the fluctuation range of the loss value after a preset number of consecutive iterations is less than a preset threshold, and the overlap between the bird's-eye view segmentation prediction result and the corresponding bird's-eye view segmentation label reaches a preset ratio, training is stopped, and the trained multimodal fusion bird's-eye view segmentation network is obtained. The initial multimodal fusion bird's-eye view segmentation network is a pre-built multimodal semantic segmentation basic network with an encoder and segmentation head structure, and configured with iteratively updatable network weights and bias parameters.
8. A multimodal fusion road detection device, characterized in that, The device includes: The multimodal feature extraction module is used to perform preprocessing operations on the original multi-view images captured by the vehicle surround view camera to obtain preprocessed multi-view images. The multi-view images are then input into a pre-trained multimodal fusion bird's-eye view segmentation network to extract visual features at different scales layer by layer, resulting in multi-scale image feature maps. The perspective transformation module is used to perform spatial transformation and feature aggregation processing on multi-scale image feature maps from camera perspective to bird's-eye view based on preset camera intrinsic and extrinsic parameters, so as to obtain a globally unified bird's-eye view feature representation. The initial segmentation prediction module is used to call the multimodal fusion bird's-eye view segmentation network to perform pixel-level multi-class road surface semantic classification prediction on the bird's-eye view feature representation, so as to obtain the initial bird's-eye view road surface segmentation result containing multiple road surface types. The diffusion model optimization module is used to take the initial bird's-eye view road segmentation result as a conditional input to the pre-trained diffusion model, perform repair and optimization, and obtain a refined bird's-eye view road segmentation map. The post-processing module performs morphological denoising and temporal filtering on the refined bird's-eye view road segmentation map to obtain the final road detection result.
9. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform a multimodal fusion road detection method according to any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute and implement the multimodal fusion road detection method according to any one of claims 1-7.