A dual-geometry information perception domain generalization free space detection method

Abstract

This invention discloses a domain-generalized free space detection method with dual geometric information perception. The method includes: acquiring a training image set and training a model based on the training image set to obtain a generalized model for free space detection; acquiring images of different types of road scenes to be processed, inputting the images of different types of road scenes to be processed into the generalized model for free space detection, and outputting a target free space image. This invention achieves effective fusion of multimodal information by embedding surface normal vector information into RGB features, enhances the stability of domain-invariant features through information interaction and effective aggregation between features of different scales, and further improves the model's sensitivity and capture ability to edge information through an RGB-SN edge-aware learning strategy, thereby improving the model's generalization ability to different types of free space scenes.

Description

Technical Field

[0001] This invention relates to the field of image semantic segmentation technology, and in particular to a domain generalization free space detection method based on dual geometric information perception. Background Technology

[0002] In recent years, significant advancements in computer hardware and computer vision technologies have propelled the rise of autonomous driving in the field of intelligent vehicles. Autonomous driving systems primarily consist of four core components: environmental perception, localization and path planning, behavioral decision-making, and control execution. Environmental perception, as the cornerstone of the entire system, provides robust data support for subsequent processes, particularly in identifying safe traversable areas for autonomous vehicles. This functionality is closely related to free space detection technology. Free space detection, also known as traversable area detection, is essentially a binary classification semantic segmentation task. Its goal is to accurately divide images acquired by cameras into traversable and impassable areas, thereby providing crucial information for the safe travel of autonomous vehicles. Currently, mainstream free space detection methods are mainly based on deep learning, employing supervised learning paradigms and unsupervised domain adaptation techniques. However, both of these methods experience significant performance degradation when testing out-of-distribution scene data. Domain generalization is one effective method to address this issue. By employing certain strategies, it effectively captures domain-invariant features from one or more source domain datasets, thereby ensuring that the model maintains good test performance in multiple target domains with data distributions differing from the source domain.

[0003] However, existing methods are mainly designed for structured urban scenes and aim to handle multi-category semantic segmentation problems. They cannot be directly and effectively applied to domain-generalized free space detection tasks. While segmenting passable areas, other different categories, such as sky, vehicles, and pedestrians, also need to be segmented. During training, the model needs to learn the semantic information of various categories and does not pay special attention to the unique features of free space categories. Applying existing methods to free space detection tasks often leads to poor performance. Moreover, they are mainly designed for urban road scenes. In the real world, based on the surface features of free space, free space environments can be categorized into structured urban scenes and unstructured off-road scenes. For structured urban scenes, free space typically refers to relatively flat roads with lane markings and clear boundaries, such as highways and well-planned, structurally defined roads in towns. For unstructured off-road scenes, free space often refers to roads without lane markings and clear boundaries, such as rural trails, or areas where autonomous vehicles can safely navigate complex terrains like low grass, sand, and mud. Existing domain-generalized semantic segmentation methods do not fully consider the diverse types of areas accessible to autonomous vehicles, exploring only structured urban scenes while neglecting unstructured off-road scenes. Furthermore, they primarily rely on RGB images and their corresponding label data. Different types of free space scenes exhibit significant visual differences in RGB images, meaning that the domain-invariant features contained in RGB images are quite limited. Therefore, relying solely on RGB images makes it difficult for models to fully learn the domain-invariant features of free space, resulting in poor generalization performance across different types of free space scenes.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide a domain generalization free space detection method based on dual geometric information perception, which aims to solve the problem that the existing technology does not accurately distinguish between passable and impassable regions in RGB image pixels, and has low sensitivity to edge information of passable regions, resulting in poor generalization performance in different types of free space scenes.

[0006] To achieve the above objectives, the present invention provides a domain generalization free space detection method based on dual geometric information perception, the method comprising the following steps:

[0007] A training image set is acquired, and a model is trained based on the training image set to obtain a generalized model for free space detection. The training image set includes multiple synthetic images of structured urban road scenes, and the synthetic images of structured urban road scenes include color source domain images and corresponding surface normal vector source domain images.

[0008] Acquire different types of road scene images to be processed, input the different types of road scene images to be processed into the generalization model of free space detection, and output the target free space image;

[0009] The different types of road scene images to be processed include synthetic structured urban scene images, real structured urban scene images, and real unstructured off-road scene images.

[0010] Optionally, the domain-generalized free space detection method with dual geometric information perception, wherein acquiring a training image set and training a model based on the training image set to obtain a generalized model for free space detection specifically includes:

[0011] A training image set is obtained, a free space detection training network model is created, and a synthetic image of a structured urban road scene from the training image set is input into the free space detection training network model.

[0012] The color source domain image is style-processed to obtain a target color image. Feature extraction is performed on the target color image and the surface normal vector source domain image to obtain color features and surface normal vector features at multiple stages.

[0013] Attention maps are extracted for the color features and surface normal vector features at each stage to obtain corresponding spatial attention maps and channel attention maps. Based on the spatial attention maps and channel attention maps of the same stage, information fusion processing is performed on the color features and surface normal vector features to obtain a first fused feature, a second fused feature, a third fused feature, and a fourth fused feature.

[0014] The second fusion feature, the third fusion feature, and the fourth fusion feature are calibrated and spliced ​​to obtain a multi-scale aggregated feature. A predicted semantic segmentation map is obtained based on the multi-scale aggregated feature. Cross-entropy loss and style consistency loss are calculated on the predicted semantic segmentation map to obtain semantic segmentation loss value and style consistency loss value.

[0015] The first fusion feature and the second fusion feature are concatenated to obtain edge features. A predicted edge map is obtained based on the edge features. The predicted edge map is then subjected to binary weighted cross-entropy calculation to obtain the edge segmentation loss value.

[0016] The overall loss value is obtained based on the semantic segmentation loss value, the style consistency loss value, and the edge segmentation loss value, and the parameters of the free space detection training network model are corrected based on the overall loss value.

[0017] The next synthesized image of the structured urban road scene is input into the free space detection training network model until the training status of the free space detection training network model meets the preset conditions, thus obtaining a trained generalized model of free space detection.

[0018] Optionally, the dual geometric information-aware domain generalization free space detection method, wherein the step of performing style processing on the color source domain image to obtain a target color image, and extracting features from the target color image and the surface normal vector source domain image respectively to obtain color features and surface normal vector features at multiple stages, specifically includes:

[0019] The color source image is subjected to image style diversification processing to obtain a target color image, wherein the target color image includes a stylized image and the original image;

[0020] In the target feature extraction stage, feature extraction is performed on the target color image and the surface normal vector source domain image respectively to obtain target color features and target surface normal vector features;

[0021] The target feature extraction stage includes a first feature extraction stage, a second feature extraction stage, a third feature extraction stage, and a fourth feature extraction stage. The target color features include color features from four stages, and the target surface normal vector features include surface normal vector features from four stages.

[0022] Optionally, the domain generalization free space detection method with dual geometric information perception, wherein the step of extracting attention maps from the color features and surface normal vector features at each stage to obtain corresponding spatial attention maps and channel attention maps, specifically includes:

[0023] The target color features are convolved to obtain the first convolution feature, and attention maps are extracted from the first convolution feature according to the first formula and the second formula to obtain the first target spatial attention map and the first target channel attention map.

[0024] The target surface normal vector features are convolved to obtain the second convolution feature. Attention is extracted from the second convolution feature according to the third and fourth formulas to obtain the second target spatial attention map and the second target channel attention map.

[0025] The first target spatial attention map and the second target spatial attention map each include four spatial attention maps, and the first target channel attention map and the second target channel attention map each include four channel attention maps.

[0026] Optionally, in the dual geometric information-aware domain generalization free space detection method, the first formula is:

[0027] ;

[0028] The second formula is:

[0029] ;

[0030] The third formula is:

[0031] ;

[0032] The fourth formula is:

[0033] ;

[0034] in, for The corresponding spatial attention map, for The corresponding channel attention map, for The corresponding spatial attention map, for The corresponding channel attention map, for Sigmoid Activation function It is a 1×1 convolutional layer. For the global max pooling operation in the spatial dimension, This is a global max-pooling operation at the channel level. For the first The color features of the stage are processed by a 1×1 convolutional layer to generate corresponding convolutional features. For the first The surface normal vector features of the stage are processed by a 1×1 convolutional layer to generate the corresponding convolutional features. For the number of stages, .

[0035] Optionally, the domain generalization free space detection method with dual geometric information perception, wherein the step of performing information fusion processing on the color features and the surface normal vector features based on the spatial attention map and the channel attention map of the same stage to obtain a first fused feature, a second fused feature, a third fused feature, and a fourth fused feature specifically includes:

[0036] The first convolutional feature is multiplied element-wise with the second target spatial attention map to obtain the first target semantic feature, and the first convolutional feature is multiplied element-wise with the second target channel attention map to obtain the second target semantic feature.

[0037] The second convolutional feature is multiplied element-wise with the first target spatial attention map to obtain the first target geometric feature, and the second convolutional feature is multiplied element-wise with the first target channel attention map to obtain the second target geometric feature.

[0038] The first target semantic feature, the second target semantic feature, the first target geometric feature, and the second target geometric feature are added element by element to obtain the corresponding first fusion feature, second fusion feature, third fusion feature, and fourth fusion feature;

[0039] The first target semantic feature, the second target semantic feature, the first target geometric feature, and the second target geometric feature each include four corresponding features.

[0040] Optionally, the domain generalization free space detection method with dual geometric information perception, wherein the calibration and splicing of the second fusion feature, the third fusion feature, and the fourth fusion feature to obtain multi-scale aggregated features specifically includes:

[0041] The spatial dimensions of the second fusion feature are processed, and convolution operations are performed on the processed second fusion feature, the third fusion feature, and the fourth fusion feature to obtain the second target fusion feature, the third target fusion feature, and the fourth target fusion feature.

[0042] Attention maps are extracted from the second target fusion feature to obtain a first spatial attention map and a first channel attention map; attention maps are extracted from the third target fusion feature to obtain a second spatial attention map and a second channel attention map; and attention maps are extracted from the fourth target fusion feature to obtain a third spatial attention map and a third channel attention map.

[0043] The first spatial attention map and the second spatial attention map are multiplied element-wise to obtain the first calibrated attention map, and the second spatial attention map and the third spatial attention map are multiplied element-wise to obtain the second calibrated attention map.

[0044] The first channel attention map and the second channel attention map are multiplied element-wise to obtain the third calibration attention map. The second channel attention map and the third channel attention map are multiplied element-wise to obtain the fourth calibration attention map.

[0045] The second target fusion feature is multiplied element-wise with the first calibration attention map and the third calibration attention map to obtain the first calibration feature and the second calibration feature, respectively. The fourth target fusion feature is multiplied element-wise with the second calibration attention map and the fourth calibration attention map to obtain the third calibration feature and the fourth calibration feature, respectively.

[0046] The first calibration feature and the second calibration feature are concatenated by channels to obtain the first concatenated feature. The third calibration feature and the fourth calibration feature are concatenated by channels to obtain the second concatenated feature. The first concatenated feature, the second concatenated feature and the third target fusion feature are concatenated by channels to obtain the multi-scale aggregated feature.

[0047] Optionally, the dual-geometric information-aware domain generalization free space detection method, wherein obtaining a predicted semantic segmentation map based on the multi-scale aggregated features, and calculating cross-entropy loss and style consistency loss on the predicted semantic segmentation map respectively to obtain semantic segmentation loss values ​​and style consistency loss values, specifically includes:

[0048] The multi-scale aggregated features are subjected to pore-diffused spatial pyramid pooling to obtain target multi-scale aggregated features, and a predicted semantic segmentation map is obtained based on the target multi-scale aggregated features.

[0049] The cross-entropy loss is calculated on the predicted semantic segmentation graph according to the fifth formula to obtain the semantic segmentation loss value, and the style consistency loss is calculated on the predicted semantic segmentation graph according to the sixth formula to obtain the style consistency loss value.

[0050] The fifth formula is:

[0051] ;

[0052] The sixth formula is:

[0053] ;

[0054] in, The semantic segmentation loss function is... Let style consistency loss function be used. The length of the semantic label graph. The width of the semantic label graph. For the semantic label graph, the first pixel value, To predict the first semantic segmentation graph Predicted values ​​per pixel, Let KL divergence be the KL divergence. To predict semantic segmentation maps, Predicting the fusion features of the original image and the surface normal vector map. For predicting the fusion features of stylized images and surface normal maps, for and The mean.

[0055] Optionally, the dual geometric information-aware domain generalization free space detection method, wherein the step of concatenating the first fused feature and the second fused feature to obtain edge features, obtaining a predicted edge map based on the edge features, and performing binary weighted cross-entropy calculation on the predicted edge map to obtain an edge segmentation loss value, specifically includes:

[0056] The first fusion feature is convolved to obtain the first target fusion feature. The second fusion feature is convolved and upsampled to obtain the sampled feature. The first target fusion feature and the sampled feature are concatenated by channels to obtain the edge feature.

[0057] Based on the edge features, a predicted edge map is obtained, and the predicted edge map is subjected to binary weighted cross-entropy calculation according to the seventh formula to obtain the edge segmentation loss value.

[0058] The seventh formula is:

[0059] ;

[0060] The calculation formula is: , ;

[0061] in, The edge segmentation loss function is... This represents the total number of pixels. For category loss weights, For the edge label pixel value, To predict edge maps The Predicted values ​​per pixel, For the edge label graph, the first The pixel label value belongs to the category The number of pixels, Category.

[0062] In this invention, a training image set is acquired, and a model is trained based on the training image set to obtain a generalized model for free space detection. The training image set includes multiple synthetic images of structured urban road scenes, each including a color source domain image and a corresponding surface normal vector source domain image. Different types of road scene images to be processed are acquired and input into the generalized model for free space detection, outputting a target free space image. These different types of road scene images include synthetic structured urban scene images, real structured urban scene images, and real unstructured off-road scene images. This invention achieves effective fusion of multimodal information by embedding surface normal vector information into RGB features. Through information interaction and effective aggregation between features of different scales, the stability between RGB-SN features is enhanced. Furthermore, an RGB-SN edge perception learning strategy further improves the model's sensitivity and capture ability to edge information, thereby enhancing the generalization ability of free space scenes. The generated segmentation results can divide RGB image pixels into passable and impassable areas, providing necessary perception information for autonomous driving technology decisions. Attached Figure Description

[0063] Figure 1 These are schematic diagrams of RGB images of different types of free space scenes;

[0064] Figure 2 This is a flowchart of a preferred embodiment of the domain generalization free space detection method with dual geometric information perception in this invention;

[0065] Figure 3 This is a schematic diagram of the network architecture corresponding to the domain generalization free space detection method with dual geometric information perception in this invention;

[0066] Figure 4 This is a schematic diagram of RGB and SN images of different datasets in an embodiment of the present invention;

[0067] Figure 5 This is a schematic diagram of a robust surface normal vector embedding module according to a preferred embodiment of the present invention;

[0068] Figure 6 This is a schematic diagram of the cross-scale feature robustness enhancement module of a preferred embodiment of the present invention. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0070] It should be noted that if the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0071] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0072] Existing domain generalization methods are primarily designed for structured urban scenes, aiming to handle multi-class semantic segmentation problems. However, they cannot be directly and effectively applied to domain generalization free space detection tasks. Firstly, they mainly target multi-class semantic segmentation, requiring segmentation of other different categories such as sky, vehicles, and pedestrians while segmenting passable areas. During training, the model needs to learn semantic information from various categories, without paying additional attention to the unique features of free space categories. Therefore, direct application to free space detection tasks often leads to poor performance. Secondly, existing methods are mainly designed for urban road scenes. In the real world, based on the surface features of free space, free space environments can be divided into structured urban scenes and unstructured off-road scenes, such as... Figure 1 As shown, for Figure 1 In the structured urban scenario of (a), free space typically refers to relatively flat roads with lane lines and clear boundaries, such as highways and well-planned, structurally defined roads in towns; for Figure 1 In unstructured off-road scenarios (b), free space often refers to roads without lane markings and clear boundaries, such as rural trails, or areas where autonomous vehicles can safely navigate complex terrains like low grass, sand, and mud. However, existing methods do not fully consider the diversity of traversable areas for autonomous vehicles, exploring only structured urban scenarios while neglecting unstructured off-road scenarios. Finally, existing methods primarily rely on RGB images and their corresponding label data; different types of free space scenarios exhibit significant visual differences in RGB images, such as... Figure 1As shown in (a), in structured urban scenes, the free space surfaces of synthetic scenes and real scenes exhibit significant changes in visual features such as texture and color. In addition, in real scenes, the free space surface types of structured scenes and unstructured scenes differ significantly, and these differences indicate that the domain-invariant features contained in RGB images are quite limited. Therefore, relying solely on RGB images, the model cannot fully learn the domain-invariant features of free space, which leads to the model's difficulty in achieving good generalization effects in different types of free space scenes.

[0073] This invention compares RGB images with surface normal vector images and yields the following important results: First, the free space surface exhibits significant appearance differences in RGB images across different datasets, but the surface normal vector information remains similar and is less susceptible to interference from factors such as exposure, shadows, and puddles, indicating that surface normal vector information is more domain-invariant and robust. Second, the edges of free space often extend forward or meander, reflecting that edge information can serve as another domain-invariant feature of free space. Finally, for datasets of unstructured off-road scenarios, the edges of free space are often difficult to discern, indicating that the corresponding network model needs to focus more on learning edge information. A domain-generalized free space detection method based on dual geometric information perception is proposed. Considering the domain invariance and robustness of SN (Surface Normal), firstly, RSNE (Robust Surface Normal Embedding Module) is proposed, embedding SN information into RGB features and combining RGB and SN information to learn the domain-invariant features of free space. Secondly, CFRE (Cross Feature Robustness Enhancement) is designed. The module (cross-scale feature robustness enhancement module) further enhances the stability of RGB-SN features through the interaction and fusion of shallow edge information and deep semantic features. Finally, given the similarity of free space edge information, EAL (Edge-Aware Learning Strategy, RGB-SN edge-aware learning strategy) is adopted to explicitly enhance the model's sensitivity to and ability to capture edge information.

[0074] The preferred embodiment of the present invention describes a domain generalization free space detection method based on dual geometric information perception, such as... Figure 2 As shown, the domain generalization free space detection method based on dual geometric information perception includes the following steps:

[0075] Step S10: Obtain a training image set and train the model based on the training image set to obtain a generalized model for free space detection. The training image set includes multiple synthetic images of structured urban road scenes, and the synthetic images of structured urban road scenes include color source domain images and corresponding surface normal vector source domain images.

[0076] Step S10 includes:

[0077] Step S11: Obtain a training image set, create a free space detection training network model, and input a synthetic image of a structured urban road scene from the training image set into the free space detection training network model.

[0078] Step S12: Perform style processing on the color source domain image to obtain the target color image; extract features from the target color image and the surface normal vector source domain image respectively to obtain color features and surface normal vector features at multiple stages.

[0079] Step S13: Extract attention maps for the color features and surface normal vector features of each stage to obtain corresponding spatial attention maps and channel attention maps, and fuse the color features and surface normal vector features based on the spatial attention maps and channel attention maps of the same stage to obtain a first fused feature, a second fused feature, a third fused feature and a fourth fused feature;

[0080] Step S14: The second fusion feature, the third fusion feature and the fourth fusion feature are calibrated and spliced ​​to obtain a multi-scale aggregated feature. A predicted semantic segmentation map is obtained based on the multi-scale aggregated feature. Cross-entropy loss and style consistency loss are calculated on the predicted semantic segmentation map to obtain semantic segmentation loss value and style consistency loss value.

[0081] Step S15: The first fusion feature and the second fusion feature are concatenated to obtain edge features. A predicted edge map is obtained based on the edge features, and a binary weighted cross-entropy is calculated on the predicted edge map to obtain the edge segmentation loss value.

[0082] Step S16: Obtain the overall loss value based on the semantic segmentation loss value, the style consistency loss value, and the edge segmentation loss value, and correct the parameters of the free space detection training network model based on the overall loss value;

[0083] Step S17: Input the next synthesized image of the structured urban road scene into the free space detection training network model until the training status of the free space detection training network model meets the preset conditions, and obtain the trained free space detection generalization model.

[0084] Specifically, in this embodiment of the invention, a free space detection training network model is created based on a domain generalization free space detection method with dual geometric information perception, such as... Figure 3 As shown, the free space detection training network model mainly includes RSNE (Robust Surface Normal Embedding Module), CFRE (Cross-Scale Feature Robustness Enhancement Module), and EAL (RGB-SN Edge Awareness Learning Strategy). During the training phase of the free space detection training network model, a training image set needs to be acquired for model training. This training image set includes different types of road scene images, and the road scene images include color source domain images, i.e. , and the corresponding surface normal vector source domain image, i.e. After that and The input is fed into the free space detection training network model. After input, to reduce the impact of RGB image style on the domain generalization free space detection task, based on the publicly available method, the domain generalization semantic segmentation algorithm SHADE is used to perform image style diversification in the shallow layer of the RGB encoder to obtain the target color image. The target color image includes a stylized image and the original image, and style consistency loss is used in the prediction layer of the decoder to align the style of the target color image. Then, the target color image and the surface normal vector source domain image are respectively processed through a backbone network with the same structure to extract features, thereby obtaining corresponding target color features, i.e., RGB features, at different extraction stages. This indicates that the target surface normal vector features, i.e., SN features, are represented by... It means that, among them, This represents the number of stages in the feature extraction phase. .

[0085] In the field of domain-generalized semantic segmentation, most mainstream algorithms currently extract multi-class domain-invariant features from RGB images, without considering the unique attributes and type diversity of free-space classes; for example... Figure 4 As shown, Figure 4The synthetic scene datasets include R2D (The Ready-to-Drive Dataset), SYNTHIA (The SYNTHIA-San Francisco Dataset), and SELMA (The SEmantic Large-Scale Multimodal Acquisitions Dataset), while the real-world scene datasets include KIYYI-Road (The KITTI-Road Dataset, KITTI, Karlsruhe Institute of Technology and Toyota Technological Institute) and ORFD (The Off-Road Freespace Detection Dataset). ORFD includes ORFD1 and ORFD2, representing paved roads and unpaved roads (dirt roads, sand roads, etc.), respectively. Compared to RGB images, the surface normal vector information in SN (Surface Normal Vector) images better characterizes the coplanarity of free space and is more robust to interference factors such as overexposure, shadow occlusion, and puddle reflections. Furthermore, RGB and SN images have some complementarity; for example, free space and planar lawns may be difficult to distinguish in SN images, but in RGB images, they are clearly distinguishable. The differences in appearance are quite obvious; therefore, this invention focuses on the complementarity between semantic (RGB modal) features and geometric (SN modal) features. Through the RSNE module (Robust Surface Normal Embedding module), it aims to promote the cross-synergistic effect between RGB and SN modalities to improve the stability of free space features in different scenarios. Channel attention and spatial attention mechanisms can capture key contextual information in feature maps in different dimensions. This RSNE module uses the contextual information of one modality to guide the feature learning of another modality, thereby achieving information complementarity between modalities. The specific structure of the RSNE module is as follows: Figure 5 As shown, in the RSNE module, and The source domain images of the target color image and the target surface normal vector source domain image are respectively represented in the th... The RGB and SN features extracted in each stage; and The first convolutional feature is generated by passing each feature through a 1×1 convolutional layer. Second convolutional features, i.e. Subsequently, the RSNE module uses SA (Spatial Attention) and CA (Channel Attention) to extract... The first target space attention map, i.e. And the attention map of the first target channel, i.e. The corresponding extraction formulas are as follows:

[0086] ;

[0087] ;

[0088] in, for Sigmoid Activation function It is a 1×1 convolutional layer. For the global max pooling operation in the spatial dimension, For the global max pooling operation along the channel dimension, the first target spatial attention map includes four spatial attention maps, namely... , , and The first target channel attention map includes four channel attention maps, namely , , and ; and extract The second target space attention map, namely The second target channel attention map, i.e. The corresponding extraction formulas are as follows:

[0089] ;

[0090] ;

[0091] The second target spatial attention map includes four spatial attention maps, namely , , and The second target channel attention map includes four channel attention maps, namely , , and Subsequently, the RSNE module adopted... respectively with and The feature multiplication method guides the feature learning of the RGB modality to obtain the first target semantic feature, namely... and the second target semantic features, namely The corresponding formulas are as follows:

[0092] ;

[0093] ;

[0094] in, This indicates an element-wise multiplication operation. ; then adopted and and The feature multiplication method guides the feature learning of the SN modality to obtain the first target geometric feature, namely... and the geometric features of the second target, namely The corresponding formulas are as follows:

[0095] ;

[0096] ;

[0097] Finally, the RSNE module embeds the SN modal information into the RGB features using element-wise addition, achieving multimodal information fusion and obtaining the first fused feature. Second fusion feature Third fusion feature and the fourth fusion feature The corresponding calculation formula is:

[0098] ;

[0099] in, This represents element-wise addition, for example, the first fusion feature. for:

[0100] And so on.

[0101] Subsequently, in different datasets, the shape or trend of change of free space often exhibits similarities, primarily reflected in its edge information, such as... Figure 4 As shown; therefore, edge information can serve as another domain-invariant feature, effectively enhancing the generalization performance of the model. Simultaneously, the features extracted at each stage of the backbone network encompass different types of information. Shallow (high-resolution) features are rich in local information such as edges, while deep (low-resolution) features contain more abstract global semantic cues. Given that both shallow edge information and deep semantic features possess domain-invariant properties, this embodiment of the invention utilizes a CFRE module (Cross-Scale Feature Robustness Module) to effectively aggregate multi-scale features, thereby enhancing the stability of RGB-SN features. Compared to simple channel concatenation, this CFRE module delves into the correlation between features of different scales, promoting information interaction between features of different scales through the calibration and enhancement effect of intermediate layer features on shallower and deeper features. The specific structure of this CFRE module is shown below. Figure 6 As shown, , and As input to this CFRE module, in order to clearly illustrate the feature interaction process in this embodiment of the invention, an intermediate size is used. and enhance the shallow layer The manipulation of semantic clues in the text will be introduced using examples; firstly... After DOWN (downsampling layer) to obtain and After achieving the same space dimensions, then with Each target fusion feature is obtained by passing it through a 1×1 convolutional layer, resulting in the same number of channels. Features integrated with the third objective, namely Subsequently, the attention module in the RSNE module is used to extract... The spatial attention map and the channel attention map are used to obtain the first spatial attention map, i.e. And the first channel attention map, i.e. ,extract From the spatial attention map and the channel attention map, we obtain the second spatial attention map, i.e. Second channel attention map, i.e. The attention map is then interactively calibrated using a multiplication method, with the specific formula as follows:

[0102] ;

[0103] ;

[0104] in, For the first calibrated attention map, For the second calibration attention map; then, respectively with and Multiplying them yields the first calibration feature, i.e. Second calibration feature, namely and will and By splicing along the channels, the first splicing feature is obtained, namely... This strengthens shallow features. To highlight the edge details and semantic information, the corresponding formula is:

[0105] ;

[0106] in, For channel splicing operation; then and The second splicing feature is obtained through similar operations as described above. Afterwards, in the passageway , and Multi-scale aggregated features are obtained by splicing. The data is then input into the ASPP (Atrous Spatial Pyramid Pooling) module to obtain the target multi-scale aggregated features. These features are then processed by the decoder to obtain the network's predicted semantic segmentation map. Finally, cross-entropy loss and style consistency loss are calculated on the predicted semantic segmentation map, and the corresponding calculation formulas are as follows:

[0107] ;

[0108] ;

[0109] in, ;

[0110] in, The semantic segmentation loss function is... Let style consistency loss function be used. The length of the semantic label graph. The width of the semantic label graph. For the semantic label graph, the first pixel value, To predict the first semantic segmentation graph Predicted values ​​per pixel, Let KL divergence be the KL divergence. To predict semantic segmentation maps, Predicting the fusion features of the original image and the surface normal vector map. For predicting the fusion features of stylized images and surface normal maps, for and The mean, For random variables, and Let represent the probability distribution in the segmentation graph predicted by the two networks.

[0111] Subsequently, edge information learning has dual advantages in domain generalization free space detection tasks. On the one hand, edge information can be regarded as an effective domain-invariant feature; on the other hand, edge information is also crucial for refining edge segmentation results, especially in unstructured off-road scenarios where the edges of free space are usually more difficult to distinguish, and the corresponding network model needs to pay more attention to edge information learning. Therefore, this invention uses EAL (RGB-SN edge-aware learning strategy) to explicitly improve the model's sensitivity and extraction ability to edge information. Although the detailed information of shallow features helps edge segmentation, this detailed information generally carries a lot of noise. To mitigate the impact of noise on the edge segmentation results, this invention utilizes... and This generates edge features, which are then input into the edge segmentation decoder. The specific structure of the RGB-SN edge-aware learning strategy is as follows: Figure 3 As shown in the EAL section, in order to maintain the same number of channels and size, First, the data is fed into a 3×3 convolutional layer, and then the resulting features are upsampled to obtain sampled features. In contrast, Only one 3×3 convolution operation is needed to obtain the first target fusion feature. Then, the first target fusion feature and the sampled feature are concatenated along the channel dimension to obtain the edge feature. The corresponding formula is:

[0112] ;

[0113] in, It is a 3×3 convolutional layer. This represents the upsampling layer; subsequently, edge features are... The input edge segmentation decoder generates a predicted edge map. Finally, considering the extreme imbalance between the number of edge elements and non-edge elements, the edge segmentation loss is calculated using binary weighted cross-entropy, and the corresponding formula is as follows:

[0114] ;

[0115] in, , ;

[0116] The edge segmentation loss function is... This represents the total number of pixels. For category loss weights, For the edge label pixel value, To predict edge maps The Predicted values ​​per pixel, For the edge label image, the first The pixel label value belongs to the category The number of pixels, Category.

[0117] Then, the overall loss value is calculated based on the semantic segmentation loss value, the style consistency loss value, and the edge segmentation loss value according to the formula:

[0118] ;

[0119] in, For the overall loss function, and All are weight coefficients; and through the corresponding loss functions, the model can focus on different content during training, thereby obtaining the desired effect. The semantic segmentation loss function is to enable the model to identify the semantics of traversable and intraversable regions, the style consistency loss function is to prevent the model from overfitting to the style of RGB source domain images, and the edge segmentation loss function is to make the model pay more attention to the edge information of free space (i.e., traversable regions). Then, the parameters of the free space detection training network model are corrected according to the overall loss value. The next synthetic image of the structured urban road scene is input into the free space detection training network model, and the above process is repeated, which will not be elaborated here. Until the training status of the free space detection training network model meets the preset conditions, the preset conditions include the target loss value meeting the preset requirements or the number of training times reaching the preset number. The preset requirements can be determined based on the accuracy of the free space detection generalization model, which will not be explained in detail here. The preset number of training times can be the maximum number of training times of the network training model, for example, 2000 times, etc. Finally, the trained free space detection generalization model is obtained.

[0120] Step S20: Obtain different types of road scene images to be processed, input the different types of road scene images to be processed into the generalization model of free space detection, and output the target free space image.

[0121] Specifically, after obtaining the trained generalized model for free space detection, road scene images that need to be detected for free space are acquired, i.e., different types of road scene images to be processed, including synthetic structured urban scene images, real structured urban scene images, and real unstructured off-road scene images. The different types of road scene images to be processed are then input into the trained generalized model for free space detection. The generalized model for free space detection outputs the target free space image, i.e., the road scene image with completed free space detection. This invention fully utilizes geometric information, represented by surface normal vectors and edge information, to assist RGB images in free space detection and achieves good performance in multiple generalization settings from single source domain to multi-target domain. Free space detection plays an important role in environmental perception in navigation tasks. For autonomous vehicles, the ability to accurately perceive different types of free space is crucial. The segmentation results generated by this invention can accurately divide RGB image pixels into passable and impassable areas, providing necessary perception information for autonomous driving technology decisions and improving the accuracy of dividing RGB image pixels into passable and impassable areas.

[0122] This invention embeds domain-invariant surface normal vector information into RGB features through a robust surface normal vector embedding module, achieving effective fusion of multimodal information. Given that shallow and deep features respectively retain domain-invariant edge information and global semantic cues, a cross-scale feature robustness enhancement module is used to enhance the stability of RGB-SN features by effectively aggregating and interacting information between features of different scales. Furthermore, an RGB-SN edge-aware learning strategy is employed to explicitly improve the model's sensitivity and ability to capture edge information. Through quantitative and qualitative analysis, the model successfully generalizes to structured urban scenes and unstructured off-road scene datasets across multiple synthetic single-source domains, fully demonstrating the strong robustness of the model and its excellent generalization performance in different types of free-space scenes.

[0123] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0124] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0125] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A dual-geometry information-aware domain generalization free space detection method, characterized in that, The domain generalization free space detection method based on dual geometric information perception includes: A training image set is acquired, and a model is trained based on the training image set to obtain a generalized model for free space detection. The training image set includes multiple synthetic images of structured urban road scenes, and the synthetic images of structured urban road scenes include color source domain images and corresponding surface normal vector source domain images. The process of acquiring a training image set and training a model based on the training image set to obtain a generalized model for free space detection specifically includes: A training image set is obtained, a free space detection training network model is created, and a synthetic image of a structured urban road scene from the training image set is input into the free space detection training network model. The color source domain image is style-processed to obtain a target color image. Feature extraction is performed on the target color image and the surface normal vector source domain image to obtain color features and surface normal vector features at multiple stages. Attention maps are extracted for the color features and surface normal vector features at each stage to obtain corresponding spatial attention maps and channel attention maps. Information fusion is performed on the color features and surface normal vector features based on the spatial attention maps and channel attention maps of the same stage to obtain a first fused feature, a second fused feature, a third fused feature, and a fourth fused feature. The second fusion feature, the third fusion feature, and the fourth fusion feature are calibrated and spliced ​​to obtain a multi-scale aggregated feature. A predicted semantic segmentation map is obtained based on the multi-scale aggregated feature. Cross-entropy loss and style consistency loss are calculated on the predicted semantic segmentation map to obtain semantic segmentation loss value and style consistency loss value. The first fusion feature and the second fusion feature are concatenated to obtain edge features. A predicted edge map is obtained based on the edge features. The predicted edge map is then subjected to binary weighted cross-entropy calculation to obtain the edge segmentation loss value. The overall loss value is obtained based on the semantic segmentation loss value, the style consistency loss value, and the edge segmentation loss value, and the parameters of the free space detection training network model are corrected based on the overall loss value. The next synthetic image of the structured urban road scene is input into the free space detection training network model until the training status of the free space detection training network model meets the preset conditions, and a trained generalized model of free space detection is obtained. Acquire different types of road scene images to be processed, input the different types of road scene images to be processed into the generalization model of free space detection, and output the target free space image; The different types of road scene images to be processed include synthesized structured urban scene images, real structured urban scene images, and real unstructured off-road scene images.

2. The dual-geometry information-aware domain generalization free space detection method of claim 1, wherein, The process involves style processing of the color source domain image to obtain a target color image, followed by feature extraction from both the target color image and the surface normal vector source domain image to obtain color features and surface normal vector features at multiple stages. Specifically, this includes: The color source image is subjected to image style diversification processing to obtain a target color image, wherein the target color image includes a stylized image and the original image; In the target feature extraction stage, feature extraction is performed on the target color image and the surface normal vector source domain image respectively to obtain target color features and target surface normal vector features; The target feature extraction stage includes a first feature extraction stage, a second feature extraction stage, a third feature extraction stage, and a fourth feature extraction stage. The target color features include color features from four stages, and the target surface normal vector features include surface normal vector features from four stages.

3. The dual-geometry information-aware domain generalization free space detection method of claim 2, wherein, The step of extracting attention maps from the color features and surface normal vector features at each stage to obtain corresponding spatial attention maps and channel attention maps, specifically includes: The target color features are convolved to obtain the first convolution feature, and the first convolution feature is extracted with attention map according to the first formula and the second formula to obtain the first target spatial attention map and the first target channel attention map. The target surface normal vector features are convolved to obtain the second convolution feature. Attention is extracted from the second convolution feature according to the third and fourth formulas to obtain the second target spatial attention map and the second target channel attention map. The first target spatial attention map and the second target spatial attention map each include four spatial attention maps, and the first target channel attention map and the second target channel attention map each include four channel attention maps.

4. The dual-geometry information-aware domain generalization free space detection method of claim 3, wherein, The first formula is: ； The second formula is: ； The third formula is: ； The fourth formula is: ； in, for The corresponding spatial attention map, for The corresponding channel attention map, for The corresponding spatial attention map, for The corresponding channel attention map, It is the Sigmoid activation function. It is a 1×1 convolutional layer. For the global max pooling operation in the spatial dimension, This is a global max-pooling operation at the channel level. The color features of stage i are the corresponding convolutional features generated by a 1×1 convolutional layer. The surface normal vector features of stage i are the corresponding convolutional features generated by a 1×1 convolutional layer, where i is the stage number. ={1, 2, 3, 4}.

5. The dual-geometry information-aware domain generalization free space detection method of claim 3, wherein, The step of fusing the color features and the surface normal vector features based on the spatial attention map and the channel attention map of the same stage to obtain a first fused feature, a second fused feature, a third fused feature, and a fourth fused feature specifically includes: The first convolutional feature is multiplied element-wise with the second target spatial attention map to obtain the first target semantic feature, and the first convolutional feature is multiplied element-wise with the second target channel attention map to obtain the second target semantic feature. The second convolutional feature is multiplied element-wise with the first target spatial attention map to obtain the first target geometric feature, and the second convolutional feature is multiplied element-wise with the first target channel attention map to obtain the second target geometric feature. The first target semantic feature, the second target semantic feature, the first target geometric feature, and the second target geometric feature are added element by element to obtain the corresponding first fusion feature, second fusion feature, third fusion feature, and fourth fusion feature; The first target semantic feature, the second target semantic feature, the first target geometric feature, and the second target geometric feature each include four corresponding features.

6. The dual-geometry information-aware domain generalization free-space detection method of claim 1, wherein, The step of calibrating and splicing the second, third, and fourth fusion features to obtain multi-scale aggregated features specifically includes: The spatial dimensions of the second fusion feature are processed, and convolution operations are performed on the processed second fusion feature, the third fusion feature, and the fourth fusion feature to obtain the second target fusion feature, the third target fusion feature, and the fourth target fusion feature. Attention maps are extracted from the second target fusion feature to obtain a first spatial attention map and a first channel attention map; attention maps are extracted from the third target fusion feature to obtain a second spatial attention map and a second channel attention map; and attention maps are extracted from the fourth target fusion feature to obtain a third spatial attention map and a third channel attention map. The first spatial attention map and the second spatial attention map are multiplied element-wise to obtain the first calibrated attention map, and the second spatial attention map and the third spatial attention map are multiplied element-wise to obtain the second calibrated attention map. The first channel attention map and the second channel attention map are multiplied element-wise to obtain the third calibration attention map. The second channel attention map and the third channel attention map are multiplied element-wise to obtain the fourth calibration attention map. The second target fusion feature is multiplied element-wise with the first calibration attention map and the third calibration attention map to obtain the first calibration feature and the second calibration feature, respectively. The fourth target fusion feature is multiplied element-wise with the second calibration attention map and the fourth calibration attention map to obtain the third calibration feature and the fourth calibration feature, respectively. The first calibration feature and the second calibration feature are concatenated by channels to obtain the first concatenated feature. The third calibration feature and the fourth calibration feature are concatenated by channels to obtain the second concatenated feature. The first concatenated feature, the second concatenated feature and the third target fusion feature are concatenated by channels to obtain the multi-scale aggregated feature.

7. The dual-geometry information-aware domain generalization free-space detection method of claim 1, wherein, The process of obtaining a predicted semantic segmentation map based on the multi-scale aggregated features, and then calculating cross-entropy loss and style consistency loss on the predicted semantic segmentation map to obtain semantic segmentation loss values ​​and style consistency loss values, specifically includes: The multi-scale aggregated features are subjected to pore-diffused spatial pyramid pooling to obtain target multi-scale aggregated features, and a predicted semantic segmentation map is obtained based on the target multi-scale aggregated features. The cross-entropy loss is calculated on the predicted semantic segmentation graph according to the fifth formula to obtain the semantic segmentation loss value, and the style consistency loss is calculated on the predicted semantic segmentation graph according to the sixth formula to obtain the style consistency loss value. The fifth formula is: ； The sixth formula is: ； in, The semantic segmentation loss function is... Let style consistency loss function be used. The length of the semantic label graph. The width of the semantic label graph. For the semantic label graph, the first pixel value, To predict the first semantic segmentation graph Predicted values ​​per pixel, Let KL divergence be the KL divergence. To predict semantic segmentation maps, Predicting the fusion features of the original image and the surface normal vector map. For predicting the fusion features of stylized images and surface normal maps, for and The mean.

8. The dual-geometry information-aware domain generalization free-space detection method of claim 1, wherein, The process of concatenating the first fusion feature and the second fusion feature to obtain edge features, obtaining a predicted edge map based on the edge features, and calculating the edge segmentation loss value by performing binary weighted cross-entropy calculation on the predicted edge map specifically includes: The first fusion feature is convolved to obtain the first target fusion feature. The second fusion feature is convolved and upsampled to obtain the sampled feature. The first target fusion feature and the sampled feature are concatenated by channels to obtain the edge feature. Based on the edge features, a predicted edge map is obtained, and the predicted edge map is subjected to binary weighted cross-entropy calculation according to the seventh formula to obtain the edge segmentation loss value. The seventh formula is: ； The calculation formula is: ; in, The edge segmentation loss function is... This represents the total number of pixels. For category loss weights, For the edge label pixel value, To predict edge maps The Predicted values ​​per pixel, For the edge label graph, the first The pixel label value belongs to the category The number of pixels, Category.

Images