Unstructured environment passable region segmentation method and system

Abstract

The application discloses a kind of non-structured environment passable area segmentation method and system, it is related to vehicle environment automatic perception technical field.The method includes the following steps: scene data acquisition and pre-processing;Scene attribute modeling and scene anchor point generation;Optimal transmission target distribution and fusion relationship construction;Optimal transmission solving and cross-modal fusion;Mask decoding and segmentation result output, and the segmentation result is passable area.The method provides a new scheme that can be combined modeling to scene attribute, and completes multi-modal fusion in distribution level, to improve the generalization ability of system in distribution outside scene, cross-dataset migration ability and edge device deployment capability.

Description

Technical Field

[0001] This invention relates to the field of automatic vehicle environment perception technology, and in particular to a method and system for segmenting passable areas in unstructured environments. Background Technology

[0002] In unstructured environments such as woodlands, mountains, snowfields, mud, and gravel roads, lane markings, road boundaries, and traffic signs are often lacking. Vehicles must rely on perception systems to autonomously determine safe drivable areas; therefore, drivable area analysis is a crucial foundational capability for autonomous navigation. Existing solutions can be broadly categorized into methods based on LiDAR geometric modeling, image semantic segmentation, and image-geometric information fusion. The first two methods suffer from insufficient semantic understanding and sensitivity to illumination, respectively. While multimodal fusion methods can simultaneously utilize semantic texture information and surface normal geometric structure information from images, they mostly assume that different modalities can be directly aligned within a shared feature space and primarily rely on pixel-level or patch-level feature matching. When the test scenario includes combinations of weather, time of day, and road surface type not covered in the training set, these assumptions often fail, and the model is prone to false positives and false negatives in out-of-distribution scenarios, thus affecting the safety of downstream planning and decision-making. Real-world off-road environments exhibit a significant long-tail distribution, making it impossible to exhaustively collect data to cover all scenario combinations. Summary of the Invention

[0003] The technical problem to be solved by the present invention is how to provide a method and system for segmenting unstructured environment passable areas that can perform composable modeling of scene attributes and complete multimodal fusion at the distribution level, thereby improving the system's generalization ability, cross-dataset migration ability and edge device deployment ability in out-of-distribution scenes.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a method for segmenting traversable areas in unstructured environments, comprising the following steps: Scene data acquisition and preprocessing; Scene attribute modeling and scene anchor point generation; Construction of optimal transmission target distribution and fusion relationship; Optimal transmission solution and cross-modal fusion; The mask decoding and segmentation results are output, and the segmentation result is the passable area.

[0005] This invention also discloses a system for segmenting traversable regions in unstructured environments, the system running the aforementioned method for segmenting traversable regions in unstructured environments, the system comprising: The data acquisition and preprocessing module is used to acquire and preprocess scene data; The attribute modeling and anchor point generation module is used for scene attribute modeling and scene anchor point generation. The module for constructing the distribution and fusion relationship of transmission targets is used to construct the optimal distribution and fusion relationship of transmission targets. The transport solution and cross-modal fusion module is used to achieve optimal transport solution and cross-modal fusion. The segmentation result output module is used for mask decoding and segmentation result output. The segmentation result is a passable area.

[0006] The beneficial effects of adopting the above technical solution are as follows: The method described in this invention is applicable to off-road vehicles, agricultural machinery platforms, inspection robots, unmanned transport vehicles in mining areas, and other mobile platforms operating in unstructured environments. The method generates scene anchor points by performing edge probability modeling on weather attributes, time period attributes, and road type attributes. Then, image features and surface normal features are mapped onto the semantic manifold defined by the scene anchor points through optimal transfer mapping, thereby obtaining a traversable region segmentation result with strong generalization ability for out-of-distribution scenes. This method provides a novel approach that enables composable modeling of scene attributes and multimodal fusion at the distribution level, thereby improving the system's generalization ability in out-of-distribution scenes, its cross-dataset transfer ability, and its edge device deployment ability. Attached Figure Description

[0007] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0008] Figure 1 This is the main flowchart of the method described in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the architecture of the method described in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the system described in Embodiment 2 of the present invention. Detailed Implementation

[0009] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0010] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0011] Example 1 like Figure 1As shown in the figure, this invention discloses a method for segmenting traversable areas in unstructured environments, including the following steps: S1: Scene data acquisition and preprocessing; S2: Scene attribute modeling and scene anchor point generation; S3: Construction of optimal transmission target distribution and fusion relationship; S4: Optimal transport solution and cross-modal fusion; S5: Output of mask decoding and segmentation results. The segmentation result is the passable area.

[0012] The method described in this invention is applicable to off-road vehicles, agricultural machinery platforms, inspection robots, unmanned transport vehicles in mining areas, and other mobile platforms operating in unstructured environments. Its core idea is not to force different modalities to be directly aligned in the original feature space, but rather to first utilize the semantic stability of the visual language model to generate generalizable scene anchors, and then, using these scene anchors as the target distribution, achieve distribution-level fusion of image modalities and geometric modalities through optimal transmission.

[0013] The method will be described in detail below with specific steps, and the specific process is as follows: Figure 2 As shown: Step S1: Scene data acquisition and preprocessing.

[0014] The system acquires RGB images of the current scene using a camera, obtains point cloud data using LiDAR, depth sensors, or a 3D reconstruction module, and estimates surface normals. The surface normal map is estimated based on the point cloud, and the point cloud is mapped to the image coordinate system to achieve a spatial correspondence between the image and geometric information: The image and surface normal map are synchronized in time, registered in space, scaled and normalized to obtain the multimodal sample to be input.

[0015] The surface normal map is used to characterize geometric information such as local surface orientation, boundary structure, and topographic changes.

[0016] Step S2: Scene attribute modeling and scene anchor point generation.

[0017] Image encoders using visual language models extract images global features Each for weather attributes Time period attributes and road type attributes Probabilistic modeling is performed to obtain the marginal probability distribution of each attribute: in, The temperature parameter is used to adjust the scaling of the similarity score to control the sharpness and stability of the scene attribute probability distribution.

[0018] Based on a pre-built scene prototype library and accessible and inaccessible meta-semantic embeddings, the various scene combinations are weighted and aggregated to generate scene anchors corresponding to the current environment: in, The meta-semantic embeddings representing passable and impassable features have a feature dimension of D.

[0019] The scene anchor points simultaneously encode environmental semantics and accessibility semantics, serving as prior conditions for subsequent fusion and decoding.

[0020] Step S3: Constructing the optimal transmission target distribution and fusion relationship.

[0021] The source distribution is constructed by using the multi-scale image feature map output from the image encoder and the multi-scale normal feature map output from the surface normal encoder, respectively. : Based on the pre-segmentation results of image branching and normal branching, and combined with scene anchor points, the target distribution is constructed. : Construct a cost matrix based on the distance relationship between features and scene anchor points. : The cost matrix is ​​used for subsequent optimal transmission solutions.

[0022] Step S4: Optimal transport solution and cross-modal fusion.

[0023] Solve for the optimal transport plan to the scene anchor point for both the image branch and the surface normal branch: Based on the optimal transport plan, image features and surface normal features are projected onto the semantic manifold defined by scene anchor points to obtain projected image enhancement features and normal enhancement features. A unified fusion feature representation can be obtained through weighted summation, gating fusion, or other learnable fusion methods: The fusion feature is used for subsequent mask decoding.

[0024] Step S5: Mask decoding and result output.

[0025] Encode scene anchors with learnable locations The sum is used as the initial query vector for the mask decoder; The fused features are input into the pixel decoder and mask decoder. The query vector is updated layer by layer through mask attention, self-attention, and feedforward networks, and the output is a probability map of passable regions or a binary mask. in, Indicates the features to be segmented. Represents a multi-scale feature pyramid. To achieve a minimum value, avoid a denominator of 0.

[0026] The final segmentation result is the passable area, which can be sent to the path planning, obstacle avoidance, speed planning, risk assessment, or vehicle control modules.

[0027] The method described in this invention is not limited to outputting only binary classification passable results, but can also be extended to multi-level risk zoning results, such as safe passage, cautious passage, and prohibited passage; the scene attributes in step two are not limited to weather, time of day, and road type, but can also be extended to attributes such as surface slipperiness, slope, vegetation density, water accumulation, and obstacle density; the optimal transmission in step four is not limited to the single entropy regularization form, but can also be implemented using unbalanced optimal transmission, Gromov-Wasserstein optimal transmission, or Wasserstein centroid, etc.

[0028] The method described in this invention has significant advantages over existing technologies in terms of generalization ability in out-of-distribution scenarios, cross-dataset transfer ability, multimodal fusion robustness, and edge deployment efficiency. The comparison results with the mean Intersection over Union (mIoU) index of existing traversable region segmentation methods are as follows.

[0029] Table 1 - Experimental Results of ORFD Dataset

[0030] Table 2 - Experimental Results of ORAD-3D Dataset

[0031] Table 3 - Experimental Results of ORFD Migration to ORAD-3D Dataset

[0032] The method described in this invention has the following advantages compared with existing methods: (1) Significantly improves the generalization ability of out-of-distribution scenes: By decomposing scene distribution into marginal probabilities such as weather, time period and road type, and generating scene anchors based on combined scene prototypes, this invention can maintain stable performance even in scene combinations that do not appear in the training set. Experimental data show that in the out-of-distribution scene test of the ORFD dataset, the mIoU of the corresponding method of this invention reaches 95.16%, which is about 6.35 percentage points higher than the existing state-of-the-art methods.

[0033] (2) Enhancing the robustness of multimodal fusion: Existing methods typically employ pixel-level or token-level direct alignment, which is easily affected by distribution offset. This invention projects image features and surface normal features onto the semantic manifold defined by scene anchor points through optimal transmission, completing the fusion at the distribution level, thereby improving the adaptability to unknown weather, unknown lighting, and unknown road conditions. In experimental results, this invention achieves an mIoU of 89.89% on out-of-distribution scenes of the ORAD-3D dataset, and controls the performance degradation from known scenes to unknown scenes to within approximately 1.04 percentage points.

[0034] (3) Strong combinatorial generalization and cross-dataset transfer capabilities: Since this invention does not directly learn closed dataset patterns, but reorganizes scene factors based on attribute marginal probabilities, it can transfer to a larger test set even with limited training data. Experimental results show that in cross-dataset tasks of ORFD training and ORAD-3D direct testing, the mIoU of this invention reaches 89.79%, which is about 13.99 percentage points higher than the baseline method, and the performance degradation is only about 0.07 percentage points.

[0035] (4) It has good data efficiency and interpretability: The scene anchor generation process clearly gives the probability distribution of weather, time period and road type and the corresponding combination weights, which facilitates the analysis of the decision basis of the model under different environmental conditions. At the same time, the ablation experiment shows that the introduction of scene anchors alone can bring about an improvement of mIoU of about 21.03% and 5.73% in out-of-distribution scenes of ORFD and ORAD-3D, respectively, indicating that the module has a clear effect on improving generalization ability.

[0036] (5) Possesses good real-time deployment potential: By caching the scene prototype library and avoiding repeated calls to the text encoder during the inference phase, this invention achieves a speed improvement of about 25.6% in the semantic guidance part; after further combining graph-optimized compilation and mixed-precision inference, it can reach about 21.11 FPS on Tesla T4, which is suitable for deployment on resource-constrained edge computing platforms.

[0037] Example 2 Corresponding to the method described in Example 1, such as Figure 3 As shown, Embodiment 2 of the present invention discloses a system for segmenting traversable areas in unstructured environments. The system runs the aforementioned method for segmenting traversable areas in unstructured environments. The system includes: Data acquisition and preprocessing module 101 is used for scene data acquisition and preprocessing; Attribute modeling and anchor point generation module 102 is used for scene attribute modeling and scene anchor point generation; The transmission target distribution and fusion relationship construction module 103 is used to realize the construction of the optimal transmission target distribution and fusion relationship; The transmission solution and cross-modal fusion module 104 is used to realize optimal transmission solution and cross-modal fusion; The segmentation result output module 105 is used for mask decoding and segmentation result output, and the segmentation result is a passable area.

[0038] It should be noted that the specific implementation methods of each module in the system can refer to the method described in Embodiment 1, and will not be repeated here.

[0039] Although the above embodiments have been described, those skilled in the art, once they understand the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the above descriptions are merely embodiments of the present invention and do not limit the scope of patent protection of the present invention. Any equivalent structural or procedural transformations made using the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for segmenting traversable regions in an unstructured environment, characterized in that... Includes the following steps: Scene data acquisition and preprocessing; Scene attribute modeling and scene anchor point generation; Construction of optimal transmission target distribution and fusion relationship; Optimal transmission solution and cross-modal fusion; The mask decoding and segmentation results are output, and the segmentation result is the passable area.

2. The method for segmenting traversable areas in an unstructured environment as described in claim 1, characterized in that, The scene data acquisition and preprocessing includes the following steps: The system acquires RGB images of the current scene using a camera, and obtains point cloud data using LiDAR, depth sensors, or a 3D reconstruction module to estimate the surface normal map. The surface normal map is estimated based on the point cloud, and the point cloud is mapped to the image coordinate system to achieve a spatial correspondence between the image and geometric information: The image and surface normal map are synchronized in time, registered in space, scaled and normalized to obtain the multimodal sample to be input.

3. The method for segmenting traversable areas in an unstructured environment as described in claim 2, characterized in that, The method for scene attribute modeling and scene anchor point generation includes the following steps: Image encoders using visual language models extract images global features Each for weather attributes Time period attributes and road type attributes Probabilistic modeling is performed to obtain the marginal probability distribution of each attribute: in, The temperature parameter is used to adjust the scaling of the similarity score to control the sharpness and stability of the scene attribute probability distribution. Based on a pre-built scene prototype library and accessible and inaccessible meta-semantic embeddings, the various scene combinations are weighted and aggregated to generate scene anchors corresponding to the current environment: in, The meta-semantic embeddings representing drivability and inaccessibility have a feature dimension of D. The scene anchors simultaneously encode environmental semantics and drivability semantics, serving as prior conditions for subsequent fusion and decoding.

4. The method for segmenting traversable areas in an unstructured environment as described in claim 3, characterized in that, The method for constructing the optimal transmission target distribution and fusion relationship includes the following steps: The source distributions are constructed from the multi-scale image feature map output by the image encoder and the multi-scale normal feature map output by the surface normal encoder, respectively: Based on the pre-segmentation results of image branching and normal branching, and combined with scene anchor points, the target distribution is constructed as follows: Construct a cost matrix based on the distance relationship between features and scene anchor points. : The cost matrix is ​​used for subsequent optimal transmission solutions.

5. The method for segmenting traversable areas in an unstructured environment as described in claim 4, characterized in that, The method for optimal transmission solution and cross-modal fusion includes the following steps: The optimal transport plan to the scene anchor point is solved separately for the image branch and the surface normal branch. Based on the optimal transport plan, image features and surface normal features are projected onto the semantic manifold defined by scene anchor points to obtain projected image enhancement features and normal enhancement features: A unified fusion feature representation can be obtained through weighted summation, gating fusion, or other learnable fusion methods: The fusion feature is used for subsequent mask decoding.

6. The method for segmenting traversable areas in an unstructured environment as described in claim 5, characterized in that, The specific methods for mask decoding and segmentation result output include the following steps: Encode scene anchors with learnable locations The sum is used as the initial query vector for the mask decoder: Subsequently, the feature input pixel decoder and mask decoder are fused, and the query vector is updated layer by layer through mask attention, self-attention, and feedforward networks, outputting a passable region probability map or a binary mask: in, Indicates the features to be segmented. Represents a multi-scale feature pyramid. To minimize the denominator to zero, the final segmentation result is the passable region.

7. A system for segmenting traversable regions in an unstructured environment, the system operating the method for segmenting traversable regions in an unstructured environment as described in any one of claims 1-6, characterized in that... The system includes: The data acquisition and preprocessing module is used to acquire and preprocess scene data; The attribute modeling and anchor point generation module is used for scene attribute modeling and scene anchor point generation. The module for constructing the distribution and fusion relationship of transmission targets is used to construct the optimal distribution and fusion relationship of transmission targets. The transport solution and cross-modal fusion module is used to achieve optimal transport solution and cross-modal fusion. The segmentation result output module is used for mask decoding and segmentation result output. The segmentation result is a passable area.

Images