A method for road recognition based on multi-view perception SAM

Abstract

The application provides a road recognition method based on multi-view perception SAM, comprising the following steps: transforming a training road image into global and multiple local views through multi-view transformation, inputting a frozen SAM encoder embedded with a multi-scale space aggregation adapter, and extracting global and local coding features; obtaining a local view mask through splicing fusion and up-sampling of a local view mask predictor, encoding the mask into dense prompt features by a prompt encoder, fusing the features with global features, and obtaining a training recognition result through bidirectional cross attention processing and up-sampling of a mask decoder; determining a joint loss according to the training result, the local mask and a real label, and updating only the adapter, the mask predictor and the decoder parameters to obtain a trained model; and inputting a to-be-recognized road image into the trained model through multi-view transformation to obtain a road region recognition result. The application can complete road region recognition without relying on artificial interactive prompts and reduce the parameter update range required for road field adaptation.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation technology, specifically relating to a road recognition method based on multi-view perception (SAM). Background Technology

[0002] Road region recognition is used to determine drivable road areas from road scene images and is fundamental to autonomous driving environmental perception, path planning, and vehicle control. Due to variations in road scene scale, lighting, occlusion, and complex intersections, road recognition models need to utilize both global structure and local details to stably output road region segmentation results.

[0003] Existing convolutional neural networks rely on local convolutions to extract road features, but are limited by a finite receptive field, making it difficult to fully capture long-distance dependencies. While expanding the receptive field or deepening the network can improve global modeling, it significantly increases the number of parameters and computational cost, making it difficult to meet the real-time requirements of automotive embedded platforms. Visual transformers establish long-distance associations through self-attention mechanisms, but the computational complexity increases quadratically when processing high-resolution road images, resulting in high computational overhead. The Segment Anything Model (SAM), as a large-scale visual pre-trained foundational model, has general image segmentation capabilities. Its image encoder, cue encoder, and mask decoder can jointly complete cue-guided segmentation processing. However, directly applying SAM to road recognition still faces many challenges. On the one hand, road images and pre-trained natural images differ significantly in data distribution, scene structure, and imaging characteristics, making it difficult for SAM to effectively transfer general segmentation knowledge to road scenes. On the other hand, SAM typically relies on external cues such as human points, boxes, or masks during training and inference, which cannot be provided manually in real time in actual autonomous driving deployments, limiting its in-vehicle usability.

[0004] To alleviate reliance on manual prompts, existing methods introduce additional prompt generation modules or auxiliary encoder branches to achieve automatic prompting. However, these methods often overemphasize prompt generation, neglecting the global structural information in the original road image. This results in a disconnect between the generated prompts and the global road structure, affecting the final segmentation accuracy. Furthermore, existing adapters often directly employ linear transformation structures from the field of natural language processing, which are insufficient for representing the complex and varied scale changes and spatial structures in road images, making it difficult to fully model the multi-scale features of road scenes with a limited number of trainable parameters. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides a road recognition method based on Multi-View Awareness (SAM). The technical problem to be solved by this invention is achieved through the following technical solution: This invention provides a road recognition method based on multi-view perception (SAM), comprising: Acquire training road images and their corresponding real road labels, perform multi-view transformation on the training road images, and obtain training multi-view input data including a global view and multiple local views; The training multi-view input data is input into the road recognition model to be trained, and global view coding features and local view coding features are extracted by a frozen SAM image encoder with an embedded multi-scale spatial aggregation adapter. The local view encoded features are spatially stitched, fused across views, and upsampled by a local view mask predictor to obtain a local view mask; the local view mask is encoded by a cue encoder to obtain dense cue features, and the dense cue features are fused with the global view encoded features to obtain fused features; The fused features are subjected to bidirectional cross-attention processing and upsampling by a mask decoder to obtain the training road recognition results. The joint loss is determined based on the training road recognition results, the local view mask, and the real road label. The parameters of the multi-scale spatial aggregation adapter, the local view mask predictor, and the mask decoder are updated based on the joint loss to obtain the trained road recognition model. A multi-view transformation is performed on the road image to be identified to obtain multi-view input data to be identified. The multi-view input data to be identified is then input into the trained road recognition model to obtain the road area recognition result.

[0006] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention provides a road recognition method based on Multi-View Awareness (SAM). By converting training road images into a global view and multiple local views, the road recognition model simultaneously acquires the complete road structure and local region details. By embedding a multi-scale spatial aggregation adapter into the frozen SAM image encoder, road domain feature adaptation is achieved without updating the original parameters of the image encoder. Local view masks are generated by spatially stitching, cross-view fusion, and upsampling the encoded features of local views, transforming the spatial correspondences between local views into dense cues that can be processed by the cue encoder. By fusing the dense cue features with the encoded features of the global view and using a mask decoder for bidirectional cross-attention processing, local road cues can constrain the decoding of the global road region. By determining the joint loss and updating the model parameters based on the training road recognition results, local view masks, and real road labels, the training road recognition results are simultaneously constrained by both the local view masks and real road labels. Ultimately, this invention can complete road region recognition without relying on manual interactive prompts and reduces the parameter update range required for road domain adaptation, thereby reducing the risk of overfitting with limited training data and improving the model's generalization ability and deployment efficiency in complex road scenarios.

[0007] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0008] Figure 1 This is a schematic diagram illustrating the implementation process of a road recognition method based on multi-view perception (SAM) provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the hardware entity of an electronic device to which embodiments of the present invention are applied. Detailed Implementation

[0009] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0010] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating the implementation process of a road recognition method based on Multi-View Awareness (SAM) provided in an embodiment of the present invention. Figure 1 As shown, the road recognition method based on Multi-View Awareness (SAM) provided in this embodiment of the invention may include the following steps: Step S101: Obtain the training road image and its corresponding real road label, and perform multi-view transformation on the training road image to obtain training multi-view input data including a global view and multiple local views.

[0011] In one possible implementation, a multi-view transformation is performed on the training road image to obtain training multi-view input data including a global view and multiple local views. This includes: flipping, resizing, and normalizing the training road image to obtain a preprocessed road image; using the preprocessed road image as the global view and dividing the preprocessed road image into multiple local views according to adjacent spatial regions; adjusting the global view and each local view to the same resolution and combining them along the batch dimension to obtain the training multi-view input data.

[0012] For example, the training road image can be a road scene image with pixel-level real road labels.

[0013] After obtaining the training road image and its corresponding real road label, the same spatial transformation as the training road image can be performed on the real road label to ensure that the pixels of the real road label correspond to the pixels of the training road image.

[0014] The training road image can be randomly flipped vertically, randomly flipped horizontally, scaled, and normalized to obtain a preprocessed road image. The complete preprocessed road image is used as a global view, and the preprocessed road image is divided into four local views (upper left, upper right, lower left, and lower right) according to adjacent spatial regions (or other numbers of local views according to actual needs). The global view and each local view are adjusted to the same resolution (e.g., 256×256 pixels, or other values ​​set according to actual needs), and the global view and each local view are combined along the batch dimension to form the training multi-view input data.

[0015] Step S102: Input the training multi-view input data into the road recognition model to be trained, and extract global view coding features and local view coding features by using a frozen SAM image encoder with an embedded multi-scale spatial aggregation adapter.

[0016] For example, the SAM image encoder may consist of multiple transformer modules that keep their pre-trained parameters frozen during training and embed a multi-scale spatial aggregation adapter (MSA-Adapter) in each transformer module.

[0017] In one possible implementation, the multi-scale spatial aggregation adapter includes an adaptive normalization layer, a downprojection layer, a multi-scale spatial aggregation module, and an upprojection layer. The frozen SAM image encoder, embedded with the multi-scale spatial aggregation adapter, extracts global view coding features and local view coding features, including: performing layer normalization on the input features of the frozen SAM image encoder through the adaptive normalization layer, and adjusting the layer normalization result and input features respectively through learnable layer normalization weight parameters and input weight parameters to obtain adaptive normalized features; reducing the channel dimension of the adaptive normalized features through the downprojection layer to obtain downprojection features; processing the downprojection features sequentially through the multi-scale spatial aggregation module and the upprojection layer to obtain upprojection features; performing residual fusion of the upprojection features and input features to obtain multi-scale spatial aggregation features, and determining the multi-scale spatial aggregation features corresponding to each view as global view coding features or local view coding features according to the view type (global or local view) in the training multi-view input data.

[0018] In one possible implementation, the downprojection features are processed sequentially by a multi-scale spatial aggregation module and an upprojection layer to obtain the upprojection features. This includes: extracting features from the downprojection features by progressively increasing the receptive field through a serial convolutional branch in the multi-scale spatial aggregation module, resulting in multiple cascaded scale features; extracting features from the downprojection features by different receptive fields through parallel convolutional branches in the multi-scale spatial aggregation module, resulting in multiple parallel scale features; determining multi-scale fusion features based on each cascaded scale feature and each parallel scale feature; and restoring the channel dimension of the multi-scale fusion features through the upprojection layer to obtain the upprojection features.

[0019] For example, a multi-scale spatial aggregation adapter can be constructed by sequentially connecting an adaptive normalization layer, a lower projection layer, a multi-scale spatial aggregation module, and an upper projection layer. The input features can be first normalized using the adaptive normalization layer, and then learnable layer normalization weight parameters can be utilized. and input weight parameters Adjust the layer normalization result and the original input feature separately, and add the two adjusted results to obtain the adaptive normalized feature. The calculation relationship is as follows: ; in, For the input features of the multi-scale spatial aggregation adapter, Representation layer normalization operation, For layer normalized weight parameters, For input weight parameters, This is an adaptive normalized feature.

[0020] The lower projection layer uses pointwise convolution to reduce adaptive normalized features. The channel dimension is used to obtain the downprojection features. .

[0021] The multi-scale spatial aggregation module includes both serial and parallel convolution branches. The serial convolution branch sequentially uses 3×3, 5×5, and 7×7 convolution kernels to extract features at progressively larger receptive fields, yielding multiple cascaded scale features and intermediate scale features within the cascaded process. The parallel convolution branch processes the projected features using the aforementioned convolution kernels. Parallel scale features are obtained. These cascaded scale features include: a first-scale feature obtained by performing a first-layer convolution operation on the downprojected features using a 3×3 convolution kernel; a second-scale feature obtained by performing a second-layer convolution operation on the first-scale features using a 5×5 convolution kernel; and a third-scale feature obtained by performing a third-layer convolution operation on the second-scale features using a 7×7 convolution kernel. After concatenating all cascaded scale features, intermediate scale features, and parallel scale features along the channel dimension, a multi-scale fused feature is obtained through point-by-point convolution. The calculation relationship is as follows: ; in, This represents pointwise convolution. Indicates channel splicing. Indicates the first-scale feature. This represents the second-scale feature. , and Represents parallel scale characteristics. Indicates the third-scale feature. , and This represents the intermediate-scale features generated by cascaded convolutions. This represents the multi-scale fusion features. The upper projection layer recovers the multi-scale fusion features. The channel dimension is calculated and residual fusion is performed with the input features. The above processing steps are then applied to the global view and the local view respectively to obtain the global view encoded features and the local view encoded features.

[0022] Step S103: Spatial stitching, cross-view fusion, and upsampling are performed on the local view encoded features by the local view mask predictor to obtain the local view mask; the local view mask is encoded by the cue encoder to obtain dense cue features, and the dense cue features are fused with the global view encoded features to obtain fused features.

[0023] In one possible implementation, the Local View Mask Predictor (LVMP) includes a feature stitching module, a gated multilayer perceptron, and an upsampling module. The LVMP performs spatial stitching, cross-view fusion, and upsampling on the local view encoded features to obtain the local view mask. This process includes: using the feature stitching module to stitch the corresponding local view encoded features based on the spatial location of each local view in the training road image, obtaining complete local encoded features; using the gated multilayer perceptron to perform nonlinear feature transformation and gate weight generation on the complete local encoded features, fusing the transformed features and gate weights element-wise to obtain cross-view fused features; extracting channel features from the cross-view fused features through deep convolution and expanding the channel dimensions of the obtained channel features; allocating the expanded channel features to the spatial dimension through pixel rearrangement to obtain upsampled features; and using the upsampling module to process the upsampled features using pointwise convolution and the sigmoid activation function to obtain the local view mask.

[0024] For example, the feature stitching module stitches together the corresponding local view encoded features based on the spatial position of each local view in the original training road image to obtain complete local encoded features. A branch of a gated multilayer perceptron encodes complete local features. A nonlinear feature transformation is performed, and another branch generates gated weights. The outputs of the two branches are multiplied element-wise and then randomly deactivated to obtain the cross-view fusion features. The calculation relationship is as follows: ; in, For cross-view fusion features, For complete local coding features, This represents pointwise convolution. This represents the activation function. This indicates a random deactivation operation. Gating weights are used to select effective road features in different local regions, enabling the stitched local features to exchange information across regions.

[0025] The upsampling module first processes cross-view fusion features through depthwise convolution. Then, the channel dimensions are expanded and pixel rearrangement is used to redistribute the channel information to the spatial dimension. Subsequently, pointwise convolution is used to compress the channels into mask channels, and the Sigmoid activation function is used to generate a local view mask. The calculation relationship is as follows: ; in, For local view mask, For cross-view fusion features, This represents the Sigmoid activation function. This represents pointwise convolution. Represents depthwise convolution. This indicates pixel rearrangement. The aforementioned local view mask covers the complete road image space formed by the recombination of multiple local views.

[0026] In one possible implementation, a cue encoder encodes a local view mask to obtain dense cue features, and then fuses these dense cue features with global view encoded features to obtain fused features. This process includes: using a cue encoder to downsample the local view mask using convolution to obtain downsampled mask features with the same spatial resolution as the global view encoded features; performing a nonlinear transformation on the downsampled mask features based on a Gaussian error linear unit activation function to obtain dense cue features; and adding the dense cue features and global view encoded features element-wise to obtain the fused features.

[0027] For example, the cue encoder receives a local view mask generated by a local view mask predictor as dense cue input. In one embodiment, the dense cue encoding module in the cue encoder can apply two layers of convolutions with a stride of 2 to the local view mask. By performing stepwise downsampling, the spatial resolution of the mask is reduced from 256×256 to 64×64, ensuring consistency with the spatial resolution of the global view encoded features output by the frozen SAM image encoder, thus obtaining the downsampled mask features. In this implementation, stepwise downsampling allows for the gradual extraction of high-level semantic information from the mask while reducing resolution, minimizing the loss of detail caused by direct large-step downsampling.

[0028] A Gaussian Error Linear Unit (GELU) activation function can be applied to the downsampled mask features to perform a nonlinear transformation, thereby enhancing the expressive power of the mask features and introducing nonlinear factors to obtain dense cue features. These dense cue features carry the spatial prior and road area distribution information from the local view mask.

[0029] The dense cue features and the global view encoded features can be fused element-wise, embedding road region spatial constraints from the local view mask into the global view encoded features, resulting in a fused feature. This fused feature simultaneously contains both global road structure context and local road region cue information.

[0030] In one embodiment, if no external mask input is available, the cue encoder can use a learnable maskless embedding to replace the mask feature and fuse it with the global view encoded feature to ensure model compatibility under conditions of missing cues. It should be noted that in this embodiment, the local view mask is used as the automatically generated dense cue, eliminating the need for manually provided point cues, box cues, or mask cues.

[0031] Step S104: Perform bidirectional cross-attention processing and upsampling on the fused features through a mask decoder to obtain the training road recognition result.

[0032] In one possible implementation, the fused features are subjected to bidirectional cross-attention processing via a mask decoder, including: inputting a cue token sequence consisting of a sparse cue embedding, an output mask token, and an intersection-union (IU) prediction token, along with the fused features, into the mask decoder; performing self-attention processing on the cue token sequence to obtain an updated cue token; performing token-to-image cross-attention processing with the updated cue token as the query and the fused features as the key and value to obtain an image-enhanced cue token; and performing image-to-token cross-attention processing with the fused features as the query and the image-enhanced cue token as the key and value to obtain decoded image features.

[0033] For example, the mask decoder can employ a bidirectional transformer structure. The bidirectional transformer performs self-attention processing on the cue token sequence, enabling information exchange between tokens to eliminate ambiguity and obtain an updated cue token. Then, it sequentially performs token-to-image cross-attention processing with the updated cue token as the query and fused features as the key and value, allowing the cue information to query road-related regions in the image and obtain an image-enhanced cue token. Finally, it performs image-to-token cross-attention processing with fused features as the query and the image-enhanced cue token as the key and value, enabling image features to be updated according to the cue information. This achieves bidirectional interaction between image features and cue features, obtaining decoded image features, and using the image-enhanced cue token as the decoded cue token. This bidirectional cross-attention mechanism can achieve efficient collaboration between cue information and image features while maintaining a lightweight model.

[0034] In one possible implementation, the fused features are upsampled by a mask decoder to obtain the training road recognition result, including: upsampling the decoded image features to obtain road image features with restored spatial resolution; determining the image enhancement cue token as the decoding cue token, mapping the output mask token in the decoding cue token to dynamic convolution kernel weights, and performing a dot product operation between the dynamic convolution kernel weights and the road image features to obtain a candidate road segmentation mask; mapping the intersection-union prediction token in the decoding cue token to a mask quality prediction value, and determining the training road recognition result from the candidate road segmentation mask based on the mask quality prediction value.

[0035] In one embodiment, the decoded image features are upsampled twice via transposed convolution to restore the spatial resolution from 64×64 to 256×256, resulting in road image features with restored spatial resolution. The output mask token is mapped to dynamic convolution kernel weights via a multilayer perceptron, and these weights are then multiplied by the restored spatial resolution road image features to generate candidate road segmentation masks for the road region. Simultaneously, the intersection-union (IU) prediction token is mapped to a scalar value via the multilayer perceptron as a mask quality prediction value, used to predict the IU between the current candidate road segmentation mask and the real road label. The highest quality segmentation result is then selected from the candidate road segmentation masks as the training road recognition result. In the above embodiment, the mask decoder outputs a binary segmentation mask for the road region, representing the recognition result between the road region and non-road regions.

[0036] Step S105: Determine the joint loss based on the training road recognition results, local view mask, and real road labels. Update the parameters of the multi-scale spatial aggregation adapter, local view mask predictor, and mask decoder based on the joint loss to obtain the trained road recognition model.

[0037] In one possible implementation, the joint loss is determined based on the training road recognition result, the local view mask, and the real road label, including: determining the contrast consistency loss based on the pixel-by-pixel difference between the training road recognition result and the local view mask; determining the segmentation loss based on the pixel-by-pixel classification difference and the degree of region overlap between the training road recognition result and the real road label; and determining the joint loss based on the contrast consistency loss and the segmentation loss.

[0038] In one possible implementation, the parameters of the multi-scale spatial aggregation adapter, local view mask predictor, and mask decoder are updated based on the joint loss to obtain the trained road recognition model. This includes: keeping the original parameters of the frozen SAM image encoder unchanged, updating the parameters of the multi-scale spatial aggregation adapter, local view mask predictor, and mask decoder according to the gradient of the joint loss; repeatedly extracting global view encoding features and local view encoding features, generating local view masks, decoding the training road recognition results, and updating the parameters until the training termination condition is met to obtain the trained road recognition model.

[0039] For example, the contrastive consistency loss, used to constrain the consistency between local view auto-completion and the final segmentation result, can be determined by the following formula: ; in, Total number of pixels For the first Predicted pixel values ​​at each location, For the local view mask at the 1st The pixel value at each position. To compare the loss of consistency.

[0040] The segmentation loss includes binary cross-entropy loss and Dice loss. The binary cross-entropy loss, used to measure pixel-by-pixel classification differences, can be determined by the following formula: ; in, For binary cross-entropy loss, For the first The actual road label pixel value at each location Total number of pixels For the first The predicted pixel value at each location.

[0041] Dice loss, determined based on the overlap between the predicted and actual road regions, is used to reduce the impact of sample imbalance between road and non-road regions and can be expressed as: ; in, For Dice's loss, For the first The actual road label pixel value at each location Total number of pixels For the first The predicted pixel value at each location.

[0042] The binary cross-entropy loss and the Dice loss can be combined with equal weights to obtain the segmentation loss: ; in, To divide the loss, For binary cross-entropy loss, This is a loss for Dice.

[0043] The segmentation loss and the contrast consistency loss can be added together to obtain the joint loss used for model training: ; in, To divide the loss, To compare consistency loss, This is a joint loss.

[0044] During training, the original parameters of the SAM image encoder can be kept unchanged, based on the joint loss. The gradient updates the parameters of the multi-scale spatial aggregation adapter, local view mask predictor, and mask decoder until the training termination condition is met, resulting in the trained road recognition model. The training termination condition can be: reaching a preset maximum number of training rounds (e.g., 300 rounds, or other values ​​set according to actual conditions); the validation set metrics not improving for multiple consecutive rounds (the intersection-over-union (IoU) or Dice similarity coefficient of the road recognition results can be calculated on the validation set; if no improvement is seen for multiple consecutive rounds (e.g., 20-50 rounds), the training termination condition is met); or the decrease in the joint loss on the validation set is less than a preset threshold (e.g., 10). -5 (or other values ​​set according to the actual situation), etc.

[0045] Step S106: Perform multi-view transformation on the road image to be identified to obtain multi-view input data to be identified, and input the multi-view input data to be identified into the trained road recognition model to obtain the road area recognition result.

[0046] For example, the same resizing, normalization, and multi-view transformations as in the training phase can be performed on the road image to be identified. The trained road recognition model sequentially obtains global view encoding features, local view encoding features, local view masks, fusion features, and candidate road segmentation masks, and outputs the road region recognition result based on the mask quality prediction value. The road region recognition result is a pixel-level classification result of the road region and non-road regions.

[0047] In one embodiment, image training is performed on a graphics processing unit (GPU) based on the PyTorch deep learning framework. The training images are uniformly adjusted to 256×256 pixels, and the AdamW optimizer and cosine annealing warm restart learning rate scheduling strategy are employed, with an initial learning rate of 2×10⁻⁶. -5 The batch size was 4, and the maximum number of training epochs was 300. To reduce the impact of random fluctuations during training, all experiments were run independently three times, and the average results were taken. During training, the SAM image encoder was frozen, and only the parameters of the multi-scale spatial aggregation adapter, local view mask predictor, and mask decoder were updated. The total number of trainable parameters was only about 2.75M, accounting for about 2.9% of the total number of model parameters.

[0048] The experiments used the Cityscapes urban road dataset, the KITTI road benchmark dataset, and the BDD100K driving dataset, and evaluated the road region recognition results using the intersection-over-union ratio (IoU) and Dice similarity coefficient. Experimental results show that multi-view input can supplement local information of distant roads and complex intersection areas; the multi-scale spatial aggregation adapter can complete road domain feature adaptation under the condition of freezing the SAM image encoder; and local view masks can provide dense cues to the cue encoder without human interaction.

[0049] This invention provides a road recognition method based on Multi-View Awareness (SAM). By converting training road images into a global view and multiple local views, the road recognition model simultaneously acquires the complete road structure and local region details. By embedding a multi-scale spatial aggregation adapter into the frozen SAM image encoder, road domain feature adaptation is achieved without updating the original parameters of the image encoder. Local view masks are generated by spatially stitching, cross-view fusion, and upsampling the encoded features of local views, transforming the spatial correspondences between local views into dense cues that can be processed by the cue encoder. By fusing the dense cue features with the encoded features of the global view and using a mask decoder for bidirectional cross-attention processing, local road cues can constrain the decoding of the global road region. By determining the joint loss and updating the model parameters based on the training road recognition results, local view masks, and real road labels, the training road recognition results are simultaneously constrained by both the local view masks and real road labels. Ultimately, this invention can complete road region recognition without relying on manual interactive prompts and reduces the parameter update range required for road domain adaptation, thereby reducing the risk of overfitting with limited training data and improving the model's generalization ability and deployment efficiency in complex road scenarios.

[0050] Please see Figure 2 , Figure 2 This is a schematic diagram of the hardware entity of an electronic device to which an embodiment of the present invention applies. The electronic device provided in this embodiment includes a processor 201, a communication interface 202, a memory 203, and a communication bus 204, wherein the processor 201, the communication interface 202, and the memory 203 communicate with each other via the communication bus 204. Memory 203 is used to store computer programs; When the processor 201 executes the program stored in the memory 203, it implements the steps in the above method embodiments.

[0051] The communication bus 204 mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 204 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus.

[0052] The communication interface 202 is used for communication between the above-mentioned electronic device and other devices.

[0053] The memory 203 may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory 203 may also be at least one storage device located remotely from the aforementioned processor.

[0054] The processor 201 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0055] The method provided in this invention can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc. No limitation is made herein; any electronic device that can implement this invention falls within the protection scope of this invention.

[0056] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps provided in the above-described method embodiments.

[0057] For the embodiments of the device / electronic device / storage medium, since they are basically similar to the method embodiments, the description is relatively simple. For specific details and beneficial effects, please refer to the description of the method embodiments.

[0058] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0059] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus (devices), or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects, all of which are collectively referred to herein as "modules" or "systems." Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The computer program may be stored / distributed in a suitable medium, provided with or as part of other hardware, or may take other distribution forms, such as via the Internet or other wired or wireless telecommunications systems.

[0060] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0061] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0062] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0063] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A road recognition method based on multi-view perception (SAM), characterized in that, include: Acquire training road images and their corresponding real road labels, perform multi-view transformation on the training road images, and obtain training multi-view input data including a global view and multiple local views; The training multi-view input data is input into the road recognition model to be trained, and global view coding features and local view coding features are extracted by a frozen SAM image encoder with an embedded multi-scale spatial aggregation adapter. The local view encoded features are spatially stitched, fused across views, and upsampled by a local view mask predictor to obtain a local view mask; the local view mask is encoded by a cue encoder to obtain dense cue features, and the dense cue features are fused with the global view encoded features to obtain fused features; The fused features are subjected to bidirectional cross-attention processing and upsampling by a mask decoder to obtain the training road recognition results. The joint loss is determined based on the training road recognition results, the local view mask, and the real road label. The parameters of the multi-scale spatial aggregation adapter, the local view mask predictor, and the mask decoder are updated based on the joint loss to obtain the trained road recognition model. A multi-view transformation is performed on the road image to be identified to obtain multi-view input data to be identified. The multi-view input data to be identified is then input into the trained road recognition model to obtain the road area recognition result.

2. The road recognition method based on multi-view perception SAM according to claim 1, characterized in that, The process of performing multi-view transformation on the training road image to obtain training multi-view input data including a global view and multiple local views includes: The training road images are flipped, resized, and normalized to obtain preprocessed road images; The preprocessed road image is used as the global view, and the preprocessed road image is divided into multiple local views according to adjacent spatial regions; The global view and each of the local views are adjusted to the same resolution and combined along the batch dimension to obtain the training multi-view input data.

3. The road recognition method based on multi-view perception SAM according to claim 1, characterized in that, The multi-scale spatial aggregation adapter includes an adaptive normalization layer, a lower projection layer, a multi-scale spatial aggregation module, and an upper projection layer; the extraction of global view coding features and local view coding features by the frozen SAM image encoder embedded with the multi-scale spatial aggregation adapter includes: The input features of the frozen SAM image encoder are normalized by the adaptive normalization layer, and the layer normalization result and the input features are adjusted by the learnable layer normalization weight parameter and the input weight parameter respectively to obtain the adaptive normalized features. The channel dimension of the adaptive normalized feature is reduced by the lower projection layer to obtain the lower projection feature; The lower projection features are processed sequentially by the multi-scale spatial aggregation module and the upper projection layer to obtain the upper projection features; the upper projection features are fused with the input features by residual to obtain the multi-scale spatial aggregation features; and according to the view type in the training multi-view input data, the multi-scale spatial aggregation features corresponding to each view are respectively determined as the global view encoding features or the local view encoding features.

4. The road recognition method based on multi-view perception SAM according to claim 3, characterized in that, The process of sequentially processing the lower projection features through the multi-scale spatial aggregation module and the upper projection layer to obtain the upper projection features includes: The downprojection features are extracted stepwise by increasing the receptive field through the serial convolution branch in the multi-scale spatial aggregation module, resulting in multiple cascaded scale features. The parallel convolutional branches in the multi-scale spatial aggregation module extract features from the downprojection features with different receptive fields to obtain multiple parallel scale features; and the multi-scale fusion features are determined based on each of the cascaded scale features and each of the parallel scale features. The channel dimension of the multi-scale fusion feature is recovered through the upper projection layer to obtain the upper projection feature.

5. The road recognition method based on multi-view perception SAM according to claim 1, characterized in that, The local view mask predictor includes a feature stitching module, a gated multilayer perceptron, and an upsampling module; The process of obtaining a local view mask by spatially concatenating, cross-view fusion, and upsampling the local view encoded features using a local view mask predictor includes: The feature stitching module stitches together the corresponding local view coding features according to the spatial position of each local view in the training road image to obtain complete local coding features. The gated multilayer perceptron performs nonlinear feature transformation and gate weight generation on the complete local encoded features, and then fuses the transformed features and gate weights element by element to obtain cross-view fused features. The channel features of the cross-view fused features are extracted by deep convolution, and the channel dimension of the obtained channel features is expanded. Upsampled features are obtained by distributing the expanded channel features to the spatial dimension through pixel rearrangement. The upsampling module processes the upsampled features using pointwise convolution and the Sigmoid activation function to obtain the local view mask.

6. The road recognition method based on multi-view perception SAM according to claim 1, characterized in that, The process involves encoding the local view mask using a cue encoder to obtain dense cue features, and then fusing these dense cue features with the global view encoded features to obtain fused features, including: The local view mask is downsampled using convolution through the prompt encoder to obtain a downsampled mask feature with the same spatial resolution as the global view encoded feature. The dense cue features are obtained by performing a nonlinear transformation on the downsampled mask features based on the Gaussian error linear unit activation function; The dense cue feature is added element by element to the global view encoding feature to obtain the fused feature.

7. The road recognition method based on multi-view perception SAM according to claim 1, characterized in that, The fused features are subjected to bidirectional cross-attention processing via a mask decoder, including: The cue token sequence, consisting of sparse cue embedding, output mask token, and intersection-union prediction token, is input into the mask decoder along with the fusion feature; The prompt token sequence is subjected to self-attention processing to obtain an updated prompt token; Using the updated hint token as the query and the fusion feature as the key and value, perform token-to-image cross-attention processing to obtain the image enhancement hint token; Using the fused features as the query and the image enhancement hint token as the key and value, cross-attention processing from image to token is performed to obtain the decoded image features.

8. The road recognition method based on multi-view perception SAM according to claim 7, characterized in that, The fused features are upsampled using a mask decoder to obtain the trained road recognition results, including: The decoded image features are upsampled to obtain road image features with restored spatial resolution; The image enhancement cue token is determined as the decoding cue token, the output mask token in the decoding cue token is mapped to the dynamic convolution kernel weight, and the dynamic convolution kernel weight is multiplied by the road image features to obtain the candidate road segmentation mask; The intersection-union prediction token in the decoding hint token is mapped to a mask quality prediction value, and the training road recognition result is determined from the candidate road segmentation mask based on the mask quality prediction value.

9. The road recognition method based on multi-view perception SAM according to claim 1, characterized in that, The step of determining the joint loss based on the trained road recognition results, the local view mask, and the real road labels includes: The contrast consistency loss is determined based on the pixel-by-pixel difference between the trained road recognition results and the local view mask. The segmentation loss is determined based on the pixel-by-pixel classification difference and the degree of region overlap between the trained road recognition results and the real road labels. The joint loss is determined based on the contrast consistency loss and the segmentation loss.

10. The road recognition method based on multi-view perception SAM according to claim 1, characterized in that, The process of updating the parameters of the multi-scale spatial aggregation adapter, the local view mask predictor, and the mask decoder based on the joint loss to obtain the trained road recognition model includes: Keeping the original parameters of the frozen SAM image encoder unchanged, the parameters of the multi-scale spatial aggregation adapter, the local view mask predictor, and the mask decoder are updated according to the gradient of the joint loss; Repeatedly extract global view encoding features and local view encoding features, generate local view masks, decode the training road recognition results and update parameters until the training termination condition is met, to obtain the trained road recognition model.

Images