A city scene semantic segmentation method based on data fusion and electronic equipment

Abstract

The application relates to a kind of city scene semantic segmentation method and electronic equipment based on data fusion, will obtain the data of city scene and input into heterogeneous feature fusion model, output mask prediction and category prediction, obtain the final semantic segmentation result;Wherein data includes color information and auxiliary information;The heterogeneous feature fusion model includes the mixed feature decoupling encoder connected in turn, multi-scale heterogeneous feature fusion block and mask decoder;The mixed feature decoupling encoder is used to encode the features of input color information and auxiliary information, respectively decoupling and enhancing from global scale and local scale, obtain global feature and local feature;The multi-scale heterogeneous feature fusion block is fused to obtain multi-scale fusion feature after decoupling and enhancing global feature and local feature, and the mask decoder is iteratively decoded to generate semantic segmentation result.The multi-scale fusion feature is fused to obtain multi-scale fusion feature after decoupling and enhancing global feature and local feature, and the mask decoder is iteratively decoded to generate semantic segmentation result.Compared with the prior art, the application has the advantages of improving the precision and robustness of semantic segmentation.

Description

Technical Field

[0001] This invention relates to the field of urban scene semantic segmentation, and in particular to an urban scene semantic segmentation method and electronic device based on data fusion. Background Technology

[0002] Accurate parsing and semantic segmentation of urban scenes are crucial for autonomous driving and mobile robotics technologies. This perception capability ensures the safe operation of autonomous driving systems, especially in complex urban environments. Traditionally, urban scene perception has relied primarily on color image data, as color images provide rich visual information about the environment. However, single-modal networks that depend solely on color images often struggle to handle scenes with low light, high dynamic range, or complex backgrounds.

[0003] With advancements in deep learning technology, models based on convolutional neural networks (CNNs) and Transformers have made significant progress in semantic image segmentation tasks. These techniques improve the accuracy and robustness of scene parsing by learning deep features of images. Nevertheless, many challenges remain when facing extreme weather conditions or low-light environments. For example, single color information may not provide sufficient detail in nighttime or backlit conditions, thus affecting the overall perceptual effect.

[0004] To address these issues, some studies have begun exploring the use of auxiliary information sources to enhance the semantic segmentation performance of RGB images. This auxiliary information includes, but is not limited to, depth information, thermal imaging, and surface normals. This data fusion method can collect information from different sensors and, through internal algorithmic processing, synthesize a more comprehensive scene resolution view. For example, depth information can help the system better understand the spatial relationships between objects, thermal imaging can highlight organisms or other heat sources in low-light conditions, and normal vector information can provide important clues about surface structure—all of which are difficult for RGB images to provide individually.

[0005] While existing multi-data source fusion methods have improved the performance of semantic segmentation, they still face several challenges in data processing and feature fusion. First, the characteristics of different data sources vary significantly, making the design of an efficient feature extraction and fusion strategy crucial for achieving high-quality data fusion. Furthermore, processing multiple data sources typically requires higher computational resources and more complex model structures, which may limit the deployment of algorithms in practical applications. For example, traditional fusion strategies often employ simple feature overlay or concatenation methods. While intuitive, these methods often overlook interactions and conflicts between features, resulting in low-quality fused features.

[0006] A search revealed Chinese invention patent publication number CN 117372698 A, which discloses a data fusion method, apparatus, and medium for semantic segmentation of road scenes. This method acquires the RGB information and corresponding normal vector information of a road scene as input to a heterogeneous feature fusion model for the road scene. The trained heterogeneous feature fusion model then outputs semantic category prediction masks for drivable areas and damaged road areas. The heterogeneous feature fusion model performs the following steps: S1, using two parallel encoders to encode the RGB information and normal vector information respectively, and fusing the encoded features using a multi-source feature fusion module to generate unified multi-source features; S2, using a deformable attention pixel decoder to generate multi-scale interactive features and mask features from the fused multi-source features; S3, initializing a constant number of query embeddings and using the mask features with the initialized... The process involves: S4. Initializing the attention mask using the query embedding combined with a masked multilayer perceptron; S5. Optimizing the query embedding using a specific scale feature from the multi-scale interaction features, guided by the attention mask, based on a stacked Transformer decoder; S6. Inputting the optimized query embedding into both the category multilayer perceptron and the masked multilayer perceptron to generate category predictions and mask embeddings; S7. Performing tensor multiplication on the mask embeddings and mask features to generate mask predictions. After filtering out empty categories at a specific threshold, post-processing is performed, and pixel-wise semantic prediction results are generated by tensor multiplication with the category predictions; S8. Scaling the mask predictions to obtain a new attention mask; S9. Repeating steps S4-S7 a preset number of times to continuously optimize the semantic segmentation results, and using the output of the last optimization as the final prediction result. This existing patent suffers from problems such as insufficient semantic segmentation accuracy, poor applicability (or generalization) to different types of heterogeneous data, and excessively high deployment costs due to the large number of model parameters.

[0007] Improving the accuracy of semantic segmentation in urban scenes has become a technical problem that needs to be solved. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a semantic segmentation method and electronic device for urban scenes based on data fusion.

[0009] The objective of this invention can be achieved through the following technical solutions:

[0010] According to one aspect of the present invention, a semantic segmentation method for urban scenes based on the fusion of color and auxiliary information is provided. The method includes: firstly acquiring data of the urban scene, including color information and auxiliary information, as input to a heterogeneous feature fusion model; and then using the trained heterogeneous feature fusion model to output mask prediction and category prediction to obtain the final semantic segmentation result.

[0011] The heterogeneous feature fusion model includes a hybrid feature decoupling encoder, a multi-scale heterogeneous feature fusion block, and a mask decoder connected in sequence. The hybrid feature decoupling encoder is used to encode the input color information and auxiliary information, decoupling and enhancing them at the global and local scales respectively to obtain global and local features. The multi-scale heterogeneous feature fusion block fuses the decoupled and enhanced global and local features to obtain multi-scale fused features. The mask decoder iteratively decodes the multi-scale fused features to generate semantic segmentation results.

[0012] Preferably, the auxiliary information includes depth, surface normal vector, thermal imaging information, and polarized light information.

[0013] More preferably, the depth is obtained by a stereo imaging system, a depth camera, or a lidar; the surface normal vector is obtained by a depth-normal vector estimator; the thermal imaging information is captured by a thermal sensor; and the polarization information is obtained by a polarization camera acquiring multiple images at different polarization angles and calculating the corresponding polarization angles.

[0014] Preferably, the process of obtaining the global and local features includes: firstly, in the hybrid feature decoupling encoder, a shared weight encoder is used to encode four layers of features at different scales; then, a global feature enhancement module and a local feature enhancement module are used to decouple and enhance the four-stage features generated by the shared weight encoder at the global and local scales, respectively.

[0015] More preferably, the shared weight encoder is a backbone encoder with shared weights and based on large kernel convolution.

[0016] More preferably, the weights of the global feature enhancement module and the local feature enhancement module are independent; the global feature enhancement module uses block embedding and multi-head self-attention sub-modules to enhance global features; the local feature enhancement module uses depthwise separable convolutional layers to enhance local features.

[0017] Preferably, the process of obtaining the multi-scale fusion features includes:

[0018] The decoupled global and local features are fed into a dual-branch multi-scale heterogeneous feature fusion block for fusion. The global features are fused using a global feature fusion module based on cross-attention and channel attention, while the local features are fused using a local feature fusion module based on convolution. The fused global and local features are first summed element-wise, and then a fusion feature generation and enhancement module based on spatial attention is used to unify them into multi-scale fused features.

[0019] The fusion process involves fusing features at each scale separately.

[0020] More preferably, during multi-scale fusion, a learnable coefficient with an initial value of 1 is added to the local features to dynamically adjust the weights of the global and local features.

[0021] Preferably, the semantic segmentation result generation process is as follows: the mask decoder first uses deformable attention to generate multi-scale interaction features and mask features, then uses several layers of stacked Transformer decoders to generate mask predictions and category predictions, and multiplies the two to obtain the final semantic segmentation result.

[0022] According to another aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described thereon.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] (1) This invention introduces a variety of auxiliary information, including data types such as depth, surface normal vector, thermal information and polarization information that are widely used in the industry. The auxiliary information and RGB information are used together as input to the heterogeneous feature fusion model. In the encoding stage, an efficient backbone encoder based on a weight sharing strategy is used, which saves a lot of parameters compared with the traditional weight separation encoding strategy. At the same time, an innovative global feature enhancement module based on self-attention and a local feature enhancement module based on convolutional attention are developed to enhance the decoupled global features and local features respectively. The network accuracy is maintained through attention operations at different scales. The performance is better than the encoder based on the weight separation strategy, which helps to improve the accuracy and robustness of semantic segmentation in complex urban scenarios.

[0025] (2) In the semantic understanding task of complex urban scenes, this invention proposes the idea of ​​feature decoupling for the first time, and uses Transformer attention and convolutional blocks with different receptive fields in parallel to fuse global and local features to obtain better heterogeneous feature fusion.

[0026] (3) This invention uses a query embedding-based Transformer decoder, drawing on the application of the Transformer architecture in the field of object detection. Unlike traditional pixel-by-pixel classification semantic segmentation methods, this technology adopts an ensemble prediction paradigm. It generates category-level semantic classification results by predicting a class instance mask with the same number of query embeddings at once, that is, by using the class confidence shared between different instances. This achieves the semantic segmentation task of urban scenes. This method not only improves the efficiency of prediction, but also improves the accuracy of semantic segmentation.

[0027] (4) The model of the present invention can be adapted to a variety of backbone encoders with different architectures, such as encoders based on convolutional networks and encoders based on Transformers, or even lighter encoders based on depthwise separable convolutions, and can achieve accurate segmentation results. It can be freely replaced according to actual hardware resources. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the network model architecture for semantic segmentation of urban scenes in this invention;

[0029] Figure 2(a) is a schematic diagram of the color images of the network model in this invention running on the Cityscapes dataset;

[0030] Figure 2(b) is a schematic diagram of the surface normal information of the network model in this invention running on the Cityscapes dataset;

[0031] Figure 2(c) is a schematic diagram of the pixel-level semantic segmentation results of the network model in the Cityscapes dataset in this invention;

[0032] Figure 3(a) is a schematic diagram of the color images of the network model in this invention running on the MFNet dataset;

[0033] Figure 3(b) is a schematic diagram of the thermal imaging information of the network model running on the MFNet dataset in this invention;

[0034] Figure 3(c) is a schematic diagram of the pixel-level semantic segmentation results of the network model in this invention running on the MFNet dataset;

[0035] Figure 4 This is a schematic diagram of the semantic segmentation process for urban scenes in this invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0037] This invention proposes a semantic segmentation method for urban scenes based on data fusion, specifically designed for semantic segmentation of urban scenes. The method employs a dual-branch heterogeneous feature fusion framework, effectively integrating information from different sensors and improving perception accuracy and robustness in complex environments. Furthermore, an attention-based feature enhancement module is introduced, which optimizes the feature fusion effect by appropriately recalibrating features at different levels. This algorithm not only improves the efficiency of multi-source data fusion but also reduces computational complexity and model parameters, making it more suitable for deployment on resource-constrained mobile devices.

[0038] Through the above technical improvements, this invention not only demonstrates superior performance on the color-representation normal vector task, but also exhibits excellent versatility and adaptability on datasets of color-thermal information and color-polarization information. This achievement indicates that in autonomous driving, mobile robotics, and other applications requiring advanced environmental perception, this invention achieves high-precision semantic understanding at a relatively low cost, and has broad application prospects.

[0039] This embodiment relates to a semantic segmentation method for urban scenes based on data fusion. To address the semantic understanding problem of complex urban scenes, a heterogeneous feature fusion model is proposed, which uses data fusion based on scene color (RGB) information and corresponding auxiliary information as input, and outputs a pixel-level semantic mask.

[0040] like Figure 1 This heterogeneous feature fusion model uses a Transformer-related architecture and a convolutional neural network, including a hybrid feature decoupling encoder, a multi-scale heterogeneous feature fusion block, and a mask decoder connected in sequence. The hybrid feature decoupling encoder includes a shared weight encoder, a global feature enhancement module, and a local feature enhancement module. The multi-scale heterogeneous feature fusion block (a two-branch multi-scale heterogeneous feature fusion block) includes a global feature fusion module, a local feature fusion module, and a fusion feature generation and enhancement module for fusing global and local features.

[0041] First, a shared-weight encoder (using a weight-sharing encoder) encodes features from the RGB and auxiliary information inputs, and then decouples and enhances the resulting features using weight-independent global and local feature enhancement modules, respectively. Subsequently, the decoupled global and local features are fed into a dual-branch, multi-scale heterogeneous feature fusion block, which utilizes Transformer attention and convolutional attention with different receptive fields to fuse features. Finally, the resulting multi-scale fused features are fed into a mask decoder for iterative decoding. A non-linear transformation is applied to the iterative features to generate the corresponding semantic mask, achieving accurate semantic segmentation.

[0042] Specific technical solutions are as shown in 2.

[0043] It mainly includes the following steps:

[0044] Step S1: Obtain the auxiliary information of the required urban scene. The depth can be obtained by sensors such as stereo imaging system, depth camera, and lidar. The normal vector information can be obtained by the existing depth-normal vector estimator. The thermal imaging information can be obtained by thermal sensor. The polarization light information can be obtained by polarization camera acquiring multiple images at different polarization angles and calculating the corresponding polarization angles.

[0045] Step S2: Use the RGB information of the urban scene and the corresponding auxiliary information as input to the network encoder and feed them into the hybrid feature decoupling encoder;

[0046] Step S3: In the hybrid feature decoupling encoder, a shared weight encoder (a backbone network bridge based on large kernel convolution with shared weights) is used to encode heterogeneous information to obtain four layers of features at different scales.

[0047] Step S4: Use the global feature enhancement module to decouple the four layers of features at different scales generated by the shared weight encoder from the global scale;

[0048] Step S5: Use the local feature enhancement module to decouple the four layers of features at different scales generated by the shared weight encoder from the local scale.

[0049] Step S6: The decoupled global and local features are fed into a dual-branch multi-scale heterogeneous feature fusion block for fusion, wherein the features obtained in each encoding stage are fused separately.

[0050] Step S7: In the multi-scale heterogeneous feature fusion block, the global features are fused using a global feature fusion module based on cross attention and channel attention;

[0051] Step S8: In the multi-scale heterogeneous feature fusion block, local features are fused using a local feature fusion module based on convolution modules;

[0052] Step S9: In the multi-scale heterogeneous feature fusion block, the fused global and local features are first summed element by element, and then the spatial attention-based fusion feature generation and enhancement module is used to unify them into fused features.

[0053] Step S10: Feed the obtained multi-scale fused features into the mask decoder. Specifically, the mask decoder first uses deformable attention to generate multi-scale interaction features and mask features, and then uses several layers of stacked Transformer decoders to generate mask predictions and category predictions. The two are multiplied to obtain the final semantic prediction result.

[0054] The four layers of features at different scales are generated by the four stages of the shared weight encoder.

[0055] Step S11: During the network training phase, coarse feature extraction is achieved using encoder weights pre-trained on large-scale data. The feature fusion module and decoder are trained using different learning rates. At the same time, the encoder is fine-tuned so that the network gradually fits the objective function that can achieve urban scene understanding.

[0056] Step S12: When training the model, the proposed decoder is trained based on specific weight coefficients, using the cross-entropy loss function and the geometric similarity loss function as evaluation criteria.

[0057] Step S13: When training the model, after each round, use the validation set to evaluate the model's performance and save the weights of the best-performing model for subsequent inference.

[0058] Step S14: When using the model for inference, input the RGB image of the urban scene and auxiliary information, and the network outputs the semantic segmentation result of the urban scene after computation.

[0059] S1, S2, S3, S4, S5, S6, S7, S8, S9, and S10 represent the main structure of the network model, while S11, S12, S13, and S14 represent the training-testing steps when the network is applied to a specific scenario.

[0060] Furthermore, data fusion refers to using scene RGB information and corresponding auxiliary information (including but not limited to depth, surface normal vector, thermal information and polarized light information) as network input in parallel, and finally outputting a pixel-level multi-class semantic category prediction mask for urban scenes;

[0061] Furthermore, the RGB information and corresponding auxiliary information are encoded using a weight-sharing encoder;

[0062] Furthermore, the weighted global feature enhancement module uses block embedding and multi-head self-attention submodules to enhance features;

[0063] Furthermore, the weight-independent local feature enhancement module uses depthwise separable convolutional layers to enhance features;

[0064] Furthermore, the dual-branch multi-scale heterogeneous feature fusion block includes a parallel global feature fusion module and a local feature fusion module, as well as a unified feature generation and enhancement module that fuses global and local features; wherein: the global feature fusion module is only responsible for fusing the global features output by the global feature enhancement module, and it includes a cross attention module and a channel attention module; the local feature fusion module is only responsible for fusing the local features output by the local feature enhancement module, and it uses convolutional layers and gating mechanisms to fuse heterogeneous local features;

[0065] Furthermore, when summing the elements of the fused global and local features to obtain the fused features, a learnable coefficient with an initial value of 1 is added to the local features, allowing the algorithm to dynamically adjust the weights of the global and local features;

[0066] Furthermore, the fusion feature generation and enhancement modules calculate spatial attention coefficients in both height and width directions using global average pooling.

[0067] Furthermore, the fusion feature generation and enhancement module utilizes Hadamard multiplication to multiply the spatial attention coefficients with the fusion features;

[0068] Furthermore, the mask decoder includes a pixel decoder module based on deformable attention to further enhance the multi-scale fused features;

[0069] Furthermore, the mask decoder includes a stacked Transformer decoder that receives these enhanced multi-scale features to generate the final semantic prediction mask;

[0070] Furthermore, the stacked Transformer decoder initializes a constant number of query embeddings and iteratively incorporates multi-scale fused features into the query features through a multi-layered mask-guided cross-attention mechanism.

[0071] Furthermore, each decoder layer combines a certain scale feature from the multi-scale features as a key (K) and a value (V), and optimizes the query embedding (Q) under the guidance of the attention mask to generate class prediction and mask embedding;

[0072] Furthermore, in the final stage of decoding, the processed mask embedding is multiplied by the mask features using tensor multiplication to generate mask predictions. After post-processing to filter out empty categories, the semantic prediction results are generated pixel-by-pixel through tensor multiplication with the category predictions.

[0073] Furthermore, a decoder based on a fully convolutional network (FCN) is used to process the 1 / 4 scale features in parallel to directly predict the semantic classification results and enhance the local semantics of the high-scale feature map.

[0074] Furthermore, open-source encoder pre-trained weights obtained on the ImageNet-1k dataset are used as the initial weights for the shared weight encoder at the beginning of network training.

[0075] Furthermore, the loss function used for training is:

[0076]

[0077] in Represents mask loss, Represents the class loss, λ mask and λ cls It is the weighting coefficient of mask loss and category loss; and Representing the cross-entropy loss function and the set similarity loss function, respectively, they are defined as:

[0078]

[0079] Where y and Let represent the semantic label and prediction result of the i-th sample in the j-th category, respectively. ∈ is a very small constant to prevent the denominator from being zero. N is the number of samples and C is the number of categories.

[0080] In the specific implementation of this invention, the network needs to be fine-tuned according to the specific use scenario to obtain the most suitable network parameter weights, and then model inference in the specific scenario can be performed to achieve a more accurate prediction effect. Directly using pre-trained weights cannot guarantee that the network will fully perform at its best.

[0081] This embodiment also relates to the application of a semantic segmentation method for urban scenes based on the fusion of color and auxiliary information.

[0082] The CityScapes dataset is a widely used urban scene dataset containing 2975 high-resolution stereo images for training. The surface normal information used in the experiments was extracted from the depth images using RAFT-Stereo. The RGB images from the CityScapes dataset (Figure 2(a)) and the surface normal information (Figure 2(b)) were input into the trained semantic segmentation network, resulting in the pixel-level semantic segmentation result shown in Figure 2(c), which is generated by the semantic segmentation network.

[0083] The MFNet dataset is a widely used dataset of urban driving scene data, consisting of 1569 pairs of synchronized RGB and thermal images. In this embodiment, the RGB images (Figure 3(a)) and thermal information (Figure 3(b)) of the MFNet dataset are also input into the trained semantic segmentation network to obtain the result shown in Figure 3(c), which is the pixel-level semantic segmentation result generated by the semantic segmentation network.

[0084] In summary, this invention addresses the need for urban environment perception in challenging scenarios such as mobile robots and autonomous vehicles by proposing a heterogeneous feature fusion network that integrates RGB and auxiliary information (such as depth, surface normal vectors, thermal information, and polarized light information). Through a multi-scale perception and heterogeneous feature decoupling strategy, the accuracy and robustness of urban scene analysis are effectively improved.

[0085] This network uses a backbone network based on a large-kernel convolutional neural network to encode RGB color images and auxiliary information as heterogeneous features. Simultaneously, it uses a global feature enhancement module based on a self-attention mechanism and a local feature enhancement module based on depthwise separable convolution to decouple the heterogeneous features into global and local features. Then, a dual-branch feature fusion module extracts and fuses global and local features from the heterogeneous data across multiple receptive fields. Specifically, for global features, a global feature fusion module based on cross-attention and channel attention is used; for local features, a local feature fusion module based on convolution is used. The fused global and local features are first summed element-wise, and then a fusion feature generation and enhancement module based on spatial attention is used to unify them into a fused feature. Finally, the fused multi-scale features are fed into a Transformer-based mask classification decoder, generating corresponding semantic masks through iterative decoding to achieve accurate semantic segmentation. This invention overcomes the problem of poor universality and generality of mainstream data fusion networks when fusing different types of auxiliary information. This invention maintains good applicability and accuracy when processing different types of data.

[0086] This invention, as a leading semantic segmentation method for urban scenes, performs excellently in road drivable area detection, road damage detection, and even more general urban scene semantic segmentation. It is beneficial to improve the environmental perception capabilities of application fields such as mobile robots and autonomous driving in urban scenes, and provide more reliable real-time environmental data for subsequent planning and decision-making processes.

[0087] The electronic device of this invention includes a central processing unit (CPU) and a graphics processing unit (GPU), which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from storage units into random access memory (RAM). The RAM can also store various programs and data required for device operation. The CPU, GPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0088] Multiple components in the device are connected to the I / O interface, including: input units such as keyboards and mice; output units such as various types of displays and speakers; storage units such as disks and optical discs; and communication units such as network interface cards (NICs), modems, and wireless transceivers. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0089] The processing unit executes the various methods and processes described above, such as methods S1 to S14. For example, in some embodiments, methods S1 to S14 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of methods S1 to S14 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S14 by any other suitable means (e.g., by means of firmware).

[0090] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0091] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0092] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0093] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A semantic segmentation method for urban scenes based on data fusion, characterized in that, The method includes: first, acquiring data of urban scenes, including color information and auxiliary information, as input to a heterogeneous feature fusion model; then, using the trained heterogeneous feature fusion model to output mask prediction and category prediction, and obtaining the final semantic segmentation result. The heterogeneous feature fusion model includes a hybrid feature decoupling encoder, a multi-scale heterogeneous feature fusion block, and a mask decoder connected in sequence. The hybrid feature decoupling encoder is used to encode features of the input RGB color image and auxiliary information, decoupling and enhancing them at the global and local scales respectively to obtain global features and local features. The multi-scale heterogeneous feature fusion block fuses the decoupled and enhanced global and local features to obtain multi-scale fused features. The mask decoder iteratively decodes the multi-scale fused features to generate semantic segmentation results. The process of obtaining the global and local features includes: firstly, in the hybrid feature decoupling encoder, a shared weight encoder is used to encode four layers of features at different scales; then, a global feature enhancement module and a local feature enhancement module are used to decouple and enhance the four-stage features generated by the shared weight encoder at the global and local scales, respectively. The weights of the global feature enhancement module and the local feature enhancement module are independent; the global feature enhancement module uses block embedding and multi-head self-attention sub-modules to enhance global features; the local feature enhancement module uses depthwise separable convolutional layers to enhance local features. The process of obtaining the multi-scale fusion features includes: The decoupled global and local features are fed into a dual-branch multi-scale heterogeneous feature fusion block for fusion. The global features are fused using a global feature fusion module based on cross-attention and channel attention, while the local features are fused using a local feature fusion module based on convolutional layers and gating mechanisms. The fused global and local features are first summed element-wise, and then a fusion feature generation and enhancement module based on spatial attention is used to unify them into multi-scale fused features. The fusion process involves fusing features at each scale separately.

2. The urban scene semantic segmentation method based on data fusion according to claim 1, characterized in that, The auxiliary information includes depth, surface normal vector, thermal imaging information, and polarized light information.

3. The urban scene semantic segmentation method based on data fusion according to claim 2, characterized in that, The depth is obtained by a stereo imaging system, a depth camera, or a lidar; the surface normal vector is obtained by a depth-normal vector estimator; the thermal imaging information is captured by a thermal sensor; and the polarization information is obtained by a polarization camera acquiring multiple images at different polarization angles and calculating the corresponding polarization angles.

4. The urban scene semantic segmentation method based on data fusion according to claim 1, characterized in that, The shared weight encoder is a backbone encoder with shared weights and based on large kernel convolution.

5. The urban scene semantic segmentation method based on data fusion according to claim 1, characterized in that, In multi-scale fusion, a learnable coefficient with an initial value of 1 is added to the local features to dynamically adjust the weights of the global and local features.

6. The urban scene semantic segmentation method based on data fusion according to claim 1, characterized in that, The semantic segmentation result generation process is as follows: the mask decoder first uses deformable attention to generate multi-scale interaction features and mask features, then uses several layers of stacked Transformer decoders to generate mask predictions and category predictions, and multiplies the two to obtain the final semantic segmentation result.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 6.

Images