Semantic segmentation method and device with enhanced depth estimation, equipment and medium

Abstract

The invention provides a depth estimation enhanced semantic segmentation method and device, equipment and a medium. The method comprises the following steps: extracting a corresponding depth image from an acquired RGB image by using a depth estimation large model; constructing a scene perception model, wherein the scene perception model comprises a coding layer and a decoding layer; the coding layer comprises an RGB image coding branch and a depth image coding branch, and is used for correcting and fusing the characteristics of the RGB mode and the depth mode output by the RGB feature extraction layer and the depth feature extraction layer based on data fusion modules arranged between the coding layers respectively to obtain the first fusion characteristics of each layer; a first fusion feature obtained after fusion of the data fusion modules except the last layer of data fusion module is input to a multi-scale fusion decoding module, step-by-step recovery of a feature map is achieved through a multi-scale feature fusion module and a frequency perception feature fusion device of the multi-scale fusion decoding module, and finally a semantic segmentation result is obtained. According to the method, scene perception is accurate, and meanwhile, only the minimum cost is needed.

Description

Technical Field

[0001] The present invention relates to the field of semantic segmentation technology, and in particular to a semantic segmentation method, apparatus, device and medium with enhanced depth estimation. Background Art

[0002] With the rapid development of urban rail transit, the demand for accurate perception of the railway environment has increased significantly for tasks such as condition monitoring, intrusion detection, and autonomous driving. Single-modal semantic segmentation methods are typically simple and efficient, but are susceptible to interference from environmental changes. Multimodal semantic segmentation tasks utilize multiple sensors to complement each other, resulting in higher accuracy and robustness. For example, RGB-D semantic segmentation utilizes depth information to complement the color information and depth geometry of an image. However, in railway environments, obtaining accurate and reliable depth information while balancing cost is a primary consideration for practical applications of scene perception systems for urban rail transit.

[0003] Existing semantic segmentation methods for rail scenes only focus on the train operation area and do not perform multi-category segmentation on the entire scene. Multimodal segmentation methods for rail scenes are even more limited by the limitations of the dataset and are rarely explored. In practical applications, depth information in rail transit scenes is often difficult to obtain. Usually, depth information is obtained through depth cameras or lidar. The depth map generated by the depth camera has a limited range (usually no more than 20 meters) and is expensive. LiDAR adds an additional computational burden due to the large number of point clouds. In addition, the sparsity of the point cloud also makes it unable to provide sufficiently rich and dense depth information. Therefore, how to simply obtain reliable depth information is also an issue that needs to be considered.

[0004] At the technical level, complex scene perception in rail transit environments still faces the following challenges: first, the noise in the image and the uncertainty of the object scale, such as Figure 1 The following is a schematic diagram of the problem of the dominant characteristics of rail transit scenarios, where Figure 1 (a) The two pictures (above and below) are scenes in foggy and rainy weather. Figure 1 (b) Group of pictures shows the scene under two lighting conditions. Figure 1 (c) The image group is a scene with uncertain object size. For example, due to the large amount of noise caused by weather conditions such as rain, snow, and fog, as well as motion blur and rapid changes in illumination caused by high-speed trains, these factors bring great difficulties to the data fusion of multi-source sensors. Second, in the track scene, such as Figure 1(c) In the image group, important objects to be perceived (such as utility poles, traffic lights, and railroad tracks) are often complex and easily confused. In a typical layered encoder-decoder network structure, information loss of these objects is even more severe, resulting in boundary displacement problems, which interferes with fine-grained semantic segmentation in rail transit scenes.

[0005] Based on the above analysis, there is an urgent need for a low-cost, high-precision scene understanding method suitable for rail transit scenarios to improve the automation and accuracy of environmental perception in intelligent rail transit scenarios. Summary of the Invention

[0006] To overcome the problems existing in the related art, the present disclosure provides a semantic segmentation method, apparatus, device and medium with enhanced depth estimation to solve the technical problems in the related art.

[0007] One or more embodiments of this specification provide a semantic segmentation method with enhanced depth estimation, including the following steps:

[0008] Use the depth estimation model to extract the corresponding depth image from the collected RGB image, and obtain the RGB image and depth image of the same scene;

[0009] Construct a scene perception model, which includes a coding layer and a decoding layer; the coding layer includes an RGB image coding branch and a depth image coding branch, which process RGB images and depth images respectively; based on a data fusion module respectively set between each coding layer, it is used to correct and fuse the RGB modality and depth modality features output by the RGB feature extraction layer and the depth feature extraction layer to obtain the first fused feature of each layer;

[0010] The first fusion feature obtained after fusion of the data fusion modules except the last layer of data fusion module is input into the multi-scale fusion decoding module. The multi-scale feature fusion module of the multi-scale fusion decoding module realizes the interactive mapping of different resolution features between the first fusion features of each decoding layer and the first fusion features of the adjacent layer, and obtains the second fusion feature that fuses the adjacent layers and different resolutions. Each second fusion feature is output as each coding layer and jumps to each decoder layer to set a frequency-aware feature fusion device. Except for the frequency-aware feature fusion device corresponding to the deepest decoding layer, each frequency-aware feature fusion device receives the feature output by the next decoding layer and fuses it with the second fusion feature to obtain a third fusion feature. The third fusion feature is added to the up-sampled feature of the corresponding decoding layer and input to the next decoding layer. The first fusion feature obtained after fusion of the last data fusion module is input into the multi-scale ASPP module, and the output of the multi-scale ASPP module is input to the frequency-aware feature fusion device and decoder corresponding to the deepest decoding layer to realize step-by-step recovery of the feature map, and finally obtain the semantic segmentation result.

[0011] Furthermore, the data fusion module is used to add the first fusion features obtained by processing to the original RGB modality and depth modality respectively to achieve feature enhancement, and the enhanced RGB modality is jump-connected to the next RGB feature extraction layer, and the enhanced depth modality is jump-connected to the next depth feature extraction layer respectively, and then the first fusion features fused by each data fusion module except the last layer of data fusion module are input into the decoder.

[0012] Furthermore, the data fusion module includes a complementary calibration module and a fusion and enhancement module;

[0013] The complementary calibration module is responsible for calibrating the corresponding feature pixel positions between the RGB modality and the depth modality and filtering the noise information in the modality; the fusion and enhancement module splices the calibrated RGB modality and the depth modality in the channel dimension, obtains the multi-dimensional information of RGB features and depth features through different gated attention mechanisms, and then aggregates them and adds the calibrated RGB features and depth features respectively to enhance the image, and the aggregated features are introduced into the decoder.

[0014] Furthermore, the complementary calibration module CRM implements the feature processing process specifically including the following steps:

[0015] First, the input RGB image and depth image are spliced ​​together, and then the attention vector of the spliced ​​features is extracted through the coordinate attention mechanism;

[0016] The two attention vectors are concatenated and fused, and the fused feature maps are processed sequentially through the KAN network, batch normalization layer, activation function layer, and split layer to obtain two independent tensors of the rectified RGB image and two independent tensors of the rectified depth image. These are then multiplied with the RGB image and depth image to obtain filtered RGB features and depth features respectively.

[0017] The attention cross strategy is adopted to obtain the calibrated RGB features and the calibrated depth features.

[0018] Furthermore, the image processing process of the fusion and enhancement module specifically includes the following steps:

[0019] The calibrated RGB features and the calibrated depth features are spliced ​​in the channel dimension;

[0020] The spliced ​​features are processed based on the gated attention mechanism, aggregation, separation and activation to obtain the calibrated RGB feature spatial attention weights and the depth feature spatial attention weights respectively;

[0021] The spliced ​​features are then processed based on the linear gated attention mechanism, aggregation, separation and activation to obtain the global attention weights of the calibrated RGB features and the global attention weights of the depth features respectively;

[0022] Based on the obtained spatial attention weights and global attention weights, the first fused features of each coding layer are obtained by weighted fusion with the calibrated RGB features and depth features;

[0023] The first fusion features are fused with the calibrated RGB features and depth features to obtain enhanced RGB features and enhanced depth features. The enhanced RGB features and enhanced depth features are passed to the next RGB image coding layer and depth image coding layer to achieve layer-by-layer encoding, and the first fusion features of each layer are input to the decoder.

[0024] Furthermore, the multi-scale fusion decoding module implements the feature interaction process specifically including the following steps:

[0025] Based on the first fusion features obtained by the data fusion module of each layer as input, the multi-scale feature fusion module realizes the interactive fusion of each first fusion feature with the first fusion feature corresponding to the adjacent coding layer to realize the region of interest of the first fusion features of the adjacent layers, thereby obtaining high-resolution features that fuse the feature information of adjacent layers and different resolutions, and respectively introduced into the decoding layer corresponding to the decoder for residual supplement.

[0026] Furthermore, the multi-scale feature fusion module implements the feature interaction process specifically including the following steps:

[0027] The first fused features of different resolutions are adjusted to the same size through convolution or transposed convolution operations, and the first fused features of the target layer are enhanced through the first coordinate attention mechanism. The enhanced first fused features are extracted through the second coordinate attention mechanism to obtain the weights of the region of interest, mapped and applied to the first fused features of the adjacent layer, and then fused with the enhanced first fused features to obtain the second fused features.

[0028] Furthermore, the scene perception model consists of two parts: a teacher model and a student model. The backbone networks of the teacher model and the student model use the ResNet series model; the convolutional layers that realize attention fusion in the data fusion module and the multi-scale feature interaction module in the student model are replaced by necked linear layers, and the loss function is composed of the basic loss and the cross-image global dependency loss.

[0029] One or more embodiments of this specification provide a semantic segmentation device with enhanced depth estimation, a data acquisition and processing module for extracting a corresponding depth image from a collected RGB image using a large depth estimation model, and obtaining an RGB image and a depth image of the same scene;

[0030] A model building module is used to build a scene perception model, which includes an encoding layer and a decoding layer;

[0031] The coding layer includes an RGB image coding branch and a depth image coding branch, which process RGB images and depth images respectively; a data fusion module is set between each coding layer to correct and fuse the features of the RGB mode and depth mode output by each RGB feature extraction layer and the depth feature extraction layer to obtain the first fusion feature of each layer; the first fusion feature obtained after fusion of the data fusion module except the last layer of data fusion module is input to the multi-scale fusion decoding module, and the multi-scale fusion decoding module realizes the interactive mapping of different resolution features between the first fusion feature of each decoding layer and the first fusion feature of the adjacent layer, and obtains the second fusion feature that fuses adjacent layers and different resolutions, and each The second fusion feature is output as each encoding layer and jumps to each decoder layer to set a frequency perception feature fusion device. Except for the frequency perception feature fusion device corresponding to the deepest decoding layer, each frequency perception feature fusion device receives the feature output by the next decoding layer and fuses it with the second fusion feature to obtain a third fusion feature. The third fusion feature is added to the up-sampled feature of the corresponding decoding layer and then input to the next decoding layer; the first fusion feature obtained after the last data fusion module is input to the multi-scale ASPP module, and the output of the multi-scale ASPP module is input to the frequency perception feature fusion device and decoder corresponding to the deepest decoding layer to realize step-by-step recovery of the feature map and obtain the semantic segmentation result;

[0032] The semantic segmentation module is used to obtain the depth image from the collected RGB image based on the data acquisition and processing module, and then input it into the scene perception model constructed by the model construction module to obtain the semantic segmentation result.

[0033] One or more embodiments of the present specification provide a method including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the semantic segmentation method with enhanced depth estimation as described in any one of the above items is implemented.

[0034] One or more embodiments of this specification provide a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the semantic segmentation method with enhanced depth estimation as described in any one of the above items is implemented.

[0035] The present disclosure provides a depth estimation-enhanced semantic segmentation method, device, equipment and medium, which have the advantages of expanding RGB data to RGB-D mode through an unsupervised depth estimation large model, thereby constructing RGB-D data, and then designing cross-modal data complementary calibration fusion modules for the noise problem and multi-scale features of the railway environment, so as to enhance the weight of the dominant features in the railway, reduce the interference of irrelevant background or low-value areas, capture diverse image information, and the set multi-scale feature interactive fusion module improves the segmentation accuracy and robustness of the model; then, the frequency-aware feature fuser set in the decoding layer is used to embed frequency-aware fusion into decoding, and the final semantic output is obtained through a fine-grained local frequency-aware fusion decoding strategy, which effectively reduces the mutual interference of different frequencies and improves the boundary segmentation performance. The device provided in this embodiment is suitable for scenes with high noise, multi-category, multi-scale and difficult-to-acquire depth information, such as railways, and can effectively deal with the high noise problem in the railway environment, while accurately perceiving the scene, with minimal cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0037] Figure 1 A group of schematic diagrams of problems existing in the dominant features of rail transit scenarios provided for one or more embodiments of this specification;

[0038] Figure 2 A flowchart of a semantic segmentation method with enhanced depth estimation provided by one or more embodiments of this specification;

[0039] Figure 3 A diagram of the network structure of a scene perception model provided for one or more embodiments of this specification;

[0040] Figure 4 A structural diagram of a data fusion module provided for one or more embodiments of this specification;

[0041] Figure 5 A structural diagram of a multi-scale feature fusion module provided for one or more embodiments of this specification;

[0042] Figure 6 A structural diagram of the frequency-aware local fusion decoding process provided in one or more embodiments of this specification;

[0043] Figure 7A diagram showing the framework of a knowledge distillation method for semantic segmentation provided in one or more embodiments of this specification;

[0044] Figure 8 Eight scene graph semantic segmentation result graphs implemented by the method of the present invention according to one or more embodiments of this specification, wherein (a) row of graphs are eight scene graphs with semantic segmentation, and (b) row of graphs are semantic segmentation result graphs obtained for each of the eight scene graphs;

[0045] Figure 9 A system block diagram for implementing a semantic segmentation method provided in one or more embodiments of this specification;

[0046] Figure 10 A block diagram of a semantic segmentation device with enhanced depth estimation provided by one or more embodiments of this specification;

[0047] Figure 11 A schematic diagram of the structure of a computer device provided in one or more embodiments of this specification. DETAILED DESCRIPTION

[0048] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0049] The present invention will be described in detail below with reference to specific implementation methods and the accompanying drawings.

[0050] Method Example

[0051] According to an embodiment of the present invention, a semantic segmentation method with enhanced depth estimation is provided, such as Figure 2 FIG. 1 is a flowchart of a semantic segmentation method with enhanced depth estimation provided in this embodiment. The semantic segmentation method with enhanced depth estimation according to an embodiment of the present invention includes:

[0052] Step S1, data acquisition and processing: Use the depth estimation model to extract the corresponding depth image from the collected RGB image to obtain the RGB image and depth image of the same scene.

[0053] In this embodiment, existing methods often rely on using a comprehensive multi-source sensor system to obtain accurate RGB images and depth images. This equipment for acquiring accurate depth maps is usually expensive, and the hardware requirements of the depth sensor equipment determine that this method is only applicable to indoor and close-range scenes. For outdoor scenes, conventional sensors cannot obtain accurate depth maps. To address the difficulty in accurately extracting depth data for railway scenes, the present invention also includes a process for modal expansion of the RGB image to obtain the corresponding depth map, as follows:

[0054] Step S11, collecting RGB images through a camera sensor;

[0055] In step S12, for each RGB image in the RGB image set, a corresponding depth image is extracted using a large depth estimation model to form a corresponding RGB-D image, thereby obtaining a corresponding image pair data set. In this embodiment, the large depth estimation model includes but is not limited to using a Depth Anything V2 model to obtain a reliable depth map that strictly corresponds to the image from the RGB image through the large depth estimation model.

[0056] To verify whether the introduction of estimated depth improves the performance of semantic segmentation, this embodiment also conducted a comparative experiment on the Cityspaces dataset. The method of this embodiment introduces a large depth estimation model to obtain the depth map of the RGB image. Combined with the original RGB image, the semantic segmentation performance is much higher than the model using only RGB images. At the same time, it can also achieve performance close to that of using real depth maps. Moreover, this embodiment relies only on a single hardware device, the camera. In environments where it is difficult to accurately measure depth maps, it is a compromise solution that balances cost and performance.

[0057] Table 1. Performance comparison between this method and existing methods under different image data conditions

[0058]

[0059]

[0060] Step S2: data complementary calibration and fusion processing; build scene perception model, refer to Figure 3 As shown in the figure, it is a network structure diagram of the scene perception model provided in this embodiment. The scene perception model includes an encoding layer and a decoding layer; the encoding layer includes an RGB image encoding branch and a depth image encoding branch, which process RGB images and depth images respectively; based on the data fusion module respectively arranged between each encoding layer, it is used to correct the features of the RGB mode and the depth mode output by the fusion RGB feature extraction layer and the depth feature extraction layer, obtain the first fusion feature of each layer, and input it to the decoder.

[0061] In this embodiment, the encoder is provided with a first encoding branch and a second encoding branch for extracting RGB image features and depth image features, respectively. The first encoding branch and the second encoding branch are provided with multiple layers of RGB feature extraction layers and depth feature extraction layers, respectively, which can effectively capture the spatial relationship in the image, accelerate the training process and improve the effect of subsequent feature fusion.

[0062] In this embodiment, the data fusion module is also used to add the first fusion features obtained by processing to the original RGB modality and depth modality respectively to achieve feature enhancement, and the enhanced RGB modality is jump-connected to the next RGB feature extraction layer, and the enhanced depth modality is jump-connected to the next depth feature extraction layer respectively, and the first fusion features fused by each data fusion module except the last layer of data fusion module are input into the decoder.

[0063] In this embodiment, the data fusion module includes a complementary calibration module (CRM) and a fusion and enhancement module (FEM). The complementary calibration module CRM is responsible for calibrating the corresponding feature pixel positions between the RGB modality and the depth modality and filtering the noise information in the modality. The fusion and enhancement module FEM splices the calibrated RGB modality and the depth modality in the channel dimension, obtains multi-dimensional information of RGB features and depth features through different gated attention mechanisms, aggregates them, and adds the calibrated RGB features and depth features respectively to enhance the image, and the aggregated features are introduced into the decoder.

[0064] In order to eliminate the influence of noise on data and perform data fusion, the data fusion module of this embodiment acts on each coding layer, such as Figure 4 As shown in FIG, a data fusion module framework diagram provided in this embodiment takes the RGB features and depth features output by each convolutional layer as input, and utilizes the complementary characteristics of the RGB features and depth features to perform weighted fusion to obtain fusion features that have both color texture and geometric information. In order to take into account the influence of noise in railway scenes, which may cause information loss in a certain area of ​​the image, a complementary calibration module CRM is provided to filter noise, calibrate data features, and fully capture the complementary characteristics between modalities. Specifically, the feature processing process implemented by the complementary calibration module CRM includes the following:

[0065] Step 110 , firstly, the input RGB image and depth image are spliced ​​together, and then the attention vector of the spliced ​​features is extracted through the coordinate attention mechanism;

[0066] In step 120, the two attention vectors are concatenated and fused, and the fused feature map is processed sequentially through the KAN network (Kolmogorov-Arnold Networks), batch normalization layer, activation function layer, and split layer to obtain two independent tensors of the rectified RGB image and two independent tensors of the rectified depth image, namely, the attention feature weights of the rectified RGB image in the horizontal and vertical directions, and the attention feature weights of the rectified depth image in the horizontal and vertical directions, respectively. The filtered RGB features and depth features are then multiplied with the RGB image and the depth image respectively. In this way, the RGB image and the depth image contain both the relationship between channels and encode precise spatial position information.

[0067] Step 130: Use the attention cross strategy to obtain the calibrated RGB feature RGB re And the depth feature DEP after calibration re .

[0068] In this embodiment, to address the problem of more serious object information loss and boundary displacement when a typical hierarchical encoder-decoder network structure is used in a rail transit environment with complex scene perception requirements, especially under poor environmental conditions, which interferes with fine-grained semantic segmentation in rail transit scenes, the FEM image processing process of the fusion and enhancement module provided in this embodiment specifically includes the following steps:

[0069] Step 210: calibrate the RGB features re and deep features DEP re Perform stitching in the channel dimension;

[0070] Step 220: The spliced ​​features are processed based on the gated attention mechanism, aggregation, separation and activation to obtain the calibrated RGB feature space attention weights G r and the spatial attention weight G of the deep features d ;

[0071] In step 230, the concatenated features are then subjected to linear gated attention mechanism, aggregation, separation and activation processing to obtain the global attention weights L of the calibrated RGB features. r and the global attention weight L of the deep features d , to perceive the spatial and global information of each modality;

[0072] In step 240 , the first fused features of each coding layer are obtained by weighted fusion of the obtained spatial attention weights and the global attention weights corresponding to the calibrated RGB features and the depth features.

[0073] In one embodiment, feature fusion obtains the first fusion feature M i This is achieved by the following formula, where i represents the coding layer number:

[0074] M i =RGB re (i,j) ·[G r (i,j) +L r (i,j) ]+DEP re (i,j) ·[G a (i,j) +L d (i,j) ];

[0075] Among them, RGB re and DEP re These are the RGB features and depth features obtained after calibration.

[0076] In step 250, the first fused features are fused with the calibrated RGB features and depth features to obtain enhanced RGB features and depth features. The enhanced RGB features and enhanced depth features are passed to the next RGB image coding layer and depth image coding layer to achieve layer-by-layer coding, and the first fused features M of each layer are input to the decoder.

[0077] In this embodiment, the feedback enhancement module (FEM) uses two gated attention mechanisms to capture long-range semantic information, integrate local and global information, enhance the weight of dominant features in the railway (such as telephone poles, signal lights, etc.), reduce interference from irrelevant background or low-value areas, and capture diverse image information. The output of each encoder layer is then fused with RGB features and depth features of different scales to obtain fused features that have both color, texture, and geometric information, thereby enhancing the contribution of key areas. The fused features are fused with the RGB features and depth features of the current layer, and then jump-connected to the RGB feature extraction layer and depth feature extraction layer of the next layer, thereby enhancing the expressive power of image information, realizing feature enhancement operations, and realizing step-by-step fusion of enhanced features, so that the data features of high-attention areas are propagated in the convolutional layer, that is, some of the original data features are retained. At the same time, complementary features are indirectly added to the original features through the fusion results, thereby achieving data enhancement. With the layer-by-layer encoding, this complementary effect is updated and passed on at each layer.

[0078] Step S3, multi-scale feature interactive fusion processing; the first fusion feature obtained after fusion of the data fusion modules except the last layer data fusion module is input into the multi-scale fusion decoding module, and the multi-scale feature fusion module of the multi-scale fusion decoding module realizes interactive mapping of different resolution features between the first fusion feature of each decoding layer and the first fusion feature of the adjacent layer, and obtains a second fusion feature that fuses adjacent layers and different resolutions, and each second fusion feature is output as each coding layer and jumps to each decoder layer. A frequency-aware feature fusion device is set, except for the frequency-aware feature fusion device corresponding to the deepest decoding layer. Each frequency-aware feature fusion device receives the feature output by the next decoding layer and fuses it with the second fusion feature to obtain a third fusion feature. The third fusion feature is added to the up-sampled feature of the corresponding decoding layer and input to the next decoding layer; the first fusion feature obtained after fusion by the last data fusion module is input into the multi-scale ASPP module, and the ASPP module output is input to the decoder and also input to the frequency-aware feature fusion device corresponding to the deepest decoding layer to realize step-by-step recovery of the feature map, and finally obtain the semantic segmentation result.

[0079] In one embodiment, conventional methods such as ASPP for extracting multi-scale features only extract features at receptive fields of varying sizes and do not learn the correlation between cross-layer features. This embodiment takes into account the complementary information between features of varying resolutions in different coding layers. Therefore, in the decoding phase, this embodiment designs a multi-scale fusion decoding module specifically for railway scene scale issues. To maximize the utilization of feature information from convolutions of varying depths and minimize data feature loss during the convolution process, a multi-scale feature fusion module is constructed using cross-layer feature mapping. The output features of the coding layer are cross-fused across a range of scales, and finally, attention fusion is performed with the output features of the coding layer itself through a KAN network, enabling the output features of the coding layer to fuse with regions of interest in the output features of adjacent coding layers.

[0080] In order to utilize the feature information of different resolutions as effectively as possible, this scheme designs a multi-scale fusion decoding module to reduce the data loss during the decoding process, such as Figure 5 As shown in the figure, it is a structural diagram of the multi-scale feature fusion module provided by this embodiment. In the multi-scale feature fusion module, the first fusion features m0 to m1 are obtained based on the data fusion modules of each layer. i-1 As input, i represents the sequence number of the coding layer and the decoding layer, and the multi-scale feature fusion module realizes the interactive fusion of each first fusion feature with the first fusion feature corresponding to the adjacent coding layer to realize the fusion of the region of interest of the first fusion feature of the adjacent layer, thereby obtaining a new high-resolution feature that fuses the feature information of adjacent layers and different resolutions as the second fusion feature M0~M i-1 , M0~M i-1The 0~i-1 decoding layers corresponding to the decoder will be introduced respectively for residual supplement.

[0081] In this embodiment, the process of the multi-scale feature fusion module processing the first fusion feature of the adjacent coding layer is referred to Figure 5 For each feature mapping process (for example, mapping the features of layer l (as the target layer) to layer l+1 (adjacent layer)), the first fused features of different resolutions are first adjusted to the same size through convolution or transposed convolution operations, and then the first fused features of the target layer are enhanced through the first coordinate attention mechanism. The enhanced first fused features are then extracted through the second coordinate attention mechanism to extract the weights of the regions of interest, mapped and applied to the first fused features of the adjacent layer, and then fused with the enhanced first fused features to obtain the second fused features. In this way, the two sets of coordinate attention mechanisms interactively map the first fused features of different resolutions between the two encoding layers, and finally the output of each encoding layer contains its own enhanced features and features of interest to the adjacent layer.

[0082] In a specific embodiment, the feature processing process of the multi-scale feature fusion module is as follows:

[0083] w1=σ(split{F kan (c[p x (m l ),p y (m l )])})

[0084] w2=σ(split{F kan (c[p x (m rect ),p y (m rect )])});

[0085] Among them, m rect Represents the self-attention enhancement feature, w1 and w2 are the attention weights extracted by the first coordinate attention mechanism and the second coordinate attention mechanism respectively. This feature mapping process can be used with formula g (j,i) Description, that is, the output of the interaction between the first fusion feature of each layer and the adjacent first fusion feature map is:

[0086] m out =g (l,l+1) (m l ,m l+1 )=cov(w2×m l+1 +m rect );

[0087] Finally, all feature outputs of the same coding layer are added together to obtain feature information M that integrates multiple layers and different resolutions. i , Mi It can be expressed uniformly as:

[0088]

[0089] In this embodiment, according to the formula Where i≠j,|ij|=1, for i=0, that is, the first coding layer, the only j value that satisfies i≠j and |ij|=1 is j=1, so,

[0090] M0=m0+g (1,0) (m1,m0);

[0091] For i=1, i.e., the second coding layer, the j values ​​satisfying i≠j and |ij|=1 are j=0 and j=2. Therefore,

[0092] M1=m1+g (0,1) (m0,m1)+g (2,1) (m2,m1);

[0093] For i=2, i.e. the third coding layer, the only j value that satisfies i≠j and |ij|=1 is j=1. Therefore,

[0094] M2=m2+g (1,2) (m1,m2).

[0095] Next, M0~M2 will be introduced into the corresponding decoding layers of the decoder for residual supplementation, and the decoder will be used to convert the fused feature maps of different levels into the final semantic map.

[0096] In the traditional decoding process, the supplementary features M obtained by each coding layer are i Directly introducing the corresponding decoding layer through the residual connection method, the direct addition process does not take into account the mutual interference between features of different resolutions. During the decoding process, the frequency interference between features of different resolutions will lead to inconsistency within the category and cause the boundary displacement problem of segmentation. To address this problem, this embodiment embeds frequency perception fusion (FreqFusion) into the decoding process, such as Figure 6 , which is a structural diagram of the frequency-aware local fusion decoding process provided by this embodiment.

[0097] Based on the above steps, the data fusion module is used to extract the deep aggregated feature m3 and the multi-scale feature fusion module is used to fuse the features M0~M2 through multi-scale fusion. Subsequently, m3 is used as the decoding input to the decoder, and M2~M0 are sequentially used as the supplementary features of local fusion in each decoding layer. m3 is first processed by the spatial pyramid pooling module (ASPP) to extract context information and obtain feature f3. The low-resolution feature f3 and the high-resolution feature M2 are frequency-aware fused using the frequency-aware feature fusion device set corresponding to the deepest decoding layer, replacing the traditional residual connection that only adds. The fused feature is then added to the upsampled feature obtained by processing the low-resolution feature f3 in the deepest decoding layer to obtain f2, and then input to the previous decoding layer for upsampling, and the frequency-aware feature fusion device set corresponding to the previous decoding layer is used to achieve frequency fusion with the input high-resolution feature and is still added to the upsampled feature obtained by the corresponding decoding layer. Similarly, compared with the traditional residual connection, the frequency-aware fusion device set at each decoding layer can reduce the mutual interference of different frequencies and improve the segmentation performance of the boundary. Therefore, the output features of each decoding layer can be expressed as:

[0098] f i-1 =Feq(f i ,M i-1 )+up(f i ).

[0099] In one embodiment, a multi-scale cross entropy loss is used as loss supervision, and a weighted weight is applied to each loss according to the level of the feature layer:

[0100]

[0101] Among them, loss i is the loss of the i-th decoding layer feature, and λ0~λ3 are 1, 0.8, 0.6 and 0.4 respectively.

[0102] Finally, this scheme uses multi-scale cross entropy loss as loss supervision and applies weighted weights to each loss according to the height of the feature layer:

[0103]

[0104] Among them, loss i is the loss of the i-th layer feature, and λ0~λ3 are 1, 0.8, 0.6 and 0.4 respectively.

[0105] Among them, loss i is the loss of the i-th decoding layer feature, and λ0~λ3 are 1, 0.8, 0.6 and 0.4 respectively.

[0106] To improve the inference speed of the model, the scene perception model in this embodiment consists of two parts: a teacher model and a student model. A knowledge distillation network is used to achieve network lightweighting, so as to transfer the knowledge of the teacher model to a smaller and more efficient student model while maintaining performance as much as possible. The backbone network of the model uses the ResNet series. Figure 7 , which is a framework structure diagram of the knowledge distillation method for semantic segmentation provided in this embodiment. In this embodiment, ResNet-101 is selected as the base layer of the teacher model, and the student model uses the ResNet-18 model with fewer layers. The convolutional layers that implement attention fusion in the data fusion module and the multi-scale feature interaction module are replaced with necked linear layers to reduce the feature size, reduce the amount of model parameter calculation, and improve the performance of the model. The loss function consists of a basic loss and a cross-image global dependency loss. The model distillation method uses the CIRKD method to adjust and train the network parameters of the student model. The basic loss includes the semantic segmentation cross entropy loss and the class probability distillation loss, and the global dependency loss includes pixel-to-pixel and pixel-to-region distillation losses. The final distillation total loss function is:

[0107] loss sum =L task +L kd +αL memory-p2p +βL memory-p2r ;

[0108] Among them, α and β are weight coefficients, L task is the cross entropy loss for semantic segmentation tasks, L kd is the class probability distillation loss, L memory-p2p is the pixel-to-pixel distillation loss, L memory-p2r is the pixel-to-region distillation loss.

[0109] refer to Figure 8 As shown, there are eight scene graph semantic segmentation result graphs implemented by the method of the present invention provided in this embodiment, wherein the (a) row graphs are 8 scene graphs with semantic segmentation, and the (b) row graphs are semantic segmentation result graphs corresponding to the 8 scenes respectively.

[0110] To demonstrate the effectiveness of the proposed embodiment, refer to Table 2 below. This embodiment uses the joint mean intersection over union (MIoU) and pixel accuracy (PixAcc) as evaluation indicators on the same rail transit scene data, and conducts comparative tests on various existing advanced RGB segmentation methods and RGB-D segmentation methods. The comparison results are shown in Table 3. The performance of the model in this embodiment achieves the highest segmentation accuracy with an MIoU of 72.81% and a PixAcc of 89.69%.

[0111] Table 2. Performance comparison between this method model and existing method models

[0112]

[0113]

[0114] To verify the effectiveness of the different modules of the model in this embodiment, specific ablation experiments were conducted. For fair comparison, we used two identical encoding branches and simple additive fusion as the baseline. Using estimated depth information improved the MIoU segmentation metric from 65.84% to 67.26%, demonstrating that although the depth map is derived from depth estimation, it can still serve as a supplementary modal feature to improve semantic segmentation performance. In addition, using the data fusion module alone improved MIoU by 2.59% to 69.85%, and using the multi-scale feature fusion module alone improved MIoU by 3.2% to 70.46%, indicating that both the data fusion module and the multi-scale interactive fusion module can improve segmentation performance. When the data fusion module and the multi-scale feature fusion module are used simultaneously, the MIoU is 72.51%, an increase of 5.25%, indicating that the two modules can synergistically promote segmentation performance.

[0115] Finally, using the frequency-aware feature fuser proposed in the present invention, the MIoU reached 72.81%, which is 0.3% higher than the traditional decoding method (72.51%). Therefore, the frequency-aware feature fuser processing principle set in the present invention can optimize the segmentation ability of the model from the perspective of frequency filtering.

[0116] Table 3. Ablation experiments on the railway dataset

[0117]

[0118] Through dataset expansion, RGB-D semantic segmentation, and knowledge distillation, the method in this embodiment has developed a lightweight semantic segmentation model suitable for complex scenarios, such as rail transit. This model can handle complex traffic scenarios, such as train operation scenes, stations, underground sections, and tunnels. It can also understand scenes in harsh weather conditions and can cope with interference caused by rain, snow, and fog, showing strong robustness. It can accurately identify important objects such as railway operation areas, tracks, utility poles, and trains.

[0119] refer to Figure 9As shown in the figure, it is a system block diagram of the semantic segmentation method provided by this embodiment. The present invention first expands RGB data to RGB-D mode through an unsupervised depth estimation large model, thereby constructing an RGB-D dataset. Secondly, in response to the noise problem and multi-scale features of the railway environment, cross-modal data complementary calibration fusion modules are designed to enhance the weight of the dominant features in the railway, reduce the interference of irrelevant background or low-value areas, capture diverse image information, and set up a multi-scale feature interactive fusion module to improve the segmentation accuracy and robustness of the model; then, the frequency-aware feature fuser set in the decoding layer is used to embed frequency-aware fusion (FreqFusion) into the decoding, and the final semantic output is obtained through a fine-grained local frequency-aware fusion decoding strategy, which effectively reduces the mutual interference of different frequencies and improves the boundary segmentation performance. The present invention is suitable for scenes with high noise, multi-category, multi-scale and difficult-to-acquire depth information, such as railways. It can effectively handle the high noise problem in the railway environment and accurately perceive the scene at the same time with minimal cost; finally, the knowledge distillation algorithm for semantic segmentation is used to lightweight the model to improve the model inference speed.

[0120] This invention aims to utilize a large depth estimation model to expand image data to RGB-D modality to improve the accuracy and robustness of scene perception. It aims to combine the complementary advantages of multimodal data to address the complex and changing environments and high noise levels found in rail transit scenarios, and to address the limited and limited research on rail scene perception in existing technologies. Relying only on visual sensor information regarding the train's direction of travel, this method achieves efficient, accurate, and robust scene understanding, enhancing the safety and stability of automated train driving systems. It can provide sufficient visual understanding for intelligent transportation systems, high-precision track mapping, positioning and navigation, and other tasks, helping to promote the intelligent development of urban rail transit systems and having important implications for their sustainable development.

[0121] Device embodiment

[0122] According to an embodiment of the present invention, a semantic segmentation device with enhanced depth estimation is provided. Figure 10 FIG. 1 is a block diagram of a semantic segmentation apparatus with enhanced depth estimation provided by this embodiment. The semantic segmentation apparatus with enhanced depth estimation according to an embodiment of the present invention includes:

[0123] The data acquisition and processing module 10 is used to extract the corresponding depth image from the collected RGB image using a depth estimation model, and obtain the RGB image and depth image of the same scene.

[0124] A model building module 20 is used to build a scene perception model based on the scene perception model, the scene perception model includes a coding layer and a decoding layer;

[0125] The coding layer includes an RGB image coding branch and a depth image coding branch, which process RGB images and depth images respectively. Based on the data fusion modules respectively set between the coding layers, it is used to correct and fuse the features of the RGB mode and the depth mode output by each RGB feature extraction layer and the depth feature extraction layer to obtain the first fusion feature of each layer. The first fusion feature obtained after fusion of the data fusion modules except the last layer is input to the multi-scale fusion decoding module. The multi-scale feature fusion module of the multi-scale fusion decoding module realizes the interactive mapping of the first fusion feature of each decoding layer with the first fusion feature of the adjacent layer to obtain the second fusion feature that fuses the adjacent layers and different resolutions. And each second fusion feature is output as each encoding layer and jumps to each decoder layer to set a frequency-aware feature fusion device. Except for the frequency-aware feature fusion device corresponding to the deepest decoding layer, each frequency-aware feature fusion device receives the features output by the next decoding layer, and fuses them with the second fusion feature to obtain a third fusion feature. The third fusion feature is added to the up-sampled feature of the corresponding decoding layer and input to the next decoding layer; the first fusion feature obtained after the fusion of the last data fusion module is input to the multi-scale ASPP module, and the output of the multi-scale ASPP module is input to the frequency-aware feature fusion device and decoder corresponding to the deepest decoding layer, so as to realize step-by-step recovery of the feature map and finally obtain the semantic segmentation result.

[0126] The semantic segmentation module 30 is used to obtain a depth image from the collected RGB image based on the data acquisition and processing module 10, and then input it into the scene perception model constructed by the model construction module 20 to obtain a semantic segmentation result.

[0127] The depth estimation enhanced semantic segmentation device provided in this embodiment expands RGB data to RGB-D mode through an unsupervised depth estimation large model, thereby constructing RGB-D data. Then, in response to the noise problem and multi-scale features of the railway environment, cross-modal data complementary calibration fusion modules are designed respectively to enhance the weight of the dominant features in the railway, reduce the interference of irrelevant background or low-value areas, and capture diverse image information. The multi-scale feature interactive fusion module set up improves the segmentation accuracy and robustness of the model; then, the frequency-aware feature fuser set up in the decoding layer is used to embed frequency-aware fusion into decoding, and the final semantic output is obtained through a fine-grained local frequency-aware fusion decoding strategy, which effectively reduces the mutual interference of different frequencies and improves the boundary segmentation performance. The device provided in this embodiment is suitable for scenes such as railways with high noise, multi-category, multi-scale and difficult-to-acquire depth information. It can effectively handle the high noise problem in the railway environment, and while accurately perceiving the scene, only requires minimal cost; finally, the model is lightweighted using a knowledge distillation algorithm for semantic segmentation to improve the model inference speed.

[0128] In this embodiment, the data fusion module includes a complementary calibration module CRM and a fusion and enhancement module FEM. The complementary calibration module CRM is responsible for calibrating the corresponding feature pixel positions between the RGB modality and the depth modality and filtering the noise information in the modality. The fusion and enhancement module FEM splices the calibrated RGB modality and the depth modality in the channel dimension, obtains multi-dimensional information of RGB features and depth features through different gated attention mechanisms, aggregates them, and adds the calibrated RGB features and depth features respectively to enhance the image, and the aggregated features are introduced into the decoder.

[0129] In this embodiment, the multi-scale feature fusion module processes the first fusion feature of the adjacent coding layer. Figure 5 For each feature mapping process (for example, mapping the features of layer l (as the target layer) to layer l+1 (adjacent layer)), the first fused features of different resolutions are first adjusted to the same size through convolution or transposed convolution operations, and then the first fused features of the target layer are enhanced through the first coordinate attention mechanism. The enhanced first fused features are then extracted through the second coordinate attention mechanism to extract the area of ​​interest of the first fused features of the adjacent coding layer and mapped to the adjacent layer, and then fused with the obtained enhanced first fused features to obtain the second fused features; in this way, the two sets of coordinate attention mechanisms interactively map the first fused features of different resolutions between the two coding layers, and finally the output part of each coding layer contains its own enhanced features and features of interest to the adjacent layer.

[0130] The embodiment of the present invention is an apparatus embodiment corresponding to the above-mentioned method embodiment. The specific operations of the processing steps of each module can be understood by referring to the description of the method embodiment, and will not be repeated here.

[0131] like Figure 11 As shown, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the semantic segmentation method with enhanced depth estimation in the above-mentioned embodiment is implemented.

[0132] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the semantic segmentation method with enhanced depth estimation in the above-mentioned embodiment.

[0133] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0134] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without making any creative efforts.

[0135] In addition, the functional modules in the various embodiments of the present disclosure may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and the contents not described in detail in the specification of the present invention are common knowledge to those skilled in the art.

Claims

1. A semantic segmentation method with enhanced depth estimation, characterized in that: Including steps: Use the depth estimation model to extract the corresponding depth image from the collected RGB image, and obtain the RGB image and depth image of the same scene; Construct a scene perception model, which includes a coding layer and a decoding layer; the coding layer includes an RGB image coding branch and a depth image coding branch, which process RGB images and depth images respectively; based on a data fusion module respectively set between each coding layer, it is used to correct and fuse the RGB modality and depth modality features output by the RGB feature extraction layer and the depth feature extraction layer to obtain the first fused feature of each layer; The first fusion feature obtained after fusion of the data fusion modules except the last layer of data fusion module is input into the multi-scale fusion decoding module. The multi-scale feature fusion module of the multi-scale fusion decoding module realizes the interactive mapping of different resolution features between the first fusion features of each decoding layer and the first fusion features of the adjacent layer, and obtains the second fusion feature that fuses the adjacent layers and different resolutions. Each second fusion feature is output as each coding layer and jumps to each decoder layer to set a frequency-aware feature fusion device. Except for the frequency-aware feature fusion device corresponding to the deepest decoding layer, each frequency-aware feature fusion device receives the feature output by the next decoding layer and fuses it with the second fusion feature to obtain the third fusion feature. The third fusion feature is added to the up-sampled feature of the corresponding decoding layer and input to the next decoding layer. The first fusion feature obtained after fusion of the last data fusion module is input into the multi-scale ASPP module, and the output of the multi-scale ASPP module is input to the frequency-aware feature fusion device and decoder corresponding to the deepest decoding layer to realize step-by-step restoration of the feature map, and finally obtain the semantic segmentation result.

2. The semantic segmentation method with enhanced depth estimation according to claim 1, wherein: The data fusion module is used to add the first fusion features obtained by processing to the original RGB mode and depth mode respectively to achieve feature enhancement, and the enhanced RGB mode is jump-connected to the next RGB feature extraction layer, and the enhanced depth mode is jump-connected to the next depth feature extraction layer respectively, and then the first fusion features fused by each data fusion module except the last layer of data fusion module are input into the decoder.

3. The semantic segmentation method with enhanced depth estimation according to claim 1, wherein: The data fusion module includes a complementary calibration module and a fusion and enhancement module; The complementary calibration module is responsible for calibrating the corresponding feature pixel positions between the RGB modality and the depth modality and filtering the noise information in the modality; The fusion and enhancement module concatenates the calibrated RGB modality and the depth modality in the channel dimension, obtains multi-dimensional information of RGB features and depth features through different gated attention mechanisms, aggregates them, and adds the calibrated RGB features and depth features respectively to enhance the image, and the aggregated features are introduced into the decoder.

4. The semantic segmentation method with enhanced depth estimation according to claim 3, wherein: The complementary calibration module implements the feature processing process specifically including the following steps: The input RGB image and depth image are spliced ​​together, and the attention vector of the spliced ​​features is extracted through the coordinate attention mechanism; The two attention vectors are concatenated and fused, and the fused feature maps are processed sequentially through the KAN network, batch normalization layer, activation function layer, and split layer to obtain two independent tensors of the rectified RGB image and two independent tensors of the rectified depth image. These are then multiplied with the RGB image and depth image to obtain filtered RGB features and depth features respectively. The attention cross strategy is adopted to obtain the calibrated RGB features and the calibrated depth features.

5. The semantic segmentation method with enhanced depth estimation according to claim 3 or 4, wherein: The image processing process of the fusion and enhancement module specifically includes the following steps: The calibrated RGB features and the calibrated depth features are spliced ​​in the channel dimension; The spliced ​​features are processed based on the gated attention mechanism, aggregation, separation and activation to obtain the calibrated RGB feature spatial attention weights and the depth feature spatial attention weights respectively; The spliced ​​features are then processed based on the linear gated attention mechanism, aggregation, separation and activation to obtain the global attention weights of the calibrated RGB features and the global attention weights of the depth features respectively; Based on the obtained spatial attention weights and global attention weights, the first fused features of each coding layer are obtained by weighted fusion with the calibrated RGB features and depth features; The first fusion features are fused with the calibrated RGB features and depth features to obtain enhanced RGB features and enhanced depth features. The enhanced RGB features and enhanced depth features are passed to the next RGB image coding layer and depth image coding layer to achieve layer-by-layer encoding, and the first fusion features of each layer are input to the decoder.

6. The semantic segmentation method with enhanced depth estimation according to claim 1, wherein: The multi-scale fusion decoding module implements the feature interaction process specifically including the following steps: Based on the first fusion features obtained by the data fusion module of each layer as input, the multi-scale feature fusion module realizes the interactive fusion of each first fusion feature with the first fusion feature corresponding to the adjacent coding layer to realize the fusion of the region of interest of the first fusion features of the adjacent layers, thereby obtaining high-resolution features that fuse the feature information of adjacent layers and different resolutions, and respectively introduced into the decoding layer corresponding to the decoder for residual supplement.

7. The semantic segmentation method with enhanced depth estimation according to claim 1 or 6, wherein: The multi-scale feature fusion module implements the feature interaction process specifically including the following steps: The first fused features of different resolutions are adjusted to the same size through convolution or transposed convolution operations, and the first fused features of the target layer are enhanced through the first coordinate attention mechanism. The enhanced first fused features are extracted through the second coordinate attention mechanism to obtain the weights of the region of interest, mapped and applied to the first fused features of the adjacent layer, and then fused with the enhanced first fused features to obtain the second fused features.

8. The semantic segmentation method with enhanced depth estimation according to claim 1, wherein: The scene perception model consists of two parts: a teacher model and a student model. The backbone networks of the teacher model and the student model use the ResNet series model. The convolutional layers that implement attention fusion in the data fusion module and the multi-scale feature interaction module in the student model are replaced by necked linear layers, and the loss function consists of a basic loss and a cross-image global dependency loss.

9. A semantic segmentation device with enhanced depth estimation, characterized in that: include: The data acquisition and processing module is used to extract the corresponding depth image from the collected RGB image using the depth estimation model, and obtain the RGB image and depth image of the same scene; A model building module is used to build a scene perception model, which includes an encoding layer and a decoding layer; The coding layer includes an RGB image coding branch and a depth image coding branch, which process RGB images and depth images respectively; a data fusion module is set between each coding layer to correct and fuse the features of the RGB mode and depth mode output by each RGB feature extraction layer and the depth feature extraction layer to obtain the first fusion feature of each layer; the first fusion feature obtained after fusion of the data fusion module except the last layer of data fusion module is input to the multi-scale fusion decoding module, and the multi-scale fusion decoding module realizes the interactive mapping of different resolution features between the first fusion feature of each decoding layer and the first fusion feature of the adjacent layer, and obtains the second fusion feature that fuses adjacent layers and different resolutions, and each The second fusion feature is output as each encoding layer and jumps to each decoder layer to set a frequency perception feature fusion device. Except for the frequency perception feature fusion device corresponding to the deepest decoding layer, each frequency perception feature fusion device receives the feature output by the next decoding layer and fuses it with the second fusion feature to obtain a third fusion feature. The third fusion feature is added to the up-sampled feature of the corresponding decoding layer and then input to the next decoding layer; the first fusion feature obtained after the last data fusion module is input to the multi-scale ASPP module, and the output of the multi-scale ASPP module is input to the frequency perception feature fusion device and decoder corresponding to the deepest decoding layer to realize step-by-step recovery of the feature map and obtain the semantic segmentation result; The semantic segmentation module is used to obtain the depth image from the collected RGB image based on the data acquisition and processing module, and then input it into the scene perception model constructed by the model construction module to obtain the semantic segmentation result.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the depth estimation enhanced semantic segmentation method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the depth estimation enhanced semantic segmentation method according to any one of claims 1 to 8 is implemented.

Images