Semantic segmentation method based on stereoscopic image super-resolution reconstruction guidance

Abstract

The invention discloses a semantic segmentation method based on stereoscopic image super-resolution reconstruction guidance, and relates to the technical field of computer vision, and the method comprises the steps: firstly designing a semantic-super-resolution joint network with a double-branch structure; and simultaneously generating a high-resolution semantic segmentation map and a super-resolution reconstruction result of the left view through the semantic segmentation branch and the super-resolution branch. And secondly, in order to effectively utilize parallax information in the three-dimensional image to assist semantic segmentation, a parallax perception fusion module is designed, and semantic features are enhanced based on the parallax information, so that the segmentation accuracy is improved. Besides, in consideration of the fact that the super-resolution branch contains richer detail information and can guide the semantic segmentation task to learn better high-resolution representation, a semantic-detail interaction module is designed, the semantic-detail interaction module and the super-resolution branch can fully pay attention to complementary information clues of each other, and therefore performance improvement is further achieved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a semantic segmentation method guided by stereo image super-resolution reconstruction. Background Technology

[0002] Semantic segmentation is a core task in computer vision. By classifying each pixel of an image, it helps computers understand image content and make decisions, and has been widely applied in scenarios such as autonomous driving, robot navigation, medical image analysis, and remote sensing mapping. With the development of deep learning, semantic segmentation methods have made significant progress. Typical structures include encoder-decoder structures and multi-branch structures. In the encoder-decoder structure, the encoder extracts hierarchical deep features, and the decoder gradually restores spatial details. Representative methods include U-Net, SegNet, PSPNet, the DeepLab series, and SCTNet, which have good feature representation and context modeling capabilities. The multi-branch structure, based on the global semantic information extracted by the backbone network, uses auxiliary branches to enhance boundary or detailed features, balancing real-time performance and accuracy. Typical methods include BiSeNet, DDRNet, and PIDNet.

[0003] However, the reliance on high-resolution input in existing technologies contradicts practical applications. Most current semantic segmentation methods depend on high-resolution input to extract sufficient spatial details and boundary information to achieve ideal segmentation results. But in practical applications, due to factors such as limited hardware resources, complex acquisition environments, and insufficient sensor performance, input images are often low-resolution, leading to the loss of texture information and blurred boundaries, significantly reducing segmentation performance. In the field of autonomous driving, the performance degradation caused by low-resolution input may prevent the system from accurately identifying pedestrians, vehicles, traffic signs, etc. on the road, increasing driving risks. Unimodal dependency is ill-suited for complex scenes: Most methods focus on unimodal image processing, lacking auxiliary information such as depth, optical flow, or semantic priors. In complex scenes with similar textures, varying lighting, and cluttered backgrounds, unimodal dependency makes it difficult for models to adequately distinguish between targets and backgrounds, easily leading to missegmentation or missed segmentation. In medical image analysis, unimodal processing struggles to handle complex human tissue images; missegmentation or missed segmentation may affect doctors' accurate diagnosis and delay treatment.

[0004] Therefore, a semantic segmentation method based on stereo image super-resolution reconstruction is proposed to solve the above problems. Summary of the Invention

[0005] In view of this, the technical problem to be solved by the present invention is to propose a semantic segmentation method based on stereo image super-resolution reconstruction guidance, so as to solve the problems of significant performance degradation of existing technology under low resolution input and lack of robustness caused by insufficient single-modal features.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a semantic segmentation method based on stereo image super-resolution reconstruction guidance, comprising: S1, Input data preparation: Acquire a low-resolution (LR) stereo image, the stereo image including a left view. and right view ,in, and As network input data, H and W are the height and width of the image, respectively, and r represents the method scaling factor; S2, Constructing a Single-View Encoding: A symmetric encoder with shared weights is constructed based on ResNet34. The residual blocks are replaced with NBT1D, and the max-pooling layer after the first convolution is removed to extract features from the left and right views respectively. , ; S3, Stereo Image Super-Resolution Branch Processing: First, aggregate through the Pyramid Pooling Module (PPM). , Multi-scale information is then processed by an improved parallax attention module (MPAM) to generate a parallax attention map. Finally, the super-resolution decoder recovers the feature size from the enhanced features, resulting in segmentation features augmented with disparity information. ; S4, semantic segmentation branch processing: After extracting multi-scale semantic information using the pyramid pooling module (PPM), it is combined with the disparity attention map. Semantic features are enhanced by DAFM, and then processed by a semantic decoder (with the same structure but different parameters as the super-resolution decoder) to obtain the refined segmentation features. ; S5, Feature Interaction: Left View Super-Resolution Refinement Features Segmentation features refined by the decoder The input is fed into the Detailed Interaction Module (SDIM), where it is fused through channel interaction, spatial interaction, and adaptive integration to generate the final interactive features. , ; S6, Output and Optimization: The super-resolution head and semantic segmentation head will respectively generate... , The figure shows the construction of a joint loss of "cross-entropy + MAE" to optimize the network until convergence.

[0007] Preferably, the single-view encoder described in S2 includes a symmetric left-view encoder and a right-view encoder with shared weights. Both the left-view encoder and the right-view encoder are based on a pre-trained ResNet34 network, with the residual blocks in the ResNet34 network replaced by non-bottleneck 1D blocks (NBT1D), and the max-pooling layer after the first convolutional layer in the ResNet34 encoder removed. The data is input into the left view encoder to obtain the left view features. The right view The input is fed into the right view encoder to obtain the right view features. ,in and The size is one-sixteenth of the input image size, and the feature extraction formula is:

[0008]

[0009] In the formula, This represents the feature extraction operation of a single-view encoder.

[0010] Preferably, the stereo image super-resolution branching process described in S3 specifically includes: S3.1, Multi-scale feature aggregation: and The shared pyramid pooling module (PPM) is input separately. Through multi-size average pooling and convolution operations (1×1, 2×2, 4×4, 8×8) in the PPM, contextual information at different scales is aggregated to obtain the left view features after multi-scale processing. and right view features ,in This represents the feature processing operations of the Pyramid Pooling Module (PPM). S3.2, Cross-view feature enhancement: will and Input the improved parallax attention module (MPAM) to generate a parallax attention map. and left-view super-resolution features enhanced across views The operating formula is:

[0011] in, Represents the operation of the pyramid pooling module (PPM). This represents the improved parallax attention module (MPAM) feature enhancement operation. Represents a parallax attention map; S3.3, Super-resolution Feature Decoding: Construct a super-resolution decoder, which consists of three decoder modules and a bilinear upsampling layer. Each decoder module contains a 3×3 convolutional layer, three NBT1D blocks, and a bilinear upsampling layer; The input super-resolution decoder restores the feature map resolution to match the input image size through progressive 2x upsampling, resulting in the super-resolution refined features of the left view. The decoding formula is:

[0012] in, This indicates the decoder used for super-resolution reconstruction; Preferably, the semantic segmentation branching process described in S4 specifically includes: S4.1, Multi-scale semantic feature extraction: Inputting the pyramid pooling module (PPM) in S3.1 yields the left-view semantic features after multi-scale processing. ; S4.2, Parallax Perception Feature Enhancement: Construct a parallax perception fusion module (DAFM) to... And obtained from S3.1 The input is a disparity-aware fusion module (DAFM). Through disparity acquisition and disparity perception operations of the DAFM, disparity-enhanced semantic features of the left view are generated. The operating formula is:

[0013] In the formula, Represents the parallax perception fusion module. These are segmentation features enhanced with disparity information; S4.3, Semantic Feature Decoding: Construct a semantic segmentation decoder, which has the same structure as the super-resolution decoder in S3.31 but does not share parameters; Input the semantic segmentation decoder to obtain the semantic refinement features of the left view. The decoding formula is:

[0014] In the formula, This represents the decoder used for semantic segmentation, which has the same structure as the decoder in the stereo image super-resolution branch, but does not share parameters; This represents the segmentation features refined by the decoder.

[0015] Preferably, S5 specifically includes constructing a Semantic Detail Interaction Module (SDIM), which uses the data obtained in S3.1. And obtained from S4.3 The input is the Detail Interaction Module (SDIM). Through the SDIM's channel interaction, spatial interaction, and adaptive fusion operations, the final super-resolution interactive features are obtained. and semantic interaction features The operating formula is:

[0016] In the formula, Semantic-detailed interaction module.

[0017] Preferably, S6 specifically includes constructing a super-resolution head and a semantic segmentation head. The super-resolution head contains two 3×3 convolutional layers and one sub-pixel convolutional layer, and the semantic segmentation head contains a bilinear upsampling layer and a softmax layer. Input a super-resolution head to generate a high-resolution (HR) reconstructed image of the left view. ,Will Input semantic segmentation header to generate a high-resolution semantic segmentation map for the left view. ,in, and Where r is the super-resolution magnification, the generation formula is:

[0018]

[0019] In the formula, This represents a super-resolution head, which mainly consists of two 3×3 kernel convolutional layers and one subpixel convolutional layer; This represents the semantic segmentation header.

[0020] Preferably, the disparity-aware fusion module (DAFM) described in S4.2 includes: Disparity Acquisition: For disparity acquisition, due to the disparity attention map from the stereo image super-resolution branch... The disparity map is obtained by calculating the similarity of features in the left and right views along the horizontal epipolar line. Therefore, the index of the point with the highest similarity can be used to roughly obtain the disparity map. Next, the disparity map is subjected to min-max normalization and then fed into a 3×3 convolutional layer and an NBT1D block to extract disparity features. The specific process is shown in the following formula.

[0021]

[0022]

[0023] In the formula, Indicates the position of the right view Features and left view position The degree of matching of features. Indicates the position of the right view features relative to the left view. The index of the most relevant feature at a given location; This represents a convolutional layer operation with a kernel size of 3×3; Represents a non-bottleneck 1D block. Represents the extracted disparity features; Parallax perception: Input a 1×1 convolutional layer to generate a feature map Q, and then... A 3×3 deep convolutional layer is input to generate a feature map K; Q and K are multiplied pixel by pixel and activated by the sigmoid function to obtain a disparity-aware attention map, which is then combined with... Pixel-by-pixel multiplication yields preliminary enhanced semantic features. The specific formula is as follows:

[0024] In the formula, This indicates a pixel-by-pixel multiplication operation. Represents the sigmoid function; Indicates parallax features; Will and The fused features are obtained by adding them pixel by pixel. Then, the fused features are input into the channel attention module (CA) to filter key channel information and obtain the final enhanced semantic features. The calculation formula is:

[0025] In the formula, This represents the channel attention mechanism.

[0026] Preferably, the specific operations of the Detailed Interaction Module (SDIM) in S5 include: S5.1 Channel Interaction: Input semantic features and super-resolution features The concatenation is performed along the channel dimension to obtain concatenated features; these concatenated features are then input into a global average pooling layer and a global max pooling layer to generate a global average pooling vector. and global max pooling vector ;Will and Cascaded along the channel dimension, inputting two fully connected layers and a sigmoid function sequentially to generate channel weight vectors. The specific calculation formula is as follows: In the formula, Represents the ReLU function. Represents the sigmoid function; and This represents the weight matrix of two fully connected layers. It is the channel compression ratio; Decomposed into semantic channel weights and super-resolution channel weights , respectively with semantic features and super-resolution features Adjustments are made by multiplying pixel by pixel to obtain the semantic features after channel interaction. and super-resolution features The calculation formula is:

[0027]

[0028] S5.2, Spatial Interaction: Global average pooling and max pooling are used to process semantic features and super-resolution features respectively to generate four feature maps. Unlike channel interactions, these operations are performed along the channel dimension to preserve spatial context information. To better utilize complementary information, the four feature maps are concatenated along the channel dimension and then fed into a dilated convolutional layer with a kernel size of 7×7 and a dilation rate of 3 to learn spatial relationships. Finally, a sigmoid layer is used to generate a spatial attention map. The specific formula for the operation is shown below.

[0029] In the formula, This represents the dilated convolutional layer operation; by using dilated convolutional layers, the receptive field can be expanded, thereby capturing a wider range of contextual information and helping to generate more reliable spatial attention maps. Next... Divided into two attention maps and These are used to adjust semantic features and super-resolution features, respectively. The specific formulas for the process are shown below.

[0030]

[0031] In the formula, and These represent the semantic and super-resolution features after spatial interaction, respectively; Finally, the features adjusted through channel and spatial interaction learning are adaptively fused with the original features to obtain the final features for segmentation and reconstruction. and The specific formula for the process is shown below.

[0032]

[0033] In the formula, These are four learnable parameters used to adjust the intensity of channel interaction and spatial interaction.

[0034] Preferably, the semantic segmentation branch processing obtained in S4 includes semantic segmentation loss and super-resolution loss. The semantic segmentation loss adopts a conventional multi-class cross-entropy loss to measure the difference between the predicted class distribution and the true label, and its expression is as follows:

[0035] In the formula, C represents the number of categories. It is the truth label corresponding to category i. The probability of class i predicted for the semantic segmentation branch; The super-resolution loss is defined as the mean absolute error (MAE) between the reconstructed stereo image and the true HR stereo image, and the formula is as follows:

[0036] In the formula, The left view representing the super-resolution branch reconstruction of a stereo image; This represents the corresponding real HR image.

[0037] Compared with existing technologies, the semantic segmentation method based on stereo image super-resolution reconstruction provided by this invention has the following advantages: 1. Significantly Improved Image Quality: Unlike traditional methods, this solution innovatively integrates multiple attention mechanisms and cleverly introduces disparity information into the network to assist image reconstruction. In both spatial and channel dimensions, it can jointly and accurately extract intra-view features. Simultaneously, the multi-head cross-attention mechanism within the transformer can efficiently transfer complementary information between the left and right views. Subsequently, by fusing disparity maps and RGB images across modalities and refining cross-view interactions using a stereo cross-attention module (SCAM), it powerfully achieves deep interaction and fusion of intra-view and cross-view information. This unique design helps the network more accurately recover textures and details in images, thereby significantly improving the quality of reconstructed images. For example, in the field of medical imaging, it allows doctors to more clearly observe subtle features of lesions, providing strong support for accurate diagnosis; in autonomous driving scenarios, it enables vehicles to more accurately identify various objects on the road, ensuring driving safety. 2. Significantly Reduced Computational Costs: Currently, most semantic segmentation methods, in pursuit of high performance, often use high-resolution inputs, which undoubtedly brings huge computational demands and severely limits their application on resource-constrained devices. This solution proposes a flexible and concise dual-branch network framework (DSRL). This framework consists of three parts: Semantic Segmentation Super-Resolution (SSSR), Single Image Super-Resolution (SISR), and Feature Similarity (FA) modules. It can maintain high-resolution representation while using low-resolution inputs, greatly reducing the computational complexity of the model. Furthermore, the SISR branch optimized during training can be removed from the network during the inference phase with almost no additional computational cost. Taking the semantic segmentation task on the CityScapes dataset as an example, under similar computational cost (FLOPs), this solution can achieve an mIoU improvement of ≥2%; in the human pose estimation task, with the same FLOPs, it can achieve an mAP improvement of ≥2%, and can still maintain mAP even when FLOPs are reduced by 30%. This advantage makes this solution have broad application prospects in resource-constrained scenarios such as mobile devices and embedded devices. 3. Enhanced Applicability Across Multiple Scenarios: The proposed method boasts excellent versatility, easily adaptable to various tasks and scenarios. In semantic segmentation tasks within autonomous driving, it helps vehicles more accurately identify targets such as roads, pedestrians, and traffic signs, enhancing the safety and reliability of autonomous driving. In industrial inspection, it enables more precise identification and classification of surface defects in products, improving the efficiency and accuracy of product quality inspection. In intelligent security, it assists monitoring systems in more clearly identifying suspicious individuals and abnormal behaviors, strengthening security capabilities. Furthermore, for binocular super-resolution tasks, this solution introduces a diffusion model and stereo physical constraints for collaborative modeling. While improving image details and perception quality, it effectively maintains binocular consistency, achieving a good balance between distortion and perception. This drives binocular super-resolution technology towards a more realistic and natural direction, making it suitable for scenarios requiring high-precision stereo vision, such as virtual reality and robot navigation. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the network structure of a semantic segmentation method based on stereo image super-resolution reconstruction guided by the present invention; Figure 2 The present invention is illustrated in (a) a decoder and (b) a decoder module. Figure 3 This is a detailed structural diagram of the disparity perception fusion module (DAFM) of the present invention; Figure 4 This is a detailed structural diagram of the Semantic-Detail Interaction Module (SDIM) of the present invention; Figure 5This is a schematic diagram showing the segmentation results of the present invention on the Cityscapes validation set. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0041] For an example, please refer to... Figures 1 to 5 As shown: To address the problems mentioned in the technical solutions, this application provides a semantic segmentation method guided by stereo image super-resolution reconstruction. The network is trained and evaluated using the Cityscapes dataset. Cityscapes is a large dataset focused on urban visual scene understanding, providing binocular images, disparity maps, and semantic annotations for the left view. The dataset contains 2975 training images, 500 validation images, and 1525 test images. These images were collected from road scenes in 50 cities, covering different seasons, and all images are 2048×1024 pixels in size. Typically, semantic segmentation tasks involve dividing the images in this dataset into 19 classes. Since the dataset only provides semantic annotations for the left view and not for the right view, this invention only implements semantic segmentation and super-resolution reconstruction for the left view.

[0042] Step 1, Experiment Setup; During the training phase, bicubic interpolation was first used to downsample the left and right images in the Cityscapes training set by a factor of 2 to obtain LR images. To increase the diversity of training samples, this invention employs a data augmentation strategy of random scaling, cropping, and flipping. The scaling range is set to [1.0, 1.4], the cropping size is 512×1024, and horizontal flipping is performed. Simultaneously, HR images and semantic annotation maps are also augmented accordingly. All experiments in this invention were implemented on a GeForce RTX 3090 GPU using the PyTorch framework, and the model was optimized using the SGD optimizer with a momentum of 0.9 and a weight decay of 0.0001. The batch size was set to 2, the initial learning rate was 0.01, and a single-cycle learning rate adjustment strategy was used to dynamically adjust the learning rate, with a total of 108 training cycles.

[0043] During the testing phase, the left and right views in the validation set are first downsampled by a factor of 2, and then input into the network for prediction. Considering the highly imbalanced class distribution in the Cityscapes dataset (e.g., roads and buildings have a much larger proportion than traffic lights), the mean Intersection-over-Union (mIoU) metric, used to measure the overlap between the predicted results and the true labels, can reduce the dominance of large classes, thus improving the fairness of the evaluation results. Therefore, this method uses mIoU as the evaluation metric to assess model performance. Furthermore, current mainstream semantic segmentation methods generally only use mIoU to evaluate the model's performance on the Cityscapes dataset; using this metric also helps ensure the fairness and consistency of the comparison. A higher mIoU value indicates better segmentation performance.

[0044] Step two, experimental results; This invention is compared with some existing representative semantic segmentation methods, including eight single-modal semantic segmentation methods, which rely solely on the left-view RGB image as input; three RGB-D semantic segmentation methods, which use both the left-view RGB image and the depth map as input; two multi-task methods based on stereo matching and semantic segmentation; and two semantic segmentation methods based on SISR guidance.

[0045] Table 1 shows the mIoU of each method on the Cityscapes validation set. The table reveals that the performance of each method gradually improves with increasing input resolution. When the resolution increases from 512×1024 to 1024×2048, the mIoU can be improved by approximately 4% to 5%, which fully demonstrates the positive impact of high-resolution images on semantic segmentation performance. With an input resolution of 512×1024, the proposed method outperforms other single-input semantic segmentation methods, even surpassing STDC2 and CFSCN at 768×1536 resolution and FPANet at 1024×2048 resolution. Compared to RGB-D semantic segmentation methods, at 512×1024 resolution, the mIoU of the proposed method is 2.4% and 1.2% higher than ESANet and GRDNet, respectively. This indicates that by introducing a stereo image super-resolution task, the proposed method can effectively improve segmentation performance under low-resolution input. Compared to the performance of multi-task methods DSNet and PMT based on stereo matching and semantic segmentation at 1024×2048 resolution, the proposed method also achieves competitive results. Compared to SISR-guided semantic segmentation methods DSRL and DS2F, the proposed method also shows a significant improvement in mIoU. These results fully demonstrate the effectiveness of using stereo image super-resolution reconstruction tasks to assist semantic segmentation. Stereo image super-resolution reconstruction tasks not only provide disparity information for semantic segmentation but also provide rich high-resolution detail information, thereby effectively improving segmentation performance.

[0046] also, Figure 5 This paper demonstrates the segmentation performance of the proposed method on the Cityscapes validation set, comparing it to SCTNet (the method with the highest mIoU at 512×1024 resolution). Figure 5 It can be observed that the method proposed in this invention performs better in segmenting small targets, such as traffic signs, utility poles, and traffic lights, and is also superior in distinguishing foreground and background, for example, it can more accurately distinguish between buildings and signs of similar colors. Figure 5 The first line), and vehicles and bicycles ( Figure 5 (Third line).

[0047] Table 1: mIoU of each method on the Cityscapes validation set

[0048] Table 2 shows the FLOPs required for each method when processing an input image with a resolution of 512×1024. As can be seen from the table, since the proposed method needs to simultaneously complete both semantic segmentation and stereo image super-resolution during the training and testing phases, although its performance is improved compared to single-input semantic segmentation methods and RGBD semantic segmentation methods, its computational cost is also correspondingly higher. Compared to the SISR-guided semantic segmentation methods DSRL and DS2F (both using Deeplabv3 as a baseline), the proposed method achieves higher performance with lower computational cost.

[0049] Table 2: FLOPs required for each method when the input resolution is 512×1024

[0050] To verify the effectiveness of each design in the method proposed in this invention, a series of ablation experiments were conducted, which will be described in detail below.

[0051] To verify the effectiveness of the proposed method in semantic segmentation tasks, this section first trains a standalone semantic segmentation network as a baseline, with input and output having the same resolution. Then, bilinear interpolation is added to the baseline network to form the semantic segmentation branch (SSB) in the proposed method, ensuring a high-resolution semantic segmentation map output. Next, a stereo image super-resolution branch (SRB), a disparity-aware fusion module (DAFM), and a semantic-detail interaction module (SDIM) are gradually introduced into the SSB. The experimental results are shown in Table 3. The results in the table show that when using a high-resolution semantic segmentation map as the label, the SSB improves the performance from 75.7% to 75.9%. By introducing SRB, mIoU is effectively improved by 1.1%, demonstrating the effectiveness of using super-resolution tasks to assist segmentation. When DAFM and SDIM are further combined, the overall performance is further improved to 77.4% (a 1.7% improvement over the baseline), illustrating the importance of transferring disparity information from stereo images and detail information recovered by the super-resolution branch to the semantic segmentation task.

[0052] Table 3: Impact of each component in the proposed method on segmentation performance

[0053] To verify the effectiveness of the proposed DAFM in this invention, DAFM was first removed from the semantic segmentation branch (w / o DAFM). Then, the disparity perception part in DAFM was replaced with simple pixel-by-pixel summation (Fusion by summation) or channel-by-channel concatenation (Fusion by concatenation), and then fed into the channel attention module to achieve disparity information fusion guidance for semantic segmentation. The experimental results are shown in Table 4. The results show that removing DAFM reduces segmentation accuracy, demonstrating the guiding role of disparity information in semantic segmentation. When disparity features are fused using simple summation or concatenation, the segmentation accuracy not only fails to improve but even decreases. This may be because disparity features and semantic features belong to different modalities and have significant differences; simple fusion methods easily interfere with the network's learning of effective features. The proposed DAFM, through attention-based disparity perception, transmits disparity information to the semantic segmentation branch, enabling the network to automatically strengthen features in regions related to disparity changes, thereby achieving effective guidance.

[0054] Table 4: Impact of DAFM on segmentation performance

[0055] To verify the effectiveness of the SDIM proposed in this invention, SDIM was first removed from the network (without SDIM), meaning that semantic features and super-resolution features did not interact after passing through the decoder. Subsequently, this section removed spatial interaction (SDIM with only CI) and channel interaction (SDIM with only SI) from SDIM to verify the specific impact of each component of SDIM on segmentation performance. The experimental results are shown in Table 5. The results show that when SDIM is completely removed (without SDIM), the mIoU on the Cityscapes validation set decreases, indicating that feature interaction has a positive effect on improving model performance. Further observation revealed that when only spatial interaction or channel interaction is retained, the mIoU decreases further compared to when SDIM is not present. This phenomenon may be because when spatial interaction is used alone, the model may focus too much on local regions, leading to insufficient utilization of global information and affecting the comprehensive understanding of semantic information in complex scenes, thus reducing segmentation performance. While channel interaction strengthens the extraction of global features, insufficient attention to local spatial information may result in inaccurate object edges and localization, which is also detrimental to improving segmentation performance. In summary, the SDIM module enhances the model's feature extraction capabilities by combining channel interaction and spatial interaction, and simultaneously modeling global and local information.

[0056] Table 5: Impact of SDIM on segmentation performance

[0057] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0058] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A semantic segmentation method based on stereo image super-resolution reconstruction guidance, characterized in that, include: S1, Input data preparation: Acquire a low-resolution stereoscopic image, the stereoscopic image including a left view. and right view ,in, and As network input data, H and W are the height and width of the image, respectively, and r represents the method scaling factor; S2, Constructing a Single-View Encoding: A symmetric encoder with shared weights is constructed based on ResNet34. The residual blocks are replaced with NBT1D, and the max-pooling layer after the first convolution is removed to extract features from the left and right views respectively. , ; S3, Stereo Image Super-Resolution Branch Processing: First, aggregate through a pyramid pooling module. , Multi-scale information is then processed by an improved disparity attention module to generate a disparity attention map. Finally, the super-resolution decoder recovers the feature size from the enhanced features, resulting in segmentation features augmented with disparity information. ; S4, semantic segmentation branch processing: After extracting multi-scale semantic information using the pyramid pooling module, it is combined with the disparity attention map. Semantic features are enhanced by DAFM and then processed by a semantic decoder to obtain refined segmentation features. ; S5, Feature Interaction: Left View Super-Resolution Refinement Features Segmentation features refined by the decoder The input is fed into the detailed interaction module, where it is fused through channel interaction, spatial interaction, and adaptive integration to generate the final interaction features. , ; S6, Output and Optimization: The super-resolution head and semantic segmentation head will respectively generate... , The figure shows the construction of a joint loss of "cross-entropy + MAE" to optimize the network until convergence.

2. The semantic segmentation method based on stereo image super-resolution reconstruction as described in claim 1, characterized in that, The single-view encoder described in S2 includes a symmetric, weight-shared left-view encoder and a right-view encoder. Both the left-view encoder and the right-view encoder are based on a pre-trained ResNet34 network, with the residual blocks in the ResNet34 network replaced by non-bottleneck 1D blocks (NBT1D). Simultaneously, the max-pooling layer after the first convolutional layer in the ResNet34 encoder is removed. The left-view encoder... The data is input into the left view encoder to obtain the left view features. , right view The input is fed into the right view encoder to obtain the right view features. ,in and The size is one-sixteenth of the input image size, and the feature extraction formula is: In the formula, This represents the feature extraction operation of a single-view encoder.

3. The semantic segmentation method based on stereo image super-resolution reconstruction as described in claim 1, characterized in that, The stereo image super-resolution branching process described in S3 specifically includes: S3.1, Multi-scale feature aggregation: and The shared pyramid pooling module is input separately. Through multi-size average pooling and convolution operations (1×1, 2×2, 4×4, 8×8) in the pyramid pooling module, contextual information at different scales is aggregated to obtain the left view features after multi-scale processing. and right view features ,in This represents the feature processing operations of the pyramid pooling module; S3.2, Cross-view feature enhancement: will and Input the improved parallax attention module to generate a parallax attention map. and left-view super-resolution features enhanced across views The operating formula is: in, Represents pyramid pooling module operations. This represents an improved feature enhancement operation for the parallax attention module. Represents a parallax attention map; S3.3, Super-resolution Feature Decoding: Construct a super-resolution decoder, which consists of three decoder modules and a bilinear upsampling layer. Each decoder module contains a 3×3 convolutional layer, three NBT1D blocks, and a bilinear upsampling layer; The input super-resolution decoder restores the feature map resolution to match the input image size through progressive 2x upsampling, resulting in the super-resolution refined features of the left view. The decoding formula is: in, This indicates the decoder used for super-resolution reconstruction.

4. The semantic segmentation method based on stereo image super-resolution reconstruction as described in claim 1, characterized in that, The semantic segmentation branch processing described in S4 specifically includes: S4.1, Multi-scale semantic feature extraction: Inputting the pyramid pooling module in S3.1 yields the left-view semantic features after multi-scale processing. ; S4.2, Parallax Perception Feature Enhancement: Construct a parallax perception fusion module to... And obtained from S3.1 The input is to the disparity-aware fusion module, which performs disparity acquisition and disparity perception operations to generate disparity-enhanced semantic features for the left view. The operating formula is: In the formula, Represents the parallax perception fusion module. These are segmentation features enhanced with disparity information; S4.3, Semantic Feature Decoding: Construct a semantic segmentation decoder, which has the same structure as the super-resolution decoder in S3.31 but does not share parameters; Input the semantic segmentation decoder to obtain the semantic refinement features of the left view. The decoding formula is: In the formula, This represents the decoder used for semantic segmentation, which has the same structure as the decoder in the stereo image super-resolution branch, but does not share parameters; This represents the segmentation features refined by the decoder.

5. A semantic segmentation method based on stereo image super-resolution reconstruction as described in claim 1, characterized in that, S5 specifically includes building a semantic detail interaction module, which takes the results obtained in S3.1 and uses them to build the semantic detail interaction module. And obtained from S4.3 The input detail interaction module, through its channel interaction, spatial interaction, and adaptive fusion operations, yields the final super-resolution interactive features. and semantic interaction features The operating formula is: In the formula, Semantic-detailed interaction module.

6. The semantic segmentation method based on stereo image super-resolution reconstruction as described in claim 1, characterized in that, S6 specifically includes constructing a super-resolution head and a semantic segmentation head. The super-resolution head contains two 3×3 convolutional layers and one sub-pixel convolutional layer. The semantic segmentation head contains a bilinear upsampling layer and a softmax layer. Input a super-resolution head to generate a high-resolution reconstructed image of the left view. ,Will Input semantic segmentation header to generate a high-resolution semantic segmentation map for the left view. ,in, and Where r is the super-resolution magnification, the generation formula is: In the formula, This represents a super-resolution head, which mainly consists of two 3×3 kernel convolutional layers and one subpixel convolutional layer; This represents the semantic segmentation header.

7. The semantic segmentation method based on stereo image super-resolution reconstruction as described in claim 1, characterized in that, The parallax perception fusion module described in S4.2 includes: Disparity Acquisition: For disparity acquisition, due to the disparity attention map from the stereo image super-resolution branch... The disparity map is obtained by calculating the similarity of features in the left and right views along the horizontal epipolar line. Therefore, the index of the point with the highest similarity can be used to roughly obtain the disparity map. Next, the disparity map is subjected to min-max normalization and then fed into a 3×3 convolutional layer and an NBT1D block to extract disparity features. The specific process is shown in the following formula. In the formula, Indicates the position of the right view Features and left view position The degree of matching of features. Indicates the position of the right view features relative to the left view. The index of the most relevant feature at a given location; This represents a convolutional layer operation with a kernel size of 3×3; Represents a non-bottleneck 1D block. Represents the extracted disparity features; Parallax perception: Input a 1×1 convolutional layer to generate a feature map Q, and then... A 3×3 deep convolutional layer is input to generate a feature map K; Q and K are multiplied pixel by pixel and activated by the sigmoid function to obtain a disparity-aware attention map, which is then combined with... Pixel-by-pixel multiplication yields preliminary enhanced semantic features. The specific formula is as follows: In the formula, This indicates a pixel-by-pixel multiplication operation. Represents the sigmoid function; Indicates parallax features; Will and The fused features are obtained by adding them pixel by pixel. These fused features are then input into the channel attention module to filter key channel information, resulting in the final enhanced semantic features. The calculation formula is: In the formula, This represents the channel attention mechanism.

8. The semantic segmentation method based on stereo image super-resolution reconstruction as described in claim 1, characterized in that, The specific operations of the detailed interaction module described in S5 include: S5.1 Channel Interaction: Input semantic features and super-resolution features The concatenation is performed along the channel dimension to obtain concatenated features; these concatenated features are then input into a global average pooling layer and a global max pooling layer to generate a global average pooling vector. and global max pooling vector ;Will and Cascaded along the channel dimension, inputting two fully connected layers and a sigmoid function sequentially to generate channel weight vectors. The specific calculation formula is as follows: In the formula, Represents the ReLU function. Represents the sigmoid function; and This represents the weight matrix of two fully connected layers. It is the channel compression ratio; Decomposed into semantic channel weights and super-resolution channel weights , respectively with semantic features and super-resolution features Adjustments are made by multiplying pixel by pixel to obtain the semantic features after channel interaction. and super-resolution features The calculation formula is: S5.2, Spatial Interaction: Global average pooling and max pooling are used to process semantic features and super-resolution features respectively to generate four feature maps. Unlike channel interactions, these operations are performed along the channel dimension to preserve spatial context information. To better utilize complementary information, the four feature maps are concatenated along the channel dimension and then fed into a dilated convolutional layer with a kernel size of 7×7 and a dilation rate of 3 to learn spatial relationships. Finally, a sigmoid layer is used to generate a spatial attention map. The specific formula for the operation is shown below. In the formula, This represents the dilated convolutional layer operation; by using dilated convolutional layers, the receptive field can be expanded, thereby capturing a wider range of contextual information and helping to generate more reliable spatial attention maps. Next... Divided into two attention maps and These are used to adjust semantic features and super-resolution features, respectively. The specific formulas for the process are shown below. In the formula, and These represent the semantic and super-resolution features after spatial interaction, respectively; Finally, the features adjusted through channel and spatial interaction learning are adaptively fused with the original features to obtain the final features for segmentation and reconstruction. and The specific formula for the process is shown below. In the formula, These are four learnable parameters used to adjust the intensity of channel interaction and spatial interaction.

9. A semantic segmentation method based on stereo image super-resolution reconstruction as described in claim 1, characterized in that, The semantic segmentation branch processing obtained in S4 includes semantic segmentation loss and super-resolution loss. The semantic segmentation loss adopts a conventional multi-class cross-entropy loss, which is used to measure the difference between the predicted class distribution and the true label. Its expression is as follows: In the formula, C represents the number of categories. It is the truth label corresponding to category i. The probability of class i predicted for the semantic segmentation branch; The super-resolution loss is defined as the mean absolute error (MAE) between the reconstructed stereo image and the true HR stereo image, and the formula is as follows: In the formula, The left view representing the super-resolution branch reconstruction of a stereo image; This represents the corresponding real HR image.

Images