A semantic discrete monocular depth estimation method based on adaptive pixel clustering

Abstract

The application provides a semantic discrete monocular depth estimation method based on adaptive pixel clustering, and relates to the technical field of image processing; specifically comprising: S1, preparing a data set to train a model; S2, constructing a semantic discrete monocular depth estimation network based on adaptive pixel clustering; S3, constructing a depth estimation encoder; S4, designing a multi-scale deformable feature refinement module to further refine the features of the depth estimation encoder; S5, designing a semantic discrete decoder; S6, inputting the RGB images and corresponding depth maps in the NYU and KITTI data sets into the semantic discrete monocular depth estimation network based on adaptive pixel clustering for training, and optimizing the network model weight through a scale invariant loss function, so as to obtain a monocular depth estimation model and save the model; S6, inputting an RGB image, performing inference through the trained monocular depth estimation model, and returning a predicted depth map. The application can pay attention to the continuity of the internal depth of an object and the discreteness of the depth between objects.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a semantic discrete monocular depth estimation method based on adaptive pixel clustering. Background Technology

[0002] Monocular depth estimation based on ordered regression has proven to be a promising alternative to geometric constraints. Fu et al. proposed dividing the depth range according to an incremental interval method and designed a pixel-to-pixel ordered regression loss function to simulate the ordered relationship between these depth sub-intervals. Bhat et al. proposed an adaptive bins strategy for ordered regression monocular depth estimation, and later proposed using all layers of the decoder to predict the local neighbor depth distribution at each pixel. Shao et al. proposed an iterative elastic bin depth interval discretization strategy, improving upon previous ordered regression methods and achieving state-of-the-art performance.

[0003] Monocular depth estimation has made significant progress compared to traditional methods after a long period of development. Current mainstream approaches combine it with deep learning, with state-of-the-art methods often employing convolutional neural networks or multi-scale Transformers as their backbone networks. Most methods are either geometrically constrained regression tasks or explicitly discrete depth range ordered regression tasks. However, due to the complex geometry of real-world scenes, it's difficult to model various arbitrary depth patterns using the geometric constraints of a single image. Furthermore, explicitly discretely distributing the depth values ​​of individual pixels may disrupt the inherent depth distribution relationships between adjacent pixels. In addition, depth distributions in real-world scenes are both continuous and discrete; the depth between objects is often discrete, with only the depth values ​​within an object exhibiting a certain degree of continuity. Current techniques cannot simultaneously consider the continuity of depth within objects and the discreteness of depth between objects.

[0004] Therefore, a semantic discrete monocular depth estimation method based on adaptive pixel clustering is proposed to solve the above problems. Summary of the Invention

[0005] This invention provides a semantically discrete monocular depth estimation method based on adaptive pixel clustering. It combines an adaptive pixel clustering module for semantic discretization of deep features from the encoder with a cross-semantic alignment module to align the discretized feature maps to a larger resolution, thus addressing both the continuity of depth within objects and the discreteness of depth between objects. Simultaneously, a deformable feature pyramid module with sparse attention is used for multi-scale feature fusion, resolving the quadratic complexity issue introduced by the standard Transformer as a depth estimation decoder.

[0006] To achieve the above objectives, this invention provides a semantic discrete monocular depth estimation method based on adaptive pixel clustering, specifically including the following steps:

[0007] S1. Prepare the dataset to train the model. The dataset includes the indoor NYU dataset and the outdoor KITTI dataset for depth estimation.

[0008] S2. Construct a semantic discrete monocular depth estimation network based on adaptive pixel clustering. This network consists of three parts: a depth estimation encoder, a multi-scale deformable feature refinement module, and a semantic discrete decoder.

[0009] S3. Construct a depth estimation encoder by using a multi-scale visual Transformer pre-trained on the ImageNet dataset as a depth estimation encoder to extract multi-scale features.

[0010] S4. Design a deformable feature refinement module to further refine the features of the depth estimation encoder through multi-scale deformation attention;

[0011] S5. Design a semantic discrete decoder, which includes an adaptive pixel clustering module, a deformable feature pyramid module, and a cross-semantic alignment module.

[0012] S6. Match the RGB images from the NYU and KITTI datasets with their corresponding depth values. Figure 1 The input is fed into a semantic discrete monocular depth estimation network based on adaptive pixel clustering for training, and the network model weights are optimized through a scale-invariant loss function to obtain and save the monocular depth estimation model;

[0013] S7. Input an RGB image, perform inference through a trained monocular depth estimation model, and return the predicted depth map.

[0014] Preferably, in step S1, the NYU dataset includes a training set and a test set. The training set contains 24,231 samples, and the test set contains 654 samples. The NYU dataset includes 464 indoor scenes, which consist of RGB images and quasi-density depth images with a resolution of 640×480. The KITTI dataset provides stereo images of outdoor scenes and corresponding 3D laser scanning depth maps. This dataset includes 23,488 pairs of training images and 697 test images. The resolution of the RGB images in the KITTI dataset is approximately 1241×376, but the corresponding depth images have a large amount of missing data and low density.

[0015] Preferably, step S2 specifically includes the following steps: S1: Extracting multi-scale encoder features using a depth estimation encoder;

[0016] S2: The multi-scale encoder features are further refined using the multi-scale deformable feature refinement module;

[0017] S3: Utilize a semantic discrete decoder for multi-scale feature fusion and semantic discretization.

[0018] Preferably, in step S3, the Swin Transformer pre-trained on the ImageNet dataset is used as the encoder. The Swin Transformer includes four variants: Swin-T, Swin-S, Swin-B, and Swin-L. Swin-L is used as the encoder for monocular depth estimation to extract multi-scale features.

[0019] Preferably, in step S4, the encoded features at different resolutions are refined using a multi-scale deformable feature refinement module.

[0020] set up Let the multi-scale feature mapping of the input be... The normalized coordinates of the reference point for each query element q are represented by the multi-scale deformable feature refinement module as follows:

[0021]

[0022] Among them, z q Let m represent the attention head index, l represent the input feature layer index, k represent the sampling point, and Δp represent the content features. mlqk and A mlqk W represents the sampling offset and attention weight of the k-th sampling point in the l-th feature layer and the m-th attention head, respectively. m and W' m Represents the learnable weights, φ l Represents normalized coordinates The processing function, x l This represents the characteristics of the l-th layer encoder.

[0023] Preferably, step S5 specifically includes the following steps:

[0024] S51: The adaptive pixel clustering module refines the minimum resolution features extracted by the encoder and then clusters them to obtain a feature map composed of semantic discrete blocks with scene spatial information.

[0025] S52: The deformable feature pyramid module consists of three identical layers. It uses deformable cross attention to perform feature fusion in the form of a feature pyramid, which refines features from encoders at different resolutions and decoders at different levels.

[0026] The mathematical expression for deformable cross attention is as follows:

[0027]

[0028] Where M represents the total number of attention heads, K represents the total number of sampling keys, and K << HW, W m , and W mqk Represents learnable weights; q indexes a z with content features. q The query element, Δp mqk and σ(W) mqk x) represents the sampling offset and attention weight of the k-th sampling point in the m-th attention head, respectively, where σ represents the softmax function, and v and x represent input features from different sources.

[0029] S53: The cross-semantic alignment module aligns the discretized feature maps from the adaptive pixel clustering module to a high-resolution feature space. This module includes a channel convolutional attention module that models its channel features. The channel convolutional attention module consists of depthwise convolutions, Gaussian error linear units, SE blocks, and ordinary convolutions. Cross-semantic cross-attention is calculated by fusing the output features of the channel convolutional attention module with the features fused by the deformable feature pyramid module. The cross-semantic alignment module is represented as:

[0030] CCA(x) = Conv 1×1 (SE(GELU(DW 3×3 (x))))

[0031]

[0032] CSA(Q i ,K,V) i =Softmax(Q i K T / d)V

[0033] DW 3×3 Represents a 3×3 depthwise convolution, Conv 1×1 This represents a 1×1 ordinary convolution. Let represent the linear projection function, which upsamples the (i+1)th layer output of the cross-semantic alignment module and uses it as the input of the i-th layer of the deformable feature pyramid module for feature fusion with the refined features of the encoder at the same resolution. Let x represent the output feature of the adaptive pixel clustering module. Q represents the output feature of the i-th layer of the deformable feature pyramid module. i K and V represent the query, key, and value of cross-attention, respectively. T Let K denote the transpose of matrix K, and d denote the scaling factor.

[0034] Preferably, step S51 includes three processes: clustering, intra-cluster feature aggregation, and aggregated feature allocation;

[0035] The mathematical expressions for clustering and intra-cluster feature aggregation are as follows:

[0036]

[0037] Where σ represents the softmax function. This represents the dot product operation, where α and β are both learnable parameters, A represents the features after pixel aggregation within each cluster, and C... v and C x Let S represent different sets of centroids, and S represent the similarity matrix. i F represents a subset of the similarity matrix S. v and F x These represent different sets of pixels, and L represents the normalization factor.

[0038] Preferably, in step S6: the RGB images from the NYU and KITTI datasets are compared with their corresponding depth values. Figure 1 The input is fed into a semantic discrete monocular depth estimation network based on adaptive pixel clustering for training. The model weights are then optimized using a scale-invariant loss function to obtain and save the monocular depth estimation model. The scale-invariant loss function is shown below:

[0039]

[0040] Where, d i g represents the true depth of the ground. i This represents the logarithmic difference between the true depth and the predicted depth. λ represents the prediction depth, T represents the number of valid pixels in the true label, and λ is a hyperparameter that controls the scale invariance of the loss; λ = 1 indicates complete scale invariance, and λ = 0 indicates the ordinary L2 loss function.

[0041] Preferably, in step S7, during the inference stage, an RGB image is input, and the trained monocular depth estimation model is used for inference to return a predicted depth map.

[0042] Therefore, this invention employs the aforementioned semantic discrete monocular depth estimation method based on adaptive pixel clustering. It combines an adaptive pixel clustering module for semantic discretization of deep features from the encoder with a cross-semantic alignment module to align the discretized feature maps to a larger resolution. This approach addresses both the continuity of depth within objects and the discreteness of depth between objects. Simultaneously, a deformable feature pyramid module with sparse attention is used for multi-scale feature fusion, resolving the quadratic complexity issue introduced by the standard Transformer as a depth estimation decoder.

[0043] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating a semantic discrete monocular depth estimation method based on adaptive pixel clustering according to the present invention.

[0045] Figure 2 This is a network structure diagram of a semantic discrete monocular depth estimation method based on adaptive pixel clustering according to the present invention.

[0046] Figure 3 This is a graph showing the results of this invention on the NYU dataset;

[0047] Figure 4 This is a graph showing the results of this invention on the KITTI dataset. Detailed Implementation

[0048] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0049] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0050] Example

[0051] like Figure 1 As shown, this invention provides a semantic discrete monocular depth estimation method based on adaptive pixel clustering, specifically including the following steps:

[0052] S1. Prepare the datasets for model training. The datasets include the indoor NYU dataset and the outdoor KITTI dataset for depth estimation. In step S1, the NYU dataset contains a training set and a test set. The training set contains 24,231 samples, and the test set contains 654 samples. The NYU dataset includes 464 indoor scenes, which consist of RGB images and quasi-density depth images with a resolution of 640×480. The KITTI dataset provides stereo images of outdoor scenes and corresponding 3D laser scanning depth maps. This dataset includes 23,488 pairs of training images and 697 test images. The resolution of the RGB images in the KITTI dataset is approximately 1241×376, but the corresponding depth images have a large amount of missing data and low density.

[0053] S2. Construct a semantic discrete monocular depth estimation network based on adaptive pixel clustering. The network consists of three parts: a depth estimation encoder, a multi-scale deformable feature refinement module, and a semantic discrete decoder. In step S2, the depth estimation encoder is first used to extract multi-scale encoder features, then the multi-scale deformable feature refinement module is used to further refine the multi-scale encoder features, and finally the semantic discrete decoder is used to perform multi-scale feature fusion and semantic discretization.

[0054] S3. Construct a depth estimation encoder. Extract multi-scale features by using a multi-scale visual Transformer pre-trained on the ImageNet dataset as the depth estimation encoder. In step S3, the Swing Transformer pre-trained on the ImageNet dataset is used as the encoder. The Swing Transformer includes four variants: Swing-T, Swing-S, Swing-B, and Swing-L. Swing-L is used as the encoder for monocular depth estimation to extract multi-scale features.

[0055] S4. Design a deformable feature refinement module to further refine the features of the depth estimation encoder through multi-scale deformable attention; in step S4, the multi-scale deformable feature refinement module is used to refine the encoded features at different resolutions.

[0056] set up Let the multi-scale feature mapping of the input be... The normalized coordinates of the reference point for each query element q are represented by the multi-scale deformable feature refinement module as follows:

[0057]

[0058] Among them, z q Let m represent the attention head index, l represent the input feature layer index, k represent the sampling point, and Δp represent the content features. mlqk and Amlqk W represents the sampling offset and attention weight of the k-th sampling point in the l-th feature layer and the m-th attention head, respectively. m and W' m Represents the learnable weights, φ l Represents normalized coordinates The processing function, x l This represents the characteristics of the l-th layer encoder.

[0059] S5. Design a semantic discrete decoder, which includes an adaptive pixel clustering module, a deformable feature pyramid module, and a cross-semantic alignment module; step S5 specifically includes the following steps:

[0060] S51: The adaptive pixel clustering module refines the minimum resolution features extracted by the encoder and then clusters them to obtain a feature map composed of semantic discrete blocks with scene spatial information at different spatial locations; step S51 includes three processes: clustering, intra-cluster feature aggregation, and aggregated feature allocation;

[0061] For each input feature X∈R C×H×W , where H and W represent the height and width of the input feature, respectively, and C represents the number of channels of the input feature.

[0062] First, let X∈R C×H×W Linear projection to P x ∈R C×H×W and P v ∈R C×H×W Similarity calculations are performed, and the resulting pixel set F is obtained after reshaping. x ∈R N×D and F v ∈R N×D , where N = H × W represents all pixels and D represents the sequence length;

[0063] By using two different depthwise convolutions with a kernel size of 7×7, a stride of 3, and a padding size of half the kernel size, P... x and P v The resolution is reduced to 1 / 3 of its original value. During this process, a 7×7 sliding window is used to adaptively assign a center point to each pixel within the window, and then a reshaping operation is performed to obtain the center point set Cx∈R. MxD and C v ∈R M×D Where M = H / 3 × W / 3 represents the number of clusters and the number of centers within each cluster. Calculate the set F of all pixels. x ∈R N×D With the center point set C x ∈R M×D Pairwise cosine similarity between S∈R M×NThis allows us to obtain the similarity between all center points and each pixel, for the similarity matrix S∈R. M×N Perform a Mask operation to retain only the maximum value in the M-dimensional similarity matrix and set the rest to zero. Based on this result, assign each pixel to the center with the most similarity, thus obtaining M different clusters.

[0064] In extreme cases, some clusters may not have a single pixel. It is necessary to dynamically aggregate the cluster features into a single pixel based on the similarity between all points in the cluster and the center, and then perform a softmax operation on the inter-cluster dimension to maintain a discrete relationship between the aggregated feature points.

[0065] Specifically, it involves masking the similarity matrix S and then comparing the result with F. v By performing a dot product operation, we can obtain the feature A∈R after aggregating the pixels within each cluster. M×D The dot product operation includes two processes: assigning pixels to individual clusters and cluster feature aggregation. This is because matrix multiplication involves multiplication and addition between vectors. Multiplication is the process of assigning pixels to the corresponding clusters, while addition is the process of aggregating all pixels in a cluster into a single pixel.

[0066] The mathematical expressions for clustering and intra-cluster feature aggregation are as follows:

[0067]

[0068] Where σ represents the softmax function. This represents the dot product operation, where α and β are both learnable parameters, A represents the features after pixel aggregation within each cluster, and C... v and C x Let S represent different sets of centroids, and S represent the similarity matrix. i F represents a subset of the similarity matrix S. v and F x These represent different sets of pixels, and L represents the normalization factor.

[0069] In the extreme case where there are no pixels, aggregate feature A∈R M×D With the center point set C v ∈R M×D Adding them together ensures the effectiveness of the aggregated features;

[0070] The aggregated features are dynamically assigned to each point within a cluster based on similarity, so that each point can share the features of all points within the cluster, while maintaining discrete relationships between different clusters.

[0071] Specifically, this is represented by the similarity matrix S∈R. M×N The result after applying the mask and the aggregated features A∈R M×DDot product operations are performed. After reshaping, the feature map resolution is restored to the original input size, ensuring that the output feature map contains scene spatial information and is composed of semantically discrete blocks at different spatial locations, rather than a single feature embedding. The result of the aggregated feature allocation is then fed into an MLP to obtain the final output features of the adaptive pixel clustering module.

[0072] S52: The deformable feature pyramid module consists of three identical layers. It uses deformable cross attention to perform feature fusion in the form of a feature pyramid, which refines features from encoders at different resolutions and decoders at different levels.

[0073] The mathematical expression for deformable cross attention is as follows:

[0074]

[0075] Where M represents the total number of attention heads, K represents the total number of sampling keys, and K << HW, W m , and W mqk Represents learnable weights; q indexes a z with content features. q The query element, Δp mqk and σ(W) mqk x) represents the sampling offset and attention weight of the k-th sampling point in the m-th attention head, respectively, where σ represents the softmax function, and v and x represent input features from different sources.

[0076] S53: The cross-semantic alignment module aligns the discretized feature maps from the adaptive pixel clustering module to a higher-resolution feature space. This module includes a channel convolutional attention module that models its channel features. The channel convolutional attention module consists of depthwise convolutions, Gaussian error linear units, SE blocks, and ordinary convolutions. Cross-semantic cross-attention is calculated by fusing the output features of the channel convolutional attention module with the features fused by the deformable feature pyramid module. The cross-semantic alignment module is represented as:

[0077] CCA(x) = Conv 1×1 (SE(GELU(DW 3×3 (x))))

[0078]

[0079] CSA(Q i ,K,V) i =Softmax(Q i K T / d)V

[0080] DW 3×3 Represents a 3×3 depthwise convolution, Conv1×1 This represents a 1×1 ordinary convolution. Let represent the linear projection function, which upsamples the (i+1)th layer output of the cross-semantic alignment module and uses it as the input of the i-th layer of the deformable feature pyramid module for feature fusion with the refined features of the encoder at the same resolution. Let x represent the output feature of the adaptive pixel clustering module. Q represents the output feature of the i-th layer of the deformable feature pyramid module. i K and V represent the query, key, and value of cross-attention, respectively. T / d represents the transpose of matrix K divided by a scaling factor d.

[0081] S6. Match the RGB images from the NYU and KITTI datasets with their corresponding depth values. Figure 1 The input is fed into a semantic discrete monocular depth estimation network based on adaptive pixel clustering for training, and the network model weights are optimized through a scale-invariant loss function to obtain and save the monocular depth estimation model;

[0082] In step S6: The RGB images from the NYU and KITTI datasets are compared with their corresponding depth values. Figure 1 The input is fed into the semantic discrete monocular depth estimation network based on adaptive pixel clustering for training, and the model weights are optimized using a scale-invariant loss function to obtain and save the monocular depth estimation model. The scale-invariant loss function is shown below:

[0083]

[0084] Where, d i g represents the true depth of the ground. i This represents the logarithmic difference between the true depth and the predicted depth. λ represents the prediction depth, T represents the number of valid pixels in the true label, and λ is a hyperparameter that controls the scale invariance of the loss; λ = 1 indicates complete scale invariance, and λ = 0 indicates the ordinary L2 loss function.

[0085] S7. Input an RGB image, perform inference using the trained monocular depth estimation model, and return the predicted depth map. In step S6, during the inference stage, input an RGB image, perform inference using the trained monocular depth estimation model, and return the predicted depth map.

[0086] Comparative Example

[0087] This approach uses commonly used metrics in the field of monocular depth estimation to perform quantitative evaluations on the NYU and KITTI datasets, including absolute relative error (Abs Rel), squared relative error (Squ Rel), linear root mean square error (RMSE), exponential root mean square error (RMSE log), average log10 error, and accuracy.

[0088] This approach qualitatively and quantitatively compares the proposed method with nine previous state-of-the-art methods on the NYU indoor dataset, and the results are as follows: Figure 3 As shown in Table 1, the proposed method surpasses previous state-of-the-art methods. Furthermore, the larger object size in the indoor dataset facilitates semantic discretization by the adaptive pixel clustering module. Figure 3 The proposed method was qualitatively compared with previous methods such as AdaBins and NeWCRFs. Visualizations show that the proposed method estimates depth more accurately; for example, the depth values ​​of the book (row 1) and carpet (rows 2 and 4) in the RGB image should match the room floor. Furthermore, the proposed method estimates objects more clearly, such as the bench (row 2), laptop (row 3), and fan (row 5) in the RGB image.

[0089] Table 1

[0090] TransDepth ResNet-50+ViT 0.106 0.365 0.045 0.900 0.983 0.996 DPT VIT-Hybrid 0.110 0.357 0.045 0.904 0.988 0.998 LapDepth ResNext-101 0.110 0.393 0.047 0.885 0.979 0.995 AdaBins EfficientNet-B5 0.103 0.364 0.044 0.903 0.984 0.997 NeWCRFs Swin-L 0.095 0.334 0.041 0.922 0.992 0.998 NDDepth Swin-L 0.087 0.311 0.038 0.936 0.991 0.998 IEBins Swin-L 0.087 0.314 0.038 0.936 0.992 0.998 MG Swin-L 0.087 0.311 - 0.933 - - VA-DepthNet Swin-L 0.086 0.304 - 0.937 0.992 - This invention Swin-L 0.086 0.313 0.037 0.939 0.993 0.998

[0091] This approach qualitatively and quantitatively compares the proposed method with nine previous state-of-the-art methods on the outdoor dataset KITTI, and the results are as follows: Figure 4 As shown in Table 2, the experimental results of the proposed method on KITTI did not surpass the previous state-of-the-art methods, but only achieved comparable performance. There are two possible reasons for this: First, KITTI is an outdoor dataset containing many trees, shrubs, and other objects with numerous fine branches in the images; however, after multiple downsampling, the information of these branches may have been lost in the encoder feature map with the lowest resolution. Second, the batch size of the experiments using this method may be too small for the KITTI dataset. Figure 4 The proposed method was qualitatively compared with previous methods such as AdaBins and NeWCRFs. Visualizations show that the proposed method can more accurately estimate the depth information of more distant objects.

[0092] Table 2

[0093]

[0094] Therefore, this invention employs the aforementioned semantic discrete monocular depth estimation method based on adaptive pixel clustering. It combines an adaptive pixel clustering module for semantic discretization of deep features from the encoder with a cross-semantic alignment module to align the discretized feature maps to a larger resolution. This approach addresses both the continuity of depth within objects and the discreteness of depth between objects. Simultaneously, a deformable feature pyramid module with sparse attention is used for multi-scale feature fusion, resolving the quadratic complexity issue introduced by the standard Transformer as a depth estimation decoder.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A semantic discrete monocular depth estimation method based on adaptive pixel clustering, characterized in that, Specifically, the following steps are included: S1. Prepare the dataset to train the model. The dataset includes the indoor NYU dataset and the outdoor KITTI dataset for depth estimation. S2. Construct a semantic discrete monocular depth estimation network based on adaptive pixel clustering, which includes three parts: a depth estimation encoder, a multi-scale deformable feature refinement module, and a semantic discrete decoder. S3. Construct a depth estimation encoder by using a multi-scale visual Transformer pre-trained on the ImageNet dataset as a depth estimation encoder to extract multi-scale features. S4. Design a multi-scale deformable feature refinement module to further refine the features of the depth estimation encoder; S5. Design a semantic discrete decoder, which includes an adaptive pixel clustering module, a deformable feature pyramid module, and a cross-semantic alignment module. Step S5 specifically includes the following steps: S51: The adaptive pixel clustering module refines the minimum resolution features extracted by the encoder and then clusters them to obtain a feature map composed of semantic discrete blocks with scene spatial information. S52: The deformable feature pyramid module consists of three identical layers. It uses deformable cross attention to perform feature fusion in the form of a feature pyramid, which refines features from encoders at different resolutions and decoders at different levels. The mathematical expression for deformable cross attention is as follows: Where M represents the total number of attention heads, K represents the total number of sampling keys, and K HW, and Indicates learnable weights; q Index a content-based feature z q The query element, and They represent the first m The first thing to note is the head. k The sampling offset and attention weight of each sampling point, where This represents the softmax function. and Represents input features from different sources; S53: The cross-semantic alignment module aligns the discretized feature maps from the adaptive pixel clustering module to a high-resolution feature space. This module includes a channel convolutional attention module that models its channel features. The channel convolutional attention module consists of depthwise convolutions, Gaussian error linear units, SE blocks, and ordinary convolutions. Cross-semantic cross-attention is calculated by fusing the output features of the channel convolutional attention module with the features fused by the deformable feature pyramid module. The cross-semantic alignment module is represented as: in This represents a 3×3 depthwise convolution. This represents a 1×1 ordinary convolution. Represents a linear projection function, which will project the first semantically aligned module across the [module name missing]. i+1 The layer output, after upsampling, is used as the first layer of the deformable feature pyramid module. i The layer input is fused with the refined features of the encoder at the same resolution; x This represents the output features of the adaptive pixel clustering module. The first module of the deformable feature pyramid represents the deformable feature. i The output features of the layer These represent the query, key, and value for cross-attention, respectively. Representation matrix K transpose, d Indicates the scaling factor; S6. Input the RGB images and corresponding depth maps from the NYU and KITTI datasets into a semantic discrete monocular depth estimation network based on adaptive pixel clustering for training. Optimize the network model weights using a scale-invariant loss function to obtain and save the monocular depth estimation model. S7. Input an RGB image, perform inference through a trained monocular depth estimation model, and return the predicted depth map.

2. The semantic discrete monocular depth estimation method based on adaptive pixel clustering according to claim 1, characterized in that: In step S1, the NYU dataset contains a training set and a test set. The training set contains 24,231 samples, and the test set contains 654 samples. The NYU dataset includes 464 indoor scenes, which include RGB images and quasi-density depth images with a resolution of 640×480. The KITTI dataset provides stereo images of outdoor scenes and corresponding 3D laser scanning depth maps. The KITTI dataset includes 23,488 pairs of training images and 697 test images. The resolution of the RGB images in the KITTI dataset is 1241×376.

3. The semantic discrete monocular depth estimation method based on adaptive pixel clustering according to claim 1, characterized in that: In step S2, specifically Includes the following steps: S1: Extract multi-scale encoder features using a depth estimation encoder; S2: The multi-scale encoder features are further refined using the multi-scale deformable feature refinement module; S3: Utilize a semantic discrete decoder for multi-scale feature fusion and semantic discretization.

4. The semantic discrete monocular depth estimation method based on adaptive pixel clustering according to claim 1, characterized in that: In step S3, the Swin Transformer pre-trained on the ImageNet dataset is used as the encoder. The Swin Transformer includes four variants: Swin-T, Swin-S, Swin-B, and Swin-L. Swin-L is used as the encoder for monocular depth estimation to extract multi-scale features.

5. The semantic discrete monocular depth estimation method based on adaptive pixel clustering according to claim 1, characterized in that: In step S4, the encoded features at different resolutions are refined using the multi-scale deformable feature refinement module. set up Let the multi-scale feature mapping of the input be... ∈[0,1] 2 For each query element q Normalized coordinates of the reference point q Index a content-based feature The query element, represented by the multi-scale deformable feature refinement module, is as follows: in, Indicates content characteristics, m Note the header index. l Indicates the input feature layer index. k Indicates the sampling point. mlqk and A mlqk They represent the first l The first feature layer and the first m The first thing to note k The sampling offset and attention weight of each sampling point and Indicates learnable weights, Represents normalized coordinates The processing function, Indicates the first l Layer encoder features.

6. The semantic discrete monocular depth estimation method based on adaptive pixel clustering according to claim 1, characterized in that: Step S51 includes three processes: clustering, intra-cluster feature aggregation, and aggregated feature allocation. The mathematical expressions for clustering and intra-cluster feature aggregation are as follows: in This represents the softmax function. Represents the dot product operation. All are learnable parameters. A This represents the feature after pixel aggregation within each cluster. and They represent different sets of centroids. S Represents the similarity matrix. Representing the similarity matrix S A subset, and Each represents a different set of pixels. L This represents the normalization factor.

7. The semantic discrete monocular depth estimation method based on adaptive pixel clustering according to claim 1, characterized in that: In step S6: RGB images and corresponding depth maps from the NYU and KITTI datasets are input together into a semantic discrete monocular depth estimation network based on adaptive pixel clustering for training. The model weights are then optimized using a scale-invariant loss function to obtain and save the monocular depth estimation model. The scale-invariant loss function is shown below: in, Indicates the true depth of the ground. This represents the logarithmic difference between the true depth and the predicted depth. Indicates the predicted depth. T This represents the number of valid pixels in the actual label. It is a hyperparameter that controls the scale invariance of the loss; =1 indicates complete scale invariance. =0 indicates the L2 loss function.

8. The semantic discrete monocular depth estimation method based on adaptive pixel clustering according to claim 1, characterized in that: In step S7, during the inference phase, an RGB image is input, and the trained monocular depth estimation model is used for inference to return a predicted depth map.

Images