ViT-based monocular depth estimation and semantic segmentation method and system

Abstract

The invention discloses a monocular depth estimation and semantic segmentation method and system based on ViT, and relates to the technical field of artificial intelligence, and the method comprises the steps: receiving a to-be-processed image, inputting the to-be-processed image into a pre-established encoder based on a ViT backbone network, segmenting the to-be-processed image into blocks which are not overlapped, and carrying out the segmentation of the to-be-processed image; mapping the square blocks into a feature space to obtain lexical elements; inputting the lexical elements into a Reassemble block of an encoder of a pre-established ViT backbone network for combination processing to obtain a feature representation of a class image, and processing the feature representation of the class image through a Fusion block of a convolutional decoder to obtain a feature representation of a processed image; splicing the feature representation of the similar image and the feature representation of the processed image to obtain a spliced feature representation, inputting the spliced feature representation into two continuous convolution units of a convolution decoder for up-sampling operation processing, and obtaining a final feature map through continuous operation processing of a Fusion block; and inputting the final feature map into a pre-constructed monocular depth estimation and semantic segmentation task head, and calculating to obtain the total loss.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a ViT-based monocular depth estimation and semantic segmentation method and system. Background Art

[0002] Scene depth estimation and semantic segmentation play an important role in the field of machine vision. It enables machines to enhance their perception and understanding of real three-dimensional scenes. Therefore, it has a wide range of applications in tasks such as robot navigation, autonomous driving, and virtual reality. From models based on manually designed features to models trained based on binocular camera images, and then to models trained based on monocular camera images, it reflects the pursuit of a higher level of artificial intelligence.

[0003] The common practice for handling these tasks is to divide the model into two parts, an encoder and a decoder. The encoder is usually based on an image classification network, also known as a backbone network, which is always pre-trained on a large image dataset such as ImageNet. The decoder is responsible for combining the feature representations output by the encoder and generating the final prediction result.

[0004] The convolution-based backbone network extracts feature representations of different scales by downsampling the input image. The downsampling operation can gradually increase the receptive field and aggregate low-level features to form more abstract high-level features, while ensuring that the network model is affordable for computational and memory overheads. However, the downsampling operation exhibits obvious shortcomings when processing intensive tasks, causing the loss of feature resolution and granularity in the deep stage of the model, which is difficult to recover during the decoder processing. Summary of the invention

[0005] In order to solve the deficiencies mentioned in the above background technology, the purpose of the present invention is to provide a monocular depth estimation and semantic segmentation method and system based on ViT.

[0006] In a first aspect, the purpose of the present invention can be achieved by the following technical solution: a monocular depth estimation and semantic segmentation method based on ViT, the method comprising the following steps:

[0007] Receive an image to be processed, input the image to be processed into a pre-established encoder based on the ViT backbone network, divide the image to be processed into non-overlapping blocks, and map the blocks into a feature space to obtain word units;

[0008] The word unit is input into the Reassemble block of the encoder of the pre-established ViT backbone network for combination processing to obtain the feature representation of the class image, and the feature representation of the class image is processed by the Fusion block of the convolution decoder to obtain the feature representation of the processed image;

[0009] The feature representation of the class image is concatenated with the feature representation of the processed image to obtain a concatenated feature representation, which is input into two consecutive convolution units of the convolution decoder for upsampling operation, and the final feature map is obtained through continuous operation of the Fusion block;

[0010] The final feature map is input into the pre-built monocular depth estimation and semantic segmentation task head to calculate the total loss.

[0011] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the encoder based on the ViT backbone network processes the image into p 2 non-overlapping blocks, which are then flattened into tensors and mapped into embedding representations using linear layers.

[0012] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the process of generating embedding in the encoder based on the ViT backbone network is as follows: a picture of H×W pixels is processed into a series of word units in represents the readout word, and D represents the feature dimension of each word;

[0013] The input word is passed through l transformer layers to form a feature representation t l , where l represents the output of the lth layer.

[0014] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the expression of the decomposition of the Reassemble block is as follows:

[0015]

[0016] Among them, s represents the ratio of the features restored to the input image to be processed, Indicates the dimension of the output features.

[0017] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the convolution decoder converts N p +1 word is mapped into a set of N p word units are concatenated into image-like feature representations. The operation is shown as follows:

[0018]

[0019] This operation is responsible for processing the readout token.

[0020] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the method for processing the readout word unit is to splice the information of the readout word unit to all other word units, and the operation expression is as follows:

[0021]

[0022] After the Read operation, the obtained N is embedded according to the position p The word units are reassembled into image-like feature representations, and spatial concatenation is applied to obtain a D-channel image with a size of The feature map of the operation is as follows:

[0023]

[0024] Finally, the features are passed to the Resample layer, which scales the feature representation to a size of D channels. The feature representation of .

[0025] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a loss function of the pre-built monocular depth estimation and semantic segmentation task head is as follows:

[0026] The dataset D contains the original image x i , and the corresponding depth estimation map And the annotated semantic segmentation map Loss Function Loss estimated from depth and semantic segmentation loss composition;

[0027]

[0028] For the monocular depth estimation task, the loss function used is the scale and translation invariant cropping loss:

[0029]

[0030] Among them, y is the true value, is the predicted value. is the loss of a single data point, selected as the L2 loss function, S is the set of selected data points, the set of data point indices after removing outliers, and |S represents the number of elements in the set S;

[0031] The construction process of set S is as follows:

[0032]

[0033] Where N is the total number of data points and α is the removal ratio;

[0034] The mathematical expression of the masked mean square error MSE loss function is as follows:

[0035]

[0036] in, is the total loss, is the predicted value, y is the true value, m is the mask matrix, M = ∑ i,j m i,j is the number of valid elements of each sample, and reduction is the reduction operation performed on the loss of each sample.

[0037] In a second aspect, in order to achieve the above-mentioned object, the present invention discloses a monocular depth estimation and semantic segmentation system based on ViT, comprising:

[0038] A coding extraction module is used to receive the image to be processed, input the image to be processed into a pre-established encoder based on the ViT backbone network, divide the image to be processed into non-overlapping blocks, and map the blocks into a feature space to obtain word units;

[0039] A decoding processing module is used to input the word unit into the Reassemble block of the encoder of the pre-established ViT backbone network for combination processing to obtain the feature representation of the class image, and process the feature representation of the class image through the Fusion block of the convolution decoder to obtain the feature representation of the processed image;

[0040] A splicing processing module is used to splice the feature representation of the class image with the feature representation of the processed image to obtain a spliced ​​feature representation, and input the spliced ​​feature representation into two consecutive convolution units of the convolution decoder for upsampling operation processing, and obtain the final feature map through continuous operation processing of the Fusion block;

[0041] The monocular depth estimation and semantic segmentation module is used to input the final feature map into the pre-built monocular depth estimation and semantic segmentation task head to calculate the total loss.

[0042] In combination with the second aspect, in some implementations of the second aspect, the system further includes: the encoding extraction module processes the image into p based on the encoder of the ViT backbone network 2 non-overlapping squares, which are then flattened into tensors and mapped into embedding representations using linear layers;

[0043] The process of generating embeddings in the encoder of the ViT backbone network is as follows: an H×W pixel image is processed into a series of word units Where N p = represents the readout word, and D represents the feature dimension of each word;

[0044] The input word is passed through l transformer layers to form a feature representation t l , where l represents the output from the lth layer;

[0045] The expression of the decoding processing module Reassemble block decomposition is as follows:

[0046]

[0047] Among them, s represents the ratio of the features restored to the input image to be processed, Indicates the dimension of the output features;

[0048] The decoding processing module convolution decoder converts N p +1 word is mapped into a set of N p word units are concatenated into image-like feature representations. The operation is shown as follows:

[0049]

[0050] This operation is responsible for processing the readout token.

[0051] The method of processing the readout word in the decoding processing module is to splice the information of the readout word to all other words. The operation expression is as follows:

[0052]

[0053] After the Read operation, the obtained N is embedded according to the position p The word units are reassembled into image-like feature representations, and spatial concatenation is applied to obtain a D-channel image with a size of The feature map of the operation is as follows:

[0054]

[0055] Finally, the features are passed to the Resample layer, which scales the feature representation to a size of D channels. The feature representation of

[0056] Monocular Depth Estimation and Semantic Segmentation Module The loss function of the pre-built monocular depth estimation and semantic segmentation task head is as follows:

[0057] The dataset D contains the original image x i , and the corresponding depth estimation map And the annotated semantic segmentation map Loss Function Loss estimated from depth and semantic segmentation loss composition;

[0058]

[0059] For the monocular depth estimation task, the loss function used is the scale and translation invariant cropping loss:

[0060]

[0061] Among them, y is the true value, is the predicted value. is the loss of a single data point, selected as the L2 loss function, S is the set of selected data points, the set of data point indices after removing outliers, and |S represents the number of elements in the set S;

[0062] The construction process of set S is as follows:

[0063]

[0064] Where N is the total number of data points and α is the removal ratio;

[0065] The mathematical expression of the masked mean square error MSE loss function is as follows:

[0066]

[0067] in, is the total loss, is the predicted value, y is the true value, m is the mask matrix, M = ∑ i,j m i,j is the number of valid elements of each sample, and reduction is the reduction operation performed on the loss of each sample.

[0068] In another aspect of the present invention, in order to achieve the above-mentioned purpose, a terminal device is disclosed, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, the ViT-based monocular depth estimation and semantic segmentation method as described above is adopted.

[0069] Beneficial effects of the present invention:

[0070] The present invention uses the DPT model based on the ViT backbone network to realize the joint model of monocular depth estimation and image semantic segmentation, and integrates the feature representations extracted by different transformer layers. The fine-grained and more consistent global depth estimation brought by ViT can solve the problem of granularity and resolution lost in the process of deepening the network of CNNs, which is important for predicting intensive tasks. The introduction of a combined loss function with a balancing coefficient can accelerate the convergence speed when training monocular depth estimation and semantic segmentation tasks at the same time, and bring certain performance improvements to the two tasks. The experiment achieved the expected effect, which can effectively estimate the depth of objects in the image and effectively segment humans in complex scene images. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0072] Figure 1 It is a schematic flow chart of the method of the present invention;

[0073] Figure 2 This is an overview of the ViT-based monocular depth estimation and semantic segmentation joint model of the present invention;

[0074] Figure 3 It is the Reassemble structure diagram of the present invention;

[0075] Figure 4 It is the Fusion structure diagram of the present invention;

[0076] Figure 5 It is the original image, depth estimation annotation image and semantic segmentation annotation image in the data set of the present invention;

[0077] Figure 6 It is the model prediction diagram under scenario 1 of the present invention;

[0078] Figure 7 It is the model prediction diagram under scenario 2 of the present invention;

[0079] Figure 8 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0080] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0081] Embodiment 1:

[0082] like Figure 1 As shown, a monocular depth estimation and semantic segmentation method based on ViT is characterized in that the method comprises the following steps:

[0083] S101: receiving an image to be processed, inputting the image to be processed into a pre-established encoder based on the ViT backbone network, dividing the image to be processed into non-overlapping blocks, and mapping the blocks to a feature space to obtain word units;

[0084] Among them, the encoder based on the ViT backbone network, from a macro perspective, ViT operates on the word-like embedding features of the image, divides the image into non-overlapping blocks, and then maps these blocks into the feature space, playing the role of word units in natural language processing (NLP) tasks. The transformer network uses multiple superimposed multi-head attention mechanism blocks to process these word units, and the transformer network maintains the same number of word units throughout the calculation process. Because these word units correspond one-to-one to the blocks into which the image is divided, this means that the ViT encoder maintains the dimensions of the initial embedding in all transformer stages. In addition, the multi-head attention mechanism is essentially a global operation, so each word unit will affect the rest of the words. Therefore, the transformer network has a global receptive field at each stage after the initial embedding is generated. This is a significant advantage compared to convolutional networks. Convolutional neural networks gradually increase the receptive field of the network model after performing continuous convolution and downsampling layers on the feature map.

[0085] ViT processes the image into p 2 non-overlapping blocks, which are then flattened into tensors and mapped into embedding representations using linear layers.

[0086] The transformer does not preserve the spatial position of these individual tokens in the image, because the transformer is essentially a set-to-set function, so these tokens need to be concatenated with a learnable positional embedding to restore their positions in the original image when generating the final feature representation. Using the approach used in NLP work, ViT adds an additional special token to be used as the label [cls] in the classification task, and this special token is called the readout token. The process of generating the embedding is as follows: an H×W pixel image is processed into a series of tokens in represents the readout word, and D represents the feature dimension of each word.

[0087] The input word is passed through l transformer layers to form a feature representation t l , where l represents the output of the lth layer.

[0088] The backbone network used in this paper is the ViT-Base version. ViT-Base uses a block-based embedding process and 12 transformer layers. This paper uses p = 16, which means that the image is divided into 16×16 blocks, a total of 576 blocks. Each block is flattened into a 768-dimensional embedding representation through a linear projection, which is based on the design of the ViT-Base model.

[0089] S102: input the word unit to the Reassemble block of the encoder of the pre-established ViT backbone network for combination processing to obtain a feature representation of the class image, and process the feature representation of the class image through the Fusion block of the convolution decoder to obtain a feature representation of the processed image;

[0090] In the decoding stage, the words extracted from different transformer layers are combined through the Reassemble block, aiming to convert these features into a higher-dimensional space, which can then be used as the input of the Fusion block and can be input into the task head after processing.

[0091] The Reassemble block consists of three different sub-blocks. The overall structure of the model is as follows: Figure 2 The following is the expression that decomposes the Reassemble block.

[0092]

[0093] Among them, s represents the ratio of the features restored to the input image to be processed, Indicates the dimension of the output features.

[0094] First, N p+1 token is mapped into a group of N p tokens, which can be concatenated later to form an image-like feature representation. The operation is illustrated as follows:

[0095]

[0096] This operation is actually responsible for processing the readout token. The method for processing the readout token in this paper is to splice the information of the readout token onto all other tokens. The operation expression is as follows.

[0097]

[0098] After the Read operation, according to the position embedding, the obtained N p tokens can be recombined into an image-like feature representation. Applying the spatial splicing operation, a feature map with D channels and a size of is obtained. The operation is illustrated as follows:

[0099]

[0100] Finally, this feature is passed to the Resample layer, as Figure 3 shown. The feature representation is scaled to a feature representation with D channels and a size of First, a 1×1 convolution is used to obtain a feature representation with channels, and then a 3×3 convolution (when s≥p) or a 3×3 transposed convolution (when s<p) is used to implement the downsampling operation and the upsampling operation respectively. In this experiment, the features extracted at four different resolutions in four different stages of the transformer backbone network are recombined. The features extracted from deeper transformer layers are of lower resolution, and the features extracted from shallower transformer layers are of higher resolution. The transformer layers selected in this paper are l = {3, 6, 9, 12}. After the Read operation, a feature representation of is generated.

[0101]

[0102] The transformed feature map is used for the Fusion block of the corresponding layer. These Fusions are based on the structure of RefineNet, as Figure 4 shown. These features first have to go through the Reassemble block to obtain an image-like feature representation and the feature representation processed by the Fusion block of the previous layer. After splicing the two feature representations, two consecutive convolutional units are then applied and these feature representations are processed with a 2x upsampling operation at each Fusion stage.

[0103] S103: splicing the class image feature representation and the processed image feature representation to obtain a spliced ​​feature representation, inputting the spliced ​​feature representation into two consecutive convolution units of the convolution decoder for upsampling operation processing, and obtaining a final feature map through continuous operation processing of the Fusion block;

[0104] Finally, the feature map generated by the last fusion layer is used as the input of the task processing head module. Each task head consists of a small deconvolution block and an upsampling layer.

[0105] S104: Input the final feature map into the pre-built monocular depth estimation and semantic segmentation task head to calculate the total loss.

[0106] The dataset D contains the original image x i , and the corresponding depth estimation map And the annotated semantic segmentation map Loss Function Loss estimated from depth and semantic segmentation loss This paper introduces a balance coefficient λ to weigh the loss of monocular depth estimation and semantic segmentation to speed up training and improve the quality of results. Through experiments, it is found that λ of 0.6 can achieve a good balance. In the process of training, the purpose is to make Minimum.

[0107]

[0108] For the monocular depth estimation task, the loss function used is the scale-and-shift invariant trimmed loss in order to obtain a smooth depth estimation map. The semantic segmentation task uses the masked mean square error (MSE) loss function to minimize the distance between the true value and the predicted value.

[0109] The scale- and translation-invariant clipping loss function is a robust loss function that is often used to process data in the presence of outliers. Its basic idea is to reduce the impact of outliers on model training by removing some data points (usually extreme values), so that the loss function is insensitive to the scaling and displacement of the data. Its calculation formula is as follows:

[0110]

[0111] Among them, y is the true value, is the predicted value. is the loss of a single data point, which is selected as the L2 loss function. S is the set of selected data points, the set of data point indices after removing outliers. S| represents the number of elements in the set S.

[0112] The construction process of set S is as follows:

[0113]

[0114] Where N is the total number of data points and α is the removal ratio.

[0115] The mathematical expression of the masked mean square error (MSE) loss function is as follows:

[0116]

[0117] in, is the total loss, is the predicted value, y is the true value, m is the mask matrix, M = ∑ i,j m i,j is the number of valid elements of each sample, and reduction is the reduction operation performed on the loss of each sample.

[0118] Specifically, the present invention is further described below by way of embodiments:

[0119] 3.1 Dataset

[0120] This experiment is trained on three datasets, which are public datasets on Kaggle, namely inria_FOD, posetrack_FOD and nyuv2_FOD. The three datasets contain a total of 3969 images, and each dataset contains three folders. They are the original image images folder, the monocular depth estimation annotated image depths folder and the semantic segmentation annotated image segmentations folder. The original image is a 24-bit color image, the depth estimation annotated image is an 8-bit grayscale image, and the semantic segmentation is a 32-bit image, such as Figure 5 As shown in the figure, they are the original image and the monocular depth estimation annotation;

[0121] 3.2 Experimental Setup

[0122] The CPU model of the device used in this experiment is AMD Ryzen 55600, the graphics card is NVIDIA GeForce RTX40608G, the memory is 32G, and the platform is Windows11 (64bit). The deep learning framework used is Pytorch, and the python version is 3.10. The batch_size is set to 4, the epochs is set to 20, the training set accounts for 0.6, the validation set accounts for 0.2, and the test set accounts for 0.2. The image size is scaled to 384×384 during training.

[0123] Figure 6 and Figure 7 The following are the prediction results of the model in different scenarios. The prediction results correspond to the input image, the output depth estimation map (the higher the grayscale value of the image, the closer it is to the camera), and the semantic segmentation map (the pixels containing humans in the image are marked).

[0124] Embodiment 2: In the second aspect, as Figure 8 As shown, in order to achieve the above-mentioned purpose, the present invention discloses a monocular depth estimation and semantic segmentation system based on ViT, comprising:

[0125] The code extraction module 11 is used to receive the image to be processed, input the image to be processed into a pre-established encoder based on the ViT backbone network, divide the image to be processed into non-overlapping blocks, and map the blocks into the feature space to obtain word units;

[0126] A decoding processing module 12 is used to input the word unit into the Reassemble block of the encoder of the pre-established ViT backbone network for combination processing to obtain a feature representation of the class image, and process the feature representation of the class image through the Fusion block of the convolution decoder to obtain a feature representation of the processed image;

[0127] A splicing processing module 13 is used to splice the feature representation of the class image with the feature representation of the processed image to obtain a spliced ​​feature representation, input the spliced ​​feature representation into two consecutive convolution units of the convolution decoder for upsampling operation processing, and obtain a final feature map through continuous operation processing of the Fusion block;

[0128] The monocular depth estimation and semantic segmentation module 14 is used to input the final feature map into the pre-built monocular depth estimation and semantic segmentation task head to calculate the total loss.

[0129] In combination with the second aspect, in some implementations of the second aspect, the system further includes: the encoding extraction module 11 processes the image into p based on the encoder of the ViT backbone network 2non-overlapping squares, which are then flattened into tensors and mapped into embedding representations using linear layers;

[0130] The process of generating embedding in the encoder of the ViT backbone network in the encoding extraction module 11 is as follows: a H×W pixel image is processed into a series of word units in represents the readout word, and D represents the feature dimension of each word;

[0131] The input word is passed through l transformer layers to form a feature representation t l , where l represents the output from the lth layer;

[0132] The expression of the decoding processing module 12Reassemble block decomposition is as follows:

[0133]

[0134] Among them, s represents the ratio of the features restored to the input image to be processed, Indicates the dimension of the output features;

[0135] Decoding processing module 12 convolution decoder converts N p +1 word is mapped into a set of N p word units are concatenated into image-like feature representations. The operation is shown as follows:

[0136]

[0137] This operation is responsible for processing the readout token.

[0138] The method of processing the readout word unit in the decoding processing module 12 is to splice the information of the readout word unit to all other word units. The operation expression is as follows:

[0139]

[0140] After the Read operation, the obtained N is embedded according to the position p The word units are reassembled into image-like feature representations, and spatial concatenation is applied to obtain a D-channel image with a size of The feature map of the operation is as follows:

[0141]

[0142] Finally, the features are passed to the Resample layer, which scales the feature representation to a size of D channels. The feature representation of

[0143] The loss function of the monocular depth estimation and semantic segmentation task head pre-built by the monocular depth estimation and semantic segmentation module 14 is as follows:

[0144] The dataset D contains the original image x i , and the corresponding depth estimation map And the annotated semantic segmentation map Loss Function Loss estimated from depth and semantic segmentation loss composition;

[0145]

[0146] For the monocular depth estimation task, the loss function used is the scale and translation invariant cropping loss:

[0147]

[0148] Among them, y is the true value, is the predicted value. is the loss of a single data point, selected as the L2 loss function, S is the set of selected data points, the set of data point indices after removing outliers, and S| represents the number of elements in the set S;

[0149] The construction process of set S is as follows:

[0150]

[0151] Where N is the total number of data points and α is the removal ratio;

[0152] The mathematical expression of the masked mean square error MSE loss function is as follows:

[0153]

[0154] in, is the total loss, is the predicted value, y is the true value, m is the mask matrix, M = ∑ i,j m i,j is the number of valid elements of each sample, and reduction is the reduction operation performed on the loss of each sample.

[0155] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0156] It needs to be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0157] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0158] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

Claims

1. A monocular depth estimation and semantic segmentation method based on ViT, characterized in that: The method comprises the following steps: Receive an image to be processed, input the image to be processed into a pre-established encoder based on the ViT backbone network, divide the image to be processed into non-overlapping blocks, and map the blocks into a feature space to obtain word units; The word unit is input into the Reassemble block of the encoder of the pre-established ViT backbone network for combination processing to obtain the feature representation of the class image, and the feature representation of the class image is processed by the Fusion block of the convolution decoder to obtain the feature representation of the processed image; The feature representation of the class image is concatenated with the feature representation of the processed image to obtain a concatenated feature representation, which is input into two consecutive convolution units of the convolution decoder for upsampling operation, and the final feature map is obtained through continuous operation of the Fusion block; The final feature map is input into the pre-built monocular depth estimation and semantic segmentation task head to calculate the total loss.

2. The ViT-based monocular depth estimation and semantic segmentation method according to claim 1, characterized in that: The encoder based on the ViT backbone network processes the image into p 2 non-overlapping blocks, which are then flattened into tensors and mapped into embedding representations using linear layers.

3. The ViT-based monocular depth estimation and semantic segmentation method according to claim 2, characterized in that: The process of generating embeddings in the encoder based on the ViT backbone network is as follows: a H×W pixel image is processed into a series of word units in represents the readout word, and D represents the feature dimension of each word; The input word is passed through l transformer layers to form a feature representation t l , where l represents the output of the lth layer.

4. The ViT-based monocular depth estimation and semantic segmentation method according to claim 1, characterized in that: The expression decomposed by the Reassemble block is as follows: Among them, s represents the ratio of the features restored to the input image to be processed, Indicates the dimension of the output features.

5. The ViT-based monocular depth estimation and semantic segmentation method according to claim 1, characterized in that: The convolutional decoder converts N p +1 word is mapped into a set of N p word units are concatenated into image-like feature representations. The operation is shown as follows: This operation is responsible for processing the readout token.

6. The ViT-based monocular depth estimation and semantic segmentation method according to claim 5, characterized in that: The method for processing the readout word is to splice the information of the readout word to all other words. The operation expression is as follows: After the Read operation, the obtained N is embedded according to the position p The word units are reassembled into image-like feature representations, and spatial concatenation is applied to obtain a D-channel image with a size of The feature map of the operation is as follows: Finally, the features are passed to the Resample layer, which scales the feature representation to a size of D channels. The feature representation of .

7. The ViT-based monocular depth estimation and semantic segmentation method according to claim 1, characterized in that: The loss function of the pre-built monocular depth estimation and semantic segmentation task head is as follows: The dataset D contains the original image x i , and the corresponding depth estimation map And the annotated semantic segmentation map Loss Function Loss estimated from depth and semantic segmentation loss composition; For the monocular depth estimation task, the loss function used is the scale and translation invariant cropping loss: Among them, y is the true value, is the predicted value, is the loss of a single data point, selected as the L2 loss function, S is the set of selected data points, the set of data point indices after removing outliers, and S| represents the number of elements in the set S; The construction process of set S is as follows: Where N is the total number of data points and α is the removal ratio; The mathematical expression of the masked mean square error MSE loss function is as follows: in, is the total loss, is the predicted value, y is the true value, m is the mask matrix, M = ∑ i,j m i,j is the number of valid elements of each sample, and reduction is the reduction operation performed on the loss of each sample.

8. A monocular depth estimation and semantic segmentation system based on ViT, characterized in that: include: A coding extraction module is used to receive the image to be processed, input the image to be processed into a pre-established encoder based on the ViT backbone network, divide the image to be processed into non-overlapping blocks, and map the blocks into a feature space to obtain word units; A decoding processing module is used to input the word unit into the Reassemble block of the encoder of the pre-established ViT backbone network for combination processing to obtain the feature representation of the class image, and process the feature representation of the class image through the Fusion block of the convolution decoder to obtain the feature representation of the processed image; A splicing processing module is used to splice the feature representation of the class image with the feature representation of the processed image to obtain a spliced ​​feature representation, and input the spliced ​​feature representation into two consecutive convolution units of the convolution decoder for upsampling operation processing, and obtain the final feature map through continuous operation processing of the Fusion block; The monocular depth estimation and semantic segmentation module is used to input the final feature map into the pre-built monocular depth estimation and semantic segmentation task head to calculate the total loss.

9. The ViT-based monocular depth estimation and semantic segmentation system according to claim 8, characterized in that: The encoding extraction module processes the image into p based on the encoder of the ViT backbone network 2 non-overlapping squares, which are then flattened into tensors and mapped into embedding representations using linear layers; The process of generating embeddings in the encoder of the ViT backbone network is as follows: an H×W pixel image is processed into a series of word units in represents the readout word, and D represents the feature dimension of each word; The input word is passed through l transformer layers to form a feature representation t l , where l represents the output from the lth layer; The expression of the decoding processing module Reassemble block decomposition is as follows: Among them, s represents the ratio of the features restored to the input image to be processed, Indicates the dimension of the output features; The decoding processing module convolution decoder converts N p +1 word is mapped into a set of N p word units are concatenated into image-like feature representations. The operation is shown as follows: This operation is responsible for processing the readout token; The method of processing the readout word in the decoding processing module is to splice the information of the readout word to all other words. The operation expression is as follows: After the Read operation, the obtained N is embedded according to the position p The word units are reassembled into image-like feature representations, and spatial concatenation is applied to obtain a D-channel image with a size of The feature map of the operation is as follows: Finally, the features are passed to the Resample layer, which scales the feature representation to a size of D channels. The feature representation of Monocular Depth Estimation and Semantic Segmentation Module The loss function of the pre-built monocular depth estimation and semantic segmentation task head is as follows: The dataset D contains the original image x i , and the corresponding depth estimation map And the annotated semantic segmentation map Loss Function Loss estimated from depth and semantic segmentation loss composition; For the monocular depth estimation task, the loss function used is the scale and translation invariant cropping loss: Among them, y is the true value, is the predicted value. is the loss of a single data point, selected as the L2 loss function, S is the set of selected data points, the set of data point indices after removing outliers, and |S represents the number of elements in the set S; The construction process of set S is as follows: Where N is the total number of data points and α is the removal ratio; The mathematical expression of the masked mean square error MSE loss function is as follows: in, is the total loss, is the predicted value, y is the true value, m is the mask matrix, M = ∑ i,j m i,j is the number of valid elements of each sample, and reduction is the reduction operation performed on the loss of each sample.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: The memory stores a computer program that can be run on a processor. When the processor loads and executes the computer program, the ViT-based monocular depth estimation and semantic segmentation method according to any one of claims 1 to 7 is adopted.

Images