Monocular depth estimation based depth map compression method and device, equipment and medium

Abstract

The application is suitable for the technical field of image processing, and provides a depth map compression method based on monocular depth estimation, which comprises the following steps: according to a received color image and an original depth map corresponding to the color image, the original depth map is compressed by a depth map compression model with a preset symmetrical structure and composed of a monocular depth estimation module, a depth compression module and a mask generation and processing module, so as to obtain a corresponding compressed depth map, thereby reducing the data distortion degree under the condition of low code rate and improving the compression performance and compression effect of the depth map.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and particularly relates to a method, apparatus, device and medium for depth map compression based on monocular depth estimation. Background Technology

[0002] In recent years, with the continuous development of the sensor field, a large number of novel 3D cameras are being invented and applied. Depth information is spatial information acquired by 3D cameras, which reflects the distance relationship between objects in the current scene and the camera, and is usually stored and applied in the form of depth images. The data distribution of depth maps differs from that of common natural images. On the one hand, the content of depth maps mainly includes edge information of objects and information on smooth object surface areas; on the other hand, the bit depth of depth maps is generally higher than that of traditional natural images. The introduction of depth information has broadened the dimension of the two-dimensional information acquired by traditional cameras and has been widely used in fields such as autonomous driving and augmented reality. In practical applications, massive amounts of depth information often need to be transmitted from collection to application. Due to limitations such as bandwidth, this depth information needs to be compressed, but data compression either has the problem of limited compression ratio or data distortion.

[0003] Existing depth map compression methods primarily involve making a series of adjustments to the depth map for application in technically mature image encoders. Due to the correlation between color images and depth maps of the same scene, methods using color image information to guide depth map compression have also attracted researchers' attention. For example, Forster et al. proposed introducing color image auxiliary information during the depth map decoding process to improve the reconstruction effect; Uruma et al. divided the color image into a series of superpixels and set masks based on the superpixels, transforming the depth map into a sparse matrix for compression, and then reusing this information to reconstruct the depth map at the decoding end. In recent years, based on the powerful feature extraction capabilities of deep learning models such as convolutional neural networks, researchers have begun to focus on using deep learning models to extract depth information from color images. For example, Eigen et al. first performed global depth feature prediction from coarse to fine, and then optimized the depth information of local regions; Alhashim et al. used transfer learning to design a simple encoder-decoder network to predict depth maps, using a pre-trained model as a depth feature extractor; Bhat et al. proposed applying the transformer structure to monocular depth estimation, dividing the predicted depth range into several containers with widths that vary according to image features, and linearly combining the center values ​​of each container to obtain the final predicted depth value.

[0004] Existing depth map compression methods primarily compress the complete depth map. Although these methods employ various measures to improve compression performance, global depth map compression still suffers from severe distortion at low bitrates. Furthermore, traditional image encoders mainly aim to achieve compression by eliminating information insensitive to human vision, while depth maps are primarily used in machine vision tasks rather than human vision tasks. Therefore, directly using traditional image encoders for depth map compression may destroy some non-visual information, such as spatial structure information, thus affecting the performance of downstream tasks. On the other hand, the results of depth prediction directly using deep learning models on color images still exhibit distortion compared to the true depth map. Specifically, this manifests as discrepancies in depth values ​​and the inability to accurately predict the depth values ​​of some objects. Therefore, depth prediction results are currently difficult to replace true depth information in many application scenarios. Summary of the Invention

[0005] The purpose of this invention is to provide a depth map compression method, apparatus, device, and medium based on monocular depth estimation, aiming to solve the problems of severe data distortion, low compression performance, and poor compression effect after depth map compression due to the inability of existing technologies to provide an effective depth map compression method.

[0006] On one hand, the present invention provides a depth map compression method based on monocular depth estimation, the method comprising the following steps:

[0007] Receive an input color image and a raw depth map corresponding to the color image;

[0008] Based on the received color image and the original depth map, the original depth map is compressed using a preset depth map compression model to obtain a corresponding compressed depth map. The depth map compression model has a symmetrical structure and consists of a monocular depth estimation module, a depth compression module, and a mask generation and processing module.

[0009] Preferably,

[0010] The step of compressing the original depth map using a preset depth map compression model includes:

[0011] The color image is processed by the monocular depth estimation module to obtain the corresponding predicted depth map;

[0012] Calculate the residual information between the predicted depth map and the original depth map;

[0013] The residual information is processed by the depth compression module to obtain the corresponding residual depth map;

[0014] The original depth map is processed by the mask generation and processing module to obtain the corresponding image mask;

[0015] According to the preset reconstruction formula, the predicted depth map, the residual depth map, and the image mask are combined to obtain the compressed depth map.

[0016] Preferably, the step of processing the color image through the monocular depth estimation module includes:

[0017] The depth features of the color image are extracted by the encoder in the monocular depth estimation module;

[0018] Based on the extracted depth features, the predicted depth map is generated by the decoder in the monocular depth estimation module.

[0019] Preferably, the step of processing the residual information through the deep compression module includes:

[0020] The residual information is transformed by the encoder in the deep compression module to obtain the corresponding latent representation;

[0021] The residual depth map is obtained by reconstructing the latent representation using the decoder in the depth compression module.

[0022] Preferably, after the step of transforming the residual information by the encoder in the deep compression module and before the step of reconstructing the latent representation by the decoder in the deep compression module, the method further includes:

[0023] The correlations in the latent representation are estimated and eliminated by the super-prior module in the deep compression module.

[0024] On the other hand, the present invention provides a depth map compression device based on monocular depth estimation, the device comprising:

[0025] An image input unit is configured to receive an input color image and a corresponding original depth map; and

[0026] The image compression unit is used to compress the original depth map according to the received color image and the original depth map using a preset depth map compression model to obtain a corresponding compressed depth map. The depth map compression model is a symmetrical structure and consists of a monocular depth estimation module, a depth compression module, and a mask generation and processing module.

[0027] Preferably, the image compression unit includes:

[0028] The first processing unit is used to process the color image through the monocular depth estimation module to obtain the corresponding predicted depth map;

[0029] An information calculation unit is used to calculate the residual information between the predicted depth map and the original depth map;

[0030] The second processing unit is used to process the residual information through the depth compression module to obtain the corresponding residual depth map;

[0031] The third processing unit is used to process the original depth map through the mask generation and processing module to obtain the corresponding image mask; and

[0032] An image combination unit is used to combine the predicted depth map, the residual depth map, and the image mask according to a preset reconstruction formula to obtain the compressed depth map.

[0033] Preferably, the first processing unit includes:

[0034] A depth feature extraction unit is used to extract depth features of the color image through the encoder in the monocular depth estimation module; and

[0035] The depth map generation unit is used to generate the predicted depth map based on the extracted depth features through the decoder in the monocular depth estimation module.

[0036] On the other hand, the present invention also provides an image processing apparatus, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps described in the above-described depth map compression method based on monocular depth estimation.

[0037] On the other hand, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the above-described method for depth map compression based on monocular depth estimation.

[0038] This invention compresses the original depth map based on the received color image and its corresponding original depth map using a depth map compression model with a preset symmetrical structure, consisting of a monocular depth estimation module, a depth compression module, and a mask generation and processing module, to obtain the corresponding compressed depth map. This reduces data distortion at low bitrates and improves the compression performance and effect of the depth map. Attached Figure Description

[0039] Figure 1This is a flowchart illustrating the implementation of the depth map compression method based on monocular depth estimation provided in Embodiment 1 of the present invention.

[0040] Figure 2 This is a schematic diagram of the depth map compression model in the depth map compression method based on monocular depth estimation provided in Embodiment 1 of the present invention;

[0041] Figure 3 This is a flowchart illustrating the implementation of the depth map compression method based on monocular depth estimation provided in Embodiment 2 of the present invention.

[0042] Figure 4 This is a schematic diagram of the depth map compression device based on monocular depth estimation provided in Embodiment 3 of the present invention;

[0043] Figure 5 This is a preferred structural schematic diagram of the depth map compression device based on monocular depth estimation provided in Embodiment 3 of the present invention; and

[0044] Figure 6 This is a schematic diagram of the image processing device provided in Embodiment 4 of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0046] The specific implementation of the present invention will be described in detail below with reference to specific embodiments:

[0047] Example 1:

[0048] Figure 1 The implementation flow of the depth map compression method based on monocular depth estimation provided in Embodiment 1 of the present invention is shown. For ease of explanation, only the parts related to the embodiments of the present invention are shown, and are described in detail below:

[0049] In step S101, the input color image and the original depth map corresponding to the color image are received.

[0050] In step S102, based on the received color image and the original depth map, the original depth map is compressed using a preset depth map compression model to obtain the corresponding compressed depth map. The depth map compression model has a symmetrical structure and consists of a monocular depth estimation module, a depth compression module, and a mask generation and processing module.

[0051] This invention is applicable to image processing platforms, devices, or systems, such as personal computers and servers. In this invention, the depth map compression model is a lossy compression model comprising two parts: an encoding end and a decoding end with a symmetrical structure. It consists of a monocular depth estimation module, a depth compression module, and a mask generation and processing module. Specifically, as shown... Figure 2 As shown, the encoding end of the depth map compression model includes an encoder / decoder for the monocular depth estimation module, an encoder for the depth compression module, and a mask generator and encoder for the mask generation and processing module. The decoding end of the depth map compression model includes an encoder / decoder for the monocular depth estimation module, a decoder for the depth compression module, and a decoder for the mask generation and processing module. When a user inputs a color image obtained instantly, stored locally, or stored in the cloud, along with the corresponding original depth map, into the trained depth map compression model, the model automatically processes and compresses the color image and the original depth map accordingly, finally outputting a processed compressed depth map. The original depth map corresponding to the color image can be obtained through a 3D camera.

[0052] In this embodiment of the invention, based on the received color image and its corresponding original depth map, the original depth map is compressed using a depth map compression model with a preset symmetrical structure, consisting of a monocular depth estimation module, a depth compression module, and a mask generation and processing module, to obtain the corresponding compressed depth map. This reduces data distortion at low bitrates and improves the compression performance and effect of the depth map.

[0053] Example 2:

[0054] Figure 2 The implementation flow of the depth map compression method based on monocular depth estimation provided in Embodiment 2 of the present invention is shown. For ease of explanation, only the parts related to the embodiments of the present invention are shown, and are described in detail below:

[0055] In step S301, the color image is processed by the monocular depth estimation module to obtain the corresponding predicted depth map.

[0056] In this embodiment of the invention, the monocular depth estimation module adopts an encoder-decoder network based on transfer learning. The monocular depth estimation module can automatically extract depth features from the input color image and use these depth features to generate a predicted depth map of the corresponding scene. The predicted depth map contains the edges of most objects and the approximate depth information of their surfaces.

[0057] When processing a color image using a monocular depth estimation module, the processing of the color image is preferably achieved through the following steps:

[0058] (1) Extract the depth features of the color image through the encoder in the monocular depth estimation module.

[0059] In this embodiment of the invention, the encoder in the monocular depth estimation module uses a modified DenseNet-169 network as a feature extractor. When the user inputs a color image into the encoder of this module, the corresponding depth features can be obtained. The modified DenseNet-169 network is obtained by removing the top-level network used for image classification tasks in the original DenseNet-169 network.

[0060] (2) Based on the extracted depth features, a predicted depth map is generated by the decoder in the monocular depth estimation module.

[0061] In this embodiment of the invention, the decoder in the monocular depth estimation module performs a series of convolution and upsampling operations on the depth features, and concatenates the upsampled, interpolated feature map with the corresponding size feature map in the encoder of the monocular depth estimation module to transmit the high-resolution feature map information in the encoder to the decoder, and finally generates a predicted depth map. This enriches the feature map information of the decoder by constructing a skip connection structure, thereby improving the depth prediction effect.

[0062] The color image is processed through the above steps (1) and (2), thereby improving the similarity between the predicted depth map and the original depth map and enhancing the depth prediction effect.

[0063] More preferably, the monocular depth estimation module employs a loss function. Training is performed, where D is the original depth map. To predict the depth map, Indicate D and The degree of similarity between the two is determined by β, a parameter used to adjust the weights of different metrics in the loss function. A larger β indicates that L is more closely related to the current training objective. D The importance of indicators is greater, L G and L SSIM The two metrics are less important; conversely, when β is smaller, it indicates that the current training task emphasizes L more. G and L SSIM In this invention, the index β is set to 0.1, and L... D The Mean Absolute Error (MAE) represents the difference between the original depth value and the predicted depth value, and is used to measure the distortion of the depth value. G This is used to measure the loss of high-frequency information such as object edges, using the formula... The calculation yields g, where g xG represents the difference in gradient along the x-component between the original depth map and the predicted depth map. y This represents the difference in gradient along the y-component between the original depth map and the predicted depth map, where n represents the number of pixels in D, p is the pixel index, and D... p This represents the p-th pixel in D. express The p-th pixel in L SSIM It is a transformation of the Structural Similarity Index (SSIM), using the formula... The calculations show that the predicted depth map achieves high fidelity in both depth value and high-frequency detail. This means that while minimizing the difference between the predicted depth value and the original depth value, it suppresses the distortion of high-frequency details in the predicted depth map, thereby improving the similarity between the predicted depth map and the original depth map.

[0064] In step S302, the residual information between the predicted depth map and the original depth map is calculated.

[0065] In this embodiment of the invention, through the formula Calculate the predicted depth map The residual information R between the original depth map D and the original depth map is used as error compensation information between the predicted depth map and the original depth map. This residual information is used instead of the complete depth map as the compression and transmission object, thereby avoiding the destruction of structural information caused by compressing the complete depth map. This ensures that most of the depth information in the subsequently reconstructed depth map comes from the depth estimation of the color map.

[0066] In step S303, the residual information is processed by the depth compression module to obtain the corresponding residual depth map.

[0067] In this embodiment of the invention, the deep compression module follows the principle of transform coding and includes a codec body and an auxiliary super-prior module. When processing residual information through the deep compression module, preferably, the processing of the residual information is achieved through the following steps:

[0068] (1) The input residual information is transformed by the encoder in the deep compression module to obtain the corresponding latent representation.

[0069] In this embodiment of the invention, the encoder of the deep compression module consists of a series of linear-nonlinear transformation modules. Each module uses convolutional layers and generalized division normalization (GDN) layers to perform linear and nonlinear transformations on the input residual information to eliminate the correlation between data. After the encoder's transformation processing, the residual information with strong information correlation is converted into a potential representation with weaker correlation and more compactness.

[0070] More preferably, the correlations in the latent representation are estimated and eliminated by the super-prior module in the deep compression module, thereby reducing the information entropy of the latent representation and achieving a better compression effect.

[0071] (2) The latent representation is reconstructed by the decoder in the depth compression module to obtain the residual depth map.

[0072] In this embodiment of the invention, the decoder and encoder in the depth compression module form a symmetrical structure. The decoder in the depth compression module is composed of an inverse transform module corresponding to the encoder in the depth compression module. It mainly includes a deconvolution layer and an inverse transform layer of GDN (i.e., inverse generalized division normalization layer (IGDN)). After the latent representation obtained at the encoding end of the depth map compression model is transmitted to the decoding end, it is restored by the decoder in the depth compression module, and finally the reconstructed depth residual information is obtained. This depth residual information is the corresponding residual depth map.

[0073] By performing the above steps (1) and (2), the residual information is processed, and compression optimization is performed to a great extent according to the characteristics of the residual information, so as to ensure that a good compression effect can still be achieved under low bit rate conditions.

[0074] More preferably, the rate-distortion function L = B + λL is used in the deep compression module. recon Mean Squared Error (MSE) is used as the loss function for training. As a distortion metric, where L is the overall loss function in the compression process, and B represents the bitrate required for the compressed potential representation, L recon Residual depth map representing reconstruction The distortion between the residual image R and the original input residual information, where λ is a parameter used to adjust... The parameter λ, along with the weights R, indicates that the training objective of the current model is to obtain better image reconstruction results. This means that the required bitrate B will increase. Conversely, the smaller the λ value, the training objective of the current model is to minimize the bitrate B required for compression encoding, which will result in a decrease in reconstruction quality. By setting different λ values, models with different compression effects can be trained, thereby achieving a balance between bitrate and reconstructed image quality, and realizing the goal of obtaining the highest possible quality reconstructed image at the lowest possible bitrate.

[0075] In step S304, the original depth map is processed by the mask generation and processing module to obtain the corresponding image mask.

[0076] In this embodiment of the invention, the original depth map may contain invalid depth regions. In this case, the original depth map is processed by the mask generation and processing module to obtain an image mask that marks the valid information in the original depth map. At the encoding end of the depth map compression model, the encoder in the mask generation and processing module uses a binary compression encoder (Joint Bi-level Image experts Group, abbreviated as JBIG) to perform lossless compression on the obtained image mask. Subsequently, the compressed image mask is transmitted to the decoding end of the depth map compression model for decompression.

[0077] In step S305, the predicted depth map, residual depth map, and image mask are combined according to the preset reconstruction formula to obtain a compressed depth map.

[0078] In this embodiment of the invention, at the decoding end of the depth map compression model, firstly, the compressed residual depth map transmitted from the encoding end of the depth map compression model is reconstructed by the decoder of the depth compression module to obtain the decompressed residual depth map. Subsequently, according to the reconstruction formula The residual depth map The predicted depth map is compared with the prediction of the monocular depth estimation model deployed at the decoding end. The images are then combined, and finally, combined with the image mask M decompressed by the decoder in the mask generation and processing module to obtain the reconstructed compressed depth map.

[0079] In this embodiment of the invention, a monocular depth estimation module processes a color image to obtain a corresponding predicted depth map. The residual information between the predicted depth map and the original depth map is calculated. The residual information is then processed by a depth compression module to obtain a corresponding residual depth map. The original depth map is then processed by a mask generation and processing module to obtain a corresponding image mask. Based on a preset reconstruction formula, the predicted depth map, the residual depth map, and the image mask are combined to obtain a compressed depth map. This achieves high-quality compressed and reconstructed images even at low bitrates, improving the performance of depth map compression.

[0080] Example 3:

[0081] Figure 4 The structure of the depth map compression device based on monocular depth estimation provided in Embodiment 3 of the present invention is shown. For ease of explanation, only the parts related to the embodiments of the present invention are shown, including:

[0082] Image input unit 41 is used to receive an input color image and a raw depth map corresponding to the color image.

[0083] The image compression unit 42 is used to compress the original depth map according to the received color image and the original depth map through a preset depth map compression model to obtain the corresponding compressed depth map. The depth map compression model has a symmetrical structure and consists of a monocular depth estimation module, a depth compression module, and a mask generation and processing module.

[0084] This invention is applicable to image processing platforms, devices, or systems, such as personal computers and servers. In this invention, the depth map compression model is a lossy compression model comprising two parts: an encoding end and a decoding end with a symmetrical structure. It consists of a monocular depth estimation module, a depth compression module, and a mask generation and processing module. When a user inputs a color image obtained instantly, stored locally, or stored in the cloud, along with the corresponding original depth map, into the trained depth map compression model, the model automatically processes and compresses the color image and the original depth map, finally outputting a processed compressed depth map. The original depth map corresponding to the color image can be obtained through a 3D camera.

[0085] like Figure 5 As shown, preferably, the image compression unit 42 includes:

[0086] The first processing unit 421 is used to process the color image through the monocular depth estimation module to obtain the corresponding predicted depth map.

[0087] In this embodiment of the invention, the monocular depth estimation module adopts an encoder-decoder network based on transfer learning. The monocular depth estimation module can automatically extract depth features from the input color image and use these depth features to generate a predicted depth map of the corresponding scene. The predicted depth map contains the edges of most objects and the approximate depth information of their surfaces.

[0088] Preferably, the monocular depth estimation module employs a loss function. Training is performed, where D is the original depth map. To predict the depth map, Indicate D and The degree of similarity between the two is determined by β, a parameter used to adjust the weights of different metrics in the loss function. A larger β indicates that L is more closely related to the current training objective. D The importance of indicators is greater, L G and L SSIM The two metrics are less important; conversely, when β is smaller, it indicates that the current training task emphasizes L more. G and L SSIM In this invention, the index β is set to 0.1, and L... D The Mean Absolute Error (MAE) represents the difference between the original depth value and the predicted depth value, and is used to measure the distortion of the depth value. G This is used to measure the loss of high-frequency information such as object edges, using the formula... The calculation yields g, where g x G represents the difference in gradient along the x-component between the original depth map and the predicted depth map. y This represents the difference in gradient along the y-component between the original depth map and the predicted depth map, where n represents the number of pixels in D, p is the pixel index, and D... p This represents the p-th pixel in D. express The p-th pixel in L SSIM It is a transformation of the Structural Similarity Index (SSIM), using the formula... The calculations show that the predicted depth map achieves high fidelity in both depth value and high-frequency detail. This means that while minimizing the difference between the predicted depth value and the original depth value, it suppresses the distortion of high-frequency details in the predicted depth map, thereby improving the similarity between the predicted depth map and the original depth map.

[0089] The information calculation unit 422 is used to calculate the residual information between the predicted depth map and the original depth map.

[0090] In this embodiment of the invention, through the formula Calculate the predicted depth map The residual information R between the original depth map D and the original depth map is used as error compensation information between the predicted depth map and the original depth map. This residual information is used instead of the complete depth map as the compression and transmission object, thereby avoiding the destruction of structural information caused by compressing the complete depth map. This ensures that most of the depth information in the subsequently reconstructed depth map comes from the depth estimation of the color map.

[0091] The second processing unit 423 is used to process the residual information through the depth compression module to obtain the corresponding residual depth map.

[0092] In a preferred embodiment of the invention, the rate-distortion function L = B + λL is used in the deep compression module. recon Mean Squared Error (MSE) is used as the loss function for training. As a distortion metric, where L is the overall loss function in the compression process, and B represents the bitrate required for the compressed potential representation, L recon Residual depth map representing reconstruction The distortion between the residual image R and the original input residual information, where λ is a parameter used to adjust... The parameter λ, along with the weights R, indicates that the training objective of the current model is to obtain better image reconstruction results. This means that the required bitrate B will increase. Conversely, the smaller the λ value, the training objective of the current model is to minimize the bitrate B required for compression encoding, which will result in a decrease in reconstruction quality. By setting different λ values, models with different compression effects can be trained, thereby achieving a balance between bitrate and reconstructed image quality, and realizing the goal of obtaining the highest possible quality reconstructed image at the lowest possible bitrate.

[0093] The third processing unit 424 is used to process the original depth map through the mask generation and processing module to obtain the corresponding image mask.

[0094] In this embodiment of the invention, the original depth map may contain invalid depth regions. In this case, the original depth map is processed by the mask generation and processing module to obtain an image mask that marks the valid information in the original depth map. At the encoding end of the depth map compression model, the encoder in the mask generation and processing module uses a binary compression encoder (Joint Bi-level Image experts Group, abbreviated as JBIG) to perform lossless compression on the obtained image mask. Subsequently, the compressed image mask is transmitted to the decoding end of the depth map compression model for decompression.

[0095] The image combination unit 425 is used to combine the predicted depth map, the residual depth map and the image mask according to a preset reconstruction formula to obtain a compressed depth map.

[0096] In this embodiment of the invention, at the decoding end of the depth map compression model, firstly, the compressed residual depth map transmitted from the encoding end of the depth map compression model is reconstructed by the decoder of the depth compression module to obtain the decompressed residual depth map. Subsequently, according to the reconstruction formula The residual depth map The predicted depth map is compared with the prediction of the monocular depth estimation model deployed at the decoding end. The images are then combined, and finally, combined with the image mask M decompressed by the decoder in the mask generation and processing module to obtain the reconstructed compressed depth map.

[0097] More preferably, the first processing unit 421 includes:

[0098] The depth feature extraction unit 4211 is used to extract depth features of a color image through the encoder in the monocular depth estimation module.

[0099] In this embodiment of the invention, the encoder in the monocular depth estimation module uses a modified DenseNet-169 network as a feature extractor. When the user inputs a color image into the encoder of this module, the corresponding depth features can be obtained. The modified DenseNet-169 network is obtained by removing the top-level network used for image classification tasks in the original DenseNet-169 network.

[0100] The depth map generation unit 4212 is used to generate a predicted depth map based on the extracted depth features through the decoder in the monocular depth estimation module.

[0101] In this embodiment of the invention, the decoder in the monocular depth estimation module performs a series of convolution and upsampling operations on the depth features, and concatenates the upsampled, interpolated feature map with the corresponding size feature map in the encoder of the monocular depth estimation module to transmit the high-resolution feature map information in the encoder to the decoder, and finally generates a predicted depth map. This enriches the feature map information of the decoder by constructing a skip connection structure, thereby improving the depth prediction effect.

[0102] More preferably, the second processing unit 423 includes:

[0103] The information transformation unit 4231 is used to transform the residual information through the encoder in the deep compression module to obtain the corresponding latent representation.

[0104] In this embodiment of the invention, the encoder of the deep compression module consists of a series of linear-nonlinear transformation modules. Each module uses convolutional layers and generalized division normalization (GDN) layers to perform linear and nonlinear transformations on the input residual information to eliminate the correlation between data. After the encoder's transformation processing, the residual information with strong information correlation is converted into a potential representation with weaker correlation and more compactness.

[0105] The information reconstruction unit 4232 is used to reconstruct the latent representation through the decoder in the depth compression module to obtain the residual depth map.

[0106] In this embodiment of the invention, the decoder and encoder in the depth compression module form a symmetrical structure. The decoder in the depth compression module is composed of an inverse transform module corresponding to the encoder in the depth compression module. It mainly includes a deconvolution layer and an inverse transform layer of GDN (i.e., inverse generalized division normalization layer (IGDN)). After the latent representation obtained at the encoding end of the depth map compression model is transmitted to the decoding end, it is restored by the decoder in the depth compression module, and finally the reconstructed depth residual information is obtained. This depth residual information is the corresponding residual depth map.

[0107] More preferably, the depth map compression device of this embodiment further includes:

[0108] Information elimination unit 4233 is used to estimate and eliminate the correlation in the latent representation through the super-prior module in the deep compression module.

[0109] In this embodiment of the invention, each unit of the depth map compression device based on monocular depth estimation can be implemented by a corresponding hardware or software unit. Each unit can be an independent hardware or software unit, or it can be integrated into a hardware or software unit. This is not intended to limit the invention.

[0110] Example 4:

[0111] Figure 6 The structure of the image processing device provided in Embodiment 4 of the present invention is shown. For ease of explanation, only the parts related to the embodiments of the present invention are shown.

[0112] The image processing device 6 of this embodiment includes a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, it implements the steps described in the above-described embodiment of the depth map compression method based on monocular depth estimation, for example... Figure 1The steps S101 to S102 are shown. Alternatively, when the processor 60 executes the computer program 62, it implements the functions of each unit in the above-described device embodiments, for example... Figure 4 The functions of units 41 to 42 shown.

[0113] In this embodiment of the invention, based on the received color image and its corresponding original depth map, the original depth map is compressed using a depth map compression model with a preset symmetrical structure, consisting of a monocular depth estimation module, a depth compression module, and a mask generation and processing module, to obtain the corresponding compressed depth map. This reduces data distortion at low bitrates and improves the compression performance and effect of the depth map.

[0114] The image processing device in this embodiment of the invention can be a personal computer or a server. The steps implemented by the processor 60 in the image processing device 6 when executing the computer program 62 to implement the depth map compression method based on monocular depth estimation can be referred to the description of the foregoing method embodiments, and will not be repeated here.

[0115] Example 5:

[0116] In this embodiment of the invention, a computer-readable storage medium is provided, which stores a computer program. When executed by a processor, the computer program implements the steps in the above-described embodiments of the depth map compression method based on monocular depth estimation. For example... Figure 1 The steps S101 to S102 are shown. Alternatively, when the computer program is executed by the processor, it implements the functions of each unit in the above-described device embodiments, for example... Figure 4 The functions of units 41 to 42 shown.

[0117] In this embodiment of the invention, based on the received color image and its corresponding original depth map, the original depth map is compressed using a depth map compression model with a preset symmetrical structure, consisting of a monocular depth estimation module, a depth compression module, and a mask generation and processing module, to obtain the corresponding compressed depth map. This reduces data distortion at low bitrates and improves the compression performance and effect of the depth map.

[0118] The computer-readable storage medium in embodiments of the present invention may include any entity or device capable of carrying computer program code, a recording medium, such as ROM / RAM, disk, optical disk, flash memory, etc.

[0119] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A depth map compression method based on monocular depth estimation, characterized in that, The method includes the following steps: Receive an input color image and a raw depth map corresponding to the color image; Based on the received color image and the original depth map, the original depth map is compressed using a preset depth map compression model to obtain a corresponding compressed depth map. The depth map compression model has a symmetrical structure and consists of a monocular depth estimation module, a depth compression module, and a mask generation and processing module. The step of compressing the original depth map using a preset depth map compression model includes: The color image is processed by the monocular depth estimation module to obtain the corresponding predicted depth map; Calculate the residual information between the predicted depth map and the original depth map; The residual information is processed by the depth compression module to obtain the corresponding residual depth map; The original depth map is processed by the mask generation and processing module to obtain the corresponding image mask; According to the preset reconstruction formula, the predicted depth map, the residual depth map, and the image mask are combined to obtain the compressed depth map.

2. The method as described in claim 1, characterized in that, The step of processing the color image through the monocular depth estimation module includes: The depth features of the color image are extracted by the encoder in the monocular depth estimation module; Based on the extracted depth features, the predicted depth map is generated by the decoder in the monocular depth estimation module.

3. The method as described in claim 1, characterized in that, The step of processing the residual information through the deep compression module includes: The residual information is transformed by the encoder in the deep compression module to obtain the corresponding latent representation; The residual depth map is obtained by reconstructing the latent representation using the decoder in the depth compression module.

4. The method as described in claim 3, characterized in that, After the step of transforming the residual information by the encoder in the deep compression module and before the step of reconstructing the latent representation by the decoder in the deep compression module, the method further includes: The correlations in the latent representation are estimated and eliminated by the super-prior module in the deep compression module.

5. A depth map compression device based on monocular depth estimation, characterized in that, The device includes: An image input unit is configured to receive an input color image and a corresponding original depth map; and An image compression unit is used to compress the original depth map according to the received color image and the original depth map using a preset depth map compression model to obtain a corresponding compressed depth map. The depth map compression model is a symmetrical structure and consists of a monocular depth estimation module, a depth compression module, and a mask generation and processing module. The image compression unit includes: The first processing unit is used to process the color image through the monocular depth estimation module to obtain the corresponding predicted depth map; An information calculation unit is used to calculate the residual information between the predicted depth map and the original depth map; The second processing unit is used to process the residual information through the depth compression module to obtain the corresponding residual depth map; The third processing unit is used to process the original depth map through the mask generation and processing module to obtain the corresponding image mask; and An image combination unit is used to combine the predicted depth map, the residual depth map, and the image mask according to a preset reconstruction formula to obtain the compressed depth map.

6. The apparatus as claimed in claim 5, characterized in that, The first processing unit includes: A depth feature extraction unit is used to extract depth features of the color image through the encoder in the monocular depth estimation module; and The depth map generation unit is used to generate the predicted depth map based on the extracted depth features through the decoder in the monocular depth estimation module.

7. The apparatus as claimed in claim 5, characterized in that, The second processing unit includes: The information transformation unit is used to transform the residual information through the encoder in the deep compression module to obtain the corresponding latent representation; The information reconstruction unit is used to reconstruct the latent representation through the decoder in the depth compression module to obtain the residual depth map.

8. The apparatus as claimed in claim 7, characterized in that, The device further includes: The information elimination unit is used to estimate and eliminate the correlations in the latent representation through the super-prior module in the deep compression module.

9. An image processing apparatus, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 4.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 4.

Images