Depth map super-resolution method based on implicit neural representation guided by a single RGB image

Abstract

The present invention belongs to the technical field of image quality enhancement and discloses a depth map super-resolution method based on implicit neural representation under the guidance of a single RGB image. The method includes establishing a network model of implicit neural representation, where the network model is used to map input pixel coordinates to depth values ​​corresponding to the pixel coordinates; inputting a pixel coordinate matrix of a known low-resolution depth map and a pixel coordinate matrix of a high-resolution depth map to be reconstructed into the network model respectively to obtain a predicted low-resolution depth map and a high-resolution depth map; adopting a pixel-by-pixel loss function between the predicted low-resolution depth map and the known low-resolution depth map, and an image consistency loss function between the predicted high-resolution depth map and the known high-resolution RGB map under the joint constraints, the generalization ability of the depth map super-resolution can be improved; at the same time, the present invention makes full use of the information of the RGB image, and the artifacts and blurring in the generated depth image are significantly reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of image quality enhancement, and more specifically, relates to a depth map super-resolution method based on implicit neural representation under the guidance of a single RGB image. Background Art

[0002] Depth information reveals the three-dimensional structure of a scene, and accurate depth maps are a key attribute for various vision tasks. However, due to cost and technical issues, the resolution of depth maps is often lower than that of the corresponding RGB images, which limits their application. Therefore, the RGB-guided depth map super-resolution task is to use a high-resolution RGB image of the same scene to guide and improve the resolution of a low-resolution depth map, generating a high-resolution depth map.

[0003] With the development of deep learning technology, the current mainstream RGB-image-guided deep super-resolution method is to build an end-to-end deep learning network to establish a mapping from low-resolution depth maps and high-resolution RGB images to high-resolution depth maps. This method relies on RGB-D datasets with high-resolution depth maps for supervised training. However, the difficulty in obtaining large, scene-rich RGB-D datasets with high-resolution depth maps has limited the generalization ability of these supervised deep learning methods.

[0004] In recent years, some self-supervised algorithms based on single sample pairs have been proposed. These algorithms do not rely on datasets, but only require a single low-resolution depth map and a corresponding single high-resolution RGB image to complete the super-resolution process of the depth map. These methods generally train the network by downsampling the predicted high-resolution depth map and then calculating the loss with the low-resolution depth map. However, this fixed downsampling process cannot simulate the downsampling process corresponding to different actual scenes, resulting in insufficient generalization ability of these methods. In addition, during the training process, these methods generally constrain network parameters based on the information in the low-resolution depth map, and have limited utilization of the high-resolution RGB image. The resulting depth map is prone to blurring, edge artifacts, or texture duplication. Summary of the Invention

[0005] In response to the above defects or improvement needs of the prior art, the present invention provides a depth map super-resolution method based on implicit neural representation under the guidance of a single RGB image, which aims to improve the generalization ability of depth map super-resolution.

[0006] To achieve the above objectives, according to a first aspect of the present invention, a depth map super-resolution method based on implicit neural representation guided by a single RGB image is provided, comprising:

[0007] Establishing a network model of implicit neural representation, wherein the network model is used to map input pixel coordinates to depth values ​​corresponding to the pixel coordinates;

[0008] Inputting the pixel coordinate matrix of the known low-resolution depth map into the network model to obtain a predicted low-resolution depth map;

[0009] Inputting the pixel coordinate matrix of the high-resolution depth map to be reconstructed into the network model to obtain a predicted high-resolution depth map;

[0010] The network model is trained with minimizing a loss function as a convergence goal to obtain a trained network model; wherein the loss function includes a pixel-by-pixel loss function between the predicted low-resolution depth map and the known low-resolution depth map, and an image consistency loss function between the predicted high-resolution depth map and the known high-resolution RGB image; wherein the known high-resolution RGB image is a single high-resolution RGB image corresponding to the same scene as the known low-resolution depth map;

[0011] The pixel coordinate matrix of the high-resolution depth map to be reconstructed is input into the trained network model to obtain the required high-resolution depth map.

[0012] Furthermore, the image consistency loss function includes the plane loss L ps and marginal loss L es ;

[0013] The plane loss L ps is the feature loss between the gradient map of the predicted high-resolution depth map and the gradient map of the known high-resolution RGB map;

[0014] The edge loss L es The calculation methods include:

[0015] Calculate the two-dimensional curvature map of the predicted high-resolution depth map, and convert the two-dimensional curvature map into the image according to a preset edge curvature threshold. Figure 2 The binary image after dilation is used as a mask;

[0016] Multiplying the mask by the known gradient map of the high-resolution RGB image to obtain the gradient map of the filtered high-resolution RGB image;

[0017] Calculate the feature loss between the gradient map of the high-resolution RGB image after filtering and the gradient map of the predicted high-resolution depth image to obtain the edge loss L es .

[0018] Furthermore, the plane loss L ps and the edge loss Les They are:

[0019]

[0020] Where I is a known high-resolution RGB image, H and W are its length and width respectively; D H×W is the predicted high-resolution depth map; and Corresponding to the gradient operators in the x and y directions; is the gradient of the high-resolution RGB image after filtering, Pick or Corresponding to the gradient of the high-resolution RGB image after filtering in the x and y directions; |D H×W -(F r×r ·D H×W )| is the predicted high-resolution depth map D H×W The two-dimensional curvature map of F r×r is an average filter with kernel size r; T c is the preset edge curvature threshold; Φ(~) is the indicator function, which returns 1 when the condition is met, otherwise it returns 0. The result is used to characterize the two-dimensional curvature map after binarization; Dilate(~) is the dilation function.

[0021] Furthermore, the pixel-by-pixel loss function L mse for:

[0022]

[0023] Among them, D h×w is the predicted low-resolution depth map, D h×w is a known low-resolution depth map, h and w are the length and width of the known low-resolution depth map or the predicted low-resolution depth map.

[0024] Furthermore, the pixel coordinates input into the network model are normalized and dynamically position-encoded pixel coordinates; wherein the formula for dynamically position-encoding the normalized pixel coordinates is:

[0025] γ(x)=[Wcos(2πBx),Wsin(2πBx)] T

[0026] Where, is the normalized two-dimensional pixel coordinate, with a value range of (-1,1); is the pixel coordinate after encoding, m is the preset encoding dimension, is the encoding frequency, and the encoding frequency of the i-th dimension is [b i,0 ,b i,1], and the encoding frequency B is a random quantity that obeys a normal distribution with a mean of 0 and a variance of σ; W is a dynamic weight that changes dynamically with the training of the network model. There are m components in total, and the weight of the i-th dimension is ω i ∈{0,1}, when the encoding frequency of the i-th dimension exceeds the preset frequency threshold T f , then the weight of the corresponding i-th dimension is set to 0, otherwise it is 1.

[0027] Furthermore, the preset frequency threshold T f for:

[0028]

[0029] Among them, k is the current iteration number of the network model, and k1 and k2 are the preset iteration number thresholds.

[0030] Furthermore, the network model is a multi-layer perceptron network.

[0031] According to a second aspect of the present invention, there is provided a depth map super-resolution system based on implicit neural representation guided by a single RGB image, comprising a computer-readable storage medium and a processor;

[0032] The computer-readable storage medium is used to store executable instructions;

[0033] The processor is configured to read the executable instructions stored in the computer-readable storage medium to execute the depth map super-resolution method according to any one of the first aspects.

[0034] According to a third aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the depth map super-resolution method as described in any one of the first aspects is implemented.

[0035] According to a fourth aspect of the present invention, a computer program product is provided. When the computer program product is run on a computer, the computer is enabled to execute the depth map super-resolution method according to any one of the first aspects.

[0036] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0037] (1) The depth map super-resolution method based on implicit neural representation under the guidance of a single RGB image in the present invention uses a paired single high-resolution RGB image and a low-resolution depth map to train a network model of implicit neural representation to complete the super-resolution process of the depth map. Specifically, a mapping from image space coordinates to depth values ​​is established by constructing a network model of implicit neural representation, and the pixel coordinate matrix of the low-resolution depth map is input into the network model. The output of the network model is a predicted low-resolution depth map with the same size as the pixel coordinate matrix of the low-resolution depth map; similarly, the pixel coordinate matrix of the high-resolution depth map to be reconstructed is input into the network model, and the output of the network model is a predicted high-resolution depth map with the same size as the pixel coordinate matrix of the high-resolution depth map to be reconstructed. During the training process of the network model, the network model is trained based on the pixel-by-pixel loss between the predicted low-resolution depth map and the known low-resolution depth map, and the image consistency loss between the predicted high-resolution depth map and the known high-resolution RGB map, so that a trained network model can be obtained without the need for corresponding downsampling operations. In this way, the method of the present invention can adapt to depth data of different scenes and has strong generalization ability.

[0038] (2) Furthermore, the designed consistency loss function L s Including plane loss L ps and marginal loss L es , construct the plane loss L based on the gradient map corresponding to the predicted high-resolution depth map and the gradient map corresponding to the known high-resolution RGB map ps , ensuring that the smooth areas in the RGB image also appear smooth in the generated depth map. Regarding the learning of edge information, considering that the edge information in the high-resolution RGB image contains not only geometric edge information but also texture edge information, in order to filter out the texture edge information, a binary mask map is constructed based on the predicted high-resolution depth map, and the binary mask map is multiplied with the gradient map of the known high-resolution RGB image to filter out the edges of the texture in the gradient map of the high-resolution RGB image. At this time, the gradient map of the filtered high-resolution RGB image is considered to retain only the geometric edge information. The network model is trained based on the feature loss between the gradient map of the filtered high-resolution RGB image and the gradient map of the predicted high-resolution depth map, which ensures that the geometric edges in the RGB image appear as sharp edges in the generated high-resolution depth map. In this way, the consistency loss function designed by the present invention effectively utilizes the depth clues in the RGB image, so that the generated depth map has a smooth transition and sharp edges, showing better super-resolution capabilities.

[0039] (3) As a preference, by normalizing the pixel coordinates of the low-resolution depth map and the pixel coordinate matrix of the high-resolution depth map to be reconstructed and then performing dynamic position encoding, the dimension of each two-dimensional pixel coordinate is expanded, which can accelerate the convergence speed of the network model.

[0040] (4) Furthermore, the present invention sets the frequency threshold T f It is designed as a piecewise function related to the training round k, and different frequency thresholds T are selected at different iteration stages. f Specifically, in the initial stage of iteration (k<k1), a lower frequency domain threshold is designed. For the coding frequency of the i-th dimension, only the weight corresponding to the relatively low coding frequency will be set to 1, and the weight corresponding to the second highest or very high frequency will be set to 0. That is, in this stage, the network will first fit the low-frequency information; as the number of iterations increases, the frequency threshold becomes higher and higher, and the network model gradually learns high-frequency information. The design of the frequency threshold corresponding to the later stage of iteration (k>k2) can ensure that the weights of all random quantities are 1, and the fitting of all frequency band information can be performed. The frequency threshold T f The design of (k) ensures a stable transition from low frequency to high frequency during the network model training process, maximizing the ability of the implicit neural representation network model to represent complex functions.

[0041] (5) Preferably, the network model of implicit neural representation constructed by the present invention is a multi-layer perceptron network, which has a stronger fitting ability for high-dimensional information after dynamic position encoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flow chart of a single-sample RGB image-guided depth image super-resolution method based on implicit neural representation in an embodiment of the present invention.

[0043] Figure 2 Schematic diagram of the loss function of the network model during the training process in an embodiment of the present invention.

[0044] Figure 3 Schematic diagram of the overall architecture of the implicit neural representation network in an embodiment of the present invention.

[0045] Figure 4 Schematic diagram of a pixel coordinate matrix provided in an embodiment of the present invention.

[0046] Figure 5 A comparison chart of the visualization results of the indoor dataset tested in an embodiment of the present invention.

[0047] Figure 6 This is a comparison chart of the visualization results of the test in the real data set provided by the embodiment of the present invention, where: Figure 6(a) is the original low-resolution depth map, (b) is the super-resolution result of the MMSR method, (c) is the super-resolution result of the P2P method, and (d) is the super-resolution result of the present invention. DETAILED DESCRIPTION

[0048] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to illustrate the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0049] Example 1

[0050] like Figure 1 and Figure 2 As shown, the depth map super-resolution method based on implicit neural representation under the guidance of a single RGB image in an embodiment of the present invention mainly includes:

[0051] Obtain a single low-resolution depth map and a corresponding single high-resolution RGB image corresponding to a scene to be super-resolved, and use them as the corresponding known low-resolution depth map and high-resolution RGB image respectively;

[0052] Establishing an implicit neural representation network model, which is used to map the input pixel coordinates to the depth values ​​corresponding to the pixel coordinates;

[0053] The pixel coordinate matrix of the known low-resolution depth map is input into the above-constructed network model to obtain a predicted low-resolution depth map; wherein the pixel coordinate matrix is ​​a matrix composed of the coordinates of each pixel in the coordinate order;

[0054] Input the pixel coordinate matrix of the high-resolution depth map to be reconstructed into the above-constructed network model to obtain the predicted high-resolution depth map;

[0055] The network model constructed above is trained with minimization of the loss function as the convergence goal. When the goal converges or a preset number of training times is reached, the training is stopped to obtain a trained network model. The loss function includes a pixel-by-pixel loss function between the predicted low-resolution depth map and the known low-resolution depth map, and an image consistency loss function between the predicted high-resolution depth map and the known high-resolution RGB map.

[0056] The pixel coordinate matrix of the high-resolution depth map to be reconstructed is input into the trained network model to obtain the required high-resolution depth map.

[0057] The depth map super-resolution method based on implicit neural representation under the guidance of a single RGB image in the present invention uses a paired single high-resolution RGB image and a low-resolution depth map to train a network model of implicit neural representation to complete the super-resolution process of the depth map. Specifically, a mapping of image space coordinates to depth values ​​is established by constructing a network model of implicit neural representation, and the pixel coordinate matrix of the low-resolution depth map is input into the network model. The output of the network model is a predicted low-resolution depth map with the same size as the pixel coordinate matrix of the low-resolution depth map; similarly, the pixel coordinate matrix of the high-resolution depth map to be reconstructed is input into the network model, and the output of the network model is a predicted high-resolution depth map with the same size as the pixel coordinate matrix of the high-resolution depth map to be reconstructed. During the training process of the network model, the network model training is performed based on the pixel-by-pixel loss between the predicted low-resolution depth map and the low-resolution depth map, and the image consistency loss between the predicted high-resolution depth map and the high-resolution RGB map, so that a trained network model can be obtained, and there is no need to perform corresponding downsampling operations. In this way, the method of the present invention can adapt to depth data of different scenes and has strong generalization ability.

[0058] Preferably, before inputting the pixel coordinates into the above-constructed network model, the method further includes:

[0059] Normalize pixel coordinates;

[0060] Dynamically position-encode the normalized pixel coordinates to expand the dimension of each two-dimensional pixel coordinate;

[0061] The encoded pixel coordinates are input into the network model constructed above.

[0062] By normalizing the pixel coordinate matrix of the low-resolution depth map and the pixel coordinate matrix of the high-resolution depth map to be reconstructed and then performing dynamic position encoding, the dimension of each two-dimensional pixel coordinate is expanded, which can speed up the convergence of the network model.

[0063] In the embodiment of the present invention, the dynamic position encoding formula used is:

[0064] γ(x)=[Wcos(2πBx),Wsin(2πBx)] T

[0065] in, is the normalized two-dimensional pixel coordinate, with a value range of (-1,1); is the pixel coordinate after encoding, m is the preset encoding dimension, is the encoding frequency, and the encoding frequency of the i-th dimension is [b i,0 ,b i,1], and the encoding frequency B is a random quantity, which obeys the normal distribution with mean 0 and variance σ; W is the dynamic weight, which changes dynamically with the training of the network model. There are m components in total, and the weight of the i-th dimension is ω i ∈{0,1}, when the encoding frequency of the i-th dimension exceeds the preset frequency threshold T f , then the weight of the corresponding i-th dimension is set to 0, otherwise it is 1.

[0066] As a further design of the present invention, the preset frequency threshold T f for:

[0067]

[0068] Among them, k is the current number of iterations (training rounds) of the network model, k1 and k2 are preset training round thresholds, which are selected based on experience in practical applications.

[0069] In the embodiment of the present invention, the frequency threshold T f It is designed as a piecewise function related to the training round k, and different frequency thresholds T are selected at different iteration stages. f Specifically, in the initial stage of iteration (k<k1), a lower frequency domain threshold is designed. For the coding frequency of the i-th dimension, only the weight corresponding to the relatively low coding frequency will be set to 1, and the weight corresponding to the second highest or very high frequency will be set to 0. That is, in this stage, the network will first fit the low-frequency information; as the number of iterations increases, the frequency threshold becomes higher and higher, and the network model gradually learns high-frequency information. The design of the frequency threshold corresponding to the later stage of iteration (k>k2) can ensure that the weights of all random quantities are 1, and the fitting of all frequency band information can be performed. The frequency threshold T f The design of (k) ensures a stable transition from low frequency to high frequency during the network model training process, maximizing the ability of the implicit neural representation network model to represent complex functions.

[0070] Preferably, the implicit neural representation network model constructed in embodiments of the present invention is a multilayer perceptron (MLP) network, which has stronger fitting power for high-dimensional information after dynamic position encoding. Specifically, the MLP network is constructed with m hidden layers, each containing n neurons; a GeLU function is used for activation before each hidden layer. In other embodiments, other feedforward artificial neural network models may also be used.

[0071] In the embodiment of the present invention, the pixel-by-pixel loss function L between the predicted low-resolution depth map and the known low-resolution depth map is mse for:

[0072]

[0073] Among them, h and w are the length and width of the low-resolution depth map, D h×w The low-resolution depth map predicted by the network model, D h×w is a known low-resolution depth map. In other embodiments, the pixel-by-pixel loss may also adopt a loss function such as L1 loss.

[0074] As a preference, the image consistency loss function L between the predicted high-resolution depth map and the known high-resolution RGB map is s Including plane loss L ps and marginal loss L es :

[0075] L s =L ps +L es

[0076] Plane loss L ps It is the feature loss between the gradient map of the predicted high-resolution depth map and the gradient map of the known high-resolution RGB image.

[0077] Margin loss L es The calculation methods include:

[0078] Calculate the two-dimensional curvature map of the predicted high-resolution depth map, and convert the two-dimensional curvature into the image according to the preset edge curvature threshold. Figure 2 The binary image after dilation is used as a mask, i.e., a binary mask map.

[0079] Multiply the mask by the known gradient map of the high-resolution RGB image to obtain the gradient map of the filtered high-resolution RGB image;

[0080] The edge loss is obtained by calculating the feature loss between the gradient map of the filtered high-resolution RGB image and the gradient map of the predicted high-resolution depth image.

[0081] Specifically, in the embodiment of the present invention, the plane loss L ps and marginal loss L es They are defined as:

[0082]

[0083]

[0084] Where I is a known high-resolution RGB image with length and width of H and W respectively, and D H×W High-resolution depth map predicted by the network model; and Corresponding to the gradient operators in the x and y directions; and Corresponding to the gradient of the known high-resolution RGB image I in the x and y directions; is the gradient of the masked RGB image, that is, the gradient of the high-resolution RGB image after filtering, Pick or Corresponding to the gradient of the high-resolution RGB image after filtering in the x and y directions; |D H×W -(F r×r ·D H×W )| is the predicted high-resolution depth map D H×W The two-dimensional curvature map is essentially a region with drastic grayscale changes. After average filtering, the difference with the original image (the predicted high-resolution depth map) becomes larger; the local region with uniform grayscale is the opposite. The purpose is to distinguish the region with drastic grayscale changes, that is, the depth edge region; the local region with uniform grayscale is the smooth region, thus distinguishing the edge region from the smooth region. r×r is an average filter with kernel size r; T c is the preset edge curvature threshold; Φ(~) is an indicator function that returns 1 if the condition is met and 0 otherwise. The result is used to characterize the two-dimensional curvature map after binarization; Dilate(~) is a dilation function that is used to dilate the two-dimensional curvature map after binarization to obtain the required binary mask map.

[0085] Therefore, the final loss function Loss is:

[0086] Loss = L mse +αL s

[0087] Among them, α is the proportional coefficient of the loss function.

[0088] In the embodiment of the present invention, the designed consistency loss function L s Including plane loss L ps and marginal loss L es , considering that the smooth texture area in the known high-resolution RGB image (the area with a gradient of 0) should also show smoothness in the corresponding position in its predicted high-resolution depth image, the plane loss L is constructed based on the gradient image corresponding to the predicted high-resolution depth image and the gradient image corresponding to the known high-resolution RGB image. ps, ensuring that the smooth areas in the RGB image also appear smooth in the generated depth map. Regarding the learning of edge information, considering that the edge information in the high-resolution RGB image contains not only geometric edge information but also texture edge information, in order to filter out the texture edge information, a binary mask map is constructed based on the predicted high-resolution depth map, and the binary mask map is multiplied with the gradient map of the known high-resolution RGB image to filter out the edges of the texture in the gradient map of the high-resolution RGB image. At this time, the gradient map of the filtered high-resolution RGB image is considered to retain only the geometric edge information. The network model is trained based on the feature loss between the gradient map of the filtered high-resolution RGB image and the gradient map of the predicted high-resolution depth map, which ensures that the geometric edges in the RGB image appear as sharp edges in the generated high-resolution depth map. In this way, the consistency loss function designed by the present invention effectively utilizes the depth clues in the RGB image, so that the generated depth map has a smooth transition and sharp edges, showing better super-resolution capabilities.

[0089] The method of the present invention is further described below by taking a low-resolution depth image with a size of 112*112 and a high-resolution RGB image with a size of 448*448 as an example.

[0090] A. A network model for implicit neural representation was built using a multilayer perceptron (MLP). The network consisted of four hidden layers, each containing 256 neurons. A GeLU function was used for activation before each hidden layer. The pixel coordinate matrix underwent dynamic position encoding before being input into the network. During dynamic position encoding, m was 128, σ was 1.0, and k1 and k2 were 1000 and 5000, respectively.

[0091] The overall architecture of the implicit neural representation network model is as follows Figure 3 shown.

[0092] B. For a scene to be super-resolutioned, the existing observations are a single 112*112 depth map and a corresponding 448*448 RGB image. This single sample pair is used as the constraint data and the loss function is calculated:

[0093] B1. Dynamically encode the pixel coordinate matrix of the 112*112 depth map. The schematic diagram of the pixel coordinate matrix is ​​as follows: Figure 4 As shown in the figure. After dynamic position encoding, each coordinate is expanded from two dimensions to 256 dimensions. The upgraded pixel coordinate matrix is ​​input into the network model, and the network outputs a low-resolution depth prediction map of size 112*112. The depth prediction map is compared with the existing low-resolution depth map to calculate the pixel-by-pixel loss function. In the loss calculation, the length h and width w of the low-resolution depth map are both 112.

[0094] B2. Dynamically position-encode the 448*448 pixel coordinate matrix, expanding each coordinate from two dimensions to 256 dimensions. The upgraded pixel coordinate matrix is ​​input into the network model, and the network outputs a 448*448 high-resolution depth prediction image. The structural consistency loss function is calculated for the high-resolution depth prediction image and the high-resolution RGB image. In the loss calculation, the height H and width W of the high-resolution RGB image are both 448. The filter size r is 5, and T c The threshold is 0.05 and the kernel of the dilation function is 5.

[0095] B3. The final loss function is the sum of the pixel-by-pixel loss function and the structural consistency loss function Loss = L mse +αL s , with the minimization of the final loss function as the convergence goal, the network model of the implicit neural representation is trained; α is the loss function proportional coefficient, which is 0.02.

[0096] B4. Train the network model for 10,000 iterations. Use the Adam optimizer with an initial learning rate of 1e-3. As the iterations progress, the learning rate decays to 1e-6 using a cosine decay strategy. The loss function scaling factor α is set to 0 before 2,000 iterations and to 0.02 thereafter.

[0097] C. Application stage: The 448*448 pixel coordinate matrix is ​​dynamically encoded and input into the trained network model, and the network outputs a 448*448 high-resolution depth map.

[0098] D. The above-mentioned depth map super-resolution algorithm is evaluated on a simulated indoor dataset. Specifically, an RGB-D dataset is obtained, which contains a high-resolution depth map of 448*448 size and a corresponding RGB map of the same size. According to the nearest downsampling method, the high-resolution depth map is downsampled to 224*224, 112*112, and 56*56 sizes, respectively, to obtain low-resolution depth maps that are downsampled 2 times, 4 times, and 8 times, respectively. The low-resolution depth map and the high-resolution RGB map constitute a pair of single sample pairs. Using each sample pair to train the implicit neural representation network model, a high-resolution depth result map of 448*448 size can be generated. The generated high-resolution depth map is compared with the original high-resolution depth map to evaluate the effectiveness of the super-resolution algorithm.

[0099] D1. Selecting an appropriate evaluation index. In the embodiment of the present invention, the root mean square error (RMSE) evaluation index is used. The smaller the RMSE, the smaller the error.

[0100] D2. Select an appropriate comparison algorithm. In this embodiment of the present invention, the most advanced RGB-guided depth map super-resolution networks in the field of image quality enhancement, namely P2P, MMSR, SSGNet, SVLRM, FDSR, DCTNet, AHMF, RSAG, and geoDSR, were selected for comparison. The first three algorithms (P2P, MMSR, and SSGNet) are all single-sample-pair RGB-guided depth map super-resolution methods that do not require any training datasets and use each sample pair to train the network model. The remaining methods are all supervised RGB-guided depth map super-resolution methods that are trained using the common NYU dataset.

[0101] D3. Objective and subjective evaluation of the super-resolution capability of the present invention. Table 1 shows the super-resolution results in four indoor datasets: MiddleBury, Lu, HyperSim and Hammer. Figure 5 The comparison of the residual map calculated by each method is shown. The residual map refers to the difference between the generated high-resolution depth map and the original true high-resolution depth map.

[0102] Table 1 Super-resolution results of different methods on four indoor datasets

[0103]

[0104] The 2×, 4×, and 8× in Table 1 represent low-resolution depth maps obtained by downsampling the high-resolution depth map by 2 times, 4 times, and 8 times, respectively; the bold data and underlined data in each column represent the optimal data and suboptimal data in that column, respectively.

[0105] As can be seen from Table 1, the method of the present invention (INR-DSR) achieves good reconstruction results in all data sets, especially in the case of low magnification, the method of the present invention outperforms most supervised algorithms. Figure 5 As can be seen from the figure, the difference of the method of the present invention is the smallest. This shows that the method of the present invention has good effects on various data sets, thus proving the generalization ability of the present invention.

[0106] E. The above-mentioned depth map super-resolution algorithm is evaluated on a real dataset. First, the real dfc_2018 depth dataset is obtained, which only contains high-resolution RGB images and low-resolution depth maps. The resolution of the depth map is 10 times lower than that of the RGB image. Secondly, the depth map is cropped to 34*34 size, corresponding to the RGB image size of 340*340, and the outliers in the depth map are manually removed. Then, the self-supervised P2P and MMSR methods are compared. For three different scenarios, the subjective evaluation results are as follows: Figure 6 As shown. Among them, Figure 6(a) is the original low-resolution depth map, (b) is the super-resolution result of the MMSR method, (c) is the super-resolution result of the P2P method, and (d) is the super-resolution result of the present invention. Figure 6 It can be seen from the figure that compared with MMSR and P2P, the proposed method obtains smooth results in flat areas and can restore sharp edges, achieving good reconstruction effects.

[0107] Example 2

[0108] An embodiment of the present invention provides a depth map super-resolution system based on implicit neural representation under the guidance of a single RGB image, including a computer-readable storage medium and a processor;

[0109] Computer-readable storage media for storing executable instructions;

[0110] The processor is configured to read the executable instructions stored in the computer-readable storage medium to execute the depth map super-resolution method of any one of the first aspects. For related technical solutions, see the description in Example 1, which will not be repeated here.

[0111] Example 3

[0112] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the depth map super-resolution method according to any one of the first aspects is implemented. For related technical solutions, see the description in Example 1 and will not be repeated here.

[0113] Example 4

[0114] An embodiment of the present invention provides a computer program product that, when executed on a computer, causes the computer to perform the depth map super-resolution method according to any one of the first aspects. For related technical solutions, see the description in Example 1, which will not be repeated here.

[0115] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A depth map super-resolution method based on implicit neural representation guided by a single RGB image, characterized by: include: Establishing a network model of implicit neural representation, wherein the network model is used to map input pixel coordinates to depth values ​​corresponding to the pixel coordinates; Inputting the pixel coordinate matrix of the known low-resolution depth map into the network model to obtain a predicted low-resolution depth map; Inputting the pixel coordinate matrix of the high-resolution depth map to be reconstructed into the network model to obtain a predicted high-resolution depth map; The network model is trained with minimizing a loss function as a convergence goal to obtain a trained network model; wherein the loss function includes a pixel-by-pixel loss function between the predicted low-resolution depth map and the known low-resolution depth map, and an image consistency loss function between the predicted high-resolution depth map and the known high-resolution RGB image; wherein the known high-resolution RGB image is a single high-resolution RGB image corresponding to the same scene as the known low-resolution depth map; Inputting the pixel coordinate matrix of the high-resolution depth map to be reconstructed into the trained network model to obtain the required high-resolution depth map; The pixel coordinates input into the network model are normalized and dynamically position-encoded pixel coordinates; wherein the formula for dynamically position-encoding the normalized pixel coordinates is: Where, is the normalized two-dimensional pixel coordinate, with a value range of (-1, 1); is the encoded pixel coordinate, is the preset encoding dimension, is the encoding frequency, and the encoding frequency of the i-th dimension is , and the coding frequency is a random quantity with a mean of 0 and a variance of Normal distribution; is the dynamic weight, which changes dynamically with the training of the network model. components, and the weight of the i-th dimension is , when the encoding frequency of the i-th dimension exceeds the preset frequency threshold , then the corresponding weight of the i-th dimension is set to 0, otherwise it is 1; The preset frequency threshold for: in, is the current iteration number of the network model, and is the preset iteration threshold.

2. The depth map super-resolution method according to claim 1, wherein: The image consistency loss function includes plane loss and marginal loss ; The plane loss is the feature loss between the gradient map of the predicted high-resolution depth map and the gradient map of the known high-resolution RGB map; The edge loss The calculation methods include: Calculating a two-dimensional curvature map of the predicted high-resolution depth map, binarizing the two-dimensional curvature map according to a preset edge curvature threshold, performing a dilation operation, and using the dilated binary image as a mask; Multiplying the mask by the known gradient map of the high-resolution RGB image to obtain the gradient map of the filtered high-resolution RGB image; Calculate the feature loss between the gradient map of the high-resolution RGB image after filtering and the gradient map of the predicted high-resolution depth map to obtain the edge loss .

3. The depth map super-resolution method according to claim 2, wherein: The plane loss and the edge loss They are: in, is a known high-resolution RGB image, and are its length and width respectively; is the predicted high-resolution depth map; and Corresponding representation and Directional gradient operator; is the gradient of the high-resolution RGB image after filtering, Pick or , which corresponds to the high-resolution RGB image after filtering. and Directional gradient; For the predicted high-resolution depth map The two-dimensional curvature map of The kernel size is Average filter; is the preset edge curvature threshold; It is an indicator function that returns 1 when the condition is met, otherwise it returns 0. Its result is used to represent the two-dimensional curvature map after binarization; is the expansion function.

4. The depth map super-resolution method according to claim 1, wherein: The pixel-by-pixel loss function for: in, is the predicted low-resolution depth map, is a known low-resolution depth map, and The length and width of the known low-resolution depth map or the predicted low-resolution depth map.

5. The depth map super-resolution method according to claim 1, wherein: The network model is a multi-layer perceptron network.

6. A depth map super-resolution system based on implicit neural representation guided by a single RGB image, characterized by: comprising a computer-readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is configured to read the executable instructions stored in the computer-readable storage medium to execute the depth map super-resolution method according to any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the depth map super-resolution method according to any one of claims 1 to 5 is implemented.

8. A computer program product, characterized in that When the computer program product is run on a computer, the computer is enabled to execute the depth map super-resolution method according to any one of claims 1 to 5.

Images