Training method and device of depth image reconstruction model, electronic equipment

Abstract

The application provides a training method and device of a depth image reconstruction model, electronic equipment and a storage medium, which are used to improve the accuracy and speed of training of the depth image reconstruction model, and further improve the accuracy of a depth image predicted by the depth image reconstruction model. The method comprises the following steps: obtaining a first depth image corresponding to each scene image in each group of scene images by using a trained teacher model; filtering abnormal pixel points in the first depth image in the same group according to a spatial geometric projection error and a re-projection error to obtain a depth filtering image corresponding to each first depth image; calculating a first loss value of the depth filtering image corresponding to each scene image in each group of scene images and a second depth image, wherein the second depth image is obtained by using a depth image reconstruction model; and updating the depth image reconstruction model according to the first loss value.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a training method, apparatus, electronic device, and storage medium for a depth image reconstruction model. Background Technology

[0002] Unsupervised / self-supervised MVS (multi-view stereo vision) models transform the multi-view depth estimation task into an image reconstruction task by leveraging the photometric consistency between multiple views. The reference view is reconstructed based on the predicted depth map, thereby achieving low constraint on the predicted depth map.

[0003] Currently, supervised MVS methods require training on datasets with deep ground truth, which are difficult to obtain and limited in size, thus restricting the generalization of such methods in different scenarios. Methods that utilize external information such as optical flow require the creation of optical flow pseudo-labels for a large amount of unlabeled training data, consuming a lot of training time and storage space, resulting in a complex training process. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a training method, apparatus, electronic device and storage medium for a depth image reconstruction model, which can improve the accuracy and speed of training the depth image reconstruction model, thereby improving the accuracy of the depth image predicted by the depth image reconstruction model.

[0005] In a first aspect, embodiments of this application provide a method for training a depth image reconstruction model, the method comprising:

[0006] The trained teacher model is used to obtain the first depth image corresponding to each scene image in each group of scene images; the first depth image includes the depth value and probability value of each pixel.

[0007] Based on spatial geometric projection error and reprojection error, abnormal pixels in the first depth image within the same group are filtered to obtain the depth-filtered image corresponding to each first depth image.

[0008] Calculate the first loss value of the depth-filtered image and the second depth image corresponding to each scene image in each group of scene images, where the second depth image is obtained through the depth image reconstruction model;

[0009] The depth image reconstruction model is updated based on the first loss value.

[0010] In one possible implementation, each set of scene images includes a reference image and an auxiliary image taken from an adjacent viewpoint to the reference image, and the training process of the teacher model is as follows:

[0011] A set of scene images is input into the teacher model to obtain the first depth image of the reference image, as well as the image features corresponding to the reference image and the auxiliary image respectively;

[0012] Using the first depth image of the reference image, a differentiable transformation is performed on the auxiliary image to obtain a reconstructed auxiliary image, and the image features of the auxiliary image are then differentiable to obtain the reconstructed image features.

[0013] A second loss value is calculated based on the reconstructed auxiliary image, the features of the reconstructed image, the reference image, and the image features of the reference image;

[0014] The teacher model is updated according to the second loss value until the accuracy of the teacher model reaches a preset value, and the last updated teacher model is used as the trained teacher model.

[0015] In one possible implementation, calculating the second loss value based on the reconstructed auxiliary image, the features of the reconstructed image, the reference image, and the image features of the reference image includes:

[0016] Calculate the image loss value of each reconstructed auxiliary image compared to the reference image, and calculate the feature loss value of each reconstructed image feature compared to the image feature of the reference image;

[0017] The second loss value is obtained by calculating based on all the image loss values ​​and the feature loss values.

[0018] In one possible implementation, obtaining the first depth image corresponding to each scene image in each group of scene images through the trained teacher model includes:

[0019] Each scene image in each group of scene images is used as a reference image, and the remaining scene images are used as auxiliary images. These are then input into the trained teacher model to obtain the first depth image corresponding to each scene image in each group of scene images.

[0020] In one possible implementation, calculating the first loss value of the depth-filtered image and the second depth image corresponding to each scene image in each group of scene images includes:

[0021] For each set of scene images, each scene image in the set is input into the depth image reconstruction model to obtain the second depth image corresponding to each scene image in the set.

[0022] For each set of scene images, calculate the first loss value between the depth-filtered image and the second depth image corresponding to the same scene image.

[0023] In one possible implementation, the teacher model is a multi-view stereo vision network (MVS) model.

[0024] In one possible implementation, the method further includes:

[0025] A set of images of the scene to be reconstructed is obtained, the set of images of the scene to be reconstructed includes a reference image and an auxiliary image whose shooting angle is adjacent to that of the reference image;

[0026] The group of scene images to be reconstructed is input into a trained depth image reconstruction model to obtain a depth image corresponding to the reference image in the scene images to be reconstructed.

[0027] Secondly, embodiments of this application also provide a training apparatus for a depth image reconstruction model, the training apparatus for the depth image reconstruction model comprising:

[0028] The acquisition module is used to obtain the first depth image corresponding to each scene image in each group of scene images through the trained teacher model; the first depth image includes the depth value and probability value of each pixel.

[0029] The filtering module is used to filter out abnormal pixels in the first depth images within the same group based on spatial geometric projection error and reprojection error, so as to obtain the depth-filtered images corresponding to each first depth image.

[0030] The calculation module is used to calculate the first loss value of the depth-filtered image and the second depth image corresponding to each scene image in each group of scene images, wherein the second depth image is obtained through the depth image reconstruction model;

[0031] An update module is used to update the depth image reconstruction model based on the first loss value.

[0032] Fifthly, embodiments of this application also provide an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the training method for the depth image reconstruction model as described in any of the first aspects.

[0033] In a sixth aspect, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the training method for the depth image reconstruction model as described in any of the first aspects.

[0034] This application provides a training method, apparatus, electronic device, and storage medium for a depth image reconstruction model. The method includes: obtaining a first depth image corresponding to each scene image in each group of scene images using a trained teacher model; the first depth image includes the depth value and probability value of each pixel; filtering out abnormal pixels in the first depth images belonging to the same group based on spatial geometric projection error and reprojection error to obtain a depth-filtered image corresponding to each first depth image; calculating a first loss value for the depth-filtered image and the second depth image corresponding to each scene image in each group of scene images, wherein the second depth image is obtained through the depth image reconstruction model; and updating the depth image reconstruction model based on the first loss value. Compared to existing technologies where the MVS model requires a dataset of depth values ​​and the creation of optical flow pseudo-labels, this application does not require a dataset of depth values ​​or the creation of optical flow pseudo-labels when training the depth image reconstruction model. That is, after obtaining the first depth images corresponding to each scene image, this application filters out abnormal pixels in the same group of depth images to ensure the accuracy of the depth images. Then, it calculates the first loss value of the second depth image obtained by the depth image reconstruction model and the corresponding depth-filtered image, and updates the depth image reconstruction model based on the first loss value. Thus, this invention can improve the accuracy and speed of training the depth image reconstruction model, thereby improving the accuracy of the depth images predicted by the depth image reconstruction model. Attached Figure Description

[0035] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 The flowchart illustrating the training process of the depth image reconstruction model provided in this application embodiment is shown.

[0037] Figure 2 The training structure diagram of the depth image reconstruction model provided in the embodiments of this application is shown;

[0038] Figure 3 A flowchart illustrating the teacher model training process provided in an embodiment of this application is shown.

[0039] Figure 4 The diagram shows the teacher model training structure provided in an embodiment of this application;

[0040] Figure 5 A diagram of a training apparatus for a depth image reconstruction model provided in an embodiment of this application is shown;

[0041] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0043] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0044] This application proposes a training method for a deep image reconstruction model. It utilizes a knowledge distillation-based MVS self-supervised training method (hereinafter referred to as KD-MVS) to train an MVS network on an unlabeled dataset. The method mainly comprises two stages: self-supervised teacher model training and knowledge distillation-based student model training. Specifically, the teacher model is trained in a self-supervised manner based on photometric and feature metric consistency. Then, knowledge from the teacher model is extracted into the student model through probabilistic knowledge distillation. Under validated knowledge supervision, the student model achieves better performance than the teacher model.

[0045] The self-supervised teacher model training phase involves training the teacher model by forcing photometric and feature metric consistency between the reference view and the reconstructed view. The reconstructed view can be obtained through homography transformation based on the estimated depth. Specifically, given the camera intrinsics and pose, KD-MVS transforms the RGB (photometric information) and features from adjacent viewpoints to the reference view based on the depth values ​​predicted by the MVS network, performing image and feature reconstruction. The depth values ​​predicted by the MVS network are constrained by ensuring that the reconstructed results are consistent with the images and features under the real reference view, thereby achieving the goal of training the MVS network. The real reference view image is the actual captured image, and the features under the real reference view are the features extracted by the MVS network itself.

[0046] Unlike existing self-supervised MVS methods that only use photometric consistency, this application proposes using internally extracted features for consistency measurement. Specifically, it utilizes high-dimensional features extracted by the MVS network itself as an evaluation criterion for view reconstruction quality (the reconstruction process is described above). These high-dimensional features have stronger information representation capabilities than RGB images and can better judge the quality of view reconstruction. Furthermore, using internal features extracted by the MVS network itself yields better results compared to high-dimensional features extracted by other external pre-trained models (e.g., VGG and ResNet). Experiments verify that the internal feature measurement loss is more suitable for MVS and can help the self-supervised teacher model generate relatively complete and accurate depth maps.

[0047] The student model training phase based on knowledge distillation mainly includes two steps: pseudo-probabilistic knowledge generation and student model training. First, the teacher model infers raw depth maps on unlabeled training data. These raw depth maps are the teacher model's original output and typically contain a lot of noise and inaccurate outliers. To remove these outliers, multi-view cross-validation can be used. Specifically, assuming accurate camera parameters, each pixel is projected into 3D space based on the predicted depth from a reference view, then back-projected to adjacent views to obtain the corresponding pixel coordinates. These pixels are then projected into 3D space again based on the predicted depth values ​​at these coordinates from adjacent views. The resulting 3D points have spatial geometric errors compared to those projected from the reference view. Furthermore, the second set of 3D points is back-projected back to the reference view to obtain pixel coordinates, which have reprojection errors compared to the original reference pixel coordinates. Filtering is performed based on spatial geometric errors and reprojection errors; points that do not meet the threshold conditions are considered outliers and are removed. Cross-validation is then performed to filter unreliable outliers. Finally, a pseudo-probability distribution for the teacher model is generated through probabilistic encoding. By forcing the student model's predicted probability distribution to be similar to the pseudo-probability distribution, probabilistic knowledge can be transferred to the student model. Specifically, the pseudo-probability distribution contains the teacher model's prediction information at that point, that is, the prediction result (probability distribution) made by the teacher model based on its own knowledge. By using a loss function to force the student model's probability prediction at the same point to be the same as the teacher model, it is equivalent to transferring the teacher model's knowledge to the student model.

[0048] The following is a detailed description of a training method for a depth image reconstruction model provided in the embodiments of this application.

[0049] Reference Figure 1 The diagram shown is a flowchart illustrating a training method for a depth image reconstruction model provided in this application embodiment. The specific execution process of the training method for the depth image reconstruction model is as follows:

[0050] S101. Using the trained teacher model, obtain the first depth image corresponding to each scene image in each group of scene images.

[0051] The scene image can be an RGB image, and the first depth image includes the depth value and probability value of each pixel. The probability value is used to represent the probability of the predicted depth value of each pixel in the first depth image obtained by the teacher model.

[0052] In this embodiment, each set of scene images includes scene images taken from multiple angles. Scene images with shooting angles less than a certain value can be identified as adjacent scene images. Specifically, these adjacent angles can be less than 5 degrees or 10 degrees; this embodiment does not impose a specific limitation on this. Specifically, in this embodiment, any one image can be selected from each set of scene images as a reference image, and the other images as auxiliary images. The reference image is the image that needs to be used to predict the first depth image using the trained teacher model, and the auxiliary images are used to assist in updating the training teacher model.

[0053] Specifically, this teacher model can utilize image features extracted by the MVS network itself as an evaluation metric for depth images. These image features have stronger information representation capabilities compared to RGB images, enabling a better assessment of the quality of view reconstruction. Furthermore, using image features extracted by the MVS network itself yields better results compared to image features extracted by other external pre-trained models (such as VGG and ResNet). The proposed image feature metric loss is more suitable for MVS and can help the self-supervised teacher model generate relatively complete and accurate first depth images.

[0054] It should be noted that the teacher model in this embodiment can be an existing model, or a model trained with labeled sample data. For example, the teacher model can be trained with multiple sets of scene images and the corresponding depth images of each set of sample images; it can also be trained with unlabeled sample images. The teacher model without labels will be explained in subsequent steps.

[0055] Specifically, obtaining the first depth image corresponding to each scene image in each group of scene images through the trained teacher model includes: taking each scene image in each group of scene images as a reference image and the remaining scene images as auxiliary images, and inputting them into the trained teacher model to obtain the first depth image corresponding to each scene image in each group of scene images; or inputting each scene image in each group of scene images into the trained teacher model to obtain the first depth image corresponding to each scene image in each group of scene images.

[0056] For example, given a set of scene images comprising four images: scene image A, scene image B, scene image C, and scene image D, each scene image can be used as a reference image, and the remaining scene images as auxiliary images. These are then input into a trained teacher model to obtain the first depth image corresponding to each scene image in the set. Specifically, scene image A can be used as the reference image, and the remaining scene images as auxiliary images, input into the trained teacher model to obtain the first depth image of scene image A; scene image B can be used as the reference image, and the remaining scene images as auxiliary images, input into the trained teacher model to obtain the first depth image of scene image B; scene image C can be used as the reference image, and the remaining scene images as auxiliary images, input into the trained teacher model to obtain the first depth image of scene image C; and scene image D can be used as the reference image, and the remaining scene images as auxiliary images, input into the trained teacher model to obtain the first depth image of scene image D.

[0057] For example, a set of scene images includes four scene images, namely scene image E, scene image F, scene image G, and scene image H. Scene images E, F, G, and H can be input into the trained teacher model to obtain the first depth images corresponding to scene images E, F, G, and H respectively.

[0058] S102. Based on the spatial geometric projection error and reprojection error, filter out abnormal pixels in the first depth image within the same group to obtain the depth-filtered image corresponding to each first depth image.

[0059] It should be noted that the first depth images predicted by the teacher model on unlabeled training data are the output results obtained directly from the teacher model. In other words, the first depth images predicted by the teacher model usually contain a lot of noise and inaccurate outliers. In order to remove these outliers, this application uses multi-view cross-validation to filter outliers.

[0060] Specifically, assuming accurate camera parameters, this application projects each pixel into 3D space based on the predicted depth of a first depth image, then backprojects it to adjacent viewpoints to obtain the corresponding pixel coordinates. Next, based on the predicted depth values ​​corresponding to these coordinates in the adjacent viewpoints, these pixels are projected into 3D space. At this point, the 3D points projected from the reference view have spatial geometric errors. Furthermore, the second set of 3D points is backprojected to the reference view to obtain pixel coordinates, which have reprojection errors compared to the original reference pixel coordinates. Points that do not meet the threshold conditions are filtered based on the spatial geometric errors and reprojection errors, thus removing abnormal pixels from the first depth image.

[0061] S103. Calculate the first loss value of the depth-filtered image and the second depth image corresponding to each scene image in each group of scene images. The second depth image is obtained through the depth image reconstruction model.

[0062] In this embodiment, each set of scene images is input into the depth image reconstruction model to obtain the second depth image of the reference image.

[0063] Specifically, calculating the first loss value of the depth-filtered image and the second depth image corresponding to each scene image in each group of scene images includes:

[0064] S1031. For each group of scene images, each scene image in the group of scene images is input into the depth image reconstruction model to obtain the second depth image corresponding to each scene image in the group of scene images.

[0065] For example, in this embodiment, a set of scene images includes three scene images, namely scene image A, scene image B, and scene image C. The set of scene images (with scene image A, scene image B, and scene image C as reference images in the set of scene images) is input into the depth image reconstruction model to obtain the second depth images corresponding to the three scene images (with scene image A, scene image B, and scene image C as reference images).

[0066] S1032. For each group of scene images, calculate the first loss value between the depth-filtered image and the second depth image corresponding to the same scene image.

[0067] It should be noted that the first loss value calculated in this embodiment is the difference between the second depth image and the depth-filtered image corresponding to the same scene image in the same set of scene images. For example, if scene image A in a set of scene images is a reference image, then the second depth image corresponding to scene image A obtained by the depth image reconstruction model and the depth image corresponding to scene image A obtained by the trained teacher model are obtained. Then, abnormal pixels in the depth image are filtered to obtain the depth-filtered image. Finally, the first loss value between the second depth image and the depth-filtered image corresponding to scene image A is calculated.

[0068] S104. Update the depth image reconstruction model based on the first loss value.

[0069] Specifically, after repeatedly updating the depth image reconstruction model, this application can verify the accuracy of the depth image reconstruction model. If the accuracy does not meet the requirements, it continues to execute S1031 to S1032, that is, continuously updating the depth image reconstruction model until the accuracy of the depth image reconstruction model meets the requirements, at which point the updating of the depth image reconstruction model is stopped, and a trained depth image reconstruction model is obtained.

[0070] like Figure 2 The diagram illustrates the training process of the depth image reconstruction model. The upper part, the Teacher model, represents the teacher model in this embodiment, while the lower part, the Student model, represents the depth image reconstruction model in this embodiment. This application first generates a pseudo-probability distribution for the teacher model through probabilistic encoding. Then, by forcing the prediction probability distribution of the depth image reconstruction model to be similar to the pseudo-probability distribution, probabilistic knowledge can be transferred to the student model. Specifically, the pseudo-probability distribution contains the teacher model's prediction information for each pixel in the scene image. This prediction information consists of the depth value and probability value of the corresponding pixel, i.e., the prediction result made by the teacher model based on its own knowledge (the predicted first depth image). By using a loss function to force the depth image reconstruction model to have the same probability prediction at the same pixel as the teacher model, it is equivalent to transferring the teacher model's knowledge to the depth image reconstruction model.

[0071] In one embodiment of the present invention, the method further includes: acquiring a group of scene images to be reconstructed, the group of scene images to be reconstructed including a reference image and an auxiliary image angularly adjacent to the reference image; inputting the group of scene images to be reconstructed into a trained depth image reconstruction model to obtain a depth image corresponding to the reference image in the scene images to be reconstructed. That is, this embodiment obtains a depth image through a trained depth image reconstruction model. Because the high-depth image reconstruction model in this embodiment has high accuracy and speed, the depth image obtained through the high-depth image reconstruction model has high efficiency and accuracy.

[0072] This application provides a training method for a depth image reconstruction model. Compared to the existing MVS model, which requires a dataset of depth values ​​and the creation of optical flow pseudo-labels, this application does not require a dataset of depth values ​​or the creation of optical flow pseudo-labels when training the depth image reconstruction model. That is, after obtaining the first depth images corresponding to each scene image through the trained teacher model, this application filters out abnormal pixels in the same group of depth images to ensure the accuracy of the depth images. Then, it calculates the first loss value of the second depth image obtained by the depth image reconstruction model and the corresponding depth-filtered image, and updates the depth image reconstruction model based on the first loss value. Thus, this invention can improve the accuracy and speed of training the depth image reconstruction model, thereby improving the accuracy of the depth images predicted by the depth image reconstruction model.

[0073] Reference Figure 3 The diagram shown is a flowchart of a teacher model provided in an embodiment of this application. The specific execution process of the teacher model training process is as follows:

[0074] S301. Input a set of scene images into the teacher model to obtain the first depth image of the reference image, as well as the image features corresponding to the reference image and the auxiliary image respectively;

[0075] Each set of scene images includes a reference image and an auxiliary image that is angularly adjacent to the reference image.

[0076] S302. Using the first depth image of the reference image, perform a differentiable transformation on the auxiliary image to obtain a reconstructed auxiliary image, and perform a differentiable transformation on the image features of the auxiliary image to obtain the reconstructed image features.

[0077] S303. Calculate a second loss value based on the reconstructed auxiliary image, the features of the reconstructed image, the reference image, and the image features of the reference image.

[0078] Specifically, calculating the second loss value based on the reconstructed auxiliary image, the reconstructed image features, the reference image, and the image features of the reference image includes:

[0079] S3031. Calculate the image loss value of each reconstructed auxiliary image compared to the reference image, and calculate the feature loss value of each reconstructed image feature compared to the image feature of the reference image;

[0080] S3032. Calculate the second loss value based on all the image loss values ​​and the feature loss values.

[0081] Specifically, the second loss value is calculated based on all the image loss values ​​and the feature loss values, including: adding all the image loss values ​​and the feature loss values ​​and averaging them to obtain the second loss value; or averaging all the image loss values ​​and the feature loss values ​​using weighted averages to obtain the second loss value.

[0082] For example, a set of scene images includes scene image A, scene image B, and scene image C. Scene image A is used as a reference image, and scene images B and C are used as auxiliary images. These are input into the teacher model to obtain the first depth image of reference image A, and the image features corresponding to scene images A, B, and C, respectively. Based on the first depth image of reference image A, differentiable transformations are performed on the images and image features of scene images B and C to obtain the reconstruction auxiliary images and reconstruction image features corresponding to scene images B and C, respectively. The image loss values ​​of the depth image of scene image A with the reconstruction auxiliary images of scene images B and C, respectively, and the feature loss values ​​of the image features of scene image A with the reconstruction image features of scene images B and C, respectively, are calculated. Finally, all the image loss values ​​and feature loss values ​​are summed and averaged to obtain the second loss value.

[0083] For example, a set of scene images includes scene image E, scene image F, and scene image G. Using scene image E as a reference image and scene images F and G as auxiliary images, the model is input to obtain the first depth image of scene image E, and the corresponding image features of scene images E, F, and G. Based on the depth image of scene image E, differentiable transformations are performed on the images and image features of scene images F and G to obtain the corresponding reconstruction auxiliary images and reconstruction image features of scene images F and G, respectively. The image loss values ​​of the depth image of scene image E with the reconstruction auxiliary images of scene images F and G, and the feature loss values ​​of the image features of scene image E with the reconstruction image features of scene images F and G are calculated. Finally, a weighted average of all the image loss values ​​and feature loss values ​​is taken to obtain the second loss value. If the weight corresponding to the image loss value is 0.45 and the weight corresponding to the feature loss value is 0.55, then the second loss value = (the sum of all image loss values ​​multiplied by 0.45 + the sum of all image feature loss values ​​multiplied by 0.45) / 3.

[0084] S304. Update the teacher model according to the second loss value until the accuracy of the teacher model reaches a preset value, and use the last updated teacher model as the trained teacher model.

[0085] The preset value can be set according to actual needs, such as 90%, 95% or 98%, etc. This embodiment does not make a specific limitation on this.

[0086] like Figure 4 The diagram shown illustrates the structure of the teacher model provided in this embodiment, where the NVS model is the depth image reconstruction model in this embodiment. Specifically, given the camera intrinsics and pose, KD-MVS transforms the RGB (luminance information) and features of the auxiliary image onto the reference image based on the depth image predicted by the MVS network (teacher model) to reconstruct the image and features. The depth image predicted by the MVS network is constrained to match the result obtained through the reconstruction (i.e., differentiable warping) with the image and features of the actual reference image. Specifically, a second loss value is calculated based on the reconstructed auxiliary image, the reconstructed image features, the reference image, and the image features of the reference image. The teacher model is then updated based on this second loss value, ensuring that the depth image obtained by the teacher model meets the expected results. This achieves the goal of training the MVS network (depth image reconstruction model) based on the depth image obtained by the teacher model. The reference image is the actual captured image, and the image features under the reference image are features extracted by the MVS network itself.

[0087] Based on the same inventive concept, this application also provides a training device for a depth image reconstruction model corresponding to the training method of the depth image reconstruction model. Since the principle of the device in this application is similar to the training method of the depth image reconstruction model described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0088] Reference Figure 5 The diagram shown is a schematic of a training apparatus for a depth image reconstruction model provided in an embodiment of this application. The training apparatus for the depth image reconstruction model includes:

[0089] The acquisition module 51 is used to obtain the first depth image corresponding to each scene image in each group of scene images through the trained teacher model; the first depth image includes the depth value and probability value of each pixel.

[0090] The filtering module 52 is used to filter abnormal pixels in the first depth images in the same group according to the spatial geometric projection error and the reprojection error, so as to obtain the depth filtered images corresponding to each first depth image.

[0091] The calculation module 53 is used to calculate the first loss value of the depth-filtered image and the second depth image corresponding to each scene image in each group of scene images, wherein the second depth image is obtained by the depth image reconstruction model;

[0092] The update module 54 is used to update the depth image reconstruction model based on the first loss value.

[0093] In one possible implementation, each set of scene images includes a reference image and an auxiliary image taken from an adjacent viewpoint to the reference image, and the device further includes:

[0094] The acquisition module 51 is used to input a set of scene images into the teacher model to obtain the first depth image of the reference image, as well as the image features corresponding to the reference image and the auxiliary image respectively;

[0095] Reconstruction module 55 is used to perform differentiable transformations on the auxiliary image using the first depth image of the reference image to obtain a reconstructed auxiliary image, and to perform differentiable transformations on the image features of the auxiliary image to obtain reconstructed image features.

[0096] Calculation module 53 is used to calculate a second loss value based on the reconstruction auxiliary image, the features of the reconstructed image, the reference image, and the image features of the reference image;

[0097] The update module 54 is used to update the teacher model according to the second loss value until the accuracy of the teacher model reaches a preset value, and the last updated teacher model is used as the trained teacher model.

[0098] In one possible implementation, the computing module 53 is specifically used for:

[0099] Calculate the image loss value of each reconstructed auxiliary image compared to the reference image, and calculate the feature loss value of each reconstructed image feature compared to the image feature of the reference image;

[0100] The second loss value is obtained by calculating based on all the image loss values ​​and the feature loss values.

[0101] In one possible implementation, the acquisition module 51 is specifically used for:

[0102] Each scene image in each group of scene images is used as a reference image, and the remaining scene images are used as auxiliary images. These are then input into the trained teacher model to obtain the first depth image corresponding to each scene image in each group of scene images.

[0103] In one possible implementation, the computing module 53 is specifically used for:

[0104] For each set of scene images, each scene image in the set is input into the depth image reconstruction model to obtain the second depth image corresponding to each scene image in the set.

[0105] For each set of scene images, calculate the first loss value between the depth-filtered image and the second depth image corresponding to the same scene image.

[0106] In one possible implementation, the teacher model is a multi-view stereo vision network (MVS) model.

[0107] In one possible implementation, the acquisition module 51 is further configured to acquire a group of scene images to be reconstructed, the group of scene images to be reconstructed including a reference image and an auxiliary image adjacent to the shooting angle of the reference image; and input the group of scene images to be reconstructed into a trained depth image reconstruction model to obtain a depth image corresponding to the reference image in the scene images to be reconstructed.

[0108] like Figure 6 As shown in the embodiment of this application, an electronic device 130 includes a processor 131, a memory 132, and a bus. The memory 132 stores machine-readable instructions that can be executed by the processor 131. When the electronic device is running, the processor 131 communicates with the memory 132 through the bus. The processor 131 executes the machine-readable instructions to perform the steps of the training method of the depth image reconstruction model described above.

[0109] Specifically, the memory 132 and processor 131 can be general-purpose memory and processor, without any specific limitations. When the processor 131 runs the computer program stored in the memory 132, it can execute the training method of the depth image reconstruction model.

[0110] Corresponding to the training method of the depth image reconstruction model described above, this application embodiment also provides a computer-readable storage medium storing a computer program, which, when run by a processor, executes the steps of the training method of the depth image reconstruction model described above.

[0111] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0112] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0113] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0114] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the training method for the depth image reconstruction model described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0115] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A training method for a depth image reconstruction model, characterized in that, The method includes: The trained teacher model is used to obtain the first depth image corresponding to each scene image in each group of scene images; the first depth image includes the depth value and probability value of each pixel; the probability value is used to represent the probability of the predicted depth value of each pixel in the first depth image obtained by the teacher model. Based on spatial geometric projection error and reprojection error, abnormal pixels in the first depth image within the same group are filtered to obtain the depth-filtered image corresponding to each first depth image. Calculate the first loss value of the depth-filtered image and the second depth image corresponding to each scene image in each group of scene images, where the second depth image is obtained through the depth image reconstruction model; The depth image reconstruction model is updated based on the first loss value.

2. The method according to claim 1, characterized in that, Each set of scene images includes a reference image and an auxiliary image taken from an adjacent viewpoint. The training process of the teacher model is as follows: A set of scene images is input into the teacher model to obtain the first depth image of the reference image, as well as the image features corresponding to the reference image and the auxiliary image respectively; Using the first depth image of the reference image, a differentiable transformation is performed on the auxiliary image to obtain a reconstructed auxiliary image, and the image features of the auxiliary image are then differentiable to obtain the reconstructed image features. A second loss value is calculated based on the reconstructed auxiliary image, the features of the reconstructed image, the reference image, and the image features of the reference image; The teacher model is updated according to the second loss value until the accuracy of the teacher model reaches a preset value, and the last updated teacher model is used as the trained teacher model.

3. The method according to claim 2, characterized in that, The step of calculating the second loss value based on the reconstructed auxiliary image, the features of the reconstructed image, the reference image, and the image features of the reference image includes: Calculate the image loss value of each reconstructed auxiliary image compared to the reference image, and calculate the feature loss value of each reconstructed image feature compared to the image feature of the reference image; The second loss value is obtained by calculating based on all the image loss values ​​and the feature loss values.

4. The method according to claim 2, characterized in that, The process of obtaining the first depth image corresponding to each scene image in each group of scene images through the trained teacher model includes: Each scene image in each group of scene images is used as a reference image, and the remaining scene images are used as auxiliary images. These are then input into the trained teacher model to obtain the first depth image corresponding to each scene image in each group of scene images.

5. The method according to claim 1 or 2, characterized in that, The calculation of the first loss value of the depth-filtered image and the second depth image corresponding to each scene image in each group of scene images includes: For each set of scene images, each scene image in the set is input into the depth image reconstruction model to obtain the second depth image corresponding to each scene image in the set. For each set of scene images, calculate the first loss value between the depth-filtered image and the second depth image corresponding to the same scene image.

6. The method according to claim 1, characterized in that, The teacher model is a multi-view stereo vision network (MVS) model.

7. The method according to any one of claims 1-4, characterized in that, The method further includes: A set of images of the scene to be reconstructed is obtained, the set of images of the scene to be reconstructed includes a reference image and an auxiliary image whose shooting angle is adjacent to that of the reference image; The group of scene images to be reconstructed is input into a trained depth image reconstruction model to obtain a depth image corresponding to the reference image in the scene images to be reconstructed.

8. A training device for a depth image reconstruction model, characterized in that, The device includes: The acquisition module is used to obtain the first depth image corresponding to each scene image in each group of scene images through the trained teacher model; the first depth image includes the depth value and probability value of each pixel; the probability value is used to represent the probability of the predicted depth value of each pixel in the first depth image obtained by the teacher model; The filtering module is used to filter out abnormal pixels in the first depth images within the same group based on spatial geometric projection error and reprojection error, so as to obtain the depth-filtered images corresponding to each first depth image. The calculation module is used to calculate the first loss value of the depth-filtered image and the second depth image corresponding to each scene image in each group of scene images, wherein the second depth image is obtained through the depth image reconstruction model; An update module is used to update the depth image reconstruction model based on the first loss value.

9. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is in operation, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the training method for the depth image reconstruction model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the training method for the depth image reconstruction model as described in any one of claims 1 to 7.

Images