Absolute depth map generation method and device, equipment and storage medium
By iteratively training the depth estimation model with public and private data sets, the accuracy and robustness of monocular depth estimation technology in dealing with complex scenarios and dynamic objects is solved, achieving higher generalization and robustness.
Patent Information
- Application Number
- CN202311168548.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-11
- Publication Date
- 2025-03-11
AI Technical Summary
Existing deep learning-based monocular depth estimation technology has accuracy and robustness problems when dealing with long-distance objects, lack of texture areas, complex scenes, dynamic objects and pavement estimation.
By using the public and private data sets as training sample sets, the initial model is iteratively trained to obtain the depth estimation model, improving the generalization and robustness of the model, thereby generating an accurate absolute depth map.
It significantly improves the generalization and robustness of the depth estimation model, can handle complex scenes and dynamic objects more accurately, and improves the accuracy of road surface estimation.
Smart Images
Figure CN119600125A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to the field of computer vision technology, and in particular to a method, apparatus, device and storage medium for generating an absolute depth map. Background Art
[0002] Monocular depth estimation based on deep learning is an important research direction in the field of computer vision. It uses deep learning technology to estimate the depth value of each pixel from a single RGB image, which can provide support for a variety of tasks, such as augmented reality and autonomous driving. At present, monocular depth estimation algorithms based on deep learning are mainly divided into two types according to the prediction method of depth map: regression-based and classification-based. The regression-based method directly predicts the depth value of each pixel and can generate a continuous depth map. The classification-based method divides the depth value into several discrete categories, predicts the depth segment to which each pixel belongs through a classifier, and then generates a depth map.
[0003] Monocular depth estimation technology based on deep learning has made important progress, but there are still many challenges and difficulties. The existing monocular depth estimation technology based on deep learning has the following main problems: for distant objects in the scene or areas without sufficient texture, the predicted depth value may be relatively inaccurate; the model requires a large amount of labeled data for training, but the cost of annotating depth values is high, and the quality and scale of the labeled data set are limited; in some complex scenes, such as occluded or reflected objects, the performance of the model may be affected; existing methods have difficulty in handling dynamic objects, and monocular depth estimation technology based on deep learning is difficult to capture dynamically changing objects; road surface estimation is difficult, especially in complex scenes where multiple planes intersect, such as road intersections, it is difficult for the model to extract a single depth value only through RGB images; the robustness and generalization ability of the model need to be improved, and the performance on heterogeneous data sets may not be as good as other visual tasks such as object detection or classification.
[0004] Therefore, how to improve the accuracy and robustness of the model and how to deal with the problem of depth estimation in specific scenarios are important issues that need to be solved in this field. Summary of the invention
[0005] The embodiments of the present invention provide an absolute depth map generation method, apparatus, device and storage medium, so as to make full use of the characteristics of large scale, strong sample diversity and wide coverage of public data sets, and iteratively train the initial model as a training sample set to obtain a depth estimation model, thereby greatly improving the generalization and robustness of the depth estimation model, and then accurately obtaining the absolute depth map of the image through the depth estimation model.
[0006] According to one aspect of the present invention, there is provided a method for generating an absolute depth map, comprising:
[0007] Get the initial image;
[0008] The initial image is input into a depth estimation model to obtain an absolute depth map corresponding to the initial image, wherein the depth estimation model is obtained by iteratively training an initial model with a training sample set, and the training sample set includes: a public data set and a private data set.
[0009] According to another aspect of the present invention, there is provided an absolute depth map generating device, the device comprising:
[0010] An acquisition module, used for acquiring an initial image;
[0011] An input module is used to input the initial image into a depth estimation model to obtain an absolute depth map corresponding to the initial image, wherein the depth estimation model is obtained by iteratively training an initial model with a training sample set, and the training sample set includes: a public data set and a private data set.
[0012] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:
[0013] at least one processor; and
[0014] a memory communicatively connected to the at least one processor; wherein,
[0015] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the absolute depth map generation method described in any embodiment of the present invention.
[0016] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the absolute depth map generation method described in any embodiment of the present invention when executed.
[0017] The embodiment of the present invention uses the public data set and the private data set as training sample sets, iteratively trains the initial model to obtain a depth estimation model, then obtains the initial image, inputs the initial image into the depth estimation model, and finally obtains the absolute depth map corresponding to the initial image. Through the technical solution of the present invention, it is possible to fully utilize the characteristics of the public data set, such as large scale, strong sample diversity, and wide coverage, and iteratively train the initial model as a training sample set to obtain a depth estimation model, which greatly improves the generalization and robustness of the depth estimation model, and then accurately obtains the absolute depth map of the image through the depth estimation model.
[0018] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0020] Figure 1 is a flow chart of a method for generating an absolute depth map in an embodiment of the present invention;
[0021] Figure 2 is a schematic diagram of a method for iteratively training an initial model with a training sample set to obtain a depth estimation model in an embodiment of the present invention;
[0022] Figure 3 is a schematic structural diagram of an absolute depth map generating device in an embodiment of the present invention;
[0023] Figure 4 It is a structural schematic diagram of an electronic device for implementing the absolute depth map generation method according to an embodiment of the present invention. DETAILED DESCRIPTION
[0024] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0026] Embodiment 1
[0027] Figure 1 is a flow chart of a method for generating an absolute depth map in an embodiment of the present invention. This embodiment is applicable to the case of generating an absolute depth map. The method can be executed by an absolute depth map generating device in an embodiment of the present invention. The device can be implemented in software and / or hardware, such as Figure 1 As shown, the method specifically comprises the following steps:
[0028] S101, obtaining an initial image.
[0029] In this embodiment, the initial image may be any image. For example, the initial image may be an image taken by a camera.
[0030] S102: Input the initial image into a depth estimation model to obtain an absolute depth map corresponding to the initial image.
[0031] In this embodiment, the depth estimation model can be a model for estimating the depth value of each pixel in the initial image, and then generating an absolute depth map according to the depth value of each pixel. It can be known that the absolute depth of the image can be represented by an RGB color picture, and the value of the pixel represents the real distance from the camera. Among them, the absolute depth map corresponding to the initial image can be a map that can represent the real distance of each pixel in the initial image from the camera.
[0032] Among them, the depth estimation model is obtained by iteratively training the initial model through the training sample set. It should be noted that the training sample set can be a sample set used to train the initial model, and the initial model can be an untrained neural network model. Specifically, the training sample set includes: a public data set and a private data set. It should be explained that a public data set can be a data set composed of any scaled or unscaled depth map. In this embodiment, the public data set can be represented as dataset_public; a private data set can be a data set that only contains scaled depth maps. In this embodiment, the public data set can be represented as dataset_private.
[0033] Specifically, the public dataset and the private dataset are used as training sample sets, the initial model is iteratively trained to obtain a trained depth estimation model, and then the acquired initial image is input into the trained depth estimation model to obtain an absolute depth map corresponding to the initial image.
[0034] The embodiment of the present invention uses the public data set and the private data set as training sample sets, iteratively trains the initial model to obtain a depth estimation model, then obtains the initial image, inputs the initial image into the depth estimation model, and finally obtains the absolute depth map corresponding to the initial image. Through the technical solution of the present invention, it is possible to fully utilize the characteristics of the public data set, such as large scale, strong sample diversity, and wide coverage, and iteratively train the initial model as a training sample set to obtain a depth estimation model, which greatly improves the generalization and robustness of the depth estimation model, and then accurately obtains the absolute depth map of the image through the depth estimation model.
[0035] Optionally, the depth estimation model includes: a target encoder, a target decoder, and a target global scale network.
[0036] Among them, the target encoder can be an encoder in a trained depth estimation model, which can be used to extract image features; the target decoder can be an encoder in a trained depth estimation model, which can be used to generate a scale-free depth map corresponding to the image; the target global scale network can be a global scale network in a trained depth estimation model, which can be used to estimate global scale information.
[0037] Input the initial image into the depth estimation model to obtain the absolute depth map corresponding to the initial image, including:
[0038] The initial image is input into the target encoder to obtain the image features of the initial image.
[0039] In this embodiment, the target encoder can be a deformable convolutional neural network. The initial image is input into the deformable convolutional neural network to obtain the image feature FI of the initial image. s, where s is the image feature of different scales. In this embodiment, s∈[0,3].
[0040] The image features of the initial image are input into the target decoder to obtain a relative depth map corresponding to the initial image.
[0041] It can be known that the relative depth of an image can be represented by a grayscale image, which represents the relative distance between pixels, not the actual depth value.
[0042] In this embodiment, the target decoder can be designed as an upsampling method, that is, as the number of layers increases, the scale of the feature map gradually increases, and the target decoder includes multi-scale feature maps from the feature extraction network, which are respectively connected in series at the channel level with the feature maps of the same scale of the target decoder. Specifically, inputting the image features of the initial image into the target decoder can generate a relative depth map D′ corresponding to the initial image.
[0043] The image features of the initial image are input into the target global scale network to obtain the global scale factor corresponding to the initial image.
[0044] In this embodiment, the global scale factor may include a global scale amount and a global offset amount.
[0045] In this embodiment, the target global scale network can be composed of three layers of fully connected layers or convolutional layers, and the input is the image feature FI of the initial image provided by the target encoder s , the output is the global scale scale and the global offset shift.
[0046] An absolute depth map corresponding to the initial image is generated according to the global scale factor corresponding to the initial image and the relative depth map corresponding to the initial image.
[0047] Specifically, D can be used to represent the absolute depth map corresponding to the initial image. The calculation method of the absolute depth map D corresponding to the initial image can be expressed as: D=scale*D′+shift, wherein scale represents the global scale included in the global scale factor, shift represents the global offset included in the global scale factor, and D′ represents the relative depth map corresponding to the initial image.
[0048] Optionally, the public data set includes: image samples and depth maps corresponding to the image samples. The image samples include: image samples of the first type and / or image samples of the second type. Among them, the image samples of the first type can be any depth map without scale, and the image samples of the second type can be any depth map with scale. If the image sample is an image sample of the first type, the depth map corresponding to the image sample is the relative depth map corresponding to the image sample of the first type; if the image sample is an image sample of the second type, the depth map corresponding to the image sample is the absolute depth map corresponding to the image sample of the second type. The private data set includes: image samples of the second type and the absolute depth map corresponding to the image samples of the second type.
[0049] Optionally, iteratively train the initial model using the training sample set, including:
[0050] Build an initial model.
[0051] The initial model is iteratively trained through the public data set to obtain the first weight parameter.
[0052] The first weight parameter may be a weight obtained by iteratively training the initial model using a public data set.
[0053] First, the initial model is trained using a public dataset, also known as scale-free pre-training. Specifically, SSIL (Scale-and Shift-Invariant Losses), VNL (Virtual Normal Losses), and DGL (Deep Gradient Losses) can be used for joint supervision according to certain weights, and the learning rate can be set to 2e -4 , after training, the first weight parameter is obtained. The overall loss function of this stage is: loss1 = SSIL + 10 * VNL + 0.1 * DGL. The first stage of training is completed. The purpose of the first stage of training is to allow the initial model to learn features with strong robustness and generalization from a large-scale public data set.
[0054] A first model is created according to the first weight parameter and the initial model.
[0055] It should be explained that the first model may be a model obtained by initializing the initial model with the first weight parameter.
[0056] Specifically, the initial model is initialized using the first weight parameter to obtain the first model.
[0057] Obtain a target relative depth map corresponding to the second type of image samples in the training sample set.
[0058] The target relative depth map may be a relative depth map corresponding to a scaled depth map image sample in a training sample set.
[0059] Specifically, a separate large depth estimation model can be used to predict the relative depth of the second type of image samples in the training sample set to obtain a target relative depth map corresponding to the second type of image samples in the training sample set. This embodiment does not limit the separate large depth estimation model.
[0060] A target sample set is generated according to the training sample set and the target relative depth map corresponding to the second type of image samples in the training sample set.
[0061] The target sample set may be a sample set consisting of a training sample set and a target relative depth map corresponding to image samples of the second type in the training sample set, and may be used to train the first model.
[0062] Specifically, the training sample set and the target relative depth map corresponding to the second type of image samples in the training sample set form a target sample set.
[0063] The first model is iteratively trained according to the target sample set to obtain a second weight parameter.
[0064] The second weight parameter may be a weight obtained by iteratively training the first model using a target sample set.
[0065] Secondly, the first model is trained in the second stage using public and private datasets, also known as scale-free joint training.
[0066] In actual operation, before iteratively training the first model with the public data set and the private data set, the public data set and the private data set may be selected in a balanced manner. Exemplarily, the probability of data selection may be determined based on the inverse of the sample size of the public data set and the private data set, that is, 1 / N1 and 1 / N2 are probability factors, where N1 and N2 are the sample sizes of the private data set and the public data set, respectively. It should be noted that this data selection method is not unique and can be set according to actual conditions.
[0067] Specifically, the same loss function as the first stage training can be used, namely SSIL, VNL and DGL, and the learning rate can be set to 1e -4 , it is sufficient to be 1 / 10 of the first training stage, and the second weight parameter is obtained after training. The overall loss function of this stage is: loss2 = SSIL + 10*VNL + 0.1*DGL, and the second stage of training is completed. The purpose of the second stage of training is to better adapt to the domain of private data while maintaining strong robust features.
[0068] A second model is created based on the second weight parameters and the initial model.
[0069] It should be explained that the second model may be a model obtained by initializing the initial model with the second weight parameter.
[0070] Specifically, the second weight parameter is used to initialize the initial model to obtain the second model.
[0071] The second model is iteratively trained according to the private dataset to obtain a depth estimation model.
[0072] Finally, the private dataset is used to train the second model in the third stage, also known as scale-only branch training, in which a network branch is used to learn the scale information in the private dataset. Specifically, the private dataset is used to iteratively train the second model to obtain a trained depth estimation model.
[0073] Optionally, the initial model is iteratively trained using a public data set to obtain a first weight parameter, including:
[0074] The image samples in the public dataset are input into the initial model to obtain the first predicted relative depth map.
[0075] The first predicted relative depth map may be a relative depth map corresponding to an image sample in a public data set predicted by an initial model.
[0076] Specifically, the image samples in the public data set are input into the initial model for prediction, and a first predicted relative depth map corresponding to the image samples in the public data set is output.
[0077] The parameters of the initial model are trained according to a first function formed by the first predicted relative depth map and the depth map corresponding to the image sample.
[0078] It should be noted that the first function may be a loss function formed by the first predicted relative depth map and the depth map corresponding to the image sample. The first function includes: at least one of a loss function SSIL, a loss function VNL and a loss function DGL.
[0079] In this embodiment, the first function can be expressed as loss1, and the first function loss1 can be expressed as: first function loss1=loss function SSIL+10*loss function VNL+0.1*loss function DGL, wherein the weight corresponding to the loss function SSIL is 1, the weight corresponding to the loss function VNL is 10, and the weight corresponding to the loss function DGL is 0.1. The parameters of the initial model are trained according to the first function loss1.
[0080] Return to the operation of inputting the image samples in the public dataset into the initial model to obtain the first predicted relative depth map until the first weight parameter is obtained.
[0081] Optionally, the target sample set includes: image samples of the first type in the training sample set, relative depth maps corresponding to the image samples of the first type, image samples of the second type, and target relative depth maps corresponding to the image samples of the second type.
[0082] Before generating the target sample set according to the training sample set and the target relative depth map corresponding to the second type of image samples in the training sample set, the method further includes:
[0083] The second type of image samples in the training sample set are input into the target model to obtain a target relative depth map corresponding to the second type of image samples in the training sample set.
[0084] In this embodiment, the target model may be a separate large depth estimation model, which may be used to predict relative depth, and this embodiment does not limit this.
[0085] Specifically, a separate large depth estimation model is used to predict the relative depth of the second type of image samples in the training sample set to obtain a target relative depth map corresponding to the second type of image samples in the training sample set, thereby using the target relative depth map to perform scale-free supervision on the second type of image samples.
[0086] Optionally, iteratively training the first model according to the target sample set to obtain the second weight parameter includes:
[0087] The image samples in the target sample set are input into the first model to obtain a second predicted relative depth map.
[0088] The second predicted relative depth map may be a relative depth map corresponding to image samples in the target sample set predicted by the first model.
[0089] Specifically, the image samples in the target sample set are input into the first model for prediction, and a second predicted relative depth map corresponding to the image samples in the target sample set is output.
[0090] The parameters of the first model are trained according to a second function formed by the second predicted relative depth map and the relative depth map corresponding to the image sample, and / or the parameters of the first model are trained according to a second function formed by the second predicted relative depth map and the target relative depth map.
[0091] It should be noted that the second function may be a loss function formed by the second predicted relative depth map and the relative depth map corresponding to the image sample, and / or a loss function formed by the second predicted relative depth map and the target relative depth map. The second function includes: at least one of a loss function SSIL, a loss function VNL, and a loss function DGL.
[0092] In this embodiment, the second function can be expressed as loss2, and the second function loss2 can be expressed as: second function loss2=loss function SSIL+10*loss function VNL+0.1*loss function DGL, wherein the weight corresponding to the loss function SSIL is 1, the weight corresponding to the loss function VNL is 10, and the weight corresponding to the loss function DGL is 0.1. The parameters of the first model are trained according to the second function loss2.
[0093] Return to the operation of inputting the image samples in the training sample set into the first model to obtain the second predicted relative depth map until the second weight parameter is obtained.
[0094] Optionally, iteratively training a second model according to a private data set to obtain a depth estimation model includes:
[0095] The second type of image samples in the private dataset are input into the second model to obtain a predicted absolute depth map.
[0096] The predicted absolute depth map may be an absolute depth map corresponding to the second type of image samples in the private dataset predicted by the second model.
[0097] Specifically, the image samples of the second type in the private data set are input into the second model for prediction, and the predicted absolute depth map corresponding to the image samples of the second type in the private data set is output.
[0098] The parameters of the second model are trained according to a depth difference loss function formed by the predicted absolute depth map and the absolute depth map corresponding to the second type of image sample.
[0099] The depth difference loss function may be a loss function formed by predicting the absolute depth map and the absolute depth map corresponding to the second type of image sample. In this embodiment, the depth difference loss function may be an absolute value loss function or a mean square error loss function.
[0100] Return to the operation of inputting the second type of image samples in the private dataset into the second model to obtain a predicted absolute depth map until a depth estimation model is obtained.
[0101] In this embodiment, the private dataset is used to train the second model in the third stage, which is also called scale-only training. Specifically, the second weight parameter can be used to initialize the network, and then the learnable parameters involved in the encoder and decoder in the network are frozen, that is, no adjustment is made, and only the global scale branch network is trained. The learning rate can be set to 1e -4 , it only needs to be 1 / 10 of that in the first training stage. In this stage, the absolute value loss function or the mean square error loss function can be used. The overall loss function in this stage is: loss3=L1, or loss3=MSE, where L1 represents the absolute value loss function and MSE represents the mean square error loss function.
[0102] Optionally, the second model includes: an initial encoder, an initial decoder, and an initial global scale network.
[0103] Among them, the initial encoder can be an encoder in an untrained second model, which can be used to extract image features; the initial decoder can be an encoder in an untrained second model, which can be used to generate a scale-free depth map corresponding to the image; the initial global scale network can be a global scale network in an untrained second model, which can be used to estimate global scale information.
[0104] Inputting the second type of image samples in the private dataset into the second model to obtain a predicted absolute depth map, including:
[0105] The second type of image samples in the private data set are input into the initial encoder to obtain image features of the second type of image samples.
[0106] In this embodiment, the initial encoder may specifically be a deformable convolutional neural network, and the image features of the second type of image samples in the private data set may be obtained by inputting the second type of image samples into the deformable convolutional neural network.
[0107] The image features of the second type of image samples are input into the initial decoder to obtain a predicted relative depth map corresponding to the second type of image samples.
[0108] The predicted relative depth map may be a relative depth map corresponding to the second type of image samples predicted by the initial decoder.
[0109] In this embodiment, the initial decoder can be designed as an upsampling method, that is, as the number of layers increases, the scale of the feature map gradually increases, and the initial decoder includes multi-scale feature maps from the feature extraction network, which are respectively connected in series at the channel level with the feature maps of the same scale of the initial decoder. Specifically, inputting the image features of the second type of image samples into the initial decoder can generate a predicted relative depth map corresponding to the second type of image samples.
[0110] The image features of the second type of image samples are input into the initial global scale network to obtain the global scale factors corresponding to the second type of image samples.
[0111] In this embodiment, the global scale factor may include a global scale amount and a global offset. In this embodiment, the target global scale network may be composed of 3 layers of fully connected layers or convolutional layers, the input is the image features of the second type of image samples provided by the initial encoder, and the output is the global scale amount and the global offset.
[0112] A predicted absolute depth map corresponding to the image samples of the second type is generated according to the predicted relative depth map corresponding to the image samples of the second type and the global scale factor corresponding to the image samples of the second type.
[0113] Specifically, the calculation method of the predicted absolute depth map corresponding to the second type of image samples can be expressed as: predicted absolute depth map corresponding to the second type of image samples = global offset * predicted relative depth map corresponding to the second type of image samples + global offset.
[0114] As an exemplary description of an embodiment of the present invention, Figure 2 FIG. 1 is a schematic diagram of a method for iteratively training an initial model with a training sample set to obtain a depth estimation model in an embodiment of the present invention. Figure 2 As shown, the training sample set includes a public data set and a private data set, and the initial network can be understood as including an initial encoder and an initial decoder. The process of iteratively training the initial model through the training sample set to obtain the depth estimation model can include three stages. The first training stage is Figure 2 The purpose is to allow the network to learn features with strong robustness and generalization from a large-scale public dataset. The second training phase is Figure 2 The purpose is to better adapt to the field of private data while maintaining strong robust features; the third training stage is Figure 2 The purpose is to use a network branch to specifically learn the scale information in the private dataset.
[0115] The first training stage, also known as scale-free pre-training, is to input image samples in the public dataset into the initial model to obtain a first predicted relative depth map corresponding to the image sample, and train the parameters of the initial model according to a first function formed by the first predicted relative depth map and the depth map corresponding to the image sample, wherein the first function includes at least one of the loss function SSIL, the loss function VNL, and the loss function DGL, and the learning rate is set to 2e -4 After training, the first weight parameter is obtained, and the first model is created according to the first weight parameter and the initial model. The overall loss function of this stage is: loss1 = SSIL + 10*VNL + 0.1*DGL.
[0116] The second training stage, also known as scale-free joint training: the public data set and the private data set are mixed, and the training sample set is extracted based on the data balancing strategy during training. The first model is initialized using the first weight parameter L1, and a separate large depth estimation model is used to obtain the target relative depth map for the image samples of the private data set in the training sample set extracted based on the data balancing strategy, so that the private data is scale-free supervised using the target relative depth map, and the training sample set and the target relative depth map are combined into a target sample set. The image samples in the target sample set are input into the first model to obtain a second predicted relative depth map, and the parameters of the first model are trained according to the second function formed by the second predicted relative depth map and the relative depth map corresponding to the image sample, and / or the parameters of the first model are trained according to the second function formed by the second predicted relative depth map and the target relative depth map, wherein the second function includes: at least one of the loss function SSIL, the loss function VNL and the loss function DGL, and the same loss function as the first training stage is used, and the learning rate is set to 1e -4 , which satisfies 1 / 10 of the first training stage. After training, the second weight parameter is obtained, and the second model is created according to the second weight parameter and the initial model. The overall loss function of this stage is: loss2 = SSIL + 10 * VNL + 0.1 * DGL.
[0117] The third training stage, also called scale-only branch training: only the private dataset is used for training. During this stage of training, the second weight parameter is used to initialize the network, and then the learnable parameters involved in the initial encoder and initial decoder in the network are frozen, that is, no adjustment is made, and only the initial global scale branch network is trained. The image samples in the private dataset are input into the initial encoder to obtain image features, the image features are input into the initial decoder to obtain the predicted relative depth map, the image features are input into the initial global scale network to obtain the global scale factor, and the predicted absolute depth map is generated according to the predicted relative depth map and the global scale factor. The parameters of the second model are trained according to the depth difference loss function formed by the predicted absolute depth map and the absolute depth map corresponding to the image samples in the private dataset, and the learning rate is set to 1e -4 , which is 1 / 10 of the first training stage. In this stage, the absolute value loss function (L1) or the mean square error loss function (MSE) can be used to finally obtain the depth estimation model. The overall loss function in this stage is: loss3 = L1 or loss3 = MSE.
[0118] The network structure and training method proposed in the technical solution of this embodiment make full use of the characteristics of large scale, strong sample diversity and wide coverage of public data sets, and improve the generalization and robustness of the depth estimation model by decoupling the depth estimation problem into scale-free depth and global scale.
[0119] The technical solution of this embodiment is to obtain an image, input the image into the encoder of the depth estimation model to extract image features, and then generate a scale-free depth map through the decoder, and at the same time generate a global scale factor through the global scale branch, and then multiply the scale-free depth map and the scale factor through a multiplication operation to obtain a scaled depth map. The technical solution of this embodiment can solve the problem that the predicted depth value is inaccurate when using deep learning for monocular depth estimation and the model is difficult to extract a single depth value only through RGB images, and can obtain a more accurate scaled depth map.
[0120] Embodiment 2
[0121] Figure 3 is a schematic diagram of the structure of an absolute depth map generation device in an embodiment of the present invention. This embodiment is applicable to the case of absolute depth map generation. The device can be implemented in software and / or hardware. The device can be integrated in any device that provides the function of absolute depth map generation, such as Figure 3 As shown, the absolute depth map generating device specifically includes: an acquisition module 201 and an input module 202.
[0122] Wherein, the acquisition module 201 is used to acquire an initial image;
[0123] The input module 202 is used to input the initial image into the depth estimation model to obtain an absolute depth map corresponding to the initial image, wherein the depth estimation model is obtained by iteratively training the initial model with a training sample set, and the training sample set includes: a public data set and a private data set.
[0124] Optionally, the depth estimation model includes: a target encoder, a target decoder, and a target global scale network;
[0125] The input module 202 includes:
[0126] A first input unit, used to input the initial image into a target encoder to obtain image features of the initial image;
[0127] A second input unit, used to input the image features of the initial image into a target decoder to obtain a relative depth map corresponding to the initial image;
[0128] A third input unit is used to input the image features of the initial image into a target global scale network to obtain a global scale factor corresponding to the initial image;
[0129] The first generating unit is configured to generate an absolute depth map corresponding to the initial image according to a global scale factor corresponding to the initial image and a relative depth map corresponding to the initial image.
[0130] Optionally, the public data set includes: image samples and depth maps corresponding to the image samples, the image samples include: image samples of the first type and / or image samples of the second type, if the image samples are image samples of the first type, then the depth map corresponding to the image samples is a relative depth map corresponding to the image samples of the first type, if the image samples are image samples of the second type, then the depth map corresponding to the image samples is an absolute depth map corresponding to the image samples of the second type, and the private data set includes: image samples of the second type and absolute depth maps corresponding to the image samples of the second type.
[0131] Optionally, the input module 202 includes:
[0132] Establishing unit, used to establish initial model;
[0133] A first iterative training unit, configured to iteratively train the initial model using the public data set to obtain a first weight parameter;
[0134] A first creation unit, configured to create a first model according to the first weight parameter and the initial model;
[0135] An acquisition unit, configured to acquire a target relative depth map corresponding to the image samples of the second type in the training sample set;
[0136] A second generating unit, configured to generate a target sample set according to the training sample set and a target relative depth map corresponding to the image samples of the second type in the training sample set;
[0137] A second iterative training unit, configured to iteratively train the first model according to the target sample set to obtain a second weight parameter;
[0138] A second creation unit, configured to create a second model according to the second weight parameter and the initial model;
[0139] The third iterative training unit is used to iteratively train the second model according to the private data set to obtain a depth estimation model.
[0140] Optionally, the first iterative training unit is specifically used for:
[0141] Inputting image samples in the public dataset into the initial model to obtain a first predicted relative depth map;
[0142] Training the parameters of the initial model according to a first function formed by the first predicted relative depth map and the depth map corresponding to the image sample, wherein the first function includes: at least one of a loss function SSIL, a loss function VNL, and a loss function DGL;
[0143] Return to the operation of inputting the image samples in the public dataset into the initial model to obtain the first predicted relative depth map until the first weight parameter is obtained.
[0144] Optionally, the target sample set includes: image samples of the first type in the training sample set, relative depth maps corresponding to the image samples of the first type, image samples of the second type, and target relative depth maps corresponding to the image samples of the second type;
[0145] The input module 202 further includes:
[0146] The fourth input unit is used to input the second type of image samples in the training sample set into the target model before generating the target sample set based on the training sample set and the target relative depth map corresponding to the second type of image samples in the training sample set, so as to obtain the target relative depth map corresponding to the second type of image samples in the training sample set.
[0147] Optionally, the second iterative training unit is specifically used for:
[0148] Inputting the image samples in the target sample set into the first model to obtain a second predicted relative depth map;
[0149] Training the parameters of the first model according to a second function formed by the second predicted relative depth map and the relative depth map corresponding to the image sample, and / or training the parameters of the first model according to a second function formed by the second predicted relative depth map and the target relative depth map, wherein the second function includes: at least one of a loss function SSIL, a loss function VNL, and a loss function DGL;
[0150] Return to the operation of inputting the image samples in the training sample set into the first model to obtain the second predicted relative depth map until a second weight parameter is obtained.
[0151] Optionally, the third iterative training unit includes:
[0152] An input subunit, configured to input an image sample of a second type in the private dataset into the second model to obtain a predicted absolute depth map;
[0153] A training subunit, configured to train parameters of the second model according to a depth difference loss function formed by predicting the absolute depth map and the absolute depth map corresponding to the second type of image samples;
[0154] The operation subunit is used to return to the operation of inputting the second type of image samples in the private data set into the second model to obtain a predicted absolute depth map until a depth estimation model is obtained.
[0155] Optionally, the second model includes: an initial encoder, an initial decoder, and an initial global scale network;
[0156] The input subunit is specifically used for:
[0157] Inputting the second type of image samples in the private data set into the initial encoder to obtain image features of the second type of image samples;
[0158] Inputting the image features of the second type of image samples into the initial decoder to obtain a predicted relative depth map corresponding to the second type of image samples;
[0159] Inputting the image features of the second type of image samples into the initial global scale network to obtain a global scale factor corresponding to the second type of image samples;
[0160] The predicted absolute depth map corresponding to the image samples of the second type is generated according to the predicted relative depth map corresponding to the image samples of the second type and the global scale factor corresponding to the image samples of the second type.
[0161] The above-mentioned product can execute the absolute depth map generation method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the absolute depth map generation method.
[0162] Embodiment 3
[0163] Figure 4 A schematic diagram of the structure of an electronic device 30 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0164] like Figure 4As shown, the electronic device 30 includes at least one processor 31, and a memory connected to the at least one processor 31, such as a read-only memory (ROM) 32, a random access memory (RAM) 33, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 31 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 32 or the computer program loaded from the storage unit 38 to the random access memory (RAM) 33. In the RAM 33, various programs and data required for the operation of the electronic device 30 can also be stored. The processor 31, the ROM 32, and the RAM 33 are connected to each other through a bus 34. An input / output (I / O) interface 35 is also connected to the bus 34.
[0165] A number of components in the electronic device 30 are connected to the I / O interface 35, including: an input unit 36, such as a keyboard, a mouse, etc.; an output unit 37, such as various types of displays, speakers, etc.; a storage unit 38, such as a disk, an optical disk, etc.; and a communication unit 39, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 39 allows the electronic device 30 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0166] The processor 31 may be a variety of general and / or dedicated processing components with processing and computing capabilities. Some examples of the processor 31 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 31 performs the various methods and processes described above, such as the absolute depth map generation method:
[0167] Get the initial image;
[0168] The initial image is input into a depth estimation model to obtain an absolute depth map corresponding to the initial image, wherein the depth estimation model is obtained by iteratively training an initial model with a training sample set, and the training sample set includes: a public data set and a private data set.
[0169] In some embodiments, the absolute depth map generation method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 38. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 30 via the ROM 32 and / or the communication unit 39. When the computer program is loaded into the RAM 33 and executed by the processor 31, one or more steps of the absolute depth map generation method described above may be performed. Alternatively, in other embodiments, the processor 31 may be configured to perform the absolute depth map generation method in any other appropriate manner (e.g., by means of firmware).
[0170] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0171] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0172] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0173] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0174] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0175] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
[0176] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.
[0177] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for generating an absolute depth map, characterized in that: include: Get the initial image; The initial image is input into a depth estimation model to obtain an absolute depth map corresponding to the initial image, wherein the depth estimation model is obtained by iteratively training an initial model with a training sample set, and the training sample set includes: a public data set and a private data set.
2. The method according to claim 1, characterized in that The depth estimation model includes: a target encoder, a target decoder and a target global scale network; Inputting the initial image into a depth estimation model to obtain an absolute depth map corresponding to the initial image includes: Inputting the initial image into a target encoder to obtain image features of the initial image; Inputting the image features of the initial image into a target decoder to obtain a relative depth map corresponding to the initial image; Inputting the image features of the initial image into a target global scale network to obtain a global scale factor corresponding to the initial image; An absolute depth map corresponding to the initial image is generated according to the global scale factor corresponding to the initial image and the relative depth map corresponding to the initial image.
3. The method according to claim 1, characterized in that The public data set includes: image samples and depth maps corresponding to the image samples, the image samples include: image samples of the first type and / or image samples of the second type, if the image samples are image samples of the first type, then the depth map corresponding to the image samples is the relative depth map corresponding to the image samples of the first type, if the image samples are image samples of the second type, then the depth map corresponding to the image samples is the absolute depth map corresponding to the image samples of the second type, the private data set includes: image samples of the second type and the absolute depth map corresponding to the image samples of the second type.
4. The method according to claim 3, characterized in that The initial model is iteratively trained using the training sample set, including: Build an initial model; Iteratively train the initial model through the public data set to obtain a first weight parameter; Creating a first model according to the first weight parameter and the initial model; Obtain a target relative depth map corresponding to the second type of image samples in the training sample set; Generate a target sample set according to the training sample set and the target relative depth map corresponding to the second type of image samples in the training sample set; Iteratively train the first model according to the target sample set to obtain a second weight parameter; Creating a second model according to the second weight parameter and the initial model; The second model is iteratively trained according to the private data set to obtain a depth estimation model.
5. The method according to claim 4, characterized in that Iteratively training the initial model through the public data set to obtain a first weight parameter includes: Inputting image samples in the public dataset into the initial model to obtain a first predicted relative depth map; Training the parameters of the initial model according to a first function formed by the first predicted relative depth map and the depth map corresponding to the image sample, wherein the first function includes: at least one of a loss function SSIL, a loss function VNL, and a loss function DGL; Return to the operation of inputting the image samples in the public dataset into the initial model to obtain the first predicted relative depth map until the first weight parameter is obtained.
6. The method according to claim 5, characterized in that The target sample set includes: image samples of the first type in the training sample set, relative depth maps corresponding to the image samples of the first type, image samples of the second type, and target relative depth maps corresponding to the image samples of the second type; Before generating a target sample set according to the training sample set and the target relative depth map corresponding to the second type of image samples in the training sample set, the method further includes: The image samples of the second type in the training sample set are input into the target model to obtain a target relative depth map corresponding to the image samples of the second type in the training sample set.
7. The method according to claim 6, characterized in that Iteratively training the first model according to the target sample set to obtain a second weight parameter includes: Inputting the image samples in the target sample set into the first model to obtain a second predicted relative depth map; Training the parameters of the first model according to a second function formed by the second predicted relative depth map and the relative depth map corresponding to the image sample, and / or training the parameters of the first model according to a second function formed by the second predicted relative depth map and the target relative depth map, wherein the second function includes: at least one of a loss function SSIL, a loss function VNL, and a loss function DGL; Return to the operation of inputting the image samples in the training sample set into the first model to obtain the second predicted relative depth map until a second weight parameter is obtained.
8. The method according to claim 7, characterized in that Iteratively training the second model according to the private data set to obtain a depth estimation model includes: Inputting a second type of image sample in the private dataset into the second model to obtain a predicted absolute depth map; Training the parameters of the second model according to a depth difference loss function formed by the predicted absolute depth map and the absolute depth map corresponding to the second type of image sample; Return to the operation of inputting the second type of image samples in the private dataset into the second model to obtain a predicted absolute depth map until a depth estimation model is obtained.
9. The method according to claim 8, characterized in that The second model includes: an initial encoder, an initial decoder, and an initial global scale network; Inputting the second type of image samples in the private dataset into the second model to obtain a predicted absolute depth map, comprising: Inputting the second type of image samples in the private data set into the initial encoder to obtain image features of the second type of image samples; Inputting the image features of the second type of image samples into the initial decoder to obtain a predicted relative depth map corresponding to the second type of image samples; Inputting the image features of the second type of image samples into the initial global scale network to obtain a global scale factor corresponding to the second type of image samples; The predicted absolute depth map corresponding to the image samples of the second type is generated according to the predicted relative depth map corresponding to the image samples of the second type and the global scale factor corresponding to the image samples of the second type.
10. An absolute depth map generating device, characterized in that: include: An acquisition module, used for acquiring an initial image; An input module is used to input the initial image into a depth estimation model to obtain an absolute depth map corresponding to the initial image, wherein the depth estimation model is obtained by iteratively training an initial model with a training sample set, and the training sample set includes: a public data set and a private data set.
11. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so as to enable the at least one processor to perform the absolute depth map generation method according to any one of claims 1 to 9.
12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the absolute depth map generation method according to any one of claims 1 to 9 when executed.