Lifelong learning based monocular depth estimation method and related devices

Abstract

Embodiments of the present application disclose a monocular depth estimation method based on lifelong learning and related equipment. The target monocular depth estimation model is a multi-branch perception framework composed of a domain-shared encoder and a specific domain decoder, which can predict the depth map and uncertainty map of the input image to exclude the performance decline caused by outliers in the depth map taken by the depth sensor during the training process. To further overcome catastrophic forgetting, a regularization constraint is adopted, and knowledge distillation loss and replay loss terms are used to overcome significant domain gaps. The specific domain decoder will learn with data from a new domain; other decoders trained in the old domain will be regularized by depth consistency and uncertainty consistency and replay loss, and the regularization terms complement each other to improve the stability and plasticity of lifelong learning, thereby achieving more accurate monocular depth estimation of images.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of image processing, in particular to a monocular depth estimation method based on lifelong learning and related equipment. BACKGROUND

[0002] In recent years, monocular depth estimation has made significant progress. The ultimate goal is to replace depth sensors and provide a more affordable way of perceiving scene depth. With the continuous progress of its technology, monocular depth estimation can provide a safe and reliable depth estimation algorithm, thereby replacing depth sensors in some fields. Therefore, it has important industrial value and economic benefits.

[0003] However, the current algorithm implementation can only learn a depth convolutional neural network (CNN) in the target environment through a large amount of data-driven training to estimate the absolute depth of the scene. However, due to poor generalization ability for out-of-distribution data, learning-based methods are often criticized and questioned. The recent trend is to solve the problem of poor generality by covering as many possible domains as possible. However, it is impossible to exhaust all possible data patterns in the real world. When there are some new data patterns or target domains, the pre-trained model must be retrained from scratch, resulting in a huge waste of time and cost.

[0004] On the other hand, since there is a big difference between image recognition and monocular depth estimation, how to perform lifelong learning on monocular depth estimation is largely unknown. Most of the existing multi-domain learning methods only infer relative depth maps to solve domain gaps. In addition, only a few studies have tried to develop depth estimation algorithms based on the lifelong learning paradigm, but these methods cannot infer the absolute depth map of the scene due to unsupervised learning. In addition, such methods only work in cases where the domain difference is small and cannot handle cases where the domain difference is large.

[0005] Overall, when performing lifelong learning, the following two major challenges will lead to catastrophic forgetting (i.e., forgetting learned knowledge after updating the trained model in a new domain), one is the significant domain gap, visual images and depth images are significantly different in different domains, so a trained model cannot be transferred between two domains; the second is the depth scale difference, the scene depth scale is usually related to the domain, such as indoor 0-10 meters and outdoor 0-100 meters, so the model cannot be transferred between two domains with different scales. SUMMARY

[0006] Embodiments of the present application provide a monocular depth estimation method based on lifelong learning and related equipment for performing more accurate image monocular depth estimation.

[0007] The first aspect of the embodiment of the present application provides a monocular depth estimation method based on lifelong learning, the method is applied to a computer device, and the method comprises the following steps:

[0008] obtaining a target monocular depth estimation model trained in advance, wherein the target monocular depth estimation model comprises a multi-domain shared encoder and a plurality of specific domain decoders connected with the multi-domain shared encoder;

[0009] obtaining a target image of a target domain, and inputting the target image into the target monocular depth estimation model, so that the target monocular depth estimation model performs the following operations:

[0010] extracting features of the target image by using the multi-domain shared encoder to obtain image features;

[0011] calculating the image features by using a decoder corresponding to the target domain in the plurality of specific domain decoders to obtain a depth estimation map of the target domain;

[0012] outputting the depth estimation map of the target domain.

[0013] The second aspect of the embodiment of the present application provides a computer device, which comprises the following:

[0014] an obtaining unit, configured to obtain a target monocular depth estimation model trained in advance, wherein the target monocular depth estimation model comprises a multi-domain shared encoder and a plurality of specific domain decoders connected with the multi-domain shared encoder;

[0015] a monocular depth estimation unit, configured to obtain a target image of a target domain, and input the target image into the target monocular depth estimation model, so that the target monocular depth estimation model performs the following operations: extracting features of the target image by using the multi-domain shared encoder to obtain image features; and calculating the image features by using a decoder corresponding to the target domain in the plurality of specific domain decoders to obtain a depth estimation map of the target domain;

[0016] an output unit, configured to output the depth estimation map of the target domain.

[0017] The third aspect of the embodiment of the present application provides a computer device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method of the first aspect when executing the computer program.

[0018] The fourth aspect of the embodiment of the present application provides a computer storage medium, which stores instructions, and the instructions make a computer execute the method of the first aspect when executed on the computer.

[0019] From the above technical solution, the embodiments of the present application have the following advantages:

[0020] The target monocular depth estimation model is a multi-branch perception framework composed of a domain-shared encoder and a domain-specific decoder, which can predict the depth map and its uncertainty map of the input image to exclude the performance decline caused by outliers in the depth map taken by the depth sensor during the training process. To further overcome catastrophic forgetting, regularization constraints are adopted, and knowledge distillation loss and replay loss terms are used to overcome significant domain gaps. The domain-specific decoder will be learned with data from the new domain; other decoders trained in the old domain will be regularized by depth consistency and uncertainty consistency and replay loss, and the regularization terms complement each other to improve the stability and plasticity of lifelong learning, thereby achieving more accurate image monocular depth estimation. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 A flowchart of the monocular depth estimation method based on lifelong learning in the embodiments of the present application is shown.

[0022] Figure 2 A model structure diagram of the target monocular depth estimation model in the embodiments of the present application is shown.

[0023] Figure 3 An application scenario diagram of the target monocular depth estimation model in the embodiments of the present application is shown.

[0024] Figure 4 A depth map captured by a sensor in the embodiments of the present application is shown. Figure 1 A display effect diagram is shown.

[0025] Figure 5 A structure diagram of a computer device in the embodiments of the present application is shown.

[0026] Figure 6 Another structure diagram of a computer device in the embodiments of the present application is shown. DETAILED DESCRIPTION

[0027] The embodiments of the present application provide a monocular depth estimation method based on lifelong learning and related devices, which are used to perform more accurate image monocular depth estimation.

[0028] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] The monocular depth estimation method based on lifelong learning in the embodiments of this application is described below:

[0030] Before proceeding with a detailed description of the embodiments of this application, the following concepts will be defined.

[0031] Monocular depth estimation: estimating the distance between an object and the camera from an image captured by a single camera. This technique is commonly used in obstacle avoidance, 3D reconstruction, and autonomous driving.

[0032] Domain: Also known as a field, it refers to a specific data distribution. For example, in monocular depth estimation tasks, the depth range indoors is generally 0 to 10 meters, while some outdoor scenes have a depth range of 0 to 100 meters. Therefore, scenes with different depth ranges belong to different domains.

[0033] Lifelong learning refers to enabling a model to continuously or incrementally learn. That is, after learning in some domains, the model continues to learn in new domains. When learning in new domains, lifelong learning uses little or no data from the old domains. Its purpose is to enhance plasticity (achieving the highest possible accuracy in new domains) and preserve stability (maintaining as much accuracy as possible in older domains).

[0034] Please see Figure 1 One embodiment of the monocular depth estimation method based on lifelong learning in this application includes:

[0035] 101. Obtain a pre-trained target monocular depth estimation model, wherein the target monocular depth estimation model includes a multi-domain shared encoder and multiple domain-specific decoders connected to the multi-domain shared encoder;

[0036] The method of the embodiment can be applied to a computer device, which can be a server, a terminal, or any computer device capable of performing data processing. When the computer device is a terminal, it can be a personal computer (PC), a desktop computer, or any terminal device. When the computer device is a server, it can be a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server providing basic cloud computing services such as cloud database, cloud computing, big data, and artificial intelligence platform.

[0037] When performing monocular depth estimation, the computer device can obtain a pre-trained target monocular depth estimation model, which includes a multi-domain shared encoder and multiple specific domain decoders connected to the multi-domain shared encoder.

[0038] As shown in Figure 2 , the multi-domain shared encoder included in the target monocular depth estimation model is composed of multiple residual convolution modules such as Res1, Res2, Res3, and Res4, and a multi-scale feature fusion module MFF, which together form the multi-domain shared encoder. Wherein, x denotes the output image, g(x) denotes the output of the multi-domain shared encoder, and the three-dimensional cuboid denotes the feature map, whose thickness represents the number of channels, and the other two dimensions represent the length and width of the feature map, i.e., the scale of the feature map.

[0039] And Figure 2 The model structures f1, f2, and f3 on the right are different specific domain decoders, each of which is used to output the depth map and the uncertainty map of the image. For example, in Figure 3 , a multi-domain shared encoder g is used for feature extraction, and multi-branch decoders are used for specific domain depth estimation. The learning of each decoder is to estimate the depth map of a specific domain with a fixed depth range. Figure 3 The learning framework of the visualization model in three different domains is given in . The model starts with a depth decoder f1, which is used for learning in the domain , and dynamically expands its decoders f2 and f3, thereby sequentially learning in and .

[0040] Of course, Figure 2 and Figure 3 only 3 specific domain decoders are used as examples, and in fact, the number of domains corresponding to multiple specific domain decoders is not limited, for example, there can be 4 specific domain decoders, or 5 specific domain decoders, and so on.

[0041] 102. Obtain the target image of the target domain, and input the target image into the target monocular depth estimation model so that the target monocular depth estimation model performs the following operations: use the multi-domain shared encoder to extract features from the target image to obtain image features; use the decoder corresponding to the target domain among the multiple domain-specific decoders to calculate the depth estimation map of the target domain.

[0042] 103. Output the depth estimation map of the target domain;

[0043] Considering real-world applications, the model needs to estimate depth in real time. Therefore, this application aims to construct a lightweight model framework. Figure 2 and Figure 3 As shown, for the encoder, this application employs a ResNet-34-based backbone network to pre-define multi-scale features and uses a multi-feature fusion (MFF) module for feature fusion. The decoder consists of four 5×5 convolutional layers, with two layers (conv_d1 and conv_d2) used for depth estimation and the other two layers (conv_u1 and conv_u2) used for inferring the model's prediction uncertainty. In this framework, the shared encoder has 21.81M parameters; each domain-specific decoder has 0.21M parameters. When learning three domains, this framework boasts over 97% shared parameters to improve computational efficiency.

[0044] After obtaining the target monocular depth estimation model, the computer device can acquire a target image of the target domain and input the target image of the target domain into the target monocular depth estimation model, thereby enabling the target monocular depth estimation model to perform the following operations:

[0045] Image features are obtained by using a multi-domain shared encoder of a target monocular depth estimation model to extract features from the target image;

[0046] The depth estimation map of the target domain is obtained by calculating the image features using the decoder corresponding to the target domain from among the decoders of multiple specific domains of the target monocular depth estimation model.

[0047] Therefore, the computer device can obtain the depth estimation map of the target domain output by the target monocular depth estimation model.

[0048] In the process of training the monocular depth estimation model, in order to achieve better technical performance in monocular depth estimation, this application proposes the following aspects to optimize the monocular depth estimation model, which will be described in detail below:

[0049] First, in-depth acquisition of scenarios based on uncertainty;

[0050] Since one of the goals of lifelong learning is to minimize the storage requirement of the training data of the old target domain, the ability or knowledge acquired by the model in the old domain will continue to degrade when the model learns with new training data in the new target domain. Therefore, during the model training process, two requirements need to be met, one is to acquire the ability of depth estimation in the new target domain, and the other is to retain the ability of depth estimation learned in the old target domain and prevent knowledge forgetting. To this end, the present application proposes an uncertainty-based scene depth acquisition to meet the first aspect of the requirement, and proposes an uncertainty-based consistency constraint and data playback mechanism to meet the second aspect of the requirement.

[0051] Therefore, in a preferred embodiment of the present embodiment, the training step of the target monocular depth estimation model includes:

[0052] Obtaining a plurality of sets of training samples, each set of training samples including image samples of a domain and real domain depth maps corresponding to the image samples;

[0053] Obtaining an initial monocular depth estimation model, the initial monocular depth estimation model including a multi-domain shared encoder, a plurality of specific domain decoders connected to the multi-domain shared encoder, the multi-domain shared encoder including a plurality of residual convolution modules and a multi-scale feature fusion module connected to the plurality of residual convolution modules;

[0054] Inputting the plurality of sets of training samples into the initial monocular depth estimation model, so that the initial monocular depth estimation model performs the following operations:

[0055] Using the plurality of residual convolution modules to extract features of the image samples to obtain a plurality of intermediate features of the image samples at different scales;

[0056] Using the multi-scale feature fusion module to fuse the plurality of intermediate features at different scales to obtain a fused feature;

[0057] Using the plurality of specific domain decoders to calculate the fused feature to obtain a predicted domain depth map corresponding to the image samples;

[0058] Constructing an uncertainty-based depth loss function according to the real domain depth map and the predicted domain depth map, adjusting the model parameters of the initial monocular depth estimation model according to the loss value of the uncertainty-based depth loss function, stopping the model training when the uncertainty-based depth loss function meets the convergence condition, and obtaining the target monocular depth estimation model.

[0059] In the uncertainty-based scene depth acquisition proposed in the present application, given a target domain wherein RGB images and their corresponding depth maps, which can then be directly utilized with supervised learning to let the model learn to estimate depth maps of the target domain.

[0060] As shown in Figure 4 Depth maps captured by sensors are usually sparse and noisy near image edges. To eliminate the influence of outliers and improve robustness, the present application adopts an uncertainty-based (model prediction confidence) depth loss function to train the target monocular depth estimation model:

[0061]

[0062] wherein, s t correspond to the up-sampled image data of the model in the domain , i.e. x t , the estimated depth map and uncertainty map, i.e. In this way, the model can acquire the ability to perform monocular depth perception in the target domain

[0063] Therefore, the present application proposes an efficient multi-branch model framework, which can realize lifelong, cross-domain and absolute scene scale monocular depth learning. It is particularly important to note that the method proposed in the present application is the first to realize multi-domain absolute depth estimation through lifelong learning.

[0064] Second, uncertainty-based old domain knowledge retention;

[0065] When a new domain appears, the monocular depth estimation model will correspondingly increase a new depth decoder of a specific domain, and learn its parameters with a loss function. However, this will change the original parameters of the encoder of the monocular depth estimation model, thereby causing the depth estimation in the old domain to fail, i.e. leading to catastrophic forgetting. In order to prevent forgetting, the present application embodiment proposes uncertainty-based old domain knowledge retention to avoid catastrophic forgetting of the monocular depth estimation model when learning a new domain.

[0066] ​Therefore, in another preferred embodiment of the present embodiment, the target monocular depth estimation model is a monocular depth estimation model of the old domain. After obtaining the target monocular depth estimation model, the computer device can obtain image samples of the new domain, input the image samples of the new domain into the target monocular depth estimation model, so that the target monocular depth estimation model uses the plurality of residual convolution modules to extract features of the image samples of the new domain to obtain a plurality of intermediate features of different scales of the image samples of the new domain, and uses the multi-scale feature fusion module to fuse the plurality of intermediate features of different scales to obtain fused features, and uses the plurality of specific field decoders to calculate the fused features to obtain a first predicted depth map and a first predicted uncertainty map corresponding to the image samples of the new domain;

[0067] obtain a monocular depth estimation model corresponding to the new domain, input the image samples of the new domain into the monocular depth estimation model of the new domain, so that the monocular depth estimation model of the new domain uses the plurality of residual convolution modules to extract features of the image samples of the new domain to obtain a plurality of intermediate features of different scales of the image samples of the new domain, and uses the multi-scale feature fusion module to fuse the plurality of intermediate features of different scales to obtain fused features, and uses the plurality of specific field decoders to calculate the fused features to obtain a second predicted depth map and a second predicted uncertainty map corresponding to the image samples of the new domain;

[0068] According to the first predicted depth map and the first predicted uncertainty map, the second predicted depth map and the second predicted uncertainty map, a consistency loss function is constructed, and the monocular depth estimation model of the new domain is adjusted according to the loss value of the consistency loss function, until the loss value of the consistency loss function is reduced to a preset numerical range, the model training is stopped, and a target monocular depth estimation model of the new domain is obtained.

[0069] In related schemes, when a new field occurs, a new specific field depth decoder f t+1 is correspondingly added, and the parameters thereof are learned using the above loss function. However, this will change the parameters of the encoder, thereby causing the depth estimation on the old field to fail, that is, causing catastrophic forgetting. In order to prevent forgetting, the model parameters need to be regularized, and in the method of the present application, the model trained on the old field is used as an expert teacher, and the prediction results thereof in each field are used to impose consistency constraints.

[0070] Therefore, g', f1',..., f t ' can be the encoder and the decoder trained on the old domain , and g, f1,..., f t are the model encoder and the plurality of specific field decoders that need to be updated on the new domain, and the following operations are specifically performed based on the uncertainty-based old domain knowledge:

[0071] 1. For each old domain With the trained model, i.e. g', f i ', predict the depth map t+1 (i.e. the aforementioned first predicted depth map) and uncertainty map (i.e. the aforementioned first predicted uncertainty map) from the image data of the new domain, i.e. x

[0072] 2. For each old domain, with the new model, i.e. g, f i , predict the depth map t+1 (i.e. the aforementioned second predicted depth map) and uncertainty map (i.e. the aforementioned second predicted uncertainty map) from the image data of the new domain, i.e. x

[0073] 3. Minimize the consistency loss, including depth consistency and uncertainty consistency. The consistency loss can be represented as follows:

[0074]

[0075]

[0076]

[0077] Therefore, according to the above loss function, the target monocular depth estimation model can be trained to enable the target monocular depth estimation model obtained by training to acquire the ability of depth estimation in the new target domain, retain the depth estimation ability learned in the old target domain, prevent knowledge forgetting, and thus more accurately perform monocular depth estimation of images.

[0078] Third, memory enhancement based on data playback mechanism;

[0079] The embodiment of the present application also proposes a memory enhancement method based on a data playback mechanism to retain the depth estimation ability learned by the monocular depth estimation model in the old target domain, further prevent knowledge forgetting, and maintain the depth estimation performance of the monocular depth estimation model in the old domain.

[0080] Therefore, in another preferred embodiment of the present embodiment, the computer device can also obtain a training sample, which includes an image sample of the old domain;

[0081] ​​Image samples from the old domain are input into the target monocular depth estimation model, which uses multiple residual convolution modules to extract features from the image samples from the old domain, obtaining multiple intermediate features at different scales of the image samples from the old domain. A multi-scale feature fusion module is used to fuse the intermediate features at multiple different scales to obtain fused features. Multiple domain-specific decoders are used to calculate the fused features to obtain the third predicted depth map and the third predicted uncertainty map corresponding to the image samples from the old domain.

[0082] Based on the third predicted depth map and the third predicted uncertainty map, a loss function is constructed. The target monocular depth estimation model is adjusted according to the loss value of the loss function until the loss value of the loss function is reduced to a preset value range, at which point model training stops.

[0083] In the relevant plan, if new fields The image is related to an old field. Since the images have the same distribution, consistency loss will affect the images stored in the neighborhood. The knowledge is fully valid. Otherwise, its performance tends to degrade due to the domain gap. If there is a significant domain shift between the new and old domains, the model's performance on the old domain may drop significantly. To address this issue, this application adopts a replay strategy similar to many classic lifelong learning methods, randomly retaining a limited amount of training data (500 images) from each previous domain by periodically and repeatedly reviewing historical data, and replaying this data for learning the new domain.

[0084] Replay loss can be understood as a supplement to the above-mentioned old domain knowledge preservation techniques based on uncertainty. The specific implementation steps are as follows:

[0085] 1. For each legacy domain For the few images that are retained, the depth map predicted by the model for each image is calculated. (i.e., the aforementioned third prediction depth map) and uncertainty map s i (i.e., the aforementioned third prediction uncertainty diagram).

[0086] 2. Since a small number of images and their corresponding true values ​​are retained, the consistency loss can be minimized. This replay loss can be expressed as:

[0087]

[0088] Therefore, combining prediction consistency regularization and replay strategy can overcome catastrophic forgetting. The former applies both depth and uncertainty consistency, while the latter retains a small portion of data from the old domain and replays it during the learning of the new domain, thus achieving more accurate monocular depth estimation of images.

[0089] In the model deployment process of the monocular depth estimation model, an embodiment of the present application proposes to identify the minimum distance between a given image and each domain in the feature space to achieve the selection of the online specific domain depth decoder.

[0090] Therefore, in another preferred embodiment of the present embodiment, the computer device can further obtain image samples of each domain, input the image samples of each domain into the target monocular depth estimation model, so that the target monocular depth estimation model uses multiple residual convolution modules to extract features of the image samples of each domain to obtain multiple intermediate features of different scales of the image samples of each domain, and uses a multi-scale feature fusion module to fuse the multiple intermediate features of different scales to obtain fused features;

[0091] According to the fused features corresponding to the image samples of each domain, the average feature approximation of each domain is calculated;

[0092] According to the distance of the input image sample in the feature space from each domain according to the fused features corresponding to the image samples of each domain, the domain corresponding to the minimum distance is determined as the target domain of the input image sample. Therefore, the decoder corresponding to the target domain of the input image sample can be further determined, which can be used for the decoding calculation process of the fused features corresponding to the input image sample, and then the depth map and the uncertainty map of the input image sample are obtained.

[0093] In related schemes, after completing the incremental learning of all target domains, in an ideal case, the model can correctly estimate the depth map of any image sampled from multiple target domains. Since the model has multiple branch decoders, an actual challenge is how to identify the domain of the image and automatically select the corresponding decoder in the inference process. To solve this problem, the present application proposes to identify the minimum distance between a given image and each domain in the feature space to achieve the selection of the online specific domain depth decoder. The specific execution process is as follows:

[0094] 1. Since a subset of each domain (i.e., replay data, assuming that the subset is 500 pictures) is reserved, first, the average feature approximation of each domain is calculated using the replay data, i.e.,

[0095]

[0096] wherein, is the fused feature corresponding to the kth picture in the 500 pictures of the domain .

[0097] 2. The distance d of each input image sample in the feature space from each domain is calculated i , then the depth decoder of the specific domain should select the domain corresponding to the minimum d i , and its expression can be represented as:

[0098]

[0099] s.t.d i ||g(x)-u i ||2

[0100] wherein g(x) is the fusion feature corresponding to the input image sample.

[0101] Therefore, when the model is deployed, the minimum distance according to the average feature of each domain is proposed to automatically select the decoder of a specific domain for the image, which can achieve more accurate monocular depth estimation of the image.

[0102] The depth map captured in the real world is obviously different in different domains, and their data quality and scale are related to the domain. Therefore, the model must have multiple prediction branches, each branch is used to predict a different range of depth map. For this purpose, the embodiment proposes a multi-branch perception framework, which consists of a domain-shared encoder and a domain-specific decoder. For the input image of each domain, not only the depth map is predicted, but also the uncertainty map is predicted to exclude the performance decline caused by the outliers in the depth map taken by the depth sensor during the training process. The framework allows monocular depth estimation across multiple domains with absolute scale. To further overcome catastrophic forgetting, regularization constraints are adopted to overcome significant domain gaps using knowledge distillation loss and replay loss terms. When learning in a new domain, the framework dynamically grows a decoder specific to this particular domain. The domain-specific decoder will learn with the data of the new domain; other decoders trained in old domains will be regularized by depth consistency and uncertainty consistency, and replay loss. These regularization terms complement each other to improve the stability and plasticity of lifelong learning. In the inference process, the corresponding domain-specific decoder needs to be dynamically selected. The present application proposes to compare the distance of the image and each domain in the feature space, and then the minimum distance corresponds to the domain to which the image belongs. Through extensive numerical studies, it is shown that the method proposed in the present application can achieve good efficiency, stability and plasticity, and the accuracy on the benchmark dataset is 8%-15% higher than that of existing methods.

[0103] The monocular depth estimation method based on lifelong learning in the embodiment of the present application is described above, and the computer device in the embodiment of the present application is described below. Please refer to Figure 5 , one embodiment of the computer device in the embodiment of the present application includes:

[0104] The acquisition unit 501 is configured to acquire a target monocular depth estimation model trained in advance, wherein the target monocular depth estimation model includes a multi-domain shared encoder and a plurality of domain-specific decoders connected with the multi-domain shared encoder.

[0105] The monocular depth estimation unit 502 is configured to acquire a target image of a target domain, and input the target image into the target monocular depth estimation model, so that the target monocular depth estimation model performs the following operations: performing feature extraction on the target image by using the multi-domain shared encoder to obtain image features; and performing calculation on the image features by using a decoder corresponding to the target domain in the plurality of specific domain decoders to obtain a depth estimation map of the target domain.

[0106] The output unit 503 is configured to output the depth estimation map of the target domain.

[0107] In a preferred embodiment of the present embodiment, the computer device further comprises:

[0108] The training unit 504 is configured to perform a training step of the target monocular depth estimation model, and the training step comprises:

[0109] acquiring a plurality of groups of training samples, each group of training samples comprising an image sample of a domain and a real domain depth map corresponding to the image sample;

[0110] acquiring an initial monocular depth estimation model, the initial monocular depth estimation model comprising a multi-domain shared encoder, a plurality of specific domain decoders connected to the multi-domain shared encoder, the multi-domain shared encoder comprising a plurality of residual convolution modules and a multi-scale feature fusion module connected to the plurality of residual convolution modules;

[0111] inputting the plurality of groups of training samples into the initial monocular depth estimation model, so that the initial monocular depth estimation model performs the following operations:

[0112] performing feature extraction on the image sample by using the plurality of residual convolution modules to obtain a plurality of intermediate features of different scales of the image sample;

[0113] performing feature fusion on the plurality of intermediate features of different scales by using the multi-scale feature fusion module to obtain a fused feature;

[0114] performing calculation on the fused feature by using the plurality of specific domain decoders to obtain a predicted domain depth map corresponding to the image sample;

[0115] constructing an uncertainty-based depth loss function according to the real domain depth map and the predicted domain depth map, adjusting model parameters of the initial monocular depth estimation model according to a loss value of the uncertainty-based depth loss function, and stopping model training when the uncertainty-based depth loss function meets a convergence condition to obtain the target monocular depth estimation model.

[0116] In a preferred implementation of the embodiment, the target monocular depth estimation model is a monocular depth estimation model of an old domain;

[0117] The training unit 504 is further configured to:

[0118] obtain image samples of a new domain, input the image samples of the new domain into the target monocular depth estimation model, so that the target monocular depth estimation model uses the plurality of residual convolution modules to extract features of the image samples of the new domain, to obtain a plurality of intermediate features of different scales of the image samples of the new domain, and uses the multi-scale feature fusion module to fuse the plurality of intermediate features of different scales, to obtain fused features, and uses the plurality of specific domain decoders on the old domain to calculate the fused features to obtain a first predicted depth map and a first predicted uncertainty map corresponding to the image samples of the new domain;

[0119] obtain a monocular depth estimation model corresponding to the new domain, input the image samples of the new domain into the monocular depth estimation model of the new domain, so that the monocular depth estimation model of the new domain uses a plurality of residual convolution modules to extract features of the image samples of the new domain, to obtain a plurality of intermediate features of different scales of the image samples of the new domain, and uses a multi-scale feature fusion module to fuse the plurality of intermediate features of different scales, to obtain fused features, and uses a plurality of specific domain decoders on the old domain to calculate the fused features to obtain a second predicted depth map and a second predicted uncertainty map corresponding to the image samples of the new domain;

[0120] According to the first predicted depth map and the first predicted uncertainty map, the second predicted depth map and the second predicted uncertainty map, a consistency loss function is constructed, and the monocular depth estimation model of the new domain is adjusted according to the loss value of the consistency loss function, until the loss value of the consistency loss function is reduced to a preset numerical range, the model training is stopped, and the target monocular depth estimation model of the new domain is obtained.

[0121] In a preferred implementation of the embodiment, the training unit 504 is further configured to:

[0122] obtain training samples, the training samples including image samples of the old domain;

[0123] input the image sample of the old domain into the target monocular depth estimation model, so that the target monocular depth estimation model uses the plurality of residual convolution modules to extract features of the image sample of the old domain, obtains a plurality of intermediate features of different scales of the image sample of the old domain, and uses the multi-scale feature fusion module to fuse the plurality of intermediate features of different scales to obtain a fused feature, and uses the plurality of specific field decoders to calculate the fused feature to obtain a third predicted depth map and a third predicted uncertainty map corresponding to the image sample of the old domain;

[0124] According to the third predicted depth map and the third predicted uncertainty map, a loss function is constructed, and the target monocular depth estimation model is adjusted according to a loss value of the loss function until the loss value of the loss function is reduced to a preset numerical range, and the model training is stopped.

[0125] In a preferred embodiment of the present embodiment, the computer device further comprises:

[0126] The model deployment unit 505 is configured to obtain image samples of each domain, and input the image samples of each domain into the target monocular depth estimation model, so that the target monocular depth estimation model uses the plurality of residual convolution modules to extract features of the image samples of each domain, obtains a plurality of intermediate features of different scales of the image samples of each domain, and uses the multi-scale feature fusion module to fuse the plurality of intermediate features of different scales to obtain a fused feature.

[0127] According to the fused feature corresponding to the image sample of each domain, an average feature approximation value of each domain is calculated.

[0128] According to the average feature approximation value, a distance of the input image sample from each domain in the feature space is calculated, and a domain corresponding to the smallest distance is determined as a target domain of the input image sample.

[0129] In the present embodiment, the operations performed by each unit in the computer device are similar to those described in the foregoing Figure 1 embodiments described in the foregoing

[0130] In this embodiment, the target monocular depth estimation model is a multi-branch perception framework composed of a domain-shared encoder and a domain-specific decoder, which can predict the depth map and uncertainty map of the input image to exclude the performance decline caused by outliers in the depth map taken by the depth sensor during the training process. To further overcome catastrophic forgetting, a regularization constraint is adopted, which uses knowledge distillation loss and replay loss terms to overcome significant domain gaps. The domain-specific decoder will be learned with data from the new domain; other decoders trained in the old domain will be regularized by depth consistency and uncertainty consistency and replay loss, and the regularization terms complement each other to improve the stability and plasticity of lifelong learning, so as to achieve more accurate monocular depth estimation of images.

[0131] The computer device in the embodiments of the present application is described below. Please refer to Figure 6 An embodiment of the computer device in the embodiments of the present application includes:

[0132] The computer device 600 can include one or more central processing units (CPUs) 601 and a memory 605 in which one or more application programs or data are stored.

[0133] The memory 605 can be volatile storage or persistent storage. The programs stored in the memory 605 can include one or more modules, each of which can include a series of instruction operations in the computer device. Further, the central processing unit 601 can be configured to communicate with the memory 605 and execute the series of instruction operations in the memory 605 on the computer device 600.

[0134] The computer device 600 can also include one or more power supplies 602, one or more wired or wireless network interfaces 603, one or more input / output interfaces 604, and / or one or more operating systems, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0135] The central processing unit 601 can execute the operations performed by the computer device in the embodiments described above, and details are not repeated here. Figure 1 The central processing unit 601 can execute the operations performed by the computer device in the embodiments described above, and details are not repeated here.

[0136] The embodiments of the present application also provide a computer storage medium, one embodiment of which includes: the computer storage medium stores instructions, which, when executed on a computer, cause the computer to perform the operations performed by the computer device in the embodiments described above. Figure 1 The central processing unit 601 can execute the operations performed by the computer device in the embodiments described above, and details are not repeated here.

[0137] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0138] In several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0139] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0140] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0141] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, read-only memory), a random access memory (RAM, random access memory), a magnetic disk or an optical disk, and various program code storage media.

Claims

1. A monocular depth estimation method based on lifelong learning, characterized in that, The method is applied to a computer device, and the method includes: Obtain a pre-trained target monocular depth estimation model, wherein the target monocular depth estimation model includes a multi-domain shared encoder and multiple domain-specific decoders connected to the multi-domain shared encoder; Obtain the target image in the target domain, and input the target image into the target monocular depth estimation model so that the target monocular depth estimation model performs the following operations: The multi-domain shared encoder is used to extract features from the target image to obtain image features; Using the decoder corresponding to the target domain among the multiple domain-specific decoders, the image features are calculated to obtain a depth estimation map of the target domain; Output a depth estimation map of the target domain; The training steps of the target monocular depth estimation model include: Multiple sets of training samples are obtained, each set of training samples including an image sample of a domain and the corresponding real domain depth map of the image sample; An initial monocular depth estimation model is obtained, which includes a multi-domain shared encoder and multiple domain-specific decoders connected to the multi-domain shared encoder. The multi-domain shared encoder includes multiple residual convolutional modules and a multi-scale feature fusion module connected to the multiple residual convolutional modules. The multiple sets of training samples are input into the initial monocular depth estimation model, so that the initial monocular depth estimation model performs the following operations: The image samples are used to extract features using the multiple residual convolution modules to obtain multiple intermediate features of the image samples at different scales; The multi-scale feature fusion module is used to fuse the intermediate features at multiple different scales to obtain fused features; The fusion features are calculated using the multiple domain-specific decoders to obtain the predicted domain depth map corresponding to the image sample; An uncertainty-based depth loss function is constructed based on the real domain depth map and the predicted domain depth map. The model parameters of the initial monocular depth estimation model are adjusted according to the loss value of the uncertainty-based depth loss function until the uncertainty-based depth loss function meets the convergence condition, at which point model training stops, and the target monocular depth estimation model is obtained.

2. The method according to claim 1, characterized in that, The target monocular depth estimation model is the old domain monocular depth estimation model; After obtaining the target monocular depth estimation model, the method further includes: Image samples from a new domain are acquired and input into the target monocular depth estimation model. The target monocular depth estimation model uses multiple residual convolution modules to extract features from the image samples of the new domain, obtaining multiple intermediate features of different scales of the image samples of the new domain. The multi-scale feature fusion module is then used to fuse the intermediate features of the multiple different scales to obtain fused features. The multiple domain-specific decoders on the old domain are used to calculate the fused features to obtain a first predicted depth map and a first predicted uncertainty map corresponding to the image samples of the new domain. Obtain the monocular depth estimation model corresponding to the new domain, input the image samples of the new domain into the monocular depth estimation model of the new domain, so that the monocular depth estimation model of the new domain uses multiple residual convolution modules to extract features from the image samples of the new domain, obtain multiple intermediate features of the image samples of the new domain at different scales, and uses a multi-scale feature fusion module to fuse the intermediate features at multiple different scales to obtain fused features. Use multiple domain-specific decoders on the old domain to calculate the fused features to obtain the second predicted depth map and the second predicted uncertainty map corresponding to the image samples of the new domain. Based on the first predicted depth map and the first predicted uncertainty map, the second predicted depth map and the second predicted uncertainty map, a consistency loss function is constructed. The monocular depth estimation model of the new domain is adjusted according to the loss value of the consistency loss function until the loss value of the consistency loss function is reduced to a preset numerical range, at which point model training is stopped, and the target monocular depth estimation model of the new domain is obtained.

3. The method according to claim 2, characterized in that, The method further includes: Obtain training samples, which include image samples from the old domain; The image samples from the old domain are input into the target monocular depth estimation model, so that the target monocular depth estimation model uses the multiple residual convolution modules to extract features from the image samples from the old domain, obtains multiple intermediate features of the image samples from the old domain at different scales, and uses the multi-scale feature fusion module to fuse the intermediate features of the multiple different scales to obtain fused features. The multiple domain-specific decoders are used to calculate the fused features to obtain the third predicted depth map and the third predicted uncertainty map corresponding to the image samples from the old domain. Based on the third predicted depth map and the third predicted uncertainty map, a loss function is constructed, and the target monocular depth estimation model is adjusted according to the loss value of the loss function until the loss value of the loss function is reduced to a preset value range, at which point model training stops.

4. The method according to claim 3, characterized in that, The method further includes: Image samples of each domain are acquired and input into the target monocular depth estimation model, so that the target monocular depth estimation model uses the multiple residual convolution modules to extract features from the image samples of each domain, obtaining multiple intermediate features of different scales for the image samples of each domain, and uses the multi-scale feature fusion module to fuse the multiple intermediate features of different scales to obtain fused features. Calculate the average feature approximation value for each domain based on the fusion features corresponding to the image samples in each domain; The distance of the input image sample to each domain in the feature space is calculated based on the average feature approximation value, and the domain corresponding to the smallest distance is determined as the target domain of the input image sample.

5. A computer device, characterized in that, The computer device includes: The acquisition unit is used to acquire a pre-trained target monocular depth estimation model, wherein the target monocular depth estimation model includes a multi-domain shared encoder and multiple domain-specific decoders connected to the multi-domain shared encoder. A monocular depth estimation unit is used to acquire a target image of a target domain and input the target image into the target monocular depth estimation model, so that the target monocular depth estimation model performs the following operations: using the multi-domain shared encoder to extract features from the target image to obtain image features; using the decoder corresponding to the target domain among the multiple domain-specific decoders to calculate the depth estimation map of the target domain. The output unit is used to output a depth estimation map of the target domain; The computer device also includes: A training unit is configured to perform the training steps of the target monocular depth estimation model, the training steps including: Multiple sets of training samples are obtained, each set of training samples including an image sample of a domain and the corresponding real domain depth map of the image sample; An initial monocular depth estimation model is obtained, which includes a multi-domain shared encoder and multiple domain-specific decoders connected to the multi-domain shared encoder. The multi-domain shared encoder includes multiple residual convolutional modules and a multi-scale feature fusion module connected to the multiple residual convolutional modules. The multiple sets of training samples are input into the initial monocular depth estimation model, so that the initial monocular depth estimation model performs the following operations: The image samples are used to extract features using the multiple residual convolution modules to obtain multiple intermediate features of the image samples at different scales; The multi-scale feature fusion module is used to fuse the intermediate features at multiple different scales to obtain fused features; The fusion features are calculated using the multiple domain-specific decoders to obtain the predicted domain depth map corresponding to the image sample; An uncertainty-based depth loss function is constructed based on the real domain depth map and the predicted domain depth map. The model parameters of the initial monocular depth estimation model are adjusted according to the loss value of the uncertainty-based depth loss function until the uncertainty-based depth loss function meets the convergence condition, at which point model training stops, and the target monocular depth estimation model is obtained.

6. The computer device according to claim 5, characterized in that, The target monocular depth estimation model is the old domain monocular depth estimation model; The training unit is also used for: Image samples of a new domain are acquired and input into the target monocular depth estimation model. The target monocular depth estimation model uses the multiple residual convolution modules to extract features from the image samples of the new domain, obtaining multiple intermediate features of the image samples of the new domain at different scales. The multi-scale feature fusion module is used to fuse the intermediate features at the multiple different scales to obtain fused features. The multiple domain-specific decoders are used to calculate the fused features to obtain a first predicted depth map and a first predicted uncertainty map corresponding to the image samples of the new domain. Obtain the monocular depth estimation model corresponding to the new domain, input the image samples of the new domain into the monocular depth estimation model of the new domain, so that the monocular depth estimation model of the new domain uses multiple residual convolution modules to extract features from the image samples of the new domain, obtain multiple intermediate features of the image samples of the new domain at different scales, and uses a multi-scale feature fusion module to fuse the intermediate features at multiple different scales to obtain fused features. Use multiple domain-specific decoders to calculate the fused features to obtain the second predicted depth map and the second predicted uncertainty map corresponding to the image samples of the new domain. Based on the first predicted depth map and the first predicted uncertainty map, the second predicted depth map and the second predicted uncertainty map, a consistency loss function is constructed. The monocular depth estimation model of the new domain is adjusted according to the loss value of the consistency loss function until the loss value of the consistency loss function is reduced to a preset numerical range, at which point model training is stopped, and the target monocular depth estimation model of the new domain is obtained.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 4.

8. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed on the computer, cause the computer to perform the method as described in any one of claims 1 to 4.

Images