Video depth estimation method and device, equipment, storage medium and program product

Through the lightweight depth estimation model, combining depth estimation and front background segmentation, the performance limitation problem of convolutional neural networks in complex image processing is solved, and the accuracy of depth estimation and the real-timeness of the model is improved, which is suitable for mobile devices.

CN119006556APending Publication Date: 2024-11-22PENINSULA INFORMATION TECH INC
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410972133.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The performance of the convolutional neural network model is limited when processing complex images, resulting in poor depth estimation accuracy, high computing power requirements, large storage resources occupy space, making it difficult to deploy on mobile devices, and cannot be suitable for real-time business scenarios.

Method used

Using a lightweight depth estimation model, the pre-built lightweight depth estimation model is trained by obtaining image frames from the training dataset input to the depth estimation model and the front background segmentation model. This model includes an encoder and a decoder for output depth estimation. Segment results with the previous background to reduce computing power and storage resource requirements.

Benefits of technology

It improves the understanding and prediction capabilities of the depth estimation model for complex scenarios, improves accuracy, is suitable for real-time business scenarios, and meets the deployment needs of mobile devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006556A_ABST
    Figure CN119006556A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a video depth estimation method and device, equipment, a storage medium and a program product, and the method comprises the steps: inputting image frames in a training data set into a set depth estimation model and a foreground and background segmentation model through obtaining the training data set, obtaining a depth estimation image and a foreground and background segmentation image corresponding to the image frame; a pre-constructed lightweight depth estimation model is trained based on the training data set, the depth estimation map and the foreground and background segmentation map, the lightweight depth estimation model comprises an encoder, a first decoder and a second decoder, the first decoder and the second decoder are connected with the encoder, the first decoder is used for outputting a depth estimation result, and the second decoder is used for outputting a depth estimation result; the second decoder is used for outputting a foreground and background segmentation result; and obtaining a to-be-processed image frame, and inputting the to-be-processed image frame into the trained lightweight depth estimation model to obtain a target depth estimation map. According to the scheme, the accuracy of depth estimation is improved, the method is suitable for a real-time service scene, and the deployment requirement of mobile equipment is met.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a video depth estimation method, apparatus, device, storage medium, and program product. Background Art

[0002] With the rapid development of computer vision technology, depth estimation, as a key technology in the field of computer vision, can be used to solve the problem of distance perception of objects in the real world and promote the application of intelligent systems in the fields of autonomous driving, intelligent monitoring, augmented reality, etc. Among them, depth estimation involves inferring the distance information of objects in the scene from images or videos. With the development of deep learning technology, depth estimation has made great progress in recent years. Deep learning-based methods can use deep neural networks to learn features and patterns from large amounts of data, capture semantic information in images, and complete depth estimation tasks. Among deep learning methods, convolutional neural networks have become the main technical means for depth estimation. The convolutional neural network can directly learn depth information from the input image through end-to-end training without the need for manual feature design or complex preprocessing steps.

[0003] However, due to the influence of factors such as the size, shape and occlusion of objects in the image, the generation of depth estimation maps is a highly nonlinear and multimodal problem. The performance of the convolutional neural network model used in related technologies is easily limited when processing complex images, resulting in poor accuracy. In addition, due to the high computing power requirements of the model and the large space occupied by storage resources, it is difficult to deploy on mobile devices and is not suitable for real-time business scenarios, and needs to be improved. Summary of the invention

[0004] The embodiments of the present application provide a video depth estimation method, apparatus, device, storage medium and program product, which solve the problems in the related art that the convolutional neural network model has limited performance and poor accuracy when processing complex images, and is difficult to deploy on mobile devices and cannot be applied to real-time business scenarios due to the high computing power requirements of the model and the large space occupied by storage resources. The constructed depth estimation model can better understand the semantic information in the scene, improve the depth estimation model's understanding and prediction capabilities for complex scenes, improve the accuracy of depth estimation, and can reduce computing power and storage resource requirements, so that it can be applied to real-time business scenarios and meet the deployment requirements of mobile devices.

[0005] In a first aspect, an embodiment of the present application provides a video depth estimation method, the method comprising:

[0006] Obtain a training data set, input the image frames in the training data set into a set depth estimation model and a foreground and background segmentation model, and obtain a depth estimation map and a foreground and background segmentation map corresponding to the image frames;

[0007] Based on the training data set, the depth estimation map and the foreground and background segmentation map, a pre-constructed lightweight depth estimation model is trained, wherein the lightweight depth estimation model comprises an encoder and a first decoder and a second decoder respectively connected to the encoder, wherein the first decoder is used to output a depth estimation result, and the second decoder is used to output a foreground and background segmentation result;

[0008] Obtain an image frame to be processed, and input the image frame to be processed into the trained lightweight depth estimation model to obtain a target depth estimation map.

[0009] In a second aspect, an embodiment of the present application further provides a video depth estimation device, the device comprising:

[0010] A first acquisition module is configured to acquire a training data set;

[0011] A labeling module is configured to input the image frames in the training data set into a set depth estimation model and a foreground and background segmentation model to obtain a depth estimation map and a foreground and background segmentation map corresponding to the image frames;

[0012] A model training module, configured to train a pre-constructed lightweight depth estimation model based on the training data set, the depth estimation map and the foreground and background segmentation map, wherein the lightweight depth estimation model includes an encoder and a first decoder and a second decoder respectively connected to the encoder, the first decoder is used to output a depth estimation result, and the second decoder is used to output a foreground and background segmentation result;

[0013] A second acquisition module is configured to acquire an image frame to be processed;

[0014] The depth estimation module is configured to input the image frame to be processed into the trained lightweight depth estimation model to obtain a target depth estimation map.

[0015] In a third aspect, an embodiment of the present application further provides a video depth estimation device, the device comprising:

[0016] one or more processors;

[0017] a storage device configured to store one or more programs,

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the video depth estimation method described in the embodiment of the present application.

[0019] In a fourth aspect, an embodiment of the present application further provides a non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the video depth estimation method described in the embodiment of the present application when executed by a computer processor.

[0020] In a fifth aspect, an embodiment of the present application further provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor of the device reads and executes the computer program from the computer-readable storage medium, so that the device executes the video depth estimation method described in the embodiment of the present application.

[0021] In an embodiment of the present application, a training data set is obtained, and image frames in the training data set are input into a set depth estimation model and a foreground and background segmentation model to obtain a depth estimation map and a foreground and background segmentation map corresponding to the image frame; a pre-constructed lightweight depth estimation model is trained based on the training data set, the depth estimation map, and the foreground and background segmentation map, wherein the lightweight depth estimation model includes an encoder and a first decoder and a second decoder respectively connected to the encoder, the first decoder being used to output a depth estimation result, and the second decoder being used to output a foreground and background segmentation result; an image frame to be processed is obtained, and the image frame to be processed is input into the trained lightweight depth estimation model to obtain a target depth estimation map. In the above scheme, by using the open source depth estimation model and foreground and background segmentation model to generate the depth estimation map and foreground and background segmentation map corresponding to the training data set, the training data set can be efficiently labeled, the true value label corresponding to the training data set can be accurately obtained, and the data labeling cost can be reduced; by training the lightweight depth estimation model with decoders respectively corresponding to the depth estimation and foreground and background segmentation, the foreground and background segmentation can be reasonably used to provide semantic information of the foreground and background for the depth estimation, so that the model can better understand the semantic information in the scene, improve the depth estimation model's understanding and prediction capabilities for complex scenes, and improve the accuracy of depth estimation; and the lightweight depth estimation model provided in this embodiment can reduce computing power and storage resource requirements, can be suitable for real-time business scenarios, and meet the deployment requirements of mobile devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A flowchart of a video depth estimation method provided in an embodiment of the present application;

[0023] Figure 2 A schematic diagram of the structure of a pre-built lightweight depth estimation model provided in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of the effect of applying the video depth estimation method provided in an embodiment of the present application to perform depth estimation;

[0025] Figure 4 A flowchart of a video depth estimation method including a training process of a lightweight depth estimation model provided in an embodiment of the present application;

[0026] Figure 5 A flowchart of a video depth estimation method including a multi-stage training process of a lightweight depth estimation model provided in an embodiment of the present application;

[0027] Figure 6 A flowchart of another method for video depth estimation including a multi-stage training process of a lightweight depth estimation model provided in an embodiment of the present application;

[0028] Figure 7 A flowchart of a video depth estimation method including a process of generating a training data set provided in an embodiment of the present application;

[0029] Figure 8 A structural block diagram of a video depth estimation device provided in an embodiment of the present application;

[0030] Fig. 9 A schematic diagram of the structure of a video depth estimation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the embodiments of the present application, rather than to limit the embodiments of the present application. It should also be noted that, for ease of description, only parts related to the embodiments of the present application are shown in the accompanying drawings, rather than all structures.

[0032] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0033] The video depth estimation method provided in the embodiment of the present application can efficiently complete the labeling task of the training data set by using an open source large model, and by training the lightweight depth estimation model that integrates the corresponding foreground and background segmentation settings, a lightweight and efficient depth estimation model can be obtained, which is suitable for real-time business scenarios and meets the deployment requirements of mobile devices. Relevant application scenarios may include: video conferencing, video calls, indoor live broadcasts, outdoor live broadcasts, and short videos. Taking the live broadcast scene as an example, by applying the video depth estimation method provided in this embodiment, the depth estimation of the video frame can be effectively performed, and accurate depth information can be provided for the depth reconstruction of the live broadcast screen, which is conducive to improving the stereoscopic sense of the video. The several application scenarios listed above are only exemplary and explanatory. In practical applications, the video depth estimation method can also be used in depth estimation tasks in other scenarios, and the embodiment of the present application does not limit this. The video depth estimation method, device, equipment, storage medium and program product provided in the embodiment of the present application are intended to solve the problem that the performance of the convolutional neural network model in the related art is limited when processing complex images, resulting in poor accuracy, and the model has high computing power requirements and large storage resource space, which is difficult to deploy on mobile devices and cannot be applied to real-time business scenarios.

[0034] In the video depth estimation method provided in the embodiment of the present application, the executor of each step can be a computer device, which refers to any electronic device with data calculation, processing and storage capabilities, such as mobile phones, PCs (Personal Computers), tablet computers and other terminal devices, and can also be servers and other devices, which are not limited in the embodiment of the present application.

[0035] Figure 1 A flowchart of a video depth estimation method provided in an embodiment of the present application. Figure 1 As shown, the following steps are included:

[0036] Step S101, obtaining a training data set, inputting image frames in the training data set into a set depth estimation model and a foreground and background segmentation model, and obtaining a depth estimation map and a foreground and background segmentation map corresponding to the image frame.

[0037] Among them, the training data set can be derived from a benchmark data set, and the benchmark data set can include an open source data set and an open source video data set. The open source data set can include a KITTI data set, a NYU Depth V2 data set, and a CityScapes data set, etc. The open source video data can be obtained by crawling Internet videos through a crawler script. In one embodiment, the benchmark data set can be divided into a training data set, a verification data set, and a test data set according to a preset ratio. For example, the preset ratio can be 8:1:1, wherein the training data set can be used in the training phase of the pre-built lightweight depth estimation model in the embodiment of the present application, the verification data set can be used for model effect verification at different stages of the training process, and the test data set can be used for subsequent model accuracy evaluation. In one embodiment, the depth estimation model can be an open source large model, such as a Depth-Anything-Large model, a DinoV2 model, and a MiDaS model, etc. The foreground and background segmentation model can be an RVM (Robust Video Matting) model, a DVM (Deep Video Matting, Deep Video Matting) model, etc., which is not limited in this application. By using an open source depth estimation model and foreground-background segmentation model, it is possible to perform high-precision depth estimation and foreground-background image segmentation on the training data set, provide reliable true value labels for model training, and reduce the high cost of manually annotated data. Among them, the depth estimation map can be a map that records the depth information of each pixel in the image frame, and the depth information is the distance between the object surface corresponding to the pixel and the image acquisition device. The foreground-background segmentation map can be a map that records the probability value of each pixel in the image frame being the foreground or background. For example, a probability value of 0 can represent that the pixel is in the foreground, and a probability value of 1 can represent that the pixel is in the background. Then the probability value at (0,1) can represent the degree to which the pixel is close to the foreground or background.

[0038] Step S102, training a pre-constructed lightweight depth estimation model based on a training data set, a depth estimation map, and a foreground and background segmentation map, wherein the lightweight depth estimation model includes an encoder and a first decoder and a second decoder connected to the encoder respectively, the first decoder is used to output a depth estimation result, and the second decoder is used to output a foreground and background segmentation result.

[0039] Among them, the encoder in the lightweight depth estimation model can be constructed based on a lightweight neural network architecture, for example, a MobileNetV3 model, a MobileNetV2 model, a ShuffleNet model, and a SINet model. In one embodiment, the encoder may include multiple lightweight convolution modules connected in series. Taking the encoder as an example of using the MobileNetV3 model for construction, the lightweight convolution module of the MobileNetV3 model can be used, and its lightweight convolution module includes a 1x1 convolution layer, a depth-separable convolution layer, and an activation function layer connected in sequence, and the depth-separable convolution layer may include a depth convolution and a point-by-point convolution for efficient feature extraction. Optionally, the encoder may also include other components of the MobileNetV3 model, for example, a convolution layer, a batch normalization layer, an activation function layer, and a global average pooling layer connected after the lightweight convolution module. Optionally, in order to reduce the computational complexity of the model, the number of channels of the MobileNetV3 model may also be appropriately reduced to meet the computing power and performance requirements of the live broadcast business and achieve the purpose of model lightweighting. The above encoder is only an exemplary description, and developers can also use other lightweight neural network architectures to construct encoders, which is not limited in this application. In one embodiment, the first decoder and the second decoder respectively include multiple upsampling modules connected in series, and each upsampling module includes a convolutional layer, a normalization layer, an activation function layer, a convolutional gated loop layer and an upsampling layer connected in sequence. Among them, each upsampling module can be used to upsample features and reconstruct images. The convolutional gated loop layer in the upsampling module is a neural network structure that combines a convolutional neural network and a gated loop unit, which integrates the convolutional neural network's ability to capture spatial features and the gated loop unit's ability to model time series data, and can effectively capture the spatial and temporal information in the video sequence. When processing image frames, the convolutional gated loop layer can receive the hidden state of the current frame and the previous frame, and update the current hidden state through a gating mechanism, thereby transmitting information between frames, so that the results generated by the network are relatively stable in time, and can make full use of the time domain information of the video frame sequence, and perform depth estimation more accurately and stably.

[0040] Since depth estimation involves complex and diverse business scenarios, taking the live broadcast business scenario as an example, the live broadcast video usually uses the anchor as the foreground, and the distance between the anchor and the camera is relatively close. Depth estimation requires accurate assessment of the distance from the background to the camera. Therefore, the foreground and background segmentation task can be used to assist the training of the depth estimation task to improve the model's depth estimation accuracy for the background area. Among them, the foreground and background segmentation task involves pixel-level classification, which has a more accurate understanding of the semantic information in the video screen, can provide semantic information of the foreground and background for depth estimation, and assist the model in learning depth information. Especially in scenes involving occlusion, pixel-level segmentation can provide supplementary information for occlusion.

[0041] in, Figure 2 A schematic diagram of the structure of a pre-built lightweight depth estimation model provided in an embodiment of the present application is shown in FIG. Figure 2As shown, the lightweight depth estimation model 101 is provided with an encoder 1011 and a first decoder 1012 and a second decoder 1013 respectively connected to the encoder 1011, the first decoder 1012 is used to output the depth estimation result, and the second decoder 1013 is used to output the foreground and background segmentation result. By inputting the training data set into the lightweight depth estimation model according to the preset batch size, the depth estimation prediction and the foreground and background segmentation prediction can be performed simultaneously, and then the corresponding loss function value is calculated based on the depth estimation result and the depth estimation map, and the corresponding loss function value is calculated based on the foreground and background segmentation result and the foreground and background segmentation map, and the two loss function values ​​obtained are fused according to the preset ratio to obtain the target loss function value, and finally based on the target loss function value, the network parameters can be iteratively updated through the back propagation algorithm and the related optimization algorithm until the set stop condition is reached, for example, the maximum number of iterations or the loss convergence is reached, and the developer can select the corresponding back propagation algorithm and the related optimization algorithm according to the model network architecture of the actual application scenario, which is not limited in this application. Since the foreground and background segmentation prediction and the depth estimation prediction share encoder parameters, thus, by training the lightweight depth estimation model to achieve a better depth estimation prediction effect, while ensuring the foreground and background segmentation prediction effect, it can be beneficial to assist in optimizing the encoder parameters of the depth estimation prediction, so that the feature extraction of the depth estimation prediction has the ability to distinguish the foreground and background, and can improve the accuracy of the depth estimation. In one embodiment, the joint training of the depth estimation task and the foreground and background segmentation task can be performed based on different learning rates, and the weight update amplitude of the model during the training process can be controlled to adapt to the model performance requirements of different scenes. In one embodiment, the training of the depth estimation task can be performed separately by selecting the network parameters of the second decoder corresponding to the foreground and background segmentation to enhance the learning effect of the depth estimation task. In one embodiment, the joint training of the depth estimation task and the foreground and background segmentation task, as well as the separate training of the depth estimation task, can also be combined to set a multi-stage model training process, gradually improve the model performance, and achieve balance and optimization between different prediction tasks. For example, a two-stage training process of joint training + separate training can be set, and for another example, a three-stage training process of joint training + separate training + joint training can be set, and different training stages can be trained with the same or different learning rates. The above training process is only an exemplary description. Developers can make adaptive settings in the training phase according to the business requirements of the actual application scenario and the model accuracy requirements, and this application does not limit this.

[0042] Step S103: Obtain an image frame to be processed, and input the image frame to be processed into a trained lightweight depth estimation model to obtain a target depth estimation map.

[0043] Among them, the image frame to be processed can be a video image that needs to be depth estimated in a business scenario, and the business scenario can include live video quality assessment, depth of field reconstruction, etc., which is not limited in this application. By using the trained lightweight depth estimation model, the depth of the image frame to be processed can be efficiently and accurately estimated, providing reliable reference depth information for the business scenario. Figure 3 A schematic diagram of the effect of applying the video depth estimation method provided in the embodiment of the present application to perform depth estimation, as shown in FIG. Figure 3 As shown, after the image frame 201 to be processed is input into the trained lightweight depth estimation model 101, a target depth estimation map 202 can be obtained.

[0044] As mentioned above, by obtaining a training data set, the image frames in the training data set are input into the set depth estimation model and the foreground and background segmentation model to obtain the depth estimation map and the foreground and background segmentation map corresponding to the image frame; the pre-constructed lightweight depth estimation model is trained based on the training data set, the depth estimation map and the foreground and background segmentation map, wherein the lightweight depth estimation model includes an encoder and a first decoder and a second decoder respectively connected to the encoder, the first decoder is used to output the depth estimation result, and the second decoder is used to output the foreground and background segmentation result; the image frame to be processed is obtained, and the image frame to be processed is input into the trained lightweight depth estimation model to obtain the target depth estimation map. In the above scheme, by using the open source depth estimation model and foreground and background segmentation model to generate the depth estimation map and foreground and background segmentation map corresponding to the training data set, the training data set can be efficiently labeled, the true value label corresponding to the training data set can be accurately obtained, and the data labeling cost can be reduced; by training the lightweight depth estimation model with decoders respectively corresponding to the depth estimation and foreground and background segmentation, the foreground and background segmentation can be reasonably used to provide semantic information of the foreground and background for the depth estimation, so that the model can better understand the semantic information in the scene, improve the depth estimation model's understanding and prediction capabilities for complex scenes, and improve the accuracy of depth estimation; and the lightweight depth estimation model provided in this embodiment can reduce computing power and storage resource requirements, can be suitable for real-time business scenarios, and meet the deployment requirements of mobile devices.

[0045] Figure 4 A flowchart of a video depth estimation method including a training process of a lightweight depth estimation model provided in an embodiment of the present application. Figure 4 As shown, the following steps are included:

[0046] Step S201: obtain a training data set, input the image frames in the training data set into a set depth estimation model and a foreground and background segmentation model, and obtain a depth estimation map and a foreground and background segmentation map corresponding to the image frame.

[0047] Step S202: input the image frames in the training data set into a pre-built lightweight depth estimation model to obtain a first depth prediction map and a first foreground and background prediction map, wherein the lightweight depth estimation model includes an encoder and a first decoder and a second decoder respectively connected to the encoder, the first decoder is used to output a depth estimation result, and the second decoder is used to output a foreground and background segmentation result.

[0048] The first decoder in the lightweight depth estimation model is used to output a first depth prediction map, and the second decoder is used to output a first foreground and background prediction map.

[0049] Step S203, calculating the first loss function value corresponding to the first depth prediction map and the depth estimation map based on the set loss function, and calculating the second loss function value corresponding to the foreground and background segmentation map and the first foreground and background prediction map.

[0050] In one embodiment, the loss function includes at least one of a structural similarity loss function, an absolute value loss function, a mean square error loss function, and a Laplace loss function. When more than two loss functions are used, the result values ​​calculated using different loss functions are summed to obtain the loss function value. In one embodiment, the training process can simultaneously use a combination of a structural similarity loss function, an absolute value loss function, a mean square error loss function, and a Laplace loss function for loss calculation. Optionally, the use of a structural similarity loss function can effectively consider the structural similarity of the image, ensuring that the depth estimation map generated by the model is closer in structure to the original input image frame, which helps to improve the accuracy and stability of the depth estimation, and makes the depth estimation map generated by the model closer to the real scene. Among them, the specific formula of the structural similarity loss function is as follows:

[0051]

[0052] Among them, x and y are the predicted result map and the true value label map respectively, μ x and μ y is the average value of x and y, σ x and σ y is the standard deviation of x and y, σ xy is the covariance of x and y, C1 and C2 are preset constants.

[0053] Optionally, the absolute value loss function can be used to focus on the average deviation between the predicted value and the true value to improve the robustness of the model. The specific formula of the absolute value loss function is as follows:

[0054]

[0055] Where N is the number of samples, y true is the true value, y predis the predicted value.

[0056] Optionally, the mean square error loss function can be used to focus on the error between the predicted value and the true value. This loss function is differentiable and easy to optimize. The specific formula of the mean square error loss function is as follows:

[0057]

[0058] Where N is the number of samples, y true is the true value, y pred is the predicted value.

[0059] Optionally, the Laplace loss function can be more robust to noise and better preserve the edge information of the image. The specific formula of the Laplace loss function is as follows:

[0060]

[0061] Where N is the number of samples, y true is the true value, y pred is the predicted value, e is the natural base, and σ is the standard deviation.

[0062] In one embodiment, the loss function may be obtained by summing the result values ​​calculated respectively using the structural similarity loss function, the absolute value loss function, the mean square error loss function, and the Laplace loss function, wherein the formula for calculating the loss function is as follows:

[0063] Loss=SSIMLoss+L1Loss+MSELoss+LaplacianLoss,

[0064] Among them, Loss is the loss function value, SSIMLoss is the structural similarity loss function value, L1Loss is the absolute value loss function value, MSELoss is the mean square error loss function value, and LaplacianLoss is the Laplace loss function value. Therefore, by combining different loss functions, it is beneficial for the lightweight depth estimation model in this embodiment to not only accurately fit the real data, but also maintain the structural characteristics of the image, thereby improving the performance and generalization ability of the model.

[0065] Step S204, iteratively updating the network parameters of the lightweight depth estimation model based on the first loss function value, the second loss function value and the preset first learning rate until the loss converges.

[0066] Among them, the present embodiment is to perform joint training of depth estimation task and foreground and background segmentation task, and the network parameters corresponding to the encoder of the lightweight depth estimation model are shared, while the network parameters obtained by the first decoder and the second decoder for depth estimation task and foreground and background segmentation task training are different. Optionally, the loss function value in the training process can be the weighted sum of the first loss function value and the second loss function value, and the specific weight can be adaptively adjusted by the developer according to the actual training effect, which is not limited in this application. According to the loss function value, the iterative update of network parameters can be performed by the back propagation algorithm and the related optimization algorithm until the loss converges. In addition, by setting the first learning rate, the weight update amplitude of the model in the training process can be reasonably controlled. For example, the first learning rate can be set to 0.01 to preliminarily learn the model weights and speed up the convergence speed of the model. Of course, the developer can also perform adaptive settings of other learning rates according to the actual model performance requirements, which is not limited in this application.

[0067] Step S205 : obtaining an image frame to be processed, and inputting the image frame to be processed into a trained lightweight depth estimation model to obtain a target depth estimation map.

[0068] As mentioned above, by jointly training the pre-built lightweight depth estimation model for the depth estimation task and the foreground and background segmentation task, the foreground and background segmentation task can be effectively utilized to assist in optimizing the network parameters of the model, so that the model can reasonably combine the semantic information of the foreground and background, thereby improving the accuracy of depth estimation.

[0069] Figure 5 A flowchart of a video depth estimation method including a multi-stage training process of a lightweight depth estimation model provided in an embodiment of the present application. Figure 5 As shown, the following steps are included:

[0070] Step S301: obtain a training data set, input the image frames in the training data set into the set depth estimation model and foreground and background segmentation model, and obtain a depth estimation map and a foreground and background segmentation map corresponding to the image frame.

[0071] Step S302: input the image frames in the training data set into a pre-built lightweight depth estimation model to obtain a first depth prediction map and a first foreground and background prediction map, wherein the lightweight depth estimation model includes an encoder and a first decoder and a second decoder respectively connected to the encoder, the first decoder is used to output a depth estimation result, and the second decoder is used to output a foreground and background segmentation result.

[0072] Step S303: Calculate the first loss function value corresponding to the first depth prediction map and the depth estimation map based on the set loss function, and calculate the second loss function value corresponding to the foreground and background segmentation map and the first foreground and background prediction map.

[0073] Step S304, iteratively updating the network parameters of the lightweight depth estimation model based on the first loss function value, the second loss function value and the preset first learning rate until the loss converges.

[0074] Step S305 , freeze the network parameters of the second decoder in the lightweight depth estimation model, input the image frames in the training data set into the lightweight depth estimation model, and obtain a second depth prediction map.

[0075] Among them, based on the joint training of the depth estimation task and the foreground and background segmentation task performed in steps S302-S304, by freezing the network parameters of the second decoder in the lightweight depth estimation model and separately training the network parameters of the first decoder, enhanced training for the depth estimation task can be achieved.

[0076] Step S306: Calculate a third loss function value corresponding to the second depth prediction map and the depth estimation map based on the loss function.

[0077] Step S307, iteratively update the remaining network parameters of the lightweight depth estimation model based on the third loss function value and a preset second learning rate until the loss converges, wherein the second learning rate is less than the first learning rate.

[0078] Among them, according to the third loss function value, the network parameters of the encoder and the first decoder can be iteratively updated through the back propagation algorithm and the related optimization algorithm until the loss converges. In addition, the second learning rate is less than the first learning rate. For example, the first learning rate can be set to 0.01, and the second learning rate can be set to 0.001. The model weights are preliminarily learned based on the first learning rate, and the learning effect of the model for the depth estimation task is enhanced based on the second learning rate, thereby improving the accuracy of the depth estimation. Of course, developers can also make adaptive settings of other learning rates according to the actual model performance requirements, which is not limited in this application.

[0079] Step S308: Obtain the image frame to be processed, and input the image frame to be processed into the trained lightweight depth estimation model to obtain a target depth estimation map.

[0080] As described above, by jointly training the lightweight depth estimation model for the depth estimation task and the foreground and background segmentation task based on the first learning rate, the purpose of preliminary learning the model weights can be achieved, and by freezing the network parameters of the second decoder in the lightweight depth estimation model and performing intensive training of the depth estimation task on the lightweight depth estimation model based on the second learning rate, the depth estimation capability of the model can be enhanced and the depth estimation accuracy can be improved.

[0081] Figure 6A flowchart of another method for video depth estimation including a multi-stage training process of a lightweight depth estimation model provided in an embodiment of the present application. Figure 6 As shown, the following steps are included:

[0082] Step S401: obtain a training data set, input the image frames in the training data set into a set depth estimation model and a foreground and background segmentation model, and obtain a depth estimation map and a foreground and background segmentation map corresponding to the image frame.

[0083] Step S402: input the image frames in the training data set into a pre-built lightweight depth estimation model to obtain a first depth prediction map and a first foreground and background prediction map, wherein the lightweight depth estimation model includes an encoder and a first decoder and a second decoder respectively connected to the encoder, the first decoder is used to output a depth estimation result, and the second decoder is used to output a foreground and background segmentation result.

[0084] Step S403: Calculate the first loss function value corresponding to the first depth prediction map and the depth estimation map based on the set loss function, and calculate the second loss function value corresponding to the foreground and background segmentation map and the first foreground and background prediction map.

[0085] Step S404, iteratively updating the network parameters of the lightweight depth estimation model based on the first loss function value, the second loss function value and the preset first learning rate until the loss converges.

[0086] Step S405 , freezing the network parameters of the second decoder in the lightweight depth estimation model, inputting the image frames in the training data set into the lightweight depth estimation model, and obtaining a second depth prediction map.

[0087] Step S406: Calculate a third loss function value corresponding to the second depth prediction map and the depth estimation map based on the loss function.

[0088] Step S407, iteratively update the remaining network parameters of the lightweight depth estimation model based on the third loss function value and a preset second learning rate until the loss converges, wherein the second learning rate is less than the first learning rate.

[0089] Step S408: training the pre-constructed lightweight depth estimation model based on the training data set, the depth estimation map, the foreground and background segmentation map, and a preset third learning rate, wherein the third learning rate is less than the second learning rate.

[0090] Among them, based on the joint training of the depth estimation task and the foreground and background segmentation task performed in steps S302-S304, and the separate training of the depth estimation task performed in steps S305-S307, the joint training of the depth estimation task and the foreground and background segmentation task can be performed with a smaller learning rate to more finely adjust the network parameters of the model.

[0091] Step S409: Obtain the image frame to be processed, and input the image frame to be processed into the trained lightweight depth estimation model to obtain a target depth estimation map.

[0092] As mentioned above, by setting up cross-multi-stage training of joint training and separate training, and gradually optimizing the model by gradually decaying the learning rate, the model parameters can be adjusted more finely, which is conducive to improving the performance of the model in complex scenarios and optimizing the stability of the model.

[0093] Figure 7 A flowchart of a video depth estimation method including a process of generating a training data set is provided in an embodiment of the present application. Figure 7 As shown, the following steps are included:

[0094] Step S501: obtain open source video data and an open source data set, and filter out a target video set from the open source video data based on a preset detection model and a preset category distribution.

[0095] The open source video data may be obtained by crawling Internet videos in advance through crawler scripts, and the open source data sets may include KITTI data sets, NYU Depth V2 data sets, and City Scapes data sets. The preset detection model may be used to identify video scenes, character information, background information, texture information, etc. of the open source video data, and provide accurate reference information for video classification. The preset category distribution may be the scene categories, character action categories, and foreground and background categories that the target video set needs to cover, as well as the quantity requirements of each category to ensure uniform coverage of different categories. The scene categories may include but are not limited to indoor scenes, outdoor scenes, semi-open spaces, etc., and can cover actual scenes with different depth ranges at long and short distances to ensure that the model has the ability to generalize in various real environments and accurately capture depth information at different distances; the character action categories may include but are not limited to common actions in daily life such as singing, chatting, dancing, and cooking to ensure that the model's depth estimation ability for different actions in different scenes is fully trained and evaluated; the foreground and background categories may involve characters, objects, and various relationships between them to ensure that the model can effectively understand and analyze complex scene structures. In one embodiment, the preset detection model may include a scene classification model, a behavior recognition model, a foreground-background segmentation model, and a sharpness estimation model, wherein the scene classification model can be used to classify the video to ensure that the number of different scenes in the target video set is evenly distributed, the behavior recognition model can be used to classify the behavior of the characters in the video to ensure the diversity of the behavior of the characters in the target video set, the foreground-background segmentation model can be used to evaluate the background pixel ratio of the video to ensure the diversity of the background ratio in the target video set, and the sharpness estimation model can be used to evaluate the texture complexity of the video to ensure the diversity of the texture complexity in the target video set. In one embodiment, the number of faces in the video can also be determined by a face detection algorithm, and videos involving multi-person scenes can be removed from the target video set, and the resolution of the video can be determined by a resolution detection algorithm, and videos involving extremely small resolutions can be removed from the target video set to avoid this part of the data affecting the model training effect. In one embodiment, the target video set can also be sharpened to enhance the video clarity, which is conducive to improving the subsequent model training effect.

[0096] Step S502: extract frames from the target video set according to a preset frame interval to obtain multiple video frame sequences.

[0097] Among them, the value range of the preset frame interval can be set between 1 frame and 8 frames, and the developer can make an adaptive selection based on the actual training requirements of the model, and this application does not limit it here.

[0098] Step S503: merge the open source data set and multiple video frame sequences to obtain a training data set.

[0099] Step S504: obtain a training data set, input the image frames in the training data set into the set depth estimation model and foreground and background segmentation model, and obtain a depth estimation map and a foreground and background segmentation map corresponding to the image frame.

[0100] Step S505, training a pre-constructed lightweight depth estimation model based on a training data set, a depth estimation map, and a foreground and background segmentation map, wherein the lightweight depth estimation model includes an encoder and a first decoder and a second decoder respectively connected to the encoder, the first decoder is used to output a depth estimation result, and the second decoder is used to output a foreground and background segmentation result.

[0101] Step S506: Obtain an image frame to be processed, and input the image frame to be processed into the trained lightweight depth estimation model to obtain a target depth estimation map.

[0102] As mentioned above, by building a comprehensive and representative training dataset based on open source video data and open source datasets, it can meet the diverse data needs covering different scenes, different character actions, and different foreground and background, which is conducive to improving the generalization ability and adaptability of the model.

[0103] Figure 8 This is a structural block diagram of a video depth estimation device provided in an embodiment of the present application. The device is configured to execute the video depth estimation method provided in the above embodiment and has the corresponding functional modules and beneficial effects of the execution method. Figure 8 As shown, the device comprises:

[0104] A first acquisition module 301 is configured to acquire a training data set;

[0105] The labeling module 302 is configured to input the image frames in the training data set into the set depth estimation model and the foreground and background segmentation model to obtain the depth estimation map and the foreground and background segmentation map corresponding to the image frame;

[0106] A model training module 303 is configured to train a pre-built lightweight depth estimation model based on a training data set, a depth estimation map, and a foreground-background segmentation map, wherein the lightweight depth estimation model includes an encoder and a first decoder and a second decoder respectively connected to the encoder, the first decoder is used to output a depth estimation result, and the second decoder is used to output a foreground-background segmentation result;

[0107] A second acquisition module 304 is configured to acquire an image frame to be processed;

[0108] The depth estimation module 305 is configured to input the image frame to be processed into the trained lightweight depth estimation model to obtain a target depth estimation map.

[0109] As mentioned above, by obtaining a training data set, the image frames in the training data set are input into the set depth estimation model and the foreground and background segmentation model to obtain the depth estimation map and the foreground and background segmentation map corresponding to the image frame; the pre-constructed lightweight depth estimation model is trained based on the training data set, the depth estimation map and the foreground and background segmentation map, wherein the lightweight depth estimation model includes an encoder and a first decoder and a second decoder respectively connected to the encoder, the first decoder is used to output the depth estimation result, and the second decoder is used to output the foreground and background segmentation result; the image frame to be processed is obtained, and the image frame to be processed is input into the trained lightweight depth estimation model to obtain the target depth estimation map. In the above scheme, by using the open source depth estimation model and foreground and background segmentation model to generate the depth estimation map and foreground and background segmentation map corresponding to the training data set, the training data set can be efficiently labeled, the true value label corresponding to the training data set can be accurately obtained, and the data labeling cost can be reduced; by training the lightweight depth estimation model with decoders respectively corresponding to the depth estimation and foreground and background segmentation, the foreground and background segmentation can be reasonably used to provide semantic information of the foreground and background for the depth estimation, so that the model can better understand the semantic information in the scene, improve the depth estimation model's understanding and prediction capabilities for complex scenes, and improve the accuracy of depth estimation; and the lightweight depth estimation model provided in this embodiment can reduce computing power and storage resource requirements, can be suitable for real-time business scenarios, and meet the deployment requirements of mobile devices.

[0110] In a possible embodiment, the model training module 303 is further configured to:

[0111] Input the image frames in the training data set into a pre-built lightweight depth estimation model to obtain a first depth prediction map and a first foreground and background prediction map;

[0112] Calculate the first loss function value corresponding to the first depth prediction map and the depth estimation map based on the set loss function, and calculate the second loss function value corresponding to the foreground and background segmentation map and the first foreground and background prediction map;

[0113] The network parameters of the lightweight depth estimation model are iteratively updated based on the first loss function value, the second loss function value, and a preset first learning rate until the loss converges.

[0114] In a possible embodiment, the model training module 303 is further configured to:

[0115] Freeze the network parameters of the second decoder in the lightweight depth estimation model, input the image frames in the training data set into the lightweight depth estimation model, and obtain a second depth prediction map;

[0116] Calculate the third loss function value corresponding to the second depth prediction map and the depth estimation map based on the loss function;

[0117] The remaining network parameters of the lightweight depth estimation model are iteratively updated based on the third loss function value and a preset second learning rate until the loss convergence is reached, wherein the second learning rate is less than the first learning rate.

[0118] In a possible embodiment, the model training module 303 is further configured to:

[0119] A pre-built lightweight depth estimation model is trained based on a training data set, a depth estimation map, a foreground-background segmentation map, and a preset third learning rate, wherein the third learning rate is less than the second learning rate.

[0120] In one possible embodiment, the encoder includes multiple lightweight convolution modules, the first decoder and the second decoder respectively include multiple upsampling modules, and each upsampling module includes a convolution layer, a normalization layer, an activation function layer, a convolution gated loop layer and an upsampling layer.

[0121] In a possible embodiment, it further includes a data set generation module configured to:

[0122] Obtain open source video data and open source data sets, and filter out target video sets from the open source video data based on a preset detection model and preset category distribution;

[0123] Extract frames from the target video set according to a preset frame number interval to obtain multiple video frame sequences;

[0124] The open source dataset and multiple video frame sequences are merged to obtain the training dataset.

[0125] In a possible embodiment, the loss function includes at least one of a structural similarity loss function, an absolute value loss function, a mean square error loss function, and a Laplace loss function. When more than two loss functions are used, the result values ​​calculated using different loss functions are summed to obtain the loss function value.

[0126] Fig. 9 A schematic diagram of the structure of a video depth estimation device provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the device includes a processor 401, a memory 402, an input device 403 and an output device 404; the number of processors 401 in the device can be one or more. Fig. 9 A processor 401 is taken as an example; the processor 401, memory 402, input device 403 and output device 404 in the device can be connected by a bus or other means. Fig. 9The example of connection via bus is taken. The memory 402, as a computer-readable storage medium, can be configured to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the video depth estimation method in the embodiment of the present application. The processor 401 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 402, that is, implements the above-mentioned video depth estimation method. The input device 403 can be configured to receive input digital or character information, and generate key signal input related to user settings and function control of the device. The output device 404 may include a display device such as a display screen.

[0127] The embodiment of the present application further provides a non-volatile storage medium containing computer executable instructions, which are configured to execute a video depth estimation method described in the above embodiment when executed by a computer processor, including:

[0128] Obtain a training data set, input the image frames in the training data set into the set depth estimation model and the foreground and background segmentation model, and obtain the depth estimation map and the foreground and background segmentation map corresponding to the image frame; train the pre-built lightweight depth estimation model based on the training data set, the depth estimation map and the foreground and background segmentation map, wherein the lightweight depth estimation model includes an encoder and a first decoder and a second decoder respectively connected to the encoder, the first decoder is used to output the depth estimation result, and the second decoder is used to output the foreground and background segmentation result; obtain the image frame to be processed, input the image frame to be processed into the trained lightweight depth estimation model, and obtain the target depth estimation map.

[0129] It is worth noting that in the embodiment of the above-mentioned video depth estimation device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not configured to limit the protection scope of the embodiments of the present application.

[0130] In some possible implementations, various aspects of the method provided in the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is configured to enable the computer device to perform the steps of the method according to various exemplary embodiments of the present application described above in this specification. For example, the computer device may execute the video depth estimation method described in the embodiment of the present application. The program product may be implemented in any combination of one or more readable media.

Claims

1. A video depth estimation method, characterized in that: include: Obtain a training data set, input the image frames in the training data set into a set depth estimation model and a foreground and background segmentation model, and obtain a depth estimation map and a foreground and background segmentation map corresponding to the image frames; Based on the training data set, the depth estimation map and the foreground and background segmentation map, a pre-constructed lightweight depth estimation model is trained, wherein the lightweight depth estimation model comprises an encoder and a first decoder and a second decoder respectively connected to the encoder, wherein the first decoder is used to output a depth estimation result, and the second decoder is used to output a foreground and background segmentation result; Obtain an image frame to be processed, and input the image frame to be processed into the trained lightweight depth estimation model to obtain a target depth estimation map.

2. The video depth estimation method according to claim 1, characterized in that: The training of the pre-built lightweight depth estimation model based on the training data set, the depth estimation map and the foreground and background segmentation map includes: Inputting the image frames in the training data set into a pre-built lightweight depth estimation model to obtain a first depth prediction map and a first foreground and background prediction map; Calculate the first loss function values ​​corresponding to the first depth prediction map and the depth estimation map based on the set loss function, and calculate the second loss function values ​​corresponding to the foreground and background segmentation map and the first foreground and background prediction map; The network parameters of the lightweight depth estimation model are iteratively updated based on the first loss function value, the second loss function value and a preset first learning rate until loss convergence is reached.

3. The video depth estimation method according to claim 2, characterized in that: After iteratively updating the network parameters of the lightweight depth estimation model based on the first loss function value, the second loss function value, and a preset first learning rate until loss convergence is reached, the method further includes: Freezing network parameters of a second decoder in the lightweight depth estimation model, inputting image frames in the training data set into the lightweight depth estimation model to obtain a second depth prediction map; Calculate a third loss function value corresponding to the second depth prediction map and the depth estimation map based on the loss function; The remaining network parameters of the lightweight depth estimation model are iteratively updated based on the third loss function value and a preset second learning rate until loss convergence is reached, and the second learning rate is less than the first learning rate.

4. The video depth estimation method according to claim 3, characterized in that: After iteratively updating the remaining network parameters of the lightweight depth estimation model based on the third loss function value and the preset second learning rate until the loss converges, the method further includes: A pre-constructed lightweight depth estimation model is trained based on the training data set, the depth estimation map, the foreground and background segmentation map, and a preset third learning rate, where the third learning rate is less than the second learning rate.

5. The video depth estimation method according to claim 1, characterized in that: The encoder includes multiple lightweight convolution modules, the first decoder and the second decoder respectively include multiple upsampling modules, each of the upsampling modules includes a convolution layer, a normalization layer, an activation function layer, a convolution gated loop layer and an upsampling layer.

6. The video depth estimation method according to claim 1, characterized in that: Before obtaining the training data set, the method further includes: Obtaining open source video data and an open source data set, and filtering out a target video set from the open source video data based on a preset detection model and a preset category distribution; Performing frame extraction processing on the target video set according to a preset frame number interval to obtain multiple video frame sequences; The open source data set and the multiple video frame sequences are combined to obtain a training data set.

7. The video depth estimation method according to claim 2, characterized in that: The loss function includes at least one of a structural similarity loss function, an absolute value loss function, a mean square error loss function, and a Laplace loss function. When more than two loss functions are used, the result values ​​calculated using different loss functions are summed to obtain the loss function value.

8. A video depth estimation device, characterized in that: include: A first acquisition module is configured to acquire a training data set; A labeling module is configured to input the image frames in the training data set into a set depth estimation model and a foreground and background segmentation model to obtain a depth estimation map and a foreground and background segmentation map corresponding to the image frames; A model training module, configured to train a pre-constructed lightweight depth estimation model based on the training data set, the depth estimation map and the foreground and background segmentation map, wherein the lightweight depth estimation model includes an encoder and a first decoder and a second decoder respectively connected to the encoder, the first decoder is used to output a depth estimation result, and the second decoder is used to output a foreground and background segmentation result; A second acquisition module is configured to acquire an image frame to be processed; The depth estimation module is configured to input the image frame to be processed into the trained lightweight depth estimation model to obtain a target depth estimation map.

9. A video depth estimation device, the device comprising: one or more processors; A storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the video depth estimation method according to any one of claims 1 to 7.

10. A non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the video depth estimation method according to any one of claims 1 to 7 when executed by a computer processor.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the video depth estimation method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Image processing method and device, electronic equipment and computer readable storage medium

    CN113313646A

  • Fan blade damage identification method and system based on multi-task association

    CN116503612A

  • Semantic-guided monocular depth estimation method

    CN116523987A

  • Student model training method and device, material processing method and device and electronic equipment

    CN117494762A