Dynamic scene depth estimation method and device based on scene flow

Abstract

The invention provides a dynamic scene depth estimation method and device based on a scene flow. The method comprises the following steps: acquiring an RGB image of a target scene collected by a camera at the current moment; performing non-parametric distortion on the RGB image at the current moment to obtain a plurality of distorted images at the current moment; processing the RGB image at the current moment based on the plurality of distorted images at the previous moment and the plurality of distorted images at the current moment to obtain feature representation of the RGB image at the current moment fused with multi-view information; processing the RGB image at the current moment and the feature representation of the fused multi-view information of the RGB image at the current moment by using a pre-trained depth estimation network to obtain a depth estimation map of the RGB image at the current moment; wherein the depth estimation network is obtained based on scene flow training. According to the invention, the accuracy and robustness of depth estimation in a dynamic scene are improved.

Description

Technical Field

[0001] This application relates to the field of dynamic depth estimation technology, and in particular to a dynamic scene depth estimation method and apparatus based on scene flow. Background Technology

[0002] Current depth estimation methods for dynamic scenes have the following drawbacks:

[0003] Reliance on dense viewpoint input: Some methods rely on high-cost multi-camera arrays or continuous video input for training and inference, which is difficult to adapt to the situation of limited equipment and sparse viewpoints in actual deployment.

[0004] Uncontrolled mask quality: When dealing with phenomena such as non-rigid motion, severe occlusion, or object disappearance, existing methods often need to rely on semantic segmentation or masking mechanisms, which increases model complexity and has the problem of unstable mask generation quality.

[0005] Weak temporal consistency: Some methods do not explicitly model motion changes in the time dimension, resulting in phenomena such as depth jumps and image flickering in long videos, affecting rendering continuity and realism.

[0006] Lack of ability to model 3D motion fields: Although some scene flow modeling methods exist, they usually have limited ability to model non-rigid body motion and lack the ability to flexibly express the complex motion structure of the scene.

[0007] Sensitive to camera pose accuracy: Most multi-view methods perform calculations under the assumption of accurate pose, but in practical applications, pose estimation errors often affect matching quality and geometric consistency. Summary of the Invention

[0008] In view of this, this application provides a dynamic scene depth estimation method and apparatus based on scene flow. This method can adapt to sparse viewpoint input and has the ability to process depth estimation of complex dynamic scenes. It can improve the robustness and visual quality of dynamic new viewpoint synthesis, thereby solving the above-mentioned technical problems of existing depth estimation methods.

[0009] In a first aspect, embodiments of this application provide a dynamic scene depth estimation method based on scene flow, including:

[0010] Acquire the RGB image of the target scene captured by the camera at the current moment;

[0011] The RGB image at the current moment is non-parametrically warped to obtain multiple warped images at the current moment;

[0012] Based on multiple distorted images from the previous time step and multiple distorted images from the current time step, the RGB image at the current time step is processed to obtain a feature representation of the fused multi-view information of the RGB image at the current time step.

[0013] The depth estimation network is pre-trained to process the feature representation of the RGB image at the current time and the fused multi-view information of the RGB image at the current time, so as to obtain the depth estimation map of the RGB image at the current time; wherein, the depth estimation network is trained based on scene flow.

[0014] In one possible implementation, the RGB image at the current moment is non-parametrically warped to obtain multiple warped images at the current moment; including:

[0015] Construct a set of depth hypotheses called Depths, which includes count depth planes uniformly sampled between 3 meters and 40 meters, where the i-th depth plane is denoted as d. i , 1≤i≤count;

[0016] Based on the current RGB image and the depth hypothesis set Depths, construct a pixel-level depth hypothesis tensor D containing count levels. t ;

[0017] The pixel-level depth assumption tensor D t The i-th level D t,i Mapped to 3D point cloud data

[0018]

[0019] Where K is the camera's intrinsic parameter matrix, and t is the current time.

[0020] Based on 3D point cloud data Given the camera parameters, calculate the i-th distorted image. Its pixel (u,v) is:

[0021]

[0022] Among them, (X) c ,Y c Z c (3D point cloud) The three-dimensional coordinates of each point in the camera coordinate system, f x and f y For the camera's focal length, (c x ,c y () represents the two-dimensional coordinates of the main point.

[0023] In one possible implementation, based on multiple distorted images from the previous time step and multiple distorted images from the current time step, the RGB image at the current time step is processed to obtain a feature representation of the fused multi-view information of the RGB image at the current time step; including:

[0024] Calculate the similarity value SSIM i,t :

[0025]

[0026] Wherein, the RGB image I at the current moment t The dimensions are H×W; Let i be the i-th distorted image from the previous time step; SSIM(,) is the structural similarity calculation function;

[0027] Using convolutional layers to apply SSIM (similarity) i,t Perform a convolution operation to obtain the image difference tensor Diff. i,t :

[0028] Based on image difference tensor Diff i,t Calculate the weight of each pixel in different depth planes. i,t :

[0029]

[0030] Where α is a hyperparameter used to control the decay rate of the exponential function; min i Diff i,t Diff i,t The minimum value in;

[0031] Calculate the multi-view cost volume C, which represents the matching cost of pixels at each depth level. multi :

[0032]

[0033] The multi-view cost volume C is processed through three layers of consecutive convolution operations. multi Processing is performed to generate multi-frame cue representation F multi1 The size is h×w, where h=H / 4, w=W / 4;

[0034] F multi1 The three sets of parallel convolutional layers are mapped to query feature Q, key feature K, and value feature V, respectively.

[0035] Attention graph Att is calculated using the dot product attention mechanism. map , is used to represent the correlation strength between different spatial locations in a multi-view cost volume, and its expression is:

[0036] Attmap =Softmax(Q⊙K)

[0037] Where the symbol ⊙ represents element-wise matrix multiplication; Softmax(·) is the activation function;

[0038] Using Att map We weight the value feature V to obtain the attention-enhancing feature Att. multi :

[0039] Att multi =Softmax(V⊙Att) map )

[0040] Attention-enhancing features Att multi and multi-perspective cost body C multi The images are fused to obtain the RGB image I at the current moment. t Feature representation F that integrates multi-perspective information use_multi :

[0041] F use_multi =Conv(Upsample(Att) multi ))+γ·Conv(C multi )

[0042] Among them, Upsample(Att) multi ) for Att multi Upsampling is performed using bilinear interpolation to obtain attention-enhanced features of size H×W, where γ is a parameter.

[0043] In one possible implementation, the depth estimation network includes an image backbone network, a stitching unit, a multilayer encoder, and a multilayer decoder connected in sequence.

[0044] The pre-trained depth estimation network is used to process the feature representation of the current RGB image and the fused multi-view information of the current RGB image to obtain the depth estimation map of the current RGB image, including:

[0045] Extracting the RGB image I at the current time using an image backbone network t Image features at different scales in group J: 1≤j≤J, where J is the number of scales;

[0046] Using splicing units to fuse multi-view information feature representation F use_multi with I t The fused features F are obtained by concatenating the data along the channel dimension. concat ;

[0047] F fusion feature F concatThe input is a multi-layer encoder, which performs encoding processing to obtain the first intermediate feature representation E output by each layer encoder. j :

[0048] E j =Encoder j (E j-1 )

[0049] Among them, Encoder j Let E represent the j-th layer encoder. j E0 represents the output of the (j-1)th layer encoder, which is the input of the j-th layer encoder, and E0 = F. concat ;

[0050] The J sets of image features at different scales and the first intermediate features output from each encoder layer are input into a multi-layer decoder for decoding processing to obtain the second intermediate features D output from each decoder layer. j :

[0051]

[0052] Among them, Decoder j Denotes the j-th layer decoder, DE0 = E J ;

[0053] The final output DE of the multi-layer decoder J Perform a single-layer convolution operation to obtain the depth estimate map of the RGB image at the current time step.

[0054] In one possible implementation, the method further includes:

[0055] Establish a training set, which includes K consecutive RGB image samples;

[0056] Each frame of RGB image samples is subjected to non-parametric distortion to obtain multiple distorted images for each frame;

[0057] Based on multiple distorted images of the (k-1)th frame and multiple distorted images of the (k+1)th frame, the RGB image sample of the kth frame is processed to obtain the feature representation of the fused multi-view information of the RGB image sample of the kth frame, where 2≤k≤K-1.

[0058] The feature representation of the fused multi-view information of the k-th frame RGB image sample is processed using a depth estimation network to obtain the depth map Dep of the k-th frame RGB image sample. k ;

[0059] The feature representation of the fused multi-view information of the (k+1)th frame RGB image sample is processed using a depth estimation network to obtain the depth map Dep of the (k+1)th frame RGB image sample.k+1 ;

[0060] Dep based on the RGB image samples of the k-th frame k Dep and the depth map of the (k+1)th frame RGB image samples k+1 The optical flow loss L is calculated using the scene flow guidance method. optical and parallax loss L disparity ;

[0061] Calculate the scene flow loss value L scene_flow :

[0062] L scene_flow =L optical +L disparity

[0063] Using scene flow loss value L scene_flow Update the parameters of the depth estimation network.

[0064] In one possible implementation, the depth map Dep is based on the RGB image samples of the k-th frame. k Dep and the depth map of the (k+1)th frame RGB image samples k+1 The optical flow loss L is calculated using the scene flow guidance method. optical and parallax loss L disparity ;include:

[0065] The depth map Dep of the RGB image samples of the k-th frame k Mapping to the camera coordinate system yields the point cloud sample for the k-th frame.

[0066]

[0067]

[0068] Among them, K -1 Let K be the inverse of the camera's intrinsic parameter matrix K; R is the rotation matrix R from the camera coordinate system to the world coordinate system in the k-th frame. k The transpose of T; k Let be the translation vector of the camera in the world coordinate system for the k-th frame;

[0069] The depth map Dep of the (k+1)th frame RGB image samples k+1 Mapping to the camera coordinate system, we obtain the point cloud sample of the (k+1)th frame.

[0070]

[0071] in, R is the rotation matrix R from the camera coordinate system to the world coordinate system in the (k+1)th frame.k+1 The transpose of T; k+1 Let be the translation vector of the camera in the world coordinate system for the (k+1)th frame;

[0072] From the point cloud sample of the kth frame Select N points 1≤n≤N; from the (k+1)th frame point cloud sample Select the corresponding N points Form N point pairs

[0073] The scene flow estimation network is used to process N point pairs to obtain the 3D scene flow SF of the nth point pair. k→k+1,n , 1≤n≤N; SF k→k+1,n Including the nth point of the point cloud sample in the kth frame The offset of the three position coordinates;

[0074] The nth point of the point cloud sample in the kth frame. 3D coordinates Add SF k→k+1,n To obtain the nth point 3D coordinates of the point cloud sample in frame k+1 Will Projecting the image onto the (k+1)th frame of the RGB image sample yields the two-dimensional coordinates of the nth point in the (k+1)th frame of the RGB image sample.

[0075] Using a scene flow estimation network to sample the RGB image of the k-th frame The pixel coordinates of the nth point are processed to obtain its pixel coordinates in the (k+1)th frame of the RGB image sample.

[0076] Calculate optical flow loss L optical :

[0077]

[0078] Calculate disparity loss L disparity :

[0079]

[0080] Where, d k+1,n For point cloud samples from frame k+1 Extract the depth value of the nth point, Dep k+1,n To extract from the depth map k+1 The depth value of the nth point is extracted.

[0081] Secondly, embodiments of this application provide a dynamic scene depth estimation device based on scene flow, comprising:

[0082] The acquisition unit is used to acquire the RGB image of the target scene captured by the camera at the current moment;

[0083] The first processing unit is used to perform non-parametric distortion on the RGB image at the current moment to obtain multiple distorted images at the current moment.

[0084] The second processing unit is used to process the RGB image at the current time based on multiple distorted images from the previous time and multiple distorted images at the current time, so as to obtain the feature representation of the fused multi-view information of the RGB image at the current time.

[0085] The depth estimation unit is used to process the feature representation of the RGB image at the current time and the fused multi-view information of the RGB image at the current time using a pre-trained depth estimation network to obtain the depth estimation map of the RGB image at the current time; wherein, the depth estimation network is trained based on scene flow.

[0086] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of embodiments of this application.

[0087] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the methods of embodiments of this application.

[0088] This application improves the accuracy and robustness of depth estimation in dynamic scenes. Attached Figure Description

[0089] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0090] Figure 1 A flowchart illustrating the dynamic scene depth estimation method based on scene flow provided in this application embodiment;

[0091] Figure 2 A flowchart illustrating the processing of the scene flow estimation network provided in this application embodiment;

[0092] Figure 3 A functional structure diagram of the scene flow-based dynamic scene depth estimation device provided in the embodiments of this application;

[0093] Figure 4 A functional structure diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0094] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0095] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0096] First, a brief introduction to the design concept of the embodiments of this application will be given.

[0097] Existing techniques in the field of dynamic depth estimation mainly fall into two categories: learning-based depth estimation methods and geometry-based multi-view modeling methods.

[0098] Early learning-based methods largely relied on supervised learning, training deep networks using a large number of labeled depth maps. While these methods achieved high accuracy, they were highly dependent on data and had limited generalization ability. In recent years, self-supervised methods have gradually emerged, automatically extracting depth information from videos through mechanisms such as image reconstruction loss and edge smoothing constraints, exhibiting better adaptability and scalability. Other methods introduce semantic masks or optical flow estimation to handle dynamic regions, improving the robustness of depth estimation.

[0099] In multi-view-based geometric estimation methods, the mainstream approaches are multi-view stereo vision and scene flow estimation. Multi-view stereo vision methods reconstruct scene depth by achieving geometric consistency across multiple camera views, making them suitable for static environments. However, to handle the more common dynamic scenes in reality, some methods combine time-series modeling and scene flow networks to attempt to model motion fields in 3D space for depth estimation. Furthermore, methods based on neural implicit representations, such as NeRF and its extended versions, are also used for modeling and rendering dynamic scenes.

[0100] A typical dynamic view synthesis system consists of multiple modules, including feature extraction, depth estimation, view transformation, and image rendering. The depth estimation module is usually built on a convolutional neural network and incorporates camera pose and reprojection mechanisms to achieve pixel matching between different viewpoints. To improve the quality of the synthesized image, spatial or temporal feature fusion techniques, such as feature warping and spatiotemporal consistency constraints, are often used to enhance the continuity and realism of image generation. The following are representative methods:

[0101] Lite-Mono network: It transforms the depth estimation task into an image reconstruction task, guides the learning of deep information based on reconstruction error, and proposes a hybrid structure by combining CNN and Vision Transformer to achieve lightweight model.

[0102] RM-Depth Network: Explicitly models scene flow and object motion, achieving robust handling of complex motions in dynamic scenes. It uses scene flow to estimate object motion and self-motion, mitigating the impact of object motion by estimating the 3D motion field, depth, and self-motion of moving objects.

[0103] Multi-task networks: In recent years, multi-task network structures have emerged that jointly model depth estimation, optical flow, normal vector estimation and semantic segmentation, and enhance the synergistic effect of each task through feature sharing mechanisms.

[0104] This application addresses the issue of uncontrolled mask quality in dynamic scenes: To improve the accuracy of depth estimation in dynamic regions, it considers the concept of "spatiotemporal geometric information guidance" and employs a dynamic region 3D motion field modeling method. By estimating the 3D motion field of the dynamic region through a scene flow estimation network, it can eliminate dependence on the mask. Finally, it considers using a spatiotemporally geometrically consistent depth estimation algorithm. By modeling the 3D motion field, it can better handle dynamic regions and effectively guide depth estimation.

[0105] This application achieves self-supervised depth estimation by introducing scene flow loss, utilizing the geometric consistency between image frames as a supervision signal, without relying on true depth labels. The scene flow loss consists of optical flow consistency loss and disparity consistency loss. The former measures the deviation in image space between the predicted 2D optical flow and the optical flow projected from the 3D scene flow, while the latter compares the difference between the depth values ​​predicted by the depth estimation module and the sparse depth generated from the 3D point cloud. Through this loss function, the model can effectively learn depth information on unlabeled data by constraining the geometric and motion consistency between different time frames, thus achieving self-supervised training.

[0106] This application improves the accuracy and robustness of depth estimation in dynamic scenes, and is applicable to fields such as autonomous driving and robot perception.

[0107] After introducing the application scenarios and design concepts of the embodiments of this application, the technical solutions provided by the embodiments of this application will be described below.

[0108] like Figure 1 As shown, this application provides a dynamic scene depth estimation method based on scene flow, including:

[0109] Step 101: Acquire the RGB image of the target scene captured by the camera at the current moment;

[0110] Step 102: Perform non-parametric warping on the RGB image at the current moment to obtain multiple warped images at the current moment;

[0111] Step 103: Based on multiple distorted images from the previous time step and multiple distorted images from the current time step, process the RGB image from the current time step to obtain the feature representation of the fused multi-view information of the RGB image from the current time step.

[0112] Step 104: Use the pre-trained depth estimation network to process the feature representation of the RGB image at the current time and the fused multi-view information of the RGB image at the current time to obtain the depth estimation map of the RGB image at the current time; wherein, the depth estimation network is trained based on scene flow.

[0113] In some embodiments, the RGB image at the current moment is non-parametrically warped to obtain multiple warped images at the current moment; including:

[0114] Construct a set of depth hypotheses called Depths, which includes count depth planes uniformly sampled between 3 meters and 40 meters, where the i-th depth plane is denoted as d. i , 1≤i≤count;

[0115] Based on the current RGB image and the depth hypothesis set Depths, construct a pixel-level depth hypothesis tensor D containing count levels. t ;

[0116] The pixel-level depth assumption tensor D t The i-th level D t,i Mapped to 3D point cloud data

[0117]

[0118] Where K is the camera's intrinsic parameter matrix, and t is the current time.

[0119] Based on 3D point cloud data Given the camera parameters, calculate the i-th distorted image. Its pixel (u,v) is:

[0120]

[0121] Among them, (X) c ,Y c Z c (3D point cloud) The three-dimensional coordinates of each point in the camera coordinate system, f x and f y For the camera's focal length, (c x ,c y () represents the two-dimensional coordinates of the main point.

[0122] In some embodiments, based on multiple distorted images from the previous time step and multiple distorted images from the current time step, the RGB image at the current time step is processed to obtain a feature representation of the RGB image at the current time step that fuses multi-view information; including:

[0123] Calculate the similarity value SSIM i,t :

[0124]

[0125] Wherein, the RGB image I at the current moment t The dimensions are H×W; Let i be the i-th distorted image from the previous time step; SSIM(,) is the structural similarity calculation function;

[0126] Using convolutional layers to apply SSIM (similarity) i,t Perform a convolution operation to obtain the image difference tensor Diff. i,t :

[0127] Based on image difference tensor Diff i,t Calculate the weight of each pixel in different depth planes. i,t :

[0128]

[0129] Where α is a hyperparameter used to control the decay rate of the exponential function; min i Diff i,t Diff i,t The minimum value in;

[0130] Calculate the multi-view cost volume C, which represents the matching cost of pixels at each depth level. multi :

[0131]

[0132] The multi-view cost volume C is processed through three layers of consecutive convolution operations. multi Processing is performed to generate multi-frame cue representation F multi1 The size is h×w, where h=H / 4, w=W / 4;

[0133] F multi1 The three sets of parallel convolutional layers are mapped to query feature Q, key feature K, and value feature V, respectively.

[0134] Attention graph Att is calculated using the dot product attention mechanism. map , is used to represent the correlation strength between different spatial locations in a multi-view cost volume, and its expression is:

[0135] Att map =Softmax(Q⊙K)

[0136] Where the symbol ⊙ represents element-wise matrix multiplication; Softmax(·) is the activation function;

[0137] Using Att map We weight the value feature V to obtain the attention-enhancing feature Att. multi :

[0138] Att multi =Softmax(V⊙Att) map )

[0139] Attention-enhancing features Att multi and multi-perspective cost body C multi The images are fused to obtain the RGB image I at the current moment. t Feature representation F that integrates multi-perspective information use_multi :

[0140] F use_multi =Conv(Upsample(Att) multi ))+γ·Conv(C multi )

[0141] Among them, Upsample(Att) multi ) for Att multi Bilinear interpolation is used for upsampling to obtain attention-enhanced features of size H×W, where γ is a parameter used to adjust the proportion of the original cost volume features in the fusion process.

[0142] Feature representation F that integrates multi-perspective information use_multi It can effectively improve the semantic consistency and feature representation ability across perspectives, providing more discriminative fusion features for subsequent depth estimation or 3D reconstruction.

[0143] In some embodiments, the depth estimation network includes an image backbone network, a stitching unit, a multilayer encoder, and a multilayer decoder connected in sequence;

[0144] The pre-trained depth estimation network is used to process the feature representation of the current RGB image and the fused multi-view information of the current RGB image to obtain the depth estimation map of the current RGB image, including:

[0145] Extracting the RGB image I at the current time using an image backbone network t Image features at different scales in group J: 1≤j≤J, where J is the number of scales;

[0146] Using splicing units to fuse multi-view information feature representation F use_multi with I t The fused features F are obtained by concatenating the data along the channel dimension. concat ;

[0147] F fusion feature F concat The input is a multi-layer encoder, which performs encoding processing to obtain the first intermediate feature representation E output by each layer encoder. j :

[0148] E j =Encoder j (E j-1 )

[0149] Among them, Encoder j Let E represent the j-th layer encoder. j E0 represents the output of the (j-1)th layer encoder, which is the input of the j-th layer encoder, and E0 = F. concat ;

[0150] The J sets of image features at different scales and the first intermediate features output from each encoder layer are input into a multi-layer decoder for decoding processing to obtain the second intermediate features D output from each decoder layer. j :

[0151]

[0152] Among them, Decoder j Denotes the j-th layer decoder, DE0 = E J ;

[0153] The final output DE of the multi-layer decoder J Perform a single-layer convolution operation to obtain the depth estimate map of the RGB image at the current time step.

[0154] This application introduces the fusion of multi-scale image features and coded features during the decoding process, which not only preserves the spatial structure information of high-resolution images, but also improves the accuracy and robustness of depth prediction.

[0155] In some embodiments, the method further includes:

[0156] Establish a training set, which includes K consecutive RGB image samples;

[0157] Each frame of RGB image samples is subjected to non-parametric distortion to obtain multiple distorted images for each frame;

[0158] Based on multiple distorted images of the (k-1)th frame and multiple distorted images of the (k+1)th frame, the RGB image sample of the kth frame is processed to obtain the feature representation of the fused multi-view information of the RGB image sample of the kth frame, where 2≤k≤K-1.

[0159] The feature representation of the fused multi-view information of the k-th frame RGB image sample is processed using a depth estimation network to obtain the depth map Dep of the k-th frame RGB image sample. k ;

[0160] The feature representation of the fused multi-view information of the (k+1)th frame RGB image sample is processed using a depth estimation network to obtain the depth map Dep of the (k+1)th frame RGB image sample. k+1 ;

[0161] Dep based on the RGB image samples of the k-th frame k Dep and the depth map of the (k+1)th frame RGB image samples k+1 The optical flow loss L is calculated using the scene flow guidance method. optical and parallax loss L disparity ;

[0162] Calculate the scene flow loss value L scene_flow :

[0163] L scene_flow =L optical +L disparity

[0164] Using scene flow loss value L scene_flow Update the parameters of the depth estimation network.

[0165] Optical flow loss measures the deviation between the point location projected from the scene flow into the 2D image space and the pixel coordinates estimated by optical flow. Parallax loss constrains the difference between the sparse depth values ​​generated by the scene flow and the sparse depth values ​​generated by the depth estimation module combined with optical flow.

[0166] Through the above loss function design, this embodiment effectively enhances the model's ability to learn about three-dimensional geometric changes in dynamic scenes and improves the generalization of depth estimation without the need for external supervision information.

[0167] In some embodiments, the depth map Dep is based on the RGB image samples of the k-th frame. k Dep and the depth map of the (k+1)th frame RGB image samples k+1 The optical flow loss L is calculated using the scene flow guidance method. optical and parallax loss L disparity ;include:

[0168] The depth map Dep of the RGB image samples of the k-th frame k Mapping to the camera coordinate system yields the point cloud sample for the k-th frame.

[0169]

[0170] Among them, K -1 Let K be the inverse of the camera's intrinsic parameter matrix K; R is the rotation matrix R from the camera coordinate system to the world coordinate system in the k-th frame. k The transpose of T; k Let be the translation vector of the camera in the world coordinate system for the k-th frame;

[0171] The depth map Dep of the (k+1)th frame RGB image samples k+1 Mapping to the camera coordinate system, we obtain the point cloud sample of the (k+1)th frame.

[0172]

[0173] in, R is the rotation matrix R from the camera coordinate system to the world coordinate system in the (k+1)th frame. k+1 The transpose of T; k+1 Let be the translation vector of the camera in the world coordinate system for the (k+1)th frame;

[0174] From the point cloud sample of the kth frame Select N points 1≤n≤N; from the (k+1)th frame point cloud sample Select the corresponding N points Form N point pairs

[0175] The scene flow estimation network is used to process N point pairs to obtain the 3D scene flow SF of the nth point pair. k→k+1,n , 1≤n≤N; SF k→k+1,n Including the nth point of the point cloud sample in the kth frame The offset of the three position coordinates;

[0176] The nth point of the point cloud sample in the kth frame. 3D coordinates Add SF k→k+1,n To obtain the nth point 3D coordinates of the point cloud sample in frame k+1 Will Projecting the image onto the (k+1)th frame of the RGB image sample yields the two-dimensional coordinates of the nth point in the (k+1)th frame of the RGB image sample.

[0177] Using a scene flow estimation network to sample the RGB image of the k-th frame The pixel coordinates of the nth point are processed to obtain its pixel coordinates in the (k+1)th frame of the RGB image sample.

[0178] Calculate optical flow loss L optical :

[0179]

[0180] Calculate disparity loss L disparity :

[0181]

[0182] Where, d k+1,n For point cloud samples from frame k+1 Extract the depth value of the nth point, Dep k+1,n To extract from the depth map k+1 The depth value of the nth point is extracted.

[0183] In the process of dynamic scene depth estimation, in order to effectively handle the 3D motion information of dynamic regions, a scene flow estimation network is proposed to model the 3D motion of dynamic regions.

[0184] like Figure 2 As shown, the scene flow estimation network in this embodiment includes a superpoint generation and iterative update mechanism. Utilizing the characteristic that superpoints can sense similar motion within a local area, initial superpoints are first generated based on the initial scene flow. Subsequently, the "superpoint-guided scene flow optimization" and "scene flow-guided superpoint generation" modules are executed sequentially for iterative updates to improve scene flow accuracy.

[0185] In the initialization phase, a three-layer convolutional network is used to extract features of adjacent point clouds, calculate the feature similarity matrix, and combine it with the three-dimensional spatial similarity to generate a scene flow weight matrix, thereby predicting the corresponding points and their features and obtaining the initial scene flow estimation results.

[0186] In the "scene flow-guided superpoint generation" stage, the source point cloud is first downsampled to obtain the cluster index, cluster center features, and coordinates. Then, the index of each point relative to its cluster center is obtained through K-nearest neighbor query. Further, the feature differences and coordinate differences between each point and its corresponding superpoint are compared. These differences are then encoded through a convolutional layer and summed to calculate the bidirectional association weights, which are then normalized to obtain the association matrix between points and superpoints. Using this association matrix, the point's features, coordinates, and scene flow information are updated to the corresponding superpoint.

[0187] In the "Superpoint-Guided Scene Flow Optimization" stage, a GRU module is used to refine the scene flow based on the current scene flow, point features, and association graph. Specifically, this includes: transforming the point cloud based on the current scene flow and calculating local correlation features; constructing the GRU input by combining input features, correlation features, the current scene flow, and association information; updating the hidden state through GRU and generating a residual scene flow, which is then normalized and added to the current scene flow to obtain the optimized updated scene flow. After three iterations, the final scene flow estimation result is output.

[0188] Based on the same inventive concept, this application provides an apparatus, see below. Figure 3 As shown, the apparatus 200 provided in this application embodiment includes at least:

[0189] The acquisition unit 201 is used to acquire the RGB image of the target scene captured by the camera at the current moment;

[0190] The first processing unit 202 is used to perform non-parametric distortion on the RGB image at the current moment to obtain multiple distorted images at the current moment;

[0191] The second processing unit 203 is used to process the RGB image at the current time based on multiple distorted images at the previous time and multiple distorted images at the current time, so as to obtain the feature representation of the fused multi-view information of the RGB image at the current time.

[0192] The depth estimation unit 204 is used to process the feature representation of the RGB image at the current time and the fused multi-view information of the RGB image at the current time using a pre-trained depth estimation network to obtain a depth estimation map of the RGB image at the current time; wherein, the depth estimation network is trained based on scene flow.

[0193] It should be noted that the principle of the scene flow-based dynamic scene depth estimation device 200 provided in this application embodiment to solve the technical problem is similar to the method provided in this application embodiment. Therefore, the implementation of the scene flow-based dynamic scene depth estimation device 200 provided in this application embodiment can refer to the implementation of the method provided in this application embodiment, and the repeated parts will not be described again.

[0194] Based on the same inventive concept, this application also provides an electronic device, see reference. Figure 4 As shown, the electronic device 300 provided in this application embodiment includes at least: a processor 301, a memory 302, and a computer program stored on the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, it implements the dynamic scene depth estimation method based on scene flow provided in this application embodiment.

[0195] The electronic device 300 provided in this application embodiment may further include a bus 303 connecting different components (including processor 301 and memory 302). The bus 303 represents one or more types of bus structures, including memory bus, peripheral bus, local area bus, etc.

[0196] The memory 302 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 3021 and / or cache memory 3022, and may further include read-only memory (ROM) 3023.

[0197] The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, including but not limited to: an operating subsystem, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0198] Electronic device 300 can also communicate with one or more external devices 304 (e.g., keyboard, remote control, etc.), and with one or more devices that enable a user to interact with electronic device 300 (e.g., mobile phone, computer, etc.), and / or with any device that enables electronic device 300 to communicate with one or more other electronic devices 300 (e.g., router, modem, etc.). This communication can be performed through input / output (I / O) interface 305. Furthermore, electronic device 300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 306. Figure 4 As shown, network adapter 306 communicates with other modules of electronic device 300 via bus 303. It should be understood that, although... Figure 4As not shown, other hardware and / or software modules may be used in conjunction with electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) subsystems, tape drives, and data backup storage subsystems.

[0199] It should be noted that, Figure 4 The electronic device 300 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0200] This application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the scene flow-based dynamic scene depth estimation method provided in this application. Specifically, the executable program can be built into or installed in the electronic device 300, so that the electronic device 300 can implement the scene flow-based dynamic scene depth estimation method provided in this application by executing the built-in or installed executable program.

[0201] The scene flow-based dynamic scene depth estimation method provided in this application embodiment can also be implemented as a program product. The program product includes program code. When the program product can run on the electronic device 300, the program code is used to make the electronic device 300 execute the scene flow-based dynamic scene depth estimation method provided in this application embodiment.

[0202] The program product provided in this application embodiment can be any combination of one or more readable media, wherein the readable media can be a readable signal medium or a readable storage medium, and the readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0203] The program product provided in this application embodiment can be a CD-ROM and include program code, and can also run on a computing device. However, the program product provided in this application embodiment is not limited thereto. In this application embodiment, the readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0204] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0205] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0206] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of this application do not depart from the spirit and scope of the technical solutions of this application, and should all be covered within the scope of the claims of this application.

Claims

1. A dynamic scene depth estimation method based on scene flow, characterized in that, include: Acquire the RGB image of the target scene captured by the camera at the current moment; The RGB image at the current moment is non-parametrically warped to obtain multiple warped images at the current moment; Based on multiple distorted images from the previous time step and multiple distorted images from the current time step, the RGB image at the current time step is processed to obtain a feature representation of the fused multi-view information of the RGB image at the current time step. The depth estimation network is pre-trained to process the feature representation of the RGB image at the current time and the fused multi-view information of the RGB image at the current time, so as to obtain the depth estimation map of the RGB image at the current time; wherein, the depth estimation network is trained based on scene flow.

2. The method according to claim 1, characterized in that, The RGB image at the current moment is non-parametrically warped to obtain multiple warped images at the current moment; including: Construct a set of depth hypotheses called Depths, which includes count depth planes uniformly sampled between 3 meters and 40 meters, where the i-th depth plane is denoted as d. i , 1≤i≤count; Based on the current RGB image and the depth hypothesis set Depths, construct a pixel-level depth hypothesis tensor D containing count levels. t ; The pixel-level depth assumption tensor D t The i-th level D t,i Mapped to 3D point cloud data Where K is the camera's intrinsic parameter matrix, and t is the current time. Based on 3D point cloud data Given the camera parameters, calculate the i-th distorted image. Its pixel (u,v) is: Among them, (X) c ,Y c Z c (3D point cloud) The three-dimensional coordinates of each point in the camera coordinate system, f x and f y For the camera's focal length, (c x ,c y () represents the two-dimensional coordinates of the main point.

3. The method according to claim 2, characterized in that, Based on multiple distorted images from the previous time step and multiple distorted images from the current time step, the RGB image at the current time step is processed to obtain a feature representation of the RGB image at the current time step that fuses multi-view information; including: Calculate the similarity value SSIM i,t : Wherein, the RGB image I at the current moment t The dimensions are H×W; Let i be the i-th distorted image from the previous time step; SSIM(,) is the structural similarity calculation function; Using convolutional layers to apply SSIM (similarity) i,t Perform a convolution operation to obtain the image difference tensor Diff. i,t : Based on image difference tensor Diff i,t Calculate the weight of each pixel in different depth planes. i,t : Where α is a hyperparameter used to control the decay rate of the exponential function; min i Diff i,t Diff i,t The minimum value in; Calculate the multi-view cost volume C, which represents the matching cost of pixels at each depth level. multi : The multi-view cost volume C is processed through three layers of consecutive convolution operations. multi Processing is performed to generate multi-frame cue representation F multi1 The size is h×w, where h=H / 4, w=W / 4; F multi1 The three sets of parallel convolutional layers are mapped to query feature Q, key feature K, and value feature V, respectively. Attention graph Att is calculated using the dot product attention mechanism. map , is used to represent the correlation strength between different spatial locations in a multi-view cost volume, and its expression is: To map =Softmax(Q⊙K) Where the symbol ⊙ represents element-wise matrix multiplication; Softmax(·) is the activation function; Using Att map We weight the value feature V to obtain the attention-enhancing feature Att. multi : To multi =Softmax(V⊙At map ) Attention-enhancing features Att multi and multi-perspective cost body C multi The images are fused to obtain the RGB image I at the current moment. t Feature representation F that integrates multi-perspective information use_multi : F use_multi =Conv(Upsample(Att multi ))+γ·Conv(C multi ) Among them, Upsample(Att) multi ) for Att multi Upsampling is performed using bilinear interpolation to obtain attention-enhanced features of size H×W, where γ is a parameter.

4. The method according to claim 3, characterized in that, The depth estimation network comprises an image backbone network, a stitching unit, a multi-layer encoder, and a multi-layer decoder connected in sequence. The pre-trained depth estimation network is used to process the feature representation of the current RGB image and the fused multi-view information of the current RGB image to obtain the depth estimation map of the current RGB image, including: Extracting the RGB image I at the current time using an image backbone network t Image features at different scales in group J: 1≤j≤J, where J is the number of scales; Using splicing units to fuse multi-view information feature representation F use_multi with I t The fused features F are obtained by concatenating the data along the channel dimension. concat ; F fusion feature F concat The input is a multi-layer encoder, which performs encoding processing to obtain the first intermediate feature representation E output by each layer encoder. j : AND j = Encoder j (AND j-1 ) Among them, Encoder j Let E represent the j-th layer encoder. j E0 represents the output of the (j-1)th layer encoder, which is the input of the j-th layer encoder, and E0 = F. concat ; The J sets of image features at different scales and the first intermediate features output from each encoder layer are input into a multi-layer decoder for decoding processing to obtain the second intermediate features D output from each decoder layer. j : Among them, Decoder j Denotes the j-th layer decoder, DE0 = E J ; The final output DE of the multi-layer decoder J Perform a single-layer convolution operation to obtain the depth estimate map of the RGB image at the current time step.

5. The method according to claim 1, characterized in that, The method further includes: Establish a training set, which includes K consecutive RGB image samples; Each frame of RGB image samples is subjected to non-parametric distortion to obtain multiple distorted images for each frame; Based on multiple distorted images of the (k-1)th frame and multiple distorted images of the (k+1)th frame, the RGB image sample of the kth frame is processed to obtain the feature representation of the fused multi-view information of the RGB image sample of the kth frame, where 2≤k≤K-1. The feature representation of the fused multi-view information of the k-th frame RGB image sample is processed using a depth estimation network to obtain the depth map Dep of the k-th frame RGB image sample. k ; The feature representation of the fused multi-view information of the (k+1)th frame RGB image sample is processed using a depth estimation network to obtain the depth map Dep of the (k+1)th frame RGB image sample. k+1 ; Dep based on the RGB image samples of the k-th frame k Dep and the depth map of the (k+1)th frame RGB image samples k+1 The optical flow loss L is calculated using the scene flow guidance method. optical and parallax loss L disparity ; Calculate the scene flow loss value L scene_flow : L scene_flow L optical +L disparity Using scene flow loss value L scene_flow Update the parameters of the depth estimation network.

6. The method according to claim 5, characterized in that, Dep based on the RGB image samples of the k-th frame k Dep and the depth map of the (k+1)th frame RGB image samples k+1 The optical flow loss L is calculated using the scene flow guidance method. optical and parallax loss L disparity ; include: The depth map Dep of the RGB image samples of the k-th frame k Mapping to the camera coordinate system yields the point cloud sample for the k-th frame. Among them, K -1 Let K be the inverse of the camera's intrinsic parameter matrix K; R is the rotation matrix R from the camera coordinate system to the world coordinate system in the k-th frame. k The transpose of T; k Let be the translation vector of the camera in the world coordinate system for the k-th frame; The depth map Dep of the (k+1)th frame RGB image samples k+1 Mapping to the camera coordinate system, we obtain the point cloud sample of the (k+1)th frame. in, R is the rotation matrix R from the camera coordinate system to the world coordinate system in the (k+1)th frame. k+1 The transpose of T; k+1 Let be the translation vector of the camera in the world coordinate system for the (k+1)th frame; From the point cloud sample of the kth frame Select N points 1≤n≤N; from the (k+1)th frame point cloud sample Select the corresponding N points Form N point pairs The scene flow estimation network is used to process N point pairs to obtain the 3D scene flow SF of the nth point pair. k→k+1,n , 1≤n≤N; SF k→k+1,n Including the nth point of the point cloud sample in the kth frame The offset of the three position coordinates; The nth point of the point cloud sample in the kth frame. 3D coordinates Add SF k→k+1,n To obtain the nth point 3D coordinates of the point cloud sample in frame k+1 Will Projecting the image onto the (k+1)th frame of the RGB image sample yields the two-dimensional coordinates of the nth point in the (k+1)th frame of the RGB image sample. Using a scene flow estimation network to sample the RGB image of the k-th frame The pixel coordinates of the nth point are processed to obtain its pixel coordinates in the (k+1)th frame of the RGB image sample. Calculate optical flow loss L optical : Calculate disparity loss L disparity : Where, d k+1,n For point cloud samples from frame k+1 Extract the depth value of the nth point, Dep k+1,n To extract from the depth map k+1 The depth value of the nth point is extracted.

7. A dynamic scene depth estimation device based on scene flow, characterized in that, include: The acquisition unit is used to acquire the RGB image of the target scene captured by the camera at the current moment; The first processing unit is used to perform non-parametric distortion on the RGB image at the current moment to obtain multiple distorted images at the current moment. The second processing unit is used to process the RGB image at the current time based on multiple distorted images from the previous time and multiple distorted images at the current time, so as to obtain the feature representation of the fused multi-view information of the RGB image at the current time. The depth estimation unit is used to process the feature representation of the RGB image at the current time and the fused multi-view information of the RGB image at the current time using a pre-trained depth estimation network to obtain the depth estimation map of the RGB image at the current time; wherein, the depth estimation network is trained based on scene flow.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-6.

Images