A method and apparatus for dynamic scene depth estimation based on scene flow

Abstract

The application provides a dynamic scene depth estimation method and device based on scene flow, wherein the method comprises: acquiring an RGB image of a target scene collected by a camera at a current time; performing non-parametric warping on the RGB image at the current time to obtain a plurality of warped images at the current time; processing the RGB image at the current time based on the plurality of warped images at the previous time and the plurality of warped images at the current time to obtain a feature representation of the RGB image at the current time with fused multi-view information; processing the RGB image at the current time and the feature representation of the RGB image at the current time with fused multi-view information by using a pre-trained depth estimation network to obtain a depth estimation map of the RGB image at the current time; wherein the depth estimation network is trained based on scene flow. The application improves the accuracy and robustness of depth estimation in a dynamic scene.

Description

Technical Field

[0001] This application relates to the field of dynamic depth estimation technology, and in particular to a dynamic scene depth estimation method and apparatus based on scene flow. Background Technology

[0002] Current depth estimation methods for dynamic scenes have the following drawbacks:

[0003] Reliance on dense viewpoint input: Some methods rely on high-cost multi-camera arrays or continuous video input for training and inference, which is difficult to adapt to the situation of limited equipment and sparse viewpoints in actual deployment.

[0004] Uncontrolled mask quality: When dealing with phenomena such as non-rigid motion, severe occlusion, or object disappearance, existing methods often need to rely on semantic segmentation or masking mechanisms, which increases model complexity and has the problem of unstable mask generation quality.

[0005] Weak temporal consistency: Some methods do not explicitly model motion changes in the time dimension, resulting in phenomena such as depth jumps and image flickering in long videos, affecting rendering continuity and realism.

[0006] Lack of ability to model 3D motion fields: Although some scene flow modeling methods exist, they usually have limited ability to model non-rigid body motion and lack the ability to flexibly express the complex motion structure of the scene.

[0007] Sensitive to camera pose accuracy: Most multi-view methods perform calculations under the assumption of accurate pose, but in practical applications, pose estimation errors often affect matching quality and geometric consistency. Summary of the Invention

[0008] In view of this, this application provides a dynamic scene depth estimation method and apparatus based on scene flow. This method can adapt to sparse viewpoint input and has the ability to process depth estimation of complex dynamic scenes. It can improve the robustness and visual quality of dynamic new viewpoint synthesis, thereby solving the above-mentioned technical problems of existing depth estimation methods.

[0009] In a first aspect, embodiments of this application provide a dynamic scene depth estimation method based on scene flow, including:

[0010] Acquire the RGB image of the target scene captured by the camera at the current moment;

[0011] The RGB image at the current moment is non-parametrically warped to obtain multiple warped images at the current moment;

[0012] Based on the plurality of warped images of the previous time and the plurality of warped images of the current time, the RGB image of the current time is processed to obtain a feature representation of the RGB image of the current time with fused multi-view information.

[0013] The RGB image of the current time and the feature representation of the RGB image of the current time with fused multi-view information are processed by using a pre-trained depth estimation network to obtain a depth estimation map of the RGB image of the current time, wherein the depth estimation network is trained based on a scene flow.

[0014] In one possible implementation, the RGB image of the current time is non-parametrically warped to obtain a plurality of warped images of the current time, including:

[0015] A depth hypothesis set Depths is constructed, including count depth planes uniformly sampled from 3 meters to 40 meters, and the i-th depth plane is represented as d i , 1≤i≤count.

[0016] Based on the RGB image of the current time and the depth hypothesis set Depths, a pixel-level depth hypothesis tensor D t containing count levels is constructed.

[0017] The i-th level D t,i of the pixel-level depth hypothesis tensor D t is mapped to three-dimensional point cloud data

[0018]

[0019] wherein K is an intrinsic matrix of the camera, and t is the current time.

[0020] Based on the three-dimensional point cloud data and the camera parameters, the i-th warped image is calculated, and a pixel (u, v) of the i-th warped image is:

[0021]

[0022] wherein (X c , Y c , Z c ) is the three-dimensional coordinates of each point in the three-dimensional point cloud in the camera coordinate system, f x and f y are focal lengths of the camera, and (c x , c y ) are two-dimensional coordinates of the principal point.

[0023] In one possible implementation, based on the plurality of warped images of the previous moment and the plurality of warped images of the current moment, the RGB image of the current moment is processed to obtain a feature representation of the fusion multi-view information of the RGB image of the current moment; comprising:

[0024] The similarity value SSIM is calculated i,t :

[0025]

[0026] Wherein, the size of the RGB image I t of the current moment is HxW; is the i-th warped image of the previous moment; SSIM(,) is a structural similarity calculation function;

[0027] The similarity value SSIM i,t is convoluted by using a convolution layer to obtain an image difference tensor Diff i,t :

[0028] Based on the image difference tensor Diff i,t , the weight weight i,t of each pixel at different depth planes is calculated:

[0029]

[0030] Wherein, alpha is a hyperparameter, used to control the decay rate of the exponential function; min i Diff i,t represents the minimum value in Diff i,t ;

[0031] The multi-view cost volume C multi representing the matching cost of pixels at each depth level is calculated:

[0032]

[0033] The multi-view cost volume C multi is processed by three consecutive convolution operations to generate a multi-frame clue representation F multi1 , the size of which is h x w, wherein h = H / 4, w = W / 4;

[0034] F multi1 is mapped to query feature Q, key feature K and value feature V respectively by three groups of parallel convolution layers;

[0035] The attention map Att map is calculated by using dot product attention mechanism, which is used to represent the correlation strength between different spatial positions in the multi-view cost volume, and its expression is:

[0036] Attmap =Softmax(Q⊙K)

[0037] Where the symbol ⊙ represents element-wise matrix multiplication; Softmax(·) is the activation function;

[0038] Using Att map We weight the value feature V to obtain the attention-enhancing feature Att. multi :

[0039] Att multi =Softmax(V⊙Att) map )

[0040] Attention-enhancing features Att multi and multi-perspective cost body C multi The images are fused to obtain the RGB image I at the current moment. t Feature representation F that integrates multi-perspective information use_multi :

[0041] F use_multi =Conv(Upsample(Att) multi ))+γ·Conv(C multi )

[0042] Among them, Upsample(Att) multi ) for Att multi Upsampling is performed using bilinear interpolation to obtain attention-enhanced features of size H×W, where γ is a parameter.

[0043] In one possible implementation, the depth estimation network includes an image backbone network, a stitching unit, a multilayer encoder, and a multilayer decoder connected in sequence.

[0044] The pre-trained depth estimation network is used to process the feature representation of the current RGB image and the fused multi-view information of the current RGB image to obtain the depth estimation map of the current RGB image, including:

[0045] Extracting the RGB image I at the current time using an image backbone network t Image features at different scales in group J: 1≤j≤J, where J is the number of scales;

[0046] Using splicing units to fuse multi-view information feature representation F use_multi with I t The fused features F are obtained by concatenating the data along the channel dimension. concat ;

[0047] F fusion feature F concatinput the multi-layer encoder, perform encoding processing, and obtain a first intermediate feature representation E output by each layer of the encoder j :

[0048] E j = Encoder j (E j-1 )

[0049] wherein Encoder j represents the jth layer of the encoder, E j represents the output of the (j-1)th layer of the encoder, i.e., the input of the jth layer of the encoder, and E0=F concat ;

[0050] input the multi-layer decoder with the J sets of image features of different scales and the first intermediate features output by each layer of the encoder, perform decoding processing, and obtain a second intermediate feature D output by each layer of the decoder j :

[0051]

[0052] wherein Decoder j represents the jth layer of the decoder, and DE0=E J ;

[0053] perform single-layer convolution operation on DE J output by the multi-layer decoder, and obtain a depth estimation map of the RGB image at the current moment.

[0054] In one possible implementation, the method further includes:

[0055] establishing a training set, wherein the training set includes K consecutive RGB image samples;

[0056] performing non-parametric warping on each RGB image sample respectively, and obtaining multiple warped images for each frame;

[0057] processing the kth RGB image sample based on the multiple warped images of the (k-1)th frame and the multiple warped images of the (k+1)th frame, and obtaining a feature representation of the kth RGB image sample with fused multi-view information, wherein 2≤k≤K-1;

[0058] processing the kth RGB image sample and the feature representation of the kth RGB image sample with fused multi-view information by using the depth estimation network, and obtaining a depth map Dep k of the kth RGB image sample;

[0059] processing the (k+1)th RGB image sample and the feature representation of the (k+1)th RGB image sample with fused multi-view information by using the depth estimation network, and obtaining a depth map Depk+1 ;

[0060] depth map Dep of the kth frame RGB image sample k and depth map Dep of the k+1th frame RGB image sample k+1 , the optical flow loss L is calculated by using the scene flow guiding method optical and the disparity loss L disparity ;

[0061] the scene flow loss value L is calculated scene_flow :

[0062] L scene_flow = L optical + L disparity

[0063] the parameters of the depth estimation network are updated by using the scene flow loss value L scene_flow .

[0064] In one possible implementation, the depth map Dep of the kth frame RGB image sample k and the depth map Dep of the k+1th frame RGB image sample k+1 , the optical flow loss L is calculated by using the scene flow guiding method optical and the disparity loss L disparity ; comprising:

[0065] the depth map Dep of the kth frame RGB image sample is mapped to the camera coordinate system, to obtain the kth frame point cloud sample k

[0066]

[0067]

[0068] wherein, K -1 is the inverse matrix of the intrinsic matrix K of the camera; is the transpose matrix of the rotation matrix R k of the kth frame camera coordinate system to the world coordinate system; T k is the translation vector of the kth frame camera in the world coordinate system;

[0069] the depth map Dep of the k+1th frame RGB image sample is mapped to the camera coordinate system, to obtain the k+1th frame point cloud sample k+1

[0070]

[0071] wherein, is the rotation matrix R​​k+1 T k+1 is the translation vector of the k+1 frame camera in the world coordinate system;

[0072] from the kth frame point cloud sample select N points 1≤n≤N;from the k+1 frame point cloud sample select corresponding N points form N point pairs

[0073] use the scene flow estimation network to process the N point pairs, and obtain the three-dimensional scene flow SF k→k+1,n of the nth point pair, 1≤n≤N;SF k→k+1,n includes the offset of the three position coordinates of the nth point of the kth frame point cloud sample

[0074] add the three-dimensional coordinates of the nth point of the kth frame point cloud sample SF k→k+1,n , to obtain the three-dimensional coordinates of the nth point in the k+1 frame point cloud sample project to the k+1 frame RGB image sample to obtain the two-dimensional coordinates of the nth point in the k+1 frame RGB image sample

[0075] use the scene flow estimation network to process the pixel coordinates of the nth point in the kth frame RGB image sample , to obtain the pixel coordinates of the nth point in the k+1 frame RGB image sample

[0076] calculate the optical flow loss L optical :

[0077]

[0078] calculate the disparity loss L disparity :

[0079]

[0080] where d k+1,n is the depth value of the nth point extracted from the k+1 frame point cloud sample , Dep k+1,n is the depth value of the nth point extracted from the depth map Dep k+1 .

[0081] In a second aspect, the embodiments of the present application provide a dynamic scene depth estimation device based on scene flow, comprising:​​

[0082] an acquisition unit configured to acquire an RGB image of a target scene collected by a camera at a current time point;

[0083] a first processing unit configured to perform non-parametric warping on the RGB image at the current time point to obtain a plurality of warped images at the current time point;

[0084] a second processing unit configured to perform processing on the RGB image at the current time point based on the plurality of warped images at the previous time point and the plurality of warped images at the current time point to obtain a feature representation of the RGB image at the current time point with fused multi-view information;

[0085] a depth estimation unit configured to perform processing on the RGB image at the current time point and the feature representation of the RGB image at the current time point with fused multi-view information by using a pre-trained depth estimation network to obtain a depth estimation map of the RGB image at the current time point, wherein the depth estimation network is trained based on a scene flow.

[0086] In a third aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the method of the embodiments of the present application when executing the computer program.

[0087] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program, when run by a processor, implements the method of the embodiments of the present application.

[0088] The present application improves the accuracy and robustness of depth estimation in a dynamic scene. BRIEF DESCRIPTION OF DRAWINGS

[0089] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or prior art description. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0090] Figure 1 a flowchart of the scene flow based dynamic scene depth estimation method provided by the embodiments of the present application;

[0091] Figure 2 a processing flowchart of the scene flow estimation network provided by the embodiments of the present application;

[0092] Figure 3 a functional structure diagram of the scene flow based dynamic scene depth estimation apparatus provided by the embodiments of the present application;

[0093] Figure 4 A functional structure diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0094] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0095] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0096] First, the design idea of the embodiments of the present application is briefly introduced.

[0097] The existing technologies in the field of dynamic depth estimation mainly include two categories: learning-based depth estimation methods and geometry-based multi-view modeling methods.

[0098] In the learning-based method, early dependence is mainly on supervised learning, and a large number of labeled depth maps are used to train the depth network. Although this kind of method has high accuracy, it has strong dependence on data and limited generalization ability. In recent years, self-supervised methods have gradually emerged, which automatically extract depth information from videos through image reconstruction loss, edge smoothing constraint, etc., and have better adaptability and expansibility. Some methods introduce semantic masks or optical flow estimation to process dynamic regions and improve the robustness of depth estimation.

[0099] In the geometry-based multi-view estimation method, the mainstream adopts multi-view stereo vision method and scene flow estimation method. Among them, the multi-view stereo vision method restores the scene depth through the geometric consistency between multiple camera perspectives, which is suitable for static environment. In order to deal with more common dynamic scenes in reality, some methods combine time series modeling and scene flow network to try to model the motion field in three-dimensional space for depth estimation. In addition, methods based on neural implicit representation such as NeRF and its extended version are also used for modeling and rendering tasks of dynamic scenes.

[0100] A typical dynamic view synthesis system consists of multiple modules, including feature extraction, depth estimation, view transformation, and image rendering. The depth estimation module is usually built based on a convolutional neural network and combined with camera pose and re-projection mechanisms to achieve pixel matching between different views. To improve the quality of the synthesized image, spatial or temporal feature fusion techniques such as feature warping, spatio-temporal consistency constraints, etc. are often used to enhance the continuity and realism of image generation. The following are representative methods:

[0101] Lite-Mono network: transforms the depth estimation task into an image reconstruction task, uses reconstruction error to guide depth information learning, and proposes a hybrid structure by combining CNN and Vision Transformer to achieve model lightweight.

[0102] RM-Depth network: explicitly models scene flow and object motion, achieving robust processing of complex motion in dynamic scenes. Scene flow is used to estimate object motion and ego motion, and the influence of object motion is reduced by estimating the three-dimensional motion field of moving objects and depth and ego motion.

[0103] Multi-task network: In recent years, multi-task network structures that jointly model depth estimation, optical flow, normal vector estimation, and semantic segmentation have also appeared, enhancing the collaborative effect of each task through feature sharing mechanisms.

[0104] The present application solves the problem of uncontrolled mask quality in dynamic scenes: in the process of improving the accuracy of dynamic region depth estimation, the idea of "spatio-temporal geometric information guidance" is considered, and a three-dimensional motion field modeling method for dynamic regions is adopted. The three-dimensional motion field of the dynamic region can be estimated by the scene flow estimation network, which can eliminate the dependence on the mask. Finally, a spatio-temporal geometric consistency depth estimation algorithm is considered, which can better handle dynamic regions and effectively guide depth estimation by modeling the three-dimensional motion field.

[0105] The present application realizes self-supervised depth estimation: by introducing scene flow loss to realize self-supervised depth estimation, using the geometric consistency between image frames as a supervision signal without relying on real depth labels. The scene flow loss consists of an optical flow consistency loss and a disparity consistency loss, the former measures the deviation between the predicted two-dimensional optical flow and the optical flow obtained by projecting the three-dimensional scene flow, and the latter compares the difference between the depth values predicted by the depth estimation module and the sparse depth generated from the three-dimensional point cloud. Through this loss function, the model can constrain the geometric and motion consistency between different time frames on unlabeled data, effectively learning the depth information, and thus realizing self-supervised training.

[0106] The present application improves the accuracy and robustness of depth estimation in dynamic scenes and is suitable for automatic driving, robot perception, and other fields.

[0107] After introducing the application scenarios and design ideas of the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below.

[0108] As shown in Figure 1 , the embodiments of the present application provide a dynamic scene depth estimation method based on scene flow, comprising:

[0109] Step 101: acquiring an RGB image of a target scene collected by a camera at a current time;

[0110] Step 102: performing non-parametric warping on the RGB image at the current time to obtain a plurality of warped images at the current time;

[0111] Step 103: processing the RGB image at the current time based on the plurality of warped images at the previous time and the plurality of warped images at the current time to obtain a feature representation of the RGB image at the current time with fused multi-view information;

[0112] Step 104: processing the RGB image at the current time and the feature representation of the RGB image at the current time with fused multi-view information using a pre-trained depth estimation network to obtain a depth estimation map of the RGB image at the current time; wherein the depth estimation network is trained based on scene flow.

[0113] In some embodiments, the RGB image at the current time is warped non-parametrically to obtain a plurality of warped images at the current time; comprising:

[0114] Constructing a depth hypothesis set Depths, comprising: uniformly sampling count depth planes between 3 meters and 40 meters, the i-th depth plane is represented as d i , 1≤i≤count;

[0115] Based on the RGB image at the current time and the depth hypothesis set Depths, a pixel-level depth hypothesis tensor D t containing count levels is constructed;

[0116] Mapping the i-th level D t,i of the pixel-level depth hypothesis tensor D t to three-dimensional point cloud data

[0117]

[0118] Wherein, K is the intrinsic matrix of the camera, and t is the current time;

[0119] Based on the three-dimensional point cloud data and the camera parameters, the i-th warped image Its pixels (u, v) are:

[0120]

[0121] where (X c ,Y c ,Z c ) is a three-dimensional point cloud of each point in the camera coordinate system, f x and f y are the focal lengths of the camera, and (c x ,c y ) is the two-dimensional coordinate of the principal point.

[0122] In some embodiments, based on the plurality of warped images of the previous moment and the plurality of warped images of the current moment, the RGB image of the current moment is processed to obtain a feature representation of the fusion multi-view information of the RGB image of the current moment; comprising:

[0123] Calculate the similarity value SSIM i,t :

[0124]

[0125] where the size of the RGB image I t of the current moment is HxW; is the i-th warped image of the previous moment; SSIM(,) is a structural similarity calculation function;

[0126] Convolve the similarity value SSIM i,t using a convolution layer to obtain an image difference tensor Diff i,t :

[0127] Based on the image difference tensor Diff i,t , calculate the weight weight i,t of each pixel under different depth planes:

[0128]

[0129] where alpha is a hyperparameter used to control the decay rate of the exponential function; min i Diff i,t represents the minimum value in Diff i,t ;

[0130] Calculate the multi-view cost volume C multi representing the matching cost of the pixels under each depth level:

[0131]

[0132] The multi-view cost volume C is processed by a three-layer continuous convolution operation multi to generate a multi-frame clue representation F multi1 with a size of h x w, where h = H / 4 and w = W / 4.

[0133] F multi1 is mapped into query features Q, key features K, and value features V respectively by three sets of parallel convolution layers.

[0134] An attention map Att map is calculated using a dot-product attention mechanism to represent the correlation strength between different spatial positions in the multi-view cost volume, and its expression is as follows:

[0135] Att map = Softmax(Q⊙K)

[0136] where the symbol ⊙ represents element-wise matrix multiplication, and Softmax(·) is an activation function.

[0137] The value features V are weighted using Att map to obtain attention-enhanced features Att multi :

[0138] Att multi = Softmax(V⊙Att map )

[0139] The attention-enhanced features Att multi and the multi-view cost volume C multi are fused to obtain a feature representation F use_multi of the fused multi-view information of the RGB image I t at the current time:

[0140] F use_multi = Conv(Upsample(Att multi ))+γ·Conv(C multi )

[0141] where Upsample(Att multi ) is a bilinear interpolation for up-sampling Att multi to obtain attention-enhanced features with a size of H x W, and γ is a parameter for adjusting the proportion of the original cost volume features in the fusion process.

[0142] The feature representation F use_multi of the fused multi-view information can effectively improve the semantic consistency and feature expression ability across different views, and provide more discriminative fusion features for subsequent depth estimation or three-dimensional reconstruction.

[0143] In some embodiments, the depth estimation network comprises an image backbone network, a concatenation unit, a multi-layer encoder and a multi-layer decoder connected in sequence;

[0144] The pre-trained depth estimation network is used to process the feature representation of the current time RGB image and the fused multi-view information of the current time RGB image, to obtain a depth estimation map of the current time RGB image, comprising:

[0145] The image backbone network is used to extract J sets of image features of different scales of the current time RGB image I t : 1≤j≤J, wherein J is the number of scales;

[0146] The concatenation unit is used to concatenate the feature representation F use_multi and I t in the channel dimension to obtain the fused feature F concat ;

[0147] The fused feature F concat is input into the multi-layer encoder for encoding processing to obtain the first intermediate feature representation E j output by each layer of the encoder:

[0148] E j =Encoder j (E j-1 )

[0149] Wherein, Encoder j represents the jth layer of the encoder, E j represents the output of the j-1th layer of the encoder, i.e. the input of the jth layer of the encoder, and E0=F concat ;

[0150] The J sets of image features of different scales and the first intermediate feature output by each layer of the encoder are input into the multi-layer decoder for decoding processing to obtain the second intermediate feature D j output by each layer of the decoder:

[0151]

[0152] Wherein, Decoder j represents the jth layer of the decoder, and DE0=E J ;

[0153] The final output DE J of the multi-layer decoder is subjected to single-layer convolution operation to obtain the depth estimation map of the current time RGB image.

[0154] The application introduces the fusion of multi-scale image features and coding features in the decoding process, thereby not only maintaining the spatial structure information of high-resolution images, but also improving the accuracy and robustness of depth prediction.

[0155] In some embodiments, the method further comprises:

[0156] establishing a training set comprising continuous K frame RGB image samples;

[0157] performing non-parametric warping on each frame of RGB image samples respectively to obtain multiple warped images of each frame,

[0158] processing the kth frame of RGB image samples based on the multiple warped images of the k-1th frame and the multiple warped images of the k+1th frame to obtain a feature representation of the kth frame of RGB image samples with fused multi-view information, 2≤k≤K-1;

[0159] processing the kth frame of RGB image samples and the feature representation of the kth frame of RGB image samples with fused multi-view information using a depth estimation network to obtain a depth map Dep k of the kth frame of RGB image samples;

[0160] processing the k+1th frame of RGB image samples and the feature representation of the k+1th frame of RGB image samples with fused multi-view information using a depth estimation network to obtain a depth map Dep k+1 of the k+1th frame of RGB image samples;

[0161] based on the depth map Dep k of the kth frame of RGB image samples and the depth map Dep k+1 of the k+1th frame of RGB image samples, using a scene flow guided method to calculate an optical flow loss L optical and a disparity loss L disparity ;

[0162] calculating a scene flow loss value L scene_flow :

[0163] L scene_flow = L optical + L disparity

[0164] updating the parameters of the depth estimation network using the scene flow loss value L scene_flow .

[0165] The optical flow loss is used to measure the deviation between the point position projected by the scene flow to the two-dimensional image space and the pixel coordinates estimated by the optical flow. The disparity loss is used to constrain the difference between the sparse depth values generated by the scene flow and the sparse depth values generated by the depth estimation module combined with the optical flow.

[0166] Through the above loss function design, this embodiment effectively enhances the model's ability to learn about three-dimensional geometric changes in dynamic scenes and improves the generalization of depth estimation without the need for external supervision information.

[0167] In some embodiments, the depth map Dep is based on the RGB image samples of the k-th frame. k Dep and the depth map of the (k+1)th frame RGB image samples k+1 The optical flow loss L is calculated using the scene flow guidance method. optical and parallax loss L disparity ;include:

[0168] The depth map Dep of the RGB image samples of the k-th frame k Mapping to the camera coordinate system yields the point cloud sample for the k-th frame.

[0169]

[0170] Among them, K -1 Let K be the inverse of the camera's intrinsic parameter matrix K; R is the rotation matrix R from the camera coordinate system to the world coordinate system in the k-th frame. k The transpose of T; k Let be the translation vector of the camera in the world coordinate system for the k-th frame;

[0171] The depth map Dep of the (k+1)th frame RGB image samples k+1 Mapping to the camera coordinate system, we obtain the point cloud sample of the (k+1)th frame.

[0172]

[0173] in, R is the rotation matrix R from the camera coordinate system to the world coordinate system in the (k+1)th frame. k+1 The transpose of T; k+1 Let be the translation vector of the camera in the world coordinate system for the (k+1)th frame;

[0174] From the point cloud sample of the kth frame Select N points 1≤n≤N; from the (k+1)th frame point cloud sample Select the corresponding N points Form N point pairs

[0175] The scene flow estimation network is used to process N point pairs to obtain the 3D scene flow SF of the nth point pair. k→k+1,n , 1≤n≤N; SF k→k+1,n Including the nth point of the point cloud sample in the kth frame offsets of three position coordinates of the kth frame of point cloud samples;

[0176] the three-dimensional coordinates of the nth point of the kth frame of point cloud samples SF k→k+1,n , to obtain the three-dimensional coordinates of the nth point in the k+1th frame of point cloud samples projected to the k+1th frame of RGB image samples, to obtain the two-dimensional coordinates of the nth point in the k+1th frame of RGB image samples

[0177] the pixel coordinates of the nth point in the kth frame of RGB image samples

[0178] optical

[0179]

[0180] disparity

[0181]

[0182] wherein d k+1,n is the depth value of the nth point extracted from the k+1th frame of point cloud samples Dep k+1,n is the depth value of the nth point extracted from the depth map Dep k+1 .

[0183] In the dynamic scene depth estimation process, in order to effectively process the three-dimensional motion information of the dynamic region, a scene flow estimation network is introduced to model the three-dimensional motion of the dynamic region.

[0184] As shown in Figure 2 , the scene flow estimation network of the embodiment includes a super point generation and iterative updating mechanism. By utilizing the characteristic that a super point can perceive similar motion in a local region, an initial super point is first generated based on an initial scene flow. Subsequently, two modules of "super point guided scene flow optimization" and "scene flow guided super point generation" are sequentially executed for iterative updating, so as to improve the accuracy of the scene flow.

[0185] In the initialization stage, the features of adjacent point clouds are extracted through a three-layer convolutional network, a feature similarity matrix is calculated, and a scene flow weight matrix is generated in combination with the three-dimensional spatial similarity, and then the corresponding point and its feature are predicted to obtain the initialized scene flow estimation result.

[0186] ​​​​​​​​In the "superpoint guided scene flow generation" stage, first, the source point cloud is down-sampled to obtain clustering index, clustering center feature and coordinates, and the index of each point relative to the clustering center is obtained through K-neighbor query. Further, the feature difference and coordinate difference between each point and its corresponding superpoint are compared, and the sum is calculated after encoding by a convolution layer, the bidirectional association weight is normalized, and the association matrix between the point and the superpoint is obtained. Using the association matrix, the feature, coordinates and scene flow information of the point are updated to the corresponding superpoint.

[0187] In the "superpoint guided scene flow optimization" stage, a GRU module is used to refine the scene flow according to the current iteration scene flow, point feature and association graph. Specifically, the point cloud is transformed according to the current scene flow, and the local correlation feature is calculated; the GRU input is constructed by combining the input feature, correlation feature, current scene flow and association information; the hidden state is updated and the residual scene flow is generated by GRU, and the updated scene flow is obtained after normalization and addition of the current scene flow. After three iterations, the final scene flow estimation result is output.

[0188] Based on the same inventive concept, an apparatus is provided in the embodiments of the present application, as shown in Figure 3 The apparatus 200 provided in the embodiments of the present application at least includes:

[0189] The acquisition unit 201 is configured to acquire an RGB image of a target scene collected by a camera at a current time;

[0190] The first processing unit 202 is configured to perform non-parametric warping on the RGB image at the current time to obtain a plurality of warped images at the current time;

[0191] The second processing unit 203 is configured to process the RGB image at the current time based on the plurality of warped images at the previous time and the plurality of warped images at the current time to obtain a feature representation of the RGB image at the current time with fused multi-view information.

[0192] The depth estimation unit 204 is configured to process the RGB image at the current time and the feature representation of the RGB image at the current time with fused multi-view information by using a pre-trained depth estimation network to obtain a depth estimation map of the RGB image at the current time; wherein the depth estimation network is trained based on scene flow.

[0193] It should be noted that the principle of solving the technical problems of the dynamic scene depth estimation apparatus 200 based on scene flow provided in the embodiments of the present application is similar to the method provided in the embodiments of the present application, and therefore the implementation of the dynamic scene depth estimation apparatus 200 based on scene flow provided in the embodiments of the present application can be referred to the implementation of the method provided in the embodiments of the present application, and the repeated parts will not be described herein.

[0194] Based on the same inventive concept, the embodiment of the present application further provides an electronic device, referring to Figure 4 As shown in the figure, the electronic device 300 provided by the embodiment of the present application at least includes a processor 301, a memory 302, and a computer program stored in the memory 302 and capable of running on the processor 301, and the processor 301 implements the scene flow based dynamic scene depth estimation method provided by the embodiment of the present application when executing the computer program.

[0195] The electronic device 300 provided by the embodiment of the present application can further include a bus 303 connecting different components (including the processor 301 and the memory 302). Among them, the bus 303 represents one or more of several bus structures, including a memory bus, a peripheral bus, a local bus, etc.

[0196] The memory 302 can include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 3021 and / or a cache memory 3022, and can further include a read-only memory (ROM) 3023.

[0197] The memory 302 can further include a program tool 3025 having a set of (at least one) program modules 3024, including but not limited to: an operating subsystem, one or more application programs, other program modules, and program data, each of these examples or some combination thereof can include the implementation of a network environment.

[0198] The electronic device 300 can also communicate with one or more external devices 304 (such as a keyboard, a remote control, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 300 (such as a mobile phone, a computer, etc.), and / or communicate with any device that enables the electronic device 300 to communicate with one or more other electronic devices 300 (such as a router, a modem, etc.). Such communication can be through an input / output (I / O) interface 305. And the electronic device 300 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 306. As Figure 4 As shown in the figure, the network adapter 306 communicates with other modules of the electronic device 300 through the bus 303. It should be understood that although Figure 4Other hardware and / or software modules can be used in conjunction with electronic device 300, as desired, including, but not limited to, microcode, device drivers, redundant processors, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) subsystems, tape drives, data backup storage subsystems, etc., which are not shown.

[0199] It should be noted that, Figure 4 Electronic device 300 shown is merely an example and should not impose any limitation on the functions and usage scope of the embodiments of the present application.

[0200] The embodiments of the present application further provide a computer readable storage medium, which stores computer instructions, and the computer instructions are executed by a processor to implement the scene flow based dynamic scene depth estimation method provided by the embodiments of the present application. Specifically, the executable program can be built-in or installed in the electronic device 300, so that the electronic device 300 can implement the scene flow based dynamic scene depth estimation method provided by the embodiments of the present application by executing the built-in or installed executable program.

[0201] The scene flow based dynamic scene depth estimation method provided by the embodiments of the present application can also be implemented as a program product, which includes program codes, and the program codes are used to make the electronic device 300 execute the scene flow based dynamic scene depth estimation method provided by the embodiments of the present application when the program product can run on the electronic device 300.

[0202] The program product provided by the embodiments of the present application can adopt any combination of one or more readable media, wherein the readable media can be a readable signal medium or a readable storage medium, and the readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above, and more specifically, the more specific examples (non-exhaustive list) of the readable storage medium include: electrical connection with one or more conductive wires, portable disk, hard disk, RAM, ROM, Erasable Programmable Read Only Memory (EPROM), optical fiber, portable Compact Disc Read-Only Memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

[0203] The program product provided by the embodiments of the present application can adopt a CD-ROM and include program codes, and can also run on a computing device. However, the program product provided by the embodiments of the present application is not limited to this, and in the embodiments of the present application, the readable storage medium can be any tangible medium containing or storing programs, which can be used by or in combination with an instruction execution system, device or apparatus.

[0204] It should be noted that, although several units or sub-units of the apparatus are mentioned in the foregoing detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into units for embodiment.

[0205] In addition, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all of the shown operations must be performed to achieve the desired result. Additionally or alternatively, certain steps can be omitted, a plurality of steps can be combined into one step, and / or one step can be divided into a plurality of steps.

[0206] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it. Although the present application is described in detail with reference to the embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, and they should be covered in the scope of claims of the present application.

Claims

1. A dynamic scene depth estimation method based on scene flow, characterized in that, include: Acquire the RGB image of the target scene captured by the camera at the current moment; The RGB image at the current moment is non-parametrically warped to obtain multiple warped images at the current moment; Based on multiple distorted images from the previous time step and multiple distorted images from the current time step, the RGB image at the current time step is processed to obtain a feature representation of the fused multi-view information of the RGB image at the current time step. The depth estimation network is pre-trained to process the feature representation of the RGB image at the current time and the fused multi-view information of the RGB image at the current time to obtain the depth estimation map of the RGB image at the current time; wherein, the depth estimation network is trained based on scene flow; The RGB image at the current moment is non-parametrically warped to obtain multiple warped images at the current moment; including: Building a deep hypothesis set This includes: uniformly sampled from 3 meters to 40 meters. The depth plane, the first Each depth plane is represented as , ; Based on the current RGB image and depth hypothesis set , construct containing Each level of pixel-level depth assumption tensor ; pixel-level depth hypothesis tensor Mapped to 3D point cloud data : in, Let be the intrinsic parameter matrix of the camera, and t be the current time. Based on 3D point cloud data Given the camera parameters, calculate the i-th distorted image. Its pixels for: in, 3D point cloud The three-dimensional coordinates of each point in the camera coordinate system. For the camera's focal length, The two-dimensional coordinates of the principal point.

2. The method according to claim 1, characterized in that, Based on multiple distorted images from the previous time step and multiple distorted images from the current time step, the RGB image at the current time step is processed to obtain a feature representation of the RGB image at the current time step that fuses multi-view information; including: Calculate similarity value : Among them, the RGB image at the current moment The size is ; This is the i-th distorted image from the previous time step; This is a function for calculating structural similarity. Using convolutional layers to analyze similar values Perform a convolution operation to obtain the image difference tensor. : Based on image difference tensor Calculate the weight of each pixel in different depth planes. : in, This is a hyperparameter used to control the decay rate of the exponential function; express The minimum value in; Calculate the multi-view cost volume to represent the matching cost of pixels at each depth level. : Multi-view cost volume is processed through three layers of consecutive convolution operations. Processing is performed to generate multi-frame cue representations. Size is ,in, ; Will The query features are mapped through three sets of parallel convolutional layers. Key features Sum value characteristics ; Calculating attention maps using the dot product attention mechanism , is used to represent the correlation strength between different spatial locations in a multi-view cost volume, and its expression is: Among them, symbols This represents element-wise matrix multiplication; For activation functions; use logarithmic features Weighting is performed to obtain attention-enhancing features. : Attention-enhancing features and multi-perspective cost body The images are then fused to obtain the RGB image at the current moment. Feature representation of fusion of multi-perspective information : in, To Upsampling is performed using bilinear interpolation to obtain a size of Attention enhancement features For parameters.

3. The method according to claim 2, characterized in that, The depth estimation network comprises an image backbone network, a stitching unit, a multi-layer encoder, and a multi-layer decoder connected in sequence. The pre-trained depth estimation network is used to process the feature representation of the current RGB image and the fused multi-view information of the current RGB image to obtain the depth estimation map of the current RGB image, including: Extracting the RGB image at the current time using an image backbone network of Image features at different scales: , ,in, The number of scales; Feature representation of fused multi-view information using splicing units and By concatenating the data along the channel dimension, the fused features are obtained. ; Fusion features The input is a multi-layer encoder, which performs encoding processing to obtain the first intermediate feature representation output by each layer encoder. : in, Indicates the first Layer encoder, Indicates the first The output of the layer encoder, i.e. the first layer... The input of the layer encoder, ; Will Image features at different scales and the first intermediate features output from each encoder layer are input into a multi-layer decoder for decoding to obtain the second intermediate features output from each decoder layer. : in, Indicates the first Layer decoder, ; The final output of the multi-layer decoder Perform a single-layer convolution operation to obtain the depth estimate map of the RGB image at the current time step.

4. The method according to claim 1, characterized in that, The method further includes: Establish a training set, which includes consecutive... Frame RGB image sample; Each frame of RGB image samples is subjected to non-parametric distortion to obtain multiple distorted images for each frame; Based on multiple distorted images from frame (k-1) and frame (k+1), the RGB image sample of frame k is processed to obtain a feature representation of the RGB image sample of frame k that incorporates multi-view information. ; A depth estimation network is used to process the feature representation of the fused multi-view information of the k-th frame RGB image sample and the k-th frame RGB image sample to obtain the depth map of the k-th frame RGB image sample. ; A depth estimation network is used to process the feature representation of the fused multi-view information of the (k+1)th frame RGB image sample and the (k+1)th frame RGB image sample to obtain the depth map of the (k+1)th frame RGB image sample. ; Depth map based on RGB image samples of the k-th frame Depth map of RGB image samples in frame (k+1) Calculate optical flow loss using scene flow guidance method and parallax loss ; Calculate the scene flow loss value : Using scene flow loss value Update the parameters of the depth estimation network.

5. The method according to claim 4, characterized in that, Depth map based on RGB image samples of the k-th frame Depth map of RGB image samples in frame (k+1) Calculate optical flow loss using scene flow guidance method and parallax loss ;include: Depth map of RGB image samples in frame k Mapping to the camera coordinate system yields the point cloud sample for the k-th frame. : in, The intrinsic parameter matrix of the camera The inverse matrix; Let be the rotation matrix from the camera coordinate system to the world coordinate system in the k-th frame. The transpose of the matrix; Let be the translation vector of the camera in the world coordinate system for the k-th frame; The depth map of the RGB image samples in the (k+1)th frame. Mapping to the camera coordinate system, we obtain the point cloud sample of the (k+1)th frame. : in, The rotation matrix from the camera coordinate system to the world coordinate system in the (k+1)th frame. The transpose of the matrix; Let be the translation vector of the camera in the world coordinate system for the (k+1)th frame; From the point cloud sample of the kth frame Select N points , From the point cloud sample of frame k+1 Select the corresponding N points Form N point pairs ( ); The scene flow estimation network is used to process N point pairs to obtain the 3D scene flow of the nth point pair. , ; Including the nth point of the point cloud sample in the kth frame The offset of the three position coordinates; The nth point of the point cloud sample in the kth frame. 3D coordinates Plus To obtain the nth point 3D coordinates of the point cloud sample in frame k+1 ,Will Projecting the image onto the (k+1)th frame of the RGB image sample yields the two-dimensional coordinates of the nth point in the (k+1)th frame of the RGB image sample. ; Using a scene flow estimation network to sample the RGB image of the k-th frame The pixel coordinates of the nth point are processed to obtain its pixel coordinates in the (k+1)th frame of the RGB image sample. ; Calculate optical flow loss : Calculate disparity loss : in, For point cloud samples from frame k+1 Extract the depth value of the nth point. To obtain from depth map The depth value of the nth point is extracted.

6. A dynamic scene depth estimation device based on scene flow, characterized in that, include: The acquisition unit is used to acquire the RGB image of the target scene captured by the camera at the current moment; The first processing unit is used to perform non-parametric distortion on the RGB image at the current moment to obtain multiple distorted images at the current moment. The second processing unit is used to process the RGB image at the current time based on multiple distorted images from the previous time and multiple distorted images at the current time, so as to obtain the feature representation of the fused multi-view information of the RGB image at the current time. The depth estimation unit is used to process the feature representation of the RGB image at the current time and the fused multi-view information of the RGB image at the current time using a pre-trained depth estimation network to obtain the depth estimation map of the RGB image at the current time; wherein, the depth estimation network is trained based on scene flow; The first processing unit is specifically used for: Building a deep hypothesis set This includes: uniformly sampled from 3 meters to 40 meters. The depth plane, the first Each depth plane is represented as , ; Based on the current RGB image and depth hypothesis set , construct containing Each level of pixel-level depth assumption tensor ; pixel-level depth hypothesis tensor Mapped to 3D point cloud data : in, Let be the intrinsic parameter matrix of the camera, and t be the current time. Based on 3D point cloud data Given the camera parameters, calculate the i-th distorted image. Its pixels for: in, 3D point cloud The three-dimensional coordinates of each point in the camera coordinate system. For the camera's focal length, The two-dimensional coordinates of the principal point.

7. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-5.

Images