Video depth estimation method and device, storage medium and program product

Abstract

The invention provides a video depth estimation method and device, a storage medium and a program product, and the method comprises the steps: obtaining a first camera motion offset of a pixel point in a to-be-compensated video frame relative to a same pixel point in an adjacent frame, and an object motion offset of the pixel point in the to-be-compensated video frame relative to the same pixel point in the adjacent frame; and compensating the predicted first depth map of the to-be-compensated video frame pixel by pixel, namely acting the camera motion offset and the object motion offset corresponding to a single pixel point on the pixel point, and compensating the pixel point of the first depth map of the to-be-compensated video frame in a targeted manner. The estimated depth between the to-be-compensated video frame and the adjacent frame is stable, and when the estimated depth maps of the to-be-compensated video frame and the adjacent frame are applied to the fields of video blurring or automatic driving and the like, the to-be-compensated video frame and the adjacent frame do not flicker. And artifacts do not appear on the edges of the objects in the to-be-compensated video frame and the adjacent frame.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a video depth estimation method, device, storage medium and program product. Background Art

[0002] Depth estimation is a basic task in computer vision, which aims to estimate the depth from a 2D image. The input of this task is an RGB image, and the output is a depth map. The depth here refers to the distance from the object to the optical center of the camera, and the problem to be solved by depth estimation is to solve the distance from the object in the shooting scene to the optical center of the camera. Depth estimation has a wide range of use prospects and application value. For example, the currently hotly researched field of autonomous driving, as well as traditional 3D reconstruction, augmented reality, video blur, etc. all require the use of depth estimation related technologies.

[0003] Currently, when training a depth estimation network, the camera motion offset and object motion offset of two adjacent video frames in a continuous video frame are used as a loss function to constrain the depth estimation network. The estimated depth between continuous video frames is unstable, which causes the previous and next video frames to flicker when the estimated depth map of continuous video frames is applied to fields such as video blurring or autonomous driving, or artifacts appear at the edges of objects in the previous and next video frames. Summary of the invention

[0004] The present application provides a video depth estimation method, device, storage medium and program product, aiming to solve the technical problem that the estimated depth between consecutive video frames is unstable, resulting in flickering of the previous and next video frames when the estimated depth map of consecutive video frames is applied to fields such as video blurring or autonomous driving, or artifacts appear at the edges of objects in the previous and next video frames.

[0005] In order to achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, a video depth estimation method is provided. First, a first depth map and a first camera pose matrix of a video frame to be compensated in a plurality of consecutive video frames, and a second depth map and a second camera pose matrix of an adjacent frame of the video frame to be compensated are obtained; then, a first camera motion offset of a pixel in the video frame to be compensated is determined according to the first depth map, the first camera pose matrix, the second depth map, the second camera pose matrix and the camera intrinsic parameter, wherein the first camera motion offset is a coordinate offset of a pixel in the video frame to be compensated relative to the same pixel in an adjacent frame in the same coordinate system; an object motion offset of a pixel in the video frame to be compensated relative to the same pixel in an adjacent frame is determined; finally, the first depth map is compensated pixel by pixel according to the first camera motion offset and the object motion offset to obtain a compensated target depth map of the video frame to be compensated.

[0007] Based on the above technical solution, according to the first camera motion offset of the pixel in the video frame to be compensated relative to the same pixel in the adjacent frame, and the object motion offset of the pixel in the video frame to be compensated relative to the same pixel in the adjacent frame, the predicted first depth map of the video frame to be compensated is compensated pixel by pixel, that is, the camera motion offset and the object motion offset corresponding to a single pixel are applied to the pixel, instead of applying the camera motion offset and the object motion offset corresponding to all pixels to each pixel, and the pixel points of the first depth map of the video frame to be compensated are compensated in a targeted manner. In this way, the estimated depth between the video frame to be compensated and the adjacent frames is relatively stable. When the estimated depth maps of the video frame to be compensated and the adjacent frames are applied to fields such as video blurring or autonomous driving, the video frame to be compensated and the adjacent frames will not flicker, and no artifacts will appear on the edges of objects in the video frame to be compensated and the adjacent frames.

[0008] In a possible implementation of the first aspect, the first depth map is compensated pixel by pixel according to the first camera motion offset and the object motion offset to obtain a target depth map after compensation of the video frame to be compensated, including: inputting the first camera motion offset, the object motion offset, and the first depth map into a pre-trained convolutional network to obtain a first high-dimensional feature matrix after compensation of the video frame to be compensated; and obtaining a target depth map after compensation of the video frame to be compensated according to the first high-dimensional feature matrix.

[0009] In a possible implementation of the first aspect, obtaining a target depth map of the compensated video frame to be compensated according to the first high-dimensional feature matrix includes: obtaining a third depth map and a third camera pose matrix of the video frame to be compensated, and a fourth depth map and a fourth camera pose matrix of the adjacent frame according to the first high-dimensional feature matrix; determining a second camera motion offset of a pixel in the video frame to be compensated according to the third depth map, the third camera pose matrix, the fourth depth map, the fourth camera pose matrix, and the camera intrinsic parameter; compensating the third depth map pixel by pixel according to the second camera motion offset and the object motion offset to obtain a second high-dimensional feature matrix of the compensated video frame to be compensated; executing the same steps as the first high-dimensional feature matrix on the second high-dimensional feature matrix until a preset number of executions is reached; and obtaining a target depth map of the compensated video frame to be compensated according to the second high-dimensional feature matrix output after executing the predicted number of times.

[0010] In this embodiment, a one-stage initial network and a two-stage iterative refinement network are combined to first obtain a first depth map and a first camera pose matrix, and then the same steps as the first high-dimensional feature matrix are performed on the second high-dimensional feature matrix until a preset number of executions is reached. That is to say, the second high-dimensional feature matrix is ​​input into the network structure for multiple iterations. By refining the depth map and the camera pose matrix through multiple iterations, the stability of the depth map estimated from multiple consecutive video frames is improved.

[0011] In a possible implementation of the first aspect, the obtaining of a first depth map and a first camera pose matrix of a video frame to be compensated in a continuous plurality of video frames, and a second depth map and a second camera pose matrix of an adjacent frame of the video frame to be compensated, includes: inputting the video frame to be compensated and the adjacent frame in the continuous plurality of video frames into a trained first depth estimation network to obtain the first depth map and the second depth map; inputting the video frame to be compensated and the adjacent frame in the continuous plurality of video frames into a trained first camera pose estimation network to obtain the first camera pose matrix and the second camera pose matrix.

[0012] In a possible implementation of the first aspect, obtaining a third depth map and a third camera pose matrix of the video frame to be compensated, and a fourth depth map and a fourth camera pose matrix of the adjacent frame according to the first high-dimensional feature matrix includes: inputting the first high-dimensional feature matrix into a trained second depth estimation network to obtain the third depth map of the video frame to be compensated and the fourth depth map of the adjacent frame; inputting the first high-dimensional feature matrix into a trained second camera pose estimation network to obtain the third camera pose matrix of the video frame to be compensated and the fourth camera pose matrix of the adjacent frame.

[0013] In a possible implementation of the first aspect, the pre-trained convolutional network is a deformable convolutional network, and the compensation offset of the deformable convolutional network includes the first camera motion offset and the object motion offset. In this embodiment, the deformable convolutional network can directly know which offsets need to be compensated. Compared with the ordinary convolutional network, which needs to first learn which of the input parameters are offsets, the compensation accuracy is higher and the network structure is easier to train.

[0014] In a possible implementation manner of the first aspect, the adjacent frames of the video frame to be compensated include a previous frame of the video frame to be compensated and a subsequent frame of the video frame to be compensated.

[0015] In a possible implementation of the first aspect, determining the object motion offset of a pixel point in the video frame to be compensated relative to the same pixel point in the adjacent frame includes: inputting the video frame to be compensated and the adjacent frame into an optical flow network to obtain the object motion offset of the pixel point in the video frame to be compensated relative to the same pixel point in the adjacent frame.

[0016] In a possible implementation of the first aspect, determining the first camera motion offset of the pixel point in the video frame to be compensated according to the first depth map, the first camera pose matrix, the second depth map, the second camera pose matrix and the camera intrinsic parameters includes: calculating the first predicted three-dimensional coordinates of the pixel point in the video frame to be compensated in the world coordinate system according to the first depth map, the first camera pose matrix and the camera intrinsic parameters; calculating the second predicted three-dimensional coordinates of the pixel point in the adjacent frame in the world coordinate system according to the second depth map, the second camera pose matrix and the camera intrinsic parameters; calculating the relative pose transformation matrix of the adjacent frame relative to the video frame to be compensated according to the first camera pose matrix and the second camera pose matrix; transforming the second predicted three-dimensional coordinates according to the relative pose transformation matrix to obtain the ideal three-dimensional coordinates of the pixel point in the video frame to be compensated in the world coordinate system; calculating the first camera motion offset of the pixel point in the video frame to be compensated according to the ideal three-dimensional coordinates and the first predicted three-dimensional coordinates.

[0017] In a possible implementation of the first aspect, the first predicted three-dimensional coordinates of the pixel points in the video frame to be compensated in the world coordinate system are calculated according to the first depth map, the first camera pose matrix and the camera intrinsic parameters, including: converting the first depth map according to the camera intrinsic parameters to obtain the camera coordinates of the pixel points in the video frame to be compensated in the camera coordinate system; converting the camera coordinates according to the first camera pose matrix to obtain the first predicted three-dimensional coordinates of the pixel points in the video frame to be compensated in the world coordinate system.

[0018] In a second aspect, a video depth estimation method device is provided, the device comprising a memory and a processor, the memory being used to store instructions, and when the instructions are executed by the processor, the video depth estimation method device executes the video depth estimation method in the first aspect or any possible implementation of the first aspect.

[0019] In a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed, the video depth estimation method in the first aspect or any possible implementation method of the first aspect is implemented.

[0020] In a fourth aspect, a computer program product is provided, comprising: a computer program code, which, when executed on a computer, enables the computer to execute the video depth estimation method in the first aspect or any possible implementation of the first aspect.

[0021] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is an exemplary flow chart of a video depth estimation method according to a first aspect provided by an embodiment of the present application;

[0023] Figure 2 It is a structural diagram of a video depth estimation method according to a first aspect provided by an embodiment of the present application;

[0024] Figure 3 is an exemplary flow chart of a second aspect of a video depth estimation method provided by an embodiment of the present application;

[0025] Figure 4 is a structural diagram of a video depth estimation method according to a second aspect provided by an embodiment of the present application;

[0026] Figure 5 It is a structural diagram of a video depth estimation device according to a third aspect provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0028] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or. For example, A / B can mean A or B. The “and / or” in this article is merely a way to describe the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0029] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0030] First, the nouns involved in this application are explained:

[0031] (1) RGB image: Also known as a color image, it refers to an image in which each pixel is composed of R (red), G (green), and B (blue) components, where R, G, and B are described by different gray levels.

[0032] (2) Depth images: Depth images, also known as range images, refer to images that use the distance (depth) values ​​of each point in the scene captured by the image collector as pixel values. They directly reflect the geometric shape of the visible surface of the scene. Depth images can be used to calculate point cloud data after coordinate transformation. Point cloud data with rules and necessary information can also be inversely calculated into depth image data.

[0033] (3) Camera intrinsic parameters: refers to the parameters required to project the camera onto the imaging plane, including focal length, principal point position (imaging origin) and pixel size. There are two ways to obtain camera intrinsic parameters. The first is to obtain them through camera calibration, and the second is to use the parameters provided by the camera manufacturer. The use of camera intrinsic parameters can be divided into two cases. One is to map three-dimensional coordinates to two-dimensional images through camera intrinsic parameters; the other is to map two-dimensional pixel coordinates to three-dimensional coordinates through camera intrinsic parameters.

[0034] (4) Camera pose estimation: It is generally divided into two types: camera pose estimation based on feature point matching and camera pose estimation based on inertial measurement unit (IMU). The former is to calculate the camera pose information by assuming that the feature points in the image have a corresponding relationship in the three-dimensional space, and then by matching these feature points. The latter uses sensors such as accelerometers and gyroscopes, combined with the principle of spatial orientation transformation, to calculate the camera pose information. The camera pose information includes the rotation matrix R and the translation vector T.

[0035] (5) World coordinate system: The real-world three-dimensional space coordinate system is a three-dimensional coordinate system used to describe the camera position, with the unit being m.

[0036] (6) Camera coordinate system: Based on the lens imaging principle, the world coordinates are presented in the camera, which is a three-dimensional coordinate system. The origin of the camera coordinate system is the optical center of the camera. The z-axis coincides with the optical axis, that is, the z-axis points to the front of the camera (perpendicular to the imaging plane). The positive directions of the x-axis and y-axis are parallel to the world coordinate system. The unit is m. To convert the world coordinates into the camera coordinates, it is necessary to reduce the scale by equal proportions, plus rotation and translation, which is called rigid body transformation.

[0037] (7) Pixel coordinate system: The image formed by discrete sampling of the projected image coordinates is a two-dimensional coordinate system. The origin of the pixel coordinate system is the upper left corner of the image. The unit is pixel, which is used to represent the pixel length and pixel width of the entire screen.

[0038] At present, the methods for obtaining depth information in images include using depth sensors such as LIDAR sensors, ToF sensors, etc., or using depth estimation algorithms for solving. The problem with using sensors to obtain depth maps is that the equipment is relatively expensive and the acquisition cost is relatively high, especially for obtaining high-precision depth data. In addition, the depth data collected by the sensor is also limited in range and subject to environmental factors. Therefore, if the problem of obtaining depth data can be solved through a depth estimation algorithm, it will greatly save the cost of data acquisition and will also be conducive to the large-scale promotion and use of related applications.

[0039] Depth estimation algorithms can be divided into traditional methods and deep learning-based methods. Traditional methods rely on the extraction and matching of accurate image feature points, which will result in less than ideal depth data in some low-texture areas or when there are occlusions and moving objects in the scene, and the estimated depth map is usually sparse. Deep learning-based methods can solve the problems of traditional methods well and estimate a dense depth result. Depth estimation based on deep learning can be divided into supervised depth estimation and unsupervised depth estimation. The problem with supervised depth estimation is that a large amount of real depth data is required as supervision during the training process, but collecting such real depth data requires a lot of manpower and material resources, which is not worth the cost. Therefore, unsupervised depth estimation is the current research and development trend in this direction. Among them, the deep learning method using monocular video sequences as training data is the most promising method among all the depth estimation methods based on deep learning due to its convenience of data acquisition.

[0040] When training the depth estimation network, the camera motion offset and object motion offset of two adjacent video frames in the continuous video frames can be used as the loss function to constrain the depth estimation network. In this way, the weight of the depth estimation network is adjusted according to the camera motion offset and the object motion offset, and the depth estimation network with adjusted weights is used to predict the depth value of the pixel points of each frame in the continuous video frames. The depth value of each pixel point in the predicted depth map is related to the camera motion offset and the object motion offset of all pixels points in each video frame in the continuous video frames. In other words, the camera motion offset and the object motion offset of all pixels points in each video frame in the continuous video frames act together on each pixel point of the video frame through the adjustment of the weights in the depth estimation network, and the pixels cannot be compensated in a targeted manner. The depth estimated between continuous video frames is unstable.

[0041] In this way, when the estimated depth map of consecutive video frames is applied to the field of video blurring, the blurred front and rear video frames may flicker, or the predicted depth of the edge of the object in the front and rear frames may have errors, and the depth of the edge of the object in the front and rear frames may suddenly change, which is reflected in the appearance of artifacts at the edge of the object in the front and rear video frames. It can be seen that the estimated depth between consecutive video frames is unstable, which may lead to unsatisfactory effects of depth map-based applications (for example, video blurring, 3D reconstruction, autonomous driving, etc.).

[0042] In view of this, an embodiment of the present application provides a video depth estimation method, including: first, obtaining a first depth map and a first camera pose matrix of a video frame to be compensated in a plurality of consecutive video frames, and a second depth map and a second camera pose matrix of an adjacent frame of the video frame to be compensated; secondly, determining a first camera motion offset of a pixel in the video frame to be compensated according to the first depth map, the first camera pose matrix, the second depth map, the second camera pose matrix and the camera intrinsic parameter, the first camera motion offset being the coordinate offset of a pixel in the video frame to be compensated relative to the same pixel in an adjacent frame in the same coordinate system; and determining an object motion offset of a pixel in the video frame to be compensated relative to the same pixel in an adjacent frame; finally, compensating the first depth map pixel by pixel according to the first camera motion offset and the object motion offset to obtain a compensated target depth map of the video frame to be compensated.

[0043] In the embodiment of the present application, the predicted first depth map of the video frame to be compensated is compensated pixel by pixel according to the first camera motion offset of the pixel in the video frame to be compensated relative to the same pixel in the adjacent frame, and the object motion offset of the pixel in the video frame to be compensated relative to the same pixel in the adjacent frame, that is, the camera motion offset and the object motion offset corresponding to a single pixel are applied to the pixel, instead of applying the camera motion offset and the object motion offset corresponding to all pixels to each pixel, and the pixel points of the first depth map of the video frame to be compensated are compensated in a targeted manner. In this way, the estimated depth between the video frame to be compensated and the adjacent frames is relatively stable. When the estimated depth maps of the video frame to be compensated and the adjacent frames are applied to fields such as video blurring or autonomous driving, the video frame to be compensated and the adjacent frames will not flicker, and no artifacts will appear on the edges of objects in the video frame to be compensated and the adjacent frames.

[0044] Figure 1 is an exemplary flow chart of a video depth estimation method, Figure 2 The following is a schematic diagram of the structure of the video depth estimation method. Figure 1 and Figure 2 A video depth estimation method provided in an embodiment of the present application is described in detail.

[0045] Step 110: Obtain a first depth map and a first camera pose matrix of a video frame to be compensated in a plurality of consecutive video frames, and a second depth map and a second camera pose matrix of an adjacent frame of the video frame to be compensated.

[0046] In this embodiment, the continuous multi-frame video frames may be RGB images (hereinafter referred to as color images). In this embodiment, the video frame to be compensated may be each frame in the continuous multi-frame video frames, or may be other frames in the continuous multi-frame video frames except the first frame and the last frame.

[0047] Assuming that the continuous multiple video frames include continuous N video frames, each of the continuous N video frames can be used as a video frame to be compensated in sequence according to the time sequence of the video frames, for example, the first frame is used as a video frame to be compensated, and then the second frame is used as a video frame to be compensated, and so on. Alternatively, each frame from the second frame to the N-1th frame of the continuous N video frames is used as a video frame to be compensated.

[0048] In one example, the adjacent frame of the video frame to be compensated may be the previous frame of the video frame to be compensated. For example, when the first frame is the video frame to be compensated, the video frame to be compensated has no adjacent frame, and the first frame is not compensated at this time; when the second frame is the video frame to be compensated, the adjacent frames of the video frame to be compensated include the first frame; when the third frame is the video frame to be compensated, the adjacent frames of the video frame to be compensated include the second frame, and so on.

[0049] In another example, the adjacent frames of the video frame to be compensated include the previous frame of the video frame to be compensated and the next frame of the video frame to be compensated. For example, when the first frame is the video frame to be compensated, the adjacent frames of the video frame to be compensated include the second frame; when the second frame is the video frame to be compensated, the adjacent frames of the video frame to be compensated include the first frame and the third frame, and so on.

[0050] Exemplarily, obtaining a first depth map and a first camera pose matrix of a video frame to be compensated in a continuous plurality of video frames, and a second depth map and a second camera pose matrix of an adjacent frame of the video frame to be compensated, includes: inputting the video frame to be compensated and the adjacent frame in the continuous plurality of video frames into a trained first depth estimation network to obtain a first depth map and a second depth map; inputting the video frame to be compensated and the adjacent frame in the continuous plurality of video frames into a trained first camera pose estimation network to obtain a first camera pose matrix and a second camera pose matrix.

[0051] In this embodiment, a trained first depth estimation network is used to obtain a first depth map of a video frame to be compensated and a second depth map of an adjacent frame, and a trained first camera pose estimation network is used to obtain a first camera pose matrix of a video frame to be compensated and a second camera pose matrix of an adjacent frame. The first depth estimation network and the first camera pose estimation network can be two networks with independent structures, or the first depth estimation network and the first camera pose estimation network can share part of the network structure.

[0052] Step 120: Determine a first camera motion offset of a pixel in a video frame to be compensated according to the first depth map, the first camera pose matrix, the second depth map, the second camera pose matrix, and camera intrinsic parameters.

[0053] Since the camera's posture may have changed when shooting the video frame to be compensated and the adjacent frames, and the depth estimation network did not take into account the change relationship between the video frame to be compensated and the adjacent frames when estimating the first depth map of the compensated video frame and the second depth map of the adjacent frames, there is a coordinate offset caused by camera motion between the same pixels in the estimated first depth map and the second depth map.

[0054] In this embodiment, it is necessary to determine the coordinate offset of the pixel point in the video frame to be compensated relative to the same pixel point in the adjacent frame in the same coordinate system, that is, the first camera motion offset. Figure 2 First, multiple continuous video frames (including compensated video frames and adjacent frames) are respectively input into the first depth estimation network and the first camera pose estimation network. The first depth estimation network outputs multiple depth maps of the continuous multiple video frames, and the multiple depth maps include the first depth map of the video frame to be compensated and the second depth map of the video frame. The first camera pose estimation network outputs multiple camera pose matrices of the continuous multiple video frames, and the multiple camera pose matrices include the first camera pose matrix of the compensated video frame and the second camera pose matrix of the adjacent frame. Secondly, the first depth map, the first camera pose matrix, the second depth map, the second camera pose matrix and the camera intrinsic parameters can be input into the pre-trained residual network, and the residual network determines the first camera motion offset of the pixel points in the video frame to be compensated.

[0055] In some examples, the first camera motion offset of the pixel in the to-be-compensated video frame may also be calculated in the following manner: In this embodiment, the coordinate offset of the pixel in the to-be-compensated video frame relative to the same pixel in an adjacent frame in the same world coordinate system is determined.

[0056] First, the first predicted three-dimensional coordinates of the pixel points in the video frame to be compensated in the world coordinate system are calculated according to the first depth map, the first camera pose matrix and the camera intrinsic parameters, and the second predicted three-dimensional coordinates of the pixel points in the adjacent frame in the world coordinate system are calculated according to the second depth map, the second camera pose matrix and the camera intrinsic parameters. After that, the relative pose transformation matrix of the video frame to be compensated relative to the adjacent frame is calculated according to the first camera pose matrix and the second camera pose matrix; the second predicted three-dimensional coordinates are transformed according to the relative pose transformation matrix to obtain the ideal three-dimensional coordinates of the pixel points in the video frame to be compensated in the world coordinate system. Finally, the first camera motion offset of the pixel points in the video frame to be compensated is calculated according to the ideal three-dimensional coordinates and the first predicted three-dimensional coordinates.

[0057] Specifically, the camera intrinsic parameters include focal length f, principal point position (imaging origin) cx and cy, and pixel size dx and dy. In this embodiment, the camera intrinsic parameters can be obtained by reading the intrinsic parameters of the camera that captures multiple consecutive video frames, and for video frames for which the camera intrinsic parameters cannot be obtained, the same set of preset camera intrinsic parameters can be used.

[0058] First, according to the first depth map, the first camera pose matrix and the camera intrinsic parameters, the first predicted three-dimensional coordinates of the pixel point in the video frame to be compensated in the world coordinate system are calculated, and according to the second depth map, the second camera pose matrix and the camera intrinsic parameters, the second predicted three-dimensional coordinates of the pixel point in the adjacent frame in the world coordinate system are calculated.

[0059] In this embodiment, the first depth map in the pixel coordinate system is converted to the world coordinate system. It is necessary to first convert the first depth map in the pixel coordinate system to the camera coordinate system, and then convert the camera coordinate system to the world coordinate system. Specifically, it includes: converting the first depth map according to the camera internal parameters to obtain the camera coordinates of the pixel points in the video frame to be compensated in the camera coordinate system; converting the camera coordinates according to the first camera pose matrix to obtain the first predicted three-dimensional coordinates of the pixel points in the video frame to be compensated in the world coordinate system.

[0060] Assume that the pixel coordinates of the pixel point in the video frame to be compensated are [uv 1] T , then the camera coordinates of the pixel point in the video frame to be compensated in the camera coordinate system [xyz] T It can be calculated by the following formula (1):

[0061]

[0062] Assuming that the first camera pose matrix includes a rotation matrix R and a translation variable t, the first camera pose matrix K1 can be expressed as formula (2). The first camera pose matrix K1 is a matrix with 3 rows and 4 columns.

[0063] (2)

[0064] Then, the first predicted three-dimensional coordinates of the pixel point in the video frame to be compensated in the world coordinate system are [X1 Y1 Z1 1] T It can be calculated by the following formula (3):

[0065]

[0066] The above formula can be used to calculate the first predicted three-dimensional coordinates (i.e., world coordinates) of the pixel point in the video frame to be compensated in the world coordinate system. It is worth noting that for each pixel point in the adjacent frame, the second predicted three-dimensional coordinates in the world coordinate system [X2 Y2 Z21] T It can also be calculated in the above manner, and will not be described again here to avoid repetition.

[0067] Secondly, according to the first camera pose matrix and the second camera pose matrix, a relative pose transformation matrix of the video frame to be compensated relative to the adjacent frames is calculated.

[0068] Assume that the first camera pose matrix of the video frame to be compensated is A (a matrix with 3 rows and 4 columns), the first camera pose matrix of the adjacent frame is B (a matrix with 3 rows and 4 columns), and the relative pose transformation matrix of the video frame to be compensated relative to the adjacent frame is M, that is, BM = A, then the relative pose transformation matrix M = AB -1 It can be obtained that the relative posture transformation matrix M is a matrix with 4 rows and 4 columns.

[0069] Afterwards, the second predicted three-dimensional coordinates are transformed according to the relative posture transformation matrix to obtain the ideal three-dimensional coordinates of the pixel points in the video frame to be compensated in the world coordinate system.

[0070] According to the relative pose transformation matrix M of the video frame to be compensated relative to the adjacent frame, the second predicted three-dimensional coordinate [X2Y2 Z21] T Perform the conversion, assuming that the ideal three-dimensional coordinates of the pixel point in the video frame to be compensated in the world coordinate system after the conversion are [X0 Y0 Z01] T Then, the ideal three-dimensional coordinates [X0 Y0 Z01] T It can be calculated by the following formula (4):

[0071]

[0072] Finally, according to the ideal three-dimensional coordinates [X0 Y0 Z01] T and the first predicted 3D coordinates [X1 Y1 Z1 1] T , calculate the first camera motion offset of the pixel point in the video frame to be compensated.

[0073] The first camera motion offset can be the ideal three-dimensional coordinates of the pixel point in the video frame to be compensated [X0 Y0 Z01] T and the first predicted 3D coordinates [X1 Y1 Z1 1] T For example, the ideal three-dimensional coordinates [X0 Y0 Z01] can be calculated by the Euclidean distance calculation formula. T and the first predicted 3D coordinates [X1 Y1 Z1 1] T In practical applications, other distance calculation methods may also be used to calculate the distance between the ideal three-dimensional coordinates and the first predicted three-dimensional coordinates, which is not limited here.

[0074] In other examples, the first camera motion offset of the pixel in the to-be-compensated video frame may also be calculated in the following manner: In this embodiment, the coordinate offset of the pixel in the to-be-compensated video frame relative to the same pixel in an adjacent frame in the same camera coordinate system is determined.

[0075] First, according to the first depth map and the camera intrinsic parameters, the first predicted camera coordinates of the pixel points in the video frame to be compensated in the camera coordinate system are calculated, and according to the second depth map, the second camera pose matrix and the camera intrinsic parameters, the second predicted three-dimensional coordinates of the pixel points in the adjacent frame in the world coordinate system are calculated. After that, according to the first camera pose matrix and the second camera pose matrix, the relative pose transformation matrix of the video frame to be compensated relative to the adjacent frame is calculated; according to the relative pose transformation matrix, the second predicted three-dimensional coordinates are transformed to obtain the ideal three-dimensional coordinates of the pixel points in the video frame to be compensated in the world coordinate system. Then, the ideal three-dimensional coordinates are transformed by the first camera pose matrix to obtain the ideal camera coordinates. Finally, according to the ideal camera coordinates and the first predicted camera coordinates, the first camera motion offset of the pixel points in the video frame to be compensated is calculated.

[0076] Step 130: Determine the object motion offset of a pixel point in the to-be-compensated video frame relative to the same pixel point in an adjacent frame.

[0077] Specifically, when the camera shoots the video frame to be compensated and the adjacent frames, the movement of the object will also cause the position of the object in the compensated video frame and the adjacent frames to be different. That is to say, the pixel coordinates of the same pixel points of the same object in the video frame to be compensated and the adjacent frames are not the same. The distance between the pixel coordinates of the pixel points of the same object in the video frame to be compensated and the adjacent frames is the object motion offset.

[0078] In this embodiment, it is also necessary to determine the object motion offset of a pixel point in the to-be-compensated video frame relative to the same pixel point in an adjacent frame.

[0079] Exemplarily, the RGB images of the video frame to be compensated and the adjacent frames can be input into a pre-trained convolutional network to determine the object motion offset of the pixel points in the video frame to be compensated. The pre-trained convolutional network here can be an optical flow network. For example, the object motion offset of the pixel points in the video frame to be compensated determined by the convolutional network can be expressed as (-4, 5), which means that the pixel point in the video frame to be compensated is offset by 4 (in millimeters) in the negative axis direction of the X direction relative to the pixel coordinates corresponding to the same pixel point in the adjacent frame, and is offset by 5 in the positive axis direction of the Y direction. The object motion offset includes not only the size of the offset but also the direction of the offset.

[0080] Step 140: Input the first camera motion offset, the object motion offset, and the first depth map into a pre-trained convolutional network to obtain a first high-dimensional feature matrix after compensation of the video frame to be compensated.

[0081] See also Figure 2 , the first camera motion offset and the object motion offset, and the first depth map of the video frame to be compensated are input into a pre-trained convolutional network, and the first camera motion offset and the object motion offset corresponding to the pixel points in the video frame to be compensated are fused and compensated into the first depth map of the video frame to be compensated through the trained convolutional network to obtain a compensated first high-dimensional feature matrix.

[0082] For example, Figure 2 As shown, the pre-trained convolutional network can be a deformable convolutional network, and the compensation offset of the deformable convolutional network includes the first camera motion offset and the object motion offset. Assume that the first camera motion offset of the pixel point in the video frame to be compensated is 3, and the object motion offset of the pixel point in the video frame to be compensated is (-4, 5). The compensation idea of ​​the deformable convolutional network is: determine the pixel coordinates of the pixel point after the offset according to the first camera motion offset of the pixel point in the video frame to be compensated, and compensate the depth value of the pixel coordinates where the pixel point is located to the pixel coordinates after the offset. Assume that the depth value of the pixel coordinates where the pixel point is located is 10, and the original depth value of the pixel coordinates after the offset is 5. Then, after compensating the depth value of the pixel coordinates where the pixel point is located to the pixel coordinates after the offset, the depth value of the pixel coordinates after the offset is 15. And determine the pixel coordinates after the offset according to the object motion offset of the pixel point in the video frame to be compensated. If the pixel coordinates of the pixel point are (1, 1) and the object motion offset is (-4, 5), then the pixel coordinates after the pixel offset are (-3, 6). The depth value of the pixel coordinates where the pixel point is located is compensated to the pixel coordinates after the offset. The above is only an example. In actual applications, the deformable convolutional network can not only compensate the depth value of the offset pixel coordinates, but also compensate the depth value of the coordinates adjacent to the pixel coordinates where the pixel point is located.

[0083] In this embodiment, the deformable convolutional network can directly know which offsets need to be compensated. Compared with the ordinary convolutional network that needs to first learn which input parameters are offsets, the compensation accuracy is higher and the network structure is easier to train.

[0084] Step 150: Obtain a target depth map of the compensated video frame according to the first high-dimensional feature matrix.

[0085] See also Figure 2 , the compensated first high-dimensional feature matrix can be input into the third depth estimation network to restore the compensated target depth map of the video frame to be compensated.

[0086] In this embodiment, the predicted first depth map of the video frame to be compensated is compensated pixel by pixel according to the first camera motion offset of the pixel in the video frame to be compensated relative to the same pixel in the adjacent frame, and the object motion offset of the pixel in the video frame to be compensated relative to the same pixel in the adjacent frame, that is, the camera motion offset and the object motion offset corresponding to a single pixel are applied to the pixel, instead of applying the camera motion offset and the object motion offset corresponding to all pixels to each pixel, and the pixel points of the first depth map of the video frame to be compensated are compensated in a targeted manner. In this way, the estimated depth between the video frame to be compensated and the adjacent frames is relatively stable. When the estimated depth maps of the video frame to be compensated and the adjacent frames are applied to fields such as video blurring or autonomous driving, the video frame to be compensated and the adjacent frames will not flicker, and no artifacts will appear on the edges of objects in the video frame to be compensated and the adjacent frames.

[0087] In a second aspect, this embodiment also provides a video depth estimation method, such as Figure 3 : Shown is a flowchart of a video depth estimation method. Figure 4 The following is a schematic diagram of the structure of the video depth estimation method. Figure 3 and attached Figure 4 The video depth estimation method is described.

[0088] Step 210: Obtain a first depth map and a first camera pose matrix of a video frame to be compensated in a plurality of consecutive video frames, and a second depth map and a second camera pose matrix of an adjacent frame of the video frame to be compensated.

[0089] Step 220: Determine a first camera motion offset of a pixel in a video frame to be compensated according to the first depth map, the first camera pose matrix, the second depth map, the second camera pose matrix, and camera intrinsic parameters.

[0090] Step 230: Determine the object motion offset of a pixel point in the video frame to be compensated relative to the same pixel point in an adjacent frame

[0091] Step 240: Input the first camera motion offset, the object motion offset, and the first depth map into a pre-trained convolutional network to obtain a first high-dimensional feature matrix after compensation of the video frame to be compensated.

[0092] The above steps 210 to 240 are the same as steps 110 to 140 in the above embodiment, and will not be described again to avoid repetition.

[0093] Step 250: According to the first high-dimensional feature matrix, obtain a third depth map and a third camera pose matrix of the video frame to be compensated, and a fourth depth map and a fourth camera pose matrix of an adjacent frame.

[0094] The process of the video depth estimation method in this embodiment can be found in Figure 4 , input the first high-dimensional feature matrix into the trained second depth estimation network to obtain the third depth map of the video frame to be compensated and the fourth depth map of the adjacent frame; input the first high-dimensional feature matrix into the trained second camera pose estimation network to obtain the third camera pose matrix of the video frame to be compensated and the fourth camera pose matrix of the adjacent frame.

[0095] The structure of the second depth estimation network in this embodiment is roughly the same as that of the first depth estimation network in the first aspect embodiment described above, but due to different inputs, the weight parameters of the trained first depth estimation network and the weight parameters of the trained second depth estimation network are different. The input of the first depth estimation network is an RGB image, and the output is a depth map corresponding to the RGB image, and the input of the second depth estimation network is a high-dimensional feature matrix, and the output is a depth map corresponding to the high-dimensional feature matrix.

[0096] The structure of the second camera pose estimation network is roughly the same as that of the first camera pose estimation network, but due to different inputs, the weight parameters of the trained first camera pose estimation network and the weight parameters of the trained second camera pose estimation network are different. The input of the first camera pose estimation network is an RGB image, and the output is a camera pose matrix corresponding to the RGB image. The input of the second camera pose estimation network is a high-dimensional feature matrix, and the output is a camera pose matrix corresponding to the high-dimensional feature matrix.

[0097] Step 260: Determine a second camera motion offset of a pixel in the video frame to be compensated according to the third depth map, the third camera pose matrix, the fourth depth map, the fourth camera pose matrix, and the camera intrinsic parameters.

[0098] See also Figure 4In this embodiment, the third depth map, the third camera pose matrix, the fourth depth map, the fourth camera pose matrix and the camera intrinsic parameters are input into the pre-trained residual network to determine the second camera motion offset of the pixel points in the video frame to be compensated. In this embodiment, the method for determining the second camera motion offset according to the third depth map, the third camera pose matrix, the fourth depth map, the fourth camera pose matrix and the camera intrinsic parameters is substantially the same as the method for determining the first camera motion offset of the pixel points in the video frame to be compensated according to the first depth map, the first camera pose matrix, the second depth map, the second camera pose matrix and the camera intrinsic parameters in the above embodiment. The difference lies in the different input parameters. The specific implementation of step 260 can refer to the specific description of the above embodiment, which will not be repeated here.

[0099] Step 270: Compensate the third depth map pixel by pixel according to the second camera motion offset and the object motion offset to obtain a second high-dimensional feature matrix after compensation of the video frame to be compensated.

[0100] See also Figure 4 In this embodiment, the second camera motion offset, the object motion offset, and the third depth map can be input into a pre-trained convolutional network, and the second camera motion offset and the object motion offset corresponding to the pixel points in the video frame to be compensated are fused and compensated into the third depth map of the video frame to be compensated through the trained convolutional network to obtain a compensated second high-dimensional feature matrix.

[0101] It is worth noting that the implementation method of "inputting the second camera motion offset, the object motion offset, and the third depth map into a pre-trained convolutional network to obtain a second high-dimensional feature matrix after compensation of the video frame to be compensated" in this embodiment is roughly the same as the implementation method of "inputting the first camera motion offset, the object motion offset, and the first depth map into a pre-trained convolutional network to obtain a first high-dimensional feature matrix after compensation of the video frame to be compensated" in the above embodiment. The difference lies in the different input parameters. The specific implementation of step 270 can refer to the specific description of the above embodiment, which will not be repeated here.

[0102] Step 280: Execute the same steps as the first high-dimensional feature matrix on the second high-dimensional feature matrix until a preset number of executions is reached.

[0103] Step 290: Obtain a target depth map after compensation of the video frame to be compensated according to the second high-dimensional feature matrix output after executing the prediction times.

[0104] See also Figure 4In this embodiment, the first-stage initial network and the second-stage iterative refinement network are combined. The first-stage network is used to obtain the compensated first high-dimensional feature matrix. After that, the first high-dimensional feature matrix is ​​input into the second-stage iterative refinement network to obtain the second high-dimensional feature matrix, and the second high-dimensional feature matrix is ​​subjected to the same steps as the first high-dimensional feature matrix until the preset number of executions P is reached. That is, the second high-dimensional feature matrix is ​​input into the network structure for multiple iterations. The depth map and the camera pose matrix are refined through multiple iterations, thereby improving the stability of the depth map estimated from multiple consecutive video frames. It should be noted that the preset number of executions P can be set as needed, for example, P can be 3, 4, etc.

[0105] It should be understood that the above examples are intended to help those skilled in the art understand the embodiments of the present application, rather than to limit the embodiments of the present application to the specific numerical values ​​or specific scenarios illustrated. Those skilled in the art can obviously make various equivalent modifications or changes based on the above examples, and such modifications or changes also fall within the scope of the embodiments of the present application.

[0106] Figure 5 3 is a schematic structural block diagram of a video depth estimation device 300 provided in an embodiment of the present application. The video depth estimation device 300 includes: a processor 310 , a memory 320 , a communication interface 330 , and a bus 340 .

[0107] The processor 310 may be connected to a memory 320. The memory 320 may be used to store the program code and data. Therefore, the memory 320 may be a storage unit inside the processor 310, or an external storage unit independent of the processor 310, or a component including a storage unit inside the processor 310 and an external storage unit independent of the processor 310.

[0108] Optionally, the video depth estimation device 300 may further include a bus 340. The memory 320 and the communication interface 330 may be connected to the processor 310 via the bus 340. The bus 340 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus 340 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The fact that only one line is used in the diagram does not mean that there is only one bus or only one type of bus.

[0109] It should be understood that in the embodiment of the present application, the processor 310 may adopt a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. Alternatively, the processor 310 may adopt one or more integrated circuits to execute relevant programs to implement the technical solutions provided in the embodiment of the present application.

[0110] The memory 320 may include a read-only memory and a random access memory, and provides instructions and data to the processor 310. A portion of the processor 310 may also include a non-volatile random access memory. For example, the processor 310 may also store information on the device type.

[0111] When the video depth estimation apparatus is running, the processor 310 executes the computer-executable instructions in the memory 320 to use the hardware resources in the video depth estimation apparatus to perform the operation steps of the above-mentioned video depth estimation method.

[0112] It should be understood that the video depth estimation device 300 according to the embodiment of the present application may correspond to the method of executing the video depth estimation device 300 according to the embodiment of the present application. Figure 1 and Figure 3 The corresponding subjects in the method shown in the figure, and the above and other operations and / or functions of each module in the video depth estimation device 300 are respectively implemented Figure 1 and Figure 3 For the sake of brevity, the corresponding process of the method in will not be repeated here.

[0113] The present application also provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed, the video depth estimation method provided in the embodiment of the present application is implemented.

[0114] The present application also provides a computer program product, which includes: a computer program code, when the computer program code is run on a computer, the computer executes the video depth estimation method provided by the embodiment of the present application.

[0115] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid state drive (SSD).

[0116] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0117] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0118] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0119] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0120] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0121] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a memory (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

[0122] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiment of the present application, which should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.

Claims

1. A video depth estimation method, characterized in that: The method comprises: Acquire a first depth map and a first camera pose matrix of a video frame to be compensated in a plurality of consecutive video frames, and a second depth map and a second camera pose matrix of an adjacent frame of the video frame to be compensated; Determine a first camera motion offset of a pixel in the to-be-compensated video frame according to the first depth map, the first camera pose matrix, the second depth map, the second camera pose matrix, and camera intrinsic parameters, wherein the first camera motion offset is a coordinate offset of a pixel in the to-be-compensated video frame relative to the same pixel in the adjacent frame in the same coordinate system; Determine an object motion offset of a pixel point in the to-be-compensated video frame relative to the same pixel point in the adjacent frame; The first depth map is compensated pixel by pixel according to the first camera motion offset and the object motion offset to obtain a compensated target depth map of the video frame to be compensated.

2. The video depth estimation method according to claim 1, characterized in that: The compensating the first depth map pixel by pixel according to the first camera motion offset and the object motion offset to obtain the target depth map after compensation of the video frame to be compensated includes: Inputting the first camera motion offset, the object motion offset, and the first depth map into a pre-trained convolutional network to obtain a first high-dimensional feature matrix after compensation of the video frame to be compensated; A target depth map of the compensated video frame to be compensated is obtained according to the first high-dimensional feature matrix.

3. The video depth estimation method according to claim 2, characterized in that: The step of obtaining a target depth map of the video frame to be compensated after compensation according to the first high-dimensional feature matrix includes: According to the first high-dimensional feature matrix, obtaining a third depth map and a third camera pose matrix of the video frame to be compensated, and a fourth depth map and a fourth camera pose matrix of the adjacent frame; Determine a second camera motion offset of a pixel in the video frame to be compensated according to the third depth map, the third camera pose matrix, the fourth depth map, the fourth camera pose matrix and the camera intrinsic parameter; Compensating the third depth map pixel by pixel according to the second camera motion offset and the object motion offset to obtain a second high-dimensional feature matrix after compensation of the video frame to be compensated; Performing the same steps as those of the first high-dimensional feature matrix on the second high-dimensional feature matrix until a preset number of executions is reached; According to the second high-dimensional feature matrix output after executing the prediction times, a target depth map after compensation of the video frame to be compensated is obtained.

4. The video depth estimation method according to any one of claims 1 to 3, characterized in that: The step of acquiring a first depth map and a first camera pose matrix of a video frame to be compensated in a plurality of consecutive video frames, and a second depth map and a second camera pose matrix of an adjacent frame of the video frame to be compensated, comprises: Inputting the to-be-compensated video frame and the adjacent frames in the continuous multiple video frames into a trained first depth estimation network to obtain the first depth map and the second depth map; The video frame to be compensated and the adjacent frames in the continuous multiple video frames are input into a trained first camera pose estimation network to obtain the first camera pose matrix and the second camera pose matrix.

5. The video depth estimation method according to claim 3, characterized in that: The step of acquiring, according to the first high-dimensional feature matrix, a third depth map and a third camera pose matrix of the video frame to be compensated, and a fourth depth map and a fourth camera pose matrix of the adjacent frame comprises: Inputting the first high-dimensional feature matrix into a trained second depth estimation network to obtain the third depth map of the video frame to be compensated and the fourth depth map of the adjacent frame; The first high-dimensional feature matrix is ​​input into a trained second camera pose estimation network to obtain the third camera pose matrix of the video frame to be compensated and the fourth camera pose matrix of the adjacent frame.

6. The video depth estimation method according to claim 2, characterized in that: The pre-trained convolutional network is a deformable convolutional network, and the compensation offset of the deformable convolutional network includes the first camera motion offset and the object motion offset.

7. The video depth estimation method according to any one of claims 1 to 6, characterized in that: The adjacent frames of the video frame to be compensated include a previous frame of the video frame to be compensated and a subsequent frame of the video frame to be compensated.

8. The video depth estimation method according to any one of claims 1 to 7, characterized in that: The determining of the object motion offset of the pixel point in the to-be-compensated video frame relative to the same pixel point in the adjacent frame includes: The video frame to be compensated and the adjacent frame are input into an optical flow network to obtain an object motion offset of a pixel point in the video frame to be compensated relative to the same pixel point in the adjacent frame.

9. The video depth estimation method according to any one of claims 1 to 8, characterized in that: The determining, according to the first depth map, the first camera pose matrix, the second depth map, the second camera pose matrix, and the camera intrinsic parameter, a first camera motion offset of a pixel point in the video frame to be compensated includes: Calculate, according to the first depth map, the first camera pose matrix and the camera intrinsic parameter, a first predicted three-dimensional coordinate of a pixel point in the video frame to be compensated in a world coordinate system; Calculate the second predicted three-dimensional coordinates of the pixel point in the adjacent frame in the world coordinate system according to the second depth map, the second camera pose matrix and the camera intrinsic parameter; Calculating a relative pose transformation matrix of the adjacent frames relative to the video frame to be compensated according to the first camera pose matrix and the second camera pose matrix; The second predicted three-dimensional coordinates are transformed according to the relative posture transformation matrix to obtain the ideal three-dimensional coordinates of the pixel point in the video frame to be compensated in the world coordinate system; A first camera motion offset of a pixel point in the to-be-compensated video frame is calculated according to the ideal three-dimensional coordinates and the first predicted three-dimensional coordinates.

10. The video depth estimation method according to claim 9, characterized in that: The calculating, according to the first depth map, the first camera pose matrix and the camera intrinsic parameter, a first predicted three-dimensional coordinate of a pixel point in the video frame to be compensated in a world coordinate system comprises: Convert the first depth map according to the camera intrinsic parameter to obtain the camera coordinates of the pixel points in the video frame to be compensated in the camera coordinate system; The camera coordinates are transformed according to the first camera pose matrix to obtain first predicted three-dimensional coordinates of the pixel points in the video frame to be compensated in the world coordinate system.

11. A video depth estimation method and device, characterized in that: The device comprises a memory and a processor, wherein the memory is used to store instructions, and when the instructions are executed by the processor, the video depth estimation method device performs the video depth estimation method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed, the video depth estimation method according to any one of claims 1 to 10 is implemented.

13. A computer program product, characterized in that The computer program product comprises: a computer program code, and when the computer program code is run on a computer, the computer is enabled to perform the video depth estimation method according to any one of claims 1 to 10.

Images