Video Depth Estimation Method, Device, Storage Medium and Program Product

Abstract

The present application provides a video depth estimation method, apparatus, storage medium, and program product. In this method, according to the first camera motion offset of a pixel point in a video frame to be compensated relative to the same pixel point in an adjacent frame, and the object motion offset of the pixel point in the video frame to be compensated relative to the same pixel point in the adjacent frame, the first depth map of the predicted video frame to be compensated is compensated pixel by pixel, that is, the camera motion offset and the object motion offset corresponding to a single pixel point are applied to this pixel point, and the pixel points of the first depth map of the video frame to be compensated are compensated specifically. In this way, the estimated depth between the video frame to be compensated and the adjacent frame is relatively stable. When the estimated depth maps of the video frame to be compensated and the adjacent frame are applied to fields such as video defocusing or autonomous driving, there will be no flickering phenomenon between the video frame to be compensated and the adjacent frame, and there will be no artifacts at the object edges in the video frame to be compensated and the adjacent frame.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular, to a video depth estimation method, apparatus, storage medium, and program product. Background Art

[0002] Depth estimation is a fundamental task in computer vision, aiming to estimate depth from 2D images. The input of this task is an RGB image, and the output is a depth map. Here, the depth refers to the distance from the photographed object to the camera optical center, and the problem to be solved by depth estimation is to solve the distance from the photographed object in the shooting scene to the camera optical center. Depth estimation has broad application prospects and values. For example, in the currently popular field of autonomous driving, as well as traditional three-dimensional reconstruction, augmented reality, video defocusing, etc., related technologies of depth estimation are required.

[0003] Currently, when training a depth estimation network, the camera motion offset and object motion offset between two adjacent video frames in consecutive video frames are used as a loss function to constrain the depth estimation network. The depth estimated between consecutive video frames is unstable, resulting in a phenomenon where the front and back video frames may flicker when applying the estimated depth maps of consecutive video frames to fields such as video defocusing or autonomous driving, or artifacts may appear at the object edges in the front and back video frames. Summary of the Invention

[0004] This application provides a video depth estimation method, apparatus, storage medium, and program product, aiming to solve the technical problem that the depth estimated between consecutive video frames is unstable, resulting in a phenomenon where the front and back video frames may flicker when applying the estimated depth maps of consecutive video frames to fields such as video defocusing or autonomous driving, or artifacts may appear at the object edges in the front and back video frames.

[0005] To achieve the above object, this application adopts the following technical solutions:

[0006] In a first aspect, a video depth estimation method is provided. First, obtain the first depth map and the first camera pose matrix of the video frame to be compensated in multiple consecutive video frames, and the second depth map and the second camera pose matrix of the adjacent frame of the video frame to be compensated; then, according to the first depth map, the first camera pose matrix, the second depth map, the second camera pose matrix, and the camera internal parameters, determine the first camera motion offset of the pixel points in the video frame to be compensated, where the first camera motion offset is the coordinate offset of the pixel points in the video frame to be compensated relative to the same pixel points in the adjacent frame in the same coordinate system; determine the object motion offset of the pixel points in the video frame to be compensated relative to the same pixel points in the adjacent frame; finally, according to the first camera motion offset and the object motion offset, compensate each pixel of the first depth map to obtain the target depth map after compensation of the video frame to be compensated.

[0007] Based on the above technical solution, according to the first camera motion offset of a pixel in the video frame to be compensated relative to the same pixel in the adjacent frame, and the object motion offset of the pixel in the video frame to be compensated relative to the same pixel in the adjacent frame, the first depth map of the predicted video frame to be compensated is compensated pixel by pixel, that is, the camera motion offset and the object motion offset corresponding to a single pixel are applied to this pixel, rather than applying the camera motion offsets and object motion offsets corresponding to all pixels to each pixel. The pixels of the first depth map of the video frame to be compensated are compensated in a targeted manner. In this way, the estimated depth between the video frame to be compensated and the adjacent frame is relatively stable. When the depth maps of the estimated video frame to be compensated and the adjacent frame are applied to fields such as video defocusing or autonomous driving, there will be no flickering phenomenon between the video frame to be compensated and the adjacent frame, and there will be no artifacts at the object edges in the video frame to be compensated and the adjacent frame.

[0008] In a possible implementation manner of the first aspect, the compensating the first depth map pixel by pixel according to the first camera motion offset and the object motion offset to obtain the target depth map of the compensated video frame to be compensated includes: inputting the first camera motion offset, the object motion offset, and the first depth map into a pre-trained convolutional network to obtain a first high-dimensional feature matrix of the compensated video frame to be compensated; and obtaining the target depth map of the compensated video frame to be compensated according to the first high-dimensional feature matrix.

[0009] In a possible implementation manner of the first aspect, the obtaining the target depth map of the compensated video frame to be compensated according to the first high-dimensional feature matrix includes: obtaining a third depth map and a third camera pose matrix of the video frame to be compensated, a fourth depth map and a fourth camera pose matrix of the adjacent frame according to the first high-dimensional feature matrix; determining a second camera motion offset of a pixel in the video frame to be compensated according to the third depth map, the third camera pose matrix, the fourth depth map, the fourth camera pose matrix, and the camera intrinsics; compensating the third depth map pixel by pixel according to the second camera motion offset and the object motion offset to obtain a second high-dimensional feature matrix of the compensated video frame to be compensated; performing the same steps as the first high-dimensional feature matrix on the second high-dimensional feature matrix until a preset number of executions is reached; and obtaining the target depth map of the compensated video frame to be compensated according to the second high-dimensional feature matrix output after the prediction number of executions.

[0010] In this embodiment, the first-stage initial network and the second-stage iterative refinement network are combined. First, a first depth map and a first camera pose matrix are obtained. The same steps as those for the first high-dimensional feature matrix are performed on the second high-dimensional feature matrix until a preset number of executions is reached. That is, the second high-dimensional feature matrix is input into the network structure for multiple iterations. By iteratively refining the depth map and the camera pose matrix multiple times, the stability of the depth map estimated from consecutive video frames is improved.

[0011] In a possible implementation manner of the first aspect, obtaining the first depth map and the first camera pose matrix of the video frame to be compensated in consecutive video frames, and the second depth map and the second camera pose matrix of the adjacent frame of the video frame to be compensated includes: inputting the video frame to be compensated and the adjacent frame in the consecutive video frames into a trained first depth estimation network to obtain the first depth map and the second depth map; inputting the video frame to be compensated and the adjacent frame in the consecutive video frames into a trained first camera pose estimation network to obtain the first camera pose matrix and the second camera pose matrix.

[0012] In a possible implementation manner of the first aspect, obtaining the third depth map and the third camera pose matrix of the video frame to be compensated, and the fourth depth map and the fourth camera pose matrix of the adjacent frame according to the first high-dimensional feature matrix includes: inputting the first high-dimensional feature matrix into a trained second depth estimation network to obtain the third depth map of the video frame to be compensated and the fourth depth map of the adjacent frame; inputting the first high-dimensional feature matrix into a trained second camera pose estimation network to obtain the third camera pose matrix of the video frame to be compensated and the fourth camera pose matrix of the adjacent frame.

[0013] In a possible implementation manner of the first aspect, the pre-trained convolutional network is a deformable convolutional network, and the compensation offset of the deformable convolutional network includes the first camera motion offset and the object motion offset. In this embodiment, the deformable convolutional network can directly know which offsets need to be compensated. Compared with a general convolutional network that needs to first learn which of the input parameters are offsets, the compensation accuracy is higher and the network structure is easier to train.

[0014] In a possible implementation manner of the first aspect, the adjacent frame of the video frame to be compensated includes the previous frame and the next frame of the video frame to be compensated.

[0015] In a possible implementation of the first aspect, determining the object motion offset of a pixel in the video frame to be compensated relative to the same pixel in the adjacent frame includes: inputting the video frame to be compensated and the adjacent frame into an optical flow network to obtain the object motion offset of a pixel in the video frame to be compensated relative to the same pixel in the adjacent frame.

[0016] In a possible implementation of the first aspect, determining the first camera motion offset of a pixel in the video frame to be compensated according to the first depth map, the first camera pose matrix, the second depth map, the second camera pose matrix, and the camera intrinsics includes: calculating the first predicted three-dimensional coordinates of the pixel in the video frame to be compensated in the world coordinate system according to the first depth map, the first camera pose matrix, and the camera intrinsics; calculating the second predicted three-dimensional coordinates of the pixel in the adjacent frame in the world coordinate system according to the second depth map, the second camera pose matrix, and the camera intrinsics; calculating the relative pose transformation matrix of the adjacent frame relative to the video frame to be compensated according to the first camera pose matrix and the second camera pose matrix; converting the second predicted three-dimensional coordinates according to the relative pose transformation matrix to obtain the ideal three-dimensional coordinates of the pixel in the video frame to be compensated in the world coordinate system; and calculating the first camera motion offset of the pixel in the video frame to be compensated according to the ideal three-dimensional coordinates and the first predicted three-dimensional coordinates.

[0017] In a possible implementation of the first aspect, calculating the first predicted three-dimensional coordinates of the pixel in the video frame to be compensated in the world coordinate system according to the first depth map, the first camera pose matrix, and the camera intrinsics includes: converting the first depth map according to the camera intrinsics to obtain the camera coordinates of the pixel in the video frame to be compensated in the camera coordinate system; and converting the camera coordinates according to the first camera pose matrix to obtain the first predicted three-dimensional coordinates of the pixel in the video frame to be compensated in the world coordinate system.

[0018] In a second aspect, there is provided an apparatus for a video depth estimation method, the apparatus including a memory and a processor, the memory being configured to store instructions, and when the instructions are executed by the processor, causing the apparatus for the video depth estimation method to execute the video depth estimation method in the first aspect or any possible implementation of the first aspect.

[0019] In a third aspect, there is provided a computer-readable storage medium storing a computer program, the computer program including program instructions, and when the program instructions are executed, implementing the video depth estimation method in the first aspect or any possible implementation of the first aspect.

[0020] In a fourth aspect, a computer program product is provided, which includes computer program code that, when running on a computer, causes the computer to execute the video depth estimation method in the first aspect or any possible implementation of the first aspect.

[0021] Based on the implementations provided in the above aspects of this application, further combinations can be made to provide more implementations. Description of the Drawings

[0022] Figure 1 is an exemplary flowchart of the video depth estimation method in the first aspect provided by an embodiment of this application;

[0023] Figure 2 is a schematic structural diagram of the video depth estimation method in the first aspect provided by an embodiment of this application;

[0024] Figure 3 is an exemplary flowchart of the video depth estimation method in the second aspect provided by an embodiment of this application;

[0025] Figure 4 is a schematic structural diagram of the video depth estimation method in the second aspect provided by an embodiment of this application;

[0026] Figure 5 is a schematic structural diagram of the video depth estimation device in the third aspect provided by an embodiment of this application. Detailed Embodiments

[0027] Hereinafter, the technical solutions in this application will be described with reference to the accompanying drawings.

[0028] In the description of the embodiments of this application, unless otherwise specified, " / " means "or". For example, A / B may mean A or B; herein, "and / or" is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B may mean: A exists alone, A and B exist simultaneously, and B exists alone.

[0029] Hereinafter, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of this embodiment, unless otherwise specified, the meaning of "a plurality" is two or more.

[0030] First, the nouns involved in this application are explained:

[0031] (1) RGB image: Also known as a color image, it refers to an image in which each pixel is composed of R (red), G (green), and B (blue) components, where R, G, and B are described by different gray levels.

[0032] (2) Depth image: Depth Images, also known as Range Images, refer to an image in which the distance (depth) values of each point in the scene collected by the image collector are used as pixel values. It directly reflects the geometric shape of the visible surface of the scene. The depth image can calculate point cloud data through coordinate transformation, and the point cloud data with rules and necessary information can also be used to calculate the depth image data inversely.

[0033] (3) Camera internal parameters: Refer to the parameters required to project the camera onto the imaging plane, including focal length, principal point position (imaging origin), and pixel size, etc. There are two ways to obtain camera internal parameters. The first is through camera calibration, and the second is to use the parameters provided by the camera manufacturer. The use of camera internal parameters can be divided into two cases. One case is to map the three-dimensional coordinates to the two-dimensional image through the camera internal parameters; the other case is to map the two-dimensional pixel coordinates to the three-dimensional coordinates through the camera internal parameters.

[0034] (4) Camera pose estimation: Generally divided into two types: camera pose estimation based on feature point matching and camera pose estimation based on the Inertial Measurement Unit (IMU). The former calculates the camera pose information by assuming that there is a corresponding relationship between the feature points in the image in three-dimensional space and then through the matching of these feature points. The latter calculates the camera pose information through sensors such as accelerometers and gyroscopes, combined with the principle of spatial orientation transformation. Among them, the camera pose information includes the rotation matrix R and the translation vector T.

[0035] (5) World coordinate system: The three-dimensional coordinate system of the real world, used to describe the camera position, with the unit of m.

[0036] (6) Camera coordinate system: According to the lens imaging principle, the world coordinates are presented inside the camera. It is a three-dimensional coordinate system. The origin of the camera coordinate system is the optical center of the camera, and the z-axis coincides with the optical axis, that is, the z-axis points forward of the camera (perpendicular to the imaging plane). The positive directions of the x-axis and y-axis are parallel to the world coordinate system, with the unit of m. To convert the world coordinates into camera coordinates, it is necessary to scale down proportionally, plus rotation and translation, which is called a rigid body transformation.

[0037] (7) Pixel coordinate system: The picture formed by discretely sampling the projected image coordinates is a two-dimensional coordinate system. The origin of the pixel coordinate system is the upper left corner of the image, with the unit of pixel, used to represent the pixel length and pixel width of the entire picture.

[0038] Currently, the methods for obtaining depth information in images include using depth sensors such as LIDAR sensors, ToF sensors, etc., or using depth estimation algorithms for solution. The problem with using sensors to obtain depth maps is that the devices are relatively expensive and the acquisition cost is relatively high, especially for obtaining high-precision depth data. Moreover, the depth data collected by sensors also has range limitations and is subject to environmental factors. Therefore, if the problem of depth data acquisition can be solved through depth estimation algorithms, it will greatly save the cost of data acquisition and is also conducive to the large-scale promotion and use of related applications.

[0039] Depth estimation algorithms can be divided into traditional methods and deep learning-based methods. Traditional methods rely on the extraction and matching of accurate image feature points, which will result in less than ideal depth data in some low-texture regions or scenes with occlusion and moving objects, and the estimated depth maps are usually sparse. Deep learning-based methods can well solve the problems of traditional methods and can estimate a dense depth result. Deep learning-based depth estimation can be divided into supervised depth estimation and unsupervised depth estimation. The problem with supervised depth estimation is that a large amount of real depth data is required as supervision during the training process, but collecting such real depth data requires a lot of manpower and material resources, which is not worth the effort. Therefore, unsupervised depth estimation is the current trend in the research and development of this direction. Among them, the deep learning method using monocular video sequences as training data is the most promising method among all deep learning-based depth estimation methods due to the convenience of data acquisition.

[0040] When training a depth estimation network, the camera motion offset and object motion offset between two adjacent video frames in a continuous video frame can be used as a loss function to constrain the depth estimation network. In this way, the weights of the depth estimation network are adjusted according to the camera motion offset and object motion offset. The depth estimation network with adjusted weights is used to predict the depth values of each pixel point in the continuous video frame. The depth value of each pixel point in the predicted depth map is related to the camera motion offset and object motion offset of all pixel points in each video frame of the continuous video frame. That is to say, the camera motion offset and object motion offset of all pixel points in each video frame of the continuous video frame act on each pixel point of the video frame together through the adjustment of the weights in the depth estimation network, and cannot specifically compensate for the pixel points. The depth estimated between continuous video frames is unstable.

[0041] Thus, when applying the depth maps of consecutive video frames estimated to fields such as video defocusing, flickering may occur in the defocused front and rear video frames. Or, there are errors in the object edge depths in the predicted front and rear frames, and sudden changes occur in the object edge depths in the front and rear frames. Manifested in the results, artifacts appear at the object edges in the front and rear video frames. It can be seen that the depth estimated between consecutive video frames is unstable, which may lead to unsatisfactory effects in depth map-based applications (such as video defocusing, 3D reconstruction, autonomous driving, etc.).

[0042] In view of this, the embodiments of the present application provide a video depth estimation method, including: First, obtain the first depth map and the first camera pose matrix of the video frame to be compensated in multiple consecutive video frames, and the second depth map and the second camera pose matrix of the adjacent frame of the video frame to be compensated; Second, determine the first camera motion offset of the pixel points in the video frame to be compensated according to the first depth map, the first camera pose matrix, the second depth map, the second camera pose matrix, and the camera internal parameters. The first camera motion offset is the coordinate offset of the pixel points in the video frame to be compensated relative to the same pixel points in the adjacent frame in the same coordinate system; and determine the object motion offset of the pixel points in the video frame to be compensated relative to the same pixel points in the adjacent frame; Finally, compensate each pixel of the first depth map according to the first camera motion offset and the object motion offset to obtain the target depth map of the compensated video frame to be compensated.

[0043] In the embodiments of the present application, according to the first camera motion offset of the pixel points in the video frame to be compensated relative to the same pixel points in the adjacent frame, and the object motion offset of the pixel points in the video frame to be compensated relative to the same pixel points in the adjacent frame, each pixel of the first depth map of the predicted video frame to be compensated is compensated. That is, the camera motion offset and the object motion offset corresponding to a single pixel point are applied to this pixel point, rather than applying the camera motion offset and the object motion offset corresponding to all pixel points to each pixel point. The pixel points of the first depth map of the video frame to be compensated are compensated specifically. Thus, the depth estimated between the compensated video frame to be compensated and the adjacent frame is relatively stable. When applying the depth maps of the compensated video frame to be compensated and the adjacent frame to fields such as video defocusing or autonomous driving, there will be no flickering phenomenon between the compensated video frame to be compensated and the adjacent frame, and there will be no artifacts at the object edges in the compensated video frame to be compensated and the adjacent frame.

[0044] Figure 1 is an exemplary flowchart of the video depth estimation method, Figure 2 is a schematic structural diagram of the video depth estimation method. The following combines the attached Figure 1 and Figure 2 to introduce in detail a video depth estimation method provided by the embodiments of the present application,

[0045] Step 110: Obtain the first depth map and the first camera pose matrix of the video frame to be compensated, and the second depth map and the second camera pose matrix of the adjacent frame of the video frame to be compensated, from among a plurality of consecutive video frames.

[0046] In this embodiment, the plurality of consecutive video frames may be RGB images (hereinafter referred to as color images). In this embodiment, the video frame to be compensated may be each frame among the plurality of consecutive video frames, or may be other frames among the plurality of consecutive video frames except for the first frame and the last frame.

[0047] Assume that the plurality of consecutive video frames includes N consecutive video frames. Each frame among the N consecutive video frames may be sequentially used as the video frame to be compensated according to the time order of the video frames. For example, first use the first frame as the video frame to be compensated, and then use the second frame as the video frame to be compensated, and so on. Alternatively, each frame from the second frame to the (N - 1)-th frame among the N consecutive video frames may be used as the video frame to be compensated.

[0048] In one example, the adjacent frame of the video frame to be compensated may be the previous frame of the video frame to be compensated. For example, when the first frame is the video frame to be compensated, there is no adjacent frame for the video frame to be compensated, and in this case, the first frame is not compensated; when the second frame is the video frame to be compensated, the adjacent frame of the video frame to be compensated includes the first frame; when the third frame is the video frame to be compensated, the adjacent frame of the video frame to be compensated includes the second frame, and so on.

[0049] In another example, the adjacent frame of the video frame to be compensated includes the previous frame and the next frame of the video frame to be compensated. For example, when the first frame is the video frame to be compensated, the adjacent frame of the video frame to be compensated includes the second frame; when the second frame is the video frame to be compensated, the adjacent frame of the video frame to be compensated includes the first frame and the third frame, and so on.

[0050] Exemplarily, obtaining the first depth map and the first camera pose matrix of the video frame to be compensated, and the second depth map and the second camera pose matrix of the adjacent frame of the video frame to be compensated, includes: inputting the video frame to be compensated and the adjacent frame among the plurality of consecutive video frames into a trained first depth estimation network to obtain the first depth map and the second depth map; inputting the video frame to be compensated and the adjacent frame among the plurality of consecutive video frames into a trained first camera pose estimation network to obtain the first camera pose matrix and the second camera pose matrix.

[0051] In this embodiment, the trained first depth estimation network is used to obtain the first depth map of the video frame to be compensated and the second depth map of the adjacent frame, and the trained first camera pose estimation network is used to obtain the first camera pose matrix of the video frame to be compensated and the second camera pose matrix of the adjacent frame. The first depth estimation network and the first camera pose estimation network can be two networks with independent structures, or the first depth estimation network and the first camera pose estimation network can share some network structures.

[0052] Step 120: Determine the first camera motion offset of the pixel points in the video frame to be compensated according to the first depth map, the first camera pose matrix, the second depth map, the second camera pose matrix, and the camera internal parameters.

[0053] Since the pose of the camera may change when shooting the video frame to be compensated and the adjacent frame, and the depth estimation network does not consider the change relationship of the camera pose between the video frame to be compensated and the adjacent frame when estimating the first depth map of the compensated video frame and the second depth map of the adjacent frame, there is a coordinate offset caused by camera motion between the same pixel points of the estimated first depth map and the second depth map.

[0054] In this embodiment, it is necessary to determine the coordinate offset of the pixel points in the video frame to be compensated relative to the same pixel points in the adjacent frame in the same coordinate system, that is, the first camera motion offset. Refer to Figure 2 , first, input a series of consecutive video frames (including the compensated video frame and the adjacent frame) into the first depth estimation network and the first camera pose estimation network respectively. The first depth estimation network outputs multiple depth maps of the series of consecutive video frames, and the multiple depth maps include the first depth map of the video frame to be compensated and the second depth map of the video frame. The first camera pose estimation network outputs multiple camera pose matrices of the series of consecutive video frames, and the multiple camera pose matrices include the first camera pose matrix of the compensated video frame and the second camera pose matrix of the adjacent frame. Secondly, the first depth map, the first camera pose matrix, the second depth map, the second camera pose matrix, and the camera internal parameters can be input into a pre-trained residual network, and the residual network determines the first camera motion offset of the pixel points in the video frame to be compensated.

[0055] In some examples, the first camera motion offset of the pixel points in the video frame to be compensated can also be calculated in the following way. In this embodiment, the coordinate offset of the pixel points in the video frame to be compensated relative to the same pixel points in the adjacent frame in the same world coordinate system is determined.

[0056] First, according to the first depth map, the first camera pose matrix, and the camera internal parameters, calculate the first predicted three-dimensional coordinates of the pixel points in the compensated video frame in the world coordinate system. According to the second depth map, the second camera pose matrix, and the camera internal parameters, calculate the second predicted three-dimensional coordinates of the pixel points in the adjacent frame in the world coordinate system. Then, according to the first camera pose matrix and the second camera pose matrix, calculate the relative pose transformation matrix of the compensated video frame relative to the adjacent frame; convert the second predicted three-dimensional coordinates according to the relative pose transformation matrix to obtain the ideal three-dimensional coordinates of the pixel points in the compensated video frame in the world coordinate system. Finally, according to the ideal three-dimensional coordinates and the first predicted three-dimensional coordinates, calculate the first camera motion offset of the pixel points in the compensated video frame.

[0057] Specifically, the camera internal parameters include the focal length f, the principal point position (imaging origin) cx and cy, and the pixel sizes dx and dy. In this embodiment, the camera internal parameters can be obtained by reading the internal parameters of the camera that captures multiple consecutive video frames. For video frames for which the camera internal parameters cannot be obtained, the same set of preset camera internal parameters can be used.

[0058] First, according to the first depth map, the first camera pose matrix, and the camera internal parameters, calculate the first predicted three-dimensional coordinates of the pixel points in the compensated video frame in the world coordinate system, and according to the second depth map, the second camera pose matrix, and the camera internal parameters, calculate the second predicted three-dimensional coordinates of the pixel points in the adjacent frame in the world coordinate system.

[0059] In this embodiment, to convert the first depth map in the pixel coordinate system to the world coordinate system, it is necessary to first convert the first depth map in the pixel coordinate system to the camera coordinate system, and then convert the camera coordinate system to the world coordinate system. Specifically, it includes: converting the first depth map according to the camera internal parameters to obtain the camera coordinates of the pixel points in the compensated video frame in the camera coordinate system; converting the camera coordinates according to the first camera pose matrix to obtain the first predicted three-dimensional coordinates of the pixel points in the compensated video frame in the world coordinate system.

[0060] Assume that the pixel coordinates of the pixel points in the compensated video frame are [u v 1] T , then the camera coordinates [x y z] of the pixel points in the compensated video frame in the camera coordinate system T can be calculated by the following formula (1):

[0061]

[0062] Assume that the first camera pose matrix includes a rotation matrix R and a translation variable t. The first camera pose matrix can be expressed as formula (2) by K1, and the first camera pose matrix K1 is a 3-row and 4-column matrix.

[0063] (2)

[0064] Then, the first predicted three-dimensional coordinates [X1 Y1 Z1 1] of the pixel points in the to-be-compensated video frame in the world coordinate system T can be calculated through the following formula (3):

[0065]

[0066] Through the above formula, the first predicted three-dimensional coordinates (i.e., world coordinates) of the pixel points in the to-be-compensated video frame in the world coordinate system can be calculated. It should be noted that for the second predicted three-dimensional coordinates [X2 Y2 Z2 1] of each pixel point in the adjacent frames in the world coordinate system T can also be calculated in the above manner. To avoid repetition, it will not be elaborated here.

[0067] Secondly, according to the first camera pose matrix and the second camera pose matrix, calculate the relative pose transformation matrix of the to-be-compensated video frame relative to the adjacent frame.

[0068] Assume that the first camera pose matrix of the to-be-compensated video frame is A (a 3-row and 4-column matrix), the first camera pose matrix of the adjacent frame is B (a 3-row and 4-column matrix), and the relative pose transformation matrix of the to-be-compensated video frame relative to the adjacent frame is M. That is to say, BM = A. Then, the relative pose transformation matrix M = A^(-1)B -1 . It can be obtained that the relative pose transformation matrix M is a 4-row and 4-column matrix.

[0069] After that, according to the relative pose transformation matrix, transform the second predicted three-dimensional coordinates to obtain the ideal three-dimensional coordinates of the pixel points in the to-be-compensated video frame in the world coordinate system.

[0070] According to the relative pose transformation matrix M of the to-be-compensated video frame relative to the adjacent frame, transform the second predicted three-dimensional coordinates [X2 Y2 Z2 1] T Assume that the ideal three-dimensional coordinates of the pixel points in the to-be-compensated video frame in the world coordinate system obtained after transformation are [X0 Y0 Z0 1] T . Then, the ideal three-dimensional coordinates [X0 Y0 Z0 1] T can be calculated through the following formula (4):

[0071]

[0072] Finally, according to the ideal three-dimensional coordinates [X0 Y0 Z0 1] T and the first predicted three-dimensional coordinates [X1 Y1 Z1 1] T , calculate the first camera motion offset of the pixel points in the to-be-compensated video frame.

[0073] It should be noted that there is an error in the original text in item . The correct formula for calculating the relative pose transformation matrix M should be M = B^(-1)A instead of M = AB. The above translation has been corrected according to the correct content.The first camera motion offset can be the ideal three-dimensional coordinates [X0 Y0 Z0 1] of the pixel points in the video frame to be compensated T and the first predicted three-dimensional coordinates [X1 Y1 Z1 1] T The distance between them. For example, the Euclidean distance calculation formula can be used to calculate the ideal three-dimensional coordinates [X0 Y0 Z0 1] T and the first predicted three-dimensional coordinates [X1 Y1 Z1 1] T The distance between them. In practical applications, other distance calculation methods can also be used to calculate the distance between the ideal three-dimensional coordinates and the first predicted three-dimensional coordinates, which is not limited here.

[0074] In some other examples, the first camera motion offset of the pixel points in the video frame to be compensated can also be calculated in the following way. In this embodiment, the coordinate offset of the pixel points in the video frame to be compensated relative to the same pixel points in the adjacent frame in the same camera coordinate system is determined.

[0075] First, according to the first depth map and the camera internal parameters, calculate the first predicted camera coordinates of the pixel points in the video frame to be compensated in the camera coordinate system. According to the second depth map, the second camera pose matrix, and the camera internal parameters, calculate the second predicted three-dimensional coordinates of the pixel points in the adjacent frame in the world coordinate system. Then, according to the first camera pose matrix and the second camera pose matrix, calculate the relative pose transformation matrix of the video frame to be compensated relative to the adjacent frame; according to the relative pose transformation matrix, transform the second predicted three-dimensional coordinates to obtain the ideal three-dimensional coordinates of the pixel points in the video frame to be compensated in the world coordinate system. Then, transform the ideal three-dimensional coordinates through the first camera pose matrix to obtain the ideal camera coordinates. Finally, according to the ideal camera coordinates and the first predicted camera coordinates, calculate the first camera motion offset of the pixel points in the video frame to be compensated.

[0076] Step 130: Determine the object motion offset of the pixel points in the video frame to be compensated relative to the same pixel points in the adjacent frame.

[0077] Specifically, when the camera shoots the video frame to be compensated and the adjacent frame, due to the object motion, the position of the object in the compensated video frame and the adjacent frame is also different. That is to say, the pixel coordinates of the same pixel points of the same object in the video frame to be compensated and the adjacent frame are different, and the distance between the pixel coordinates of the pixel points of the same object in the video frame to be compensated and the adjacent frame is the object motion offset.

[0078] In this embodiment, it is also necessary to determine the object motion offset of the pixel points in the video frame to be compensated relative to the same pixel points in the adjacent frame.

[0079] Exemplarily, the RGB images of the video frame to be compensated and the adjacent frame can be input into a pre-trained convolutional network to determine the object motion offset of the pixel points in the video frame to be compensated. Here, the pre-trained convolutional network can be an optical flow network. For example, the object motion offset of the pixel points in the video frame to be compensated determined by the convolutional network can be expressed as (-4, 5), which means that the pixel coordinates of the pixel points in the video frame to be compensated relative to the same pixel points in the adjacent frame are offset 4 in the negative axis direction of the X direction (the unit can be millimeters) and 5 in the positive axis direction of the Y direction. The object motion offset includes not only the magnitude of the offset but also the direction of the offset.

[0080] Step 140: Input the first camera motion offset, the object motion offset, and the first depth map into a pre-trained convolutional network to obtain the first compensated high-dimensional feature matrix of the video frame to be compensated.

[0081] See Figure 2 , input the first camera motion offset, the object motion offset, and the first depth map of the video frame to be compensated into a pre-trained convolutional network. Through the trained convolutional network, fuse and compensate the first camera motion offset and the object motion offset corresponding to the pixel points in the video frame to be compensated into the first depth map of the video frame to be compensated to obtain the compensated first high-dimensional feature matrix.

[0082] Exemplarily, as Figure 2 shown, the pre-trained convolutional network can be a deformable convolutional network. The compensation offset of the deformable convolutional network includes the first camera motion offset and the object motion offset. Assume that the first camera motion offset of the pixel points in the video frame to be compensated is 3, and the object motion offset of the pixel points in the video frame to be compensated is (-4, 5). The compensation idea of the deformable convolutional network is: determine the pixel coordinates after the pixel points are offset according to the first camera motion offset of the pixel points in the video frame to be compensated, and compensate the depth value of the pixel coordinates where the pixel points are located to the pixel coordinates after the offset. Assume that the depth value of the pixel coordinates where the pixel points are located is 10, and the original depth value of the pixel coordinates after the offset is 5. Then, after compensating the depth value of the pixel coordinates where the pixel points are located to the pixel coordinates after the offset, the depth value of the pixel coordinates after the offset is 15. And determine the pixel coordinates after the offset according to the object motion offset of the pixel points in the video frame to be compensated. For example, if the pixel coordinates where the pixel points are located are (1, 1) and the object motion offset is (-4, 5), then the pixel coordinates after the pixel points are offset are (-3, 6), and the depth value of the pixel coordinates where the pixel points are located is compensated to the pixel coordinates after the offset. The above is only an example. In practical applications, when compensating, the deformable convolutional network can not only compensate the depth value of the pixel coordinates after the offset, but also compensate the depth value of the coordinates adjacent to the pixel coordinates where the pixel points are located.

[0083] In this embodiment, the variable convolution network can directly know which offset amounts need to be compensated. Compared with the ordinary convolution network that needs to first learn which of the input parameters are offset amounts, the compensation accuracy is higher and the network structure is easier to train.

[0084] Step 150: Obtain the target depth map of the compensated video frame to be compensated according to the first high-dimensional feature matrix.

[0085] Refer to Figure 2 , and the compensated first high-dimensional feature matrix can be input into the third depth estimation network to restore the target depth map of the compensated video frame to be compensated.

[0086] In this embodiment, according to the first camera motion offset amount of the pixel points in the video frame to be compensated relative to the same pixel points in the adjacent frame, and the object motion offset amount of the pixel points in the video frame to be compensated relative to the same pixel points in the adjacent frame, the first depth map of the predicted video frame to be compensated is compensated pixel by pixel, that is, the camera motion offset amount and the object motion offset amount corresponding to a single pixel point are applied to this pixel point, rather than applying the camera motion offset amounts and object motion offset amounts corresponding to all pixel points to each pixel point, and specifically compensating the pixel points of the first depth map of the video frame to be compensated. In this way, the estimated depth between the video frame to be compensated and the adjacent frame is relatively stable. When the estimated depth maps of the video frame to be compensated and the adjacent frame are applied to fields such as video defocusing or autonomous driving, there will be no flickering phenomenon between the video frame to be compensated and the adjacent frame, and there will be no artifacts at the object edges in the video frame to be compensated and the adjacent frame.

[0087] Second, this embodiment also provides a video depth estimation method. As Figure 3 shown is the schematic flow diagram of the video depth estimation method, Figure 4 shown is the schematic structural diagram of the video depth estimation method. The following will describe the video depth estimation method in conjunction with the attached Figure 3 and the attached Figure 4 diagrams.

[0088] Step 210: Obtain the first depth map and the first camera pose matrix of the video frame to be compensated, and the second depth map and the second camera pose matrix of the adjacent frame of the video frame to be compensated in multiple consecutive video frames.

[0089] Step 220: Determine the first camera motion offset amount of the pixel points in the video frame to be compensated according to the first depth map, the first camera pose matrix, the second depth map, the second camera pose matrix, and the camera internal parameters.

[0090] Step 230: Determine the object motion offset amount of the pixel points in the video frame to be compensated relative to the same pixel points in the adjacent frame

[0091] Step 240: Input the first camera motion offset, the object motion offset, and the first depth map into a pre-trained convolutional network to obtain the first high-dimensional feature matrix after compensation for the video frame to be compensated.

[0092] The above steps 210 to 240 are the same as steps 110 to 140 in the above embodiment. To avoid repetition, they will not be elaborated here.

[0093] Step 250: Obtain the third depth map and the third camera pose matrix of the video frame to be compensated, and the fourth depth map and the fourth camera pose matrix of the adjacent frame according to the first high-dimensional feature matrix.

[0094] The process of the video depth estimation method in this embodiment can be referred to Figure 4 , input the first high-dimensional feature matrix into the trained second depth estimation network to obtain the third depth map of the video frame to be compensated and the fourth depth map of the adjacent frame; input the first high-dimensional feature matrix into the trained second camera pose estimation network to obtain the third camera pose matrix of the video frame to be compensated and the fourth camera pose matrix of the adjacent frame.

[0095] The structure of the second depth estimation network in this embodiment is roughly the same as that of the first depth estimation network in the above first aspect embodiment. However, due to different inputs, the weight parameters of the trained first depth estimation network and the trained second depth estimation network are different. The input of the first depth estimation network is an RGB image, and the output is the depth map corresponding to the RGB image. The input of the second depth estimation network is a high-dimensional feature matrix, and the output is the depth map corresponding to the high-dimensional feature matrix.

[0096] The second camera pose estimation network is roughly the same as the above first camera pose estimation network in structure. However, due to different inputs, the weight parameters of the trained first camera pose estimation network and the trained second camera pose estimation network are different. The input of the first camera pose estimation network is an RGB image, and the output is the camera pose matrix corresponding to the RGB image. The input of the second camera pose estimation network is a high-dimensional feature matrix, and the output is the camera pose matrix corresponding to the high-dimensional feature matrix.

[0097] Step 260: Determine the second camera motion offset of the pixel points in the video frame to be compensated according to the third depth map, the third camera pose matrix, the fourth depth map, the fourth camera pose matrix, and the camera internal parameters.

[0098] Refer to Figure 4, in this embodiment, the third depth map, the third camera pose matrix, the fourth depth map, the fourth camera pose matrix, and the camera intrinsics are input into a pre-trained residual network to determine the second camera motion offset of the pixel points in the video frame to be compensated. The method for determining the second camera motion offset according to the third depth map, the third camera pose matrix, the fourth depth map, the fourth camera pose matrix, and the camera intrinsics in this embodiment is substantially the same as the method for determining the first camera motion offset of the pixel points in the video frame to be compensated according to the first depth map, the first camera pose matrix, the second depth map, the second camera pose matrix, and the camera intrinsics in the above embodiment. The difference lies in the input parameters. For the specific implementation of step 260, reference can be made to the specific description of the above embodiment and will not be elaborated here.

[0099] Step 270: Compensate each pixel of the third depth map according to the second camera motion offset and the object motion offset to obtain the compensated second high-dimensional feature matrix of the video frame to be compensated.

[0100] See Figure 4 , in this embodiment, the second camera motion offset, the object motion offset, and the third depth map can be input into a pre-trained convolutional network. Through the trained convolutional network, the second camera motion offset and the object motion offset corresponding to the pixel points in the video frame to be compensated are fused and compensated into the third depth map of the video frame to be compensated to obtain the compensated second high-dimensional feature matrix.

[0101] It should be noted that the implementation method of "inputting the second camera motion offset, the object motion offset, and the third depth map into a pre-trained convolutional network to obtain the compensated second high-dimensional feature matrix of the video frame to be compensated" in this embodiment is substantially the same as the implementation method of "inputting the first camera motion offset and the object motion offset, and the first depth map into a pre-trained convolutional network to obtain the compensated first high-dimensional feature matrix" in the above embodiment. The difference lies in the input parameters. For the specific implementation of step 270, reference can be made to the specific description of the above embodiment and will not be elaborated here.

[0102] Step 280: Perform the same steps on the second high-dimensional feature matrix as on the first high-dimensional feature matrix until a preset number of executions is reached.

[0103] Step 290: Obtain the compensated target depth map of the video frame to be compensated according to the second high-dimensional feature matrix output after the prediction number of executions.

[0104] See Figure 4, in this embodiment, a first-stage initial network and a second-stage iterative refinement network are combined. First, a compensated first high-dimensional feature matrix is obtained by using the first-stage network. Then, the first high-dimensional feature matrix is input into the second-stage iterative refinement network to obtain a second high-dimensional feature matrix, and the same steps as those for the first high-dimensional feature matrix are performed on the second high-dimensional feature matrix until a preset number of executions P is reached. That is to say, the second high-dimensional feature matrix is input into the network structure for multiple iterations, and the depth map and the camera pose matrix are refined through multiple iterations, improving the stability of the depth map estimated from consecutive multiple video frames. It should be noted that the preset number of executions P can be set as needed. For example, P can be 3, 4, etc.

[0105] It should be understood that the above examples are for helping those skilled in the art to understand the embodiments of the present application, rather than limiting the embodiments of the present application to the specific numerical values or specific scenarios illustrated. Those skilled in the art can clearly make various equivalent modifications or changes based on the above examples, and such modifications or changes also fall within the scope of the embodiments of the present application.

[0106] Figure 5 It is a schematic structural block diagram of a video depth estimation device 300 provided by an embodiment of the present application. The video depth estimation device 300 includes: a processor 310, a memory 320, a communication interface 330, and a bus 340.

[0107] Among them, the processor 310 can be connected to the memory 320. The memory 320 can be used to store the program code and data. Therefore, the memory 320 can be an internal storage unit of the processor 310, an external storage unit independent of the processor 310, or a component including an internal storage unit of the processor 310 and an external storage unit independent of the processor 310.

[0108] Optionally, the video depth estimation device 300 may further include a bus 340. Among them, the memory 320 and the communication interface 330 can be connected to the processor 310 through the bus 340. The bus 340 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 340 can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, Figure 5 only one line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0109] It should be understood that in the embodiments of the present application, the processor 310 may adopt a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. Alternatively, the processor 310 adopts one or more integrated circuits to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0110] The memory 320 may include a read-only memory and a random access memory, and provide instructions and data to the processor 310. A part of the processor 310 may also include a non-volatile random access memory. For example, the processor 310 may also store information about the device type.

[0111] When the video depth estimation device is running, the processor 310 executes the computer-executable instructions in the memory 320 to perform the operation steps of the above video depth estimation method by using the hardware resources in the video depth estimation device.

[0112] It should be understood that the video depth estimation device 300 according to the embodiments of the present application may correspond to the corresponding main body executing the methods shown in Figure 1 and Figure 3 of the present application, and the above and other operations and / or functions of each module in the video depth estimation device 300 are respectively for implementing the corresponding processes of the methods in Figure 1 and Figure 3 For the sake of brevity, they will not be described in detail here.

[0113] The present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and the computer program includes program instructions. When the program instructions are executed, the video depth estimation method provided in the embodiments of the present application is implemented.

[0114] The present application also provides a computer program product. The computer program product includes: computer program code. When the computer program code runs on a computer, the computer is caused to execute the video depth estimation method provided in the embodiments of the present application.

[0115] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage such as a server, data center, etc. that contains one or more sets of available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid state drive (SSD).

[0116] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Skilled professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0117] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be repeated here.

[0118] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0119] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units. They can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0120] In addition, in each embodiment of the present application, each functional unit can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0121] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a memory (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0122] As described above, the above is only the specific implementation manner of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the embodiments of the present application, and all should be covered by the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application should be subject to the protection scope of the claims.

Claims

1. A video depth estimation method, characterized in that, The method includes: Obtaining a first depth map and a first camera pose matrix of a video frame to be compensated, and a second depth map and a second camera pose matrix of an adjacent frame of the video frame to be compensated in a plurality of consecutive video frames; Determining a first camera motion offset of a pixel point in the video frame to be compensated according to the first depth map, the first camera pose matrix, the second depth map, the second camera pose matrix, and camera intrinsics, where the first camera motion offset is the coordinate offset of the pixel point in the video frame to be compensated relative to the same pixel point in the adjacent frame in the same coordinate system; Determining an object motion offset of a pixel point in the video frame to be compensated relative to the same pixel point in the adjacent frame; Compensating each pixel of the first depth map according to the first camera motion offset and the object motion offset to obtain a target depth map after compensation of the video frame to be compensated.

2. The video depth estimation method according to claim 1, wherein The compensating each pixel of the first depth map according to the first camera motion offset and the object motion offset to obtain a target depth map after compensation of the video frame to be compensated includes: Inputting the first camera motion offset, the object motion offset, and the first depth map into a pre-trained convolutional network to obtain a first high-dimensional feature matrix after compensation of the video frame to be compensated; Obtaining a target depth map after compensation of the video frame to be compensated according to the first high-dimensional feature matrix.

3. The video depth estimation method according to claim 2, wherein The obtaining a target depth map after compensation of the video frame to be compensated according to the first high-dimensional feature matrix includes: Obtaining a third depth map and a third camera pose matrix of the video frame to be compensated, and a fourth depth map and a fourth camera pose matrix of the adjacent frame according to the first high-dimensional feature matrix; Determining a second camera motion offset of a pixel point in the video frame to be compensated according to the third depth map, the third camera pose matrix, the fourth depth map, the fourth camera pose matrix, and the camera intrinsics; Compensating each pixel of the third depth map according to the second camera motion offset and the object motion offset to obtain a second high-dimensional feature matrix after compensation of the video frame to be compensated; Performing the same steps as the first high-dimensional feature matrix on the second high-dimensional feature matrix until a preset number of executions is reached; Obtaining a target depth map after compensation of the video frame to be compensated according to the second high-dimensional feature matrix output after the prediction number of executions.

4. The video depth estimation method according to any one of claims 1 to 3, characterized in that The obtaining a first depth map and a first camera pose matrix of a video frame to be compensated, and a second depth map and a second camera pose matrix of an adjacent frame of the video frame to be compensated in a plurality of consecutive video frames includes: Inputting the video frame to be compensated and the adjacent frame in the plurality of consecutive video frames into a trained first depth estimation network to obtain the first depth map and the second depth map; Inputting the video frame to be compensated and the adjacent frame in the plurality of consecutive video frames into a trained first camera pose estimation network to obtain the first camera pose matrix and the second camera pose matrix.

5. The video depth estimation method according to claim 3, wherein Obtaining the third depth map and the third camera pose matrix of the video frame to be compensated, and the fourth depth map and the fourth camera pose matrix of the adjacent frame according to the first high-dimensional feature matrix includes: Inputting the first high-dimensional feature matrix into a trained second depth estimation network to obtain the third depth map of the video frame to be compensated and the fourth depth map of the adjacent frame; Inputting the first high-dimensional feature matrix into a trained second camera pose estimation network to obtain the third camera pose matrix of the video frame to be compensated and the fourth camera pose matrix of the adjacent frame.

6. The video depth estimation method according to claim 2, wherein The pre-trained convolutional network is a deformable convolutional network, and the compensation offset of the deformable convolutional network includes the first camera motion offset and the object motion offset.

7. The video depth estimation method according to any one of claims 1 to 3, characterized in that The adjacent frames of the video frame to be compensated include the previous frame and the next frame of the video frame to be compensated.

8. The video depth estimation method according to any one of claims 1 to 3, characterized in that, Determining the object motion offset of a pixel point in the video frame to be compensated relative to the same pixel point in the adjacent frame includes: Inputting the video frame to be compensated and the adjacent frame into an optical flow network to obtain the object motion offset of a pixel point in the video frame to be compensated relative to the same pixel point in the adjacent frame.

9. The video depth estimation method according to any one of claims 1 to 3, characterized in that Determining the first camera motion offset of a pixel point in the video frame to be compensated according to the first depth map, the first camera pose matrix, the second depth map, the second camera pose matrix, and the camera internal parameters includes: Calculating the first predicted three-dimensional coordinates of a pixel point in the video frame to be compensated in the world coordinate system according to the first depth map, the first camera pose matrix, and the camera internal parameters; Calculating the second predicted three-dimensional coordinates of a pixel point in the adjacent frame in the world coordinate system according to the second depth map, the second camera pose matrix, and the camera internal parameters; Calculating the relative pose transformation matrix of the adjacent frame relative to the video frame to be compensated according to the first camera pose matrix and the second camera pose matrix; Converting the second predicted three-dimensional coordinates according to the relative pose transformation matrix to obtain the ideal three-dimensional coordinates of a pixel point in the video frame to be compensated in the world coordinate system; Calculating the first camera motion offset of a pixel point in the video frame to be compensated according to the ideal three-dimensional coordinates and the first predicted three-dimensional coordinates.

10. The video depth estimation method according to claim 9, wherein Calculating the first predicted three-dimensional coordinates of a pixel point in the video frame to be compensated in the world coordinate system according to the first depth map, the first camera pose matrix, and the camera internal parameters includes: Converting the first depth map according to the camera internal parameters to obtain the camera coordinates of a pixel point in the video frame to be compensated in the camera coordinate system; Converting the camera coordinates according to the first camera pose matrix to obtain the first predicted three-dimensional coordinates of a pixel point in the video frame to be compensated in the world coordinate system.

11. A video depth estimation method and apparatus, characterized in that, The device includes a memory and a processor, and the memory is used to store instructions. When the instructions are executed by the processor, the video depth estimation method device executes the video depth estimation method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and the computer program includes program instructions which, when executed, implement the video depth estimation method according to any one of claims 1 to 10.

13. A computer program product, characterized in that, The computer program product includes: computer program code which, when running on a computer, causes the computer to execute the video depth estimation method according to any one of claims 1 to 10.

Images