Video depth estimation method and device based on motion perception

Abstract

The invention provides a video depth estimation method and device based on motion perception, and the method comprises the steps: obtaining a dynamic difference part between adjacent video frames in a training sample, and generating a motion perception mask according to the dynamic difference part; obtaining a weighted loss value corresponding to the training sample according to the motion perception mask, the prediction depth and the real depth corresponding to each pixel in the training sample; training the initial depth estimation model according to the preprocessed training sample and the weighted loss value corresponding to the training sample to obtain an optimized depth estimation model; according to the method, the corresponding weighted loss value is obtained by emphatically training the dynamic pixel part in the sample, and the optimized depth estimation model is obtained by training the weighted loss value, so that the depth estimation performance of the corresponding model on the dynamic pixel part is improved, and the depth estimation accuracy of the model on the dynamic pixel part in the video frame to be detected is further improved.

Description

Technical Field

[0001] This invention relates to the field of computer image processing technology, and in particular to a method and apparatus for video depth estimation based on motion perception. Background Technology

[0002] With the widespread application of monocular depth estimation (MDE) in fields such as autonomous driving, augmented reality, and smart sports, achieving high-precision depth prediction in dynamic and complex real-world scenes has become an important research direction in this field. However, most existing MDE methods are based on relatively static backgrounds and a small number of moving objects. For sports scenes with a wide range of moving targets, frequent camera switching, and significant differences between foreground and background, their estimation accuracy and stability are difficult to guarantee.

[0003] In typical sports scenarios such as football, athletes move at high speeds while the background area remains largely static. This structural characteristic of dynamic foreground and static background poses a challenge to depth estimation, and the performance and accuracy of the model for depth estimation of video frames are somewhat lacking.

[0004] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0005] The technical problem this invention aims to solve is how to improve the accuracy of depth estimation for video frames.

[0006] This invention provides a motion-aware video depth estimation method, comprising: Obtain the dynamic difference between adjacent video frames in the training samples, and generate a motion-aware mask based on the dynamic difference. An initial depth estimation model is constructed, and preprocessed training samples are continuously input into the initial depth estimation model to continuously obtain the predicted depth corresponding to each pixel in the training samples; The weighted loss value corresponding to the training sample is obtained based on the motion-aware mask, predicted depth and true depth corresponding to each pixel in the training sample. The initial depth estimation model is trained based on the preprocessed training samples and the weighted loss values ​​corresponding to the training samples to obtain an optimized depth estimation model. The optimized depth estimation model is then used to estimate the depth of pixels in the video frame to be tested.

[0007] Preferably, the step of obtaining the dynamic difference portion between adjacent video frames in the training samples and generating a motion-aware mask based on the dynamic difference portion specifically includes: Obtain the current frame image and the previous frame image, and obtain a dynamic difference map based on the current frame image and the previous frame image; The initial depth distance is obtained based on the current frame image, and the motion-aware mask is obtained based on the initial depth distance and the dynamic difference map.

[0008] Preferably, the step of acquiring the current frame image and the previous frame image, and obtaining a dynamic difference map based on the current frame image and the previous frame image, specifically includes: The current frame image and the previous frame image are compared pixel by pixel, and the result of the difference comparison is compared with the filtering threshold. When the result of the difference comparison is greater than the filtering threshold, the pixel at the corresponding position is determined as a dynamic pixel on the dynamic difference map. When the result of the difference comparison is less than or equal to the filtering threshold, the pixel at the corresponding position is determined as a static pixel on the dynamic difference map.

[0009] Preferably, the step of obtaining the initial depth distance based on the current frame image, and obtaining the motion-aware mask based on the initial depth distance and the dynamic difference map, specifically includes: The current frame image is input into the basic depth estimation model to obtain the initial depth distance; The initial depth distance and the dynamic difference map are input into the mask model to obtain the motion-aware mask.

[0010] Preferably, the step of obtaining the weighted loss value corresponding to the training sample based on the motion-aware mask, predicted depth, and true depth corresponding to each pixel in the training sample specifically includes: Based on the motion-aware mask, predicted depth, and true depth corresponding to each pixel in the current frame image, obtain the predicted mask sequence and true mask sequence corresponding to all dynamic pixels in the current frame image, and obtain the predicted mask sequence and true mask sequence corresponding to all static pixels in the current frame image. The dynamic loss is obtained by using the predicted mask sequence and the real mask sequence corresponding to all dynamic pixels in the current frame image, and the static loss is obtained by using the predicted mask sequence and the real mask sequence corresponding to all static pixels in the current frame image. The weighted loss value is obtained by adding the dynamic loss and the static loss with corresponding weights.

[0011] Preferably, the step of obtaining the predicted mask sequence and the true mask sequence corresponding to all dynamic pixels in the current frame image based on the motion-aware mask, predicted depth, and true depth corresponding to each pixel in the current frame image specifically includes: Multiply the motion-aware mask of the corresponding dynamic pixel by the predicted depth to obtain the predicted mask sequence of the corresponding dynamic pixel, and multiply the motion-aware mask of the corresponding dynamic pixel by the true depth to obtain the true mask sequence of the corresponding dynamic pixel. Multiply the motion-aware mask of the corresponding static pixel by the predicted depth to obtain the predicted mask sequence of the corresponding static pixel, and multiply the motion-aware mask of the corresponding static pixel by the actual depth to obtain the actual mask sequence of the corresponding static pixel.

[0012] Preferably, the step of obtaining the dynamic loss based on the predicted mask sequence and the true mask sequence corresponding to all dynamic pixels in the current frame image, and obtaining the static loss based on the predicted mask sequence and the true mask sequence corresponding to all static pixels in the current frame image, specifically includes: The formula for calculating the dynamic loss is: ; in, This is a dynamic loss function, where n is the total number of pixels in the current frame image. Let i be the true mask sequence of the i-th dynamic pixel. Let be the predicted mask sequence for the i-th dynamic pixel; The formula for calculating the static loss is: ; in, This is a static loss, where n is the total number of pixels in the current frame image. Let i be the actual mask sequence of the i-th static pixel. Let be the predicted mask sequence for the i-th static pixel.

[0013] Preferred options also include: In each round of training, the root mean square error of the training samples is obtained, and the training samples are divided into easy samples and hard samples based on the root mean square error. In each round of training, the training samples are linearly weighted to obtain a training subset, which includes a first preset proportion of difficult samples and a second preset proportion of easy samples. The next round of training is then carried out using the training subset.

[0014] In a second aspect, a motion-aware video depth estimation apparatus is provided, comprising at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor to perform the motion-aware video depth estimation method.

[0015] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors to perform the method described in the first aspect.

[0016] Fourthly, a chip is provided, comprising: a processor and an interface for calling and running a computer program stored in memory, performing the method as described in the first aspect.

[0017] Fifthly, a computer program product containing instructions is provided that, when executed on a computer or processor, causes the computer or processor to perform the method as described in the first aspect.

[0018] In a sixth aspect, a motion-aware video depth estimation apparatus is provided, comprising the motion-aware video depth estimation apparatus of the second aspect, and using the motion-aware video depth estimation method of the first aspect.

[0019] Unlike existing technologies, the present invention has at least the following beneficial effects: By focusing on the dynamic pixel portion of the training samples to obtain the corresponding weighted loss value, and training the optimized depth estimation model with the weighted loss value, the performance of the corresponding model for depth estimation of dynamic pixel portion is improved, thereby improving the accuracy of the model for depth estimation of dynamic pixel portion in the test video frame. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0021] Figure 1 This is a flowchart of a motion-aware video depth estimation method provided in an embodiment of the present invention; Figure 2 This is a flowchart of a method for obtaining a motion-aware mask in a motion-aware video depth estimation method provided in an embodiment of the present invention; Figure 3 This is a flowchart of a method for obtaining weighted loss values ​​in a motion-aware video depth estimation method provided in an embodiment of the present invention; Figure 4 This is a flowchart of a method for obtaining a training subset in a motion-aware video depth estimation method provided in an embodiment of the present invention; Figure 5This is a pseudocode diagram illustrating the acquisition of a training subset in a motion-aware video depth estimation method provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of a motion-aware video depth estimation device provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0023] Unless the context otherwise requires, throughout the specification and claims, the term "comprising" is interpreted as openly inclusive, meaning "including, but not limited to." In the description of the specification, terms such as "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this disclosure. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics mentioned may be included in any suitable manner in any one or more embodiments or examples; that is, although they may be incorporated into embodiments or examples using the above terms for reasons such as order and position, it does not limit them to be incorporated in combination by a single embodiment or example.

[0024] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more. Furthermore, for example, the description may use the prefix "A" or "B" to describe the same type of nouns as two independent entities. In this case, the corresponding features defined with "A" and "B" are used only to distinguish between similar entities and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features.

[0025] In describing some embodiments, the terms "coupled," "coupled," and "connected," and their derivative expressions, may be used. For example, the term "connected" may be used in describing some embodiments to indicate that two or more components have direct physical or electrical contact with each other. Similarly, the term "coupled" may be used in describing some embodiments to indicate that two or more components have direct physical or electrical contact. However, the terms "connected" or "coupled" may also refer to two or more components that do not have direct contact with each other but still cooperate or interact with each other, such as "optical coupling," "wireless connection," etc. The embodiments disclosed herein are not necessarily limited to the scope of this invention.

[0026] In the description of this invention, the expression “A and / or B” (where A and B are used to formally represent specific features) will be used. The corresponding expression includes the following three combinations: only A, only B, and a combination of A and B.

[0027] As used in this invention, “about,” “approximately,” or “approximately” includes the stated value and the average value within an acceptable range of deviation from a particular value, wherein the acceptable range of deviation is determined by a person skilled in the art taking into account the measurement under discussion and the error associated with the measurement of the particular quantity (i.e., the limitations of the measuring device).

[0028] Example 1: This embodiment provides a motion-aware video depth estimation method, such as... Figure 1 As shown, the method flow is as follows.

[0029] In step 101, the dynamic difference portion between adjacent video frames is obtained, and a motion-aware mask is generated based on the dynamic difference portion.

[0030] In this embodiment, the video frame includes dynamic pixels and static pixels. Dynamic pixels are pixels in areas where the current video frame (referred to as the current frame in this embodiment) shows significant dynamic changes compared to the previous video frame (referred to as the previous frame in this embodiment). Static pixels are pixels in areas where the current video frame (referred to as the current frame in this embodiment) does not show significant dynamic changes compared to the previous video frame (referred to as the previous frame in this embodiment). The dynamic difference portion can refer to the pixel region that changes dynamically between two adjacent frames, i.e., the dynamic pixels in the current frame. The video frame is any frame in a moving video, typically a red-green-blue (RGB) image. Since the corresponding image cannot be directly applied to the subsequent depth estimation model, the corresponding dynamic difference portion needs to be converted into a mask that the depth estimation model can apply, i.e., the motion-aware mask.

[0031] In step 102, an initial depth estimation model is constructed, and the preprocessed training samples are continuously input into the initial depth estimation model to continuously obtain the predicted depth corresponding to each pixel in the training samples.

[0032] In this embodiment, the initial depth estimation model can be a multi-scale fusion network based on Depth Anything V2. The open-source parameters of this network model are used as the initial model before training begins in this embodiment. The training samples include video frames from the motion video and the true depth corresponding to each pixel in the video frame. The preprocessing involves processing the video frames in the training samples using data augmentation techniques such as cropping and horizontal and vertical flipping. The resolution of the cropped video frames can be 518x518, and the flipping probability of the video frames can be 0.5. Preprocessing converts the training samples into a format that the initial depth estimation model can process, obtaining the predicted depth corresponding to each pixel in the training samples, and simultaneously training the initial depth estimation model. During this process, memory layout optimization ensures the continuity of the sample vectors during storage.

[0033] In step 103, the weighted loss value corresponding to the training sample is obtained based on the motion-aware mask, predicted depth, and true depth corresponding to each pixel in the training sample.

[0034] In step 104, the initial depth estimation model is trained based on the preprocessed training samples and the weighted loss values ​​corresponding to the training samples to obtain an optimized depth estimation model. The optimized depth estimation model is then used to estimate the depth of pixels in the video frame to be tested.

[0035] In this embodiment, a weighted loss value is obtained by using the motion-aware mask corresponding to each pixel in the training samples, and the model is trained using the weighted loss value. This allows the model to converge towards the scene type of dynamic pixels in the video frame, making the model more adaptable to the depth estimation of dynamic pixels, thereby improving the accuracy of the model's depth estimation of dynamic pixels in the video frame and improving the model's processing performance for dynamic parts.

[0036] Furthermore, in this embodiment, since obtaining the dynamic difference between adjacent video frames requires comparing the images of two adjacent video frames to obtain the difference between the two images, and then obtaining the motion-aware mask, the design is as follows.

[0037] The process involves obtaining the dynamic differences between adjacent video frames in the training samples and generating a motion-aware mask based on these differences, such as... Figure 2 As shown, the method flow includes: In step 201, the current frame image and the previous frame image are obtained, and the dynamic difference map is obtained based on the current frame image and the previous frame image.

[0038] In this embodiment, the dynamic difference map is used to reflect the positions of dynamic and static pixels in the current frame.

[0039] In this embodiment, the current frame image and the previous frame image are compared pixel by pixel, and the result of the difference comparison is compared with the filtering threshold. When the result of the difference comparison is greater than the filtering threshold, the pixel at the corresponding position is determined to be a dynamic pixel on the dynamic difference map; when the result of the difference comparison is less than or equal to the filtering threshold, the pixel at the corresponding position is determined to be a static pixel on the dynamic difference map. The filtering threshold shall be set by those skilled in the art based on the actual situation.

[0040] The calculation formula for the dynamic difference graph is as follows.

[0041] ; ; in, This is the dynamic difference graph corresponding to the j-th video frame in the i-th video. The previous frame image, For the current frame image, In this embodiment, the filtering threshold can be 100, where x is the horizontal coordinate of the pixel and y is the vertical coordinate of the pixel.

[0042] In step 202, an initial depth distance is obtained based on the current frame image, and the motion-aware mask is obtained based on the initial depth distance and the dynamic difference map.

[0043] The current frame image is input into the basic depth estimation model to obtain the initial depth distance; the initial depth distance and the dynamic difference map are input into the mask model to obtain the motion-aware mask; The basic depth estimation model can be: The mask model can be... Model.

[0044] The formula for calculating the motion-aware mask is as follows.

[0045] ; ; Wherein, M is a motion-aware mask. This is the dynamic difference graph corresponding to the j-th video frame in the i-th video. Let the initial depth distance be the distance corresponding to the j-th video frame. for The model's input actions, for The input actions of the model.

[0046] In this embodiment, a motion video sequence is selected, the sequence is decomposed into video frames, and each video frame is input to the system after preprocessing. The model can then obtain the estimated initial depth distance for the corresponding video frame. After obtaining the dynamic difference map and initial depth distance for the current frame, these parameters are input into the model. Segment Anything V2 By performing mask correction within the model, the motion-aware mask corresponding to the current frame can be obtained. It should be noted that in this embodiment, the mask for dynamic pixels in the motion-aware mask can be 1, and the mask for static pixels can be 0.

[0047] Furthermore, in this embodiment, after obtaining the motion-aware mask, predicted depth, and true depth corresponding to the current frame, the loss value is calculated based on the dynamic pixels. The loss value obtained through this calculation can enable the depth estimation performance of the model to converge towards the region type that focuses on dynamic pixels during subsequent training. The final converged model can more accurately estimate the depth of the region type of dynamic pixels. Therefore, the corresponding method is designed as follows.

[0048] The weighted loss value corresponding to the training sample is obtained based on the motion-aware mask, predicted depth, and true depth corresponding to each pixel in the training sample. Figure 3 As shown, the corresponding method flow is as follows.

[0049] In step 301, based on the motion-aware mask, predicted depth, and true depth corresponding to each pixel in the current frame image, the predicted mask sequence and true mask sequence corresponding to all dynamic pixels in the current frame image are obtained, and the predicted mask sequence and true mask sequence corresponding to all static pixels in the current frame image are obtained.

[0050] Multiply the motion-aware mask of the corresponding dynamic pixel by the predicted depth to obtain the predicted mask sequence of the corresponding dynamic pixel, and multiply the motion-aware mask of the corresponding dynamic pixel by the actual depth to obtain the actual mask sequence of the corresponding dynamic pixel.

[0051] Multiply the motion-aware mask of the corresponding static pixel by the predicted depth to obtain the predicted mask sequence of the corresponding static pixel, and multiply the motion-aware mask of the corresponding static pixel by the actual depth to obtain the actual mask sequence of the corresponding static pixel.

[0052] In step 302, dynamic loss is obtained based on the predicted mask sequence and the real mask sequence corresponding to all dynamic pixels in the current frame image, and static loss is obtained based on the predicted mask sequence and the real mask sequence corresponding to all static pixels in the current frame image.

[0053] The formula for calculating the dynamic loss is: ; in, This is a dynamic loss function, where n is the total number of pixels in the current frame image. Let i be the true mask sequence of the i-th dynamic pixel. Let be the predicted mask sequence for the i-th dynamic pixel.

[0054] The formula for calculating the static loss is: ; in, This is a static loss, where n is the total number of pixels in the current frame image. Let i be the actual mask sequence of the i-th static pixel. Let be the predicted mask sequence for the i-th static pixel.

[0055] In step 303, the dynamic loss and the static loss are added together with corresponding weights to obtain the weighted loss value.

[0056] After obtaining the dynamic loss and static loss, the dynamic loss and the static loss can be added together with corresponding weights to obtain the weighted loss value. Since this embodiment requires the model to focus on depth estimation of dynamic pixels, the weight of the dynamic loss needs to be increased. In this embodiment, the calculation formula of the weighted loss value is as follows.

[0057] ; in, The weighted loss value. The weights for dynamic loss, The weights are those for the static loss; in this embodiment, It can be 5. It can be 1.

[0058] Example 2: Based on Example 1, this embodiment trains the model using training samples and corresponding weighted loss values ​​while continuously acquiring weighted loss values. The AdamW optimization algorithm can be used with a learning rate of 5e-5. The learning rate adjustment strategy adopts cosine annealing. The initial training rounds can be 120. Training continues until the model converges or reaches the predetermined number of iteration rounds to obtain the optimized depth estimation model.

[0059] Example 3: Furthermore, based on Examples 1 and 2, this example, in order to enable the trained model to adapt to a wider range of sample types while maintaining the initial convergence direction and improving the depth estimation performance for dynamic pixel parts, also involves the following design. For example... Figure 4 As shown, the method flow is as follows.

[0060] In step 401, in each round of training, the root mean square error of the training samples is obtained, and the training samples are divided into easy samples and hard samples according to the root mean square error.

[0061] In this embodiment, the formula for calculating the root mean square error is as follows.

[0062] ; in, The root mean square error, This represents the total number of pixels in the corresponding video frame of the training samples. Let i be the actual mask sequence of the i-th pixel in the video frame. Let be the predicted mask sequence for the i-th pixel in the video frame.

[0063] In this embodiment, the root mean square error (RMSE) reflects the error between the predicted depth and the true depth output after training samples are input into the model. Training samples with an RMSE greater than a preset threshold are considered hard samples, and training samples with an RMSE less than the preset threshold are considered easy samples. The preset threshold is set by those skilled in the art based on actual conditions. The model estimates the depth more accurately for easy samples and has a larger deviation for hard samples, indicating that the model is more suitable for easy sample types during training, and vice versa for hard samples. To ensure the model's adaptability to a wider range of samples during training, it is necessary to ensure that an appropriate proportion of easy and hard samples are used as training samples. While ensuring the model's convergence direction, the model is optimized towards the sample region with the current larger error rate, thereby improving the model's applicability.

[0064] In step 402, in each round of training, the training samples are linearly weighted to obtain a training subset. The training subset includes a first preset proportion of difficult samples and a second preset proportion of easy samples. The next round of training is carried out using the training subset.

[0065] In this embodiment, the first preset ratio and the second preset ratio are both set by those skilled in the art according to the actual situation. The model is trained with easy samples of the second preset ratio to avoid model divergence, while the model is trained with difficult samples of the first preset ratio to avoid premature model fitting and improve the model's generalization ability.

[0066] In this embodiment, a sample pool is constructed. ={ , },in For easy samples, For hard samples; in each training round, linear weighted sampling is performed based on the root mean square error of the samples to obtain a training subset. ;like Figure 5 The image shows the pseudocode for obtaining the training subset.

[0067] It should be noted that the training method provided in this embodiment can be further applied after the training method provided in embodiment 2 has been executed, or it can be applied directly without using the training method of embodiment 2.

[0068] Example 4: This embodiment, based on Embodiments 1-3, performs performance testing on the optimized depth estimation model.

[0069] The parameters in the optimized depth estimation model are fixed and tested on the test set. The output results are obtained by calculating the corresponding performance index using the formula.

[0070] The existing method yielded an overall error rate of 0.35% for the depth estimation model; the optimized depth estimation model based on the method provided in the above embodiments yielded an overall error rate of 0.258%. This demonstrates a significant decrease in the error rate.

[0071] Furthermore, this embodiment also conducts depth estimation tests on a stadium scene. Using the SoccerNet-Depth benchmark dataset for multi-view 3D scene understanding, based on real football match videos, and combining multi-camera synchronous perspectives with corresponding stadium geometric annotations, a dense pseudo-depth map is generated, which covers the dynamic changes of key targets such as players, the ball, and the goal.

[0072] The corresponding SoccerNet-Depth benchmark dataset provides high-quality image-depth pairs, including fast motion and occlusion in real-world scenes. It offers challenging and rich data support for 3D perception, depth estimation, pose recognition, and temporal modeling, and is one of the important datasets in current research on computer vision, action recognition, and augmented reality.

[0073] In the scenario of the SoccerNet-Depth benchmark dataset mentioned above, the method provided in the above embodiment and the basic Depth Anything V2 model were tested and compared. The test data is shown in Table 1 below. It can be seen that the method provided in this embodiment shows better performance.

[0074] Table 1. Comparison of test results between the method in this embodiment and the basic Depth Anything V2 model.

[0075] Example 5: like Figure 6 The diagram shown is a schematic representation of a motion-aware video depth estimation device according to an embodiment of the present invention. This motion-aware video depth estimation device includes one or more processors 41 and a memory 42.

[0076] Processor 41 and memory 42 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0077] The memory 42, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the motion-aware video depth estimation method in the above embodiment. The processor 41 executes the motion-aware video depth estimation method by running the non-volatile software program and instructions stored in the memory 42.

[0078] Memory 42 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 42 may optionally include memory remotely located relative to processor 41, which can be connected to processor 41 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0079] The program instructions / modules are stored in the memory 42. When executed by one or more processors 41, they perform the motion-aware video depth estimation method in the above embodiments, for example, performing the steps shown in the figures described above.

[0080] This invention also provides a computer storage medium storing computer program instructions; when executed by a processor, the computer program instructions implement the motion-aware video depth estimation method provided in this invention.

[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0082] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A video depth estimation method based on motion perception, characterized in that, include: Obtain the dynamic difference between adjacent video frames in the training samples, and generate a motion-aware mask based on the dynamic difference. An initial depth estimation model is constructed, and preprocessed training samples are continuously input into the initial depth estimation model to continuously obtain the predicted depth corresponding to each pixel in the training samples; The weighted loss value corresponding to the training sample is obtained based on the motion-aware mask, predicted depth and true depth corresponding to each pixel in the training sample. The initial depth estimation model is trained based on the preprocessed training samples and the weighted loss values ​​corresponding to the training samples to obtain an optimized depth estimation model. The optimized depth estimation model is then used to estimate the depth of pixels in the video frame to be tested.

2. The video depth estimation method based on motion perception according to claim 1, characterized in that, The step of obtaining the dynamic difference between adjacent video frames in the training samples and generating a motion-aware mask based on the dynamic difference specifically includes: Obtain the current frame image and the previous frame image, and obtain a dynamic difference map based on the current frame image and the previous frame image; The initial depth distance is obtained based on the current frame image, and the motion-aware mask is obtained based on the initial depth distance and the dynamic difference map.

3. The video depth estimation method based on motion perception according to claim 2, characterized in that, The step of acquiring the current frame image and the previous frame image, and obtaining a dynamic difference map based on the current frame image and the previous frame image, specifically includes: The current frame image and the previous frame image are compared pixel by pixel, and the result of the difference comparison is compared with the filtering threshold. When the result of the difference comparison is greater than the filtering threshold, the pixel at the corresponding position is determined as a dynamic pixel on the dynamic difference map. When the result of the difference comparison is less than or equal to the filtering threshold, the pixel at the corresponding position is determined as a static pixel on the dynamic difference map.

4. The video depth estimation method based on motion perception according to claim 2, characterized in that, The step of obtaining the initial depth distance based on the current frame image, and obtaining the motion-aware mask based on the initial depth distance and the dynamic difference map, specifically includes: The current frame image is input into the basic depth estimation model to obtain the initial depth distance; The initial depth distance and the dynamic difference map are input into the mask model to obtain the motion-aware mask.

5. The video depth estimation method based on motion perception according to claim 3, characterized in that, The step of obtaining the weighted loss value corresponding to the training sample based on the motion-aware mask, predicted depth, and true depth corresponding to each pixel in the training sample specifically includes: Based on the motion-aware mask, predicted depth, and true depth corresponding to each pixel in the current frame image, obtain the predicted mask sequence and true mask sequence corresponding to all dynamic pixels in the current frame image, and obtain the predicted mask sequence and true mask sequence corresponding to all static pixels in the current frame image. The dynamic loss is obtained by using the predicted mask sequence and the real mask sequence corresponding to all dynamic pixels in the current frame image, and the static loss is obtained by using the predicted mask sequence and the real mask sequence corresponding to all static pixels in the current frame image. The weighted loss value is obtained by adding the dynamic loss and the static loss with corresponding weights.

6. The video depth estimation method based on motion perception according to claim 5, characterized in that, The step of obtaining the predicted mask sequence and the true mask sequence for all dynamic pixels in the current frame image based on the motion-aware mask, predicted depth, and true depth corresponding to each pixel in the current frame image, and obtaining the predicted mask sequence and the true mask sequence for all static pixels in the current frame image, specifically includes: Multiply the motion-aware mask of the corresponding dynamic pixel by the predicted depth to obtain the predicted mask sequence of the corresponding dynamic pixel, and multiply the motion-aware mask of the corresponding dynamic pixel by the true depth to obtain the true mask sequence of the corresponding dynamic pixel. Multiply the motion-aware mask of the corresponding static pixel by the predicted depth to obtain the predicted mask sequence of the corresponding static pixel, and multiply the motion-aware mask of the corresponding static pixel by the actual depth to obtain the actual mask sequence of the corresponding static pixel.

7. The video depth estimation method based on motion perception according to claim 5, characterized in that, The process of obtaining dynamic loss based on the predicted mask sequence and the true mask sequence corresponding to all dynamic pixels in the current frame image, and obtaining static loss based on the predicted mask sequence and the true mask sequence corresponding to all static pixels in the current frame image, specifically includes: The formula for calculating the dynamic loss is: ； in, This is a dynamic loss function, where n is the total number of pixels in the current frame image. Let i be the true mask sequence of the i-th dynamic pixel. Let be the predicted mask sequence for the i-th dynamic pixel; The formula for calculating the static loss is: ； in, This is a static loss, where n is the total number of pixels in the current frame image. Let i be the actual mask sequence of the i-th static pixel. Let be the predicted mask sequence for the i-th static pixel.

8. The video depth estimation method based on motion perception according to claim 1, characterized in that, Also includes: In each round of training, the root mean square error of the training samples is obtained, and the training samples are divided into easy samples and hard samples based on the root mean square error. In each round of training, the training samples are linearly weighted to obtain a training subset, which includes a first preset proportion of difficult samples and a second preset proportion of easy samples. The next round of training is then carried out using the training subset.

9. A video depth estimation device based on motion perception, characterized in that, The method includes at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the motion-aware video depth estimation method according to any one of claims 1-8.

10. A non-volatile computer storage medium, characterized in that, The computer storage medium stores computer program instructions that, when executed by one or more processors, implement the motion-aware video depth estimation method as described in any one of claims 1-8.

Images