Method, system, and computer medium for scene-adaptive future depth prediction in monocular video

Abstract

Systems, methods, and computer-readable media for predicting depth of video frames are disclosed. A method can include the steps of: receiving a plurality of training data, each training data comprising a set of consecutive video frames and a depth representation of a subsequent video frame of the set of consecutive video frames; receiving a pre-trained neural network model f θ with a plurality of weights θ; when the pre-trained neural network model f θ does not converge: computing a plurality of second weights based on each set of consecutive video frames, and updating the plurality of weights θ based on the plurality of training data and the plurality of second weights; receiving a plurality of new consecutive video frames with consecutive timestamps; predicting a depth representation of a video frame immediately following the new consecutive video frames based on the updated plurality of weights θ.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Non-Provisional Application No. 17 / 203,645, filed March 16, 2021, entitled “Methods, systems and computer medium for scene-adaptive future depth prediction in monocular videos,” which is incorporated herein by reference in its entirety. Technical Field

[0003] This invention relates to predicting depth in monocular videos. Specifically, this invention relates to depth estimation for monocular videos generated using meta-learning methods. Background Technology

[0004] With the booming development of social media and the advancement of camera technology, especially in the context of smartphone devices, video has become a core form of disseminating entertainment, education, and awareness.

[0005] Hundreds of hours, or even more, of new video content are uploaded to social media-based video-sharing and streaming platforms every minute. Most of this video is monocular video shot with smartphones or other mobile devices. Various tasks and applications typically require accurate depth estimation of this video. Furthermore, depth estimation is also important when analyzing surveillance video or when vehicles need to use real-time camera video sources to generate guidance for drivers or computing systems on autonomous vehicles.

[0006] One known technique for predicting depth values ​​for video frames is future frame prediction, where future new frames are estimated based on RGB colors based on previously observed video frames or sequences. This approach tends to produce blurry and distorted future frames. Recent methods focus on extracting effective temporal representations from video frames and then predicting future frames based on these temporal representations.

[0007] In addition, some other methods focus on generating semantic maps for future frames, where the semantic map can include data representing depth, optical flow, and semantic map data, and the accuracy of the depth predicted in the semantic map is often unsatisfactory.

[0008] Video depth estimation depends on the temporal and spatial features of frames. Existing methods for predicting the depth values ​​of future video frames using machine learning models are quite limited because they are highly dependent on the training data, meaning that the generated machine learning models cannot be used to predict the depth of future frames of entirely new videos with high accuracy. Summary of the Invention

[0009] Exemplary embodiments are described that implement a neural network model trained using a meta-learning method to predict depths of future video frames. Such a neural network model, once trained using a set of training data from one or more videos, can quickly adapt to a new environment (e.g., a new scene from a new video) where there are very few samples from the new environment, meaning that the trained neural network model can be used to predict depths of future frames in the new video based on at least two or three existing video frames from the new video, improving computational efficiency in estimating depth values for the new video while saving computational resources.

[0010] The systems, methods, and computer-readable media disclosed herein can provide unique technical advantages, such as quickly and efficiently generating estimated depth values for video frames produced by monocular videos, as the system is used to quickly adapt to new video scenes using a meta-learning method through self-supervised training. The system can be used to predict depth values for future video frames based on a few recent video frames from a current video, which can be used by a computing system on an autonomous vehicle to generate future states to assist in navigation. The system is able to process previously unseen video data using only a few batches of sample data, meaning that the system does not need large-scale training data sets to adapt to new video scenes to generate estimated depth values for frames in a new video.

[0011] In one aspect, the present disclosure provides a computer-implemented method for predicting depths of video frames, which can include receiving a plurality of training data and for each D i : wherein, each of D θ , i = 1...N, respectively, represents a video frame from a plurality of t consecutive video frames having consecutive timestamps; is a depth representation of a future video frame immediately following the video frame ; receiving a pre-trained neural network model f θ ; when the pre-trained neural network model f i does not converge: computing a plurality of second weights θ' i based on the plurality of consecutive video frames and the pre-trained neural network model f θ ; updating the plurality of weights θ based on the plurality of training data and the plurality of second weights θ' i ; receiving a plurality of m new consecutive video frames predicting a depth of a future video frame immediately following the video frame Subsequent video frames Depth representation.

[0012] In some embodiments, the plurality of second weights θ' are calculated i It can be based on the formula:

[0013]

[0014] Where α represents the learning rate, Indicates based on Calculated loss, This represents the gradient operator.

[0015] In some embodiments, updating the plurality of weights θ can be based on the formula:

[0016]

[0017] Where β represents the learning rate, Indicates based on Calculated loss, This represents the gradient operator.

[0018] In some embodiments, predict video frames The depth representation may include the following steps: based on the plurality of new consecutive video frames And update the multiple weights θ, and update the multiple second weights θ' i Based on the updated multiple second weights θ' i Generate the depth representation.

[0019] In some embodiments, the plurality of second weights θ' are updated i It can be based on the formula:

[0020]

[0021] Where α is the learning rate. Indicates based on (f) θ ;D new The calculated loss, This represents the gradient operator.

[0022] In some embodiments, the pre-trained neural network model f θ The training process can include the current frame reconstruction process and the future depth prediction process.

[0023] In some embodiments, the pre-trained neural network model f θ The training process may include: receiving multiple consecutive video frames with consecutive timestamps. From the multiple consecutive video frames Extract multiple spatial features; f with random values θ Multiple initial parameters are set as the multiple weights θ; from the multiple consecutive video frames Extract multiple spatial features; during the current frame reconstruction process: reconstruct the multiple consecutive video frames based on the multiple spatial features. For each video frame in the reconstructed video frame, update the value of at least one of the plurality of weights θ; during the future depth prediction process: extract the plurality of consecutive video frames based on the plurality of spatial features. Temporal characteristics; based on the multiple consecutive video frames The time features are used to generate a video frame that immediately follows the video frame. Subsequent video frames Depth prediction; based on the video frames The depth prediction updates the value of at least one of the plurality of weights θ.

[0024] In some embodiments, extracting the temporal features may include using a 3D convolutional neural network to extract the temporal features.

[0025] In some embodiments, for one or more surfaces in the video frame, the depth representation of any video frame may include a depth value representing an estimated distance from the respective surface to the viewpoint.

[0026] In some embodiments, the depth representation of any video frame may be a depth map of the video frame.

[0027] On the other hand, a system for predicting the depth of video frames is disclosed, the system comprising: a processing unit; and a memory coupled to the processing unit, the memory storing machine-executable instructions, which, when executed by the processing unit, cause the system to: receive multiple training data. And for each D i : in, Each represents a video frame from a set of t consecutive video frames with consecutive timestamps; It is immediately following the video frame Depth representation of subsequent future video frames; receiving a pre-trained neural network model f with multiple weights θ. θ When the pre-trained neural network model f θ When convergence fails: based on each D i The plurality of consecutive video frames in i = 1...N and the pre-trained neural network model f θ Calculate multiple second weights θ'i Based on the aforementioned training data and the plurality of second weights θ' i Update the multiple weights θ; receive multiple m new consecutive video frames with consecutive timestamps. Based on updated weights θ, predict the next video frame. Subsequent video frames Depth representation.

[0028] In some embodiments, the plurality of second weights θ' are calculated i It can be based on the formula:

[0029]

[0030] Where α represents the learning rate, Indicates based on Calculated loss, This represents the gradient operator.

[0031] In some embodiments, updating the plurality of weights θ can be based on the formula:

[0032]

[0033] Where β represents the learning rate, Indicates based on Calculated loss, This represents the gradient operator.

[0034] In some embodiments, predict video frames The depth representation may include: based on the plurality of new consecutive video frames And update the multiple weights θ, and update the multiple second weights θ' i Based on the updated multiple second weights θ' i Generate the depth representation.

[0035] In some embodiments, the plurality of second weights θ' are updated i It can be based on the formula:

[0036]

[0037] Where α is the learning rate. Indicates based on Calculated loss, This represents the gradient operator.

[0038] In some embodiments, the pre-trained neural network model f θ The training process can include the current frame reconstruction process and the future depth prediction process.

[0039] In some embodiments, in the pre-trained neural network model f θ During the training process, the machine-executable instructions, when executed by the processing unit, cause the system to: receive a plurality of consecutive video frames with consecutive timestamps. From the multiple consecutive video frames Extract multiple spatial features; f with random values θ Multiple initial parameters are set as the multiple weights θ; from the multiple consecutive video frames Extract multiple spatial features; during the current frame reconstruction process: reconstruct the multiple consecutive video frames based on the multiple spatial features. For each video frame in the reconstructed video frame, update the value of at least one of the plurality of weights θ; during the future depth prediction process: extract the plurality of consecutive video frames based on the plurality of spatial features. Temporal characteristics; based on the multiple consecutive video frames The time features are used to generate a video frame that immediately follows the video frame. Subsequent video frames Depth prediction; based on the video frames The depth prediction updates the value of at least one of the plurality of weights θ.

[0040] In some embodiments, during the extraction of the temporal features, when the machine-executable instructions are executed by the processing unit, the system: extracts the temporal features using a 3D convolutional neural network.

[0041] In some embodiments, for one or more surfaces in the video frame, the depth representation of any video frame may include a depth value representing an estimated distance from the respective surface to the viewpoint.

[0042] In some embodiments, the depth representation of any video frame may be a depth map of the video frame.

[0043] On the other hand, a non-transient computer-readable medium is disclosed, storing machine-readable instructions for configuring a processing unit to predict the depth of video frames, the machine-readable instructions being used to instruct the processing unit to: receive multiple training data. And for each D i : in, Each represents a video frame from a set of t consecutive video frames with consecutive timestamps; It is immediately following the video frame Depth representation of subsequent future video frames; receiving a pre-trained neural network model f with multiple weights θ. θ When the pre-trained neural network model f θ When convergence fails: based on each D i The plurality of consecutive video frames in i = 1...N and the pre-trained neural network model f θ Calculate multiple second weights θ' i Based on the aforementioned training data and the plurality of second weights θ' i Update the multiple weights θ; in the pre-trained neural network model f θ Before convergence: Receive multiple m new consecutive video frames with consecutive timestamps. Based on updated weights θ, predict the next video frame. Subsequent video frames Depth representation. Attached Figure Description

[0044] The accompanying drawings, which illustrate exemplary embodiments of this application, will now be shown by way of example, in which:

[0045] Figure 1 These are block diagrams of exemplary systems provided by some exemplary embodiments for training neural network models to predict the depth of future video frames using meta-learning methods.

[0046] Figure 2 These are block diagrams of exemplary systems for pre-training neural network models to predict the depth of future video frames, provided by some exemplary embodiments.

[0047] Figure 3 The following are detailed exemplary systems provided by some exemplary embodiments for pre-training neural network models to predict the depth of future video frames.

[0048] Figure 4 Exemplary hardware environments for implementing systems including trained neural network models for predicting the depth of future video frames are shown in some exemplary embodiments.

[0049] Figure 5 Exemplary methods for training neural network models to predict the depth of future video frames using meta-learning methods are shown in some exemplary embodiments.

[0050] Similar reference numerals can be used to denote similar components in different accompanying drawings. Detailed Implementation

[0051] This invention is made with reference to the accompanying drawings, which illustrate embodiments. However, many different embodiments can be used, and therefore the description should not be construed as limiting it to the embodiments set forth herein. Rather, these embodiments are provided to make the invention comprehensive and complete. Throughout, similar numbers refer to similar elements, and in alternative embodiments, prime number symbols are used to indicate similar elements, operations, or steps. The individual blocks of the systems and devices shown, or the individual functional elements shown, do not necessarily require these functions to be physically independent, as communication between these elements can be achieved through message sending and receiving, function calls, shared memory, etc., without any such physical independence. Therefore, functions do not need to be implemented in physically or logically independent platforms, but are shown separately herein for ease of explanation. Different devices can have different designs, so while some devices implement some functions in fixed-function hardware, others may implement these functions in a programmable processor with code obtained from a machine-readable medium.

[0052] As used herein, "module" or "operation" can refer to a combination of hardware processing circuitry and machine-readable instructions (software and / or firmware) executable on that hardware processing circuitry. Hardware processing circuitry can include any or some combinations of microprocessors, the core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, a digital signal processor, a general-purpose processing unit, an accelerator unit, or other hardware processing circuitry. In some examples, a module can refer to hardware processing circuitry with a dedicated configuration.

[0053] As used herein, "model" can refer to a combination of hardware processing circuitry and machine-readable instructions (software and / or firmware) executable on that hardware processing circuitry, which applies processing functions to input data to generate corresponding outputs. "Machine learning model" can refer to a model that learns or trains processing functions using machine learning (ML) techniques. Machine learning models can include, but are not limited to, models based on one or more of the following architectures: convolutional neural network (CNN), recurrent neural network (RNN), long short-term memory (LSTM), and / or transformer. Other possible types of machine learning classification modules include models based on decision tree algorithms, support vector machine algorithms, and logistic regression algorithms.

[0054] As previously mentioned, current future frame prediction methods typically focus on predicting future frames based on RGB values ​​or semantic graphs. These methods generally exhibit poor adaptability, meaning that machine learning models need to be trained on new, large-scale datasets before they can be used to predict depth values ​​in new videos. This invention describes a novel method for predicting depth values ​​of future video frames without first obtaining the RGB values ​​of the future video frames, and using only a few samples (e.g., video frames) of the new video for prediction. The systems and methods disclosed herein provide a technical solution that requires fewer computational resources and less time than known methods to estimate the depth values ​​of frames in a given video.

[0055] In some exemplary embodiments, meta-learning methods are used to help train machine learning (e.g., neural network) models to quickly adapt to new environments with only a few samples. The basic concept of meta-learning is to rapidly and efficiently teach pre-trained machine learning models to generalize based on a new set of training data (e.g., video frames from a new video) when pre-trained models typically struggle to adapt to new video scenes without extensive training.

[0056] In some embodiments, during meta-learning, a pre-trained machine learning model can be trained on training data prepared for the environment (e.g., frames from a new video of the display environment) to adapt to a specific type of environment or scene, and the pre-trained machine learning model can be deployed to generate depth values ​​for future video frames in the same or similar environment in a computationally efficient manner.

[0057] Figure 1 This is a block diagram of an exemplary meta-learning system 100 provided by some exemplary embodiments for training a pre-trained neural network model 117 to predict depth values ​​129 of future video frames using a meta-learning method. System 100 includes a meta-training component 110 and a meta-testing component 120. The pre-trained machine learning model 117 may be, for example, a neural network model 117.

[0058] In some embodiments, the pre-trained neural network model 117 may be configured to receive multiple video frames F1, F2, ..., F3 with consecutive timestamps. j neural network model f θ 117, and based on multiple video frames F1, F2, ..., F j Spatial and temporal features are used to generate the next video frame F. j+1 A set of depth values. For example, a neural network model f θ117 can be pre-trained using convolutional neural networks (CNNs), recurrent neural networks (RNNs), and / or long short-term memory (LSTMs). After pre-training, the neural network model f θ 117 can have multiple video frames F1, F2, ..., F j Multiple parameters or weights learned from the spatial and / or temporal features are collectively represented by θ.

[0059] The following is for reference. Figure 2 and Figure 3 A detailed description of an exemplary process for pre-training a neural network model. Figure 2 This is a block diagram of an exemplary system 200 for pre-training a neural network model to predict the depth of future video frames, provided by some exemplary embodiments. Specifically, the exemplary system 200 is used to generate and refine a neural network model f for predicting the depth of future video frames. θ The weights θ. Multiple weights θ in the initial model f θ Before pre-training, it can be set or initialized with random values ​​and updated by the system 200 throughout the pre-training process.

[0060] System 200 may include a spatial feature extraction branch or process 203, a current frame reconstruction branch or process 205, and a future depth prediction branch or process 207. System 200 can receive multiple consecutive video frames. Multiple consecutive video frames can be represented as... Here, j represents the total number of consecutive video frames, which can be any positive natural number starting with 2. In this particular example, system 200 receives three consecutive video frames. They have consecutive timestamps t–2, t–1, and t, respectively. Third video frame. It is the latest and is recorded as the current frame.

[0061] System 200 includes multiple spatial feature extraction encoders 220a, 220b, 220c, from which multiple consecutive video frames are extracted. Each of the components is assigned a corresponding encoder. During the spatial feature extraction process 203, the spatial feature extraction encoder 220a is used to receive video frames. As input, multiple spatial features 230a and their associated weights θ are generated. F1 The spatial feature extraction encoder 220b is used to receive video frames. As input, multiple spatial features 230b and their associated weights θ are generated. F2The spatial feature extraction encoder 220c is used to receive video frames. As input, multiple spatial features 230c and their associated weights θ are generated. F3 Weight θ F1 θ F2 θ F3 It can be part of multiple weights θ and is updated throughout the current frame reconstruction process 205.

[0062] During the current frame reconstruction branch or process 205, the corresponding decoders 240a, 240b, and 240c can be used to reconstruct multiple consecutive video frames based on multiple spatial features 230a, 230b, and 230c from the spatial feature extraction process 203. Each of them. In some embodiments, decoder 240a may share weights θ with encoder 220a. F1 To reconstruct video frames The output is the reconstructed video frame 250a; the decoder 240b can share weights θ with the encoder 220b. F2 To reconstruct video frames The output is the reconstructed video frame 250b; and the decoder 240c can share weights θ with the encoder 220c. F3 To reconstruct video frames The output is the reconstructed video frame 250c. During this process, multiple weights θ can be updated based on the reconstructed video frames 250a, 250b, and 250c. F1 θ F2 θ F3 The value of at least one weight in it.

[0063] During the future depth prediction branch or process 207, a 3D convolutional neural network (CNN) 260 is used to extract multiple consecutive video frames based on multiple spatial features 230a, 230b, and 230c. Temporal features 270, decoder 280 for use based on multiple consecutive video frames The temporal features of multiple consecutive video frames 270 in the data are used to generate the next video frame. Subsequent video frames The depth prediction is 290. During this process, multiple weights θ can be updated based on the depth prediction 290. F1 θ F2 θ F3 At least one of the values. In some embodiments, the future depth prediction process 207 only starts from the latest video frame. Spatial features are acquired for cascading. System 200 can be directly used to cascade multiple consecutive video frames. The estimated depth value for subsequent video frames.

[0064] Figure 3 Some exemplary embodiments are shown to provide a pre-trained neural network model f θ 117. A detailed exemplary system 300 for predicting the depth of future video frames. (Compared to...) Figure 2 Similar to system 200, system 300 may include a spatial feature extraction branch or process 303, a current frame reconstruction branch or process 305, and a future depth prediction branch or process 307. System 300 can receive multiple consecutive video frames. Multiple consecutive video frames can be represented as... Here, j represents the total number of consecutive video frames, which can be any positive natural number starting with 2. In this particular example, system 300 receives three consecutive video frames. They have consecutive timestamps t–2, t–1, and t, respectively. Third video frame. It is the latest and is recorded as the current frame.

[0065] During the spatial feature extraction process 303, an encoder comprising multiple 2D convolutional layers 340a, 340b, 340c, 340d, and 340e can be used to extract features from the input. Spatial features 230a, 230b, and 230c are extracted from the layers. Each 2D convolutional layer 340a, 340b, 340c, 340d, and 340e may include at least one 2D convolutional neural network (CNN). For example, "3×3conv, 64" represents a 2D CNN with 3×3 kernels and 64 output channels for outputting spatial features. Another example is "3×3conv, 128," which represents a 2D CNN with 3×3 kernels and 128 output channels for outputting spatial features. Yet another example is "3×3conv, 256," which represents a 2D CNN with 3×3 kernels and 256 output channels for outputting spatial features. In some of the 2D convolutional layers 340a, 340b, 340c, and 340d that include two or more 2D CNNs, pooling layers (e.g., max-pooling filters) may be applied to downsample the spatial features generated by the 2D CNN.

[0066] Each of the 2D convolutional layers 340a, 340b, 340c, 340d, and 340e can generate a set of spatial features 330a, 330b, 330c, 330d, and 330e, which are passed to the decoder in the current frame reconstruction process 305 and the decoder in the future depth prediction process 307.

[0067] In the current frame reconstruction process 305, the decoder may include multiple deconvolutional layers 350a, 350b, 350c, and 350d to generate reconstructed video frames 250a, 250b, and 250c based on spatial features 330a, 330b, 330c, 330d, and 330e. Each of the deconvolutional layers 350a, 350b, 350c, and 350d may include a deconvolutional neural network (denoted as "deconv") and multiple 2D CNNs (e.g., "3×3conv, 512"). The numbers "1024", "512", "256", and "128" following each cascade operation (denoted as "C") represent the number of spatial features after the corresponding cascade operation.

[0068] In the future depth prediction process 307, a set of spatial features 330e from the spatial feature extraction process 303 is sent to a 3D CNN 360, which then processes the spatial features 330e to generate temporal features 270, which are then passed to the decoder. The decoder may include multiple deconvolutional layers 370a, 370b, 370c, and 370d to generate a sequence of video frames based on the spatial features 330a, 330b, 330c, 330d, and 330e. Subsequent video frames Depth prediction 290. Each of the deconvolutional layers 370a, 370b, 370c, and 370d may include a deconvolutional neural network (shown as "deconv") and multiple 2D CNNs (e.g., "3×3conv, 256"). The numbers "1024", "512", "256", and "128" after each cascade operation represent the number of spatial features after the corresponding cascade operation. In some embodiments, the future depth prediction process 307 starts only from the latest video frame. Spatial features are obtained from the data for cascading.

[0069] During processes 305 and 307, the neural network model f θ Some values ​​of the multiple weights θ in 117 can be updated based on the reconstructed video frames 250a, 250b, 250c and / or depth prediction 290.

[0070] As mentioned above, pre-trained neural network models 117 often struggle to adapt to new video scenes without extensive training. Returning to... Figure 1The meta-learning system 100 is designed to retrain a pre-trained neural network model 117 to predict the depth values ​​of frames in a new video through only a few training iterations based on a few frames 122 in the new video, as further described below. In some embodiments, the meta-learning system 100 may implement a model-agnostic meta-learning (MAML) method to update the weights θ of the pre-trained neural network model 117, which may include a meta-training component 110 and a meta-testing component 120.

[0071] Within the meta-training component 110, N input data batches 112a...112n can be obtained. Each input data batch 112a or 112n can be generated by... It means that, among them, and This is randomly sampled data. `i` represents different batches, ranging from 1 to N, where N is the total number of batches. More specifically, and It is paired data: Includes multiple consecutive video frames that are continuous in terms of RGB values ​​(e.g., ), Indicates following closely Future video frames The depth value (e.g., ground truth). The depth value can be represented by a matrix corresponding to the RGB values. For example, It can be a matrix, where each element of the matrix corresponds to a video frame. The depth value (e.g., 2 meters) of the corresponding pixel (or group of pixels). The depth value can represent the distance between the surface to which the pixel (or group of pixels) belongs and the viewpoint of the camera used to capture the video. For example, for a video frame containing a scene involving a table, the depth value could represent the distance from the pixel (or group of pixels) displaying the table surface to the camera viewpoint (e.g., 0.8 meters).

[0072] In some embodiments, the depth value of the video frame is included. This can be represented as a depth map, which can be a graphical representation based on depth values. In a depth map, depth values ​​can be depicted by one or more colors based on a predetermined set of rules. This set of rules may include, for example: depth values ​​within a first range (e.g., 0 to 2 meters) are represented by a first color (e.g., red), depth values ​​within a second range (e.g., 2.1 to 5 meters) are represented by a second color (e.g., yellow), depth values ​​within a third range (e.g., beyond 5 meters) are represented by a third color (e.g., blue), and so on.

[0073] Through the training process 115 in the meta-training component 110, N data batches 112a...112n are used to train the pre-trained neural network model f. θ 117. The model may include a current frame reconstruction phase (called inner loop update) and a future depth prediction phase (called outer loop update).

[0074] In some embodiments, the meta-training component 110 is used to train a pre-trained neural network model f θ The existing weights θ of 117 are updated to the updated weights θ'. i This makes the pre-trained neural network model f θ 117 Adapt to the new task. Updated weights θ' i It can be computed using one or more gradient descent updates on the new task.

[0075] During the current frame reconstruction phase of training process 115, the pre-trained neural network model f with L1 loss... θ 117 can be used Perform current frame reconstruction, and a gradient update can be used to update the weights of the spatial feature encoder (e.g., encoders 220a, 220b, 220c) and the spatial feature decoder (e.g., decoders 230a, 230b, 230c) from θ to θ' based on the following formula. i :

[0076]

[0077] Where α represents the learning rate or step size, Indicates based on Calculated loss, This represents the gradient operator, where N is the total number of training data batches.

[0078] At this time, θ' i It can be referred to as a set or multiple second weights, which are obtained through data. Updated pre-trained neural network model f θ The model weights are 117. This can be used to minimize the current frame of the reconstruction with ground truth (e.g., from...). The loss L1 of the frames.

[0079] Using the formula (1) above, we can obtain the result including θ' i After adapting the model weights θ', the pre-trained neural network model f with weights θ' is... θ 117 can utilize paired data for outer loop updates during the future deep forecasting phase. The outer loop update aims to ensure that the features from the spatial feature encoder are suitable for predicting the depth values ​​of future video frames. Each iteration of the outer loop update can update multiple weights θ based on the following formula:

[0080]

[0081] Where β represents the learning rate or step size, Indicates based on The calculated loss is where N is the total number of training data batches. This represents the gradient operator.

[0082] In some embodiments, The loss L1 is used to calculate the difference between the predicted future depth of a video frame and the ground truth value. The distance between them.

[0083] For example, the loss function of formula (1) can be configured based on the following formula. and the loss function of formula (2)

[0084]

[0085]

[0086] During the meta-test component 120, a pre-trained neural network model f with updated weights θ can be trained based on previously unseen data. θ 117, to generate depth predictions for future video frames quickly and efficiently. In some embodiments, previously unseen data may include multiple m new consecutive video frames 122 with consecutive timestamps, denoted as...

[0087] During the meta-test, several video frames from multiple m new consecutive video frames 122 can be used as input to perform inner-loop updates using formula (1) during the adaptive process 125, and to obtain multiple updated second weights θ'. i An adaptive neural network model is then developed. The neural network model 127 can then be applied to the remainder of multiple new consecutive video frames 122 to measure its performance. Finally, the system 100 can generate a sequence of video frames based on the neural network model 127 with updated weights θ. Subsequent video frames The depth representation 129. In some embodiments, based on a neural network model 127 with updated weights θ, video frames Subsequent video frames The depth is represented as 129.

[0088] As a non-limiting example, the following pseudocode presents the overall exemplary process performed by the meta-learning system 100. During the meta-training component 110:

[0089] - Input: θ, D = (D img D depth )

[0090] - When it does not converge:

[0091] • Sampling training for batches: D = (D img D depth In ) do:

[0092] For D i Each image D in img do:

[0093] - Inner loop update (current frame reconstruction):

[0094] ο

[0095] End For

[0096] Outer Ring Road Renewal (Future In-Depth Forecast):

[0097] ο

[0098] Output: Neural network model f θ weight θ

[0099] N can be the total number of training data batches.

[0100] During meta-test component 120:

[0101] - Input: θ, D = (D img )

[0102] - When it does not converge:

[0103] • Sample new batches of test data

[0104] ·ForD=(D img In ) do:

[0105] - Inner loop update (current frame reconstruction):

[0106] ο

[0107] End For

[0108] -reasoning:

[0109] ·Estimated_Depth_Values=f(θ'(D img ))

[0110] - Output: Estimated_Depth_Values

[0111] The output Estimated_Depth_Values ​​is the final output 129 of system 100.

[0112] Figure 4 A block diagram 400 of an exemplary processing system 400 operable for implementing the meta-learning system 100 is shown. The processing system 400 includes a processor 402 for executing computer program instructions and a memory 404 for storing executable instructions and data. The processing system can be connected to a database 430 for acquiring new video frames 416 and storing estimated depth values ​​418 for any video frames.

[0113] Processor 402 can be implemented as any processing resource capable of executing computer program instructions, such as a computer or one or more processors on one or more computing platforms. Memory 404 can be implemented as any data storage resource, such as one or more disk drives, random access memory, or volatile or non-volatile memory on one or more computing platforms.

[0114] The memory 404 stores several types of computer programs in the form of executable instructions. The memory has a set of executable instructions 410 for performing the methods described herein. The memory also stores one or more sets of instructions for training neural networks or other machine learning models to generate estimated depth values ​​for one or more video frames.

[0115] Several types of data 480 can be stored on memory 404. Data 480 may include, for example, a pre-trained neural network model f with weights θ. θ The matrix representation 412. Matrix representation 412 can include models used as neural networks (e.g., pre-trained neural network models f). θ The data 480 includes the input matrix or weights, and the matrix updated or generated by the neural network. Data 480 may also include matrix representation 414, representing multiple second weights θ' updated during the training process 115 and the adaptation process 125 in system 100. Data 480 may also include matrix representation 416 representing multiple new video frames and matrix representation 418 representing estimated depth values ​​of future video frames generated based on the multiple new video frames 416.

[0116] Figure 5Exemplary method 500 for training a neural network model to predict the depth of future video frames using meta-learning methods is shown in some exemplary embodiments. For example, method 500 may be executed by system 100.

[0117] When operating 510, the system can receive multiple training data sets. And for each D i : Each represents a video frame consisting of t consecutive video frames with consecutive timestamps, and It follows the video frame. Depth representation of subsequent future video frames.

[0118] In operation 520, the system can receive a pre-trained neural network model f with multiple weights θ. θ This neural network model can be based on Figure 2 or Figure 3 The process described in the text involves pre-training. The pre-trained neural network model f θ It can be stored in the memory of a computing system for quick retrieval and updating.

[0119] In some embodiments, the pre-trained neural network model f θ The training process can include the current frame reconstruction process and the future depth prediction process.

[0120] In some embodiments, in the pre-trained neural network model f θ During the training process, the system (which may be a separate system from system 100) can: receive multiple consecutive video frames with consecutive timestamps. From multiple consecutive video frames Extract multiple spatial features; f with random values θ Multiple initial parameters are set to multiple weights θ; from multiple consecutive video frames Extract multiple spatial features; during the current frame reconstruction process: reconstruct multiple consecutive video frames based on multiple spatial features. For each video frame in the reconstructed video frame; based on the reconstructed video frame, update the value of at least one of the multiple weights θ; during the future depth prediction process: extract multiple consecutive video frames based on multiple spatial features. Temporal characteristics; based on multiple consecutive video frames Temporal features, generating data immediately following video frames. Subsequent video frames Depth prediction based on video frames; For depth prediction, update the value of at least one weight among multiple weights θ.

[0121] In some embodiments, 3D CNNs can be used to extract temporal features during a future depth prediction process.

[0122] When the pre-trained neural network model f θ If convergence fails, operations 530 and 540 are executed. In operation 530, the system can base its decisions on each D... i Multiple consecutive video frames in i = 1...N and pre-trained neural network model f θ Calculate multiple second weights θ' i For example, calculating multiple second weights θ' i It can be used Based on the following formula (1):

[0123]

[0124] Where α represents the learning rate or step size, Indicates based on Calculated loss, This represents the gradient operator.

[0125] In operation 540, the system can be based on multiple training data. and multiple second weights θ' i Update multiple weights θ. For example, updating multiple weights θ can be based on the following formula (2):

[0126]

[0127] Where β represents the learning rate, Indicates based on The calculated loss is where N is the total number of training data batches. Let represent the gradient operator, and ∑ is the summation operator.

[0128] In operation 550, the system can receive multiple m new consecutive video frames with consecutive timestamps. These multiple m new consecutive video frames can be obtained from local storage device 404 or remote database 430 via a network connection.

[0129] In operation 560, the system can predict the next video frame based on multiple updated weights θ. Subsequent video frames Depth representation.

[0130] In some embodiments, predict video frames The depth representation may include: based on the plurality of new consecutive video frames And update the multiple weights θ, and update the multiple second weights θ' i Based on the updated multiple second weights θ'i Generate the depth representation.

[0131] In some embodiments, the plurality of second weights θ' are updated i It can be based on the formula:

[0132]

[0133] Where α is the learning rate or step size, Indicates based on Calculated loss, This represents the gradient operator.

[0134] In some embodiments, depth values ​​can be represented using a depth map.

[0135] The system and method described in this paper can rapidly and efficiently generate or predict depth values ​​for video frames generated from monocular video, as the system is designed to quickly adapt to new video scenes through self-supervised training using meta-learning methods. The system can be used to predict the depth values ​​of future video frames based on a few recent video frames from the current video. These depth values ​​can then be used by computational systems on autonomous vehicles to generate future states to assist navigation. The system is capable of processing previously unseen video data in batches using only a few sample data points, meaning that the system does not require a large-scale training dataset to adapt to new video scenes in order to generate estimated depth values ​​for frames in new videos.

[0136] Although the methods and processes are described in a specific order in this invention, one or more steps of the methods and processes may be omitted or modified as appropriate. One or more steps may be performed sequentially, but not in the order described (as the case may be).

[0137] Although the invention has been described, at least in part, those skilled in the art will understand that the invention also relates to various components for performing at least some aspects and features of the described methods by means of hardware components, software, or any combination of both. Accordingly, the technical solutions of the invention can be embodied in the form of a software product. Suitable software products can be stored in pre-recorded storage devices or other similar non-volatile or non-transitory computer-readable media, such as DVDs, CD-ROMs, USB flash drives, removable hard drives, or other storage media. The software product includes examples of instructions stored thereon that enable processing devices (e.g., personal computers, servers, or network devices) to perform the methods disclosed herein.

[0138] The invention may be embodied in other specific forms without departing from the subject matter of the claims. The exemplary embodiments described are merely illustrative in all respects and not restrictive. Selected features from one or more of the foregoing embodiments may be combined to create alternative embodiments not explicitly described, and it is understood that features suitable for such combinations are within the scope of the invention.

[0139] All values ​​and sub-ranges within the scope of disclosure are also disclosed. Furthermore, although the systems, devices, and processes disclosed and shown herein may include a specific number of elements / components, the systems, devices, and components may be modified to include more or fewer of such elements / components. For example, although any element / component disclosed may be mentioned in the singular, embodiments disclosed herein may be modified to include multiple such elements / components. The subject matter described herein is intended to cover and encompass all appropriate technical changes.

Claims

1. A computer-implemented method for predicting the depth of video frames, characterized in that, include: Receive multiple training data And for each : ,in, Each represents multiple timestamps with consecutive timestamps. t One video frame in a series of consecutive video frames; It is immediately following the video frame Depth representation of future video frames; Receive a pre-trained neural network model with multiple weights θ. f θ ; When the pre-trained neural network model f θ When it does not converge: Based on each The multiple consecutive video frames mentioned in and the pre-trained neural network model f θ Calculate multiple second weights ; Based on the aforementioned training data and the plurality of second weights Update the multiple weights θ; The training process of the pre-trained neural network model fθ includes the current frame reconstruction process and the future depth prediction process; The training process of the pre-trained neural network model fθ includes: Receive multiple data with consecutive timestamps m A new continuous video frame ; Based on updated weights θ, the prediction immediately following the video frame is obtained. Subsequent video frames Depth representation; Receive multiple consecutive video frames with consecutive timestamps. ; Will have random values f θ The multiple initial parameters are set as the multiple weights θ; From the multiple consecutive video frames Extracting multiple spatial features; During the current frame reconstruction process: Reconstructing the multiple consecutive video frames based on the multiple spatial features Each video frame in the video; Based on the reconstructed video frame, update the value of at least one of the plurality of weights θ; During the future depth prediction process: Based on the aforementioned spatial features, the multiple consecutive video frames are extracted. Temporal characteristics; Based on the multiple consecutive video frames The time features are used to generate a video frame that immediately follows the video frame. Subsequent video frames Depth prediction; Based on the video frame The depth prediction updates the value of at least one of the plurality of weights θ.

2. The method according to claim 1, characterized in that, Calculate the plurality of second weights Based on the formula: in, Indicates the learning rate. Indicates based on Calculated loss, This represents the gradient operator.

3. The method according to claim 2, characterized in that, The multiple weights θ are updated based on the formula: in, Indicates the learning rate. Indicates based on Calculated loss, This represents the gradient operator.

4. The method according to claim 3, characterized in that, Predict video frames The depth representation includes: Based on the aforementioned multiple new consecutive video frames And the updated multiple weights θ, update the multiple second weights. ; Based on updated multiple second weights Generate the depth representation.

5. The method according to claim 4, characterized in that, Update the multiple second weights Based on the formula: in, It's the learning rate. Indicates based on Calculated loss, This represents the gradient operator.

6. The method according to claim 1, characterized in that, Extracting the temporal features includes using a 3D convolutional neural network.

7. The method according to any one of claims 1 to 6, characterized in that, For one or more surfaces in the video frame, the depth representation of any video frame includes a depth value representing the estimated distance from the corresponding surface to the viewpoint.

8. The method according to claim 7, characterized in that, The depth representation of any video frame includes the depth map of the video frame.

9. A system for predicting the depth of a video frame, characterized in that, The system includes: Processing unit; A memory coupled to the processing unit stores machine-executable instructions that, when executed by the processing unit, cause the system to: Receive multiple training data And for each : ,in, Each represents multiple timestamps with consecutive timestamps. t One video frame in a series of consecutive video frames; It is immediately following the video frame Depth representation of future video frames; Receive a pre-trained neural network model with multiple weights θ. f θ ; When the pre-trained neural network model f θ When it does not converge: Based on each The multiple consecutive video frames mentioned in and the pre-trained neural network model f θ Calculate multiple second weights ; Based on the aforementioned training data and the plurality of second weights Update the multiple weights θ; Receive multiple data with consecutive timestamps m A new continuous video frame ; Based on updated weights θ, predict the next video frame. Subsequent video frames Depth representation; The training process of the pre-trained neural network model fθ includes the current frame reconstruction process and the future depth prediction process; The pre-trained neural network model f θ During the training process, when the machine-executable instructions are executed by the processing unit, the system: Receive multiple consecutive video frames with consecutive timestamps. ; Will have random values f θ The multiple initial parameters are set as the multiple weights θ; From the multiple consecutive video frames Extracting multiple spatial features; During the current frame reconstruction process: Reconstructing the multiple consecutive video frames based on the multiple spatial features Each video frame in the video; Based on the reconstructed video frame, update the value of at least one of the plurality of weights θ; During the future depth prediction process: Based on the aforementioned spatial features, the multiple consecutive video frames are extracted. Temporal characteristics; Based on the multiple consecutive video frames The time features are used to generate a video frame that immediately follows the video frame. Subsequent video frames Depth prediction; Based on the video frame The depth prediction updates the value of at least one of the plurality of weights θ.

10. The system according to claim 9, characterized in that, Calculate the plurality of second weights Based on the formula: in, Indicates the learning rate. Indicates based on Calculated loss, This represents the gradient operator.

11. The system according to claim 10, characterized in that, The multiple weights θ are updated based on the formula: in, Indicates the learning rate. Indicates based on Calculated loss, This represents the gradient operator.

12. The system according to claim 11, characterized in that, Predict video frames The depth representation includes: Based on the aforementioned multiple new consecutive video frames And the updated multiple weights θ, update the multiple second weights. ; Based on updated multiple second weights Generate the depth representation.

13. The system according to claim 12, characterized in that, Update the multiple second weights Based on the formula: （3） in, It's the learning rate. Indicates based on Calculated loss, This represents the gradient operator.

14. The system according to claim 9, characterized in that, During the extraction of the temporal features, when the machine-executable instructions are executed by the processing unit, the system: extracts the temporal features using a 3D convolutional neural network.

15. The system according to any one of claims 9 to 14, characterized in that, For one or more surfaces in the video frame, the depth representation of any video frame includes a depth value representing the estimated distance from the corresponding surface to the viewpoint.

16. A computer-readable medium, characterized in that, The system stores machine-readable instructions for configuring the processing unit to predict the depth of video frames, the machine-readable instructions being used to cause the processing unit to perform the method according to any one of claims 1 to 8.

17. A computer program product, characterized in that, The program includes instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 8.

Images