Video prediction method, device and equipment based on dynamic routing and readable medium

Abstract

The invention provides a video prediction method, device and equipment based on dynamic routing and a readable medium, and belongs to the technical field of computers. The method comprises the following steps: inputting an original frame into a dynamic routing video prediction model to obtain an output prediction frame; specifically, in a dynamic routing video prediction model, an encoder is used for extracting spatial features from an original frame, and a dynamic routing unit is used for outputting a selection probability corresponding to each subunit in a predictor according to the spatial features; the predictor determines a selected target subunit according to the selection probability, and performs fusion processing on the spatial features output by the encoder by adopting the target subunit so as to predict deep features; and the decoder reconstructs the image frame based on the deep features and outputs a prediction frame corresponding to the original frame. The subunits, participating in feature calculation, of the predictor are dynamically adjustable, different scenes can be flexibly adapted, the prediction task of a single frame or multiple original frames can be processed, and the error accumulation effect in the multi-frame prediction task is effectively relieved.

Description

Technical Field

[0001] The present disclosure belongs to the field of computer technology, and in particular relates to a video prediction method based on dynamic routing, a video prediction device based on dynamic routing, an electronic device, and a computer-readable storage medium. Background Art

[0002] Video prediction is a machine vision task that predicts future video frames based on the information of existing video frames. It is achieved by extracting temporal and spatial features from existing continuous video frames and combining temporal and spatial information prediction. It is widely used in video coding, autonomous driving, video generation and other fields.

[0003] The current conventional scheme usually uses deep learning methods to extract features of existing video frames and predict future video frames. In order to improve model performance, a complex model combining multiple networks can be used, which leads to long training and inference time and high energy consumption, and is difficult to be widely used; in addition, only one frame can be predicted in a single prediction task. In multi-frame prediction tasks, the prediction result of the previous frame is required to infer the next frame, which may lead to error accumulation between multi-frame images and affect the prediction effect of multi-frame tasks; and most of them use static networks with fixed structures, which may lead to excessive calculations in simple scenarios or computing performance cannot meet prediction requirements in complex scenarios. Summary of the invention

[0004] The purpose of the embodiments of the present disclosure is to provide a video prediction method, device, electronic device and readable storage medium based on dynamic routing, in which a dynamic routing unit dynamically selects a sub-unit of a predictor to participate in feature calculation according to the output of an encoder, so as to flexibly adapt to different performance requirements of simple and complex scenes and effectively improve computing efficiency and accuracy; a simple model structure of an encoder, a decoder and a predictor is adopted, which can flexibly handle prediction tasks of single or multiple original frames and effectively alleviate the error accumulation effect in multi-frame prediction tasks.

[0005] In order to solve the above technical problems, the present disclosure is implemented as follows:

[0006] In a first aspect, the present disclosure provides a video prediction method based on dynamic routing, which is used for a dynamic routing video prediction model. The dynamic routing video prediction model includes an encoder, a predictor, a decoder and a dynamic routing unit. The predictor includes more than two sub-units. The method may include: inputting at least two original frames into the dynamic routing video prediction model to obtain at least two predicted frames output by the dynamic routing video prediction model; wherein, in the dynamic routing video prediction model, the encoder is used to extract spatial features from each original frame; the dynamic routing unit is used to output a selection probability corresponding to each sub-unit in the predictor according to the spatial features output by the encoder; the predictor is used to determine a selected target sub-unit in the sub-unit according to the selection probability output by the dynamic routing unit, and use the target sub-unit to fuse the spatial features output by the encoder to predict deep features; the decoder reconstructs the image frame based on the deep features and outputs at least two predicted frames corresponding to the original frame.

[0007] Optionally, the encoder and decoder are computed independently over the channel dimension of each raw frame.

[0008] Optionally, the predictor and dynamic routing units are computed over all channel dimensions.

[0009] Optionally, a skip connection is used between the encoder and the decoder.

[0010] Optionally, the encoder is also used to retain the shallow features of each layer output when extracting the spatial features of each original frame; the decoder is also used to obtain the shallow features through jump connections, and specifically to fuse them with the deep features output by the predictor to output a predicted frame.

[0011] Optionally, the loss function used by the dynamic routing video prediction model during training includes reconstruction loss; the reconstruction loss is used to quantify the difference between the real frame corresponding to the sample frame received by the dynamic routing video prediction model during training and the output prediction frame.

[0012] Optionally, the loss function used by the dynamic routing video prediction model during the training process also includes a sparse sampling loss; the sparse sampling loss is used to control the sparsity of the dynamic routing unit output during the training process.

[0013] Optionally, the number of original frames input into the dynamic routing video prediction model is the same as the number of predicted frames output by the dynamic routing video prediction model.

[0014] In a second aspect, the present disclosure also provides a video prediction device based on dynamic routing, which includes a video prediction model based on dynamic routing. The video prediction model based on dynamic routing may include: an encoder, a predictor, a decoder and a dynamic routing unit, the predictor including more than two sub-units; the encoder is used to obtain at least two original frames of the input, and to extract spatial features from each original frame; the dynamic routing unit is used to output the selection probability corresponding to each sub-unit in the predictor according to the spatial features output by the encoder; the predictor is used to determine the selected target sub-unit in the sub-unit according to the selection probability output by the dynamic routing unit, and use the target sub-unit to fuse the spatial features output by the encoder to predict deep features; the decoder reconstructs the image frame based on the deep features, and outputs at least two predicted frames corresponding to the original frame.

[0015] In a third aspect, the present disclosure provides an electronic device comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the video prediction method based on dynamic routing as in the first aspect.

[0016] In a fourth aspect, the present disclosure provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the video prediction method based on dynamic routing as in the first aspect are implemented.

[0017] In a fifth aspect, the present disclosure provides a chip, comprising a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps of the video prediction method based on dynamic routing as in the first aspect.

[0018] In a sixth aspect, the present disclosure provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute steps for implementing the video prediction method based on dynamic routing as in the first aspect.

[0019] The present disclosure provides a video prediction method based on dynamic routing, a video prediction device based on dynamic routing, an electronic device and a computer-readable storage medium. The method can input at least two frames of original frames into a dynamic routing video prediction model to obtain at least two frames of predicted frames output by the dynamic routing video prediction model; specifically, in the dynamic routing video prediction model, the encoder is used to extract spatial features from the original frame, and the dynamic routing unit is used to output the selection probability corresponding to each sub-unit in the predictor according to the spatial features; the predictor determines the selected target sub-unit according to the selection probability, and uses the target sub-unit to fuse the spatial features output by the encoder to predict the deep features; the decoder reconstructs the image frame based on the deep features and outputs the predicted frame corresponding to the original frame. The present disclosure dynamically selects the sub-unit of the predictor participating in the feature calculation according to the output of the encoder by the dynamic routing unit, so that the depth of the dynamic routing video prediction model is adjustable, thereby flexibly adapting to the different performance requirements of simple and complex scenes, and effectively improving the calculation efficiency and accuracy; the simple model structure of the encoder, decoder and predictor is adopted, which can flexibly process the prediction task of single or multiple frames of original frames, and effectively alleviate the error accumulation effect in the multi-frame prediction task. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of the steps of the video prediction method based on dynamic routing provided by an embodiment of the present disclosure;

[0021] Figure 2 A schematic diagram of multi-scale deep and shallow feature fusion provided in an embodiment of the present disclosure;

[0022] Figure 3 A schematic diagram of the training process of the dynamic routing video prediction model provided in the embodiment of the present disclosure;

[0023] Figure 4 A schematic diagram of the inference process of the dynamic routing video prediction model provided in the embodiment of the present disclosure;

[0024] Figure 5 A schematic diagram of the structure of a video prediction device based on dynamic routing provided in an embodiment of the present disclosure;

[0025] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure;

[0026] Figure 7 A hardware schematic diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0028] The terms "first", "second", etc. in the specification and claims of the present disclosure are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable when appropriate, so that the embodiments of the present disclosure can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0029] The video prediction method based on dynamic routing provided by the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0030] Figure 1 The following is a flowchart of the steps of the video prediction method based on dynamic routing provided by the embodiment of the present disclosure. Figure 1 As shown, the method can be applied to a dynamic routing video prediction model 100, which includes an encoder 110, a predictor 130, a decoder 140, and a dynamic routing unit 120, wherein the predictor 130 includes more than two subunits 131. The method can include the following steps A1 to A2.

[0031] Step A1: input at least two original frames into the dynamic routing video prediction model 100.

[0032] Step A2: Obtain at least two predicted frames output by the dynamic routing video prediction model 100.

[0033] In the embodiment of the present disclosure, at least two original frames may be input into the dynamic routing video prediction model 100, and the dynamic routing video prediction model 100 performs spatial feature extraction, deep feature prediction, and image frame reconstruction based on the deep features, thereby obtaining at least two predicted frames output by the dynamic routing video prediction model 100. Specifically, in the dynamic routing video prediction model 100, the processing from inputting the original frames to outputting the predicted frames is as follows:

[0034] The encoder 110 is used to extract spatial features from each raw frame.

[0035] In the disclosed embodiment, the encoder 110 is a submodule of the dynamic routing video prediction model 100, which can extract corresponding spatial features from each original frame and pass them to the predictor 130 and the dynamic routing unit 120. Among them, the spatial features can refer to the local information and structure in the original frame, such as texture, color, shape, etc., so that the dynamic routing video prediction model can understand the content of the original frame. The encoder 110 can adopt a multi-layer convolutional neural network structure, and downsample the original frame through the convolution layer, which can effectively reduce the amount of calculation and reduce the complexity of the model, thereby improving the training and reasoning rate.

[0036] For example, N convolutional layers are stacked in the encoder 110, and the operation of each convolutional layer includes 3x3 convolution (Conv2d), normalization (Norm2d) and nonlinear activation (σ), as shown in formula (1):

[0037] E i =σ(Norm2d(Conv2d(E i-1 ))),1≤i≤N (1)

[0038] Among them, E i-1 is the output of a module on the encoder. When i=1, E i-1 is the original input frame.

[0039] The step size setting of the convolution layer in the encoder 110 can follow the rule of "the step size of the first layer is 1, the step size of the second layer is 2, the step size of the third layer is 1, the step size of the fourth layer is 2, ... ". On this basis, after every two convolution layers, the height and width (H, W) of the feature map of the original frame will be downsampled by 2 times.

[0040] The dynamic routing unit 120 is used to output the selection probability corresponding to each sub-unit in the predictor 130 according to the spatial features output by the encoder 110 .

[0041] In the disclosed embodiment, the dynamic routing unit 120 is a submodule of the dynamic routing video prediction model 100, and can output the selection probability corresponding to each subunit in the predictor 130 according to the spatial feature output by the encoder 110. Among them, the selection probability can represent the selection of whether the subunit participates in the subsequent processing. Specifically, according to the size of the selection probability, when the selection probability is greater than or equal to the selection threshold, it can indicate that the subunit is selected, and when the selection probability is less than the selection threshold, it can indicate that the subunit is skipped in the subsequent processing; or, the output of the selection probability can take values ​​within a fixed range, and the corresponding selection or skipping is determined according to a specific value, such as the selection probability has a value range of (0, 1), when the value is 1, it indicates that the corresponding subunit is selected, and when the value is 0, it indicates that the corresponding subunit is skipped.

[0042] For example, assuming the selection threshold is 0.5, the selection probability p i When it is greater than or equal to 0.5, it is set to 1 to select the subunit, whereas when it is less than 0.5, it is set to 0 to skip the subunit.

[0043] Among them, P k represents the selection probability of the kth sub-unit output in the predictor 130 by the dynamic routing unit 120 .

[0044] In the disclosed embodiment, when the spatial features corresponding to a single frame of the encoder 110 are obtained, they can be spliced ​​in the channel dimension as the input of the dynamic routing unit 120. In the dynamic routing module, global average pooling can be performed to flatten the input features in the channel dimension, and then the number of channels can be verified through a linear layer to make it the same as the number of sub-units 131 in the predictor 130, and the selection probability corresponding to each sub-unit 131 can be obtained through activation function processing (such as Sigmoid function). On this basis, in order to prevent the dynamic routing video prediction model 100 from falling into a trivial solution, such as all sub-units 131 of the predictor 130 are selected, the selection probability can be normalized. The normalization process can make the sum of the probabilities of each sub-unit 131 equal to 0.45 times the number of sub-units 131 in the predictor 130, thereby ensuring that during the training and reasoning process, there is always a sub-unit 131 selected, as shown in the following formula (2):

[0045]

[0046] After normalization, the dynamic routing unit can be made differentiable through the Gumbel-Softmax layer. The Gumbel-Softmax layer uses a continuous approximation method for discrete variables, combining the Gumbel distribution and the Softmax function, to achieve sampling from discrete categories while maintaining differentiability. This allows the dynamic routing unit to be optimized through back propagation during training, so that the entire dynamic routing video prediction model 100 can be trained end-to-end. Gumbel-Softmax can be expressed as the following formula (3):

[0047]

[0048] Among them, p i is the input of Gumbel-Softmax, g i is the noise term drawn from the Gumbel distribution, and τ is the temperature parameter that controls the degree of binarization.

[0049] τ decreases exponentially with the number of iterations and can be expressed as Where α is the decay rate, iter represents the number of iterations, and τ0 is the initial temperature.

[0050] In the above formula (3), τ0 can be set to 1 and α to 0.99998. In the initial stage of the training process, the selection probability is not strictly constrained, so that the entire model can be effectively trained; and at the end of the training process, τ is close to 0, making the selection probability almost binary, which is consistent with the discrete reasoning process.

[0051] The predictor 130 is used to determine the selected target sub-unit in the sub-unit 131 according to the selection probability output by the dynamic routing unit, and to use the target sub-unit to perform fusion processing on the spatial features output by the encoder to predict the deep features.

[0052] In the disclosed embodiment, the deep features may represent dynamic change information between frames, such as inter-frame motion information, change rate, relationship between objects, etc. The predictor 130 may include multiple sub-units, and the output of the previous sub-unit may be used as the input of the next sub-unit for processing, and the deep features of the future image frame may be predicted based on the fusion processing of the input spatial features.

[0053] For example, the predictor 130 may first perform a 1x1 convolution operation on the input to adjust the channel dimension to 256, and then pass it through at most M and at least 1 sub-unit (SubBlock). Each sub-unit may include a 1x1 convolution layer and 4 parallel grouped convolutions, the number of groups may be 8, and the convolution kernel sizes are 3, 5, 7, and 11, respectively. In each sub-unit, the input is first processed by a 1x1 convolution layer, and then passes through 4 parallel grouped convolutions respectively, and the outputs of the 4 grouped convolutions are added to obtain the output of the sub-unit. After passing through at least 1 and at most M sub-units, the channel dimension is restored through a 1x1 convolution operation to obtain the deep features output by the predictor 130. The output P of the kth sub-unit k It can be expressed as shown in the following formula (4):

[0054] P k =SubBlock(P k-1 ),1≤k≤M (4)

[0055] Among them, P k-1 is the output of the previous sub-unit in the predictor; when k = 1, P k-1 is the output of the encoder.

[0056] Furthermore, the selection of subunits in predictor 130 is based on the selection probability p i At this time, the output P of the subunit k It can be expressed as shown in the following formula (5):

[0057] P k =p i*SubBlock(P k-1 )+(1-p i )*P k-1 (5)

[0058] At this time, the output of the kth subunit is k-1 and SubBlock(P k-1 ) is the weighted sum of .

[0059] Or, it can be expressed as the following formula (6):

[0060]

[0061] p i When it is greater than or equal to 0.5, it is set to 1 and the output SubBlock (P k-1 ); otherwise, it is set to 0 when it is less than 0.5, and the output is P k-1 , thus skipping the kth subunit.

[0062] In the above reasoning process, dynamic routing selection of the required sub-units in each video prediction task is achieved, and the path is identified and optimized based on the spatial features of the encoder output, which can effectively adapt to the computing requirements of simple or complex scenes.

[0063] The decoder 140 reconstructs the image frame based on the deep features and outputs at least two predicted frames corresponding to the original frame.

[0064] In the disclosed embodiment, the decoder 140 is a submodule of the dynamic routing video prediction model 100 and corresponds to the encoder 110. A multi-layer convolutional neural network structure may be adopted to reconstruct image frames by upsampling based on deep features through the convolutional layer to obtain output prediction frames.

[0065] For example, N convolutional layers are stacked in the decoder 140, and upsampling is performed every two convolutional layers. When upsampling is performed in formula (7), the convolutional layer used is a pixel reorganization (PixelShuffle) layer; otherwise, a 3x3 convolutional layer with a stride of 1 is used; after the convolution operation, normalization and nonlinear activation processing are performed to generate the output of each convolutional layer.

[0066] D i =σ(Norm2d(umConv2d(D i-1 +E N-i+1 ))),1≤j≤N (7)

[0067] Among them, D i-1 represents the output of a convolutional layer on decoder 140. When i=1, D i-1 represents the output of the predictor 130; it may also include the output feature E corresponding to the convolutional layer of the encoder 110N-i+1 .

[0068] In an optional embodiment of the present disclosure, the encoder 110 and the decoder 140 perform independent calculations on the channel dimension of each original frame.

[0069] In the disclosed embodiment, when the encoder 110 receives N original frames, it separately calculates and extracts spatial features on the channel dimension of each original frame; and the decoder 140 independently performs deep feature reconstruction on the channel dimension of each image frame and outputs a predicted frame.

[0070] In an optional embodiment of the present disclosure, the predictor 130 and the dynamic routing unit 120 perform calculations on all channel dimensions.

[0071] In the disclosed embodiment, the predictor 130 can perform fusion calculations based on all channel dimensions, which is beneficial to capturing dynamic evolution information in time series and improving the accuracy of multi-frame prediction tasks. The corresponding dynamic routing unit 120 also performs calculations on all channel dimensions to achieve comprehensive analysis and routing of sub-units 131 in the predictor 130 based on spatial features.

[0072] In an optional embodiment of the present disclosure, the number of original frames input into the dynamic routing video prediction model is the same as the number of predicted frames output by the dynamic routing video prediction model.

[0073] In the disclosed embodiment, the number of frames of the prediction task can be determined according to the actual business, so that the number of input original frames and output prediction frames can be flexibly adjusted. On this basis, the error accumulation in iterative prediction caused by decomposing the multi-frame prediction task into continuous single-frame prediction tasks is avoided, and multi-frame prediction can be performed directly, which has good prediction performance in long video prediction tasks.

[0074] For example, the original frame X∈(B,T,C,H,W) is input into the dynamic routing video prediction model 100, where B is the batch size, T is the number of consecutive original frames, C is the image channel, and H and W are the height and width of the image. On this basis, the original frame X is reconstructed into a tensor of (BxT,C,H,W) and input into the encoder 110, so that the encoder 110 extracts spatial features of a single frame of the original frame in the dimension of each image channel, and obtains the spatial feature E output by the encoder 110 i ∈(BxT,C,H,W).

[0075] On this basis, the spatial feature E i∈(BxT,C,H,W) is reshaped into a tensor of dimension (B,TxC,H,W) and input into the dynamic routing unit 120. The channel dimension is first reduced through two 3x3 convolution operations, and multi-frame feature maps are fused in time sequence. Finally, the selection probability of each sub-unit of the predictor 130 is obtained.

[0076] And, the spatial feature E i ∈(BxT,C,H,W) is reshaped into a tensor of dimension (B,TxC,H,W) and input into predictor 130. Based on the selection probability of each subunit of predictor 130 output by dynamic routing unit 120, the corresponding subunit is selected for operation to obtain the deep feature Z output by predictor 130 k ∈(B,TxC,H,W).

[0077] The deep features Z output by the predictor 130 k ∈(B,TxC,H,W) is reshaped into a tensor of dimension (BxT,C,H,W) and input into the decoder 140 for image frame reconstruction, so that the image frame reconstruction of the decoder 140 focuses on single frame processing and finally outputs T frame prediction frames.

[0078] In an optional embodiment of the present disclosure, a skip connection is adopted between the encoder 110 and the decoder 140 .

[0079] In the disclosed embodiment, skip connection refers to the skipping of information between corresponding layers between the encoder 110 and the decoder 140, so that the decoder 140 can avoid losing the detailed features of the object while reconstructing the deep features. Based on the deep features output by the predictor 130, the decoder 140 obtains the shallow features extracted by the predictor 130 during the processing through the skip connection, realizing the fusion of multi-scale deep and shallow information.

[0080] In an optional embodiment of the present disclosure, the encoder 110 is also used to retain the shallow features of each layer output when extracting the spatial features of each original frame; the decoder 140 is also used to obtain the shallow features through jump connections, and specifically to merge them with the deep features output by the predictor 130 to output a predicted frame.

[0081] In the disclosed embodiment, each layer of the encoder 110 and the decoder 140 may be connected via a skip connection. When the encoder 110 extracts the spatial features of each original frame, the shallow features output by each layer are retained, thereby obtaining a set of shallow features of multiple scales. The shallow features of each layer in the encoder 110 are transmitted to the decoder 140 via the skip connection, and are added point-by-point (element-wise addition) with the output of the previous layer in the decoder 140, thereby realizing the joint action of deep and shallow features.

[0082] like Figure 2 As shown, a schematic diagram of multi-scale deep and shallow feature fusion is provided. Taking the example of stacking four convolutional layers in the encoder 110 and the decoder, the input of the first convolutional layer in the decoder 140 includes the deep features of the predictor 130 input and the shallow features of the corresponding layer input of the encoder 110; on this basis, the input of each subsequent convolutional layer includes the output of the previous convolutional layer and the shallow features of the corresponding layer input of the encoder 110. Figure 2 In the above method, the fusion of deep and shallow information is achieved by adding these two parts.

[0083] In an optional embodiment of the present disclosure, the loss function used by the dynamic routing video prediction model 100 during the training process includes a reconstruction loss; the reconstruction loss is used to quantify the difference between the real frame corresponding to the sample frame received by the dynamic routing video prediction model 100 during the training process and the output prediction frame.

[0084] In the disclosed embodiment, the dynamic routing video prediction model 100 can be obtained through pre-training, by receiving sample frames and outputting predicted frames corresponding to the sample frames to learn the execution of video prediction tasks. During the training process, reconstruction loss can be used as a loss function. The reconstruction loss is used to quantify the difference between the real frame corresponding to the sample frame received by the dynamic routing video prediction model 100 during the training process and the output predicted frame, and characterizes the prediction and reconstruction ability of the dynamic routing video prediction model 100 for the image frame. The reconstruction loss can be expressed as the mean square error (MSE) as shown in the following formula (8):

[0085]

[0086] Among them, Y i represents the real frame, represents the predicted frame, and N is the total number of frames in the sample frame.

[0087] In an optional embodiment of the present disclosure, the loss function used by the dynamic routing video prediction model during the training process also includes a sparse sampling loss; the sparse sampling loss is used to control the sparsity of the dynamic routing unit output 120 during the training process.

[0088] In the disclosed embodiment, a composite loss function may also be used, which includes a sparse sampling loss on the basis of the reconstruction loss, so as to control the sparsity of the dynamic routing unit 120 during the training process and avoid converging to a trivial solution, such as all subunits 131 of the predictor 130 are selected. The sparse sampling loss can be expressed as the following formula (9):

[0089]

[0090] Among them, p$ represents the selection probability of the mth sub-unit of the predictor 130 output by the dynamic routing unit 120 , and M represents the total number of sub-units 131 in the predictor 130 .

[0091] In summary, when the loss function includes reconstruction loss and sparse sampling loss, the overall training objective can be expressed as the following formula (10):

[0092] L total =L recon +λL sparse (10)

[0093] Wherein, λ is the weight of the sparse sampling loss, allowing the dynamic routing unit to have different degrees of sparsity. In the embodiment of the present disclosure, λ can be set to 0.01.

[0094] In the disclosed embodiment, the training process uses the Adam algorithm for end-to-end training, and the initial learning rate is set to 0.001. On this basis, in order to ensure stable convergence, a learning rate decay strategy can be implemented throughout the training process, and the differentiability of the dynamic routing unit 120 enables it to be trained together with the backbone network, thereby ensuring the overall collaborative learning of the dynamic routing video prediction model 100.

[0095] like Figure 3 A schematic diagram of the training process of the dynamic routing video prediction model 100 in the embodiment of the present disclosure is provided. Figure 3 As shown, it includes the following steps 301 to 307:

[0096] Step 301: Input N consecutive sample frames.

[0097] Step 302: concatenate the N sample frames in the batch dimension, i.e., reshape the input from (B, T, C, H, W) to (BxT, C, H, W)), and input the input to the encoder 110 to extract the spatial features of each sample frame.

[0098] Step 303: splice the spatial features of each sample frame output by the encoder 110 in the channel dimension, that is, reshape the feature dimension (BxT, C, H, W) into (B, CxT, H, W) as the input of the dynamic routing unit 120, and calculate the selection probability p of each subunit 131 of the predictor 130 through the dynamic routing unit 120 i and sparse sampling loss L sparse .

[0099] Step 304: splice the spatial features of each sample frame output by the encoder 110 in the channel dimension, that is, reshape the feature dimension (BxT, C, H, W) into (B, CxT, H, W)), and convert the selection probability p output by the dynamic routing unit 120 into iAs the weight value of the corresponding subunit 131 in the predictor 130, (1-p i ) is used as the weight value for skipping the subunit 131, and weighted summation is performed to obtain the output of each subunit 131. In the predictor 130, after passing through multiple subunits 131, deep features of multiple frames of images are obtained, and deep features of future image frames are predicted.

[0100] Step 305: The decoder 140 reconstructs the image frame according to the deep features of the future image frame output by the predictor 130, outputs the corresponding predicted frame, and calculates the reconstruction loss L based on the real frame corresponding to the input sample frame and the predicted frame. recon .

[0101] Step 306: Based on the sparse sampling loss L sparse , reconstruction loss L recon Calculate the total loss function L total .

[0102] Step 307: Based on the total loss function L total , the gradient of the parameters is calculated using the back propagation algorithm, and the gradient is fed back to each layer of the dynamic routing video prediction model 100 to update the network weights so that each parameter is updated in the direction of reducing the total loss function. Among them, the Adam algorithm is used for end-to-end training, and the initial learning rate is set to 0.001.

[0103] like Figure 4 A schematic diagram of the inference process of the dynamic routing video prediction model 100 in the embodiment of the present disclosure is provided. Figure 4 As shown, it includes the following steps 401 to 405:

[0104] Step 401: input N consecutive original frames.

[0105] Step 402: concatenate the N original frames in the batch dimension, i.e., reshape the input from (B, T, C, H, W) to (BxT, C, H, W)), and input the input to the encoder 110 to extract the spatial features of each original frame.

[0106] Step 403: splice the spatial features of each original frame output by the encoder 110 in the channel dimension, that is, reshape the feature dimension (BxT, C, H, W) into (B, CxT, H, W) as the input of the dynamic routing unit 120, and calculate the selection probability p of each subunit 131 of the predictor 130 through the dynamic routing unit 120 i .

[0107] Step 404: splice the spatial features of each original frame output by the encoder 110 in the channel dimension, that is, reshape the feature dimension (BxT, C, H, W) into (B, CxT, H, W)), and convert the selection probability p output by the dynamic routing unit 120 into i As the weight value of the corresponding subunit 131 in the predictor 130, (1-p i ) is used as the weight value for skipping the subunit 131, and weighted summation is performed to obtain the output of each subunit 131. In the predictor 130, after passing through multiple subunits 131, deep features of multiple frames of images are obtained, and deep features of future image frames are predicted.

[0108] Step 405 : The decoder 140 reconstructs the image frame according to the deep features of the future image frame output by the predictor 130 , and outputs a corresponding predicted frame.

[0109] The present disclosure provides a video prediction method based on dynamic routing. The method can input at least two frames of original frames into a dynamic routing video prediction model to obtain at least two frames of predicted frames output by the dynamic routing video prediction model; specifically, in the dynamic routing video prediction model, the encoder is used to extract spatial features from the original frame, and the dynamic routing unit is used to output the selection probability corresponding to each sub-unit in the predictor according to the spatial features; the predictor determines the selected target sub-unit according to the selection probability, and uses the target sub-unit to fuse the spatial features output by the encoder to predict the deep features; the decoder reconstructs the image frame based on the deep features and outputs the predicted frame corresponding to the original frame. The present disclosure dynamically selects the sub-unit of the predictor participating in the feature calculation according to the output of the encoder by the dynamic routing unit, so that the depth of the dynamic routing video prediction model is adjustable, thereby flexibly adapting to the different performance requirements of simple and complex scenes, and effectively improving the calculation efficiency and accuracy; the simple model structure of the encoder, decoder and predictor is adopted, which can flexibly handle the prediction task of single or multiple frames of original frames, and effectively alleviate the error accumulation effect in the multi-frame prediction task.

[0110] The video prediction method based on dynamic routing provided in the embodiment of the present disclosure may also be executed by a video prediction device based on dynamic routing, or a control module in the video prediction device based on dynamic routing for executing the video prediction method based on dynamic routing. In the embodiment of the present disclosure, the video prediction device based on dynamic routing executing the video prediction method based on dynamic routing is taken as an example to illustrate the video prediction device based on dynamic routing provided in the embodiment of the present disclosure.

[0111] Figure 5 The structure diagram of the video prediction device 500 based on dynamic routing provided by the embodiment of the present disclosure is as follows: Figure 5As shown in, the device 500 may include a video prediction model based on dynamic routing, and the video prediction model based on dynamic routing includes an encoder 510, a predictor 530, a decoder 540 and a dynamic routing unit 520, and the predictor 530 includes more than two sub-units 531; the encoder 510 is used to obtain at least two frames of original frames as input, and to extract spatial features from each original frame; the dynamic routing unit 520 is used to output the selection probability corresponding to each sub-unit 531 in the predictor 530 according to the spatial features output by the encoder 510; the predictor 530 is used to determine the selected target sub-unit 531 in the sub-unit 531 according to the selection probability output by the dynamic routing unit 520, and use the target sub-unit 531 to fuse the spatial features output by the encoder 510 to predict deep features; the decoder 540 reconstructs the image frame based on the deep features, and outputs at least two predicted frames corresponding to the original frame.

[0112] In an optional embodiment of the present disclosure, the encoder 510 and the decoder 540 perform independent calculations on the channel dimension of each original frame.

[0113] In an optional embodiment of the present disclosure, the predictor 530 and the dynamic routing unit 520 are calculated in all channel dimensions.

[0114] In an optional embodiment of the present disclosure, a skip connection is adopted between the encoder 510 and the decoder 540 .

[0115] In an optional embodiment of the present disclosure, the encoder 510 is also used to retain the shallow features of each layer output when extracting the spatial features of each original frame; the decoder 540 is also used to obtain the shallow features through jump connections, and specifically to merge them with the deep features output by the predictor 530 to output a predicted frame.

[0116] In an optional embodiment of the present disclosure, the loss function used by the dynamic routing video prediction model during training includes reconstruction loss; the reconstruction loss is used to quantify the difference between the real frame corresponding to the sample frame received by the dynamic routing video prediction model during training and the output prediction frame.

[0117] In an optional embodiment of the present disclosure, the loss function used by the dynamic routing video prediction model during the training process also includes a sparse sampling loss; the sparse sampling loss is used to control the sparsity of the output of the dynamic routing unit 520 during the training process.

[0118] In an optional embodiment of the present disclosure, the number of original frames input into the dynamic routing video prediction model is the same as the number of predicted frames output by the dynamic routing video prediction model.

[0119] In the video prediction device based on dynamic routing provided in the present disclosure, at least two frames of original frames can be input into the dynamic routing video prediction model to obtain at least two frames of predicted frames output by the dynamic routing video prediction model; specifically, in the dynamic routing video prediction model, the encoder is used to extract spatial features from the original frame, and the dynamic routing unit is used to output the selection probability corresponding to each sub-unit in the predictor according to the spatial features; the predictor determines the selected target sub-unit according to the selection probability, and uses the target sub-unit to fuse the spatial features output by the encoder to predict the deep features; the decoder reconstructs the image frame based on the deep features and outputs the predicted frame corresponding to the original frame. In the present disclosure, the dynamic routing unit dynamically selects the sub-unit of the predictor that participates in the feature calculation according to the output of the encoder, so that the depth of the dynamic routing video prediction model is adjustable, thereby flexibly adapting to the different performance requirements of simple and complex scenes, and effectively improving the calculation efficiency and accuracy; the simple model structure of the encoder, decoder and predictor is adopted, which can flexibly handle the prediction task of single or multiple frames of original frames, and effectively alleviate the error accumulation effect in the multi-frame prediction task.

[0120] The video prediction device based on dynamic routing provided by the embodiment of the present disclosure can achieve Figures 1 to 4 To avoid repetition, the various processes implemented in the method embodiment will not be described again here.

[0121] Figure 6 A schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present disclosure is shown in FIG. Figure 6 As shown, the electronic device 600 may include a processor 601, a memory 602, and a program or instruction stored in the memory 602 and executable on the processor 601. When the program or instruction is executed by the processor 601, each process of the above-mentioned video prediction method embodiment based on dynamic routing is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0122] It should be noted that Figure 6 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0123] Figure 7 A hardware schematic diagram of an electronic device 700 provided in an embodiment of the present disclosure is shown in FIG. Figure 7As shown, the electronic device 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to the program stored in ROM (Read Only Memory) 702 or the program loaded from the storage part 708 to RAM (Random Access Memory) 703. Various programs and data required for system operation are also stored in RAM 703. CPU 701, ROM 702 and RAM 703 are connected to each other through bus 704. I / O (Input / Output) interface 705 is also connected to bus 704.

[0124] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a CRT (Cathode Ray Tube), an LCD (Liquid Crystal Display), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed so that a computer program read therefrom is installed into the storage section 708 as needed.

[0125] In particular, according to an embodiment of the present disclosure, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 709, and / or installed from a removable medium 711. When the computer program is executed by a central processing unit (CPU701), various functions defined in the system of the present application are executed.

[0126] The embodiment of the present disclosure also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned video prediction method embodiment based on dynamic routing is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0127] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as ROM, RAM, magnetic disk or optical disk.

[0128] The disclosed embodiment further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned video prediction method embodiment based on dynamic routing, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0129] It should be understood that the chip mentioned in the embodiments of the present disclosure may also be referred to as a system-level chip, a system chip, a chip system, or a system-on-chip chip, etc.

[0130] The embodiment of the present disclosure provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps of the video prediction method based on dynamic routing as described above, and can achieve the same technical effect. To avoid repetition, it will not be described here.

[0131] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiments of the present disclosure is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0132] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, an electronic device, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present disclosure.

[0133] The embodiments of the present disclosure are described above in conjunction with the accompanying drawings, but the present disclosure is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present disclosure, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present disclosure and the claims, all of which are within the protection of the present disclosure.

Claims

1. A video prediction method based on dynamic routing, characterized in that: The method is used for a dynamic routing video prediction model, the dynamic routing video prediction model includes an encoder, a predictor, a decoder and a dynamic routing unit, the predictor includes more than two subunits, and the method includes: Input at least two original frames into the dynamic routing video prediction model to obtain at least two predicted frames output by the dynamic routing video prediction model; wherein, In the dynamic routing video prediction model, the encoder is used to extract spatial features from each of the original frames; the dynamic routing unit is used to output the selection probability corresponding to each of the sub-units in the predictor according to the spatial features output by the encoder; the predictor is used to determine the selected target sub-unit in the sub-units according to the selection probability output by the dynamic routing unit, and use the target sub-unit to fuse the spatial features output by the encoder to predict deep features; the decoder reconstructs the image frame based on the deep features and outputs at least two predicted frames corresponding to the original frame.

2. The method according to claim 1, characterized in that The encoder and the decoder independently calculate on the channel dimension of each of the original frames; The predictor and the dynamic routing unit are computed over all of the channel dimensions.

3. The method according to claim 1, characterized in that A skip connection is adopted between the encoder and the decoder.

4. The method according to claim 3, characterized in that The encoder is also used to retain the shallow features of each layer output when extracting the spatial features of each of the original frames; The decoder is further used to obtain the shallow features through skip connections, and is specifically used to merge the shallow features with the deep features output by the predictor to output the predicted frame.

5. The method according to claim 1, characterized in that The loss function used in the training process of the dynamic routing video prediction model includes reconstruction loss; The reconstruction loss is used to quantify the difference between the real frame corresponding to the sample frame received by the dynamic routing video prediction model during the training process and the output predicted frame.

6. The method according to claim 5, characterized in that The loss function used by the dynamic routing video prediction model during training also includes sparse sampling loss; The sparse sampling loss is used to control the sparsity of the dynamic routing unit output during training.

7. The method according to claim 1, characterized in that The number of the original frames input into the dynamic routing video prediction model is the same as the number of the predicted frames output by the dynamic routing video prediction model.

8. A video prediction device based on dynamic routing, characterized in that: The device comprises a video prediction model based on dynamic routing, wherein the video prediction model based on dynamic routing comprises an encoder, a predictor, a decoder and a dynamic routing unit, wherein the predictor comprises more than two subunits; The encoder is used to obtain at least two original frames as input, and extract spatial features from each of the original frames; The dynamic routing unit is used to output the selection probability corresponding to each of the sub-units in the predictor according to the spatial features output by the encoder; The predictor is used to determine a selected target sub-unit among the sub-units according to the selection probability output by the dynamic routing unit, and to use the target sub-unit to perform fusion processing on the spatial features output by the encoder to predict deep features; The decoder reconstructs an image frame based on the deep features and outputs at least two predicted frames corresponding to the original frame.

9. An electronic device, characterized in that: The method comprises a processor, a memory and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the video prediction method based on dynamic routing as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by the processor, the video prediction method based on dynamic routing as described in any one of claims 1 to 7 is implemented.

Images