A video prediction method and system

Abstract

The embodiment of the present application discloses a video prediction method and system, the method comprising: inputting a known frame in a video into an encoder network to obtain a semantic feature map of the known frame; using a recurrent neural network to perform spatiotemporal modeling on the semantic feature map, predicting the semantic feature map corresponding to the video frame at the next moment in the feature space; inputting the predicted semantic feature map into a decoder network to obtain the video frame image at the next moment in the pixel space. The accuracy and clarity of video prediction are significantly improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of machine vision technology, and specifically to a video prediction method and system. Background Art

[0002] In recent years, with the rapid development of computer vision, analysis technology based on single images has become increasingly mature, and more and more attention has shifted to the analysis and understanding of videos. Video prediction refers to predicting subsequent frames based on the previous frames as known conditions in video analysis. Video prediction has a wide range of applications in real life. For example, in autonomous driving, video prediction methods can help vehicles predict the road conditions at the next moment so that different strategies can be adopted in advance. In addition, video prediction can also be used for more general spatiotemporal prediction methods, such as weather forecasting, urban road condition forecasting, and so on.

[0003] The problem of video prediction is unique. Unlike the problem of image generation, video prediction needs to be modeled in the time domain and all known information in the past needs to be integrated to make predictions. Unlike video classification or action recognition, video prediction needs to consider the spatial distribution of the predicted frame at the same time, which greatly increases the difficulty. On the one hand, the motion information of the pixels in the video is crucial. However, the existing technology does not consider such motion information, or uses optical flow as a supplement to the motion information. The former is not conducive to the algorithm's accurate modeling of time domain information, while the latter requires a huge amount of computing resources. On the other hand, for video prediction, adjacent contexts are more closely associated with the predicted frame. However, the existing technology often does not strengthen this association of contextual information, making the algorithm less sensitive to some sudden changes in the time domain.

[0004] A series of defects in the video prediction methods in the prior art have resulted in video prediction being unable to be widely used in production and life due to its poor effect. Therefore, a method with low overhead and good effect is needed to model motion information and improve the accuracy of video prediction. Summary of the invention

[0005] To this end, the embodiments of the present application provide a video prediction method and system to significantly improve the accuracy and clarity of video prediction.

[0006] In order to achieve the above objectives, the embodiments of the present application provide the following technical solutions:

[0007] According to a first aspect of an embodiment of the present application, a video prediction method is provided, the method comprising:

[0008] Input the known frames in the video into the encoder network to obtain the semantic feature map of the known frames;

[0009] Using a recurrent neural network to model the semantic feature map in the spatiotemporal domain, and predicting the semantic feature map corresponding to the video frame at the next moment in the feature space;

[0010] The predicted semantic feature map is input into the decoder network to obtain the video frame image at the next moment in the pixel space.

[0011] Optionally, the method further comprises:

[0012] The known frames in the video frame sequence are input one by one into the encoder network, the recurrent neural network, and the decoder network to predict the video frame image at the next moment, and the square of the difference between the predicted video frame image and the real image at the corresponding moment of the predicted frame image is used as the loss function to train the encoder network, the recurrent neural network, and the decoder network until convergence.

[0013] Optionally, the using of a recurrent neural network to perform spatiotemporal modeling on the semantic feature map to predict the semantic feature map corresponding to the video frame at the next moment in the feature space includes:

[0014] Inputting the semantic feature map into a motion information enhancement module and a context information enhancement module simultaneously;

[0015] The motion information enhancement module amplifies the motion information between the previous and next frames of the semantic feature map to obtain a motion information enhanced feature map consistent with the original input size;

[0016] The context information enhancement module extracts features from context information of the semantic feature map including the current frame and the previous set range frame to obtain a context information enhanced feature map;

[0017] Splicing the motion information enhanced feature map and the context information enhanced feature map in the channel dimension;

[0018] The spliced ​​feature map is input into the time domain information modeling module for modeling in the time domain, and the time domain information is used to predict the semantic information of the video frame at the next moment, so as to predict the semantic feature map corresponding to the video frame at the next moment.

[0019] Optionally, the motion information enhancement module amplifies the motion information between the previous and next frames of the semantic feature map to obtain a motion information enhanced feature map having the same size as the original input, including:

[0020] Perform difference processing on the semantic feature map of the video frame at the current moment and the previous moment to obtain a difference feature map with the same size as the original input;

[0021] Inputting the difference feature map into a convolutional layer to reduce the number of channels;

[0022] Perform global mean pooling on each channel in the difference feature map after reducing the number of channels to obtain a feature vector with a length equal to the number of channels;

[0023] Input the feature vector into the fully connected layer and the activation layer so that the feature vector length matches the number of channels of the original input;

[0024] Multiply the output feature vector by the original semantic feature map at the current moment to obtain an enhanced feature map;

[0025] The original semantic feature map and the enhanced feature map are added together to obtain the motion information enhanced feature map.

[0026] Optionally, the context information enhancement module extracts features from context information of the semantic feature map including the current frame and previous set range frames to obtain a context information enhanced feature map, including:

[0027] Splicing the feature map of the current frame of the semantic feature map with the feature map of the previous set range frame in the target time dimension;

[0028] The concatenated feature maps are input into the first behavior recognition model to perform spatial convolution in three dimensions: channel, width, and height, and then input into the second behavior recognition model to combine different feature maps in the time dimension to obtain a combined feature map;

[0029] The combined feature maps at different times are multiplied by their respective weights to obtain the context information enhanced feature map.

[0030] Optionally, inputting a known frame in the video into an encoder network to obtain a semantic feature map of the known frame includes:

[0031] The encoder network converts known frames in the video into feature maps with set side lengths and set number of channels;

[0032] The feature map is input into a convolutional layer, a normalization layer, and an activation layer of a first number of layers to obtain a semantic feature map of a known frame.

[0033] Optionally, inputting the predicted semantic feature map into a decoder network to obtain a video frame image at the next moment in the pixel space includes:

[0034] The decoder network restores the number of channels of the predicted semantic feature map to the set number of channels;

[0035] The predicted semantic feature map after restoring the number of channels is input into the convolution layer, the normalization layer and the activation layer of the first number of layers to obtain the video frame image at the next moment in the pixel space.

[0036] According to a second aspect of an embodiment of the present application, a video prediction system is provided, the system comprising:

[0037] The encoding module is used to input the known frames in the video into the encoder network to obtain the semantic feature map of the known frames;

[0038] A prediction module is used to perform spatiotemporal modeling on the semantic feature map using a recurrent neural network, and predict the semantic feature map corresponding to the video frame at the next moment in the feature space;

[0039] The decoding module is used to input the predicted semantic feature map into the decoder network to obtain the video frame image at the next moment in the pixel space.

[0040] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0041] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer-readable instructions are stored. The computer-readable instructions can be executed by a processor to implement the method described in the first aspect above.

[0042] In summary, the embodiment of the present application provides a video prediction method and system, which obtains the semantic feature map of the known frame by inputting the known frame in the video into the encoder network; uses the recurrent neural network to perform spatiotemporal modeling on the semantic feature map, and predicts the semantic feature map corresponding to the video frame at the next moment in the feature space; inputs the predicted semantic feature map into the decoder network to obtain the video frame image at the next moment in the pixel space. The accuracy and clarity of video prediction are significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0044] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0045] Figure 1A schematic diagram of a video prediction method flow chart provided in an embodiment of the present application;

[0046] Figure 2 A schematic diagram of the encoder network structure provided in an embodiment of the present application;

[0047] Figure 3 A schematic diagram of the architecture of a recurrent neural network provided in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of a decoder network structure provided in an embodiment of the present application;

[0049] Figure 5 A block diagram of a video prediction system provided in an embodiment of the present application;

[0050] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown;

[0051] Figure 7 A schematic diagram of a computer-readable storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0052] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] Figure 1 The video prediction method provided by the embodiment of the present application is shown, which enhances the motion information and context information of the video based on the recurrent neural network and the incentive mechanism, and significantly improves the accuracy and clarity of the video prediction. Specifically, it includes the following steps:

[0054] Step 101: input a known frame in the video into an encoder network to obtain a semantic feature map of the known frame;

[0055] Step 102: Using a recurrent neural network to perform spatiotemporal modeling on the semantic feature map, and predicting the semantic feature map corresponding to the video frame at the next moment in the feature space;

[0056] Step 103: Input the predicted semantic feature map into the decoder network to obtain the video frame image at the next moment in the pixel space.

[0057] In a possible implementation, the method further includes: inputting known frames in the video frame sequence into the encoder network, the recurrent neural network, and the decoder network one by one to predict the video frame image at the next moment, and using the square of the difference between the predicted video frame image and the real image at the corresponding moment of the predicted frame image as a loss function to train the encoder network, the recurrent neural network, and the decoder network until convergence.

[0058] In a possible implementation, in step 102, the using of a recurrent neural network to perform spatiotemporal modeling on the semantic feature map, and predicting the semantic feature map corresponding to the video frame at the next moment in the feature space, includes:

[0059] The semantic feature map is simultaneously input into a motion information enhancement module and a context information enhancement module; the motion information enhancement module amplifies the motion information between the previous and next frames of the semantic feature map to obtain a motion information enhanced feature map that is consistent with the original input size; the context information enhancement module extracts features from the context information of the semantic feature map, including the current frame and the previous set range frames, to obtain a context information enhanced feature map; the motion information enhanced feature map and the context information enhanced feature map are spliced ​​in the channel dimension; the spliced ​​feature map is input into a time domain information modeling module for modeling in the time domain, and the time domain information is used to predict the semantic information of the video frame at the next moment, so as to predict the semantic feature map corresponding to the video frame at the next moment.

[0060] In a possible implementation, the motion information enhancement module amplifies the motion information between the previous and next frames of the semantic feature map to obtain a motion information enhanced feature map consistent with the original input size, including:

[0061] The semantic feature maps of the video frames at the current moment and the previous moment are subtracted to obtain a difference feature map with the same size as the original input; the difference feature map is input into the convolution layer to reduce the number of channels; each channel in the difference feature map after the number of channels is reduced is globally mean pooled to obtain a feature vector with a length of the number of channels; the feature vector is input into the fully connected layer and the activation layer to match the length of the feature vector with the number of channels of the original input; the output feature vector is multiplied by the original semantic feature map at the current moment to obtain an enhanced feature map; the original semantic feature map and the enhanced feature map are added to obtain a motion information enhanced feature map.

[0062] In a possible implementation, the context information enhancement module extracts features from context information of the semantic feature map including the current frame and the previous set range frame to obtain a context information enhanced feature map, including:

[0063] The feature map of the current frame of the semantic feature map and the feature map of the previous set range frame are spliced ​​in the target time dimension; the spliced ​​feature map is input into the first behavior recognition model to perform spatial convolution in three dimensions of channel, width and height, and then input into the second behavior recognition model to combine different feature maps in the time dimension to obtain a combined feature map; the combined feature maps at different times are added with the product of their respective weights to obtain the context information enhanced feature map.

[0064] In a possible implementation, inputting a known frame in a video into an encoder network to obtain a semantic feature map of the known frame includes:

[0065] The encoder network converts a known frame in a video into a feature map with a set side length and a set number of channels; the feature map is input into a first number of convolutional layers, a normalization layer, and an activation layer to obtain a semantic feature map of the known frame.

[0066] In a possible implementation, inputting the predicted semantic feature map into a decoder network to obtain a video frame image at the next moment in the pixel space includes:

[0067] The decoder network restores the number of channels of the predicted semantic feature map to the set number of channels; the predicted semantic feature map after the number of channels is restored is input into the convolution layer, the normalization layer and the activation layer of the first number of layers to obtain the video frame image at the next moment in the pixel space.

[0068] It can be seen that the video prediction method based on the recurrent neural network and the excitation mechanism first inputs the video frame by frame into the encoder network to extract the spatial semantic information and obtain the feature map of each frame; then inputs the feature map of each frame into the recurrent neural network combined with the excitation mechanism to comprehensively model the information in the spatiotemporal domain, and predicts the feature map of the next moment in this feature space; finally, the predicted feature map is mapped back to the pixel space from the feature space through the decoder network to obtain the predicted frame. It can effectively solve the problem of insufficient consideration of motion information and insensitivity to sudden changes in temporal domain modeling, significantly improve the accuracy of video prediction, and improve the quality of generated images. On the one hand, considering the change relationship between the previous and next frames, the development trend of the video frames before and after is effectively captured, and the law of video frame changes is more accurately grasped. On the other hand, the relationship between adjacent frames and the predicted frames is emphasized, so that the model is more conducive to capturing the shape and motion law of objects in the video frames, and the predicted images are clearer.

[0069] The video prediction method provided in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0070] First, it is necessary to build a video prediction network, which includes an encoder network, a recurrent neural network and a decoder network.

[0071] The encoder network aims to project the image from the pixel space to a high-dimensional feature space, and the temporal changes of the video in this high-dimensional feature space can be simpler and more efficient.

[0072] In one possible implementation, Figure 2 The schematic diagram of the encoder network structure provided by the embodiment of the present application is shown, and the encoder network is composed of 5 layers of 3x3 convolution, of which 2 layers have a step size of 2 and the other 3 layers have a step size of 1. The specific structure is as follows:

[0073] (1) 2D convolution, where the convolution kernel size is 3x3, the stride is 2, the number of input channels is 1, and the number of output channels is 32.

[0074] (2) 2D convolution, where the convolution kernel size is 3x3, the stride is 1, the number of input channels is 32, and the number of output channels is 32.

[0075] (3) 2D convolution, where the convolution kernel size is 3x3, the stride is 2, the number of input channels is 32, and the number of output channels is 64.

[0076] (4) 2D convolution, where the convolution kernel size is 3x3, the stride is 1, the number of input channels is 64, and the number of output channels is 64.

[0077] (5) 2D convolution, where the convolution kernel size is 3x3, the stride is 1, the number of input channels is 64, and the number of output channels is 64.

[0078] Each 2D convolution is followed by a normalization layer GroupNorm and an activation layer LeakyRelu.

[0079] The encoder network converts the original grayscale image into a feature map with a side length of only 1 / 4 of the original image and a channel number of 64. After the image passes through the 5 layers of 2D convolution and the subsequent normalization layer and activation layer, the spatial scale of the image changes from the original 64x64 to 16x16, which is reduced by 4 times; at the same time, the number of channels increases from 1 to 64. Thus, an abstract semantic feature map covering the entire image is obtained at each position.

[0080] In a possible implementation, each layer of the recurrent neural network part includes three modules: a motion information enhancement module, a context information enhancement module, and a time domain information modeling RNN module. The feature map first enters the Motion Excitation module and the Context Excitation module at the same time, and the outputs of the two modules are concatenated as the input of the RNN module, which outputs the final result.

[0081] The Motion Excitation module is used to enhance the motion information between video frames. The Context Excitation module is used to enhance and extract the context information of the current frame. The RNN module combines the outputs of the previous two modules to model the temporal information of the video.

[0082] Figure 3 The construction process of each layer in the recurrent neural network part is shown, and the specific steps are as follows:

[0083] Step 1: Use the Motion Excitation module to process the feature map input to this layer, amplify the motion information between the previous and next frames, and obtain a feature map with the same size as the original input but enhanced motion information;

[0084] Step 2: Use the Context Excitation module to process the original input of this layer, extract features from the context of the current frame and the previous 5 frames, and obtain a feature map with enhanced context information;

[0085] Step 3: Concatenate the two feature maps obtained in the previous step in the channel dimension;

[0086] Step 4: Input the concatenated feature map into the RNN module for time domain modeling, and use the time domain information to infer the high-level semantic information of the next frame in order to predict the frame.

[0087] In the Motion Excitation module of the recurrent neural network, a form of excitation is adopted. The specific steps are as follows:

[0088] Step 1: For the current feature map X t ∈R c×h×w And the feature map X at the previous moment t-1 ∈R c×h×w , and the difference between the two is X t-1 ∈R c×h×w That is, the feature map of the current moment and the feature map of the previous moment are subtracted to obtain a feature map of the same size as the input, which serves as the carrier of motion information.

[0089] Step 2: Send the difference D to 1x1 convolution to reduce the number of channels to 1 / 4 of the original, and get S = Conv 1x1 (D),S∈R e / 4×h×w . In order to simplify subsequent calculations and ensure better generalization.

[0090] Step 3: Perform 2D global pooling on S, and set its width and height in space to 1, that is, take the average of the 2D feature maps on each channel to obtain a vector E∈Rc / 4×1×1 That is, a global mean pooling operation is performed on each channel of the difference, and the length and width in the spatial dimension are set to 1, thereby obtaining a vector with a length equal to the number of channels.

[0091] Step 4: Put the vector E into the fully connected layer to generate a set of weights W∈R of length c c×1×1 That is, the vector is input into a fully connected layer and an activation layer to increase the vector length by 4 times, so that the vector length matches the number of channels of the original input.

[0092] Step 5: Multiply weight W back to X t Used for X t The different channels in the image are enhanced, and the channels related to motion have a larger weight; finally, the product is multiplied by the original X t Add to prevent information loss and get the final result Y t =X t ×E+X t , Y t ∈R c×h×w That is, the vector is multiplied by the original feature map at the current moment to obtain the enhanced feature map; finally, the original feature map and the enhanced feature map are added.

[0093] In the Context Excitation module, the feature map input at the current moment and the feature map input at the previous 5 moments are first concatenated into a 4D tensor in a new dimension, and then the tensor is input into a 2-layer R(2+1)D module with the same structure. The Context Excitation module consists of 2 action recognition model R(2+1)D modules, each of which has 2 layers of 3x3 spatial convolutions with a step size of 1, and 2 layers of temporal convolutions with a kernel size of 3, also with a step size of 1.

[0094] The Context Excitation module combines 2D spatial domain convolution with 1D temporal domain convolution. The specific steps are as follows:

[0095] Step 1: Take the feature graph X of the current moment and the previous 5 frames t , X t-1 , …, X t-5 ∈R c×h×w Put them together in the new time dimension and get U∈R 6×c×h×w ;

[0096] Step 2: Send U to two R(2+1)D modules in succession. In each R(2+1)D module, a 3x3 spatial convolution is performed first, which operates on the three dimensions of channel, width and height without changing the size of any dimension; followed by a temporal convolution with a convolution kernel of 3, which combines different feature maps in the temporal dimension without changing the size of the temporal dimension. The feature map obtained after two R(2+1)D modules is v∈R 6×c×h×w ;

[0097] Step 3: Send V to a spatial convolution with a convolution kernel size of 6 to reduce its time scale 6 to 1, that is, reduce the dimension to a 3D tensor Z∈R c×h×w This operation can also be understood as giving different weights to feature maps at different times, multiplying them by the weights and adding them together to obtain the result after dimensionality reduction.

[0098] Before being fed into the RNN structure, the embodiment of the present application proposes that the outputs of motion excitation and context excitation are combined in the channel dimension, that is, T = [Y t , Zl, T∈R 2c×h×w .

[0099] The RNN structure adopted in the embodiment of the present application is a ConvLSTM network, which can simultaneously perform spatial and temporal modeling on a 2D feature map.

[0100] Figure 4 The decoder network structure diagram provided in the embodiment of the present application is shown, which is specifically composed of 5 layers of 3x3 deconvolution, of which 2 layers have a step size of 2 and the other 3 layers have a step size of 1. The decoder network is composed as follows:

[0101] (1) 2D deconvolution, where the convolution kernel size is 3x3, the stride is 1, the number of input channels is 64, and the number of output channels is 64.

[0102] (2) 2D deconvolution, where the convolution kernel size is 3x3, the stride is 1, the number of input channels is 64, and the number of output channels is 64.

[0103] (3) 2D deconvolution, where the convolution kernel size is 3x3, the stride is 2, the number of input channels is 64, and the number of output channels is 32.

[0104] (4) 2D deconvolution, where the convolution kernel size is 3x3, the stride is 1, the number of input channels is 32, and the number of output channels is 32.

[0105] (5) 2D deconvolution, where the convolution kernel size is 3x3, the stride is 2, the number of input channels is 32, and the number of output channels is 1.

[0106] Each 2D convolution is followed by a normalization layer GroupNorm and an activation layer LeakyRelu.

[0107] The decoder network restores the feature map with 64 channels to a grayscale image with 1 channel. After the image passes through the 5 layers of 2D deconvolution and the subsequent normalization layer and activation layer, the spatial scale is increased by 4 times from the original 16x16 to 64x64; at the same time, the number of channels is increased from 64 to 1, and the predicted video frame is obtained.

[0108] The loss function used in the embodiment of the present application is the MSE mean square error loss function. Specifically, for the predicted The actual next frame X t+1 The loss is calculated as Where m is the width and n is the height.

[0109] Specifically applied in video prediction, the known frames in the video are input into the encoder network to obtain high-level semantic features of the known frames; the high-level semantic features obtained are modeled in the spatiotemporal domain using a recurrent neural network, and the feature map corresponding to the video frame at the next moment is predicted; the feature map is further input into the decoder network to obtain the predicted image; the video sequence is input into the encoder network, the recurrent neural network, and the decoder network one by one to predict the image at the next moment, and the square of the difference between the predicted image and the real image is used as the loss function to train the network until convergence to obtain the final model.

[0110] In summary, the embodiment of the present application provides a video prediction method, which obtains a semantic feature map of the known frame by inputting a known frame in the video into an encoder network; performs spatiotemporal modeling on the semantic feature map using a recurrent neural network, and predicts the semantic feature map corresponding to the video frame at the next moment in the feature space; and inputs the predicted semantic feature map into a decoder network to obtain the video frame image at the next moment in the pixel space. The accuracy and clarity of video prediction are significantly improved.

[0111] Based on the same technical concept, the embodiment of the present application also provides a video prediction system, such as Figure 5 As shown, the system comprises:

[0112] The encoding module 501 is used to input a known frame in the video into an encoder network to obtain a semantic feature map of the known frame;

[0113] A prediction module 502 is used to perform spatiotemporal modeling on the semantic feature map using a recurrent neural network, and predict the semantic feature map corresponding to the video frame at the next moment in the feature space;

[0114] The decoding module 503 is used to input the predicted semantic feature map into the decoder network to obtain the video frame image at the next moment in the pixel space.

[0115] The present application also provides an electronic device corresponding to the method provided in the above embodiment. Figure 6 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. The electronic device 20 may include: a processor 200, a memory 201, a bus 202 and a communication interface 203, wherein the processor 200, the communication interface 203 and the memory 201 are connected via the bus 202; the memory 201 stores a computer program that can be run on the processor 200, and the processor 200 executes the method provided by any of the aforementioned embodiments of the present application when running the computer program.

[0116] The memory 201 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one physical port 203 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0117] The bus 202 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 201 is used to store programs, and the processor 200 executes the programs after receiving execution instructions. The method disclosed in any implementation of the aforementioned embodiment of the present application may be applied to the processor 200, or implemented by the processor 200.

[0118] The processor 200 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 200. The above processor 200 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 201, and the processor 200 reads the information in the memory 201 and completes the steps of the above method in combination with its hardware.

[0119] The electronic device provided in the embodiment of the present application and the method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, operated or implemented by them.

[0120] The present application also provides a computer-readable storage medium corresponding to the method provided in the above embodiment. Figure 7 The computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the method provided in any of the aforementioned embodiments is executed.

[0121] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0122] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0123] It should be noted that:

[0124] The algorithm and display provided herein are not inherently related to any particular computer, virtual device or other equipment. Various general purpose devices can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of device. In addition, the application is not directed to any specific programming language. It should be understood that the content of the application described herein can be realized using various programming languages, and the description of the specific language above is to disclose the best mode of implementation of the application.

[0125] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0126] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0127] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0128] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0129] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components in the creation device of the virtual machine according to the embodiment of the present application. The application can also be implemented as a device or device program (e.g., computer program and computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0130] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

[0131] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A video prediction method, characterized in that: The method comprises: Input the known frames in the video into the encoder network to obtain the semantic feature map of the known frames; Using a recurrent neural network to model the semantic feature map in the spatiotemporal domain, and predicting the semantic feature map corresponding to the video frame at the next moment in the feature space; The predicted semantic feature map is input into the decoder network to obtain the video frame image at the next moment in the pixel space; The method of using a recurrent neural network to model the semantic feature map in the spatiotemporal domain and predicting the semantic feature map corresponding to the video frame at the next moment in the feature space includes: The semantic feature map is simultaneously input into a motion information enhancement module and a context information enhancement module; the motion information enhancement module amplifies the motion information between the previous and next frames of the semantic feature map to obtain a motion information enhanced feature map that is consistent with the original input size; the context information enhancement module extracts features from the context information of the semantic feature map, including the current frame and the previous set range frames, to obtain a context information enhanced feature map; the motion information enhanced feature map and the context information enhanced feature map are spliced ​​in the channel dimension; the spliced ​​feature map is input into a time domain information modeling module for modeling in the time domain, and the time domain information is used to predict the semantic information of the video frame at the next moment, so as to predict the semantic feature map corresponding to the video frame at the next moment.

2. The method according to claim 1, characterized in that The method further comprises: The known frames in the video frame sequence are input one by one into the encoder network, the recurrent neural network, and the decoder network to predict the video frame image at the next moment, and the square of the difference between the predicted video frame image and the real image at the corresponding moment of the predicted frame image is used as the loss function to train the encoder network, the recurrent neural network, and the decoder network until convergence.

3. The method according to claim 1, characterized in that The motion information enhancement module amplifies the motion information between the previous and next frames of the semantic feature map to obtain a motion information enhanced feature map consistent with the original input size, including: Perform difference processing on the semantic feature map of the video frame at the current moment and the previous moment to obtain a difference feature map with the same size as the original input; Inputting the difference feature map into a convolutional layer to reduce the number of channels; Perform global mean pooling on each channel in the difference feature map after reducing the number of channels to obtain a feature vector with a length equal to the number of channels; Input the feature vector into the fully connected layer and the activation layer so that the feature vector length matches the number of channels of the original input; Multiply the output feature vector by the original semantic feature map at the current moment to obtain an enhanced feature map; The original semantic feature map and the enhanced feature map are added together to obtain the motion information enhanced feature map.

4. The method according to claim 1, characterized in that The context information enhancement module extracts features from context information of the semantic feature map including the current frame and the previous set range frame to obtain a context information enhanced feature map, including: Splicing the feature map of the current frame of the semantic feature map with the feature map of the previous set range frame in the target time dimension; The concatenated feature maps are input into the first behavior recognition model to perform spatial convolution in three dimensions: channel, width, and height, and then input into the second behavior recognition model to combine different feature maps in the time dimension to obtain a combined feature map; The combined feature maps at different times are multiplied by their respective weights to obtain the context information enhanced feature map.

5. The method according to claim 1, characterized in that The step of inputting a known frame in the video into the encoder network to obtain a semantic feature map of the known frame includes: The encoder network converts known frames in the video into feature maps with set side lengths and set number of channels; The feature map is input into a convolutional layer, a normalization layer, and an activation layer of a first number of layers to obtain a semantic feature map of a known frame.

6. The method according to claim 1, characterized in that The predicted semantic feature map is input into the decoder network to obtain the video frame image at the next moment in the pixel space, including: The decoder network restores the number of channels of the predicted semantic feature map to the set number of channels; The predicted semantic feature map after restoring the number of channels is input into the convolution layer, normalization layer and activation layer of the first number of layers to obtain the video frame image at the next moment in the pixel space.

7. A video prediction system, characterized in that: The system comprises: The encoding module is used to input the known frames in the video into the encoder network to obtain the semantic feature map of the known frames; A prediction module is used to perform spatiotemporal modeling on the semantic feature map using a recurrent neural network, and predict the semantic feature map corresponding to the video frame at the next moment in the feature space; A decoding module is used to input the predicted semantic feature map into a decoder network to obtain a video frame image at the next moment in the pixel space; The prediction module is specifically used to: input the semantic feature map into a motion information enhancement module and a context information enhancement module at the same time; the motion information enhancement module amplifies the motion information between the previous and next frames of the semantic feature map to obtain a motion information enhanced feature map that is consistent with the original input size; the context information enhancement module extracts features from the context information of the semantic feature map, including the current frame and the previous set range frames, to obtain a context information enhanced feature map; splices the motion information enhanced feature map and the context information enhanced feature map in the channel dimension; inputs the spliced ​​feature map into the time domain information modeling module for modeling in the time domain, and uses the time domain information to predict the semantic information of the video frame at the next moment, so as to predict the semantic feature map corresponding to the video frame at the next moment.

8. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and the computer-readable instructions can be executed by a processor to implement the method according to any one of claims 1-6.

Images