Video prediction method, device, computer equipment and storage medium

Abstract

The present application relates to a video prediction method, device, computer equipment and storage medium. The method comprises: obtaining a historical optical flow information sequence corresponding to a historical video; predicting according to the historical optical flow information sequence to obtain a predicted optical flow information sequence; determining the first spatiotemporal information corresponding to the predicted video according to the predicted optical flow information sequence; predicting according to the first spatiotemporal information and the historical video to obtain a plurality of predicted images, and forming the predicted video from the plurality of predicted images. The use of this method can improve the accuracy of video prediction.

Description

Technical Field

[0001] The present application relates to the field of image prediction technology, and in particular to a video prediction method, apparatus, computer equipment and storage medium. Background Art

[0002] Since videos can provide rich visual information, more and more information will be presented in the form of videos. With the development of computer technology and image processing technology, video prediction technology has emerged. Applying video prediction technology to multiple fields such as autonomous driving and weather forecasting can greatly facilitate people's work and life.

[0003] In the related art, video prediction technology usually uses the optical flow method to extrapolate historical video images to obtain future video images. However, there is a problem of inaccurate prediction when using the optical flow method to perform image extrapolation. Summary of the invention

[0004] Based on this, it is necessary to provide a video prediction method, device, computer equipment and storage medium that can improve prediction accuracy in response to the above technical problems.

[0005] A video prediction method, the method comprising:

[0006] Obtain a historical optical flow information sequence corresponding to a historical video; a historical video is composed of N historical images arranged in chronological order, and a historical optical flow information sequence includes N-1 historical optical flow information, and the historical optical flow information represents the optical flow change characteristics between a historical image and a previous historical image;

[0007] Predictions are made based on the historical optical flow information sequence to obtain a predicted optical flow information sequence; the predicted optical flow information sequence includes a plurality of predicted optical flow information, and the predicted optical flow information represents the optical flow change characteristics between each predicted image and the previous image;

[0008] Determine first spatiotemporal information corresponding to the predicted video according to the predicted optical flow information sequence; the first spatiotemporal information includes temporal features and spatial features corresponding to each predicted image;

[0009] Prediction is performed based on the first spatiotemporal information and the historical video to obtain a plurality of predicted images, and a predicted video is composed of the plurality of predicted images.

[0010] In one embodiment, the step of determining the first spatiotemporal information corresponding to the predicted video according to the predicted optical flow information sequence includes:

[0011] Deformation processing is performed according to the predicted optical flow information sequence and the Nth historical image to obtain image content information corresponding to each predicted image;

[0012] The image content information corresponding to each predicted image is input into a pre-trained coding network to obtain the temporal features and spatial features corresponding to each predicted image output by the coding network, and the temporal features and spatial features corresponding to multiple predicted images form the first spatiotemporal information.

[0013] In one embodiment, the deformation processing is performed on the predicted optical flow information sequence and the Nth historical image to obtain the image content information corresponding to each predicted image, including:

[0014] Deform the Nth historical image according to the first predicted optical flow information to obtain the image content information corresponding to the first predicted image;

[0015] The i-1th predicted image is deformed according to the i-th predicted optical flow information to obtain image content information corresponding to the i-th predicted image; wherein i is a positive integer greater than 1.

[0016] In one embodiment, the multiple predicted images are obtained by predicting based on the first spatiotemporal information and the historical video, including:

[0017] The historical video and the first spatiotemporal information are input into a video prediction network, and the video prediction network is used to extract features of the historical video. A plurality of predicted images are predicted based on the extracted second spatiotemporal information and the first spatiotemporal information; the second spatiotemporal information includes the time features and spatial features corresponding to each historical image.

[0018] In one embodiment, the video prediction network includes a feature extraction layer, and the feature extraction of the historical video using the video prediction network includes:

[0019] N historical images are respectively input into the feature extraction layer to obtain the time features and space features corresponding to each historical image output by the feature extraction layer, and the time features and space features corresponding to the multiple historical images form the second spatiotemporal information.

[0020] In one embodiment, the video prediction network includes a video prediction layer, and the multiple predicted images are obtained by predicting based on the extracted second spatiotemporal information and the first spatiotemporal information, including:

[0021] Inputting the first spatiotemporal information and the second spatiotemporal information into the video prediction layer for feature processing to obtain processed spatiotemporal information;

[0022] Prediction is performed based on the processed spatiotemporal information to obtain multiple predicted images.

[0023] In one embodiment, the first spatiotemporal information and the second spatiotemporal information are input into the video prediction layer for feature processing to obtain the processed spatiotemporal information, including:

[0024] The first spatiotemporal information and the second spatiotemporal information are input into the video prediction layer, and the video prediction layer performs splicing or replacement processing on the first spatiotemporal information and the second spatiotemporal information, and outputs the processed spatiotemporal information.

[0025] In one embodiment, before inputting both the historical video and the first spatiotemporal information into the video prediction network, the method further includes:

[0026] Obtaining a training sample set; the training sample set includes a plurality of historical sample images, a plurality of predicted sample images and sample spatiotemporal information;

[0027] The neural network is trained based on the training sample set, and a preset loss function is used to determine whether the training result output by the neural network meets the preset convergence condition; the preset loss function is a mean square error loss function;

[0028] When the training results meet the preset convergence conditions, the training is terminated to obtain a video prediction network.

[0029] In one embodiment, the above-mentioned prediction based on the historical optical flow information sequence to obtain the predicted optical flow information sequence includes:

[0030] The historical optical flow information sequence is input into the pre-trained optical flow prediction network to obtain the predicted optical flow information sequence output by the optical flow prediction network.

[0031] In one embodiment, the above-mentioned obtaining the historical optical flow information sequence corresponding to the historical video includes:

[0032] Every two adjacent historical images are input into a preset optical flow calculation model to obtain a plurality of historical optical flow information calculated by the optical flow calculation model, and the plurality of historical optical flow information constitute a historical optical flow information sequence.

[0033] A video prediction device, comprising:

[0034] The optical flow acquisition module is used to obtain the historical optical flow information sequence corresponding to the historical video; the historical video is composed of N historical images arranged in chronological order, and the historical optical flow information sequence includes N-1 historical optical flow information, and the historical optical flow information represents the optical flow change characteristics between a historical image and the previous historical image;

[0035] An optical flow prediction module is used to predict based on a historical optical flow information sequence to obtain a predicted optical flow information sequence; the predicted optical flow information sequence includes a plurality of predicted optical flow information, and the predicted optical flow information represents the optical flow change characteristics between each predicted image and a previous image;

[0036] A spatiotemporal information determination module, used to determine first spatiotemporal information corresponding to the predicted video according to the predicted optical flow information sequence; the first spatiotemporal information includes time features and spatial features corresponding to each predicted image;

[0037] The video prediction module is used to predict a plurality of predicted images according to the first spatiotemporal information and the historical video, and to form a predicted video from the plurality of predicted images.

[0038] In one embodiment, the spatiotemporal information determination module includes:

[0039] The image content determination submodule is used to perform deformation processing according to the predicted optical flow information sequence and the Nth historical image to obtain the image content information corresponding to each predicted image;

[0040] The spatiotemporal information determination submodule is used to input the image content information corresponding to each predicted image into a pre-trained coding network, obtain the temporal features and spatial features corresponding to each predicted image output by the coding network, and form the first spatiotemporal information from the temporal features and spatial features corresponding to multiple predicted images.

[0041] In one of the embodiments, the above-mentioned image content determination submodule is specifically used to perform deformation processing on the Nth historical image according to the first predicted optical flow information to obtain the image content information corresponding to the first predicted image; perform deformation processing on the i-1th predicted image according to the i-th predicted optical flow information to obtain the image content information corresponding to the i-th predicted image; wherein i is a positive integer greater than 1.

[0042] In one of the embodiments, the above-mentioned video prediction module is specifically used to input both the historical video and the first spatiotemporal information into a video prediction network, use the video prediction network to extract features of the historical video, and predict based on the extracted second spatiotemporal information and the first spatiotemporal information to obtain multiple predicted images; the second spatiotemporal information includes the time features and spatial features corresponding to each historical image.

[0043] In one of the embodiments, the video prediction network includes a feature extraction layer, and the video prediction module is specifically used to input N historical images into the feature extraction layer respectively, obtain the time features and spatial features corresponding to each historical image output by the feature extraction layer, and form the second spatiotemporal information by the time features and spatial features corresponding to multiple historical images.

[0044] In one of the embodiments, the video prediction network includes a video prediction layer, and the video prediction module is specifically used to input the first spatiotemporal information and the second spatiotemporal information into the video prediction layer for feature processing to obtain processed spatiotemporal information; and perform prediction based on the processed spatiotemporal information to obtain multiple predicted images.

[0045] In one of the embodiments, the above-mentioned video prediction module is specifically used to input the first spatiotemporal information and the second spatiotemporal information into the video prediction layer, and obtain the video prediction layer splicing or replacing the first spatiotemporal information with the second spatiotemporal information, and output the processed spatiotemporal information.

[0046] In one embodiment, the device further comprises:

[0047] A sample acquisition module is used to acquire a training sample set; the training sample set includes a plurality of historical sample images, a plurality of predicted sample images and sample spatiotemporal information;

[0048] A training module is used to train the neural network based on the training sample set, and use a preset loss function to determine whether the training result output by the neural network meets the preset convergence condition; the preset loss function is a mean square error loss function;

[0049] The network acquisition module is used to end the training when the training results meet the preset convergence conditions and obtain the video prediction network.

[0050] In one of the embodiments, the optical flow prediction module is specifically used to input the historical optical flow information sequence into a pre-trained optical flow prediction network to obtain a predicted optical flow information sequence output by the optical flow prediction network.

[0051] In one of the embodiments, the optical flow acquisition module is used to input every two adjacent historical images into a preset optical flow calculation model to obtain multiple historical optical flow information calculated by the optical flow calculation model, and the multiple historical optical flow information form a historical optical flow information sequence.

[0052] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0053] Obtain a historical optical flow information sequence corresponding to a historical video; a historical video is composed of N historical images arranged in chronological order, and a historical optical flow information sequence includes N-1 historical optical flow information, and the historical optical flow information represents the optical flow change characteristics between a historical image and a previous historical image;

[0054] Predictions are made based on the historical optical flow information sequence to obtain a predicted optical flow information sequence; the predicted optical flow information sequence includes a plurality of predicted optical flow information, and the predicted optical flow information represents the optical flow change characteristics between each predicted image and the previous image;

[0055] Determine first spatiotemporal information corresponding to the predicted video according to the predicted optical flow information sequence; the first spatiotemporal information includes temporal features and spatial features corresponding to each predicted image;

[0056] Prediction is performed based on the first spatiotemporal information and the historical video to obtain a plurality of predicted images, and a predicted video is composed of the plurality of predicted images.

[0057] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0058] Obtain a historical optical flow information sequence corresponding to a historical video; a historical video is composed of N historical images arranged in chronological order, and a historical optical flow information sequence includes N-1 historical optical flow information, and the historical optical flow information represents the optical flow change characteristics between a historical image and a previous historical image;

[0059] Predictions are made based on the historical optical flow information sequence to obtain a predicted optical flow information sequence; the predicted optical flow information sequence includes a plurality of predicted optical flow information, and the predicted optical flow information represents the optical flow change characteristics between each predicted image and the previous image;

[0060] Determine first spatiotemporal information corresponding to the predicted video according to the predicted optical flow information sequence; the first spatiotemporal information includes temporal features and spatial features corresponding to each predicted image;

[0061] Prediction is performed based on the first spatiotemporal information and the historical video to obtain a plurality of predicted images, and a predicted video is composed of the plurality of predicted images.

[0062] In the above-mentioned video prediction method, device, computer equipment and storage medium, the server obtains the historical optical flow information sequence corresponding to the historical video; predicts according to the historical optical flow information sequence to obtain the predicted optical flow information sequence; the predicted optical flow information sequence includes multiple predicted optical flow information; determines the first spatiotemporal information corresponding to the predicted video according to the predicted optical flow information sequence; predicts according to the first spatiotemporal information and the historical video to obtain multiple predicted images, and the predicted video is composed of the multiple predicted images. In the disclosed embodiment, more accurate temporal features and spatial features are extracted based on the optical flow change features, thereby improving the accuracy of video prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A diagram showing an application environment of a video prediction method in an embodiment;

[0064] Figure 2 is a schematic flow chart of a video prediction method in one embodiment;

[0065] Figure 3 A schematic diagram of a flow chart of a step of determining first spatiotemporal information corresponding to a predicted video according to a predicted optical flow information sequence in one embodiment;

[0066] Figure 4 A schematic diagram of a process of training a video prediction network in one embodiment;

[0067] Figure 5 is a structural block diagram of a video prediction device in one embodiment;

[0068] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0070] The video prediction method provided in this application can be applied to Figure 1 In the application environment shown. The application environment includes a video acquisition device 102 and a server 104, wherein the video acquisition device 102 communicates with the server 104 via a network. The video acquisition device 102 may be, but is not limited to, various cameras, laptops, smart phones, tablet computers, and portable wearable devices, and the server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0071] In one embodiment, Figure 2 As shown, a video prediction method is provided, which is applied to Figure 1 The server in the example is used to illustrate the following steps:

[0072] Step 201, obtaining a historical optical flow information sequence corresponding to a historical video.

[0073] Among them, the historical video is composed of N historical images arranged in chronological order, and the historical optical flow information sequence includes N-1 historical optical flow information, and the historical optical flow information represents the optical flow change characteristics between a historical image and the previous historical image. For example, the historical video is composed of 8 historical images, namely, historical image 1, historical image 2...historical image 8, arranged in chronological order; the historical optical flow information sequence includes 7 historical optical flow information, namely, the optical flow change characteristics between historical image 2 and historical image 1, the optical flow change characteristics between historical image 3 and historical image 2...the optical flow change characteristics between historical image 8 and historical image 7. The above-mentioned historical video can be a video in the fields of autonomous driving, weather forecasting, etc., and the embodiments of the present disclosure do not limit the number of historical videos and historical images.

[0074] The server may obtain historical videos in the following manner: the video acquisition device sends the stored historical videos to the server, and the server receives the historical videos sent by the video acquisition device. Alternatively, the video acquisition device sends the collected videos to the server, and the server stores the videos after receiving them; during prediction, the server obtains the stored historical videos locally. The disclosed embodiments do not limit the method for obtaining historical videos.

[0075] The server is pre-set with an optical flow calculation model, which can calculate the optical flow change characteristics between two images. After acquiring the historical video, the server inputs every two adjacent historical images in the historical video into the pre-set optical flow calculation model to obtain multiple historical optical flow information calculated by the optical flow calculation model, and then the multiple historical optical flow information constitute a historical optical flow information sequence.

[0076] For example, historical image 1 and historical image 2 are input into the optical flow calculation model, and the optical flow calculation model outputs the optical flow change characteristics between historical image 2 and historical image 1, that is, outputs historical optical flow information 1; then, historical image 2 and historical image 3 are input into the optical flow calculation model, and the optical flow calculation model outputs the optical flow change characteristics between historical image 3 and historical image 2, that is, outputs historical optical flow information 2. And so on, multiple historical optical flow information are output and constitute a historical optical flow information sequence.

[0077] The above optical flow calculation model can be an optical flow estimation network flownet based on deep learning, and the embodiments of the present disclosure do not limit the optical flow calculation model.

[0078] Step 202: Predict according to the historical optical flow information sequence to obtain a predicted optical flow information sequence.

[0079] The predicted optical flow information sequence includes multiple predicted optical flow information, and the predicted optical flow information characterizes the optical flow change characteristics between each predicted image and the previous image. For example, the predicted optical flow information sequence includes 3 predicted optical flow information, namely, the optical flow change characteristics between the predicted image 1 and the historical image N, the optical flow change characteristics between the predicted image 2 and the predicted image 1, and the optical flow change characteristics between the predicted image 3 and the predicted image 2.

[0080] The server pre-trains the optical flow prediction network. After obtaining the historical optical flow information sequence, the server inputs the historical optical flow information sequence into the pre-trained optical flow prediction network to obtain the predicted optical flow information sequence output by the optical flow prediction network.

[0081] The optical flow prediction network can be a recurrent convolutional neural network, such as ConvLSTM, and the embodiments of the present disclosure do not limit the optical flow prediction network. It can be understood that by using a convolutional recurrent neural network to learn the change characteristics of the optical flow itself, the time-series related characteristics of the optical flow change can be learned.

[0082] Step 203: Determine first spatiotemporal information corresponding to the predicted video according to the predicted optical flow information sequence.

[0083] The first spatiotemporal information includes temporal features and spatial features corresponding to each predicted image.

[0084] After obtaining the predicted optical flow information sequence, the server determines the temporal features and spatial features corresponding to each predicted image according to the predicted optical flow information in the predicted optical flow information sequence, and forms the first spatiotemporal information with the temporal features and spatial features corresponding to the multiple predicted images.

[0085] For example, the temporal features and spatial features corresponding to the predicted image 1 are determined based on the predicted optical flow information 1, and the temporal features and spatial features corresponding to the predicted image 2 are determined based on the predicted optical flow information 2.

[0086] Step 204 , predicting based on the first spatiotemporal information and the historical video to obtain a plurality of predicted images, and composing a predicted video from the plurality of predicted images.

[0087] The server first obtains the temporal features and spatial features of each historical image in the historical video; then, it makes predictions based on the temporal features and spatial features of each historical image and the temporal features and spatial features corresponding to each predicted image to obtain multiple predicted images; finally, the predicted video is composed of multiple predicted images.

[0088] For example, prediction is performed based on the time features and spatial features corresponding to prediction image 1 and the time features and spatial features of historical image N to obtain prediction image 1. Prediction is performed based on the time features and spatial features corresponding to prediction image 2 and the time features and spatial features corresponding to prediction image 1 to obtain prediction image 2. And so on, multiple prediction images can be obtained.

[0089] In the above video prediction method, the server obtains the historical optical flow information sequence corresponding to the historical video; performs prediction based on the historical optical flow information sequence to obtain the predicted optical flow information sequence; determines the first spatiotemporal information corresponding to the predicted video based on the predicted optical flow information sequence; performs prediction based on the first spatiotemporal information and the historical video to obtain multiple predicted images, and the predicted video is composed of the multiple predicted images. In the disclosed embodiment, more accurate temporal features and spatial features are extracted based on the optical flow change features, thereby improving the accuracy of video prediction.

[0090] In one embodiment, Figure 3As shown, the above step of determining the first spatiotemporal information corresponding to the predicted video according to the predicted optical flow information sequence may include:

[0091] Step 301 , performing deformation processing according to the predicted optical flow information sequence and the Nth historical image to obtain image content information corresponding to each predicted image.

[0092] The server performs deformation processing on the Nth historical image according to the first predicted optical flow information to obtain the image content information corresponding to the first predicted image; and performs deformation processing on the i-1th predicted image according to the i-th predicted optical flow information to obtain the image content information corresponding to the i-th predicted image; wherein i is a positive integer greater than 1.

[0093] For example, the server inputs the predicted optical flow information 1 and the historical image N into the pre-trained deformation network to obtain the image content information of the predicted image 1 output by the deformation network. After performing image prediction to obtain the predicted image 1, the predicted optical flow information 2 and the predicted image 1 are input into the deformation network to obtain the image content information of the predicted image 2 output by the deformation network. By analogy, the image content information corresponding to each predicted image can be obtained.

[0094] Each predicted image contains two components, one component is the image content information, such as the predicted image is an apple; the other component is the motion information, such as the direction of motion of the apple.

[0095] Step 302, input the image content information corresponding to each predicted image into a pre-trained coding network, obtain the temporal features and spatial features corresponding to each predicted image output by the coding network, and form the first spatiotemporal information by the temporal features and spatial features corresponding to multiple predicted images.

[0096] The server pre-trains the encoding network, and after obtaining image content information corresponding to a predicted image, inputs the image content information into the encoding network, and the encoding network outputs the temporal features and spatial features corresponding to the predicted image.

[0097] For example, the server inputs the image content information corresponding to predicted image 1 into the encoding network to obtain the temporal features and spatial features corresponding to predicted image 1. After obtaining the image content information corresponding to predicted image 2, the image content information corresponding to predicted image 2 is input into the encoding network to obtain the temporal features and spatial features corresponding to predicted image 2. Finally, the temporal features and spatial features corresponding to multiple predicted images form the first spatiotemporal information.

[0098] In the above step of determining the first spatiotemporal information corresponding to the predicted video according to the predicted optical flow information sequence, the server performs deformation processing according to the predicted optical flow information sequence and the Nth historical image to obtain the image content information corresponding to each predicted image; the image content information corresponding to each predicted image is input into the pre-trained encoding network to obtain the temporal features and spatial features corresponding to each predicted image output by the encoding network, and the temporal features and spatial features corresponding to multiple predicted images form the first spatiotemporal information. In the disclosed embodiment, the server extracts the temporal features and spatial features corresponding to the predicted image according to the image content information corresponding to the predicted image, and these temporal features and spatial features can express the predicted image more accurately, thereby improving the accuracy of video prediction.

[0099] In one embodiment, the above-mentioned step of predicting multiple predicted images based on the first spatiotemporal information and the historical video may include: inputting both the historical video and the first spatiotemporal information into a video prediction network, using the video prediction network to extract features of the historical video, and predicting multiple predicted images based on the extracted second spatiotemporal information and the first spatiotemporal information.

[0100] The second spatiotemporal information includes the time features and spatial features corresponding to each historical image.

[0101] The server pre-trains a video prediction network, which may include a feature extraction layer and a video prediction layer. In actual operation, the server inputs N historical images into the feature extraction layer respectively, obtains the time features and spatial features corresponding to each historical image output by the feature extraction layer, and the time features and spatial features corresponding to multiple historical images form the second spatiotemporal information. Then, the server inputs the first spatiotemporal information and the second spatiotemporal information into the video prediction layer for feature processing to obtain processed spatiotemporal information; prediction is performed based on the processed spatiotemporal information to obtain multiple predicted images.

[0102] For example, the server inputs historical image 1, historical image 2...historical image 8 into the feature extraction layer, and the feature extraction layer extracts the time features and spatial features corresponding to historical image 1, the time features and spatial features corresponding to historical image 2...the time features and spatial features corresponding to historical image 8. The time features h and spatial features c of each historical image can be expressed as H(h, c). The time features h and spatial features c corresponding to the predicted image extracted by the above encoding network can be expressed as H'(h, c). The first spatiotemporal information and the second spatiotemporal information are input into the video prediction layer, and the video prediction layer processes H(h, c) and H'(h, c) to obtain the processed spatiotemporal information.

[0103] In one of the embodiments, the above-mentioned inputting the first spatiotemporal information and the second spatiotemporal information into the video prediction layer for feature processing to obtain processed spatiotemporal information may include: inputting the first spatiotemporal information and the second spatiotemporal information into the video prediction layer, obtaining the video prediction layer splicing or replacing the first spatiotemporal information with the second spatiotemporal information, and outputting the processed spatiotemporal information.

[0104] For example, H(h, c) of historical image 8 is spliced ​​with H'(h, c) corresponding to predicted image 1 to obtain processed space-time information; H'(h, c) corresponding to predicted image 1 is spliced ​​with H'(h, c) corresponding to predicted image 2 to obtain another processed space-time information. Alternatively, H'(h, c) corresponding to predicted image 1 is used to perform feature replacement on H(h, c) of historical image 8 to obtain processed space-time information; H'(h, c) corresponding to predicted image 2 is used to perform feature replacement on H'(h, c) corresponding to predicted image 2 to obtain another processed space-time information. The disclosed embodiments do not limit the feature processing method.

[0105] It can be understood that, since the spatiotemporal information can more accurately represent the image, the server can obtain a more accurate predicted image by performing prediction based on the first spatiotemporal information and the second spatiotemporal information, thereby improving the accuracy of video prediction.

[0106] In one embodiment, Figure 4 As shown, before both the historical video and the first spatiotemporal information are input into the video prediction network, a training process of the video prediction network may also be included:

[0107] Step 401: Obtain a training sample set.

[0108] The training sample set includes multiple historical sample images, multiple predicted sample images and sample spatiotemporal information.

[0109] Get a sample video, and use the first N sample images in the sample video as historical sample images, and use the subsequent multiple sample images in the sample video as predicted sample images. Perform feature extraction on each historical sample image to obtain the time features and spatial features corresponding to each historical sample image; perform feature extraction on each predicted sample image to obtain the time features and spatial features corresponding to each predicted sample image; finally, obtain the sample spatiotemporal information from the time features and spatial features of the historical sample images and the time features and spatial features corresponding to the predicted sample images.

[0110] Step 402: training the neural network based on the training sample set, and using a preset loss function to determine whether the training result output by the neural network meets a preset convergence condition.

[0111] Among them, the preset loss function is the mean square error loss function, such as Among them, MSE is the mean square error, T i To predict the sample image, T i ' is the training result. The preset loss function may also adopt other functions, which is not limited in the embodiment of the present disclosure.

[0112] The server inputs a historical sample image into the neural network, and the neural network outputs the training result. The loss value between the training result and the predicted sample image is calculated using a preset loss function. If the loss value is less than a preset threshold, it is determined that the training result meets the preset convergence condition, and step 403 is executed. If the loss value is greater than or equal to the preset threshold, it is determined that the training result does not meet the preset convergence condition, and the server adjusts the parameters in the neural network and inputs the next historical sample image into the neural network after the parameter adjustment until the training result output in the neural network meets the preset convergence condition. The present disclosure embodiment does not limit the preset threshold.

[0113] Step 403: When the training result satisfies the preset convergence condition, the training is terminated to obtain a video prediction network.

[0114] After determining that the training results meet the preset convergence conditions, the training is terminated, and the neural network at the end of the training is determined as the video prediction network.

[0115] In one embodiment, the video prediction network trained by the embodiment of the present disclosure is compared with the existing convolutional recurrent neural network ConvLSTM, as shown in the following table:

[0116] Structural similarity (SSIM) prediction results on satellite cloud images:

[0117] SSIM Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step7 Step8 ConvLSTM 0.797 0.754 0.718 0.690 0.666 0.646 0.629 0.614 Embodiments of the present disclosure 0.797 0.755 0.719 0.691 0.667 0.647 0.630 0.616

[0118] The mean square error (MSE) prediction results of pixel values ​​on satellite cloud images are as follows:

[0119] SSIM Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step7 Step8 ConvLSTM 21.58 23.55 24.95 26.17 27.31 28.29 29.32 30.32 Embodiments of the present disclosure 21.84 23.43 24.67 25.77 26.77 27.73 28.72 29.67

[0120] It can be seen from the above table that the video prediction network trained using the disclosed embodiment has improved indicators such as MSE and SSIM.

[0121] In the above embodiment, a training sample set is obtained; a neural network is trained based on the training sample set, and a preset loss function is used to determine whether the training result output by the neural network meets the preset convergence condition; the training is terminated when the training result meets the preset convergence condition, and a video prediction network is obtained. The video prediction network structure adopted in the embodiment of the present disclosure has a higher prediction accuracy than that in the prior art.

[0122] In actual operation, the optical flow prediction network, the encoding network and the video prediction network can be trained separately or simultaneously. The embodiments of the present disclosure do not limit this and can be set according to actual conditions.

[0123] It should be understood that although Figure 2-Figure 4 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-Figure 4 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0124] In one embodiment, Figure 5 As shown, a video prediction device is provided, comprising:

[0125] The optical flow acquisition module 501 is used to acquire a historical optical flow information sequence corresponding to a historical video; the historical video is composed of N historical images arranged in chronological order, and the historical optical flow information sequence includes N-1 historical optical flow information, and the historical optical flow information represents the optical flow change characteristics between a historical image and a previous historical image;

[0126] The optical flow prediction module 502 is used to predict according to the historical optical flow information sequence to obtain a predicted optical flow information sequence; the predicted optical flow information sequence includes a plurality of predicted optical flow information, and the predicted optical flow information represents the optical flow change characteristics between each predicted image and the previous image;

[0127] The spatiotemporal information determination module 503 is used to determine the first spatiotemporal information corresponding to the predicted video according to the predicted optical flow information sequence; the first spatiotemporal information includes the temporal features and spatial features corresponding to each predicted image;

[0128] The video prediction module 504 is used to predict a plurality of predicted images according to the first spatiotemporal information and the historical video, and to form a predicted video from the plurality of predicted images.

[0129] In one embodiment, the spatiotemporal information determination module 503 includes:

[0130] The image content determination submodule is used to perform deformation processing according to the predicted optical flow information sequence and the Nth historical image to obtain the image content information corresponding to each predicted image;

[0131] The spatiotemporal information determination submodule is used to input the image content information corresponding to each predicted image into a pre-trained coding network, obtain the temporal features and spatial features corresponding to each predicted image output by the coding network, and form the first spatiotemporal information from the temporal features and spatial features corresponding to multiple predicted images.

[0132] In one of the embodiments, the above-mentioned image content determination submodule is specifically used to perform deformation processing on the Nth historical image according to the first predicted optical flow information to obtain the image content information corresponding to the first predicted image; perform deformation processing on the i-1th predicted image according to the i-th predicted optical flow information to obtain the image content information corresponding to the i-th predicted image; wherein i is a positive integer greater than 1.

[0133] In one of the embodiments, the above-mentioned video prediction module 504 is specifically used to input both the historical video and the first spatiotemporal information into a video prediction network, use the video prediction network to extract features of the historical video, and predict based on the extracted second spatiotemporal information and the first spatiotemporal information to obtain multiple predicted images; the second spatiotemporal information includes the time features and spatial features corresponding to each historical image.

[0134] In one embodiment, the video prediction network includes a feature extraction layer, and the video prediction module 504 is specifically used to input N historical images into the feature extraction layer respectively, obtain the time features and spatial features corresponding to each historical image output by the feature extraction layer, and form the second spatiotemporal information by the time features and spatial features corresponding to multiple historical images.

[0135] In one of the embodiments, the video prediction network includes a video prediction layer, and the video prediction module 504 is specifically used to input the first spatiotemporal information and the second spatiotemporal information into the video prediction layer for feature processing to obtain processed spatiotemporal information; and perform prediction based on the processed spatiotemporal information to obtain multiple predicted images.

[0136] In one embodiment, the video prediction module 504 is specifically used to input the first spatiotemporal information and the second spatiotemporal information into a video prediction layer, and obtain the spatiotemporal information processed by the video prediction layer by splicing or replacing the first spatiotemporal information with the second spatiotemporal information, and outputting the processed spatiotemporal information.

[0137] In one embodiment, the device further comprises:

[0138] A sample acquisition module is used to acquire a training sample set; the training sample set includes a plurality of historical sample images, a plurality of predicted sample images and sample spatiotemporal information;

[0139] A training module is used to train the neural network based on the training sample set, and use a preset loss function to determine whether the training result output by the neural network meets the preset convergence condition; the preset loss function is a mean square error loss function;

[0140] The network acquisition module is used to end the training when the training results meet the preset convergence conditions and obtain the video prediction network.

[0141] In one embodiment, the optical flow prediction module 502 is specifically used to input the historical optical flow information sequence into a pre-trained optical flow prediction network to obtain a predicted optical flow information sequence output by the optical flow prediction network.

[0142] In one embodiment, the optical flow acquisition module 501 is used to input every two adjacent historical images into a preset optical flow calculation model to obtain multiple historical optical flow information calculated by the optical flow calculation model, and the multiple historical optical flow information form a historical optical flow information sequence.

[0143] For the specific definition of the video prediction device, please refer to the definition of the video prediction method above, which will not be repeated here. Each module in the above video prediction device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0144] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store video prediction data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video prediction method is implemented.

[0145] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0146] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0147] Obtain a historical optical flow information sequence corresponding to a historical video; a historical video is composed of N historical images arranged in chronological order, and a historical optical flow information sequence includes N-1 historical optical flow information, and the historical optical flow information represents the optical flow change characteristics between a historical image and a previous historical image;

[0148] Predictions are made based on the historical optical flow information sequence to obtain a predicted optical flow information sequence; the predicted optical flow information sequence includes a plurality of predicted optical flow information, and the predicted optical flow information represents the optical flow change characteristics between each predicted image and the previous image;

[0149] Determine first spatiotemporal information corresponding to the predicted video according to the predicted optical flow information sequence; the first spatiotemporal information includes temporal features and spatial features corresponding to each predicted image;

[0150] Prediction is performed based on the first spatiotemporal information and the historical video to obtain a plurality of predicted images, and a predicted video is composed of the plurality of predicted images.

[0151] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0152] Deformation processing is performed according to the predicted optical flow information sequence and the Nth historical image to obtain image content information corresponding to each predicted image;

[0153] The image content information corresponding to each predicted image is input into a pre-trained coding network to obtain the temporal features and spatial features corresponding to each predicted image output by the coding network, and the temporal features and spatial features corresponding to multiple predicted images form the first spatiotemporal information.

[0154] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0155] Deform the Nth historical image according to the first predicted optical flow information to obtain the image content information corresponding to the first predicted image;

[0156] The i-1th predicted image is deformed according to the i-th predicted optical flow information to obtain image content information corresponding to the i-th predicted image; wherein i is a positive integer greater than 1.

[0157] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0158] The historical video and the first spatiotemporal information are input into a video prediction network, and the video prediction network is used to extract features of the historical video. A plurality of predicted images are predicted based on the extracted second spatiotemporal information and the first spatiotemporal information; the second spatiotemporal information includes the time features and spatial features corresponding to each historical image.

[0159] In one embodiment, the video prediction network includes a feature extraction layer, and the processor further implements the following steps when executing the computer program:

[0160] N historical images are respectively input into the feature extraction layer to obtain the time features and space features corresponding to each historical image output by the feature extraction layer, and the time features and space features corresponding to the multiple historical images form the second spatiotemporal information.

[0161] In one embodiment, the video prediction network includes a video prediction layer, and the processor further implements the following steps when executing the computer program:

[0162] Inputting the first spatiotemporal information and the second spatiotemporal information into the video prediction layer for feature processing to obtain processed spatiotemporal information;

[0163] Prediction is performed based on the processed spatiotemporal information to obtain multiple predicted images.

[0164] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0165] The first spatiotemporal information and the second spatiotemporal information are input into the video prediction layer, and the video prediction layer performs splicing or replacement processing on the first spatiotemporal information and the second spatiotemporal information, and outputs the processed spatiotemporal information.

[0166] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0167] Obtaining a training sample set; the training sample set includes a plurality of historical sample images, a plurality of predicted sample images and sample spatiotemporal information;

[0168] The neural network is trained based on the training sample set, and a preset loss function is used to determine whether the training result output by the neural network meets the preset convergence condition; the preset loss function is a mean square error loss function;

[0169] When the training results meet the preset convergence conditions, the training is terminated to obtain a video prediction network.

[0170] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0171] The historical optical flow information sequence is input into the pre-trained optical flow prediction network to obtain the predicted optical flow information sequence output by the optical flow prediction network.

[0172] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0173] Every two adjacent historical images are input into a preset optical flow calculation model to obtain a plurality of historical optical flow information calculated by the optical flow calculation model, and the plurality of historical optical flow information constitute a historical optical flow information sequence.

[0174] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0175] Obtain a historical optical flow information sequence corresponding to a historical video; a historical video is composed of N historical images arranged in chronological order, and a historical optical flow information sequence includes N-1 historical optical flow information, and the historical optical flow information represents the optical flow change characteristics between a historical image and a previous historical image;

[0176] Predictions are made based on the historical optical flow information sequence to obtain a predicted optical flow information sequence; the predicted optical flow information sequence includes a plurality of predicted optical flow information, and the predicted optical flow information represents the optical flow change characteristics between each predicted image and the previous image;

[0177] Determine first spatiotemporal information corresponding to the predicted video according to the predicted optical flow information sequence; the first spatiotemporal information includes temporal features and spatial features corresponding to each predicted image;

[0178] Prediction is performed based on the first spatiotemporal information and the historical video to obtain a plurality of predicted images, and a predicted video is composed of the plurality of predicted images.

[0179] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0180] Deformation processing is performed according to the predicted optical flow information sequence and the Nth historical image to obtain image content information corresponding to each predicted image;

[0181] The image content information corresponding to each predicted image is input into a pre-trained coding network to obtain the temporal features and spatial features corresponding to each predicted image output by the coding network, and the temporal features and spatial features corresponding to multiple predicted images form the first spatiotemporal information.

[0182] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0183] Deform the Nth historical image according to the first predicted optical flow information to obtain the image content information corresponding to the first predicted image;

[0184] The i-1th predicted image is deformed according to the i-th predicted optical flow information to obtain image content information corresponding to the i-th predicted image; wherein i is a positive integer greater than 1.

[0185] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0186] The historical video and the first spatiotemporal information are input into a video prediction network, and the video prediction network is used to extract features of the historical video. A plurality of predicted images are predicted based on the extracted second spatiotemporal information and the first spatiotemporal information; the second spatiotemporal information includes the time features and spatial features corresponding to each historical image.

[0187] In one embodiment, the video prediction network includes a feature extraction layer, and the computer program, when executed by a processor, further implements the following steps:

[0188] N historical images are respectively input into the feature extraction layer to obtain the time features and space features corresponding to each historical image output by the feature extraction layer, and the time features and space features corresponding to the multiple historical images form the second spatiotemporal information.

[0189] In one embodiment, the video prediction network includes a video prediction layer, and the computer program, when executed by a processor, further implements the following steps:

[0190] Inputting the first spatiotemporal information and the second spatiotemporal information into the video prediction layer for feature processing to obtain processed spatiotemporal information;

[0191] Prediction is performed based on the processed spatiotemporal information to obtain multiple predicted images.

[0192] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0193] The first spatiotemporal information and the second spatiotemporal information are input into the video prediction layer, and the video prediction layer performs splicing or replacement processing on the first spatiotemporal information and the second spatiotemporal information, and outputs the processed spatiotemporal information.

[0194] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0195] Obtaining a training sample set; the training sample set includes a plurality of historical sample images, a plurality of predicted sample images and sample spatiotemporal information;

[0196] The neural network is trained based on the training sample set, and a preset loss function is used to determine whether the training result output by the neural network meets the preset convergence condition; the preset loss function is a mean square error loss function;

[0197] When the training results meet the preset convergence conditions, the training is terminated to obtain a video prediction network.

[0198] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0199] The historical optical flow information sequence is input into the pre-trained optical flow prediction network to obtain the predicted optical flow information sequence output by the optical flow prediction network.

[0200] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0201] Every two adjacent historical images are input into a preset optical flow calculation model to obtain a plurality of historical optical flow information calculated by the optical flow calculation model, and the plurality of historical optical flow information constitute a historical optical flow information sequence.

[0202] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0203] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0204] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A video prediction method, characterized in that: The method comprises: Obtain a historical optical flow information sequence corresponding to a historical video; the historical video is composed of N historical images arranged in chronological order, the historical optical flow information sequence includes N-1 historical optical flow information, and the historical optical flow information represents the optical flow change characteristics between a historical image and a previous historical image; Predictions are made based on the historical optical flow information sequence to obtain a predicted optical flow information sequence; the predicted optical flow information sequence includes a plurality of predicted optical flow information, and the predicted optical flow information characterizes optical flow change characteristics between each predicted image and a previous image; Determine first spatiotemporal information corresponding to the predicted video according to the predicted optical flow information sequence; the first spatiotemporal information includes temporal features and spatial features corresponding to each of the predicted images; Predicting a plurality of predicted images according to the first spatiotemporal information and the historical video, and forming the predicted video from the plurality of predicted images; The step of determining first spatiotemporal information corresponding to the predicted video according to the predicted optical flow information sequence includes: Performing deformation processing according to the predicted optical flow information sequence and the Nth historical image to obtain image content information corresponding to each of the predicted images; Inputting the image content information corresponding to each of the predicted images into a pre-trained encoding network, obtaining the temporal features and spatial features corresponding to each of the predicted images output by the encoding network, and forming the first spatiotemporal information from the temporal features and spatial features corresponding to a plurality of the predicted images; The step of performing deformation processing according to the predicted optical flow information sequence and the Nth historical image to obtain image content information corresponding to each predicted image includes: Performing deformation processing on the Nth historical image according to the first predicted optical flow information to obtain image content information corresponding to the first predicted image; Perform deformation processing on the i-1th predicted image according to the i-th predicted optical flow information to obtain image content information corresponding to the i-th predicted image; wherein i is a positive integer greater than 1; The step of performing prediction according to the first spatiotemporal information and the historical video to obtain a plurality of predicted images includes: Inputting the historical video and the first spatiotemporal information into a video prediction network, extracting features from the historical video using the video prediction network to obtain time features and spatial features corresponding to each historical image, and forming second spatiotemporal information from the time features and spatial features corresponding to a plurality of historical images, and performing prediction based on the extracted second spatiotemporal information and the first spatiotemporal information to obtain a plurality of predicted images; the second spatiotemporal information includes the time features and spatial features corresponding to each historical image; The step of performing prediction according to the extracted second spatiotemporal information and the first spatiotemporal information to obtain a plurality of predicted images includes: Inputting the first spatiotemporal information and the second spatiotemporal information into a video prediction layer, obtaining the spatiotemporal information processed by the video prediction layer by splicing or replacing the first spatiotemporal information and the second spatiotemporal information, and outputting the processed spatiotemporal information; Prediction is performed based on the processed spatiotemporal information to obtain multiple predicted images.

2. The method according to claim 1, characterized in that Before inputting the historical video and the first spatiotemporal information into the video prediction network, the method further includes: Acquire a training sample set; the training sample set includes a plurality of historical sample images, a plurality of predicted sample images and sample spatiotemporal information; Training the neural network based on the training sample set, and using a preset loss function to determine whether the training result output by the neural network meets a preset convergence condition; the preset loss function is a mean square error loss function; When the training result satisfies the preset convergence condition, the training is terminated to obtain the video prediction network.

3. The method according to claim 1, characterized in that: The step of performing prediction according to the historical optical flow information sequence to obtain a predicted optical flow information sequence includes: The historical optical flow information sequence is input into a pre-trained optical flow prediction network to obtain the predicted optical flow information sequence output by the optical flow prediction network.

4. The method according to claim 1, characterized in that: The obtaining of the historical optical flow information sequence corresponding to the historical video includes: Every two adjacent historical images are input into a preset optical flow calculation model to obtain a plurality of the historical optical flow information calculated by the optical flow calculation model, and the plurality of the historical optical flow information constitute the historical optical flow information sequence.

5. A video prediction device, characterized in that: The device comprises: An optical flow acquisition module is used to acquire a historical optical flow information sequence corresponding to a historical video; the historical video is composed of N historical images arranged in chronological order, the historical optical flow information sequence includes N-1 historical optical flow information, and the historical optical flow information represents the optical flow change characteristics between a historical image and a previous historical image; An optical flow prediction module is used to perform prediction based on the historical optical flow information sequence to obtain a predicted optical flow information sequence; the predicted optical flow information sequence includes a plurality of predicted optical flow information, and the predicted optical flow information represents the optical flow change characteristics between each predicted image and the previous image; A spatiotemporal information determination module, configured to determine first spatiotemporal information corresponding to the predicted video according to the predicted optical flow information sequence; the first spatiotemporal information includes time features and spatial features corresponding to each of the predicted images; A video prediction module, configured to predict a plurality of predicted images according to the first spatiotemporal information and the historical video, and to form the predicted video from the plurality of predicted images; The spatiotemporal information determination module comprises: An image content determination submodule, used to perform deformation processing according to the predicted optical flow information sequence and the Nth historical image to obtain image content information corresponding to each predicted image; a spatiotemporal information determination submodule, configured to input the image content information corresponding to each of the predicted images into a pre-trained coding network, obtain the temporal features and spatial features corresponding to each of the predicted images output by the coding network, and form the first spatiotemporal information from the temporal features and spatial features corresponding to the plurality of predicted images; The image content determination submodule is specifically used to perform deformation processing on the Nth historical image according to the first predicted optical flow information to obtain image content information corresponding to the first predicted image; perform deformation processing on the i-1th predicted image according to the i-th predicted optical flow information to obtain image content information corresponding to the i-th predicted image; wherein i is a positive integer greater than 1; The video prediction module is specifically used to input the historical video and the first spatiotemporal information into a video prediction network, use the video prediction network to extract features of the historical video, obtain the time features and spatial features corresponding to each historical image, and form second spatiotemporal information from the time features and spatial features corresponding to multiple historical images, and perform prediction based on the extracted second spatiotemporal information and the first spatiotemporal information to obtain multiple predicted images; the second spatiotemporal information includes the time features and spatial features corresponding to each historical image; The video prediction module is specifically used to input the first spatiotemporal information and the second spatiotemporal information into the video prediction layer, obtain the video prediction layer splicing or replacing the first spatiotemporal information with the second spatiotemporal information, and output the processed spatiotemporal information; perform prediction based on the processed spatiotemporal information to obtain multiple predicted images.

6. The device according to claim 5, characterized in that The device also includes: A sample acquisition module, used to acquire a training sample set; the training sample set includes a plurality of historical sample images, a plurality of predicted sample images and sample spatiotemporal information; A training module, used to train the neural network based on the training sample set, and determine whether the training result output by the neural network meets a preset convergence condition using a preset loss function; the preset loss function is a mean square error loss function; The network acquisition module is used to end the training when the training result meets the preset convergence condition and obtain the video prediction network.

7. The device according to claim 5, characterized in that The optical flow prediction module is specifically used to input the historical optical flow information sequence into a pre-trained optical flow prediction network to obtain the predicted optical flow information sequence output by the optical flow prediction network.

8. The device according to claim 5, characterized in that The optical flow acquisition module is specifically used to input every two adjacent historical images into a preset optical flow calculation model to obtain a plurality of the historical optical flow information calculated by the optical flow calculation model, and the plurality of the historical optical flow information constitute the historical optical flow information sequence.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images