Training methods and systems for optical flow prediction models and video generation methods and systems

Abstract

This invention discloses a training method and system for an optical flow prediction model, and a video generation method and system. The training method for the optical flow prediction model includes: loading an initial optical flow prediction model, which includes a generative adversarial network (GAN) structure; obtaining a training dataset, which includes several data pairs, each consisting of a training image and its corresponding optical flow; modifying the initial optical flow prediction model, and inputting the training dataset into the modified initial optical flow prediction model for training, thereby obtaining a target optical flow prediction model. In the training method for the optical flow prediction model provided by this invention, by training the modified model, the optical flow quality output by the trained target optical flow prediction model is high. The resulting image exhibits significantly reduced checkerboard and droplet effects, better resolution, and thus a substantial improvement in the final generated video's visual quality.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a training method and system for an optical flow prediction model and a video generation method and system. Background Technology

[0002] With the development of the OTA (Online Travel Agency) industry and the advancement of smart devices, the number of images and videos generated during their development and use has grown exponentially. Simultaneously, people's demands for the diversity and quality of images and videos are also increasing. However, because the cost of video acquisition is much higher than that of image acquisition, and with the rapid increase in labor costs, manual video acquisition is insufficient to meet the needs of processing massive amounts of data. Therefore, how to generate videos from images quickly and effectively at low cost has become an important issue for relevant personnel.

[0003] With the rapid development of hardware devices (such as GPUs), deep learning has attracted attention from academia and industry in recent years, and many methods for generating videos from images based on neural networks have emerged. In existing technologies, optical flow prediction can be used to track the motion trajectory of objects in an image, thereby simulating the motion field of the object. However, the images generated by the optical flow sequences predicted by existing optical flow prediction models often exhibit severe checkerboard and droplet effects, making them unsuitable for subsequent video generation. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the defects of the images generated by the optical flow sequence predicted by the existing optical flow prediction model in the prior art, which usually have serious checkerboard effect and water droplet effect. The invention provides a training method and system for optical flow prediction model and a video generation method and system.

[0005] The present invention solves the above-mentioned technical problems through the following technical solution:

[0006] In a first aspect, the present invention provides a method for training an optical flow prediction model, the method comprising:

[0007] Load an initial optical flow prediction model, which includes a generative adversarial network structure;

[0008] Obtain a training dataset, which includes several data pairs, each consisting of a training image and its corresponding optical flow.

[0009] The initial optical flow prediction model is modified, and the training dataset is input into the modified initial optical flow prediction model for training to obtain the target optical flow prediction model;

[0010] The modification of the initial optical flow prediction model includes:

[0011] The upsampling of the initial optical flow prediction model is set to convolutional upsampling using bilinear interpolation;

[0012] Set the normalization operation of the initial optical flow prediction model to instance regularization;

[0013] Add the following function to the Loss function of the initial optical flow prediction model:

[0014]

[0015] in, This represents the predicted optical flow. Indicates the actual optical flow. This indicates the amount of optical flow generated.

[0016] Preferably, before the step of obtaining the training dataset, the method further includes:

[0017] Obtain training videos;

[0018] The training video is split into frames to obtain several training images;

[0019] The Nth frame and the αth frame thereafter of the training image are used as image pairs to input the training optical flow prediction model in order to obtain the training optical flow corresponding to the Nth frame of the training image.

[0020] The Nth frame training image and the training optical flow are used as the data pair to construct the training dataset;

[0021] Where N and α are integers greater than or equal to 2, and N is less than or equal to α.

[0022] Secondly, the present invention provides a training system for an optical flow prediction model, the training system comprising:

[0023] A model loading module is used to load an initial optical flow prediction model, which includes a generative adversarial network structure.

[0024] The training data acquisition module is used to acquire a training dataset, which includes several data pairs, each consisting of a training image and its corresponding optical flow.

[0025] The model training module is used to modify the initial optical flow prediction model and input the training dataset into the modified initial optical flow prediction model for training, so as to obtain the target optical flow prediction model.

[0026] The modification of the initial optical flow prediction model includes:

[0027] The upsampling of the initial optical flow prediction model is set to convolutional upsampling using bilinear interpolation;

[0028] Set the normalization operation of the initial optical flow prediction model to instance regularization;

[0029] Add the following function to the Loss function of the initial optical flow prediction model:

[0030]

[0031] in, This represents the predicted optical flow. Indicates the actual optical flow. This indicates the amount of optical flow generated.

[0032] Thirdly, the present invention provides a video generation method, the video generation method comprising:

[0033] Acquire data to be processed, including the target image;

[0034] The data to be processed is input into the target optical flow prediction model trained by the optical flow prediction model training method described above, so as to obtain the target optical flow sequence, which is used to characterize the motion simulation prediction corresponding to the target image;

[0035] Generate a target video based on the target image and the target optical flow sequence.

[0036] Preferably, the data to be processed further includes an image mask corresponding to the target image, and before the step of acquiring the data to be processed, the method further includes:

[0037] Obtain the image segmentation model;

[0038] The target image is input into the image segmentation model to obtain an image mask corresponding to the target image.

[0039] Preferably, generating the target video based on the target image and the target optical flow sequence includes:

[0040] Build an image translation model;

[0041] The target image and the target optical flow sequence are input into the image translation model to obtain a first image sequence;

[0042] Generate a target video based on the first image sequence.

[0043] Preferably, the image translation model includes a feature encoding module, a feature fusion module, and a feature translation module. The step of inputting the target image and the target optical flow sequence into the image translation model to obtain a first image sequence includes:

[0044] The target image is input into the feature encoding module to generate a feature matrix;

[0045] The feature matrix and the target optical flow sequence are input into the feature fusion module to generate a feature sequence;

[0046] The feature sequence is input into the feature translation module to generate a first image sequence;

[0047] And / or, after the step of obtaining the first image sequence, the method further includes:

[0048] The last frame of the first image sequence and the target optical flow sequence are input into the image translation model to generate the second image sequence;

[0049] Generate a target video based on the first image sequence and the second image sequence.

[0050] Fourthly, the present invention provides a video generation system, the video generation system comprising:

[0051] A data acquisition module is used to acquire data to be processed, including a target image.

[0052] An optical flow generation module is used to input the data to be processed into a target optical flow prediction model trained by the optical flow prediction model training method described above, so as to obtain a target optical flow sequence, wherein the target optical flow sequence is used to characterize the motion simulation prediction corresponding to the target image;

[0053] A video generation module is used to generate a target video based on the target image and the target optical flow sequence.

[0054] Fifthly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the training method of the optical flow prediction model as described above, or the video generation method as described above.

[0055] Sixthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the training method for the optical flow prediction model as described above, or the video generation method as described above.

[0056] The positive and progressive effects of this invention are as follows: In the training method of the optical flow prediction model provided by this invention, by modifying the upsampling, normalization operations and the loss function in the initial optical flow target prediction model including the generative adversarial network, and training the modified model, the optical flow quality output by the trained target optical flow prediction model is high. The checkerboard effect and water droplet effect of the generated image are significantly reduced, and it has better resolution, thus greatly improving the picture quality of the final generated video. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the first process of training the optical flow prediction model in Embodiment 1 of the present invention.

[0058] Figure 2 This is a partial flowchart illustrating the training method of the optical flow prediction model in Embodiment 1 of the present invention.

[0059] Figure 3 This is a schematic diagram of the training system for the optical flow prediction model in Embodiment 2 of the present invention.

[0060] Figure 4 This is a schematic diagram of the first process of the video generation method in Embodiment 3 of the present invention.

[0061] Figure 5 This is a schematic diagram of the second process of the video generation method in Embodiment 3 of the present invention.

[0062] Figure 6 This is a schematic diagram comparing the input and output of the target optical flow prediction model in the video generation method of Embodiment 3 of the present invention.

[0063] Figure 7 This is a schematic diagram of the specific process of step S23 of the video generation method in Embodiment 3 of the present invention.

[0064] Figure 8 This is a schematic diagram of the specific process of step S232 of the video generation method in Embodiment 3 of the present invention.

[0065] Figure 9 This is a schematic diagram showing the inputs for each step of the image translation model in the video generation method of Embodiment 3 of the present invention.

[0066] Figure 10 This is another specific flowchart of step S23 of the video generation method in Embodiment 3 of the present invention.

[0067] Figure 11 This is a schematic diagram of the video generation system in Embodiment 4 of the present invention.

[0068] Figure 12 This is a schematic diagram of the image translation model in Embodiment 4 of the present invention.

[0069] Figure 13 This is a schematic diagram of the electronic device in Embodiment 5 of the present invention. Detailed Implementation

[0070] The present invention will be further illustrated by way of embodiments below, but the present invention is not limited to the scope of the embodiments described herein.

[0071] Example 1

[0072] This embodiment discloses a training method for an optical flow prediction model, such as... Figure 1 As shown, the training method for this optical flow prediction model includes:

[0073] S11. Load the initial optical flow prediction model, which includes a generative adversarial network structure;

[0074] S12. Obtain the training dataset, which includes several data pairs, each consisting of a training image and its corresponding optical flow.

[0075] S13. Modify the initial optical flow prediction model and input the training dataset into the modified initial optical flow prediction model for training to obtain the target optical flow prediction model.

[0076] The modifications to the initial optical flow prediction model include:

[0077] The upsampling of the initial optical flow prediction model is set to convolutional upsampling using bilinear interpolation;

[0078] Set the normalization operation of the initial optical flow prediction model to instance regularization;

[0079] Add the following function to the Loss function of the initial optical flow prediction model:

[0080]

[0081] in, This represents the predicted optical flow. Indicates the actual optical flow. This indicates the amount of optical flow generated.

[0082] This embodiment uses Pix2PixHD (a generative adversarial network model for image translation) as the initial optical flow prediction model loaded in step S11 as an example for illustration. That is, the basic training method and basic model structure during model training in this embodiment are consistent with the Pix2PixHD model.

[0083] It should be noted that the Pix2PixHD model here is only an example and is not limited to the generative adversarial network model used for optical flow prediction in step S11.

[0084] In this embodiment, a dual-channel target optical flow matrix sequence corresponding to the motion field prediction of the target image needs to be generated by inputting a single three-channel target image. However, for the existing Pix2PixHD model, the number of input and output channels is the same. Therefore, the number of output channels needs to be set to dual channels in the convolutional layer so that the corresponding predicted optical flow matrix can be output through Pix2PixHD.

[0085] However, since the upsampling of the Pix2PixHD model itself uses deconvolution, simply modifying the number of output channels will result in severe carding effects in the final generated predicted optical flow. Therefore, in step S3, the upsampling is modified to bilinear interpolation upsampling and convolution operations to effectively avoid carding effects in the output predicted optical flow.

[0086] Meanwhile, the existing Instance Normalization (a normalization operation during model training) of the Pix2PixHD model can cause a droplet effect in the output predicted optical flow, affecting the subsequent use of the predicted optical flow. Therefore, in specific implementation, the modification in step S13 includes further setting the normalization operation of the Pix2PixHD model to WeightNormalization (another normalization operation during model training) to improve the resolution and effect of the output predicted optical flow of the model.

[0087] Based on the above modifications, the loss function is set accordingly. In addition to the loss function of the existing Pix2PixHD model itself, this embodiment also includes an L1 loss function that constrains the generated optical flow and the optical flow label to ensure the quality of the predicted optical flow when generating the target image.

[0088] Specifically, the L1 loss function is:

[0089]

[0090] in, This represents the predicted optical flow corresponding to the motion field prediction of the target image. This represents the corresponding actual optical flow. This indicates the amount of optical flow generated.

[0091] To further improve the quality of the predicted optical flow output by the target optical flow prediction model based on the Pix2PixHD model, in a preferred embodiment, such as Figure 2 As shown, before step S12, the following steps are also included:

[0092] S121. Obtain training videos;

[0093] S122. Split the training video into frames to obtain several training images.

[0094] S123. Use the Nth frame and the αth frame thereafter of the training image as image pairs to input the training optical flow prediction model, so as to obtain the training optical flow corresponding to the Nth frame of the training image.

[0095] S124. Use the Nth frame training image and its training optical flow as data pairs to construct a training dataset;

[0096] Where N and α are integers greater than or equal to 2, and N is less than or equal to α.

[0097] When training the modified Pix2PixHD model, the training data acquisition module uses FlowNet (an optical flow prediction algorithm based on convolutional neural networks) to estimate image pairs, so as to summarize the real optical flow motion trajectory through training videos, and use the acquired optical flow motion trajectory and its corresponding image frame to form an image-optical flow pair as training data to train the target optical flow prediction model.

[0098] Specifically, the aforementioned training data is provided by Composition, among which, It is the Nth frame of the training image obtained by splitting the training video into frames, and It is by and the αth frame thereafter The image pairs formed The result obtained after prediction using the open-source FlowNet model, i.e. .

[0099] By modifying variables N and α, several image-optical flow pairs can be obtained, thus forming a training dataset. Furthermore, since this training data includes real optical flow trajectories summarized from training videos, training the modified Pix2PixHD model based on this image-optical flow pair training dataset not only improves training efficiency but also results in higher accuracy of the predicted optical flow output by the final trained target optical flow prediction model, facilitating subsequent use.

[0100] Therefore, in this embodiment, by modifying the upsampling, normalization operations, and loss function in the initial optical flow target prediction model, including the generative adversarial network, and training the modified model, the optical flow quality output by the trained target optical flow prediction model is improved. The resulting image exhibits significantly reduced checkerboard and droplet effects, better resolution, and thus greatly enhanced image quality in the final generated video. Simultaneously, constructing training data based on image-optical flow pairs improves the accuracy of the correspondence between the predicted optical flow and the actual optical flow output by the target optical flow prediction model, thereby improving the quality of the predicted optical flow. This approach better meets the quality requirements of the generated video while increasing efficiency.

[0101] Example 2

[0102] This embodiment discloses a training system for an optical flow prediction model, such as... Figure 3 As shown, the training system for this optical flow prediction model includes:

[0103] Model loading module 1 is used to load the initial optical flow prediction model, which includes a generative adversarial network structure.

[0104] Training data acquisition module 2 is used to acquire training dataset, which includes several data pairs, each consisting of a training image and its corresponding optical flow.

[0105] Model training module 3 is used to modify the initial optical flow prediction model and input the training dataset into the modified initial optical flow prediction model for training to obtain the target optical flow prediction model;

[0106] The modifications to the initial optical flow prediction model include:

[0107] The upsampling of the initial optical flow prediction model is set to convolutional upsampling using bilinear interpolation;

[0108] Set the normalization operation of the initial optical flow prediction model to instance regularization;

[0109] Add the following function to the Loss function of the initial optical flow prediction model:

[0110]

[0111] in, This represents the predicted optical flow. Indicates the actual optical flow. This indicates the amount of optical flow generated.

[0112] This embodiment uses Pix2PixHD (a generative adversarial network model for image translation) as an example to illustrate the initial optical flow prediction model. That is, the basic training method and basic model structure during model training in this embodiment are consistent with the Pix2PixHD model.

[0113] It should be noted that the Pix2PixHD model here is only an example and does not limit the generative adversarial network models used for optical flow prediction to this.

[0114] In this embodiment, a dual-channel target optical flow matrix sequence corresponding to the motion field prediction of the target image needs to be generated by inputting a single three-channel target image. For the existing Pix2PixHD model, the number of input and output channels is the same. Therefore, when the model loading module 1 loads the Pix2PixHD model as the initial optical flow prediction model, the number of output channels in the convolutional layer needs to be set to dual channels so that the corresponding predicted optical flow matrix can be output through Pix2PixHD.

[0115] However, since the Pix2PixHD model itself uses deconvolution for upsampling, simply modifying the number of output channels will result in severe carding effects in the final generated predicted optical flow. Therefore, model training module 3 is used to modify its upsampling to bilinear interpolation upsampling and convolution operations to effectively avoid carding effects in the output predicted optical flow.

[0116] Meanwhile, the existing Instance Normalization (a normalization operation during model training) of the Pix2PixHD model can cause a droplet effect in the output predicted optical flow, affecting the subsequent use of the predicted optical flow. Therefore, in specific implementation, the model training module 3 is further used to set the normalization operation of the Pix2PixHD model to Weight Normalization (another normalization operation during model training) in order to improve the resolution and effect of the output predicted optical flow of the model.

[0117] Based on the above modifications, the model training module 3 correspondingly sets the loss function, that is, on the basis of the existing loss function of the Pix2PixHD model itself, an L1 loss function is added to constrain the generated optical flow and the optical flow label, so as to ensure the quality of the predicted optical flow when generating the target image.

[0118] Specifically, the L1 loss function is:

[0119]

[0120] in, This represents the predicted optical flow corresponding to the motion field prediction of the target image. This represents the corresponding actual optical flow. This indicates the amount of optical flow generated.

[0121] To further improve the quality of the predicted optical flow output by the target optical flow prediction model based on the Pix2PixHD model, in a preferred embodiment, the training data acquisition module 2 is further configured to:

[0122] Obtain training videos;

[0123] The training video is split into frames to obtain several training images;

[0124] The Nth frame and the αth frame thereafter of the training image are used as image pairs to input the optical flow prediction model in order to obtain the training optical flow corresponding to the Nth frame of the training image.

[0125] The training image of frame N and its training optical flow are used as data pairs to construct the training dataset;

[0126] Where N and α are integers greater than or equal to 2, and N is less than or equal to α.

[0127] When training the modified Pix2PixHD model, the training data acquisition module 2 uses FlowNet (an optical flow prediction algorithm based on convolutional neural networks) to estimate image pairs, so as to summarize the real optical flow motion trajectory through training videos, and use the acquired optical flow motion trajectory and its corresponding image frame to form an image-optical flow pair as training data to train the target optical flow prediction model.

[0128] Specifically, the aforementioned training data is provided by Composition, among which, It is the Nth frame of the training image obtained by splitting the training video into frames, and It is by and the αth frame thereafter The image pairs formed The result obtained after prediction using the open-source FlowNet model, i.e. .

[0129] By modifying variables N and α, several image-optical flow pairs can be obtained, thus forming a training dataset. Furthermore, since this training data includes real optical flow trajectories summarized from training videos, training the modified Pix2PixHD model based on this image-optical flow pair training dataset not only improves training efficiency but also results in higher accuracy of the predicted optical flow output by the final trained target optical flow prediction model, facilitating subsequent use.

[0130] Therefore, in this embodiment, the upsampling, normalization operations, and loss function in the initial optical flow target prediction model, including the generative adversarial network, are modified by the model training module, and the modified model is trained. This results in a higher quality optical flow output from the trained target optical flow prediction model, significantly reducing the checkerboard and droplet effects in the generated image, achieving better resolution, and thus greatly improving the image quality of the final generated video. Simultaneously, the training data acquisition module constructs training data based on image-optical flow pairs, improving the accuracy of the correspondence between the predicted optical flow and the actual optical flow output by the target optical flow prediction model, thereby enhancing the quality of the predicted optical flow. This improves efficiency while better meeting the quality requirements of the generated video.

[0131] Example 3

[0132] This embodiment discloses a video generation method, such as... Figure 4 As shown, the video generation method includes:

[0133] S21. Obtain the data to be processed, which includes the target image;

[0134] S22. Input the data to be processed into the target optical flow prediction model trained by the training method of the optical flow prediction model as described in Example 1, so as to obtain the target optical flow sequence. The target optical flow sequence is used to characterize the motion simulation prediction corresponding to the target image.

[0135] S23. Generate target video based on target image and target optical flow sequence.

[0136] In this embodiment, the target optical flow prediction model obtained after training the modified Pix2PixHD model as the initial optical flow prediction model is also used as an example for illustration.

[0137] Specifically, in step S22, the predicted optical flow corresponding to the data to be processed is... for:

[0138]

[0139] For the predicted optical flow obtained by the target optical flow prediction model By using interpolation and Euler's method, and replacing the rectangular integral with the trapezoidal integral, the integral formula is obtained:

[0140]

[0141] in, This is an integral function pre-defined based on prior knowledge of fluid dynamics, which will not be elaborated upon here. Therefore, the above formula can be used to simulate the motion between the output optical flow sequences, thereby ultimately forming a target optical flow sequence that includes motion field predictions of the target object in the target image. }

[0142] In a preferred embodiment, the data to be processed further includes an image mask corresponding to the target image, therefore, as Figure 5 As shown, before step S21, the above method further includes:

[0143] S211. Obtain the image segmentation model;

[0144] S212. Input the target image into the image segmentation model to obtain the image mask corresponding to the target image.

[0145] In this embodiment, the image segmentation model includes, but is not limited to, the Swin Transformer model. It should be noted that other image segmentation models can also be selected to obtain image masks that include target labels in the segmentation results. Specifically, the target labels can be, for example, rivers, lakes, seas, waterfalls, etc.

[0146] See Figure 6 The square boxes between the input and output represent the generator, discriminator, and loss function in the Pix2PixHD model. Since the target optical flow prediction model in this embodiment maintains the basic structure of the Pix2PixHD model, the structure of the model will not be described in detail here.

[0147] In practice, the input target image is a landscape image including waterfalls and rivers. The two different color blocks of the corresponding image mask correspond to the waterfall area and the river area, respectively. Both are input into the target optical flow prediction model, so that the final output can perform optical flow prediction on the waterfall area and the river area of ​​the target image. This avoids calculating the optical flow of other locations outside the target area, effectively improving the efficiency of optical flow prediction, reducing the waste of computing resources, and saving costs.

[0148] Regarding step S23, since the target optical flow sequence finally output in step S22 of this embodiment is used to characterize the prediction of the motion field of the target object in the target image, therefore, as a preferred implementation, such as Figure 7 As shown, step S23 includes:

[0149] S231. Construct an image translation model;

[0150] S232. Input the target image and target optical flow sequence into the image translation model to obtain the first image sequence;

[0151] S233. Generate the target video based on the first image sequence.

[0152] In this embodiment, a Unet-like network structure (an image semantic segmentation network) is used as the basis for the image translation model, thereby enabling the prediction of the first image sequence corresponding to the target image through the target optical flow sequence to generate the target video.

[0153] Preferably, the above image translation model includes a feature encoding module, a feature fusion module, and a feature translation module. Based on this image translation model, such as... Figure 8 and Figure 9 As shown, step S232 includes:

[0154] S2321. Input the target image into the feature encoding module to generate a feature matrix;

[0155] S2322. Input the feature matrix and the target optical flow sequence into the feature fusion module to generate a feature sequence;

[0156] S2323. Input the feature sequence into the feature translation module to generate the first image sequence.

[0157] In the specific implementation process, the feature encoding module consists of 7 improved Resblock (a module subclass of residual network). In order to adapt to the target optical flow sequence in this embodiment and obtain better output image sequence effects, the ReLU function (an activation function) is modified to the LeakeyReLU function (another activation function), BN (a normalization operation) is modified to PN (another normalization operation), and upsampling and short links are used.

[0158] During feature encoding, this module first downsamples the target image twice to obtain 1 / 4 of its original size, and then upsamples it twice to obtain a feature matrix with the same size as the original target image. For example, given an original image of size... Image I is input as the target image into the feature encoding module, and the features of image I are encoded as follows:

[0159]

[0160] And generate a The characteristic matrix of .

[0161] In the feature fusion module, Feature matrix With the target optical flow sequence { The features are fused using feature wrapping (affine transformation) to generate the wrapped features:

[0162]

[0163] Thus obtain The size is The feature matrix is ​​used to construct the warp feature sequence { }

[0164] The structure of the feature translation module is consistent with that of the feature encoding module, both consisting of seven improved Resblocks. After feature fusion is completed, the obtained features are... The size is The wrapped feature matrix is ​​input one by one into the feature translation module to obtain the translated image. and generate a Size is The images, and thus forming the first image sequence { }

[0165] This embodiment also includes the design of the loss function for the image translation model, which is consistent with the design of the loss function of CoCosNet (an image translation network model), namely:

[0166]

[0167] in, This represents the loss function of the generative adversarial network model. This represents the weights of each loss function.

[0168] The design for the other parts of the loss function described above is as follows:

[0169]

[0170] Among them, the i and j indices include Features Feature map of the layer This represents the activation of layer l in the pre-trained VGG-19 (a convolutional neural network model) model. Furthermore, since low-level features capture richer style information, such as color and texture, layers relu2_2 to relu5-2 are used here.

[0171]

[0172] in, This represents the activation of the l-th layer in the pre-trained VGG-19 model, and features from the ReLU4_2 layer onwards are used here, so these layers mainly include high-level semantic features;

[0173]

[0174] in, Represents the label image. This represents the image obtained by the image translation model, and is used to constrain it so that the output image is generated without distortion, that is, to maintain consistency in detail information.

[0175] Therefore, in this embodiment, by constructing an image translation model, it is possible to translate the target image and its corresponding target optical flow sequence to output a series of image sequences, thereby enabling the generation of video based on the image sequence splicing.

[0176] As another preferred implementation method, such as Figure 10 As shown, after step S232, the following steps are also included:

[0177] S23201. Input the last frame of the first image sequence and the target optical flow sequence into the image translation model to generate the second image sequence;

[0178] S23202. Generate a target video based on the first image sequence and the second image sequence.

[0179] Specifically, using the last frame image in the first image sequence As the target image, it is then input again into the image translation model to obtain the second image sequence. }=model( Then, several frames from the two final image sequences are stitched together to form a video, thereby overcoming the limitations on the length of the generated video caused by the model itself or other objective factors. This enables the simple and effective generation of target videos of arbitrary length, improving the efficiency of video generation, saving costs, and expanding the applicability of the video generation method based on optical flow prediction in this embodiment.

[0180] It should be understood that this can also be done recursively based on the last frame of each image sequence. Secondary image translation to ultimately generate Frame images are then stitched together to generate a video of arbitrary length.

[0181] Therefore, in this embodiment, by inputting the target image into the constructed target optical flow prediction model, a target optical flow sequence for characterizing the motion field prediction of the target object in the target image is generated. This allows the corresponding target image sequence to be generated and stitched into a target video based on the motion field prediction included in the target optical flow sequence. This avoids the distortion of intermediate prediction images that occurs during the process of predicting the next frame image from the previous frame image in a loop, and thus greatly improves the picture quality of the final generated video.

[0182] Example 4

[0183] This embodiment discloses a video generation system, such as... Figure 11 As shown, the video generation system includes:

[0184] The data acquisition module 4 is used to acquire data to be processed, including the target image.

[0185] The optical flow generation module 5 is used to input the data to be processed into the target optical flow prediction model trained by the training method of the optical flow prediction model in Example 1, so as to obtain the target optical flow sequence. The target optical flow sequence is used to characterize the motion simulation prediction corresponding to the target image.

[0186] Video generation module 6 is used to generate target video based on target image and target optical flow sequence.

[0187] In this embodiment, the target optical flow prediction model obtained after training the modified Pix2PixHD model as the initial optical flow prediction model is also used as an example for illustration. The optical flow generation module 5 includes the target optical flow prediction model.

[0188] Specifically, the predicted optical flow output by the optical flow generation module 5 for:

[0189]

[0190] For the predicted optical flow obtained by the target optical flow prediction model By using interpolation and Euler's method, and replacing the rectangular integral with the trapezoidal integral, the integral formula is obtained:

[0191]

[0192] in, This is an integral function pre-set based on prior knowledge of fluid-related factors, which will not be elaborated upon here. Therefore, the optical flow generation module 5, through the motion simulation between the output optical flow sequences corresponding to the above formula, can ultimately form a target optical flow sequence including the motion field prediction of the target object in the target image. }

[0193] In a preferred embodiment, the data to be processed further includes an image mask corresponding to the target image; therefore, the system further includes a mask generation module 7, which is used for:

[0194] Obtain the image segmentation model;

[0195] The target image is input into the image segmentation model to obtain an image mask corresponding to the target image.

[0196] In this embodiment, the image segmentation model in the mask generation module 7 includes, but is not limited to, the SwinTransformer model. It should be noted that other image segmentation models can also be selected to obtain image masks that include target labels in the segmentation results. Specifically, the target labels can be, for example, rivers, lakes, seas, waterfalls, etc.

[0197] See Figure 6 The square boxes between the input and output represent the generator, discriminator, and loss function in the Pix2PixHD model. Since the target optical flow prediction model of the optical flow generation module 5 in this embodiment maintains the basic structure of the Pix2PixHD model, the structure of the model will not be described in detail here.

[0198] In practice, the input target image is a landscape image including waterfalls and rivers. The two different color blocks of the corresponding image mask correspond to the waterfall area and the river area, respectively. Both are input into the target optical flow prediction model, so that the final output can perform optical flow prediction on the waterfall area and the river area of ​​the target image. This avoids calculating the optical flow of other locations outside the target area, effectively improving the efficiency of optical flow prediction, reducing the waste of computing resources, and saving costs.

[0199] The target optical flow sequence finally output by the optical flow generation module 5 is used to characterize the prediction of the motion field of the target object in the target image. Therefore, as a preferred embodiment, the above system also includes an image translation unit 8, which is used for:

[0200] Constructing an image translation model 81;

[0201] The target image and target optical flow sequence are input into the image translation model to obtain the first image sequence;

[0202] Generate the target video based on the first image sequence.

[0203] In this embodiment, a Unet-like network structure (an image semantic segmentation network) is used as the basis for the image translation model, thereby enabling the prediction of the first image sequence corresponding to the target image through the target optical flow sequence to generate the target video.

[0204] Preferably, such as Figure 12 As shown, the above image translation model includes a feature encoding module 811, a feature fusion module 812, and a feature translation module 813. Based on this image translation model 81, the image translation unit 8 is used for:

[0205] The target image is input into the feature encoding module 811 to generate a feature matrix;

[0206] The feature matrix and the target optical flow sequence are input into the feature fusion module 812 to generate a feature sequence;

[0207] The feature sequence is input into the feature translation module 813 to generate the first image sequence.

[0208] In the specific implementation process, the feature encoding module 811 is composed of 7 improved Resblock (a module subclass of residual network). In order to adapt to the target optical flow sequence in this embodiment and obtain better output image sequence effects, the ReLU function (an activation function) is modified to the LeakeyReLU function (another activation function), BN (a normalization operation) is modified to PN (another normalization operation), and it is constructed using upsampling, downsampling and short links.

[0209] During feature encoding, the feature encoding module 811 first obtains 1 / 4 of the original size of the target image through two downsampling operations, and then obtains a feature matrix with the same size as the original target image through two upsampling operations. For example, given an original size of... Image I is input as the target image into the feature encoding module 811, and the features of image I are encoded as:

[0210]

[0211] And generate a The characteristic matrix of .

[0212] In feature fusion module 812, Feature matrix With the target optical flow sequence { The features are fused using feature wrapping (affine transformation) to generate the wrapped features:

[0213]

[0214] Thus obtain The size is The feature matrix is ​​used to construct the warp feature sequence { }

[0215] The feature translation module 813 has the same structure as the feature encoding module 811, both consisting of seven improved Resblocks. After feature fusion is completed, the obtained features are... The size is The wrapped feature matrix is ​​input one by one into the feature translation module 813 to obtain the translated image. and generate a Size is The images, and thus forming the first image sequence {}

[0216] This embodiment also includes the design of the loss function for the image translation model 81, which is consistent with the design of the loss function of CoCosNet (an image translation network model), namely:

[0217]

[0218] in, This represents the loss function of the generative adversarial network model. This represents the weights of each loss function.

[0219] The design for the other parts of the loss function described above is as follows:

[0220]

[0221] Among them, the i and j indices include Features Feature map of the layer This represents the activation of layer l in the pre-trained VGG-19 (a convolutional neural network model) model. Furthermore, since low-level features capture richer style information, such as color and texture, layers relu2_2 to relu5-2 are used here.

[0222]

[0223] in, This represents the activation of the l-th layer in the pre-trained VGG-19 model, and features from the ReLU4_2 layer onwards are used here, so these layers mainly include high-level semantic features;

[0224]

[0225] in, Represents the label image. This represents the image obtained by the image translation model, and is used to constrain it so that the output image is generated without distortion, that is, to maintain consistency in detail information.

[0226] Therefore, in this embodiment, by constructing an image translation model 81, it is possible to translate the target image and its corresponding target optical flow sequence to output a series of image sequences, thereby enabling the generation of video based on the image sequence splicing.

[0227] As another preferred embodiment, the image translation unit 8 is also used for:

[0228] The last frame of the first image sequence and the target optical flow sequence are input into the image translation model 81 to generate the second image sequence;

[0229] Generate a target video based on the first image sequence and the second image sequence.

[0230] Specifically, using the last frame image in the first image sequence As the target image, it is input again into the image translation model 81 to obtain the second image sequence { }=model( Then, several frames from the two final image sequences are stitched together to form a video, thereby overcoming the limitations on the length of the generated video caused by the model itself or other objective factors. This enables the simple and effective generation of target videos of arbitrary length, improving the efficiency of video generation, saving costs, and expanding the applicability of the video generation method based on optical flow prediction in this embodiment.

[0231] It should be understood that this can also be done recursively based on the last frame of each image sequence. Secondary image translation to ultimately generate Frame images are then stitched together to generate a video of arbitrary length.

[0232] Therefore, in this embodiment, by inputting the target image into the constructed target optical flow prediction model, a target optical flow sequence for characterizing the motion field prediction of the target object in the target image is generated. This allows the corresponding target image sequence to be generated and stitched into a target video based on the motion field prediction included in the target optical flow sequence. This avoids the distortion of intermediate prediction images that occurs during the process of predicting the next frame image from the previous frame image in a loop, and thus greatly improves the picture quality of the final generated video.

[0233] Example 5

[0234] Figure 13 This is a schematic diagram of an electronic device provided in Embodiment 5 of the present invention. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the training method of the optical flow prediction model in Embodiment 1, or the video generation method in Embodiment 2. Figure 13 The electronic device 30 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0235] like Figure 13As shown, the electronic device 30 can be manifested in the form of a general-purpose computing device, such as a smartwatch. The components of the electronic device 30 may include, but are not limited to: at least one processor 31, at least one memory 32, and a bus 33 connecting different system components (including memory 32 and processor 31).

[0236] Bus 33 includes a data bus, an address bus, and a control bus.

[0237] The memory 32 may include volatile memory, such as random access memory (RAM) 321 and / or cache memory 322, and may further include read-only memory (ROM) 323.

[0238] The memory 32 may also include a program / utility 325 having a set (at least one) of program modules 324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0239] The processor 31 executes various functional applications and data processing by running computer programs stored in the memory 32, such as the training method of the optical flow prediction model in Embodiment 1 of the present invention, or the video generation method in Embodiment 2.

[0240] Electronic device 30 can also communicate with one or more external devices 34 (e.g., mobile phones). This communication can be performed via input / output (I / O) interface 35. Furthermore, the model-generating device can also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 36. Figure 13 As shown, network adapter 36 communicates with other modules of the model-generated device via bus 33. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the model-generated device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0241] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0242] Example 6

[0243] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the training method for the optical flow prediction model in Embodiment 1 or the video generation method in Embodiment 2.

[0244] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0245] In a possible implementation, the present invention can also be implemented as a program product comprising program code, which, when the program product is run on a terminal device, causes the terminal device to execute a training method for implementing the optical flow prediction model in Embodiment 1, or a video generation method in Embodiment 2.

[0246] The program code for executing the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0247] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but all such changes and modifications fall within the scope of protection of the present invention.

Claims

1. A training method for an optical flow prediction model, characterized in that, The training method for the optical flow prediction model includes: Load an initial optical flow prediction model, which includes a generative adversarial network structure; Obtain a training dataset, which includes several data pairs, each consisting of a training image and its corresponding optical flow. The initial optical flow prediction model is modified, and the training dataset is input into the modified initial optical flow prediction model for training to obtain the target optical flow prediction model; The modification of the initial optical flow prediction model includes: The upsampling of the initial optical flow prediction model is set to convolutional upsampling using bilinear interpolation; Set the normalization operation of the initial optical flow prediction model to instance regularization; Add the following function to the Loss function of the initial optical flow prediction model: Among them, flow pre The term "flow" represents the predicted optical flow. label represents the actual optical flow, and n represents the number of generated optical flows; Before the step of obtaining the training dataset, the following steps are also included: Obtain training videos; The training video is split into frames to obtain several training images; The Nth frame and the αth frame thereafter of the training image are used as image pairs to input the training optical flow prediction model in order to obtain the training optical flow corresponding to the Nth frame of the training image. The Nth frame training image and the training optical flow are used as the data pair to construct the training dataset; Where N and α are integers greater than or equal to 2, and N is less than or equal to α.

2. A training system for an optical flow prediction model, characterized in that, The training system for the optical flow prediction model includes: A model loading module is used to load an initial optical flow prediction model, which includes a generative adversarial network structure. The training data acquisition module is used to acquire a training dataset, which includes several data pairs, each consisting of a training image and its corresponding optical flow. The model training module is used to modify the initial optical flow prediction model and input the training dataset into the modified initial optical flow prediction model for training, so as to obtain the target optical flow prediction model. The modification of the initial optical flow prediction model includes: The upsampling of the initial optical flow prediction model is set to convolutional upsampling using bilinear interpolation; Set the normalization operation of the initial optical flow prediction model to instance regularization; Add the following function to the Loss function of the initial optical flow prediction model: Among them, flow pre The term "flow" represents the predicted optical flow. label represents the actual optical flow, and n represents the number of generated optical flows; The training data acquisition module is also used for: Obtain training videos; The training video is split into frames to obtain several training images; The Nth frame and the αth frame thereafter of the training image are used as image pairs to input the training optical flow prediction model in order to obtain the training optical flow corresponding to the Nth frame of the training image. The Nth frame training image and the training optical flow are used as the data pair to construct the training dataset; Where N and α are integers greater than or equal to 2, and N is less than or equal to α.

3. A video generation method, characterized in that, The video generation method includes: Acquire data to be processed, including the target image; The data to be processed is input into a target optical flow prediction model trained by the training method of the optical flow prediction model as described in claim 1 to obtain a target optical flow sequence, which is used to characterize motion simulation prediction corresponding to the target image; Generate a target video based on the target image and the target optical flow sequence.

4. The video generation method as described in claim 3, characterized in that, The data to be processed also includes an image mask corresponding to the target image. Before the step of acquiring the data to be processed, the method further includes: Obtain the image segmentation model; The target image is input into the image segmentation model to obtain an image mask corresponding to the target image.

5. The video generation method as described in claim 3, characterized in that, The generation of the target video based on the target image and the target optical flow sequence includes: Build an image translation model; The target image and the target optical flow sequence are input into the image translation model to obtain a first image sequence; Generate a target video based on the first image sequence.

6. The video generation method as described in claim 5, characterized in that, The image translation model includes a feature encoding module, a feature fusion module, and a feature translation module. The step of inputting the target image and the target optical flow sequence into the image translation model to obtain a first image sequence includes: The target image is input into the feature encoding module to generate a feature matrix; The feature matrix and the target optical flow sequence are input into the feature fusion module to generate a feature sequence; The feature sequence is input into the feature translation module to generate a first image sequence; And / or, after the step of obtaining the first image sequence, the method further includes: The last frame of the first image sequence and the target optical flow sequence are input into the image translation model to generate the second image sequence; Generate a target video based on the first image sequence and the second image sequence.

7. A video generation system, characterized in that, The video generation system includes: A data acquisition module is used to acquire data to be processed, including a target image. An optical flow generation module is used to input the data to be processed into a target optical flow prediction model trained by the training method of the optical flow prediction model as described in claim 1, so as to obtain a target optical flow sequence, wherein the target optical flow sequence is used to characterize the motion simulation prediction corresponding to the target image; A video generation module is used to generate a target video based on the target image and the target optical flow sequence.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the training method for the optical flow prediction model as described in claim 1, or the video generation method as described in any one of claims 3-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method of the optical flow prediction model as described in claim 1, or the video generation method as described in any one of claims 3-6.

Images