Video generation method and apparatus

Abstract

An embodiment of the present disclosure provides a video generating method and apparatus, which includes: extracting a first image feature in a first image; obtaining a plurality of intermediate image features by nonlinear interpolation according to the first image feature and a second image feature, which is an image feature of a second image; reconstructing an image by an image generation model according to the first image feature, the second image feature and the plurality of intermediate image features, and generating a target video for showing a gradual change process from the first image to the second image, thereby improving the quality of the intermediate image feature by nonlinear interpolation, and further improving the image quality of the intermediate frames of the target video, i.e., improving the video quality of the target video.

Description

[Technical Field]

[0001] This application claims priority to a Chinese patent application filed with the China Patent Office on December 24, 2021, bearing application number 202111609441.8 and titled "Video generation method and apparatus," the entire contents of which are incorporated herein by reference.

[0002] TECHNICAL FIELD Embodiments of the present disclosure relate to the field of computer technology, and more particularly to video generation methods and devices. [Background technology]

[0003] In conventional computer vision technology and deep learning technology, a video that gradually changes between two images can be generated based on two images. For example, two facial images are input into a deep learning model to generate a video that gradually changes between the two facial images, and the video frames in the video gradually change from one facial image to another. This technology can be used, for example, to generate special effects and improve the entertainment value of videos.

[0004] In the process of video generation, it is necessary to generate multiple video frames in a video based on two images, and obtain a video that gradually changes between the two images. However, currently, there is a need to improve the quality of the multiple video frames generated based on two images, especially the image quality of the intermediate frames of the video. Summary of the Invention [Means for solving the problem]

[0005] SUMMARY OF THE INVENTION The embodiments of the present disclosure provide a video generating method and apparatus to solve the problem of the need to improve the image quality of intermediate frames of a video when generating a video based on a small number of images.

[0006] According to a first aspect, an embodiment of the present disclosure provides a video generation method, comprising: extracting a first image feature in a first image; obtaining a plurality of intermediate image features by nonlinear interpolation based on the first image feature and a second image feature, the second image feature being an image feature of a second image; and reconstructing an image using an image generation model based on the first image features, the second image features, and the plurality of intermediate image features, and generating a target video to show the gradual change process from the first image to the second image.

[0007] According to a second aspect, an embodiment of the present disclosure provides a model determination method, comprising: training a neural network with a plurality of training images and an image generation model to learn deviations to adjust image features based on a feature space of the image generation model; A single training process of the neural network is as follows: generating a target image feature based on the image feature of the first training image and the image feature of the second training image; initializing the target image features based on the feature space; learning a target deviation corresponding to the initial adjustment by the neural network, and re-adjusting the initially adjusted target image feature according to the target deviation; and adjusting model parameters of the neural network based on the target deviation, the re-adjusted target image features, the first training images, and the second training images.

[0008] According to a third aspect, an embodiment of the present disclosure provides a video generation device, comprising: an extraction unit for extracting a first image feature in the first image; an interpolation unit for obtaining a plurality of intermediate image features by nonlinear interpolation based on the first image feature and a second image feature, the second image feature being an image feature of a second image; and a video generation unit for reconstructing an image using an image generation model based on the first image features, the second image features, and the plurality of intermediate image features, and generating a target video for showing the process of gradual change from the first image to the second image.

[0009] According to a fourth aspect, an embodiment of the present disclosure provides a model determination apparatus, a training unit for training a neural network with a plurality of training images and an image generation model to learn deviations for adjusting image features based on a feature space of the image generation model; A single training process of the neural network is as follows: generating a target image feature based on the image feature of the first training image and the image feature of the second training image; initializing the target image features based on the feature space; learning a target deviation corresponding to the initial adjustment by the neural network, and re-adjusting the initially adjusted target image feature according to the target deviation; and adjusting model parameters of the neural network based on the target deviation, the re-adjusted target image features, the first training images, and the second training images.

[0010] According to a fifth aspect, an embodiment of the present disclosure provides an electronic device comprising at least one processor and a memory; the memory stores computer-executable instructions; The at least one processor executes computer-executable instructions stored in the memory to cause the at least one processor to perform the video generation method described in the first aspect or various possible designs of the first aspect, or to cause the at least one processor to perform the model determination method described in the second aspect or various possible designs of the second aspect.

[0011] According to a sixth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon, the computer-executable instructions, when executed by a processor, realizing the video generation method according to the first aspect or various possible designs of the first aspect, or the model determination method according to the second aspect or various possible designs of the second aspect.

[0012] According to a seventh aspect, there is provided a computer program product in accordance with one or more embodiments of the present disclosure, the computer program product including computer executable instructions that, when executed by a processor, implement the video generation method according to the first aspect or various possible designs of the first aspect, or the model determination method according to the second aspect or various possible designs of the second aspect.

[0013] According to an eighth aspect, there is provided a computer program according to one or more embodiments of the present disclosure, which, when executed by a processor, implements the video generation method according to the first aspect or various possible designs of the first aspect, or the model determination method according to the second aspect or various possible designs of the second aspect.

[0014] The video generation method and device provided in this embodiment obtain a plurality of intermediate image features by nonlinear interpolation based on the first image features of a first image and the second image features of a second image, reconstruct an image using an image generation model based on the first image features, the second image features, and the plurality of intermediate image features to generate a target video showing the gradual change process from the first image to the second image, thereby improving the quality of the intermediate image features by nonlinear interpolation, and ensuring the similarity between the intermediate frames of the target video and the first image and the second image, thereby improving the image quality of the intermediate frames of the target video and further improving the video quality of the target video.

[0015] In order to more clearly describe the embodiments of the present disclosure or the technical solutions of the prior art, the following briefly introduces drawings that need to be used in the description of the embodiments or the prior art. It is obvious that the drawings in the following description are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without requiring any creative work. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a schematic diagram of an application scenario to which the embodiments of the present disclosure are applied. [Figure 2] FIG. 2 is a process diagram 1 of a video generation method provided by an embodiment of the present disclosure. [Figure 3a] FIG. 3a is a process diagram 2 of a video generating method provided by an embodiment of the present disclosure. [Figure 3b] FIG. 3b is a process diagram of adjusting the third image feature based on the feature space of the image generation model and the neural network in the order provided by the embodiment of the present disclosure. [Figure 4] FIG. 4 is a diagram illustrating an example of a framework for nonlinear interpolation based on feature space and neural network provided by an embodiment of the present disclosure. [Figure 5] FIG. 5 is a process diagram of a model determination method provided by an embodiment of the present disclosure. [Figure 6] FIG. 6 is a schematic diagram of a neural network training framework provided by an embodiment of the present disclosure. [Figure 7] FIG. 7 is a structural block diagram of a video generating device provided by an embodiment of the present disclosure. [Figure 8] FIG. 8 is a structural block diagram of a model determination device provided by an embodiment of the present disclosure. [Figure 9] FIG. 9 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0017] In order to clarify the objectives, technical solutions, and advantages of the embodiments of the present disclosure, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings of the embodiments of the present disclosure. It is obvious that the described embodiments are only some of the embodiments of the present disclosure, but not all of the embodiments. Any other embodiments that can be obtained by those skilled in the art based on the embodiments of the present disclosure without requiring creative work shall fall within the scope of protection of the present disclosure.

[0018] When generating a video in which two input images change gradually, the image features of the two input images are generally linearly interpolated to obtain intermediate image features, and the intermediate image features are then used to generate intermediate frames of the video. Although this method can ensure the continuity and similarity of the video frames, the intermediate image features after linear interpolation often do not match the distribution rules (or change rules) of the image features of the video scenes in the actual video, resulting in poor image quality and insufficient aesthetics and realism.

[0019] To solve the above problem, embodiments of the present disclosure provide a video generation method and apparatus, which obtains a plurality of intermediate image features through nonlinear interpolation based on a first image feature of a first image and a second image feature of a second image, and then reconstructs an image using an image generation model based on the first image feature, the second image feature, and the plurality of intermediate image features to generate a target video. The target video is used to represent the gradual change from the first image to the second image. Because the actual video scene change process is nonlinear, compared to linear interpolation, embodiments of the present disclosure use nonlinear interpolation to improve the quality of the intermediate image features and improve the quality of intermediate frames of the target video, thereby achieving nonlinear changes in the video scene of the target video and achieving greater authenticity and aesthetics. Improving the quality of the intermediate image features includes improving the authenticity of the intermediate image features and improving the similarity between the intermediate frames and the first and second images. Improving the quality of the intermediate frames of the target video includes improving the aesthetics and authenticity of the intermediate frames and improving the similarity between the intermediate frames and the first and second images.

[0020] Referring to FIG. 1, FIG. 1 is a schematic diagram of an application scenario to which the embodiments of the present disclosure are applied.

[0021] As shown in Fig. 1, in this application scenario, the relevant devices include a video generation device 101, which may be a terminal or a server, and Fig. 1 takes the video generation device 101 as an example of a server. The video generation device 101 can process two images and generate a video to show the gradual change effect between the two images.

[0022] In one embodiment, the device related to the application scene further includes an image collection device 102, which may also be a terminal or a server. For example, the terminal collects images input by a user, or the terminal collects images of the current scene using a camera, or the server collects images from the network that are published to the network and allowed for public use. Figure 1 shows an example in which the image collection device 102 is a terminal. The image collection device 102 transmits the collected images to the video generation device 101, which generates a video showing a gradual change from the collected image to another image (from the image collection device 102 or another device), or a video showing a gradual change from another image to the collected image.

[0023] The video generation device 101 and the image acquisition device 102 may be the same device or may be different devices.

[0024] When the video generating device 101 and the image collecting device 102 are the same device, for example, a user takes a selfie with a mobile phone, obtains a selfie avatar, and selects another image from the mobile phone, and the mobile phone generates a video based on the user's selfie avatar and the image selected by the user, and the video content of the video gradually changes from the user's selfie avatar to the image selected by the user.

[0025] If the video generating device 101 and the image collecting device 102 are different devices, for example, a user takes a selfie with a mobile phone, obtains a selfie avatar, and selects another image from the mobile phone, the mobile phone sends the selfie image and the image selected by the user to a server, the server generates a video and sends the video back to the mobile phone, and the video content of the video gradually changes from the user's selfie avatar to the image selected by the user.

[0026] The terminal may be a personal digital assistant (PDA) device, a mobile device (e.g., a smartphone, a tablet computer), a computing device (e.g., a personal computer (PC)), an in-vehicle device, a wearable device (e.g., a smart watch, a smart bracelet), a smart home device (e.g., a smart display device), etc. The server may be a distributed server, a centralized server, a cloud server, etc.

[0027] The following provides several embodiments of the present disclosure. The execution body of the several embodiments of the present disclosure may be an electronic device, and the electronic device may be a terminal or a server.

[0028] 2, which is a process diagram 1 of a video generating method provided by an embodiment of the present disclosure. As shown in FIG. 2, the video generating method includes steps 201 to 203.

[0029] S201: extracting a first image feature from a first image;

[0030] The first image may be an image input by a user, an image from another device, or an image captured by the current executing device. For example, if the current executing device is a terminal, the terminal may acquire the first image input by a user or the first image captured by a camera of the terminal. Also, for example, if the current executing device is a server, the server may receive the first image input by a user sent by the terminal.

[0031] The first image feature is an image feature of the first image.

[0032] In this embodiment, the encoder encodes a first image to obtain first image features, and the first image features specifically refer to the image features obtained after encoding the first image.

[0033] S202, based on the first image feature and the second image feature, which is the image feature of the second image, obtain a plurality of intermediate image features by nonlinear interpolation.

[0034] The second image is different from the first image.

[0035] The second image feature specifically refers to the image feature obtained after encoding the second image.

[0036] In one example, a plurality of images and image features obtained after encoding the plurality of images can be stored in advance. A second image feature is acquired from the image features of the stored plurality of images. In one method, a user can specify a second image from the pre-stored plurality of images and acquire the image features of the second image, i.e., the second image feature, from the image features of the plurality of images. In another method, the second image feature can be acquired from the image features of the plurality of images in a preset order (e.g., the image storage order) or randomly.

[0037] For example, in response to a user's operation of inputting a first image, the terminal displays a plurality of images selected by the user, the user selects a second image from the plurality of images, and inputs a request to the terminal to generate a video that gradually changes from the first image to the second image, and the terminal responds to the request and obtains image features of the second image, i.e., the second image features, from image features of the plurality of images stored in advance.

[0038] In another example, a second image input by a user, transmitted by another device, or captured by the current executing device can be acquired, and the second image can be encoded to obtain second image features.

[0039] In this embodiment, after the second image feature is obtained, a preset nonlinear interpolation method is used to perform nonlinear interpolation using the first and second image features as two known quantities in the nonlinear interpolation process to obtain an interpolation function, i.e., an interpolation curve. Sampling is performed between the points corresponding to the first image feature and the points corresponding to the second image feature on the interpolation curve to obtain a plurality of intermediate image features. The intermediate image features are used to generate intermediate frames of the video.

[0040] In one embodiment, equally spaced sampling in the interpolation curve results in closer variation between adjacent intermediate image features obtained by the interpolation, improving the quality of the subsequently generated video.

[0041] S203: Based on the first image features, the second image features and the plurality of intermediate image features, an image is reconstructed by an image generation model, and a target video is generated to show the process of gradual change from the first image to the second image.

[0042] The image generation model may be a neural network for image generation or image reconstruction, whose input data are encoded image features and whose output data is a reconstructed image. A trained image generation model published to the network may be used, or the image generation model may be obtained by training the neural network with training data (including multiple training images), and there is no restriction on the training process of the model.

[0043] In this embodiment, after the multiple intermediate image features are obtained, the first image feature, the second image feature, and the multiple intermediate image features are input into an image generation model, respectively, to obtain a reconstructed image corresponding to the first image feature, a reconstructed image corresponding to the second image feature, and a reconstructed image corresponding to each intermediate image feature. The multiple reconstructed images are ordered and combined according to the distribution order of the first image feature, the second image feature, and the intermediate image feature on the interpolation curve to obtain a target video. In the target video, the image of the first frame is the reconstructed image corresponding to the first image feature, the image of the last frame is the reconstructed image corresponding to the second image feature, and the image of the intermediate frame is the reconstructed image corresponding to the intermediate image feature.

[0044] In an embodiment of the present disclosure, nonlinear interpolation is performed based on first image features obtained by encoding a first image and second image features obtained by encoding a second image to obtain multiple intermediate image features, an image is reconstructed using an image generation model based on the first image features, the second image features, and the multiple intermediate image features, and the reconstructed image is output based on the image generation model.

[0045] Therefore, the method of performing nonlinear interpolation based on the image features obtained by encoding two images improves the authenticity of the intermediate image features obtained by interpolation and the similarity between the intermediate image features and the image features of the two first images, further improving the authenticity and aesthetics of the intermediate frames of the video, and improving the similarity between the intermediate frames and the images of the first and last frames, thereby improving video quality.

[0046] For the image generation model, we have several possible implementations:

[0047] In some embodiments, the image generation model is a generative adversarial network (GAN), thereby utilizing the image generation advantages of GAN to improve the quality of image reconstruction of the image generation model and improve the quality of image frames of the target video.

[0048] In some embodiments, the image generation model is a style-based architecture for GANs (StyleGAN) model or a StyleGAN2 model, thereby utilizing the image generation advantages of the StyleGAN model or the StyleGAN2 model to improve the quality of image reconstruction of the image generation model and improve the quality of image frames of the target video.

[0049] In some embodiments, the nonlinear interpolation process can be assisted by using a feature space of an image generation model or a neural network. The following examples will illustrate this assisted process.

[0050] 3a, which is a process diagram 2 of a video generating method provided by an embodiment of the present disclosure. As shown in FIG. 3a, the video generating method includes steps S301 to S305.

[0051] S301: extracting a first image feature from a first image;

[0052] The implementation principle and technical effect of S301 can be referred to the above embodiments and will not be described here.

[0053] S302: generating a third image feature based on the first image feature and a second image feature that is an image feature of the second image;

[0054] The process of acquiring the second image feature can be referred to the above embodiment and will not be described here.

[0055] In one example, an average value of the first image feature and the second image feature is determined, and the average value is the third image feature. Specifically, the feature values ​​of the first image feature and the second image feature at corresponding positions are added and then averaged to obtain the average value of the first image feature and the second image feature.

[0056] In another example, the first image feature and the second image feature are weighted together to obtain a third image feature. Weights corresponding to the first image feature and the second image feature may be preset.

[0057] S303, in turn, adjust a third image feature based on the feature space of the image generation model and a neural network for learning deviations to adjust the image feature based on the feature space.

[0058] The feature space of the image generation model can be understood as the input space of the image generation model, where the feature samples in the input space fit a given probability distribution.

[0059] In one embodiment, when the image generation model is a generative adversarial network, the feature space of the image generation model is a latent space corresponding to the image generation model, and the image features obtained by encoding the first image and the second image by the encoder are latent codes, i.e., the first image features can be referred to as the first latent code, and the second image features can be referred to as the second latent code.

[0060] In this embodiment, after the third image feature is obtained, the third image feature can be first adjusted based on the feature samples in the feature space of the image generation model, thereby making the third image feature closer to the feature samples in the feature space, thereby improving the image quality of the reconstructed image obtained by reconstructing the image based on the third image feature, i.e., improving the image quality of the intermediate frame.

[0061] In this embodiment, it is considered that there may be a certain deviation in adjusting the third image feature based on the feature space, which may reduce the similarity between the third image feature and the first image feature and the second image feature. Therefore, to solve this problem, after adjusting the third image feature based on the feature space, the third image feature is adjusted again by a neural network model, thereby improving the similarity between the third image feature and the first image feature and the second image feature.

[0062] The neural network needs to be trained to learn the deviation of adjusting image features based on the feature space. The specific training process is described in the following examples.

[0063] In one embodiment, the neural network is a full-link neural network, so that the learning task of the neural network is unified, and when the input data and the output data are both image features, the full-link neural network with many network parameters improves the accuracy of adjusting the third image feature.

[0064] In one possible implementation, referring to Fig. 3b, Fig. 3b is a process schematic diagram of adjusting the third image feature sequentially based on the feature space of the image generation model and the neural network (i.e., S303) provided by an embodiment of the present disclosure. As shown in Fig. 3b, the process of adjusting the third image feature sequentially based on the feature space of the image generation model and the neural network (i.e., one possible implementation of S303) includes steps S3031 to S3034.

[0065] S3031: obtain an average image feature in the feature space; S3032: initially adjust a third image feature based on the average image feature; S3033: input the first image feature and the second image feature into a neural network to obtain output data of the neural network that reflects the deviation of the initial adjustment; S3034: re-adjust the initially adjusted third image feature based on the output data.

[0066] The output data of the neural network reflects the feature deviations that occur after initial adjustment of the third image feature based on the average image feature in the feature space.

[0067] In this embodiment, the average image feature in the feature space can be determined based on the probability distribution of feature space matching. The probability distribution of feature space matching is, for example, a Gaussian distribution. After determining the average image feature, the average image feature is used to initially adjust the third image feature, making the third image feature similar to the average image feature and improving the quality of the third image feature. Furthermore, the first image feature and the second image feature are input into a neural network to obtain output data of the neural network, which is also an image feature. The initially adjusted third image feature is then further adjusted based on the output data of the neural network to make the third image feature similar to the first image feature and the second image feature, improving the similarity between the third image feature and the first image feature and the second image feature.

[0068] In one embodiment, initially adjusting the third image feature based on the average image feature includes determining an average value of the third image feature and the average image feature, and determining that the initially adjusted third image feature is the average value, thereby achieving feature trimming (i.e., initial adjustment) for the third image feature in a manner of determining the average value of the third image feature and the average image feature.

[0069] In one embodiment, re-adjusting the initially adjusted third image feature based on the output data includes adding the output data and the initially adjusted third image feature to obtain the re-adjusted third image feature, thereby improving the similarity between the third image feature and the first and second image features by adding the feature deviation resulting from the initial adjustment process learned by the neural network to the initially adjusted third image feature.

[0070] S304: Perform nonlinear interpolation based on the first image feature, the second image feature and the adjusted third image feature to obtain a plurality of intermediate image features.

[0071] In this embodiment, after obtaining the first image feature, the second image feature, and the finally adjusted third image feature, an interpolation curve is obtained by using the first image feature, the second image feature, and the third image feature as three known quantities through a nonlinear interpolation method, and a plurality of intermediate image features are obtained by sampling on the interpolation curve, so that in addition to the first image feature and the second image feature, the third image feature, which has high quality and is relatively similar to the first image feature and the second image feature, is further utilized in the nonlinear interpolation process, thereby effectively improving the accuracy of the nonlinear interpolation and the quality of the intermediate image features.

[0072] In one possible implementation, the nonlinear interpolation method uses cubic spline interpolation, where S304 includes the steps of: obtaining an interpolation curve by cubic spline interpolation based on the first image feature, the second image feature, and the third image feature; and sampling the interpolation curve to obtain a plurality of intermediate image features. Thus, the use of cubic spline interpolation improves the accuracy of the nonlinear interpolation and the quality of the intermediate image features.

[0073] Specifically, the third image feature is input to the cubic spline interpolation together with the first and second image features to obtain an interpolation function, i.e., an interpolation curve. Further, sampling is performed on the interpolation curve to obtain multiple intermediate image features.

[0074] S305: Based on the first image features, the second image features and the plurality of intermediate image features, an image is reconstructed by an image generation model, and a target video is generated to show the process of gradual change from the first image to the second image.

[0075] The implementation principle and technical effect of S305 can be referred to the above embodiments and will not be described here.

[0076] In an embodiment of the present disclosure, based on the first image features obtained by encoding the first image and the second image features obtained by encoding the second image, nonlinear interpolation based on feature space and neural network is used to obtain multiple intermediate image features, which effectively improves the accuracy of the nonlinear interpolation, further improves the quality of the intermediate image features, improves the image quality of the intermediate frames of the video, and further improves the video quality.

[0077] For example, refer to FIG. 4, which illustrates an example of a nonlinear interpolation framework based on a feature space and a neural network provided by an embodiment of the present disclosure. As shown in FIG. 4, first, the average value (corresponding to the third image feature in this case) of latent code 1 (corresponding to the first image feature in this case) and latent code 2 (corresponding to the second image feature in this case) is determined, and the average value is trimmed based on the feature space to obtain the trimmed average value (corresponding to the initially adjusted third image feature in this case). Next, latent code 1 and latent code 2 are input into the neural network to obtain the feature deviation output by the neural network, and the feature deviation is added to the trimmed average value (corresponding to the adjusted third image feature in this case). In this way, latent code 1, latent code 2, and the average value are finally used in spline interpolation to obtain multiple interpolation results (i.e., multiple intermediate image features).

[0078] It should be noted that the above embodiment provides a method for adjusting image features in combination with feature space and neural network. In practical applications, image features may be adjusted solely based on feature space, i.e., the feature deviation caused by feature space adjustment is ignored.

[0079] In some embodiments, to improve the effect of nonlinear interpolation, a neural network can be pre-trained to learn the deviations that adjust image features based on the feature space of the image generation model. An example of neural network training is provided below.

[0080] It should be noted that the neural network training process and the video generation process of the above embodiment may be performed on the same device or on different devices.

[0081] 5, which is a process diagram of a model determination method provided by an embodiment of the present disclosure. As shown in FIG. 5, the model determination method includes step S501.

[0082] S501, based on a plurality of training images and an image generation model, training a neural network for learning deviations to adjust image features based on a feature space of the image generation model.

[0083] In one training process of the neural network, S501 includes steps S5011 to S5014.

[0084] S5011, generating a target image feature based on the image feature of the first training image and the image feature of the second training image.

[0085] In this embodiment, two training images can be obtained from multiple training images during each training process. For ease of distinction, the two training images are referred to as the first training image and the second training image, respectively. The two training images are encoded by an encoder to obtain image features of the first training image and the second training image. A feature fusion process is performed on the image features of the first training image and the image features of the second training image to obtain target image features.

[0086] In one example, performing a feature fusion process on the image features of the first training image and the image features of the second training image to obtain the target image feature includes determining an average value of the image features of the first training image and the image features of the second training image, where the average value is the target image feature. Specifically, the average value can be obtained by adding and then averaging the eigenvalues ​​of corresponding positions on the image features of the first training image and the image features of the second training image.

[0087] In another example, the target image feature is obtained by weighting the image features of the first training image and the image features of the second training image, and weights corresponding to the image features of the first training image and the image features of the second training image can be preset.

[0088] S5012, the target image features are initially adjusted based on the feature space.

[0089] In this embodiment, the average image feature in the feature space can be determined based on the probability distribution of feature space matching, and the average image feature can be used to initially adjust the target image feature so that the target image feature is similar to the average image feature, thereby improving the quality of the target image feature.

[0090] In one embodiment, initially adjusting the target image feature based on the average image feature includes determining an average value of the target image feature and the average image feature, and determining that the initially adjusted target image feature is the average value, thereby achieving feature trimming (i.e., initial adjustment) of the target image feature in a manner of determining the average value of the target image feature and the average image feature.

[0091] S5013, learning the target deviation corresponding to the initial adjustment by the neural network, and re-adjusting the initially adjusted target image feature according to the target deviation.

[0092] In this embodiment, the image features of the first training image and the image features of the second training image are input into a neural network to obtain output data of the neural network, i.e., the target deviation corresponding to the initial adjustment is obtained through learning. Based on the target deviation corresponding to the initial adjustment obtained through learning by the neural network, the initially adjusted target image features are adjusted again to make the target image features similar to the image features of the first training image and the image features of the second training image, i.e., to improve the similarity between the target image features and the image features of the first training image and the image features of the second training image.

[0093] In one embodiment, re-adjusting the initially adjusted target image features based on the target deviations includes adding the target deviations and the initially adjusted target image features to obtain re-adjusted target image features, thereby improving the similarity between the target image features and the image features of the first training image and the image features of the second training image by adding the feature deviations generated in the initial adjustment process learned by the neural network to the initially adjusted target image features.

[0094] S5014, adjust the model parameters of the neural network based on the target deviation, the re-adjusted target image features, the first training image and the second training image.

[0095] In this embodiment, a training error of the neural network can be determined based on the target deviation, the re-adjusted target image features, the first training image, and the second training image, and the model parameters of the neural network can be adjusted based on the training error. For example, the training error can be determined based on the difference between the re-adjusted target image features and the image features of the first training image and / or the difference between the re-adjusted target image features and the image features of the second training image.

[0096] In one example, a neural network is obtained by training regularity constraints and similarity constraints, where the regularity constraints are used to minimize the difference between image features adjusted based on the neural network and image features adjusted based on the feature space (i.e., the initially adjusted target image features), and the similarity constraints are used to minimize the difference between image features adjusted based on the neural network (i.e., the re-adjusted target image features) and the image features of a first training image and the image features of a second training image.

[0097] At this time, S5014 includes a step of determining a target optimization function of the neural network according to the regularity constraint and the similarity constraint, and a step of adjusting the model parameters of the neural network based on the target optimization function, the target deviation, the re-adjusted target image features, the first training image and the second training image.

[0098] Specifically, the target optimization function of the neural network can be predetermined based on the regularity constraint and the similarity constraint. During the training process of the neural network, the function value of the target optimization function, i.e., the training error of the neural network, is determined based on the target deviation, the first training image, and the second training image. The model parameters of the neural network are optimized based on the training error. The optimization algorithm is, for example, a gradient descent algorithm.

[0099] Specifically, the image features mentioned in the above embodiments are all image features after encoding. Therefore, to improve the accuracy of model training, after obtaining the adjusted target image features again, the target image features are input into the image generation model to obtain intermediate reconstructed images (i.e., reconstructed images corresponding to the target image features). Then, feature extraction is performed on the first training image, the second training image, and the intermediate reconstructed image using a feature extraction network, respectively, to obtain the image features of the first training image, the image features of the second training image, and the image features of the intermediate reconstructed image. For example, if the first training image, the second training image, and the intermediate reconstructed image are all face images, feature extraction can be performed on these images using a face feature extraction network. Next, the difference between the image features of the intermediate reconstructed image and the image features of the first training image (features extracted by the feature extraction network) and the difference between the image features of the intermediate reconstructed image and the image features of the second training image (features extracted by the feature extraction network) are determined, and the training error is determined based on the two differences and the output data of the neural network.

[0100] In one example, the target optimization function for a neural network can be shown as follows:

number

number

number

[0101] In this way, the above steps are repeatedly performed to train the neural network multiple times.

[0102] For example, refer to Figure 6, which is a schematic diagram of a neural network training framework provided by an embodiment of the present disclosure. As shown in Figure 6, the training process includes: first determining the average value of latent code 1 (image feature obtained after encoding input image 1) and latent code 2 (image feature obtained after encoding input image 2); performing feature trimming (i.e., initial adjustment) on the average value based on the feature space of the image generation model to obtain trimmed average values; inputting latent code 1 and latent code 2 into the neural network and determining a training error for the regularity constraint based on the feature deviation output by the neural network; adding the trimmed average value to the feature deviation output by the neural network and inputting the average value into the image generation model to obtain a reconstructed image; and finally, determining the feature difference between the reconstructed image and input image 1 and the feature difference between the reconstructed image and input image 2 using a feature extraction network, and determining a training error for the similarity constraint based on the two feature differences. In this way, the model parameters of the neural network are adjusted based on the training error of the regularity constraint part and the training error of the similarity constraint part.

[0103] Corresponding to the video generation method in the above embodiment, Fig. 7 is a structural block diagram of a video generation device provided by an embodiment of the present disclosure. For convenience of explanation, only parts relevant to the embodiment of the present disclosure are shown. Referring to Fig. 7, the video generation device includes an extraction unit 701 and an interpolation unit 702.

[0104] The extraction unit 701 is used for extracting a first image feature in a first image.

[0105] The interpolation unit 702 is used for obtaining a plurality of intermediate image features by nonlinear interpolation based on the first image feature and the second image feature, which is the image feature of the second image.

[0106] The video generation unit 703 is used to reconstruct an image according to an image generation model based on the first image features, the second image features and multiple intermediate image features, and generate a target video to show the gradual change process from the first image to the second image.

[0107] In some embodiments, the interpolation unit 702 is further used for: generating a third image feature based on the first image feature and the second image feature; adjusting the third image feature based on a feature space of the image generation model and a neural network for learning a deviation to adjust the image feature based on the feature space; and performing nonlinear interpolation based on the first image feature, the second image feature, and the adjusted third image feature to obtain a plurality of intermediate image features.

[0108] In some embodiments, the interpolation unit 702 is further used for obtaining an average image feature in the feature space; initially adjusting a third image feature based on the average image feature; inputting the first image feature and the second image feature into a neural network to obtain output data of the neural network reflecting the deviation of the initial adjustment; and again adjusting the initially adjusted third image feature based on the output data.

[0109] In some embodiments, the interpolation unit 702 is further adapted to determine an average value of the third image feature and the average image feature, and to determine that the initially adjusted third image feature is the average value.

[0110] In some embodiments, the neural network is obtained by training regularity constraints and similarity constraints, where the regularity constraints are used to minimize the difference between the image features trained based on the neural network and the image features trained based on the feature space, and the similarity constraints are used to minimize the difference between the image features trained based on the neural network and the image features of the first training image and the image features of the second training image.

[0111] In some embodiments, the interpolation unit 702 is further used for obtaining an interpolation curve by cubic spline interpolation based on the first image feature, the second image feature, and the third image feature, and sampling on the interpolation curve to obtain a plurality of intermediate image features.

[0112] In some embodiments, the image generation model is a StyleGAN model or a StyleGAN2 model.

[0113] The video generation device provided in this embodiment can be used to implement the technical solutions of the above embodiments related to the video generation method, and since the realization principles and technical effects thereof are similar, they will not be described here in this embodiment.

[0114] Corresponding to the model determination method in the above embodiment, Figure 8 is a structural block diagram of a model determination device provided by an embodiment of the present disclosure. For convenience of explanation, only parts relevant to the embodiment of the present disclosure are shown. Referring to Figure 8, the model determination device includes a training unit 801.

[0115] The training unit 801 is used to train a neural network based on a plurality of training images and an image generation model to learn deviations to adjust image features based on the feature space of the image generation model.

[0116] A single training process of the neural network includes the steps of generating target image features based on the image features of a first training image and the image features of a second training image, initially adjusting the target image features based on the feature space, learning target deviations corresponding to the initial adjustments using the neural network and re-adjusting the initially adjusted target image features based on the target deviations, and adjusting model parameters of the neural network based on the target deviations, the re-adjusted target image features, the first training image, and the second training image.

[0117] In some embodiments, the training unit 801 is further used to determine a target optimization function of the neural network according to a regularity constraint and a similarity constraint, and to adjust model parameters of the neural network based on the target optimization function, the target deviation, the readjusted target image features, the first training image, and the second training image, wherein the regularity constraint is used to minimize the difference between the readjusted target image features and the initial adjusted target image features, and the similarity constraint is used to minimize the difference between the readjusted target image features and the image features of the first training image and the image features of the second training image.

[0118] The model determination device provided in this embodiment can be used to implement the technical solutions of the above embodiments related to the model determination method, and since the realization principles and technical effects thereof are similar, they will not be described in this embodiment.

[0119] 9 shows a structural schematic diagram of an electronic device 900 suitable for implementing an embodiment of the present disclosure, which may be a terminal device or a server. Examples of terminal devices include, but are not limited to, mobile devices such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (portable Android devices, PADs), portable media players (PMPs), and in-vehicle devices (e.g., in-vehicle navigation devices), as well as fixed devices such as digital TVs and desktop computers. The electronic device shown in FIG. 9 is merely an example and should not limit the functionality and scope of use of the embodiment of the present disclosure.

[0120] 9, electronic device 900 may include a processing unit (e.g., a central processor, a graphics processor, etc.) 901, which may perform various appropriate operations and processes based on programs stored in read only memory (ROM) 902 or programs loaded from storage device 908 into random access memory (RAM) 903. Various programs and data necessary for the operation of electronic device 900 are further stored in RAM 903. Processing unit 901, ROM 902, and RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to bus 904.

[0121] Typically, the following devices can be connected to the I / O interface 905: input devices 906 including, for example, a touch panel, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 907 including, for example, a liquid crystal display (LCD), speaker, oscillator, etc.; storage devices 908 including, for example, a tape, hard disk, etc.; and communication devices 909. The communication devices 909 can allow the electronic device 900 to communicate wirelessly or via wires with other devices to exchange data. While FIG. 9 shows the electronic device 900 with various devices, it should be understood that it need not implement or include all of the devices shown. Alternatively, more or fewer devices may be implemented or included.

[0122] In particular, according to embodiments of the present disclosure, the processes described with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product, which includes a computer program loaded on a computer-readable medium, the computer program including program code for performing the methods illustrated in the flowcharts. In such embodiments, the computer program may be downloaded and installed from a network via the communication device 909, or may be installed from the storage device 908, or may be installed from the ROM 902. When the computer program is executed by the processing device 901, it performs the functions specified in the methods of the embodiments of the present disclosure.

[0123] The computer-readable medium of the present disclosure may be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media include, but are not limited to, an electrical connection having one or more conductors, a portable computer magnetic disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), optical fiber, a portable compact magnetic disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a propagated data signal, either in baseband or as part of a carrier wave, loaded with computer-readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which is capable of transmitting, propagating, or transporting a program for use by or in connection with an instruction execution system, apparatus, or device.The program code contained in the computer readable medium may be transmitted over any suitable medium, including, but not limited to, wire, cable, radio frequency (RF), etc., or any suitable combination thereof.

[0124] The computer-readable medium may be included in the electronic device, or may exist independently of the electronic device.

[0125] One or more programs are loaded into the computer-readable medium, and when the one or more programs are executed by the electronic device, the electronic device performs the method shown in the above embodiments.

[0126] Computer program code for carrying out the operations of the present disclosure can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and the like, and general procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computer, partially on the user computer, as a standalone software package, partially on the user computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider).

[0127] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of systems, methods, and computer program products, as well as possible implementations of computer programs, in various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code, which includes one or more executable instructions for implementing a specified logical function. It should be noted that in some alternative implementations, the functions marked in the blocks may be executed in an order different from the order marked in the drawings. For example, two blocks shown in succession may actually be executed substantially in parallel, or may be executed in the reverse order, depending on the functionality involved. It should be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system or a combination of dedicated hardware and computer instructions to perform the specified functions or operations.

[0128] The units according to the embodiments of the present disclosure may be implemented in a software manner or in a hardware manner, and the names of the units, in some cases, do not constitute limitations on the units themselves, for example, the acquisition unit may further be described as a "target audio acquisition unit."

[0129] The functionality described herein may be performed at least in part by one or more hardware logic components. For example, without limitation, examples of types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), etc.

[0130] In the context of this disclosure, a machine-readable medium may be a tangible medium, capable of containing or storing a program used by or in connection with an instruction execution system, device, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium includes, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the above. More specific examples of machine-readable storage media include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0131] According to a first aspect, based on one or more embodiments of the present disclosure, a video generation method is provided, comprising: extracting first image features in a first image; obtaining a plurality of intermediate image features by nonlinear interpolation based on the first image features and second image features that are image features of a second image; reconstructing an image by an image generation model based on the first image features, the second image features, and the plurality of intermediate image features; and generating a target video to show a gradual change process from the first image to the second image.

[0132] According to one or more embodiments of the present disclosure, the step of obtaining a plurality of intermediate image features by nonlinear interpolation based on the first image feature and the second image feature includes the steps of generating a third image feature based on the first image feature and the second image feature, sequentially adjusting the third image feature based on a feature space of the image generation model and a neural network for learning deviations for performing image feature adjustment based on the feature space, and performing nonlinear interpolation based on the first image feature, the second image feature, and the adjusted third image feature to obtain the plurality of intermediate image features.

[0133] According to one or more embodiments of the present disclosure, the step of adjusting the third image feature based on the feature space of the image generation model and a neural network for learning deviations for adjusting image features based on the feature space includes the steps of obtaining an average image feature in the feature space, initially adjusting the third image feature based on the average image feature, inputting the first image feature and the second image feature into the neural network to obtain output data of the neural network reflecting deviations of the initial adjustment, and again adjusting the initially adjusted third image feature based on the output data.

[0134] Based on one or more embodiments of the present disclosure, the step of initially adjusting the third image feature based on the average image feature includes the steps of determining an average value of the third image feature and the average image feature, and determining that the initially adjusted third image feature is the average value.

[0135] According to one or more embodiments of the present disclosure, the neural network is obtained by training regularity constraints and similarity constraints, the regularity constraints are used to minimize the difference between image features adjusted based on the neural network and image features adjusted based on the feature space, and the similarity constraints are used to minimize the difference between image features adjusted based on the neural network and image features of a first training image and image features of a second training image.

[0136] According to one or more embodiments of the present disclosure, the step of performing nonlinear interpolation based on the first image feature, the second image feature, and the third image feature to obtain the plurality of intermediate image features includes the steps of obtaining an interpolation curve by cubic spline interpolation based on the first image feature, the second image feature, and the third image feature, and sampling on the interpolation curve to obtain the plurality of intermediate image features.

[0137] According to one or more embodiments of the present disclosure, the image generation model is a StyleGAN model or a StyleGAN2 model.

[0138] According to a second aspect, in accordance with one or more embodiments of the present disclosure, there is provided a model determination method, comprising: training a neural network based on a plurality of training images and an image generation model to learn deviations for adjusting image features based on a feature space of the image generation model, wherein a single training process of the neural network includes: generating target image features based on image features of a first training image and image features of a second training image; initially adjusting the target image features based on the feature space; learning target deviations corresponding to the initial adjustments using the neural network and re-adjusting the initially adjusted target image features based on the target deviations; and adjusting model parameters of the neural network based on the target deviations, the re-adjusted target image features, the first training images, and the second training images.

[0139] According to one or more embodiments of the present disclosure, the step of adjusting the model parameters of the neural network based on the target deviation, the readjusted target image features, the first training images, and the second training images includes the steps of: determining a target optimization function of the neural network according to a regularity constraint and a similarity constraint; and adjusting the model parameters of the neural network based on the target optimization function, the target deviation, the readjusted target image features, the first training images, and the second training images, wherein the regularity constraint is used to minimize the difference between the readjusted target image features and the initial readjusted target image features, and the similarity constraint is used to minimize the difference between the readjusted target image features and the image features of the first training images and the image features of the second training images.

[0140] According to a third aspect, based on one or more embodiments of the present disclosure, a video generation device is provided, comprising: an extraction unit for extracting first image features in a first image; an interpolation unit for obtaining a plurality of intermediate image features by nonlinear interpolation based on the first image features and second image features that are image features of a second image; and a video generation unit for reconstructing an image by an image generation model based on the first image features, the second image features, and the plurality of intermediate image features, and generating a target video to show a gradual change process from the first image to the second image.

[0141] According to a fourth aspect, in accordance with one or more embodiments of the present disclosure, a model determination device is provided, comprising: a training unit for training a neural network, based on a plurality of training images and an image generation model, to learn deviations for adjusting image features based on a feature space of the image generation model. During one training process of the neural network, a training module is used to generate target image features based on image features of a first training image and image features of a second training image, initially adjust the target image features based on the feature space, learn target deviations corresponding to the initial adjustments using the neural network, and re-adjust the initially adjusted target image features based on the target deviations, and adjust model parameters of the neural network based on the target deviations, the re-adjusted target image features, the first training images, and the second training images.

[0142] According to a fifth aspect, in accordance with one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory; the memory stores computer-executable instructions; The at least one processor executes computer executable instructions stored in the memory to cause the at least one processor to perform the video generation method described in the first aspect or various possible designs of the first aspect, or to cause the at least one processor to perform the model determination method described in the second aspect or various possible designs of the second aspect.

[0143] According to a sixth aspect, in accordance with one or more embodiments of the present disclosure, there is provided a computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by a processor, results in the video generation method according to the first aspect or various possible designs of the first aspect, or the model determination method according to the second aspect or various possible designs of the second aspect.

[0144] According to a seventh aspect, there is provided a computer program product in accordance with one or more embodiments of the present disclosure, the computer program product including computer executable instructions that, when executed by a processor, implement the video generation method according to the first aspect or various possible designs of the first aspect, or the model determination method according to the second aspect or various possible designs of the second aspect.

[0145] According to an eighth aspect, there is provided a computer program according to one or more embodiments of the present disclosure, which, when executed by a processor, implements the video generation method according to the first aspect or various possible designs of the first aspect, or the model determination method according to the second aspect or various possible designs of the second aspect.

[0146] The above description merely describes the preferred embodiments and the technical principles used in the present disclosure. Those skilled in the art should understand that the scope of the present disclosure is not limited to the technical solution constituted by a specific combination of the above technical features, and should also include other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the concept of the above disclosure. For example, the above features may be substituted with (but not limited to) technical features having similar functions disclosed in the present disclosure.

[0147] Also, although operations have been described in a particular sequence, this should not be understood as requiring these operations to be performed in the particular sequence or order shown. In certain cases, multitasking and parallel processing may be advantageous. Similarly, although the above description includes several specific implementation details, these should not be construed as limitations on the scope of the disclosure. Certain features described in the context of a single embodiment can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments alone or in any suitable subcombination.

[0148] Although the present subject matter has been described using language specific to structural features and / or logical operations of methods, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or operations described above. Rather, the specific features and operations described above are merely example forms of implementing the claims.

Claims

1. 1. A video generation method comprising: extracting a first image feature in a first image; a step of obtaining a plurality of intermediate image features by nonlinear interpolation based on the first image feature and a second image feature that is an image feature of a second image, wherein the first image feature is an image feature obtained after encoding the first image, and the second image feature is an image feature obtained after encoding the second image; and reconstructing an image by an image generation model based on the first image feature, the second image feature, and the plurality of intermediate image features, and generating a target video to show a gradual change from the first image to the second image; The step of obtaining a plurality of intermediate image features by nonlinear interpolation based on the first image feature and the second image feature includes: performing a feature fusion process based on the first image feature and the second image feature to generate a third image feature; adjusting the third image feature in turn based on a feature space of the image generation model and a neural network for learning a deviation for adjusting the image feature based on the feature space, wherein the feature space of the image generation model includes a feature sample for adjusting the third image feature a first time, thereby making the first-time adjusted third image feature closer to the feature sample, and the neural network adjusts the first-time adjusted third image feature a second time to obtain an adjusted third image feature; and performing nonlinear interpolation based on the first image feature, the second image feature, and the adjusted third image feature to obtain the plurality of intermediate image features.

2. In turn, adjusting the third image feature based on a feature space of the image generation model and a neural network for learning a deviation for adjusting the image feature based on the feature space, obtaining a mean image feature in the feature space; initial adjustment of the third image feature based on the average image feature; inputting the first image feature and the second image feature into the neural network and obtaining output data of the neural network that reflects the initially adjusted deviation; and re-adjusting the initially adjusted third image feature based on the output data.

3. The step of initially adjusting the third image feature based on the average image feature includes: determining an average value of the third image feature and the average image feature; and determining that the initially adjusted third image feature is the mean value.

4. 4. The video generation method of claim 1, wherein the neural network is obtained by training regularity constraints and similarity constraints, the regularity constraints being used to minimize the difference between image features trained based on the neural network and image features trained based on the feature space, and the similarity constraints being used to minimize the difference between image features trained based on the neural network and image features of a first training image and image features of a second training image.

5. the step of performing nonlinear interpolation based on the first image feature, the second image feature, and the adjusted third image feature to obtain the plurality of intermediate image features includes: obtaining an interpolation curve by cubic spline interpolation based on the first image feature, the second image feature, and the third image feature; A method according to any one of claims 1 to 3, comprising the step of sampling on the interpolated curve to obtain the plurality of intermediate image features.

6. The video generation method according to any one of claims 1 to 3, wherein the image generation model is a StyleGAN model or a StyleGAN2 model.

7. The video generation method of claim 1, comprising: training the neural network based on a plurality of training images and the image generation model; A single training process of the neural network is as follows: generating target image features based on the image features of the first training image and the image features of the second training image; adjusting the target image features a first time based on the feature space of the image generation model; learning a target deviation corresponding to the first adjustment by the neural network, and adjusting the target image feature adjusted in the first adjustment for a second time according to the target deviation; and adjusting model parameters of the neural network based on the target deviation, the second adjusted target image features, the first training images, and the second training images.

8. adjusting model parameters of the neural network based on the target deviation, the second adjusted target image features, the first training image, and the second training image; determining a target optimization function of the neural network according to regularity and similarity constraints; adjusting model parameters of the neural network based on the target optimization function, the target deviation, the second adjusted target image features, the first training images, and the second training images; The video generation method of claim 7, wherein the regularity constraint is used to minimize the difference between the target image features adjusted the second time and the target image features adjusted the first time, and the similarity constraint is used to minimize the difference between the target image features adjusted the second time and the image features of the first training image and the image features of the second training image.

9. 1. A video production device comprising: an extraction unit for extracting a first image feature in the first image; an interpolation unit for obtaining a plurality of intermediate image features by nonlinear interpolation based on the first image feature and a second image feature that is an image feature of a second image, wherein the first image feature is an image feature obtained after encoding the first image, and the second image feature is an image feature obtained after encoding the second image; a video generation unit for reconstructing an image by an image generation model based on the first image feature, the second image feature, and the plurality of intermediate image features, and generating a target video for showing a gradual change process from the first image to the second image; The extraction unit further comprises performing a feature fusion process based on the first image feature and the second image feature to generate a third image feature; adjusting the third image feature in turn based on a feature space of the image generation model and a neural network for learning a deviation for adjusting the image feature based on the feature space, wherein the feature space of the image generation model includes a feature sample for adjusting the third image feature a first time, thereby making the first-time adjusted third image feature closer to the feature sample, and the neural network adjusts the first-time adjusted third image feature a second time to obtain an adjusted third image feature; and performing nonlinear interpolation based on the first image feature, the second image feature, and the adjusted third image feature to obtain the plurality of intermediate image features.

10. An electronic device comprising at least one processor and a memory, the memory stores computer-executable instructions; The at least one processor executes computer-executable instructions stored in the memory, thereby causing the at least one processor to perform the video generation method of any one of claims 1 to 3 or 7 to 8.

11. A computer-readable storage medium having stored thereon computer-executable instructions that, when executed by a processor, implement the video generation method of any one of claims 1 to 3 or 7 to 8.

12. A computer program product which, when executed by a processor, implements the video generation method according to any one of claims 1 to 3 or 7 to 8.

Images