Video generating method and device

Abstract

An embodiment of the present disclosure provides a video generating method and apparatus, which includes: extracting a first image feature in a first image; obtaining a plurality of intermediate image features by nonlinear interpolation according to the first image feature and a second image feature, which is an image feature of a second image; reconstructing an image by an image generation model according to the first image feature, the second image feature and the plurality of intermediate image features, and generating a target video for showing a gradual change process from the first image to the second image, thereby improving the quality of the intermediate image feature by nonlinear interpolation, and further improving the image quality of the intermediate frames of the target video, i.e., improving the video quality of the target video.

Description

[Technical field]

[0001] This application claims priority to a Chinese patent application filed with the China Patent Office on December 24, 2021, bearing application number 202111609441.8 and titled "Video Generation Method and Apparatus," the entire contents of which are incorporated herein by reference.

[0002] TECHNICAL FIELD Embodiments of the present disclosure relate to the field of computer technology, and more particularly to video generation methods and apparatus. [Background technology]

[0003] In conventional computer vision technology, deep learning technology, a video that changes gradually between two images can be generated based on two images, for example, two face images are input into a deep learning model to generate a video that changes gradually between the two face images, and the video frames in the video change gradually from one face image to another. This technology can be used, for example, to generate special effects and improve the fun of the video.

[0004] In the process of video generation, it is necessary to generate multiple video frames in a video based on two images, and obtain a video that changes gradually between the two images. However, currently, there is a need to improve the quality of the multiple video frames generated based on two images, especially the image quality of the intermediate frames of the video. Summary of the Invention [Means for solving the problem]

[0005] SUMMARY OF THE DISCLOSURE The embodiments of the present disclosure provide a video generating method and apparatus to solve the problem of the need to improve the image quality of intermediate frames of a video when generating a video based on a small number of images.

[0006] According to a first aspect, an embodiment of the present disclosure provides a video generation method, comprising: Extracting a first image feature in a first image; obtaining a plurality of intermediate image features by nonlinear interpolation based on the first image feature and a second image feature, the second image feature being an image feature of a second image; and a step of reconstructing an image using an image generation model based on the first image features, the second image features and the plurality of intermediate image features, and generating a target video to show the process of gradual change from the first image to the second image.

[0007] According to a second aspect, an embodiment of the present disclosure provides a method for determining a model, the method comprising: training a neural network with a plurality of training images and an image generation model to learn deviations to adjust image features based on a feature space of the image generation model; A single training process of the neural network includes: generating a target image feature based on the image features of the first training image and the image features of the second training image; initializing the target image features based on the feature space; learning a target deviation corresponding to the initial adjustment by the neural network, and re-adjusting the initially adjusted target image feature according to the target deviation; and adjusting model parameters of the neural network based on the target deviation, the re-adjusted target image features, the first training images, and the second training images.

[0008] According to a third aspect, an embodiment of the present disclosure provides a video generation device, comprising: an extraction unit for extracting a first image feature in the first image; an interpolation unit for obtaining a plurality of intermediate image features by nonlinear interpolation based on the first image feature and a second image feature, the second image feature being an image feature of a second image; and a video generation unit for reconstructing an image using an image generation model based on the first image features, the second image features and the multiple intermediate image features, and generating a target video for showing a gradual change process from the first image to the second image.

[0009] According to a fourth aspect, an embodiment of the present disclosure provides a model determination apparatus, comprising: A training unit for training a neural network with a plurality of training images and an image generation model to learn deviations for adjusting image features based on a feature space of the image generation model; A single training process of the neural network includes: generating a target image feature based on the image features of the first training image and the image features of the second training image; initializing the target image features based on the feature space; learning a target deviation corresponding to the initial adjustment by the neural network, and re-adjusting the initially adjusted target image feature according to the target deviation; and adjusting model parameters of the neural network based on the target deviation, the re-adjusted target image features, the first training images, and the second training images.

[0010] According to a fifth aspect, an embodiment of the present disclosure provides an electronic device comprising at least one processor and a memory, The memory stores computer executable instructions; The at least one processor executes computer executable instructions stored in the memory to cause the at least one processor to perform the video generation method described in the first aspect or various possible designs of the first aspect, or to cause the at least one processor to perform the model determination method described in the second aspect or various possible designs of the second aspect.

[0011] According to a sixth aspect, an embodiment of the present disclosure provides a computer readable storage medium having computer executable instructions stored thereon, the computer readable storage medium being adapted to, when executed by a processor, realise the video generation method as set forth in the first aspect or various possible designs of the first aspect, or to realise the model determination method as set forth in the second aspect or various possible designs of the second aspect.

[0012] According to a seventh aspect, there is provided a computer program product in accordance with one or more embodiments of the present disclosure, the computer program product including computer executable instructions, which, when executed by a processor, realises the video generation method as set forth in the first aspect or in any of its various possible designs, or realises the model determination method as set forth in the second aspect or in any of its various possible designs.

[0013] According to an eighth aspect, there is provided a computer program in accordance with one or more embodiments of the present disclosure, which, when executed by a processor, realizes the video generation method according to the first aspect or various possible designs of the first aspect, or realizes the model determination method according to the second aspect or various possible designs of the second aspect.

[0014] The video generating method and device provided in this embodiment obtain a plurality of intermediate image features by nonlinear interpolation based on the first image feature of the first image and the second image feature of the second image, reconstruct an image by an image generation model based on the first image feature, the second image feature and the plurality of intermediate image features, and generate a target video for showing the gradual change process from the first image to the second image, thereby improving the quality of the intermediate image features by nonlinear interpolation, and improving the image quality of the intermediate frames of the target video based on ensuring the similarity between the intermediate frames of the target video and the first image and the second image, and further improving the video quality of the target video.

[0015] In order to more clearly describe the embodiments of the present disclosure or the technical solutions of the prior art, the following briefly introduces drawings that need to be used in the description of the embodiments or the prior art. It is obvious that the drawings in the following description are only some embodiments of the present disclosure, and those skilled in the art can obtain other drawings based on these drawings on the premise that no creative work is required. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a schematic diagram of an application scene to which an embodiment of the present disclosure is applied. [Diagram 2] FIG. 2 is a process diagram 1 of a video generating method provided by an embodiment of the present disclosure. [Figure 3a] FIG. 3a is a process diagram 2 of a video generating method provided by an embodiment of the present disclosure. [Figure 3b] FIG. 3b is a process diagram of adjusting a third image feature based on the feature space of an image generation model and a neural network in the order provided by an embodiment of the present disclosure. [Figure 4] FIG. 4 is a diagram illustrating an example of a framework for nonlinear interpolation based on feature space and neural network provided by an embodiment of the present disclosure. [Diagram 5] FIG. 5 is a process diagram of a model determination method provided by an embodiment of the present disclosure. [Figure 6] FIG. 6 is a schematic diagram of a neural network training framework provided by an embodiment of the present disclosure. [Figure 7] FIG. 7 is a structural block diagram of a video generating device provided by an embodiment of the present disclosure. [Figure 8] FIG. 8 is a structural block diagram of a model determination device provided by an embodiment of the present disclosure. [Figure 9] FIG. 9 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] In order to make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions of the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings of the embodiments of the present disclosure. It is obvious that the described embodiments are only some of the embodiments of the present disclosure, and not all of the embodiments. Based on the embodiments of the present disclosure, any other embodiments that a person skilled in the art can obtain without requiring creative labor shall fall within the scope of protection of the present disclosure.

[0018] When generating a video with gradual changes between two input images, generally, linear interpolation is performed on the image features of the two input images to obtain intermediate image features, and intermediate frames of the video are generated using the intermediate image features. Although this method can ensure the continuity and similarity of the video frames, the intermediate image features after linear interpolation often do not match the distribution rule (or change rule) of the image features of the video screen in the actual video, so the image quality of the intermediate frames is poor, and the aesthetics and realism are insufficient.

[0019] In order to solve the above problem, the embodiment of the present disclosure provides a video generation method and apparatus, which obtains a plurality of intermediate image features by nonlinear interpolation based on a first image feature of a first image and a second image feature of a second image, and reconstructs an image by an image generation model based on the first image feature, the second image feature, and the plurality of intermediate image features to generate a target video. The target video is used to show the process of gradually changing from a first image to a second image. Since the change process of an actual video scene is a nonlinear change, compared with the linear interpolation method, the embodiment of the present disclosure uses nonlinear interpolation to improve the quality of the intermediate image features and improve the quality of the intermediate frames of the target video, so that the video scene of the target video changes nonlinearly, and the realism and aesthetics are higher. Improving the quality of the intermediate image features includes improving the realism of the intermediate image features and improving the similarity between the intermediate images and the first image and the second image. Improving the quality of the intermediate frames of the target video includes improving the aesthetics and realism of the intermediate frames and improving the similarity between the intermediate frames and the first image and the second image.

[0020] Referring to FIG. 1, FIG. 1 is a schematic diagram of an application scenario to which the embodiments of the present disclosure are applied.

[0021] As shown in Fig. 1, in this application scenario, the relevant devices include a video generating device 101, which may be a terminal or a server, and Fig. 1 takes the video generating device 101 as an example of a server. The video generating device 101 can process two images and generate a video to show the gradual change effect between the two images.

[0022] In one embodiment, the device related to the application scene further includes an image collection device 102, and the image collection device 102 can also be a terminal or a server, for example, the terminal collects images input by a user, or the terminal collects images in a current scene by a camera, or the server collects images from a network that are published on the network and allowed for public use. Figure 1 takes the image collection device 102 as an example being a terminal. The image collection device 102 transmits the collected images to a video generation device 101, and the video generation device 101 generates a video for showing a gradual change from the collected image to another image (from the image collection device 102 or other devices), or a video for showing a gradual change from another image to the collected image.

[0023] The video generation device 101 and the image acquisition device 102 may be the same device or may be different devices.

[0024] When the video generating device 101 and the image collecting device 102 are the same device, for example, a user takes a selfie with a mobile phone, obtains a selfie avatar, and selects another image from the mobile phone, and the mobile phone generates a video based on the user's selfie avatar and the image selected by the user, and the video content of the video gradually changes from the user's selfie avatar to the image selected by the user.

[0025] When the video generating device 101 and the image collecting device 102 are different devices, for example, a user takes a selfie with a mobile phone, obtains a selfie avatar, and selects another image from the mobile phone, the mobile phone sends the selfie image and the image selected by the user to a server, the server generates a video and sends the video back to the mobile phone, and the video content of the video gradually changes from the user's selfie avatar to the image selected by the user.

[0026] The terminal may be a personal digital assistant (PDA) device, a mobile device (e.g., a smartphone, a tablet computer), a computing device (e.g., a personal computer (PC)), an in-vehicle device, a wearable device (e.g., a smart watch, a smart bracelet), a smart home device (e.g., a smart display device), etc. The server may be a distributed server, a centralized server, a cloud server, etc.

[0027] The following provides several embodiments of the present disclosure. The execution body of the several embodiments of the present disclosure may be an electronic device, and the electronic device may be a terminal or a server.

[0028] 2, which is a process diagram 1 of a video generating method provided by an embodiment of the present disclosure. As shown in FIG. 2, the video generating method includes steps 201 to 203.

[0029] S201: extracting a first image feature from a first image.

[0030] The first image may be an image input by a user, an image from another device, or an image captured by the current executing device. For example, if the current executing device is a terminal, the terminal can obtain the first image input by a user, or obtain the first image captured by a camera of the terminal. Also, for example, if the current executing device is a server, the server can receive the first image input by a user sent by the terminal.

[0031] The first image feature is an image feature of the first image.

[0032] In this embodiment, an encoder encodes a first image and obtains a first image feature, where the first image feature specifically refers to the image feature obtained after encoding the first image.

[0033] S202, based on the first image feature and the second image feature, which is the image feature of the second image, obtain a plurality of intermediate image features by nonlinear interpolation.

[0034] The second image is a different image from the first image.

[0035] The second image feature specifically refers to the image feature obtained after encoding the second image.

[0036] In one example, a plurality of images and image features obtained after encoding the plurality of images can be stored in advance. A second image feature is obtained from the image features of the stored plurality of images. In one method, a user can specify a second image from the plurality of images stored in advance, and obtain the image features of the second image, i.e., the second image feature, from the image features of the plurality of images; in another method, the second image feature can be obtained from the image features of the plurality of images in a preset order (e.g., image storage order) or randomly.

[0037] For example, in response to a user's operation of inputting a first image, a plurality of images selected by the user are displayed on the terminal, the user selects a second image from the plurality of images, and a request is input to the terminal to generate a video that gradually changes from the first image to the second image, and the terminal responds to the request and obtains image features of the second image, i.e., the second image features, from image features of the plurality of images previously stored.

[0038] In another example, a second image input by a user, transmitted by another device, or captured by the current executing device can be obtained, and the second image can be encoded to obtain second image features.

[0039] In this embodiment, after obtaining the second image feature, a preset nonlinear interpolation method is used to perform nonlinear interpolation using the first image feature and the second image feature as two known quantities in the nonlinear interpolation process to obtain an interpolation function, i.e., an interpolation curve. In the interpolation curve, sampling is performed between the points corresponding to the first image feature and the points corresponding to the second image feature to obtain a plurality of intermediate image features. The intermediate image features are used to generate intermediate frames of the video.

[0040] In one embodiment, equally spaced sampling in the interpolation curve results in closer variation between adjacent intermediate image features obtained by the interpolation, improving the quality of the subsequently generated video.

[0041] S203: reconstructing an image according to the first image feature, the second image feature and the multiple intermediate image features by an image generation model, and generating a target video to show the process of gradual change from the first image to the second image.

[0042] The image generation model may be a neural network for image generation or image reconstruction, whose input data are encoded image features and whose output data is a reconstructed image. A trained image generation model published to the network may be used, or the neural network may be trained by training data (including multiple training images) to obtain an image generation model, and there is no restriction on the training process of the model.

[0043] In this embodiment, after the multiple intermediate image features are obtained, the first image feature, the second image feature, and the multiple intermediate image features are input into the image generation model respectively to obtain a reconstructed image corresponding to the first image feature, a reconstructed image corresponding to the second image feature, and a reconstructed image corresponding to each intermediate image feature respectively. According to the distribution order of the first image feature, the second image feature, and the intermediate image feature in the interpolation curve, the multiple reconstructed images are ordered and combined to obtain a target video. In the target video, the image of the first frame is the reconstructed image corresponding to the first image feature, the image of the last frame is the reconstructed image corresponding to the second image feature, and the intermediate frame is the reconstructed image corresponding to the intermediate image feature.

[0044] In an embodiment of the present disclosure, nonlinear interpolation is performed based on first image features obtained by encoding a first image and second image features obtained by encoding a second image to obtain multiple intermediate image features, an image is reconstructed using an image generation model based on the first image features, the second image features, and the multiple intermediate image features, and a reconstructed image is output based on the image generation model.

[0045] Therefore, the method of performing nonlinear interpolation based on the image features obtained by encoding two images improves the authenticity of the intermediate image features obtained by the interpolation and the similarity between the intermediate image features and the image features of the two first images, further improves the authenticity and aesthetics of the intermediate frames of the video, and improves the similarity between the intermediate frames and the images of the first and last frames, thereby improving the video quality.

[0046] For the image generation model, we have several possible implementations:

[0047] In some embodiments, the image generation model is a generative adversarial network (GAN), thereby utilizing the advantages of GAN in terms of image generation to improve the quality of image reconstruction of the image generation model and improve the quality of image frames of the target video.

[0048] In some embodiments, the image generation model is a style-based architecture for GANs (StyleGAN) model or a StyleGAN2 model, thereby utilizing the advantages of the StyleGAN model or the StyleGAN2 model in terms of image generation to improve the quality of image reconstruction of the image generation model and improve the quality of the image frames of the target video.

[0049] In some embodiments, the nonlinear interpolation process can be assisted by using the feature space of the image generation model and a neural network. The following examples will explain the assisted process.

[0050] Please refer to Fig. 3a, which is a process diagram 2 of a video generating method provided by an embodiment of the present disclosure. As shown in Fig. 3a, the video generating method includes steps S301 to S305.

[0051] S301, extract a first image feature in a first image.

[0052] The implementation principle and technical effects of S301 can be referred to the above embodiments and will not be described here.

[0053] S302, generating a third image feature based on the first image feature and a second image feature that is an image feature of the second image.

[0054] The process of acquiring the second image feature can be referred to the above embodiment and will not be described here.

[0055] In one example, an average value of the first image feature and the second image feature is determined, and the average value is the third image feature. Specifically, the feature values ​​of the corresponding positions of the first image feature and the second image feature are added and then averaged to obtain the average value of the first image feature and the second image feature.

[0056] In another example, the first image feature and the second image feature are weighted together to obtain a third image feature, where weights corresponding to the first image feature and the second image feature may be preset.

[0057] S303, in turn, adjust a third image feature based on the feature space of the image generation model and a neural network for learning deviations to adjust the image feature based on the feature space.

[0058] The feature space of the image generation model can be understood as the input space of the image generation model, where feature samples fit a given probability distribution.

[0059] In one embodiment, when the image generation model is a generative adversarial network, the feature space of the image generation model is a latent space corresponding to the image generation model, and the image features obtained by encoding the first image and the second image by the encoder are latent codes, i.e., the first image features can be referred to as the first latent code, and the second image features can be referred to as the second latent code.

[0060] In this embodiment, after the third image feature is obtained, the third image feature can be first adjusted based on the feature samples in the feature space of the image generation model, thereby making the third image feature closer to the feature samples in the feature space, thereby improving the image quality of the reconstructed image obtained by reconstructing an image based on the third image feature, i.e., improving the image quality of the intermediate frame.

[0061] In this embodiment, it is considered that there may be a certain deviation in adjusting the third image feature based on the feature space, which may reduce the similarity between the third image feature and the first image feature and the second image feature; therefore, to solve the problem, after adjusting the third image feature based on the feature space, the third image feature is adjusted again by a neural network model, thereby improving the similarity between the third image feature and the first image feature and the second image feature.

[0062] By training the neural network, it is necessary that the neural network can learn the deviation to adjust the image features based on the feature space. The specific training process is referred to the following embodiment.

[0063] In one embodiment, the neural network is a full-link neural network, so that the learning task of the neural network is unified, and when the input data and the output data are both image features, the full-link neural network with many network parameters improves the accuracy of adjusting the third image feature.

[0064] In one possible implementation manner, refer to Fig. 3b, which is a process schematic diagram of adjusting the third image feature based on the feature space of the image generation model and the neural network in sequence provided by the embodiment of the present disclosure (i.e., S303). As shown in Fig. 3b, the process of adjusting the third image feature based on the feature space of the image generation model and the neural network in sequence (i.e., one possible implementation manner of S303) includes steps S3031 to S3034.

[0065] S3031, obtain an average image feature in the feature space; S3032, initially adjust a third image feature based on the average image feature; S3033, input the first image feature and the second image feature into a neural network, and obtain output data of the neural network reflecting the deviation of the initial adjustment; S3034, re-adjust the initially adjusted third image feature based on the output data.

[0066] The output data of the neural network reflects the feature deviations that occur after initial tuning of the third image feature based on the average image feature in the feature space.

[0067] In this embodiment, the average image feature in the feature space can be determined based on the probability distribution of the matching of the feature space. The probability distribution of the matching of the feature space is, for example, a Gaussian distribution. After the average image feature is determined, the average image feature is used to initially adjust the third image feature, so that the third image feature is similar to the average image feature, and the quality of the third image feature is improved. Furthermore, the first image feature and the second image feature are input into a neural network to obtain output data of the neural network, and the output data of the neural network is also an image feature. Based on the output data of the neural network, the initially adjusted third image feature is again adjusted to make the third image feature similar to the first image feature and the second image feature, and the similarity between the third image feature and the first image feature and the second image feature is improved.

[0068] In one embodiment, adjusting the third image feature based on the average image feature includes determining an average value of the third image feature and the average image feature, and determining that the adjusted third image feature is the average value, thereby achieving feature trimming (i.e., adjustment) for the third image feature in the manner of determining the average value of the third image feature and the average image feature.

[0069] In one embodiment, re-adjusting the initially adjusted third image feature based on the output data includes adding the output data and the initially adjusted third image feature to obtain a re-adjusted third image feature, thereby improving the similarity between the third image feature and the first image feature and the second image feature by adding the feature deviation according to the initial adjustment process learned by the neural network to the initially adjusted third image feature.

[0070] S304, performing nonlinear interpolation based on the first image feature, the second image feature and the adjusted third image feature to obtain a plurality of intermediate image features.

[0071] In this embodiment, after obtaining the first image feature, the second image feature and the finally adjusted third image feature, the first image feature, the second image feature and the third image feature are taken as three known quantities, an interpolation curve is obtained by a nonlinear interpolation method, and a plurality of intermediate image features are obtained by sampling on the interpolation curve, so that besides the first image feature and the second image feature, the third image feature, which has high quality and is relatively similar to the first image feature and the second image feature, is further utilized in the nonlinear interpolation process, so that the accuracy of the nonlinear interpolation is effectively improved and the quality of the intermediate image features is also improved.

[0072] In one possible implementation, the nonlinear interpolation method uses cubic spline interpolation, where S304 includes the steps of: obtaining an interpolation curve by cubic spline interpolation according to the first image feature, the second image feature and the third image feature; and sampling on the interpolation curve to obtain a plurality of intermediate image features. Thus, the accuracy of the nonlinear interpolation is improved by using the cubic spline interpolation, and the quality of the intermediate image features is improved.

[0073] Specifically, the third image feature is input to the cubic spline interpolation together with the first image feature and the second image feature to obtain an interpolation function, that is, an interpolation curve. Further, sampling is performed on the interpolation curve to obtain a plurality of intermediate image features.

[0074] S305: reconstructing an image by an image generation model based on the first image features, the second image features and the multiple intermediate image features, and generating a target video to show the process of gradual change from the first image to the second image.

[0075] The implementation principle and technical effect of S305 can be referred to the above embodiments and will not be described here.

[0076] In an embodiment of the present disclosure, based on a first image feature obtained by encoding a first image and a second image feature obtained by encoding a second image, a nonlinear interpolation based on a feature space and a neural network is used to obtain multiple intermediate image features, thereby effectively improving the accuracy of the nonlinear interpolation, further improving the quality of the intermediate image features, improving the image quality of the intermediate frames of the video, and further improving the video quality.

[0077] For illustration, refer to FIG. 4, which is a diagram showing an example of a nonlinear interpolation framework based on feature space and neural network provided by an embodiment of the present disclosure. As shown in FIG. 4, firstly determine the average value (corresponding to the third image feature in this case) of latent code 1 (corresponding to the first image feature in this case) and latent code 2 (corresponding to the second image feature in this case), trim the average value based on the feature space to obtain the trimmed average value (corresponding to the initially adjusted third image feature in this case), then input latent code 1 and latent code 2 into the neural network to obtain the feature deviation output by the neural network, and add the feature deviation to the trimmed average value (corresponding to obtaining the adjusted third image feature again in this case). In this way, finally, latent code 1, latent code 2 and the average value are used for spline interpolation to obtain multiple interpolation results (i.e., multiple intermediate image features).

[0078] It should be noted that the above embodiment provides a method for adjusting image features in combination with feature space and neural network. In practical applications, image features may be adjusted solely based on feature space, that is, the feature deviation caused by the adjustment of feature space is ignored.

[0079] In some embodiments, in order to improve the effect of nonlinear interpolation, the neural network can be pre-trained so that the neural network learns the deviation to adjust the image features based on the feature space of the image generation model. An embodiment of neural network training is provided below.

[0080] It should be noted that the neural network training process and the video generation process of the above embodiment may be performed on the same device or on different devices.

[0081] Please refer to Fig. 5, which is a process diagram of a model determination method provided by an embodiment of the present disclosure. As shown in Fig. 5, the model determination method includes step S501.

[0082] S501, training a neural network for learning deviations to adjust image features based on a feature space of the image generation model according to a plurality of training images and an image generation model.

[0083] In one training process of the neural network, S501 includes steps S5011 to S5014.

[0084] S5011, generating a target image feature based on the image feature of the first training image and the image feature of the second training image.

[0085] In this embodiment, two training images can be obtained from multiple training images in each training process, and for convenience of distinction, the two training images are respectively called the first training image and the second training image. The two training images are encoded by an encoder to obtain the image features of the first training image and the image features of the second training image. The image features of the first training image and the image features of the second training image are subjected to a feature fusion process to obtain a target image feature.

[0086] In one example, performing feature fusion processing on the image features of the first training image and the image features of the second training image to obtain the target image feature includes determining an average value of the image features of the first training image and the image features of the second training image, the average value being the target image feature. Specifically, eigenvalues ​​of corresponding positions on the image features of the first training image and the image features of the second training image can be added and then averaged to obtain the average value.

[0087] In another example, the image features of the first training image and the image features of the second training image are weighted together to obtain the target image feature, and weights corresponding to the image features of the first training image and the image features of the second training image, respectively, can be preset.

[0088] S5012, the target image features are initially adjusted based on the feature space.

[0089] In this embodiment, an average image feature in the feature space can be determined based on the probability distribution of feature space matching, and the average image feature can be used to initially adjust the target image feature so that the target image feature is similar to the average image feature, thereby improving the quality of the target image feature.

[0090] In one embodiment, adjusting the target image feature based on the average image feature includes determining an average value of the target image feature and the average image feature, and determining that the adjusted target image feature is the average value, thereby achieving feature trimming (i.e., adjustment) of the target image feature in a manner of determining the average value of the target image feature and the average image feature.

[0091] S5013, learning a target deviation corresponding to the initial adjustment by a neural network, and re-adjusting the initially adjusted target image feature according to the target deviation.

[0092] In this embodiment, the image features of the first training image and the image features of the second training image are input into a neural network to obtain output data of the neural network, i.e., the target deviation corresponding to the initial adjustment is learned and obtained. Based on the target deviation corresponding to the initial adjustment learned and obtained by the neural network, the initially adjusted target image features are adjusted again to make the target image features similar to the image features of the first training image and the image features of the second training image, i.e., the similarity between the target image features and the image features of the first training image and the image features of the second training image is improved.

[0093] In one embodiment, re-adjusting the initially adjusted target image feature based on the target deviation includes adding the target deviation and the initially adjusted target image feature to obtain a re-adjusted target image feature, thereby improving the similarity between the target image feature and the image feature of the first training image and the image feature of the second training image in a manner of adding the feature deviation generated in the initial adjustment process learned by the neural network to the initially adjusted target image feature.

[0094] S5014, adjust model parameters of the neural network based on the target deviation, the re-adjusted target image features, the first training image and the second training image.

[0095] In this embodiment, a training error of the neural network can be determined based on the target deviation, the re-adjusted target image features, the first training image, and the second training image, and the model parameters of the neural network can be adjusted based on the training error, for example, the training error is determined based on the difference between the re-adjusted target image features and the image features of the first training image and / or the difference between the re-adjusted target image features and the image features of the second training image.

[0096] In one example, a neural network is obtained by training regularity constraints and similarity constraints, where the regularity constraints are used to minimize the difference between the image features adjusted based on the neural network and the image features adjusted based on the feature space (i.e., the initial adjusted target image features), and the similarity constraints are used to minimize the difference between the image features adjusted based on the neural network (i.e., the re-adjusted target image features) and the image features of a first training image and the image features of a second training image.

[0097] At this time, S5014 includes a step of determining a target optimization function of the neural network according to the regularity constraint and the similarity constraint, and a step of adjusting model parameters of the neural network based on the target optimization function, the target deviation, the re-adjusted target image features, the first training image and the second training image.

[0098] Specifically, the target optimization function of the neural network can be pre-determined according to the regularity constraint and the similarity constraint. During the training process of the neural network, the function value of the target optimization function, that is, the training error of the neural network, is determined according to the target deviation, the first training image and the second training image. The model parameters of the neural network are optimized according to the training error. The optimization algorithm is, for example, a gradient descent algorithm.

[0099] Specifically, the image features mentioned in the above embodiments are all image features after encoding. Therefore, in order to improve the accuracy of model training, after obtaining the adjusted target image features again, the target image features are input into the image generation model respectively to obtain intermediate reconstructed images (i.e., reconstructed images corresponding to the target image features), and then feature extraction is performed on the first training image, the second training image and the intermediate reconstructed image respectively by the feature extraction network, so as to obtain the image features of the first training image, the image features of the second training image and the image features of the intermediate reconstructed image. For example, if the first training image, the second training image and the intermediate reconstructed image are all face images, the face feature extraction network can be used to perform feature extraction on these images. Then, the difference between the image features of the intermediate reconstructed image and the image features of the first training image (features extracted by the feature extraction network), and the difference between the image features of the intermediate reconstructed image and the image features of the second training image (features extracted by the feature extraction network) are determined, and the training error is determined according to the two differences and the output data of the neural network.

[0100] In one example, the target optimization function for a neural network can be shown as follows:

number

number

number

[0101] In this manner, the above steps are repeatedly performed to train the neural network multiple times.

[0102] For example, refer to Fig. 6, which is a schematic diagram of the training framework of a neural network provided by an embodiment of the present disclosure. As shown in Fig. 6, the training process includes: firstly determining the average value of latent code 1 (image feature obtained after encoding input image 1) and latent code 2 (image feature obtained after encoding input image 2); according to the feature space of the image generation model, feature trimming (i.e., initial adjustment) is performed on the average value to obtain a trimmed average value; then inputting latent code 1 and latent code 2 into a neural network, and according to the feature deviation output by the neural network, a training error of the part called regularity constraint can be determined; adding the feature deviation output by the neural network to the trimmed average value, and inputting the average value into the image generation model to obtain a reconstructed image; finally, determining the feature difference between the reconstructed image and input image 1 and the feature difference between the reconstructed image and input image 2 by a feature extraction network, and according to the two feature differences, determining the training error of the part called similarity constraint. In this way, the model parameters of the neural network are adjusted based on the training error of the regularity constraint portion and the training error of the similarity constraint portion.

[0103] Corresponding to the video generation method in the above embodiment, FIG. 7 is a structural block diagram of a video generation device provided by an embodiment of the present disclosure. For convenience of explanation, only parts related to the embodiment of the present disclosure are shown. Referring to FIG. 7, the video generation device includes an extraction unit 701 and an interpolation unit 702.

[0104] The extraction unit 701 is used for extracting a first image feature in a first image.

[0105] The interpolation unit 702 is used for obtaining a plurality of intermediate image features through nonlinear interpolation based on the first image feature and the second image feature, which is the image feature of the second image.

[0106] The video generation unit 703 is used to reconstruct an image by an image generation model based on the first image features, the second image features and multiple intermediate image features, and generate a target video to show the process of gradual change from the first image to the second image.

[0107] In some embodiments, the interpolation unit 702 is further used for generating a third image feature based on the first image feature and the second image feature, adjusting the third image feature based on a feature space of the image generation model and a neural network for learning deviations to adjust the image feature based on the feature space, and performing nonlinear interpolation based on the first image feature, the second image feature, and the adjusted third image feature to obtain a plurality of intermediate image features.

[0108] In some embodiments, the interpolation unit 702 is further used for obtaining an average image feature in the feature space; initially adjusting a third image feature based on the average image feature; inputting the first image feature and the second image feature into a neural network to obtain output data of the neural network reflecting a deviation of the initial adjustment; and again adjusting the initially adjusted third image feature based on the output data.

[0109] In some embodiments, the interpolation unit 702 is further adapted to determine an average value of the third image feature and the average image feature, and to determine that the initially adjusted third image feature is the average value.

[0110] In some embodiments, the neural network is obtained by training on regularity constraints and similarity constraints, where the regularity constraints are used to minimize the difference between the image features trained based on the neural network and the image features trained based on the feature space, and the similarity constraints are used to minimize the difference between the image features trained based on the neural network and the image features of a first training image and the image features of a second training image.

[0111] In some embodiments, the interpolation unit 702 is further used for obtaining an interpolation curve by cubic spline interpolation based on the first image feature, the second image feature, and the third image feature, and sampling on the interpolation curve to obtain a plurality of intermediate image features.

[0112] In some embodiments, the image generation model is a StyleGAN model or a StyleGAN2 model.

[0113] The video generation device provided in this embodiment can be used to implement the technical solutions of the above embodiments related to the video generation method, and since the implementation principles and technical effects thereof are similar, they will not be described in this embodiment.

[0114] Corresponding to the model determination method in the above embodiment, Fig. 8 is a structural block diagram of a model determination device provided by an embodiment of the present disclosure. For convenience of explanation, only parts related to the embodiment of the present disclosure are shown. Referring to Fig. 8, the model determination device includes: a training unit 801;

[0115] The training unit 801 is used for training a neural network based on a plurality of training images and an image generation model to learn deviations for adjusting image features based on the feature space of the image generation model.

[0116] A single training process of the neural network includes the steps of generating target image features based on image features of a first training image and image features of a second training image, initially adjusting the target image features based on the feature space, learning a target deviation corresponding to the initial adjustment by the neural network and re-adjusting the initially adjusted target image features based on the target deviation, and adjusting model parameters of the neural network based on the target deviation, the re-adjusted target image features, the first training image, and the second training image.

[0117] In some embodiments, the training unit 801 is further used for determining a target optimization function of the neural network according to a regularity constraint and a similarity constraint, and adjusting model parameters of the neural network based on the target optimization function, the target deviation, the re-adjusted target image features, the first training image, and the second training image, wherein the regularity constraint is used to minimize the difference between the re-adjusted target image features and the initial adjusted target image features, and the similarity constraint is used to minimize the difference between the re-adjusted target image features and the image features of the first training image and the image features of the second training image.

[0118] The model determination equipment provided in this embodiment can be used to implement the technical solutions of the above embodiments related to the model determination method, and since the implementation principles and technical effects thereof are similar, they will not be described in this embodiment.

[0119] Referring to FIG. 9, a structural schematic diagram of an electronic device 900 suitable for implementing the embodiments of the present disclosure is shown, and the electronic device 900 may be a terminal device or a server. The terminal device includes, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDA), tablet computers (portable android devices, PAD), portable media players (portable media players, PMP), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG. 9 is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0120] As shown in Fig. 9, the electronic device 900 may include a processing unit (e.g., a central processor, a graphic processor, etc.) 901, which may perform various appropriate operations and processes based on programs stored in a read only memory (ROM) 902 or programs loaded from a storage device 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data necessary for the operation of the electronic device 900 are further stored. The processing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0121] Typically, the following devices can be connected to the I / O interface 905: input devices 906 including, for example, a touch panel, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 907 including, for example, a Liquid Crystal Display (LCD), speaker, oscillator, etc.; storage devices 908 including, for example, a tape, hard disk, etc.; and communication devices 909. The communication devices 909 can allow the electronic device 900 to communicate wirelessly or wired with other devices to exchange data. Although FIG. 9 shows the electronic device 900 having various devices, it should be understood that it is not necessary to implement or include all the devices shown. Alternatively, more or fewer devices can be implemented or included.

[0122] In particular, according to an embodiment of the present disclosure, the above process described with reference to the flowcharts can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program loaded on a computer-readable medium, the computer program including program code for performing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network by the communication device 909, or may be installed from the storage device 908, or may be installed from the ROM 902. When the computer program is executed by the processing device 901, it performs the above functions limited to the method of the embodiment of the present disclosure.

[0123] It should be noted that the computer-readable medium of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above. The computer-readable storage medium may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media include, but are not limited to, an electrical connection having one or more conductors, a portable computer magnetic disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), optical fiber, a portable compact magnetic disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program, which can be used by or in combination with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, loaded with computer-readable program code. Such a propagated data signal may take a number of forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may transmit, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device.The program code contained in the computer readable medium may be transmitted over any suitable medium, including, but not limited to, wire, cable, radio frequency (RF), or the like, or any suitable combination thereof.

[0124] The computer-readable medium may be included in the electronic device, or may exist separately from the electronic device.

[0125] One or more programs are loaded into the computer-readable medium, and when the one or more programs are executed by the electronic device, the electronic device performs the method shown in the above embodiments.

[0126] Computer program code for carrying out the operations of the present disclosure can be written in one or more program design languages ​​or combinations thereof, including object-oriented program design languages ​​such as Java, Smalltalk, C++, and the like, and general procedural program design languages ​​such as "C" or similar program design languages. The program code may be executed entirely on the user computer, partially on the user computer, as a separate software package, partially on the user computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., connected via the Internet using an Internet Service Provider).

[0127] The flowcharts and block diagrams in the drawings illustrate the architecture, functions and operations of the systems, methods and computer program products of various embodiments of the present disclosure, as well as possible implementations of the computer programs. In this regard, each block in the flowcharts or block diagrams may represent a module, program segment, or part of code, which includes one or more executable instructions for implementing a specified logic function. It should be noted that in some alternative implementations, the functions marked in the blocks may be executed in a different order than the order marked in the drawings. For example, two blocks shown in succession may actually be executed substantially in parallel or may be executed in the reverse order, depending on the related functions. It should be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts may be implemented by a dedicated system based on hardware, or by a combination of dedicated hardware and computer instructions, for implementing the specified functions or operations.

[0128] The units according to the embodiments of the present disclosure may be implemented in a software manner or in a hardware manner, and the names of the units, in some cases, do not constitute limitations on the units themselves, for example, the acquisition unit may be further described as a "target audio acquisition unit."

[0129] The functionality described herein may be performed at least in part by one or more hardware logic components. For example, and without limitation, examples of types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), etc.

[0130] In the context of this disclosure, a machine-readable medium may be a tangible medium, and may contain or store a program used by or in conjunction with an instruction execution system, device, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium includes, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the above. More specific examples of machine-readable storage media include electrical connections based on one or more conductors, portable computer disks, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable and programmable read-only memories (EPROMs or flash memories), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0131] According to a first aspect, based on one or more embodiments of the present disclosure, a video generation method is provided, comprising: extracting a first image feature in a first image; obtaining a plurality of intermediate image features by nonlinear interpolation based on the first image feature and a second image feature which is an image feature of a second image; reconstructing an image by an image generation model based on the first image feature, the second image feature and the plurality of intermediate image features, and generating a target video for showing a process of gradual change from the first image to the second image.

[0132] Based on one or more embodiments of the present disclosure, the step of obtaining a plurality of intermediate image features by nonlinear interpolation based on the first image feature and the second image feature includes the steps of generating a third image feature based on the first image feature and the second image feature, in turn adjusting the third image feature based on a feature space of the image generation model and a neural network for learning deviations for adjusting image features based on the feature space, and performing nonlinear interpolation based on the first image feature, the second image feature, and the adjusted third image feature to obtain the plurality of intermediate image features.

[0133] Based on one or more embodiments of the present disclosure, the step of adjusting the third image feature based on a feature space of the image generation model and a neural network for learning deviations for adjusting image features based on the feature space in turn includes a step of obtaining an average image feature in the feature space, a step of initially adjusting the third image feature based on the average image feature, a step of inputting the first image feature and the second image feature into the neural network to obtain output data of the neural network reflecting deviations of the initial adjustment, and a step of again adjusting the initially adjusted third image feature based on the output data.

[0134] Based on one or more embodiments of the present disclosure, the step of initially adjusting the third image feature based on the average image feature includes a step of determining an average value of the third image feature and the average image feature, and a step of determining that the initially adjusted third image feature is the average value.

[0135] Based on one or more embodiments of the present disclosure, the neural network is obtained by training regularity constraints and similarity constraints, the regularity constraints are used to minimize the difference between image features adjusted based on the neural network and image features adjusted based on the feature space, and the similarity constraints are used to minimize the difference between image features adjusted based on the neural network and image features of a first training image and image features of a second training image.

[0136] Based on one or more embodiments of the present disclosure, the step of performing nonlinear interpolation based on the first image feature, the second image feature, and the third image feature to obtain the plurality of intermediate image features includes a step of obtaining an interpolation curve by cubic spline interpolation based on the first image feature, the second image feature, and the third image feature, and a step of sampling on the interpolation curve to obtain the plurality of intermediate image features.

[0137] According to one or more embodiments of the present disclosure, the image generation model is a StyleGAN model or a StyleGAN2 model.

[0138] According to a second aspect, a model determination method is provided according to one or more embodiments of the present disclosure, comprising: training a neural network based on a plurality of training images and an image generation model to learn deviations for adjusting image features based on a feature space of the image generation model. A single training process of the neural network includes: generating a target image feature based on image features of a first training image and image features of a second training image; initially adjusting the target image feature based on the feature space; learning a target deviation corresponding to the initial adjustment by the neural network, and re-adjusting the initially adjusted target image feature based on the target deviation; and adjusting a model parameter of the neural network based on the target deviation, the re-adjusted target image feature, the first training image, and the second training image.

[0139] Based on one or more embodiments of the present disclosure, the step of adjusting model parameters of the neural network based on the target deviation, the readjusted target image features, the first training images, and the second training images includes the steps of determining a target optimization function of the neural network according to a regularity constraint and a similarity constraint, and adjusting the model parameters of the neural network based on the target optimization function, the target deviation, the readjusted target image features, the first training images, and the second training images, wherein the regularity constraint is used to minimize the difference between the readjusted target image features and the initial readjusted target image features, and the similarity constraint is used to minimize the difference between the readjusted target image features and the image features of the first training images and the image features of the second training images.

[0140] According to a third aspect, based on one or more embodiments of the present disclosure, a video generation device is provided, comprising: an extraction unit for extracting a first image feature in a first image; an interpolation unit for obtaining a plurality of intermediate image features by nonlinear interpolation based on the first image feature and a second image feature which is an image feature of a second image; and a video generation unit for reconstructing an image by an image generation model based on the first image feature, the second image feature and the plurality of intermediate image features, and generating a target video for showing a process of gradual change from the first image to the second image.

[0141] According to a fourth aspect, according to one or more embodiments of the present disclosure, a model determination device is provided, comprising: a training unit for training a neural network according to a plurality of training images and an image generation model to learn deviations for adjusting image features according to a feature space of the image generation model. In one training process of the neural network, a training module is used for generating target image features according to image features of a first training image and image features of a second training image, initially adjusting the target image features according to the feature space, learning a target deviation corresponding to the initial adjustment by the neural network, and re-adjusting the initially adjusted target image features according to the target deviations, and adjusting model parameters of the neural network according to the target deviations, the re-adjusted target image features, the first training images, and the second training images.

[0142] According to a fifth aspect, in accordance with one or more embodiments of the present disclosure, there is provided an electronic device, comprising at least one processor and a memory: the memory storing computer executable instructions; The at least one processor executes computer executable instructions stored in the memory to cause the at least one processor to perform the video generation method described in the first aspect or various possible designs of the first aspect, or to cause the at least one processor to perform the model determination method described in the second aspect or various possible designs of the second aspect.

[0143] According to a sixth aspect, in accordance with one or more embodiments of the present disclosure, there is provided a computer-readable storage medium having computer executable instructions stored thereon, which, when executed by a processor, realises the video generation method as set forth in the first aspect or various possible designs of the first aspect, or realises the model determination method as set forth in the second aspect or various possible designs of the second aspect.

[0144] According to a seventh aspect, there is provided a computer program product in accordance with one or more embodiments of the present disclosure, the computer program product comprising computer executable instructions which, when executed by a processor, realises the video generation method as set forth in the first aspect or in various possible designs of the first aspect, or realises the model determination method as set forth in the second aspect or in various possible designs of the second aspect.

[0145] According to an eighth aspect, there is provided a computer program in accordance with one or more embodiments of the present disclosure, which, when executed by a processor, realizes the video generation method according to the first aspect or various possible designs of the first aspect, or realizes the model determination method according to the second aspect or various possible designs of the second aspect.

[0146] The above description is merely a description of the preferred embodiments of the present disclosure and the technical principles used. Those skilled in the art should understand that the scope of the present disclosure is not limited to the technical solution constituted by a specific combination of the above technical features, and should include other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the concept of the above disclosure. For example, the above features are substituted with technical features having similar functions disclosed in the present disclosure (but are not limited to these).

[0147] Also, although operations have been described in a particular sequence, it should not be understood as requiring that these operations be performed in the particular sequence or order shown. In certain cases, multitasking and parallel processing may be advantageous. Similarly, although the above description includes a number of specific implementation details, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of a single embodiment can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments alone or in any suitable subcombination.

[0148] Although the present subject matter has been described using language specific to structural features and / or logical operations of a method, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or operations described above. Rather, the specific features and operations described above are merely example forms for implementing the claims.

Claims

1. 1. A method for generating video, comprising: Extracting a first image feature in a first image; obtaining a plurality of intermediate image features by nonlinear interpolation based on the first image feature and a second image feature, the second image feature being an image feature of a second image; a step of reconstructing an image using an image generation model based on the first image features, the second image features and the plurality of intermediate image features, and generating a target video for showing a gradual change from the first image to the second image.

2. The step of obtaining a plurality of intermediate image features by nonlinear interpolation based on the first image feature and the second image feature includes: generating a third image feature based on the first image feature and the second image feature; adjusting the third image feature based on a feature space of the image generation model and a neural network for learning deviations to adjust the image feature based on the feature space; and performing nonlinear interpolation based on the first image feature, the second image feature and the adjusted third image feature to obtain the plurality of intermediate image features.

3. The step of adjusting the third image feature based on a feature space of the image generation model and a neural network for learning deviations for adjusting the image feature based on the feature space, obtaining a mean image feature in the feature space; adjusting the third image feature based on the average image feature; inputting the first image feature and the second image feature into the neural network and obtaining output data of the neural network reflecting the initially adjusted deviation; and re-adjusting the initially adjusted third image feature based on the output data.

4. The step of initially adjusting the third image feature based on the average image feature comprises: determining an average value of the third image feature and the average image feature; and determining that the initially adjusted third image feature is the average value.

5. 5. The video generation method of claim 2, wherein the neural network is obtained by training regularity constraints and similarity constraints, the regularity constraints being used to minimize the difference between image features trained based on the neural network and image features trained based on the feature space, and the similarity constraints being used to minimize the difference between image features trained based on the neural network and image features of a first training image and image features of a second training image.

6. The step of performing nonlinear interpolation based on the first image feature, the second image feature, and the adjusted third image feature to obtain the plurality of intermediate image features includes: obtaining an interpolation curve by cubic spline interpolation based on the first image feature, the second image feature, and the third image feature; A method according to any one of claims 2 to 5, comprising the step of sampling on said interpolation curve to obtain said intermediate image features.

7. The video generation method according to any one of claims 1 to 6, wherein the image generation model is a StyleGAN model or a StyleGAN2 model.

8. A model determination method, comprising: training a neural network based on a plurality of training images and an image generation model to learn deviations for adjusting image features based on a feature space of the image generation model; A single training process of the neural network includes: generating a target image feature based on the image features of the first training image and the image features of the second training image; initializing the target image features based on the feature space; learning a target deviation corresponding to the initial adjustment by the neural network, and re-adjusting the initially adjusted target image feature according to the target deviation; and adjusting model parameters of the neural network based on the target deviation, the re-adjusted target image features, the first training images, and the second training images.

9. adjusting model parameters of the neural network based on the target deviation, the re-adjusted target image features, the first training images, and the second training images; determining a target optimization function of the neural network according to regularity and similarity constraints; and adjusting model parameters of the neural network based on the target optimization function, the target deviation, the re-adjusted target image features, the first training images, and the second training images; The method of claim 8, wherein the regularity constraint is used to minimize the difference between the re-adjusted target image features and the initial adjusted target image features, and the similarity constraint is used to minimize the difference between the re-adjusted target image features and the image features of the first training image and the image features of the second training image.

10. 1. A video production device comprising: an extraction unit for extracting a first image feature in the first image; an interpolation unit for obtaining a plurality of intermediate image features by nonlinear interpolation based on the first image feature and a second image feature, the second image feature being an image feature of a second image; a video generation unit for reconstructing an image using an image generation model based on the first image features, the second image features and the plurality of intermediate image features, and generating a target video for showing a gradual change from the first image to the second image.

11. 1. A model determination device, comprising: A training unit for training a neural network based on a plurality of training images and an image generation model to learn deviations for adjusting image features based on a feature space of the image generation model; A single training process of the neural network includes: generating a target image feature based on the image features of the first training image and the image features of the second training image; initializing the target image features based on the feature space; learning a target deviation corresponding to the initial adjustment by the neural network, and re-adjusting the initially adjusted target image feature according to the target deviation; and adjusting model parameters of the neural network based on the target deviation, the re-adjusted target image features, the first training images, and the second training images.

12. An electronic device comprising at least one processor and a memory, The memory stores computer executable instructions; An electronic device in which the at least one processor executes computer-executable instructions stored in the memory to cause the at least one processor to perform the video generation method described in any one of claims 1 to 7, or to cause the at least one processor to perform the model determination method described in claim 8 or 9.

13. A computer-readable storage medium having stored thereon computer-executable instructions, the computer-executable instructions being executed by a processor to implement the video generation method of any one of claims 1 to 7, or the model determination method of claim 8 or 9.

14. A computer program product comprising computer executable instructions, which, when executed by a processor, implements the video generation method according to any one of claims 1 to 7 or the model determination method according to claim 8 or 9.

15. A computer program product which, when executed by a processor, implements the video generation method according to any one of claims 1 to 7 or the model determination method according to claim 8 or 9.

Images