Video generation methods and equipment

Abstract

This disclosure provides a video generation method and apparatus. The method includes: extracting first image features from a first image; obtaining multiple intermediate image features through nonlinear interpolation based on the first image features and second image features, wherein the second image features are image features of the second image; and reconstructing an image using an image generation model based on the first image features, the second image features, and the multiple intermediate image features to generate a target video, wherein the target video is used to show the process of transitioning from the first image to the second image. Thus, by using nonlinear interpolation, the quality of the intermediate image features is improved, thereby improving the image quality of the intermediate frames of the target video, i.e., improving the video quality of the target video.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a video generation method and apparatus. Background Technology

[0002] Current computer vision and deep learning technologies can generate videos that transition between two images. For example, by inputting two facial images into a deep learning model, a video can be generated where the video frames gradually change from one facial image to another. This technology can be used for special effects generation to enhance the visual appeal of videos.

[0003] In the video generation process, multiple video frames need to be generated based on two images, resulting in a video with gradual transitions between the two images. However, the quality of generating multiple video frames based on two images, especially the image quality of the intermediate frames, needs improvement. Summary of the Invention

[0004] This disclosure provides a video generation method and apparatus to address the problem that the image quality of intermediate frames in a video needs to be improved when generating a video based on a small number of images.

[0005] In a first aspect, embodiments of this disclosure provide a video generation method, including:

[0006] Extract features from the first image;

[0007] Based on the first image feature and the second image feature, multiple intermediate image features are obtained by nonlinear interpolation, wherein the second image feature is the image feature of the second image;

[0008] Based on the first image features, the second image features, and the plurality of intermediate image features, an image reconstruction is performed using an image generation model to generate a target video, wherein the target video is used to show the process of transitioning from the first image to the second image.

[0009] Secondly, embodiments of this disclosure provide a model determination method, including:

[0010] A neural network is trained based on multiple training images and an image generation model, the neural network being used to learn the bias in adjusting image features based on the feature space of the image generation model;

[0011] The training process of the neural network includes:

[0012] Generate target image features based on the image features of the first training image and the image features of the second training image;

[0013] Based on the feature space, the features of the target image are initially adjusted;

[0014] The neural network learns the target deviation corresponding to the initial adjustment, and then adjusts the target image features again based on the target deviation.

[0015] The model parameters of the neural network are adjusted based on the target deviation, the readjusted target image features, the first training image, and the second training image.

[0016] Thirdly, embodiments of this disclosure provide a video generation device, including:

[0017] An extraction unit is used to extract features from the first image.

[0018] An interpolation unit is used to obtain multiple intermediate image features through nonlinear interpolation based on the first image features and the second image features, wherein the second image features are the image features of the second image.

[0019] The video generation unit is used to reconstruct an image based on the first image features, the second image features, and the plurality of intermediate image features using an image generation model to generate a target video, wherein the target video is used to show the process of gradation from the first image to the second image.

[0020] Fourthly, embodiments of this disclosure provide a model determination device, comprising:

[0021] A training unit is used to train a neural network based on multiple training images and an image generation model, wherein the neural network is used to learn the deviation of image feature adjustment based on the feature space of the image generation model;

[0022] The training process of the neural network includes:

[0023] Generate target image features based on the image features of the first training image and the image features of the second training image;

[0024] Based on the feature space, the features of the target image are initially adjusted;

[0025] The neural network learns the target deviation corresponding to the initial adjustment, and then adjusts the target image features again based on the target deviation.

[0026] The model parameters of the neural network are adjusted based on the target deviation, the readjusted target image features, the first training image, and the second training image.

[0027] Fifthly, embodiments of this disclosure provide an electronic device, including: at least one processor and a memory;

[0028] The memory stores computer-executed instructions;

[0029] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the video generation method as described in the first aspect or various possible designs of the first aspect, or causing the at least one processor to perform the model determination method as described in the second aspect or various possible designs of the second aspect.

[0030] In a sixth aspect, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the video generation method as described in the first aspect or various possible designs of the first aspect, or implement the model determination method as described in the second aspect or various possible designs of the second aspect.

[0031] In a seventh aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, the computer program product comprising computer execution instructions that, when executed by a processor, implement the video generation method as described in the first aspect or various possible designs of the first aspect, or implement the model determination method as described in the second aspect or various possible designs of the second aspect.

[0032] The video generation method and apparatus provided in this embodiment obtain multiple intermediate image features through nonlinear interpolation based on the first image features of the first image and the second image features of the second image. Based on the first image features, the second image features, and the multiple intermediate image features, image reconstruction is performed using an image generation model to generate a target video. The target video is used to demonstrate the process of transitioning from the first image to the second image. Thus, by using nonlinear interpolation, the quality of the intermediate image features is improved. While maintaining the similarity between the intermediate frames of the target video and the first and second images, the image quality of the intermediate frames of the target video is improved, thereby enhancing the overall video quality of the target video. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a schematic diagram illustrating an application scenario to which the embodiments of this disclosure are applicable;

[0035] Figure 2 Flowchart of the video generation method provided in the embodiments of this disclosure Figure 1 ;

[0036] Figure 3a Flowchart of the video generation method provided in the embodiments of this disclosure Figure 2 ;

[0037] Figure 3b A schematic diagram illustrating the process of adjusting the features of a third image sequentially based on the feature space of an image generation model and a neural network, as provided in this embodiment of the disclosure;

[0038] Figure 4 An example diagram of a framework for nonlinear interpolation based on feature space and neural networks provided in embodiments of this disclosure;

[0039] Figure 5 A schematic flowchart illustrating the model determination method provided in this embodiment of the disclosure;

[0040] Figure 6 A schematic diagram of the training framework for the neural network provided in this embodiment of the disclosure;

[0041] Figure 7 This is a structural block diagram of a video generation device provided in an embodiment of the present disclosure;

[0042] Figure 8 The structural block diagram of the model determination device provided in the embodiments of this disclosure;

[0043] Figure 9 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0045] When generating a transition video between two input images, the common practice is to linearly interpolate the image features of the two input images to obtain intermediate image features, which are then used to generate intermediate frames of the video. This method can ensure the continuity and similarity of video frames. However, the intermediate image features obtained after linear interpolation often do not conform to the distribution (or variation) patterns of image features in real video frames, resulting in poor image quality of the intermediate frames and insufficient aesthetics and realism.

[0046] To address the aforementioned problems, this disclosure provides a video generation method and apparatus. Based on first image features of a first image and second image features of a second image, multiple intermediate image features are obtained through nonlinear interpolation. Based on the first image features, second image features, and multiple intermediate image features, image reconstruction is performed using an image generation model to generate a target video. The target video is used to depict the process of transitioning from the first image to the second image. Since the change process of a real video image is nonlinear, compared to linear interpolation, this disclosure uses nonlinear interpolation, which improves the quality of intermediate image features and the quality of intermediate frames in the target video, resulting in a more realistic and aesthetically pleasing nonlinear video image. Improving the quality of intermediate image features includes: improving the realism of the intermediate image features and increasing the similarity between the intermediate images and the first and second images. Improving the quality of intermediate frames in the target video includes: improving the aesthetics and realism of the intermediate frames and increasing the similarity between the intermediate frames and the first and second images.

[0047] refer to Figure 1 , Figure 1 This is a schematic diagram illustrating the application scenarios to which the embodiments of this disclosure apply.

[0048] like Figure 1 As shown, in this application scenario, the devices involved include a video generation device 101, which can be a terminal or a server. Figure 1 Taking video generation device 101 as an example, two images can be processed on video generation device 101 to generate a video that displays the gradient effect between the two images.

[0049] Optionally, the device involved in this application scenario also includes an image acquisition device 102, which can be a terminal or a server. For example, the terminal acquires images input by the user, or the terminal acquires images of the current scene through a camera; or the server acquires images publicly available on the network. Figure 1 Taking image acquisition device 102 as an example, the image acquisition device 102 sends the acquired image to the video generation device 101, which generates a video to display a transition from the acquired image to another image (from image acquisition device 102 or from other devices), or a transition from another image to the acquired image.

[0050] The video generation device 101 and the image acquisition device 102 can be the same or different devices.

[0051] When the video generation device 101 and the image acquisition device 102 are the same device, for example: a user takes a selfie with a mobile phone, obtains a selfie portrait, and selects another image on the mobile phone; the mobile phone generates a video based on the user's selfie portrait and the image selected by the user, and the video content of the video is the process of gradually changing from the user's selfie portrait to the image selected by the user.

[0052] When the video generation device 101 and the image acquisition device 102 are different devices, for example: a user takes a selfie with a mobile phone, obtains a selfie portrait, and selects another image on the mobile phone; the mobile phone sends the selfie image and the image selected by the user to the server, the server generates a video and returns the video to the mobile phone, and the video is a process of gradually changing from the user's selfie portrait to the image selected by the user.

[0053] The terminal can be a personal digital assistant (PDA) device, a handheld device (such as a smartphone or tablet), a computing device (such as a personal computer (PC)), an in-vehicle device, a wearable device (such as a smartwatch or smart bracelet), or a smart home device (such as a smart display device). The server can be a distributed server, a centralized server, or a cloud server.

[0054] The following provides several embodiments of this disclosure. The implementing entity of these embodiments may be an electronic device, which may be a terminal or a server.

[0055] refer to Figure 2 , Figure 2 Flowchart of the video generation method provided in the embodiments of this disclosure Figure 1 .

[0056] like Figure 2 As shown, the video generation method includes:

[0057] S201. Extract the features of the first image from the first image.

[0058] The first image can be an image input by the user, an image from another device, or an image captured by the currently executing device. For example, when the currently executing device is a terminal, the terminal can acquire the first image input by the user, or acquire the first image captured by the camera on the terminal. Similarly, when the currently executing device is a server, the server can receive the first image input by the user sent by the terminal.

[0059] Wherein, the first image feature is the image feature of the first image.

[0060] In this embodiment, an encoder is used to encode the first image to obtain the first image features. Here, the first image features specifically refer to the image features obtained after the first image has been encoded.

[0061] S202. Based on the first image features and the second image features, multiple intermediate image features are obtained through nonlinear interpolation, and the second image features are the image features of the second image.

[0062] The second image is different from the first image.

[0063] Specifically, the second image feature refers to the image feature obtained after encoding the second image.

[0064] In one example, multiple images and their encoded image features can be pre-stored. A second image feature is obtained from the stored image features of the multiple images. In one approach, the user can specify a second image from the pre-stored multiple images, and the image features of the second image, i.e., the second image feature, can be obtained from the image features of the multiple images. In another approach, the second image feature can be obtained from the image features of the multiple images in a preset order (e.g., the image storage order) or randomly.

[0065] For example: In response to the user's input of a first image, multiple images are displayed on the terminal for the user to select; the user selects a second image from the multiple images and inputs a request on the terminal to generate a video that transitions from the first image to the second image; in response to the request, the terminal obtains the image features of the second image from the image features of the multiple images that have been stored in advance, i.e., the second image features.

[0066] In another example, a second image can be obtained from user input, sent by other devices, or captured by the currently executing device. The second image is then encoded to obtain second image features.

[0067] In this embodiment, after obtaining the second image feature, the first and second image features are used as two known quantities in the nonlinear interpolation process. A preset nonlinear interpolation method is used to perform nonlinear interpolation to obtain the interpolation function, i.e., the interpolation curve. Sampling is performed on the interpolation curve between the points corresponding to the first image feature and the points corresponding to the second image feature to obtain multiple intermediate image features. These intermediate image features are used to generate intermediate frames of the video.

[0068] Optionally, sampling can be performed at equal intervals on the interpolation curve to make the changes in features between adjacent intermediate images obtained by interpolation similar, thereby improving the quality of the subsequently generated video.

[0069] S203. Based on the first image features, the second image features, and multiple intermediate image features, image reconstruction is performed using an image generation model to generate a target video, wherein the target video is used to show the process of transitioning from the first image to the second image.

[0070] The image generation model can be a neural network used for image generation or reconstruction. Its input data is encoded image features, and its output data is the reconstructed image. A pre-trained image generation model publicly available online can be used, or the neural network can be trained using training data (including multiple training images) to obtain the image generation model. There are no restrictions on the training process of this model.

[0071] In this embodiment, after obtaining multiple intermediate image features, the first image feature, the second image feature, and the multiple intermediate image features can be input into the image generation model respectively to obtain the reconstructed image corresponding to the first image feature, the reconstructed image corresponding to the second image feature, and the reconstructed images corresponding to each intermediate image feature. The multiple reconstructed images can be sorted and combined according to the distribution order of the first image feature, the second image feature, and the intermediate image features on the interpolation curve to obtain the target video. In the target video, the first frame is the reconstructed image corresponding to the first image feature, the last frame is the reconstructed image corresponding to the second image feature, and the intermediate frames are the reconstructed images corresponding to the intermediate image features.

[0072] In this embodiment of the present disclosure, nonlinear interpolation is performed based on the first image features obtained by encoding the first image and the second image features obtained by encoding the second image to obtain multiple intermediate image features. Based on the first image features, the second image features and the multiple intermediate image features, image reconstruction is performed through an image generation model, and the reconstructed image is output by the image generation model.

[0073] Therefore, by using nonlinear interpolation of the image features obtained by encoding two images, the authenticity of the interpolated intermediate image features and the similarity between the intermediate image features and the image features of the two original images are improved. This improves the authenticity and aesthetics of the intermediate frames of the video, increases the similarity between the intermediate frames and the first and last frames, and improves the video quality.

[0074] Regarding image generation models, the following are some possible implementation examples:

[0075] In some embodiments, the image generation model is a Generative Adversarial Network (GAN), thereby leveraging the advantages of GAN in image generation to improve the image reconstruction quality of the image generation model and improve the quality of image frames in the target video.

[0076] In some embodiments, the image generation model is a Style-Based Architecture for GANs (StyleGAN) model or a StyleGAN2 model. Therefore, by leveraging the advantages of the StyleGAN or StyleGAN2 models in image generation, the image reconstruction quality of the image generation model is improved, thereby enhancing the quality of image frames in the target video.

[0077] Regarding the nonlinear interpolation process, in some embodiments, the feature space of the image generation model and neural networks can be used to assist in nonlinear interpolation. The following embodiments will describe this auxiliary process.

[0078] Reference Figure 3a , Figure 3a Flow example of the video generation method provided in the embodiments of this disclosure Figure 2 .like Figure 3a As shown, the video generation method includes:

[0079] S301. Extract the features of the first image from the first image.

[0080] The implementation principle and technical effect of S301 can be referred to the aforementioned embodiments, and will not be repeated here.

[0081] S302. Generate a third image feature based on the first image feature and the second image feature, wherein the second image feature is the image feature of the second image.

[0082] The process of obtaining the second image feature can be referred to the aforementioned embodiments and will not be repeated here.

[0083] In one example, the average value of the first image feature and the second image feature is determined, and this average value is the third image feature. Specifically, the average value of the first image feature and the second image feature can be obtained by adding the feature values ​​at corresponding positions on the first image feature and the second image feature and then averaging them.

[0084] In another example, a weighted sum of the first and second image features is performed to obtain the third image feature. The weights for the first and second image features can be preset.

[0085] S303. Based on the feature space of the image generation model and the neural network in sequence, adjust the third image features. The neural network is used to learn the deviation of image feature adjustment based on the feature space.

[0086] In this context, the feature space of the image generation model can be understood as the input space of the image generation model, where the feature samples conform to a certain probability distribution.

[0087] Optionally, when the image generation model is a generative adversarial network, the feature space of the image generation model is the latent space corresponding to the image generation model. The image features obtained by encoding the first image and the second image through the encoder are called latent codes. That is, the features of the first image can be called the first latent code, and the features of the second image can be called the second latent code.

[0088] In this embodiment, after obtaining the third image feature, the third image feature can be adjusted based on the feature samples in the feature space of the image generation model, so that the third image feature is closer to the feature samples in the feature space, thereby improving the image quality of the reconstructed image obtained by image reconstruction based on the third image feature, that is, improving the image quality of the intermediate frame.

[0089] In this embodiment, considering that the adjustment of the third image features based on the feature space may have some deviation, resulting in a decrease in the similarity between the third image features and the first and second image features, in order to solve this problem, after adjusting the third image features based on the feature space, the third image features are adjusted again through a neural network model to improve the similarity between the third image features and the first and second image features.

[0090] This requires training the neural network so that it can learn the biases in adjusting image features based on the feature space. The specific training process is described in subsequent embodiments.

[0091] Optionally, the neural network is a fully connected neural network. Therefore, when the learning task of the neural network is singular and both the input and output data are image features, a fully connected neural network with more network parameters can improve the accuracy of adjusting the third image feature.

[0092] In one possible implementation, refer to Figure 3b , Figure 3b This is a schematic diagram illustrating the process of adjusting the third image features (i.e., S303) sequentially based on the feature space of the image generation model and the neural network, as provided in an embodiment of this disclosure. Figure 3b As shown, the process of adjusting the third image features based on the feature space of the image generation model and the neural network (i.e., a possible implementation of S303) includes:

[0093] S3031. Obtain the average image features in the feature space; S3032. Make preliminary adjustments to the third image features based on the average image features; S3033. Input the first image features and the second image features into the neural network to obtain the output data of the neural network. The output data reflects the deviation of the preliminary adjustment; S3034. Make further adjustments to the third image features after the preliminary adjustment based on the output data.

[0094] The output data of the neural network reflects the feature deviation generated after the third image features are initially adjusted based on the average image features in the feature space.

[0095] In this embodiment, the average image features in the feature space can be determined based on the probability distribution conformed to by the feature space. The probability distribution conformed to by the feature space is, for example, a Gaussian distribution. After determining the average image features, the third image features are initially adjusted using these features to make them closer to the average image features, thereby improving the quality of the third image features. The first and second image features are then input into the neural network to obtain the output data of the neural network, which is also an image feature. Based on the output data of the neural network, the initially adjusted third image features are further adjusted to make them closer to the first and second image features, thereby improving the similarity between the third image features and the first and second image features.

[0096] Optionally, based on the average image features, the third image features are initially adjusted, including: determining the mean of the third image features and the average image features, and determining the initially adjusted third image as this mean. Thus, by solving for the mean of the third image features and the average image features, feature cropping (i.e., initial adjustment) of the third image features is achieved.

[0097] Optionally, based on the output data, the initially adjusted third image features are further adjusted, including adding the output data to the initially adjusted third image features to obtain the further adjusted third image features. Thus, by adding the feature bias learned by the neural network during the initial adjustment process to the initially adjusted third image features, the similarity between the third image features and the first and second image features is improved.

[0098] S304. Based on the first image features, the second image features, and the adjusted third image features, perform nonlinear interpolation to obtain multiple intermediate image features.

[0099] In this embodiment, after obtaining the first image feature, the second image feature, and the finally adjusted third image feature, the first image feature, the second image feature, and the third image feature are used as three known quantities. An interpolation curve is obtained through nonlinear interpolation, and multiple intermediate image features are sampled from the interpolation curve. Therefore, in addition to the first and second image features, the nonlinear interpolation process also utilizes a third image feature with high quality and high similarity to the first and second image features, effectively improving the accuracy of nonlinear interpolation and the quality of the intermediate image features.

[0100] In one possible implementation, the nonlinear interpolation method employs cubic spline interpolation. In this case, step S304 includes: obtaining an interpolation curve using cubic spline interpolation based on the first image features, the second image features, and the third image features; and sampling on the interpolation curve to obtain multiple intermediate image features. Thus, using cubic spline interpolation improves the accuracy of nonlinear interpolation and enhances the quality of the intermediate image features.

[0101] Specifically, the third image feature, along with the first and second image features, can be input into a cubic spline interpolation function to obtain the interpolation function, i.e., the interpolation curve. Then, sampling is performed on the interpolation curve to obtain multiple intermediate image features.

[0102] S305. Based on the first image features, the second image features, and multiple intermediate image features, an image reconstruction is performed using an image generation model to generate a target video, wherein the target video is used to show the process of transitioning from the first image to the second image.

[0103] The implementation principle and technical effects of S305 can be referred to in the aforementioned embodiments, and will not be repeated here.

[0104] In this embodiment of the disclosure, based on the first image features obtained by encoding the first image and the second image features obtained by encoding the second image, nonlinear interpolation based on feature space and neural network is used to obtain multiple intermediate image features, which effectively improves the accuracy of nonlinear interpolation, thereby improving the quality of intermediate image features, improving the image quality of intermediate frames of the video, and thus improving the video quality.

[0105] For example, see reference Figure 4 , Figure 4 An example diagram illustrating a framework for nonlinear interpolation based on feature space and neural networks provided in embodiments of this disclosure. (See diagram below.) Figure 4 As shown, first, the average value of hidden code 1 (equivalent to the first image feature) and hidden code 2 (equivalent to the second image feature) is determined (equivalent to the third image feature). This average value is then cropped based on the feature space to obtain a cropped average value (equivalent to the initially adjusted third image feature). Next, hidden code 1 and hidden code 2 are input into the neural network to obtain the feature deviation output by the neural network. Then, this feature deviation is added to the cropped average value (equivalent to obtaining a further adjusted third image feature). Finally, hidden code 1, hidden code 2, and the average value are used for spline interpolation to obtain multiple interpolation results (i.e., multiple intermediate image features).

[0106] It should be noted that the above embodiments provide a scheme for adjusting image features by combining feature space and neural network. In practical applications, image features can also be adjusted based solely on feature space, that is, the feature deviation caused by adjusting feature space can be ignored.

[0107] In some embodiments, to improve the nonlinear interpolation effect, the neural network needs to be pre-trained so that it can learn the bias in adjusting image features based on the feature space of the image generation model. Below, an embodiment of neural network training is provided.

[0108] It should be noted that the training process of the neural network and the video generation process in the aforementioned embodiments can be executed on the same device or on different devices.

[0109] Reference Figure 5 , Figure 5 A flowchart illustrating the model determination method provided in this embodiment of the disclosure. Figure 5 As shown, the model determination method includes:

[0110] S501. Train a neural network based on multiple training images and an image generation model. The neural network is used to learn the bias of image feature adjustment based on the feature space of the image generation model.

[0111] In one training process of the neural network, S501 includes the following steps:

[0112] S5011. Generate target image features based on the image features of the first training image and the image features of the second training image.

[0113] In this embodiment, during each training process, two training images can be obtained from multiple training images. For ease of distinction, these two training images are referred to as the first training image and the second training image, respectively. An encoder can be used to encode the two training images to obtain the image features of the first training image and the second training image. Feature fusion processing is then performed on the image features of the first training image and the second training image to obtain the target image features.

[0114] In one example, feature fusion processing is performed on the image features of the first training image and the image features of the second training image to obtain the target image features. This includes determining the average value of the image features of the first training image and the image features of the second training image, which is the target image feature. Specifically, the average value can be obtained by adding the feature values ​​at corresponding positions of the image features of the first training image and the image features of the second training image.

[0115] In another example, the image features of the first training image and the image features of the second training image are weighted and summed to obtain the target image features. The weights for the image features of the first training image and the image features of the second training image can be preset.

[0116] S5012. Based on the feature space, perform preliminary adjustments to the features of the target image.

[0117] In this embodiment, the average image features in the feature space can be determined based on the probability distribution conforming to the feature space. Using these average image features, the target image features are initially adjusted to bring them closer to the average image features, thereby improving the quality of the target image features.

[0118] Optionally, based on the average image features, the target image features are initially adjusted, including: determining the mean of the target image features and the average image features, and determining the fourth image after initial adjustment as this mean. Thus, by solving for the mean of the target image features and the average image features, feature cropping (i.e., initial adjustment) of the target image features is achieved.

[0119] S5013. The target deviation is initially adjusted through neural network learning, and the target image features are further adjusted based on the target deviation.

[0120] In this embodiment, the image features of the first training image and the image features of the second training image are input into the neural network to obtain the output data of the neural network, that is, the target deviation corresponding to the initial adjustment is learned. Based on the target deviation corresponding to the initial adjustment learned by the neural network, the target image features after the initial adjustment are adjusted again so that the target image features are close to the image features of the first training image and the image features of the second training image, that is, to improve the similarity between the target image features and the image features of the first training image and the image features of the second training image.

[0121] Optionally, based on the target deviation, the initially adjusted target image features are further adjusted, including adding the target deviation to the initially adjusted target image features to obtain the further adjusted target image features. Thus, by adding the feature deviation learned by the neural network during the initial adjustment process to the initially adjusted target image features, the similarity between the target image features and the image features of the first training image and the second training image is improved.

[0122] S5014. Adjust the model parameters of the neural network based on the target deviation, the adjusted target image features, the first training image, and the second training image.

[0123] In this embodiment, the training error of the neural network can be determined based on the target deviation, the readjusted target image features, the first training image, and the second training image. Based on this training error, the model parameters of the neural network can be adjusted. For example, the training error can be determined based on the difference between the readjusted target image features and the image features of the first training image, and / or the difference between the readjusted target image features and the image features of the second training image.

[0124] In one example, the neural network is trained based on regularization constraints and similarity constraints. The regularization constraints are used to minimize the difference between the image features adjusted based on the neural network and the image features adjusted based on the feature space (i.e., the target image features after initial adjustment). The similarity constraints are used to minimize the difference between the image features adjusted based on the neural network (i.e., the target image features after further adjustment) and the image features of the first training image and the second training image.

[0125] At this point, S5014 includes: determining the target optimization function of the neural network through regularization constraints and similarity constraints; and adjusting the model parameters of the neural network based on the target optimization function, target deviation, readjusted target image features, first training image, and second training image.

[0126] Specifically, the objective optimization function of the neural network can be determined in advance based on regularization constraints and similarity constraints. During the training process of the neural network, the function value of the objective optimization function, i.e., the training error of the neural network, is determined based on the target bias, the first training image, and the second training image. Based on this training error, the model parameters of the neural network are optimized. An example optimization algorithm is the gradient descent algorithm.

[0127] Specifically, since the image features mentioned in the aforementioned embodiments are all encoded image features, to improve the accuracy of model training, after obtaining the adjusted target image features, the target image features can be input into the image generation model to obtain intermediate reconstructed images (i.e., the reconstructed images corresponding to the target image features). Then, through a feature extraction network, features are extracted from the first training image, the second training image, and the intermediate reconstructed image to obtain the image features of the first training image, the image features of the second training image, and the image features of the intermediate reconstructed image. For example, when the first training image, the second training image, and the intermediate reconstructed image are all face images, a face feature extraction network can be used to extract features from these images. Next, the differences between the image features of the intermediate reconstructed image and the image features of the first training image (features extracted by the feature extraction network), and the differences between the image features of the intermediate reconstructed image and the image features of the second training image (features extracted by the feature extraction network) are determined. Based on these two differences and the output data of the neural network, the training error is determined.

[0128] In one example, the objective optimization function of a neural network can be expressed as:

[0129] minL=||Φ(G(f(w1,w2)+w3))-Φ(x1)|| 2 +||Φ(G(f(w1,w2)+w3))-Φ(x2)|| 2

[0130] +λ||f(w1, w2)||

[0131] Where x1 and x2 represent the first training image and the second training image, respectively, w1 represents the image features obtained after encoding the first training image, w2 represents the image features obtained after encoding the second training image, w3 represents the target image features, f() represents the neural network, G() represents the image generation model, Φ() represents the feature extraction network, and λ is a preset parameter.

[0132] Where, ||Φ(G(f(w1,w2)+w3))-Φ(x1)|| 2 +||Φ(G(f(w1,w2)+w3))-Φ(x2)|| 2 For similarity constraints, λ||f(w1, w2)|| is a regularization constraint.

[0133] Thus, the above steps are repeated to adjust the neural network multiple times.

[0134] For example, see reference Figure 6 , Figure 6 This is a schematic diagram of the training framework for a neural network provided in an embodiment of this disclosure. Figure 6 As shown, the training process includes: first, determining the average value of hidden code 1 (the image features obtained after encoding input image 1) and hidden code 2 (the image features obtained after encoding input image 2); based on the feature space of the image generation model, performing feature cropping (i.e., preliminary adjustment) on this average value to obtain a cropped average value; next, inputting hidden code 1 and hidden code 2 into the neural network, and determining the training error for the regularization constraint based on the feature deviation output by the neural network; then, adding the feature deviation output by the neural network to the cropped average value, and inputting this average value back into the image generation model to obtain the reconstructed image; finally, determining the feature differences between the reconstructed image and input image 1, and the feature differences between the reconstructed image and input image 2 through the feature network, and determining the training error for the similarity constraint based on these two feature differences. Thus, the model parameters of the neural network are adjusted based on the training errors for the regularization constraint and the similarity constraint.

[0135] Corresponding to the video generation method in the above embodiments, Figure 7This is a structural block diagram of a video generation device provided in an embodiment of this disclosure. For ease of explanation, only the parts relevant to the embodiments of this disclosure are shown. (Refer to...) Figure 7 The video generation device includes an extraction unit 701 and an interpolation unit 702.

[0136] Extraction unit 701 is used to extract first image features from the first image;

[0137] Interpolation unit 702 is used to obtain multiple intermediate image features by nonlinear interpolation based on the first image features and the second image features, wherein the second image features are the image features of the second image.

[0138] The video generation unit 703 is used to reconstruct an image based on the first image features, the second image features, and multiple intermediate image features through an image generation model to generate a target video, wherein the target video is used to show the process of gradually transitioning from the first image to the second image.

[0139] In some embodiments, the interpolation unit 702 is further configured to: generate a third image feature based on the first image feature and the second image feature; adjust the third image feature sequentially based on the feature space of the image generation model and the neural network, wherein the neural network is used to learn the deviation of image feature adjustment based on the feature space; and perform nonlinear interpolation based on the first image feature, the second image feature and the adjusted third image feature to obtain multiple intermediate image features.

[0140] In some embodiments, the interpolation unit 702 is further configured to: obtain average image features in the feature space; perform preliminary adjustment on the third image features based on the average image features; input the first image features and the second image features into the neural network to obtain the output data of the neural network, wherein the output data reflects the deviation of the preliminary adjustment; and perform further adjustment on the third image features after the preliminary adjustment based on the output data.

[0141] In some embodiments, the interpolation unit 702 is further configured to: determine the mean of the third image feature and the average image feature; and determine the mean of the preliminarily adjusted third image feature.

[0142] In some embodiments, the neural network is trained based on regularization constraints and similarity constraints. The regularization constraints are used to minimize the difference between the image features adjusted based on the neural network and the image features adjusted based on the feature space. The similarity constraints are used to minimize the difference between the image features adjusted based on the neural network and the image features of the first training image and the image features of the second training image.

[0143] In some embodiments, the interpolation unit 702 is further configured to: obtain an interpolation curve by cubic spline interpolation based on the first image features, the second image features, and the third image features; and sample on the interpolation curve to obtain multiple intermediate image features.

[0144] In some embodiments, the image generation model is a StyleGAN model or a StyleGAN2 model.

[0145] The video generation device provided in this embodiment can be used to execute the technical solutions of the above-described embodiments related to the video generation method. Its implementation principle and technical effects are similar, and will not be repeated here.

[0146] Corresponding to the model determination method in the above embodiments, Figure 8 A structural block diagram of the model determining device provided for embodiments of this disclosure. For ease of explanation, only the parts relevant to embodiments of this disclosure are shown. Refer to Figure 8 The model determination device includes: training unit 801.

[0147] Training unit 801 is used to train a neural network based on multiple training images and an image generation model. The neural network is used to learn the bias of image feature adjustment based on the feature space of the image generation model.

[0148] The training process of the neural network includes: generating target image features based on the image features of the first training image and the image features of the second training image; making preliminary adjustments to the target image features based on the feature space; learning to make preliminary adjustments to the corresponding target deviation through the neural network, and making further adjustments to the target image features after preliminary adjustments based on the target deviation; and adjusting the model parameters of the neural network based on the target deviation, the target image features after further adjustments, the first training image, and the second training image.

[0149] In some embodiments, the training unit 801 is further configured to: determine the target optimization function of the neural network through regularization constraints and similarity constraints; adjust the model parameters of the neural network based on the target optimization function, target deviation, readjusted target image features, first training image and second training image; wherein, the regularization constraints are used to minimize the difference between the readjusted target image features and the initially adjusted target image features, and the similarity constraints are used to minimize the difference between the readjusted target image features and the image features of the first training image and the image features of the second training image.

[0150] The model determination device provided in this embodiment can be used to execute the technical solutions of the embodiments related to the model determination method described above. Its implementation principle and technical effect are similar, and will not be repeated here.

[0151] refer to Figure 9 The diagram illustrates a structural schematic of an electronic device 900 suitable for implementing embodiments of the present disclosure. The electronic device 900 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 9 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0152] like Figure 9 As shown, the electronic device 900 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the electronic device 900. The processing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0153] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 908 including, for example, magnetic tapes, hard disks, etc.; and communication devices 909. Communication device 909 allows electronic device 900 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 9 An electronic device 900 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0154] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by a processing device 901, it performs the functions defined in the methods of embodiments of this disclosure.

[0155] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0156] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0157] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0158] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0159] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0160] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, an acquisition unit can also be described as a "unit for acquiring target audio".

[0161] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0162] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0163] In a first aspect, according to one or more embodiments of the present disclosure, a video generation method is provided, comprising: extracting first image features from a first image; obtaining a plurality of intermediate image features by nonlinear interpolation based on the first image features and second image features, wherein the second image features are image features of a second image; and reconstructing an image using an image generation model based on the first image features, the second image features, and the plurality of intermediate image features to generate a target video, wherein the target video is used to show the process of gradation from the first image to the second image.

[0164] According to one or more embodiments of this disclosure, obtaining a plurality of intermediate image features by nonlinear interpolation based on the first image features and the second image features includes: generating a third image feature based on the first image features and the second image features; adjusting the third image feature sequentially based on the feature space of the image generation model and a neural network, wherein the neural network is used to learn the deviation of image feature adjustment based on the feature space; and performing nonlinear interpolation based on the first image features, the second image features, and the adjusted third image features to obtain the plurality of intermediate image features.

[0165] According to one or more embodiments of this disclosure, the step of adjusting the third image features sequentially based on the feature space of the image generation model and a neural network, wherein the neural network is used to learn the deviation of image feature adjustment based on the feature space, includes: obtaining the average image features in the feature space; performing a preliminary adjustment on the third image features based on the average image features; inputting the first image features and the second image features into the neural network to obtain the output data of the neural network, wherein the output data reflects the deviation of the preliminary adjustment; and further adjusting the third image features after the preliminary adjustment based on the output data.

[0166] According to one or more embodiments of this disclosure, the preliminary adjustment of the third image feature based on the average image feature includes: determining the mean of the third image feature and the average image feature; and determining the preliminarily adjusted third image feature as the mean.

[0167] According to one or more embodiments of this disclosure, the neural network is trained based on regularization constraints and similarity constraints. The regularization constraints are used to minimize the difference between image features adjusted based on the neural network and image features adjusted based on the feature space. The similarity constraints are used to minimize the difference between image features adjusted based on the neural network and image features of a first training image and a second training image.

[0168] According to one or more embodiments of this disclosure, the step of performing nonlinear interpolation based on the first image feature, the second image feature, and the third image feature to obtain the plurality of intermediate image features includes: obtaining an interpolation curve by cubic spline interpolation based on the first image feature, the second image feature, and the third image feature; and sampling on the interpolation curve to obtain the plurality of intermediate image features.

[0169] According to one or more embodiments of this disclosure, the image generation model is a StyleGAN model or a StyleGAN2 model.

[0170] Secondly, according to one or more embodiments of this disclosure, a model determination method is provided, comprising: training a neural network based on a plurality of training images and an image generation model, the neural network being used to learn a deviation in image feature adjustment based on the feature space of the image generation model. One training process of the neural network includes: generating target image features based on image features of a first training image and image features of a second training image; performing preliminary adjustments on the target image features based on the feature space; learning a target deviation corresponding to the preliminary adjustments through the neural network, and further adjusting the pre-adjusted target image features based on the target deviation; and adjusting the model parameters of the neural network based on the target deviation, the further adjusted target image features, the first training image, and the second training image.

[0171] According to one or more embodiments of this disclosure, adjusting the model parameters of the neural network based on the target deviation, the readjusted target image features, and the first training image and the second training image includes: determining the target optimization function of the neural network through regularization constraints and similarity constraints; adjusting the model parameters of the neural network based on the target optimization function, the target deviation, the readjusted target image features, the first training image, and the second training image; wherein the regularization constraints are used to minimize the difference between the readjusted target image features and the initially adjusted target image features, and the similarity constraints are used to minimize the difference between the readjusted target image features and the image features of the first training image and the second training image.

[0172] Thirdly, according to one or more embodiments of this disclosure, a video generation device is provided, comprising: an extraction unit for extracting first image features from a first image; an interpolation unit for obtaining a plurality of intermediate image features by nonlinear interpolation based on the first image features and the second image features, wherein the second image features are image features of a second image; and a video generation unit for performing image reconstruction based on the first image features, the second image features, and the plurality of intermediate image features using an image generation model to generate a target video, wherein the target video is used to show the process of gradation from the first image to the second image.

[0173] Fourthly, according to one or more embodiments of this disclosure, a model determination device is provided, comprising: a training unit, configured to train a neural network based on a plurality of training images and an image generation model, the neural network being configured to learn a deviation in image feature adjustment based on the feature space of the image generation model. In one training process of the neural network, the training module is configured to: generate target image features based on image features of a first training image and image features of a second training image; perform preliminary adjustments on the target image features based on the feature space; learn a target deviation corresponding to the preliminary adjustments through the neural network, and further adjust the pre-adjusted target image features based on the target deviation; and adjust the model parameters of the neural network based on the target deviation, the further adjusted target image features, the first training image, and the second training image.

[0174] Fifthly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0175] The memory stores computer-executed instructions;

[0176] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the video generation method as described in the first aspect or various possible designs of the first aspect above, or causing the at least one processor to perform the model determination method as described in the second aspect or various possible designs of the second aspect above.

[0177] Sixthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, which, when executed by a processor, implement the video generation method as described in the first aspect or various possible designs of the first aspect above, or implement the model determination method as described in the second aspect or various possible designs of the second aspect above.

[0178] In a seventh aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, the computer program product comprising computer execution instructions that, when executed by a processor, implement the video generation method as described in the first aspect or various possible designs of the first aspect, or implement the model determination method as described in the second aspect or various possible designs of the second aspect.

[0179] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0180] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0181] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A video generation method, comprising: extracting a first image feature in a first image; generating a third image feature according to the first image feature and a second image feature, adjusting the third image feature based on a feature space of an image generation model and a neural network in sequence, the neural network being configured to learn a bias for adjusting the image feature based on the feature space to improve similarity of the third image feature with the first image feature and the second image feature, and performing non-linear interpolation according to the first image feature, the second image feature and the adjusted third image feature to obtain a plurality of intermediate image features, the second image feature being an image feature of a second image, the neural network being configured to minimize a difference between the image feature adjusted based on the neural network and the image feature adjusted based on the feature space; reconstructing an image based on the first image feature, the second image feature and the plurality of intermediate image features by the image generation model to generate a target video, wherein the target video is configured to show a process of gradually changing from the first image to the second image. 2.The video generation method of claim 1, wherein the adjusting the third image feature based on the feature space of the image generation model and the neural network in sequence, the neural network being configured to learn the bias for adjusting the image feature based on the feature space, comprises: obtaining an average image feature in the feature space; preliminarily adjusting the third image feature according to the average image feature; inputting the first image feature and the second image feature into the neural network to obtain output data of the neural network, the output data reflecting the bias of the preliminary adjustment; re-adjusting the third image feature preliminarily adjusted according to the output data. 3.The video generation method of claim 2, wherein the preliminarily adjusting the third image feature according to the average image feature comprises: determining a mean value of the third image feature and the average image feature; and determining the third image feature preliminarily adjusted as the mean value. 4.The video generation method of any one of claims 1 to 3, wherein the neural network is trained based on a regular constraint and a similarity constraint, the regular constraint being configured to minimize the difference between the image feature adjusted based on the neural network and the image feature adjusted based on the feature space, and the similarity constraint being configured to minimize a difference between the image feature adjusted based on the neural network and an image feature of a first training image and an image feature of a second training image. 5.The video generation method of any one of claims 1 to 3, wherein the performing non-linear interpolation based on the first image feature, the second image feature and the third image feature to obtain the plurality of intermediate image features comprises: obtaining an interpolation curve by cubic spline interpolation based on the first image feature, the second image feature and the third image feature; and sampling on the interpolation curve to obtain the plurality of intermediate image features.

6. The video generation method of any one of claims 1-3, wherein the image generation model is a StyleGAN model or a StyleGAN2 model.

7. A model determination method comprising: training, according to a plurality of training images and an image generation model, a neural network configured to learn a bias for adjusting image features based on a feature space of the image generation model, the neural network configured to minimize a difference between image features adjusted based on the neural network and image features adjusted based on the feature space; wherein a training process of the neural network comprises: generating a target image feature according to image features of a first training image and a second training image; preliminarily adjusting the target image feature based on the feature space; learning, by the neural network, a target bias corresponding to the preliminary adjustment, and re-adjusting the preliminarily adjusted target image feature according to the target bias; adjusting model parameters of the neural network according to the target bias, the re-adjusted target image feature, the first training image, and the second training image.

8. The model determination method according to claim 7, characterized by, The adjusting model parameters of the neural network according to the target bias, the re-adjusted target image feature, and the first training image and the second training image comprises: determining a target optimization function of the neural network by a regularization constraint and a similarity constraint; adjusting model parameters of the neural network based on the target optimization function, the target bias, the re-adjusted target image feature, the first training image, and the second training image; wherein the regularization constraint is configured to minimize a difference between the re-adjusted target image feature and the preliminarily adjusted target image feature, and the similarity constraint is configured to minimize a difference between the re-adjusted target image feature and the image features of the first training image and the second training image.

9. A video generation device comprising: an extraction unit configured to extract a first image feature in a first image; an interpolation unit configured to generate a third image feature according to the first image feature and a second image feature, adjust the third image feature based on a feature space of an image generation model and a neural network in sequence, the neural network configured to learn a bias for adjusting image features based on the feature space to improve a similarity between the third image feature and the first image feature and the second image feature, and perform non-linear interpolation according to the first image feature, the second image feature, and the adjusted third image feature to obtain a plurality of intermediate image features, the second image feature being an image feature of a second image, the neural network configured to minimize a difference between image features adjusted based on the neural network and image features adjusted based on the feature space. The video generation unit is configured to generate a target video by image reconstruction based on the first image feature, the second image feature and the plurality of intermediate image features through an image generation model, wherein the target video is used to show a process of gradually changing from the first image to the second image.

10. A model determination device, comprising: a training unit configured to train a neural network according to a plurality of training images and an image generation model, wherein the neural network is used to learn a bias for adjusting image features based on a feature space of the image generation model, and the neural network is used to minimize a difference between image features adjusted based on the neural network and image features adjusted based on the feature space; wherein a training process of the neural network comprises: generating target image features according to image features of a first training image and image features of a second training image; preliminarily adjusting the target image features based on the feature space; learning a target bias corresponding to the preliminary adjustment through the neural network, and re-adjusting the preliminarily adjusted target image features according to the target bias; adjusting model parameters of the neural network according to the target bias, the re-adjusted target image features, the first training image and the second training image.

11. An electronic device comprising: at least one processor and a memory; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the video generation method according to any one of claims 1 to 6, or so that the at least one processor executes the model determination method according to claim 7 or 8.

12. A computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the video generation method according to any one of claims 1 to 6 is implemented, or the model determination method according to claim 7 or 8 is implemented.

13. A computer program product, wherein the computer program product contains computer-executable instructions, and when a processor executes the computer-executable instructions, the video generation method according to any one of claims 1 to 6 is implemented, or the model determination method according to claim 7 or 8 is implemented.

Images