Video generation method, electronic device, storage medium, and computer program product

Abstract

This application discloses a video generation method, electronic device, storage medium, and computer program product, belonging to the field of terminal technology. Applied to electronic devices, the method includes: determining a target image in response to a video generation operation; encoding the target image using a first encoding network in a variational autoencoder to obtain a first video; and decoding the first video using a target decoding network in the variational autoencoder based on the target image to obtain a second video. The target decoding network is a decoding network pre-trained on the first decoding network in the variational autoencoder. The first decoding network, in conjunction with the first encoding network, enables the generation of video from an image. This application injects feature information of the target image into the target decoding network, thereby influencing the reconstruction process of the video to pixel space, reducing image information loss during decoding, and improving the display quality of the video.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a video generation method, electronic device, storage medium, and computer program product. Background Technology

[0002] With the development of terminal technology, electronic devices are becoming increasingly feature-rich. For example, to provide users with a better video experience, electronic devices can generate videos from images.

[0003] Currently, after acquiring an image, electronic devices can use the encoding module of a Variational Autoencoder (VAE) to extract image features into the latent space and generate video from them. Then, the VAE's decoding module restores the video from the latent space to the pixel space (spatial domain), thus obtaining a displayable video.

[0004] However, the encoding and decoding processes of images often lead to the loss of image structure and details, resulting in a loss of detail in the video and a reduction in the display quality of the video. Summary of the Invention

[0005] This application provides a video generation method, electronic device, storage medium, and computer program product, which can be used to solve the problem of reduced video quality when implementing image-to-video functionality. The technical solution is as follows:

[0006] Firstly, a method for generating video is provided for use in electronic devices, the method comprising:

[0007] In response to the video generation operation, the target image is determined;

[0008] The target image is encoded using the first coding network in the variational autoencoder to obtain the first video.

[0009] Based on the target image, the first video is decoded by the target decoding network in the variational autoencoder to obtain the second video. The target decoding network is a decoding network obtained by pre-training the first decoding network in the variational autoencoder. The first decoding network works with the first encoding network to realize the function of image-to-video generation.

[0010] In some embodiments, the first encoding network in a VAE typically includes multi-layer neural networks, which may be fully connected layers, convolutional layers, etc.

[0011] As an example, the encoding operation of the first encoding network on the target image may include: compressing the high-dimensional representation of the target image into a low-dimensional latent representation (i.e., a first video) in which key information of the target image is preserved.

[0012] As an example, an electronic device can receive a target image through a first coding network and preprocess the received target image; extract features from the preprocessed target image through the first coding network to obtain first feature information; and generate a first video based on the first feature information and a noise vector.

[0013] Thus, in the process of realizing the function of image-to-video, during the decoding stage, the electronic device adds the target image as a constraint to the target decoding network. The target image contains all the detailed features of the image. In this way, the target image affects the reconstruction process of video decoding to pixel space. Based on the target image, the first video can retain image details as much as possible, thereby reducing the loss of image information caused by the decoding network and improving the display quality of the video.

[0014] As an example of this application, the operation of an electronic device to decode a first video based on a target image using a target decoding network in a variational autoencoder to obtain a second video includes:

[0015] The target image is input into the target decoding network;

[0016] Based on the target image, the first video is decoded using a target decoding network to obtain the second video.

[0017] In other words, during the decoding stage, the electronic device does not directly decode the first video through the target decoding network, but instead inputs the target image into the target decoding network, which can then decode the first video based on the target image.

[0018] It should be noted that the network layers in the target decoding network can include fully connected layers, deconvolutional layers, etc.

[0019] Thus, by directly adding the target image to the target decoding network to decode the first video based on the target image, the loss of image information in the video is reduced, and the quality of the second video is improved.

[0020] As an example of this application, the operation of an electronic device inputting a target image into a target decoding network includes:

[0021] The first feature information is obtained by extracting features from the target image through the first coding network;

[0022] The first feature information is input into the target decoding network;

[0023] Based on this, the electronic device decodes the first video using a target decoding network according to the target image to obtain the second video, including the following operations:

[0024] Based on the first feature information, the first video is decoded using a target decoding network to obtain the second video.

[0025] As an example, since the electronic device has already extracted features from the target image through the first encoding network during the encoding stage, after feature extraction through the first encoding network, the electronic device can, on the one hand, continue to encode the first video based on the first feature information through the first encoding network; on the other hand, the electronic device can add the obtained first feature information to the target decoding network. Thus, during the decoding stage, the electronic device can directly decode the first video through the target decoding network based on the first feature information.

[0026] In this way, by directly using the first feature information extracted from the coding network, the operation of electronic devices is reduced and the efficiency of obtaining the first feature information is improved.

[0027] As an example of this application, the operation of an electronic device to decode a first video using a target decoding network based on first feature information to obtain a second video includes:

[0028] The first video is decoded step by step using a target decoding network;

[0029] During the decoding process, the first feature information and the second feature information obtained during the decoding process are fused together using a cross-attention method to obtain the second video.

[0030] It should be noted that the image features extracted by the neural network (such as a convolutional layer) near the input end of the first encoding network are different from those extracted by the neural network near the output end. For example, the neural network near the input end of the first encoding network can extract information such as color and texture from the target image, and the image features extracted by the neural network near the output end result in less loss of detail. Therefore, when the first feature information of the target image is added to the target decoding network using a cross-attention method, the electronic device can add the first feature information extracted by the neural network near the input end of the first encoding network to the neural network near the output end of the target decoding network.

[0031] In this way, by adding the first feature information extracted from the target image to the target decoding network in a cross-attention manner, the video decoding's ability to extract the original information is improved, thereby achieving the goal of preserving image details, better retaining image details, and improving video quality.

[0032] As an example of this application, the electronic device can also obtain a target decoding network through iterative training before determining the target image in response to a video generation operation.

[0033] For example, the operation of an electronic device to obtain a target decoding network through iterative training includes:

[0034] Obtain a first training set, which includes multiple sets of training videos and multiple sample images. The multiple sets of training videos correspond one-to-one with the multiple sample images, and each set of training videos is generated by a first encoding network based on the corresponding sample images.

[0035] Keeping the first encoding network unchanged, the first decoding network is iteratively trained based on the first training set;

[0036] During iterative training, the target loss value between the decoding network obtained after each training iteration and the decoding network before each training iteration is determined.

[0037] If the target loss value converges, the decoding network obtained at the convergence point is determined as the target decoding network.

[0038] Thus, since VAE can already achieve image-to-video generation, freezing the first encoder network while continuing iterative training of the first decoder network reduces the workload of iterative training and improves its efficiency. Furthermore, the ability to set multiple loss functions enhances the accuracy of training the decoder network.

[0039] In some embodiments, the electronic device can not only determine the decoded network obtained at convergence as the target decoded network when the target loss value converges, but also end the iterative training and obtain the target decoded network in other cases. For example, the electronic device can also determine the number of training iterations. If the number of training iterations is greater than or equal to a threshold, the electronic device can determine that the iterative training has ended and determine the decoded network obtained at the end of the iterative training as the target decoded network.

[0040] As an example of this application, the operation of determining the target loss value between the decoding network obtained after each training iteration and the decoding network before each training iteration in the electronic device includes:

[0041] During iterative training, the output data of the decoding network after each training session is compared with the input data of the decoding network before each training session.

[0042] Based on the output data and input data, determine the first loss value of the first loss function, the second loss value of the second loss function, and the third loss value of the third loss function;

[0043] The target loss value is determined based on the first loss value, the second loss value, and the third loss value.

[0044] For example, the first loss function can be the GAN Loss function, the second loss function can be the MSE Loss function, and the third loss function can be the VGG16 function.

[0045] In this way, by setting multiple loss functions, the first decoding network can be accurately trained iteratively, ensuring the accuracy of the target decoding network.

[0046] As an example of this application, the operation of the electronic device to determine a target loss value based on a first loss value, a second loss value, and a third loss value includes:

[0047] The sum of the first loss value, the second loss value, and the third loss value is determined as the target loss value; or,

[0048] Multiply the first loss value by the first weight to obtain the first value; multiply the second loss value by the second weight to obtain the second value; multiply the third loss value by the third weight to obtain the third value; and determine the target loss value by summing the first, second, and third values.

[0049] Thus, by determining the target loss value in different ways, the variety of ways to determine the target loss value is increased.

[0050] Secondly, a video generation apparatus is provided, which has the function of implementing the video generation method described in the first aspect. The video generation apparatus includes at least one module, which is used to implement the video generation method provided in the first aspect. The video generation apparatus may include:

[0051] The first determining module is used to determine the target image in response to the video generation operation;

[0052] The encoding module is used to encode the target image through the first encoding network in the variational autoencoder to obtain the first video;

[0053] The decoding module is used to decode the first video based on the target image through the target decoding network in the variational autoencoder to obtain the second video. The target decoding network is a decoding network obtained by pre-training the first decoding network in the variational autoencoder. The first decoding network works with the first encoding network to realize the function of image-to-video generation.

[0054] As an example of this application, the decoding module is used for:

[0055] The target image is input into the target decoding network;

[0056] Based on the target image, the first video is decoded using a target decoding network to obtain the second video.

[0057] As an example of this application, the decoding module is used for:

[0058] The first feature information is obtained by extracting features from the target image through the first coding network;

[0059] The first feature information is input into the target decoding network;

[0060] Based on the first feature information, the first video is decoded using a target decoding network to obtain the second video.

[0061] As an example of this application, the decoding module is used for:

[0062] The first video is decoded step by step using a target decoding network;

[0063] During the decoding process, the first feature information and the second feature information obtained during the decoding process are fused together using a cross-attention method to obtain the second video.

[0064] As an example of this application, the device also includes:

[0065] The acquisition module is used to acquire a first training set, which includes multiple sets of training videos and multiple sample images. The multiple sets of training videos correspond one-to-one with the multiple sample images, and each set of training videos is generated by the first encoding network based on the corresponding sample images.

[0066] The training module is used to keep the first encoding network unchanged and iteratively train the first decoding network based on the first training set;

[0067] The second determination module is used to determine the target loss value between the decoding network obtained after each training and the decoding network before each training during the iterative training process.

[0068] The third determination module is used to determine the decoding network obtained at the time of convergence as the target decoding network when the target loss value converges.

[0069] As an example of this application, the second determining module is used for:

[0070] During iterative training, the output data of the decoding network after each training session is compared with the input data of the decoding network before each training session.

[0071] Based on the output data and input data, determine the first loss value of the first loss function, the second loss value of the second loss function, and the third loss value of the third loss function;

[0072] The target loss value is determined based on the first loss value, the second loss value, and the third loss value.

[0073] As an example of this application, the second determining module is used for:

[0074] The sum of the first loss value, the second loss value, and the third loss value is determined as the target loss value; or,

[0075] Multiply the first loss value by the first weight to obtain the first value; multiply the second loss value by the second weight to obtain the second value; multiply the third loss value by the third weight to obtain the third value; and determine the target loss value by summing the first, second, and third values.

[0076] Thirdly, an electronic device is provided, comprising a processor and a memory. The memory stores a program that supports the electronic device in executing the video generation method provided in the first aspect, and stores data related to implementing the video generation method described in the first aspect. The processor is configured to execute the program stored in the memory. The electronic device may further include a communication bus for establishing a connection between the processor and the memory.

[0077] Fourthly, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the video generation method described in the first aspect.

[0078] Fifthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to execute the video generation method described in the first aspect above.

[0079] The technical effects achieved by the second, third, fourth, and fifth aspects mentioned above are similar to those achieved by the corresponding technical means in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0080] Figure 1 This is a schematic diagram of a process for generating video from an image using a VAE in a related technology provided in an embodiment of this application;

[0081] Figure 2 This is a schematic diagram illustrating an application scenario of a related technology provided in an embodiment of this application;

[0082] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0083] Figure 4This is a block diagram of a software system for an electronic device provided in an embodiment of this application;

[0084] Figure 5 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0085] Figure 6 This is a schematic flowchart of a video generation method provided in an embodiment of this application;

[0086] Figure 7 This is a schematic diagram of a process for generating video from an image using a VAE, provided in an embodiment of this application.

[0087] Figure 8 This is a schematic diagram of another process for realizing image-generated video using VAE, provided in an embodiment of this application;

[0088] Figure 9 This is a schematic diagram illustrating the addition of feature information using a cross-attention method, as provided in an embodiment of this application.

[0089] Figure 10 This is a schematic diagram of an iterative training method for a target decoding model provided in an embodiment of this application;

[0090] Figure 11 This is a schematic diagram of a loss function provided in an embodiment of this application;

[0091] Figure 12 This is a schematic diagram of a video generation device provided in an embodiment of this application. Detailed Implementation

[0092] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0093] It should be understood that "multiple" as mentioned in this application refers to two or more. In the description of this application, unless otherwise stated, " / " indicates "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist, for example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, to facilitate a clear description of the technical solutions of this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and that "first," "second," etc., do not necessarily imply differences.

[0094] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0095] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0096] With the development of terminal technology, the image-to-video function of electronic devices is becoming increasingly popular among users. Image-to-video function refers to the ability of an electronic device to generate video from at least one frame of an image.

[0097] Currently, with the increasing demand for high-resolution and high-frame-rate video, variational autoencoders (VAEs) are widely used in image-to-video generation methods for latent space compression and restoration of images and videos. This means that electronic devices can use VAEs to generate video from images. See also Figure 1 A VAE typically includes an encoder (also called an encoding module, encoding network, etc.) and a decoder (also called a decoding module, decoding network, etc.). The encoder is used to map the input image to a latent space, generating a low-dimensional representation of the video. The decoder is used to sample latent variables from the latent space and reconstruct the original image (i.e., video image reconstructed to pixel space).

[0098] However, due to information loss in the process of VAE encoding images into the latent space and decoding video into the pixel space, the quality of the final generated video is reduced. For example, see... Figure 2 In Figure (a), the electronic device can generate a representation such as image A. Figure 2 In Figure (b) of the video B, a frame C in video B can be shown as follows: Figure 2 As shown in Figure (c) of the document. Figure 2As shown in Figure (c), some details of the fallen leaves are lost in this frame image C, and the user cannot see the details of the fallen leaves clearly.

[0099] To improve the display quality of videos generated by electronic devices, this application provides a video generation method. In this method, after determining a target image, the electronic device can encode the target image using a Video Image Encoding (VAE) to obtain a first video. Subsequently, during the decoding process, the VAE can decode the first video using a target decoding network based on the target image to obtain a second video. Because the electronic device adds the target image as a constraint during the decoding of the first video, and the target image contains all the detailed features of the image, the target image influences the reconstruction process of decoding the video to pixel space, reducing image information loss caused by the decoding network and improving the display quality of the video.

[0100] In other words, in the context of image-generated video in related technologies, an electronic device acquires an original image, which is then processed by the encoding network of a VAE to obtain a video vector in the latent space. This video vector in the latent space suffers further loss of information from the original image. In order to reduce the loss of details and layout information of the original image during the reconstruction process, the embodiments of this application can inject the feature information of the original image into the decoding network. This process does not affect the video generation process, but only the video reconstruction process. That is, in the process of restoring the video vector in the latent space to the pixel space, the feature information of the original image is used as a guide through cross-attention to reduce the information loss caused by the decoding network.

[0101] Before providing a detailed explanation of the video generation method provided in the embodiments of this application, the electronic equipment involved in the embodiments of this application will be described first.

[0102] As an example, this method can be applied to electronic devices capable of generating video via VAE. As an example and not a limitation, the electronic device can be, but is not limited to, tablet computers, desktop computers, laptop computers, handheld computers, laptops, in-vehicle devices, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), mobile phones, smartwatches, etc., and this application embodiment does not limit this.

[0103] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. See also... Figure 3The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0104] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0105] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0106] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0107] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0108] In some embodiments, the processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0109] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C interfaces. The processor 110 can couple to the touch sensor 180K, charger, flash, camera 193, etc., through different I2C interfaces. For example, the processor 110 can couple to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C interface, thereby realizing the touch function of the electronic device 100.

[0110] The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple I2S interfaces. The processor 110 can be coupled to the audio module 170 through the I2S interface to realize communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the I2S interface to realize the function of answering phone calls through a Bluetooth headset.

[0111] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via the PCM interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering phone calls through a Bluetooth headset.

[0112] The UART interface is a universal serial data bus used for asynchronous communication. The UART interface can be a bidirectional communication bus. It can convert data between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface to enable music playback through Bluetooth headphones.

[0113] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to enable the electronic device 100 to capture images. The processor 110 and the display screen 194 communicate via the DSI interface to enable the electronic device 100 to display images.

[0114] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to a camera 193, a display screen 194, a wireless communication module 160, an audio module 170, a sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0115] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. USB port 130 can also be used to connect other terminals, such as AR devices.

[0116] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0117] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via a USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device 100 via the power management module 141.

[0118] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, internal memory 121, external memory, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0119] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0120] Antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0121] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0122] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0123] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0124] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. Wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. GNSS can include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0125] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0126] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is an integer greater than 1.

[0127] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0128] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's image sensor. The light signal is converted into an electrical signal, and the image sensor transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimizations on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be integrated into the camera 193.

[0129] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is an integer greater than 1.

[0130] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP performs Fourier transforms on the frequency energy.

[0131] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0132] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, text understanding, and image-to-video generation.

[0133] It should be noted that the NPU may include VAEs, and VAEs are compatible with the stablediffusion model. This application does not impose specific limitations on this aspect.

[0134] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions, such as saving music, video, and other files on the external memory card.

[0135] Internal memory 121 can be used to store computer-executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created by electronic device 100 during use (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0136] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D and application processor.

[0137] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0138] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the touch operation intensity based on pressure sensor 180A. Electronic device 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example, when a touch operation with an intensity less than the pressure threshold is applied to the SMS application icon, a command to view an SMS message is executed. When a touch operation with an intensity greater than or equal to the pressure threshold is applied to the SMS application icon, a command to create a new SMS message is executed.

[0139] The gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100. In some embodiments, the gyroscope sensor 180B can determine the angular velocity of the electronic device 100 about three axes (i.e., the x, y, and z axes). The gyroscope sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the shake of the electronic device 100, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the electronic device 100 by moving in the opposite direction, thus achieving image stabilization. The gyroscope sensor 180B can also be used in navigation and motion-sensing game scenarios.

[0140] The barometric pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates altitude using the air pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0141] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip cover. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover based on the magnetic sensor 180D. Based on the detected opening and closing state of the cover or the flip cover, the electronic device 100 can set features such as automatic flip unlocking.

[0142] The accelerometer 180E can detect the magnitude of acceleration of electronic device 100 in various directions (generally three axes). When electronic device 100 is stationary, it can detect the magnitude and direction of gravity. The accelerometer 180E can also be used to identify the attitude of electronic device 100, and can be applied to applications such as screen orientation switching and pedometers.

[0143] A distance sensor 180F is used to measure distance. Electronic device 100 can measure distance via infrared or laser. In some embodiments, during a shooting scenario, electronic device 100 can utilize the distance sensor 180F for distance measurement to achieve fast focusing.

[0144] The proximity sensor 180G may include a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The electronic device 100 emits infrared light outward through the LED. The electronic device 100 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, the electronic device 100 can determine that an object is nearby. When insufficient reflected light is detected, it can determine that no object is nearby. The electronic device 100 can use the proximity sensor 180G to detect when a user holds the electronic device 100 close to their ear for a phone call, so as to automatically turn off the screen to save power. The proximity sensor 180G can also be used in holster mode and pocket mode for automatic unlocking and locking of the screen.

[0145] The ambient light sensor 180L is used to sense the brightness of ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 194 based on the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also work with the proximity sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0146] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can utilize the characteristics of the collected fingerprints to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0147] Temperature sensor 180J is used to detect temperature. In some embodiments, electronic device 100 uses the temperature detected by temperature sensor 180J to execute a temperature handling strategy. For example, when the temperature reported by temperature sensor 180J exceeds a threshold, electronic device 100 reduces the performance of a processor located near temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is below another threshold, electronic device 100 heats battery 142 to prevent abnormal shutdown of electronic device 100 due to low temperature. In still other embodiments, when the temperature is below yet another threshold, electronic device 100 boosts the output voltage of battery 142 to prevent abnormal shutdown due to low temperature.

[0148] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touch display." Touch sensor 180K detects touch operations applied to or near it. Touch sensor 180K can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0149] The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire vibration signals from the vibrating bone segments of the human vocal cords. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure signals. In some embodiments, the bone conduction sensor 180M can also be incorporated into headphones to form bone conduction headphones. The audio module 170 can parse the voice signals from the vibrating bone segments of the vocal cords acquired by the bone conduction sensor 180M to realize voice functionality. The application processor can parse heart rate information from the blood pressure signals acquired by the bone conduction sensor 180M to realize heart rate detection functionality.

[0150] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0151] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can be corresponding to touch operations applied to different applications (such as taking photos, playing audio, etc.). Touch operations applied to different areas of the display screen 194 can also correspond to different vibration feedback effects. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0152] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0153] The software system of electronic device 100 will be described next.

[0154] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses a layered Android system as an example to illustrate the software system of electronic device 100.

[0155] Figure 4 This is a block diagram of a software system for an electronic device 100 provided in an embodiment of this application. See also... Figure 4 A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime, the system layer, and the kernel layer.

[0156] The application layer can include a series of application packages. For example... Figure 4 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0157] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example... Figure 2As shown, the application framework layer can include a window manager, content providers, a view system, a phone manager, a resource manager, and a notification manager. The window manager manages window programs. It can obtain the screen size, determine if a status bar is present, lock the screen, and capture the screen. The content provider stores and retrieves data, making this data accessible to the application. This data can include videos, images, audio, made and received phone calls, browsing history and bookmarks, and phone books. The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build the application's display interface, which can consist of one or more views, such as a view displaying SMS notification icons, a view displaying text, and a view displaying images. The phone manager provides communication functions for the electronic device 100, such as managing call status (including connection and disconnection). The resource manager provides the application with various resources, such as localized strings, icons, images, layout files, and video files. The notification manager allows the application to display notification information in the status bar, which can be used to convey informational messages and can disappear automatically after a short pause without user interaction. For example, the notification manager is used to notify users of download completions and message alerts. The notification manager can also display notifications as icons or scrolling text in the system's top status bar, such as notifications from background applications. Furthermore, the notification manager can appear as dialog boxes on the screen, such as displaying text messages in the status bar, emitting sounds, causing electronic devices to vibrate, or flashing indicator lights.

[0158] The Android Runtime consists of the core libraries and the virtual machine. The Android runtime is responsible for scheduling and managing the Android system. The core libraries consist of two parts: one part contains the functionalities that Java needs to call, and the other part is the core Android library itself. The application layer and application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0159] The system library can include multiple functional modules, such as a surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), and 2D graphics engines (e.g., SGL). The surface manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The media libraries support playback and recording of various common audio and video formats, as well as still image files. The media libraries support various audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG. The 3D graphics processing libraries are used for 3D graphics drawing, image rendering, compositing, and layer processing. The 2D graphics engine is the drawing engine for 2D graphics.

[0160] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0161] To facilitate understanding of the embodiments of this application, the application scenarios involved in the embodiments of this application will be explained below, and in the embodiments of this application, a mobile phone is used as an example for explanation.

[0162] Please refer to Figure 5 , Figure 5 This is one application scenario provided by an embodiment of this application. In one application scenario, the electronic device can obtain, for example... Figure 5 In Figure (a), image A is shown. An electronic device can encode image A using a Video Image Array (VAE) to obtain the first video; that is, the VAE can map image A into a latent space. Then, the electronic device can sample the first video from the latent space using the VAE and decode it using cross-attention based on image A to obtain the video as shown in Figure (a). Figure 5 The second video D shown in Figure (b) can have a frame E in it as follows: Figure 5 As shown in Figure (c), most of the image details, such as the details of fallen leaves, are preserved in frame E, and users can observe the details of fallen leaves.

[0163] It should be noted that, in the embodiments of this application, the following are used: Figure 5 The application scenarios shown are illustrated as examples and do not constitute a limitation on the embodiments of this application.

[0164] Based on the execution entity provided in the above embodiments, the video generation method provided in this application will be described next. Please refer to... Figure 6 , Figure 6This is a schematic diagram illustrating a video generation method, which is provided as an example and not a limitation. The method is illustrated by taking an application to an electronic device as an example, and may include some or all of the following:

[0165] Step 601: The electronic device, in response to the video generation operation, determines the target image.

[0166] It should be noted that the target image can be any image in the image library of the electronic device, or it can be any image on the network.

[0167] In one possible scenario, while browsing images in the gallery of an electronic device, if a user selects an image and then triggers a video generation operation, the electronic device, in response to the video generation operation, can designate the selected image as the target image. Alternatively, upon receiving a video generation operation triggered by the user, the electronic device can display all images in its gallery, allowing the user to select an image from the displayed images. In response to this selection operation, the electronic device will designate the selected image as the target image.

[0168] In another possible scenario, if a user selects a displayed online image while browsing images on an electronic device and then triggers a video generation operation, the electronic device can respond to the video generation operation and designate the selected online image as the target image.

[0169] It should be noted that a user may select one image or multiple images. If the user selects multiple images, the electronic device can generate a video from the selected images. However, for ease of explanation, this embodiment uses a single image as the target image for illustration; that is, this embodiment uses the generation of a video from a single image as an example to illustrate the embodiment.

[0170] As an example, the operation of an electronic device to generate video based on a target image can refer to the following steps 602-604.

[0171] Step 602: The electronic device inputs the target image into the VAE.

[0172] As an example, an electronic device can load a pre-trained VAE and then input the target image into the VAE.

[0173] In some embodiments, if a pre-trained VAE already exists before the electronic device inputs the target image into the VAE, the electronic device can directly load the VAE and input the target image into it. If the electronic device does not have a VAE that meets the video generation conditions, the electronic device can perform iterative training of the VAE first.

[0174] It should be noted that a VAE that meets the video generation requirements refers to a VAE capable of generating video from an image, and in the decoding stage, the electronic device can preserve as much image detail as possible through the VAE. In other words, a VAE that meets the video generation requirements includes a first encoding network and a target decoding network. If a VAE includes both a first encoding network and a first decoding network, then the VAE does not meet the video generation requirements.

[0175] As an example, the operation of iterative training of VAE can be referred to below. Figure 10 The contents shown in this application embodiment will not be described in detail.

[0176] As an example, a VAE may include a first encoding network and a target decoding network. When an electronic device inputs a target image into a VAE, it means that the electronic device uses the target image as input to the first encoding network.

[0177] Step 603: The electronic device encodes the target image through the encoding network in the VAE to obtain the first video.

[0178] In some embodiments, the first encoding network in the VAE typically includes multi-layer neural networks, such as fully connected layers, convolutional layers, etc. The encoding operation of the first encoding network on the target image may include: compressing the high-dimensional representation of the target image into a low-dimensional latent representation (i.e., a first video), in which key information of the target image is preserved.

[0179] As an example, an electronic device can receive a target image through a first coding network and preprocess the received target image; extract features from the preprocessed target image through the first coding network to obtain first feature information; and generate a first video based on the first feature information and a noise vector.

[0180] It should be noted that, in order to enable the first coding network to capture the overall distribution of the video data in the latent space as accurately as possible, random variables, i.e., noise vectors, are typically introduced during the coding process. For example, see [link to documentation]. Figure 7The electronic device can use the concat function to convert the initial feature information and noise vector into the distribution parameters of the latent variables (such as the mean and variance of a Gaussian distribution). These distribution parameters can be interpolated or transformed in the latent space to generate video frames with continuity and smoothness, that is, the electronic device can generate the first video in the latent space.

[0181] In some embodiments, the preprocessing operations include image expansion, elimination, normalization, and denoising. Within the encoding network, each layer of the neural network performs a non-linear transformation on the input data to extract features at different levels. The first feature information may include image texture features, color features, layout features, shape features, edge features, etc.

[0182] Step 604: The electronic device decodes the first video based on the target image through the target decoding network in the VAE to obtain the second video.

[0183] As described above, a VAE can also include a target decoding network, which can map the representation in the latent space back to the original data space (i.e., pixel space), thereby generating new sample data (i.e., the second video). Therefore, in order to obtain a displayable video, the electronic device can use the target decoding network in the VAE to decode the first video to obtain the second video.

[0184] Furthermore, since some image information may be lost during the encoding stage when the electronic device generates the first video, in order to reduce information loss during decoding and improve the display quality of the generated video, the electronic device maps the target image to the latent space through the first encoding network. After obtaining the first video, during the decoding stage, the electronic device can add the target image to the target decoding network and decode the first video based on the target image. In other words, during the decoding stage, the electronic device can use the target image as a guide to perform decoding operations on the first video through the target decoding network.

[0185] In some embodiments, the operation of an electronic device to decode a first video based on a target image through a target decoding network to obtain a second video includes: inputting the target image into the target decoding network; and decoding the first video based on the target image through the target decoding network to obtain the second video.

[0186] In other words, during the decoding stage, the electronic device does not directly decode the first video through the target decoding network, but instead inputs the target image into the target decoding network, which can then decode the first video based on the target image.

[0187] In some embodiments, after the electronic device inputs the target image into the target decoding network, it can extract features from the target image through the target decoding network to obtain third feature information. In this way, the electronic device can perform decoding operations on the first video through the target decoding network based on the third feature information to obtain the second video.

[0188] As an example, during the decoding process, the electronic device can add third feature information to the target decoding network in a cross-attention manner. That is, the electronic device uses the third feature information as an operation guide (also known as a condition constraint) for the target decoding network to perform decoding operations through cross-attention.

[0189] It should be noted that the target decoding network extracts features from the target image, and the resulting third feature information may be the same as or different from the first feature information.

[0190] In some embodiments, the electronic device can decode the first video to obtain the second video based on the third feature information through a target decoding network. This includes the electronic device receiving latent variables (or latent sequences) sampled from the latent space through the target decoding network. These latent variables are a low-dimensional representation (i.e., the first video) obtained by encoding the target image through the first encoding network. The target decoding network performs a nonlinear transformation on the latent variables and the third feature information through a series of neural network layers. During the nonlinear transformation, the third feature information is added to the target decoding network in a cross-fusion manner. After the nonlinear transformation, the target decoding network outputs the reconstructed second video.

[0191] It should be noted that the series of network layers in the target decoding network may include fully connected layers, deconvolutional layers, etc., and this application embodiment does not impose specific limitations on this.

[0192] It is worth noting that by directly adding the target image to the target decoding network to perform decoding operations on the first video based on the target image, the loss of image information in the video is reduced, thereby improving the quality of the second video.

[0193] As can be seen from the above, the first encoding network can also extract features from the target image; therefore, see [link to relevant documentation]. Figure 8 The electronic device can also reuse the first encoding network to extract features from the target image, obtaining first feature information. Based on this, the electronic device can input the first feature information into the target decoding network, and decode the first video using the target decoding network according to the first feature information to obtain the second video.

[0194] Since the electronic device has already extracted features from the target image through the first coding network during the encoding stage, after the electronic device extracts features from the target image through the first coding network during the encoding stage, see [link to relevant documentation]. Figure 7 On the one hand, the electronic device can continue to encode the first video using the first encoding network based on the first feature information; on the other hand, the electronic device can add the obtained first feature information to the target decoding network. Thus, during the decoding stage, the electronic device can directly decode the first video using the target decoding network based on the first feature information.

[0195] It is worth noting that by directly using the first feature information extracted from the coding network, the operation of electronic devices is reduced, and the efficiency of obtaining the first feature information is improved.

[0196] In some embodiments, the operation of an electronic device to decode a first video using a target decoding network based on first feature information to obtain a second video includes: progressively decoding the first video using the target decoding network; and fusing the first feature information with the second feature information obtained in each decoding process using a cross-attention approach during the decoding process to obtain the second video.

[0197] In some embodiments, the operation of an electronic device decoding a first video based on a target image through a target decoding network in a VAE to obtain a second video may include not only the methods described above, but also other methods. For example, see... Figure 9 The electronic device can perform feature extraction on the target image through a first encoding network to obtain first feature information, and then perform decoding on the first video through a target decoding network to obtain second feature information. In each encoding operation that yields a piece of first feature information, the first feature information is added to the target encoding network using a cross-attention (cross-att) approach. This first feature information serves as guiding information for a decoding operation (i.e., the aforementioned guiding condition), and in each decoding process, the first and second feature information can be fused.

[0198] It should be noted that the image features extracted by the neural network (such as convolutional layers) near the input end of the first encoding network are different from those extracted by the neural network near the output end. For example, the neural network near the input end in the first encoding network can extract information such as color and texture from the target image, while the image features extracted by the neural network near the output end suffer less loss of detail. Therefore, see [link to relevant documentation]. Figure 9When the first feature information of the target image is added to the target decoding network in a cross-attention manner, the electronic device can add the first feature information extracted by the neural network near the input end of the first encoding network to the neural network near the output end of the target decoding network.

[0199] It should also be noted that during the decoding process, the electronic device fuses the first feature information with the second feature information obtained in each decoding process in a cross-attention manner to obtain the second video. This operation can refer to the operation described above where the electronic device decodes the first video based on the third feature information through the target decoding network to obtain the second video. This application embodiment will not elaborate on this further.

[0200] It is worth noting that by adding the first feature information extracted from the target image to the target decoding network in a cross-attention manner, the video decoding's ability to extract the original information is improved, thereby achieving the goal of preserving image details, better retaining image details, and improving video quality.

[0201] As an example, after obtaining the second video through the target decoding network, the electronic device can further optimize the second video, such as by denoising and enhancing contrast, to improve the display quality of the second video.

[0202] In some embodiments, since the first frame of the first video typically contains a large amount of image information from the original image (i.e., the target image), the electronic device can decode the first video based on the target image using the target decoding network in the VAE to obtain the second video. Alternatively, the first frame of the first video can be acquired, and the second video can be obtained based on the first frame by decoding the first video using the target decoding network in the VAE.

[0203] It should be noted that the operation of the electronic device decoding the first video based on the first frame image through the target decoding network in the VAE to obtain the second video can be referred to the operation of the electronic device decoding the first video based on the target image through the target decoding network in the VAE to obtain the second video. This application embodiment will not elaborate on this step.

[0204] In the embodiments of this application, during the decoding stage of the electronic device, the target image is added as a constraint to the target decoding network in the process of realizing the function of image-generated video. The target image contains all the detailed features of the image. In this way, the target image affects the reconstruction process of video decoding to pixel space. Based on the target image, the first video can retain the image details as much as possible, thereby reducing the loss of image information caused by the decoding network and improving the display quality of the video.

[0205] Next, the process of training the target decoding network using an electronic device will be explained. Please refer to [link / reference needed]. Figure 10 This method will be illustrated using an application to electronic devices as an example.

[0206] Step 1001: Obtain the sample training set.

[0207] It should be noted that the content of the sample training set acquired by the electronic device will vary depending on the VAE acquired and / or the training method of the VAE. For example, the sample training set may include a first training set and a second training set, or the sample training set may include the first training set, or the sample training set may include the second training set.

[0208] For example, if the VAE acquired by the electronic device has not undergone any iterative training (i.e., the VAE includes an initial encoding network and an initial decoding network, neither of which has been iteratively trained), the electronic device cannot achieve image-to-video functionality through the VAE. In this case, the sample training set acquired by the electronic device can include a first training set and a second training set. The second training set is used to train the initial encoding and decoding networks to obtain a first encoding network and a first decoding network, which enable the electronic device to achieve image-to-video functionality. The first training set is used to iteratively train the first decoding network after the electronic device has achieved image-to-video functionality through the VAE, to obtain the target decoding network.

[0209] In other words, typically, the first encoding network and the first decoding network of a VAE can be trained simultaneously. In one possible approach, the electronic device can acquire a first training set and a second training set; based on the second training set, it can simultaneously train the initial encoding network and the initial decoding network to obtain a VAE capable of generating image-to-video (the VAE includes a first decoding network and a first decoding network). After obtaining the VAE capable of generating image-to-video, in order to reduce image loss during decoding and preserve image details as much as possible to improve the display quality of the generated video, the electronic device can keep the first encoding network unchanged and continue to iteratively train the first decoding network based on the first training set, thereby obtaining the aforementioned target decoding network.

[0210] It should be noted that during the iterative training of the initial decoding and encoding networks in the electronic device, the initial decoding and encoding networks can be optimized together. By minimizing the reconstruction loss and KL divergence loss (which measures the difference between the latent variable distribution and the prior distribution), the decoding network gradually learns how to more accurately generate high-quality video data from the latent space.

[0211] In some embodiments, the electronic device can directly acquire a VAE capable of generating video, i.e., the VAE acquired by the electronic device includes a first encoding network and a first decoding network. Then, in order to reduce the loss of image during the decoding process and improve the display quality of the generated video, the electronic device can acquire a first training set and iteratively train the first decoding network based on the first training set to obtain the aforementioned target decoding network.

[0212] As an example, the first training set may include multiple sets of training videos and multiple sample images, with each set of training videos corresponding to one of the multiple sample images. Each set of training videos is generated by the first encoding network based on the corresponding sample images, and each set of training videos includes multiple frames of images. The second training set includes multiple different sample images.

[0213] It should be noted that the iterative training process of VAE may be affected by a variety of factors, such as the choice of model architecture, the setting of hyperparameters, and the data preprocessing method. Furthermore, the operation of iteratively training the initial encoding network and the initial decoding network based on the second training set to obtain the first encoding network and the first decoding network can be referred to relevant technologies, and will not be described in detail in this embodiment.

[0214] To facilitate understanding of the embodiments of this application, the following embodiments will be described using the example of a VAE including a first encoding network and a first decoding network, and an electronic device iteratively training the first decoding network based on a first training set to obtain a target decoding network. The operation of iteratively training the first decoding network based on the first training set by the electronic device can be referred to the operations of steps 1002-1004 below.

[0215] Step 1002: Keep the first encoding network unchanged and iteratively train the first decoding network based on the first training set.

[0216] That is, the electronic device keeps the first encoding network unchanged and inputs each training video and corresponding sample image from the first training set into the first decoding network for iterative training.

[0217] Step 1003: During the iterative training process, determine the target loss value between the decoding network obtained after each training session and the decoding network before each training session.

[0218] In some embodiments, during iterative training, the operation of the electronic device to determine the target loss value between the decoding network obtained after each training session and the decoding network before each training session includes: during iterative training, determining the output data of the decoding network obtained after each training session and the input data of the decoding network before each training session; determining a first loss value of a first loss function, a second loss value of a second loss function, and a third loss value of a third loss function based on the output data and the input data; and determining the target loss value based on the first loss value, the second loss value, and the third loss value.

[0219] It should be noted that during the iterative training of the first decoding network, the electronic device may have at least one loss function set. In order to improve the accuracy of training, this embodiment of the application uses the setting of three loss functions as an example for illustration.

[0220] For example, see Figure 11 The first loss function can be the GAN Loss function, the second loss function can be the MSELoss function, and the third loss function can be the VGG16 function. This application does not impose specific limitations on these aspects.

[0221] It is worth noting that by setting multiple loss functions, the first decoding network can be accurately trained iteratively, ensuring the accuracy of the target decoding network.

[0222] In some embodiments, the operation of an electronic device determining a target loss value based on a first loss value, a second loss value, and a third loss value includes: determining the target loss value as the sum of the first loss value, the second loss value, and the third loss value. Alternatively, multiplying the first loss value by a first weight to obtain a first value, multiplying the second loss value by a second weight to obtain a second value, multiplying the third loss value by a third weight to obtain a third value; and determining the target loss value as the sum of the first value, the second value, and the third value.

[0223] It should be noted that the first weight, second weight, and third weight can be preset according to requirements. The first weight, second weight, and third weight can be the same or different. In general, the first weight, second weight, and third weight can be set to be different from each other.

[0224] It is worth noting that determining the target loss value through different methods increases the variety of ways to determine the target loss value.

[0225] Step 1004: If the target loss value converges, the decoding network obtained at the convergence point is determined as the target decoding network.

[0226] As an example, the target loss value converges when it is less than or equal to a first preset value, and / or when the change in the target loss value is less than or equal to a second preset value. Both the first and second preset values ​​can be preset as needed.

[0227] In some embodiments, the electronic device can not only determine the decoded network obtained at convergence as the target decoded network when the target loss value converges, but also end the iterative training and obtain the target decoded network in other cases. For example, the electronic device can also determine the number of training iterations. If the number of training iterations is greater than or equal to a threshold, the electronic device can determine that the iterative training has ended and determine the decoded network obtained at the end of the iterative training as the target decoded network.

[0228] It should be noted that the number of times threshold can also be preset according to needs. For example, the number of times threshold can be 100 times, 150 times, etc.

[0229] It should also be noted that the above description is based on the case where the VAE acquired by the electronic device includes a first encoding network and a first decoding network, and the first decoding network is trained iteratively separately. In the embodiments of this application, when the VAE includes an initial encoding network and an initial decoding network, the electronic device can also iteratively train the initial encoding network and the initial decoding network simultaneously to obtain the first encoding network and the target decoding network.

[0230] For example, the VAE acquired by the electronic device may include an initial encoding network and an initial decoding network. In this case, the electronic device can acquire a second training set and train the initial encoding network and the initial decoding network based on the second training set. During the training process, the output data of the currently trained encoding network and the input data input to the currently trained encoding network are determined as the input data of the current training of the decoding network. That is, when the electronic device acquires the second training set but not the first training set, in order to train the first encoding network and the target decoding network, the electronic device can iteratively train the initial encoding network and the initial decoding network in different ways.

[0231] In some embodiments, after obtaining the target decoding network, the electronic device can also perform video generation inference operations. For example, the electronic device can freeze the first encoding network of the VAE to maintain consistency in the latent space vector distribution between the optimized VAE (including the VAE with the target decoding network) and the original VAE (including the VAE with the first decoding network). The input test image is processed by the fourth feature information extracted from the first decoding network, and this fourth feature information is then incorporated into the target decoding network using a cross-attention approach to obtain the third video. By incorporating the fourth feature information into the target decoding network using a cross-attention approach, more intuitive constraints are provided for the target decoding module of the image-generated video.

[0232] It should be noted that the video generation inference process performed by the electronic device after obtaining the target decoding network is similar to the above. Figure 6 The process of generating video via VAE shown is the same, and will not be described in detail in this embodiment.

[0233] In some embodiments, after obtaining the VAE, the electronic device can also evaluate the video generation function of the VAE using preset evaluation indicators.

[0234] It should be noted that the preset evaluation metrics can include both subjective and objective metrics. Subjective evaluation metrics include user surveys (where users rate the quality of the generated video); objective evaluation metrics can include Peak Signal-to-Noise Ratio (PANR), Structural Similarity Index (SSIM), and / or Learned Perceptual Image Patch Similarity (LPIPS), etc. Among these, PSNR is typically used to measure the quality of image compression or reconstruction; a higher value indicates better image quality. SSIM can be used to evaluate the similarity between two images, considering factors such as brightness, contrast, and structure; a higher value indicates better image quality. LPIPS can be used to measure the perceptual similarity between images; a lower value indicates better image quality.

[0235] In this embodiment, since the VAE is already capable of generating video from images, the first encoding network is frozen while the first decoding network continues to be iteratively trained, thereby reducing the workload of iterative training and improving its efficiency. Furthermore, the accuracy of training the decoding network is improved because multiple loss functions can be set.

[0236] Figure 12This is a schematic diagram of a video generation device provided in an embodiment of this application. The device can be implemented as part or all of an electronic device by software, hardware, or a combination of both. This electronic device can be... Figure 4 The electronic device shown. See also Figure 12 The device includes: a first determining module 1201, an encoding module 1202, and a decoding module 1203.

[0237] The first determining module 1201 is used to determine the target image in response to the video generation operation;

[0238] Encoding module 1202 is used to encode the target image through the first encoding network in the variational autoencoder to obtain the first video;

[0239] The decoding module 1203 is used to decode the first video based on the target image through the target decoding network in the variational autoencoder to obtain the second video. The target decoding network is a decoding network obtained by pre-training the first decoding network in the variational autoencoder. The first decoding network works with the first encoding network to realize the function of image-to-video generation.

[0240] As an example of this application, the decoding module 1203 is used for:

[0241] The target image is input into the target decoding network;

[0242] Based on the target image, the first video is decoded using a target decoding network to obtain the second video.

[0243] As an example of this application, the decoding module 1203 is used for:

[0244] The first feature information is obtained by extracting features from the target image through the first coding network;

[0245] The first feature information is input into the target decoding network;

[0246] Based on the first feature information, the first video is decoded using a target decoding network to obtain the second video.

[0247] As an example of this application, the decoding module 1203 is used for:

[0248] The first video is decoded step by step using a target decoding network;

[0249] During the decoding process, the first feature information and the second feature information obtained during the decoding process are fused together using a cross-attention method to obtain the second video.

[0250] As an example of this application, the device also includes:

[0251] The acquisition module is used to acquire a first training set, which includes multiple sets of training videos and multiple sample images. The multiple sets of training videos correspond one-to-one with the multiple sample images, and each set of training videos is generated by the first encoding network based on the corresponding sample images.

[0252] The training module is used to keep the first encoding network unchanged and iteratively train the first decoding network based on the first training set;

[0253] The second determination module is used to determine the target loss value between the decoding network obtained after each training and the decoding network before each training during the iterative training process.

[0254] The third determination module is used to determine the decoding network obtained at the time of convergence as the target decoding network when the target loss value converges.

[0255] As an example of this application, the second determining module is used for:

[0256] During iterative training, the output data of the decoding network after each training session is compared with the input data of the decoding network before each training session.

[0257] Based on the output data and input data, determine the first loss value of the first loss function, the second loss value of the second loss function, and the third loss value of the third loss function;

[0258] The target loss value is determined based on the first loss value, the second loss value, and the third loss value.

[0259] As an example of this application, the second determining module is used for:

[0260] The sum of the first loss value, the second loss value, and the third loss value is determined as the target loss value; or,

[0261] Multiply the first loss value by the first weight to obtain the first value; multiply the second loss value by the second weight to obtain the second value; multiply the third loss value by the third weight to obtain the third value; and determine the target loss value by summing the first, second, and third values.

[0262] In this embodiment of the application, since the electronic device adds a target image as a constraint during the decoding process of the first video, and the target image contains all the detailed features of the image, the target image affects the reconstruction process of video decoding to pixel space, thereby reducing the loss of image information caused by the decoding network and improving the display quality of the video.

[0263] It should be noted that the video generation device provided in the above embodiments is only illustrated by the division of the above functional modules when generating video. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0264] The functional units and modules in the above embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of the embodiments of this application.

[0265] The video generation apparatus and video generation method embodiments provided in the above embodiments belong to the same concept. The specific working process and technical effects of the units and modules in the above embodiments can be found in the method embodiment section, and will not be repeated here.

[0266] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line, DSL) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer, or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0267] The above-described embodiments are optional embodiments provided by this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the technical scope disclosed in this application should be included within the protection scope of this application.

Claims

1. A method for generating video, characterized in that, When applied to electronic devices, the method includes: In response to the video generation operation, the target image is determined; The target image is encoded using the first coding network in the variational autoencoder to obtain the first video. Based on the target image, the first video is decoded by the target decoding network in the variational autoencoder to obtain the second video. The target decoding network is a decoding network obtained by pre-training the first decoding network in the variational autoencoder. The first decoding network and the first encoding network work together to realize the function of image-to-video generation.

2. The method as described in claim 1, characterized in that, The step of decoding the first video based on the target image using the target decoding network in the variational autoencoder to obtain the second video includes: The target image is input into the target decoding network; Based on the target image, the first video is decoded using the target decoding network to obtain the second video.

3. The method as described in claim 2, characterized in that, The step of inputting the target image into the target decoding network includes: The target image is subjected to feature extraction through the first coding network to obtain first feature information; The first feature information is input into the target decoding network; The step of decoding the first video using the target decoding network based on the target image to obtain the second video includes: Based on the first feature information, the first video is decoded using the target decoding network to obtain the second video.

4. The method as described in claim 3, characterized in that, The step of decoding the first video using the target decoding network based on the first feature information to obtain the second video includes: The first video is decoded step by step using the target decoding network; During the decoding operation, the first feature information and the second feature information obtained during the decoding process are fused together using a cross-attention method to obtain the second video.

5. The method according to any one of claims 1-4, characterized in that, Before determining the target image in response to the video generation operation, the method further includes: Obtain a first training set, which includes multiple sets of training videos and multiple sample images. The multiple sets of training videos correspond one-to-one with the multiple sample images, and each set of training videos is generated by the first encoding network based on the corresponding sample images. Keeping the first encoding network unchanged, the first decoding network is iteratively trained based on the first training set; During iterative training, the target loss value between the decoding network obtained after each training iteration and the decoding network before each training iteration is determined. If the target loss value converges, the decoding network obtained at the convergence point is determined as the target decoding network.

6. The method as described in claim 5, characterized in that, During iterative training, determining the target loss value between the decoding network obtained after each training iteration and the decoding network before each training iteration includes: During iterative training, the output data of the decoding network after each training session is compared with the input data of the decoding network before each training session. Based on the output data and input data, determine the first loss value of the first loss function, the second loss value of the second loss function, and the third loss value of the third loss function; The target loss value is determined based on the first loss value, the second loss value, and the third loss value.

7. The method as described in claim 6, characterized in that, Determining the target loss value based on the first loss value, the second loss value, and the third loss value includes: The sum of the first loss value, the second loss value, and the third loss value is determined as the target loss value; or, Multiply the first loss value by the first weight to obtain a first value; multiply the second loss value by the second weight to obtain a second value; multiply the third loss value by the third weight to obtain a third value; and determine the target loss value by summing the first value, the second value, and the third value.

8. An electronic device, characterized in that, The device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, the electronic device performs the method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, Includes a computer program, which, when run, causes the method as described in any one of claims 1-7 to be performed.

Images