Special effect video generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a special effect video generation method and device, electronic equipment and a storage medium. A to-be-processed image is acquired; based on a to-be-processed image, a target image is generated, the target image is an image generated after a target special effect is applied to a target object in the to-be-processed image, and the target object in the target image has a target effect generated by the target special effect; a video generation model is called to process the to-be-processed image and the target image, a special effect video is generated, the special effect video comprises a video first frame, a video tail frame and at least one video middle frame located between the video first frame and the video tail frame, and the video first frame and the to-be-processed image have the same image content; the video tail frame and the target image have the same image content, and the at least one video middle frame is used for representing the continuous change process of the generated target effect. The content change of the special effect video is smoother and more real, and the video quality of the special effect video is improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of Internet technology, and in particular to a special effect video generation method, device, electronic device and storage medium. Background Art

[0002] Currently, in the scenario of video generation based on artificial intelligence (AI) technology, users can generate a corresponding video by inputting a static picture and a sentence describing the video content into the video generation model, thereby achieving the effect of making the picture "move".

[0003] However, in special content generation scenarios, such as video generation for hairstyle special effects, due to the particularity of such complex special effects, the solutions in the prior art cannot generate realistic and coherent videos with such complex special effects, that is, there is a problem of poor authenticity and coherence in the generated complex special effects videos. Summary of the invention

[0004] The embodiments of the present disclosure provide a special effect video generation method, device, electronic device and storage medium to overcome the problem of poor authenticity and continuity in generating videos with special effects.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for generating a special effects video, including:

[0006] Acquire an image to be processed; generate a target image based on the image to be processed, wherein the target image is an image generated after applying a target special effect to the target object in the image to be processed, and the target object in the target image has a target effect produced by the target special effect; call a video generation model to process the image to be processed and the target image to generate a special effect video, wherein the special effect video includes a first video frame, a last video frame, and at least one video middle frame located between the first video frame and the last video frame, the first video frame has the same image content as the image to be processed, the last video frame has the same image content as the target image, and the at least one video middle frame is used to characterize a continuous change process of generating the target effect.

[0007] In a second aspect, an embodiment of the present disclosure provides a special effects video generating device, including:

[0008] An acquisition module, used for acquiring an image to be processed;

[0009] A first generating module is used to generate a target image based on the image to be processed, wherein the target image is an image generated after applying a target special effect to the target object in the image to be processed, and the target object in the target image has a target effect generated by the target special effect;

[0010] The second generation module is used to call the video generation model to process the image to be processed and the target image to generate a special effects video, wherein the special effects video includes a first video frame, a last video frame, and at least one video middle frame located between the first video frame and the last video frame, the first video frame has the same image content as the image to be processed, the last video frame has the same image content as the target image, and the at least one video middle frame is used to characterize the continuous change process of generating the target effect.

[0011] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: a processor and a memory;

[0012] The memory stores computer-executable instructions;

[0013] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the special effects video generation method described in the first aspect and various possible designs of the first aspect.

[0014] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the special effects video generation method described in the first aspect and various possible designs of the first aspect is implemented.

[0015] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the special effects video generation method described in the first aspect and various possible designs of the first aspect.

[0016] The special effects video generation method, device, electronic device and storage medium provided in the present embodiment acquire an image to be processed; generate a target image based on the image to be processed, wherein the target image is an image generated after applying a target special effect to the target object in the image to be processed, and the target object in the target image has a target effect produced by the target special effect; call a video generation model to process the image to be processed and the target image to generate a special effects video, wherein the special effects video includes a first frame of a video, a last frame of a video, and at least one intermediate frame of a video located between the first frame of the video and the last frame of the video, the first frame of the video has the same image content as the image to be processed, the last frame of the video has the same image content as the target image, and the at least one intermediate frame of the video is used to characterize the continuous change process of generating the target effect. By first generating a target image with special effects based on the image to be processed, and then calling the video generation model to process the image to be processed and the target image, a video intermediate frame representing the continuous change process of the target effect from the image to be processed to the target image is obtained, and finally a special effects video consisting of the first frame of the video, the intermediate frame of the video and the last frame of the video is generated. While the authenticity of the special effects is ensured by the last frame of the video, the content changes of the special effects video are made smoother and more realistic, thereby improving the video quality of the special effects video. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0018] Figure 1 An application scenario diagram of the special effects video generation method provided in the embodiment of the present disclosure;

[0019] Figure 2 A schematic diagram of the process of generating a special effects video provided by an embodiment of the present disclosure Figure 1 ;

[0020] Figure 3 A schematic diagram of a target image provided by an embodiment of the present disclosure;

[0021] Figure 4 for Figure 2 A flowchart of a specific implementation method of step S102 in the illustrated embodiment;

[0022] Figure 5 for Figure 4 A flowchart of a specific implementation method of step S1022 in the illustrated embodiment;

[0023] Figure 6 for Figure 2 A flowchart of a specific implementation method of step S103 in the illustrated embodiment;

[0024] Figure 7 A schematic diagram of the process of generating a special effects video provided by an embodiment of the present disclosure Figure 2 ;

[0025] Figure 8 for Figure 7 A flowchart of a specific implementation method of step S205 in the illustrated embodiment;

[0026] Fig. 9 for Figure 8 A flowchart of a specific implementation method of step S2053 in the illustrated embodiment;

[0027] Fig.10 A schematic diagram of a process for generating a special effects video provided by an embodiment of the present disclosure;

[0028] Fig.11 A structural block diagram of a special effects video generating device provided in an embodiment of the present disclosure;

[0029] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure;

[0030] Fig.13 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0032] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0033] The application scenarios of the embodiments of the present disclosure are explained below:

[0034] Figure 1A diagram of an application scenario of the special effects video generation method provided in the embodiment of the present disclosure. The special effects video generation method provided in the embodiment of the present disclosure can be applied to an application (APP, Application) with a special effects video generation function, such as a video generation application. More specifically, it can be applied to the application scenario of special effects video generation. The execution subject of this embodiment can be a terminal device that runs the above-mentioned application with a special effects video generation function, or a server that deploys the server corresponding to the above-mentioned application, or other electronic devices that perform similar functions. Among them, when the execution subject is a terminal device, the terminal device executes the method provided in this embodiment by running the above-mentioned application; when the execution subject is a server, the server of the above-mentioned application with a special effects video generation function can be partially or completely run on the server, and the method provided in this embodiment is executed on the server side, and the terminal device runs the client of the application, and the communication between the server and the terminal device is based on the server-client, so that the terminal device can obtain the execution result of the method provided in this embodiment and display it as needed.

[0035] Among them, in some embodiments, the terminal device or server can implement the special effects video generation method provided by the embodiment of the present disclosure by running various computer executable instructions or computer programs. For example, computer executable instructions can be program-level commands, machine instructions or software instructions. The computer program can be a native program or software module in the operating system; it can be a local application, that is, a program that needs to be installed in the operating system to run, or it can be a small program embedded in any APP, that is, a program that runs based on a browser environment. In summary, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form, and the specific implementation form can be configured as needed. Furthermore, in the process of implementing the special effects video generation method provided by the embodiment of the present disclosure, the terminal device can execute the method by running a computer executable instruction or computer program set locally, or it can execute the method by calling a computer executable instruction or computer program set in an external server. In some embodiments, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud storage, cloud communications, cloud databases, cloud computing, cloud functions, network services, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms. Among them, cloud services can be interactive processing services for terminal devices to call.

[0036] refer to Figure 1As shown in , taking a terminal device as an example, a target application with a special effects video generation function is running in the terminal device. The static image input by the user (such as a selfie uploaded by the user) is loaded through the target application. After that, in response to the user's trigger operation (such as clicking the "Generate" button), the cloud or locally deployed video generation model is called to convert the static image into a corresponding dynamic video. In this process, the target application can further receive a description statement input by the user (shown as ABCDEF in the figure), and control the video content of the generated video according to the description statement, so as to generate a dynamic video that meets the user's preferences and needs; or, the user controls the content of the corresponding special effects video generated based on the static image by clicking the special effects card (special effects control) provided by the target application.

[0037] In the prior art, in the scenario of special video content generation, such as special effects video generation scenarios for hairstyle special effects, clothing special effects, special effects of animals turning into humans, etc., due to the particularity of such complex special effects, it is necessary to accurately show the appearance characteristics of the complex special effects, as well as their change process. However, the solutions in the prior art only use static images as the starting point to generate subsequent video frames, which often causes the change process of the generated video frames to be uncontrolled, resulting in poor continuity between the video frames, and a large difference between the last frame of the video and the expected effect, ultimately causing the generated special effects video to have problems of incoherence and poor authenticity.

[0038] The disclosed embodiment provides a special effects video generation method to solve the above-mentioned problem.

[0039] refer to Figure 2 , Figure 2 A schematic diagram of the process of generating a special effects video provided by an embodiment of the present disclosure Figure 1 The method of this embodiment can be applied in a terminal device or a server. The special effect video generation method includes:

[0040] Step S101: Acquire an image to be processed.

[0041] Step S102: Generate a target image based on the image to be processed, wherein the target image is an image generated after applying a target special effect to a target object in the image to be processed, and the target object in the target image has a target effect generated by the target special effect.

[0042] refer to Figure 1The application scenario schematic diagram shown in the figure, in this embodiment, the terminal device is used as the execution subject to introduce the special effect video generation method provided. For example, first, after the terminal device runs the target application, it receives the image uploaded by the user through the target application, that is, the image to be processed, which is the starting point for generating the final special effect video. Specifically, the image to be processed is, for example, a life photo uploaded by the user, etc., containing a specific object object, such as a person, an animal, etc. as the main body of the image content. Afterwards, the terminal device processes the image to be processed through the capabilities provided by the target application to generate a target image, wherein the target image is the image after adding the target special effect to the image to be processed, that is, the target special effect image corresponding to the image to be processed. The target special effect is a special effect, that is, a special effect used to add a target effect of a target category to a target object, wherein an object is an object that changes due to the addition of a special effect, such as the "hair" that changes in the hairstyle special effect, the "person" in the dress-up special effect, the "animal" in the special effect of an animal turning into a person, and so on. Figure 3 A schematic diagram of a target image provided by an embodiment of the present disclosure, such as Figure 3 As shown, exemplarily, the image to be processed is a picture containing a person object A (target object). As shown in the reference figure, the hairstyle of the person object A is, for example, the first hairstyle (short hair). The terminal device generates a target image by invoking the corresponding algorithm and model to process the image to be processed. The target image still contains the person object A, but the hairstyle of the person object A is changed to the second hairstyle (long hair). The second hairstyle is the target effect of the target category generated by the target special effect. In other possible implementations, as the type of video special effects changes, the generated target image also changes accordingly. For example, in the scene of "clothing change special effects" (not shown in the figure), the target object is the person object B in the image to be processed. The person object B has a first dress. The terminal device generates a target image by invoking the corresponding algorithm and model to process the image to be processed. The target image still contains the person object B (target object), but the dress of the person object B is changed to a second dress of another style. The second dress is the target effect after the target special effect is applied to the target object in the image to be processed.

[0043] Furthermore, in other possible scenarios, the target special effects may include other video special effects such as "animal transformation into human special effects", "human transformation into animal special effects", "object material change special effects", etc. Depending on the target special effects, the generated target image also changes accordingly, and the content of the special effects video generated based on the target image also changes accordingly. That is, the solution provided in this embodiment can be applied to the generation of special effects videos in different scenarios, and can be specifically set as needed, and is not limited this time.

[0044] Furthermore, in a possible implementation, Figure 4 As shown, the specific implementation of step S102 includes:

[0045] Step S1021: obtaining a first prompt word for a target effect of a target category, where the first prompt word is used to describe an appearance feature of the target effect of the target category;

[0046] Step S1022: Processing the image to be processed and the first prompt word through the image generation model to generate a target image.

[0047] Exemplarily, in the process of generating the target image, the terminal device will call the image generation model to complete the steps of the target image. Specifically, the image generation model is a pre-trained neural network model used to regenerate the input image according to the prompt word. The terminal device first receives the first prompt word for input through the running target application. The first prompt word is used to describe the appearance characteristics of the target effect of the target category. For example, the content of the first prompt word includes: "long curly hair", "full and dense natural small curls", etc. After that, the image to be processed and the first prompt word are input into the image generation model. Using the ability of the image generation model, the image content in the image to be processed is "modified" to an image containing the target effect of the target category described by the first prompt word, that is, the target image. The above process and the process of realizing the target special effect. Among them, the first prompt word can be preset configuration information, which is dynamically determined according to the image content of the image to be processed, or it can be information input by the user, which can be set as needed.

[0048] In this embodiment, the first prompt word is used to limit the appearance characteristics of the target effect of the target category, thereby achieving precise control of the visual effect of the target special effect, making the special effect more diversified, differentiated and refined, and ultimately improving the authenticity and visual performance of the generated special effect video.

[0049] Further, in a possible implementation, the image generation model includes a first functional unit and a second functional unit, the first functional unit is used to generate a target effect of a target category, and the second functional unit is used to generate a second object matching the target effect based on the target object, and the appearance similarity between the second object and the target object is greater than a similarity threshold; Figure 5 As shown, the specific implementation of step S1022 includes:

[0050] Step S1022-1: Process the image to be processed and the first prompt word through the first functional unit to generate effect feature data representing the target effect.

[0051] Step S1022-2: Process the image to be processed and the effect feature data through the second functional unit to generate object feature data representing the second object.

[0052] Step S1022 - 3 : Generate a target image based on the effect feature data and the object feature data.

[0053] Exemplarily, in one possible implementation, the image generation model is composed of a first functional unit and a second functional unit, wherein the first functional unit is used to generate a target effect of a target category, for example, the first functional unit has the ability to edit hairstyles, for example, to generate feature data corresponding to hairstyles such as short hair, curly hair, and long straight blond hair, and the feature data can be a set of pixels, or other feature vectors or matrices that require further decoding to represent an image; the image to be processed and the first prompt word are processed by the first functional unit, and corresponding feature data is generated based on the appearance features described by the first prompt word, and the feature data can realize an image or feature description of the target effect of the target category. The second functional unit is used to generate object feature data representing the second object, that is, the second functional unit has the ability to edit object, for example, to generate feature data corresponding to the human face, animal face, and physical contour in the image to be processed, that is, to edit the target object in the image to be processed into the second object (that is, for example, to achieve a "face-changing" effect), so that the second object in the generated target image is more matched with the target effect generated by the first functional unit, thereby improving the authenticity of the image. On the other hand, a high degree of similarity is maintained between the target object and the second object, so that the appearance features of the target object in the image to be processed can be mostly retained, thereby improving the consistency between the main object (second object) in the target image after adding the target special effects and the main object (target object) in the image to be processed without adding the target special effects.

[0054] Further, in a possible implementation, the first prompt word includes a first participle, a second participle and a third participle, wherein the first participle is used to indicate the target object; the second participle is used to characterize the shape characteristics of the target effect; and the third participle is used to characterize the style characteristics of the target effect. That is, the first prompt word consists of three parts, namely the first participle, the second participle and the third participle, wherein the first participle is used to control where the image generation model generates the target effect (target object); the second participle is used to control the local appearance morphology (shape characteristics) of the target effect of the image generation model, such as hair shape, color, etc.; and the third participle is used to characterize the overall characteristics (style characteristics) of the target effect, such as hair fluffiness, curvature of hair curl, density, etc. Through the above-mentioned first participle, the second participle and the third participle, a more detailed control of the target effect can be achieved, so that the generated target image and the special effects video generated based on the target image can better meet the needs of users and have better visual effects.

[0055] Step S103: calling the video generation model to process the image to be processed and the target image to generate a special effects video, wherein the special effects video includes a first frame of the video, a last frame of the video, and at least one intermediate frame of the video between the first frame of the video and the last frame of the video, the first frame of the video has the same image content as the image to be processed, the last frame of the video has the same image content as the target image, and the at least one intermediate frame of the video is used to characterize the continuous change process of generating the target effect.

[0056] Exemplarily, after generating the target image, the image content of the image to be processed is used as the first frame content of the special effects video, and the image content of the target image is used as the last frame content of the special effects video, and the video generation model is called to generate the continuously changing video intermediate frame content between the first frame content and the last frame content, thereby obtaining a video for describing the continuous change from the first frame content to the last frame content of the target effect, i.e., the special effects video, wherein the video generation model has the ability to generate a continuously changing image between two static images based on two static images, and this ability is obtained by pre-training based on training samples, which will not be elaborated here.

[0057] Among them, in a possible implementation, such as Figure 6 As shown, the specific implementation of step S103 includes:

[0058] Step S1031: obtaining a second prompt word, where the second prompt word is used to describe the characteristics of the change process of the target effect in the special effect video;

[0059] Step S1032: calling the video generation model, processing the image to be processed and the target image based on the second prompt word, and generating a special effects video with characteristics of the change process.

[0060] Exemplarily, before calling the video generation model, the terminal device first obtains the second prompt word. Similar to the first prompt word, the second prompt word can be preset configuration information or information input by the user, and can be set as needed. Among them, the second prompt word is used to describe the characteristics of the change process of the target effect in the special effects video. For example, the content of the second prompt word includes: "The hair gradually becomes curly and fluffy, with a natural and progressive smooth dynamic." Through the above, the second prompt word can realize the control of the change process of the special effects video, so that the change process from the image content of the image to be processed to the image content of the target image is more accurate, meeting the personalized needs of users, and making the generated special effects video more realistic. Among them, it should be noted that the content of the second prompt word in this embodiment is only exemplary, and the specific content of the second prompt word can be set as needed, which will not be repeated here.

[0061] In this embodiment, by obtaining an image to be processed; based on the image to be processed, generating a target image, wherein the target image is an image after adding a target special effect to the image to be processed, the image content of the image to be processed includes a target object, and the target special effect is used to add a target effect of a target category to the target object; calling a video generation model to process the image to be processed and the target image, generating a special effect video, wherein the special effect video includes a video first frame, a video last frame, and at least one video middle frame located between the video first frame and the video last frame, the video first frame has the same image content as the image to be processed, the video last frame has the same image content as the target image, and at least one video middle frame is used to characterize the continuous change process of generating the target effect. By first generating a target image with special effects based on the image to be processed, then calling the video generation model to process the image to be processed and the target image, obtaining a video middle frame characterizing the continuous change process of the target effect from the image to be processed to the target image, and finally generating a special effect video composed of the video first frame, the video middle frame and the video last frame, while ensuring the authenticity of the special effect through the video last frame, making the content change of the special effect video more smooth and real, thereby improving the video quality of the special effect video.

[0062] refer to Figure 7 , Figure 7 A schematic diagram of the process of generating a special effects video provided by an embodiment of the present disclosure Figure 2 In this embodiment Figure 2 On the basis of the illustrated embodiment, step S103 is further refined. In this embodiment, the video generation model includes a variational autoencoder unit and a diffusion transformer unit, and the special effect video generation method includes:

[0063] Step S200, jointly training the variational autoencoder unit and the diffusion transformer unit.

[0064] Step S201: Acquire an image to be processed.

[0065] Step S202: Generate a target image based on the image to be processed, wherein the target image is an image after adding a target special effect to the image to be processed, the image content of the image to be processed includes a target object, and the target special effect is used to add a target effect of a target category to the target object.

[0066] Step S203: Processing the image to be processed and the target image through the variational autoencoder unit to generate a first feature code corresponding to the image to be processed and a second feature code corresponding to the target image, respectively.

[0067] Step S204: sequentially concatenate the first feature code, the blank frame code, and the second feature code to obtain an input feature code.

[0068] Step S205: Process the input feature code through the diffusion transformer unit to generate a special effect video.

[0069] Exemplarily, after generating the target image, the image to be processed and the target image are processed by calling the video generation model to generate a special effect video, wherein the video generation model includes a variational auto-encoder (VAE) unit and a diffusion transformer (Diffusion Transformer) unit, wherein the variational auto-encoder is a generation model that combines the ideas of the auto-encoder and the probability graph model, and is mainly used to learn the potential distribution of data and generate new data samples. The diffusion transformer is a deep learning model based on the diffusion model (Diffusion Models), which is mainly used for the generation and transformation of images and videos. In this embodiment, the variational auto-encoder unit implemented based on the variational auto-encoder is used to encode the image to be processed and the target image, respectively forming the corresponding first feature code and the second feature code corresponding to the target image; and in the subsequent steps, it is used to decode the feature code output by the diffusion transformer and restore it to the corresponding multi-frame image, thereby obtaining the final video. In this embodiment, the diffusion transformer unit implemented based on the diffusion transformer is used to predict according to the encoded feature coding to generate coding features corresponding to multiple frames of continuously changing images. In short, it is to generate a continuously changing picture between the image picture corresponding to the image to be processed and the image picture corresponding to the target image, that is, the process of continuous change of the target effect.

[0070] Specifically, in this embodiment, the terminal device first processes the image to be processed and the target image through the variational self-encoder unit, and generates a first feature code corresponding to the image to be processed and a second feature code corresponding to the target image, that is, encodes the image to be processed and the target image; then, generates a blank code with the same size as the first feature code and the second feature code, and sequentially concatenates the first feature code, the blank frame code and the second feature code to obtain the input feature code, and finally, inputs the input feature code into the diffusion transformer unit, and the diffusion transformer unit performs coding prediction to generate coding features corresponding to one or more frames of images that continuously change between the image screen of the image to be processed and the image screen of the target image. The step performed by the diffusion transformer is equivalent to the process of generating a blank code in the input feature code through the first feature code and the second feature code in the input feature code, and finally, the feature code output by the diffusion transformer is decoded into a corresponding video frame through the variational self-encoder, thereby generating a special effect video.

[0071] Furthermore, the variational autoencoder unit includes an encoder subunit and a decoder subunit, and the specific implementation of step S203 includes:

[0072] Step S2031: the encoding subunit encodes the image to be processed and the target image to generate a first feature code corresponding to the image to be processed and a second feature code corresponding to the target image, respectively.

[0073] Correspondingly, such as Figure 8 As shown, the implementation of step S205 includes:

[0074] Step S2051: Obtain a randomly initialized random feature code;

[0075] Step S2052: inputting the input feature code and the random feature code into the diffusion transformer unit, and using the diffusion transformer unit to perform multiple rounds of updating on the random feature code based on the input feature code until a preset stop condition is reached, thereby obtaining an output feature code;

[0076] Step S2053: Decode the output feature code through the decoding subunit to generate a special effects video.

[0077] Exemplarily, first, the terminal device obtains a feature coding array, vector or matrix by random initialization, and the random feature coding corresponds to N video frames, that is, the random feature coding will be used later to generate N frames of special effects video. Afterwards, the input feature coding and random feature coding generated in the previous step are used as input, and the diffusion transformer unit is called, and the random feature coding is updated for multiple rounds based on the input feature coding by the diffusion transformer unit until a preset stop condition is reached, such as the update round reaches a preset value, or the residual is less than a preset value, etc. The output feature coding output by the diffusion transformer unit is obtained, and the output feature coding is the result of multiple rounds of iterative updates of the random feature coding. In the above process, a pre-trained diffusion transformer unit is used to perform inference with the first feature code and the second feature code in the input feature code as effective information, and an output feature code including an intermediate frame code that "connects" the first feature code and the second feature code is obtained. The intermediate frame code can show the feature change from the first feature code to the second feature code. The image generated after decoding the temporal frame code can show the image change from the image to be processed to the target image, and conforms to the real law of hairstyle (hair) growth; therefore, the special effects video generated by the output feature code can coherently and realistically display the appearance process of the target special effects in the video, and has stronger visual expression.

[0078] Furthermore, in a possible implementation, the output feature code includes a first feature code, a first sequential frame code, and a second sequential frame code, wherein the first sequential frame code is generated based on the first feature code and is used to characterize the image content of N continuously changing video frames after the first frame of the special effect video; the second sequential frame code is generated based on the first sequential frame code and the second feature code and is used to characterize the image content of M continuously changing video frames at the end of the special effect video, wherein N and M are integers greater than 0. Fig. 9 As shown, the specific implementation of step S2053 includes:

[0079] Step S2053-1: decoding the first feature code by a decoding subunit to generate a first frame of the video;

[0080] Step S2053-2: decoding the first time sequence frame code by a decoding subunit to generate N video intermediate frames;

[0081] Step S2053-3: Decoding the second time sequence frame code by the decoding subunit to generate M video end frames;

[0082] Step S2053-4: Splice the first frame of the video, N middle frames of the video and M last frames of the video to generate a special effects video.

[0083] For example, Fig.10 A schematic diagram of a process for generating a special effects video provided by an embodiment of the present disclosure is shown below. Fig.10The above process is introduced in detail, as shown in the reference figure, first, the input feature code is generated through the encoding subunit, and then the input feature code and the random feature code are input into the diffusion transformer unit to obtain the output feature code output by the diffusion transformer unit, and the output feature code includes feature code F1 (first feature code), feature code F2 (first sequential frame code) and feature code F3 (second sequential frame code). Afterwards, the first feature code is directly decoded by the decoding subunit to generate the first frame of the video, that is, the first frame of the special effect video, which is shown as P1 in the figure. In this step, the restoration of the image to be processed is achieved. Afterwards, the feature code F2 is decoded by the decoding subunit to generate N video intermediate frames, where N is 4, for example, as shown in the figure, the 4 video intermediate frames are P2, P3, P4 and P5, that is, P2, P3, P4 and P5 are generated by the feature code F2, a total of 4 video intermediate values ​​located after the first frame of the video, that is, the 2nd to 5th frames of the special effect video; then, the feature code F3 is decoded by the decoding subunit to generate M video end frames, where M is 4, for example, as shown in the figure, the 4 video intermediate frames are P6, P7, P8 and P9, a total of 4 video frames located at the end of the video, that is, the 6th to 9th frames of the special effect video. Finally, the above 9 video frames are combined to remember the special effect video.

[0084] In a possible implementation, in this embodiment, prompt words, such as the first prompt word and / or the second prompt word, can be further combined to control the generation process of the target image and the special effect video. For example, the second prompt word is applied to the variational autoencoder unit and the diffusion transformer unit to control the generation process of the special effect video. Figure 2 The illustrated embodiment has been described in detail and will not be described again here.

[0085] In this embodiment, firstly, the image features of N video middle frames and M video end frames are compressed into a feature code (first temporal frame code and second temporal frame code) respectively through the trained diffusion transformer unit based on the first feature code, and then the first feature code corresponding to the first frame of the video, the first temporal frame code corresponding to the N video middle frames and the second temporal frame code corresponding to the M video end frames are restored to the first frame of the video, the N video middle frames and the M video end frames in combination with the variational autoencoder unit trained jointly with the diffusion transformer unit, so as to realize the generation of special effect video, so that the beginning and the end of the special effect video can accurately match the video to be edited that is initially input and the target video after adding special effects, that is, the content process control of the special effect video is realized, and the problems such as the change process of each video frame of the special effect video is not controlled, the video content is distorted, and the end frame of the video is greatly different from the expected effect (target special effect) are avoided, thereby improving the coherence and authenticity of the generated special effect video and providing the viewing experience of the special effect video.

[0086] In this embodiment, the implementation method of step 201 to step S202 is the same as that of the present disclosure. Figure 2 The implementation methods of step S101 and step S102 in the illustrated embodiment are the same and will not be described in detail here.

[0087] Corresponding to the special effects video generation method of the above embodiment, Fig.11 A structural block diagram of a special effects video generation device provided in an embodiment of the present disclosure. The method described in the above embodiment can be executed by the special effects video generation device, which can be implemented by software and / or hardware, and can be integrated in an electronic device with certain data processing functions. The electronic device may include but is not limited to a mobile terminal with big data processing capabilities, and a fixed terminal with big data processing capabilities such as a desktop computer and a supercomputer.

[0088] For the sake of convenience, only the parts related to the embodiments of the present disclosure are shown. Fig.11 , the special effect video generating device 3 comprises:

[0089] An acquisition module 31 is used to acquire an image to be processed;

[0090] A first generating module 32 is used to generate a target image based on the image to be processed, wherein the target image is an image generated after applying a target special effect to the target object in the image to be processed, and the target object in the target image has a target effect generated by the target special effect;

[0091] The second generation module 33 is used to call the video generation model to process the image to be processed and the target image to generate a special effect video, wherein the special effect video includes a first frame of the video, a last frame of the video, and at least one video middle frame located between the first frame of the video and the last frame of the video, the first frame of the video has the same image content as the image to be processed, the last frame of the video has the same image content as the target image, and the at least one video middle frame is used to characterize the continuous change process of generating the target effect.

[0092] According to one or more embodiments of the present disclosure, the first generation module 32 is specifically used to: obtain a first prompt word for a target effect of a target category, the first prompt word being used to describe the appearance characteristics of the target effect of the target category; and process the image to be processed and the first prompt word through an image generation model to generate a target image.

[0093] According to one or more embodiments of the present disclosure, the image generation model includes a first functional unit and a second functional unit, the first functional unit is used to generate a target effect of a target category, and the second functional unit is used to generate a second object matching the target effect based on the target object, and the appearance similarity between the second object and the target object is greater than a similarity threshold; when the first generation module 32 processes the image to be processed and the first prompt word through the image generation model to generate the target image, it is specifically used to: process the image to be processed and the first prompt word through the first functional unit to generate effect feature data representing the target effect; process the image to be processed and the effect feature data through the second functional unit to generate object feature data representing the second object; and generate the target image based on the effect feature data and the object feature data.

[0094] According to one or more embodiments of the present disclosure, the first prompt word includes a first participle, a second participle and a third participle, wherein the first participle is used to indicate a target object; the second participle is used to characterize a shape feature of a target effect; and the third participle is used to characterize a style feature of the target effect.

[0095] According to one or more embodiments of the present disclosure, the video generation model includes a variational autoencoder unit and a diffusion transformer unit, and the second generation module 33 is specifically used to: process the image to be processed and the target image through the variational autoencoder unit to generate a first feature code corresponding to the image to be processed and a second feature code corresponding to the target image, respectively; sequentially concatenate the first feature code, the blank frame code, and the second feature code to obtain the input feature code; process the input feature code through the diffusion transformer unit to generate a special effects video.

[0096] According to one or more embodiments of the present disclosure, the variational autoencoder unit includes an encoding subunit and a decoding subunit. When the second generation module 33 processes the image to be processed and the target image through the variational autoencoder unit to generate a first feature code corresponding to the image to be processed and a second feature code corresponding to the target image, respectively, the second generation module 33 is specifically used to: encode the image to be processed and the target image through the encoding subunit to generate a first feature code corresponding to the image to be processed and a second feature code corresponding to the target image, respectively; when the second generation module 33 processes the input feature code through the diffusion transformer unit to generate a special effects video, the second generation module 33 is specifically used to: obtain a randomly initialized random feature code; input the input feature code and the random feature code into the diffusion transformer unit, and use the diffusion transformer unit to update the random feature code for multiple rounds based on the input feature code until a preset stop condition is reached to obtain an output feature code; decode the output feature code through the decoding subunit to generate a special effects video.

[0097] According to one or more embodiments of the present disclosure, the output feature code includes a first feature code, a first sequential frame code, and a second sequential frame code, wherein the first sequential frame code is generated based on the first feature code and is used to characterize the image content of N continuously changing video frames after the first frame of the special effects video; the second sequential frame code is generated based on the first sequential frame code and the second feature code and is used to characterize the image content of M continuously changing video frames at the end of the special effects video.

[0098] According to one or more embodiments of the present disclosure, when the second generation module 33 decodes the output feature code through the decoding subunit to generate a special effect video, it is specifically used to: decode the first feature code through the decoding subunit to generate a first frame of the video; decode the first sequential frame code through the decoding subunit to generate N video middle frames; decode the second sequential frame code through the decoding subunit to generate M video end frames; splice the video first frame, N video middle frames and M video end frames to generate a special effect video.

[0099] According to one or more embodiments of the present disclosure, when the second generation module 33 calls the video generation model to process the image to be processed and the target image to generate a special effects video, it is specifically used to: obtain a second prompt word, and the second prompt word is used to describe the changing process characteristics of the target effect in the special effects video; call the video generation model, process the image to be processed and the target image based on the second prompt word, and generate a special effects video with changing process characteristics.

[0100] The acquisition module 31, the first generation module 32 and the second generation module 33 are connected in sequence. The special effect video generation device 3 provided in this embodiment can implement the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated in this embodiment.

[0101] Fig.12 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure is shown in FIG. Fig.12 As shown, the electronic device 4 includes:

[0102] A processor 41, and a memory 42 communicatively connected to the processor 41;

[0103] The memory 42 stores computer executable instructions;

[0104] The processor 41 executes the computer execution instructions stored in the memory 42 to implement the following Figure 2-Figure 10 The special effects video generation method in the illustrated embodiment.

[0105] Optionally, the processor 41 and the memory 42 are connected via a bus 43 .

[0106] For related instructions, please refer to Figure 2-Figure 10 The relevant descriptions and effects corresponding to the steps in the corresponding embodiments can be understood, and no further elaboration is made here.

[0107] The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, which are used to implement the present invention when executed by a processor. Figure 2-Figure 10 A special effects video generation method provided by any one of the corresponding embodiments.

[0108] The present invention provides a computer program product, including a computer program. When the computer program is executed by a processor, the present invention is realized. Figure 2-Figure 10 A special effects video generation method provided by any one of the corresponding embodiments.

[0109] In order to implement the above embodiment, the embodiment of the present disclosure also provides an electronic device.

[0110] refer to Fig.13 , which shows a schematic diagram of the structure of an electronic device 900 suitable for implementing the embodiment of the present disclosure, and the electronic device 900 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers, portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.13 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0111] like Fig.13 As shown, the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0112] Typically, the following devices may be connected to the I / O interface 905: input devices 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 908 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 909. The communication device 909 may allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.13 The electronic device 900 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0113] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0114] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0115] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0116] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0117] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0118] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0119] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit or module does not, in some cases, constitute a limitation on the unit itself.

[0120] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0121] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0122] In a first aspect, according to one or more embodiments of the present disclosure, a method for generating a special effect video is provided, comprising:

[0123] Acquire an image to be processed; generate a target image based on the image to be processed, wherein the target image is an image after adding a target special effect to the image to be processed, the image content of the image to be processed includes a target object, and the target special effect is used to add a target effect of a target category to the target object; call a video generation model to process the image to be processed and the target image to generate a special effect video, wherein the special effect video includes a first frame of the video, a last frame of the video, and at least one intermediate frame of the video between the first frame of the video and the last frame of the video, the first frame of the video has the same image content as the image to be processed, the last frame of the video has the same image content as the target image, and the at least one intermediate frame of the video is used to characterize the continuous change process of generating the target effect.

[0124] According to one or more embodiments of the present disclosure, generating a target image based on the image to be processed includes: obtaining a first prompt word for a target effect of the target category, the first prompt word being used to describe appearance features of the target effect of the target category; and processing the image to be processed and the first prompt word through an image generation model to generate the target image.

[0125] According to one or more embodiments of the present disclosure, the image generation model includes a first functional unit and a second functional unit, the first functional unit is used to generate a target effect of the target category, and the second functional unit is used to generate a second object matching the target effect based on the target object, and the appearance similarity between the second object and the target object is greater than a similarity threshold; the processing of the image to be processed and the first prompt word by the image generation model to generate the target image includes: processing the image to be processed and the first prompt word by the first functional unit to generate effect feature data representing the target effect; processing the image to be processed and the effect feature data by the second functional unit to generate object feature data representing the second object object; generating the target image based on the effect feature data and the object feature data.

[0126] According to one or more embodiments of the present disclosure, the first prompt word includes a first participle, a second participle and a third participle, wherein the first participle is used to indicate the target object; the second participle is used to characterize the shape feature of the target effect; and the third participle is used to characterize the style feature of the target effect.

[0127] According to one or more embodiments of the present disclosure, the video generation model includes a variational autoencoder unit and a diffusion transformer unit, and the calling of the video generation model to process the image to be processed and the target image to generate a special effects video includes: processing the image to be processed and the target image by a variational autoencoder unit to respectively generate a first feature code corresponding to the image to be processed and a second feature code corresponding to the target image; sequentially splicing the first feature code, the blank frame code, and the second feature code to obtain an input feature code; processing the input feature code by a diffusion transformer unit to generate the generated special effects video.

[0128] According to one or more embodiments of the present disclosure, the variational autoencoder unit includes an encoding subunit and a decoding subunit, and the variational autoencoder unit is used to process the image to be processed and the target image to generate a first feature code corresponding to the image to be processed and a second feature code corresponding to the target image, respectively, including: encoding the image to be processed and the target image by the encoding subunit to generate a first feature code corresponding to the image to be processed and a second feature code corresponding to the target image, respectively; the diffusion transformer unit is used to process the input feature code to generate the special effect video, including: obtaining a randomly initialized random feature code; inputting the input feature code and the random feature code into the diffusion transformer unit, and using the diffusion transformer unit to perform multiple rounds of updating the random feature code based on the input feature code until a preset stop condition is reached to obtain an output feature code; the decoding subunit is used to decode the output feature code to generate the special effect video.

[0129] According to one or more embodiments of the present disclosure, the output feature code includes a first feature code, a first sequential frame code and a second sequential frame code, wherein the first sequential frame code is generated based on the first feature code and is used to characterize the image content of N continuously changing video frames after the first frame of the special effects video; the second sequential frame code is generated based on the first sequential frame code and the second feature code and is used to characterize the image content of M continuously changing video frames at the end of the special effects video.

[0130] According to one or more embodiments of the present disclosure, the decoding subunit decodes the output feature code to generate the special effects video, including: decoding the first feature code by the decoding subunit to generate a first frame of the video; decoding the first sequential frame code by the decoding subunit to generate N video middle frames; decoding the second sequential frame code by the decoding subunit to generate M video end frames; and splicing the video first frame, N video middle frames and M video end frames to generate the special effects video.

[0131] According to one or more embodiments of the present disclosure, calling the video generation model to process the image to be processed and the target image to generate a special effects video includes: obtaining a second prompt word, where the second prompt word is used to describe the changing process characteristics of the target effect in the special effects video; calling the video generation model to process the image to be processed and the target image based on the second prompt word to generate a special effects video with the changing process characteristics.

[0132] In a second aspect, according to one or more embodiments of the present disclosure, a special effect video generating device is provided, including:

[0133] An acquisition module, used for acquiring an image to be processed;

[0134] A first generating module is configured to generate a target image based on the image to be processed, wherein the target image is an image after a target special effect is added to the image to be processed, the image content of the image to be processed includes a target object, and the target special effect is used to add a target effect of a target category to the target object;

[0135] The second generation module is used to call the video generation model to process the image to be processed and the target image to generate a special effects video, wherein the special effects video includes a first video frame, a last video frame, and at least one video middle frame located between the first video frame and the last video frame, the first video frame has the same image content as the image to be processed, the last video frame has the same image content as the target image, and the at least one video middle frame is used to characterize the continuous change process of generating the target effect.

[0136] According to one or more embodiments of the present disclosure, the first generation module is specifically used to: obtain a first prompt word for a target effect for the target category, the first prompt word being used to describe appearance features of the target effect for the target category; and process the image to be processed and the first prompt word through an image generation model to generate the target image.

[0137] According to one or more embodiments of the present disclosure, the image generation model includes a first functional unit and a second functional unit, the first functional unit is used to generate a target effect of the target category, and the second functional unit is used to generate a second object matching the target effect based on the target object, and the appearance similarity between the second object and the target object is greater than a similarity threshold; when the first generation module processes the image to be processed and the first prompt word through the image generation model to generate the target image, it is specifically used to: process the image to be processed and the first prompt word through the first functional unit to generate effect feature data representing the target effect; process the image to be processed and the effect feature data through the second functional unit to generate object feature data representing the second object object; and generate the target image based on the effect feature data and the object feature data.

[0138] According to one or more embodiments of the present disclosure, the first prompt word includes a first participle, a second participle and a third participle, wherein the first participle is used to indicate the target object; the second participle is used to characterize the shape feature of the target effect; and the third participle is used to characterize the style feature of the target effect.

[0139] According to one or more embodiments of the present disclosure, the video generation model includes a variational autoencoder unit and a diffusion transformer unit, and the second generation module is specifically used to: process the image to be processed and the target image through the variational autoencoder unit to respectively generate a first feature code corresponding to the image to be processed and a second feature code corresponding to the target image; sequentially splice the first feature code, the blank frame code and the second feature code to obtain the input feature code; process the input feature code through the diffusion transformer unit to generate the generated special effects video.

[0140] According to one or more embodiments of the present disclosure, the variational autoencoder unit includes an encoding subunit and a decoding subunit. When the second generation module processes the image to be processed and the target image through the variational autoencoder unit to generate a first feature code corresponding to the image to be processed and a second feature code corresponding to the target image, respectively, the second generation module is specifically used to: encode the image to be processed and the target image through the encoding subunit to generate a first feature code corresponding to the image to be processed and a second feature code corresponding to the target image, respectively; when the second generation module processes the input feature code through the diffusion transformer unit to generate the special effect video, the second generation module is specifically used to: obtain a randomly initialized random feature code; input the input feature code and the random feature code into the diffusion transformer unit, and use the diffusion transformer unit to perform multiple rounds of updating the random feature code based on the input feature code until a preset stop condition is reached, and then obtain an output feature code; decode the output feature code through the decoding subunit to generate the special effect video.

[0141] According to one or more embodiments of the present disclosure, the output feature code includes a first feature code, a first sequential frame code and a second sequential frame code, wherein the first sequential frame code is generated based on the first feature code and is used to characterize the image content of N continuously changing video frames after the first frame of the special effects video; the second sequential frame code is generated based on the first sequential frame code and the second feature code and is used to characterize the image content of M continuously changing video frames at the end of the special effects video.

[0142] According to one or more embodiments of the present disclosure, when the second generation module decodes the output feature code through the decoding subunit to generate the special effect video, it is specifically used to: decode the first feature code through the decoding subunit to generate a first frame of the video; decode the first sequential frame code through the decoding subunit to generate N video middle frames; decode the second sequential frame code through the decoding subunit to generate M video end frames; splice the video first frame, N video middle frames and M video end frames to generate the special effect video.

[0143] According to one or more embodiments of the present disclosure, when the second generation module calls the video generation model to process the image to be processed and the target image to generate a special effects video, it is specifically used to: obtain a second prompt word, where the second prompt word is used to describe the changing process characteristics of the target effect in the special effects video; call the video generation model, process the image to be processed and the target image based on the second prompt word, and generate a special effects video with the changing process characteristics.

[0144] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0145] The memory stores computer-executable instructions;

[0146] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the special effects video generation method described in the first aspect and various possible designs of the first aspect.

[0147] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the special effects video generation method described in the first aspect and various possible designs of the first aspect is implemented.

[0148] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the special effects video generation method described in the first aspect and various possible designs of the first aspect.

[0149] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0150] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0151] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A special effects video generation method, characterized in that: include: Acquire an image to be processed, wherein the image to be processed contains a target object; Based on the image to be processed, generating a target image, wherein the target image is an image generated after applying a target special effect to the target object in the image to be processed, and the target object in the target image has a target effect generated by the target special effect; Call a video generation model to process the image to be processed and the target image to generate a special effects video, wherein the special effects video includes a first video frame, a last video frame, and at least one video middle frame between the first video frame and the last video frame, the first video frame has the same image content as the image to be processed, the last video frame has the same image content as the target image, and the at least one video middle frame is used to characterize a continuous change process of generating the target effect.

2. The method according to claim 1, characterized in that The step of generating a target image based on the image to be processed comprises: Acquire a first prompt word for a target effect of the target category, where the first prompt word is used to describe an appearance feature of the target effect of the target category; The image to be processed and the first prompt word are processed by an image generation model to generate the target image.

3. The method according to claim 2, characterized in that The image generation model comprises a first functional unit and a second functional unit, wherein the first functional unit is used to generate a target effect of the target category, and the second functional unit is used to generate a second object matching the target effect based on the target object, wherein the appearance similarity between the second object and the target object is greater than a similarity threshold; The step of processing the image to be processed and the first prompt word by using an image generation model to generate the target image includes: Processing the image to be processed and the first prompt word by the first functional unit to generate effect characteristic data representing the target effect; Processing the image to be processed and the effect feature data by the second functional unit to generate object feature data representing the second object; The target image is generated based on the effect feature data and the object feature data.

4. The method according to claim 2, characterized in that: The first prompt word includes a first participle, a second participle and a third participle, wherein the first participle is used to indicate the target object; the second participle is used to characterize the shape feature of the target effect; and the third participle is used to characterize the style feature of the target effect.

5. The method according to claim 1, characterized in that The video generation model includes a variational autoencoder unit and a diffusion transformer unit, and calling the video generation model to process the image to be processed and the target image to generate a special effect video includes: Processing the image to be processed and the target image by a variational autoencoder unit to generate a first feature code corresponding to the image to be processed and a second feature code corresponding to the target image, respectively; Sequentially concatenate the first feature code, the blank frame code, and the second feature code to obtain an input feature code; The input feature code is processed by a diffusion transformer unit to generate the generated special effect video.

6. The method according to claim 5, characterized in that The variational self-encoder unit includes an encoding subunit and a decoding subunit, and the variational self-encoder unit processes the image to be processed and the target image to generate a first feature code corresponding to the image to be processed and a second feature code corresponding to the target image, respectively, including: Encoding the image to be processed and the target image by the encoding subunit to generate a first feature code corresponding to the image to be processed and a second feature code corresponding to the target image respectively; The step of processing the input feature code by a diffusion transformer unit to generate the special effect video comprises: Get randomly initialized random feature codes; Inputting the input feature code and the random feature code into the diffusion transformer unit, and using the diffusion transformer unit to perform multiple rounds of updating on the random feature code based on the input feature code until a preset stop condition is reached, thereby obtaining an output feature code; The output feature code is decoded by the decoding subunit to generate the special effect video.

7. The method according to claim 6, characterized in that The output feature code includes a first feature code, a first time sequence frame code and a second time sequence frame code, wherein: The first time-series frame code is generated based on the first feature code and is used to represent the image content of N video frames that change continuously after the first frame of the special effect video; The second temporal frame code is generated based on the first temporal frame code and the second feature code, and is used to characterize the image content of M continuously changing video frames at the end of the special effects video.

8. The method according to claim 7, characterized in that The step of decoding the output feature code by the decoding subunit to generate the special effect video includes: Decoding the first feature code by the decoding subunit to generate a first frame of the video; Decoding the first time sequence frame code by the decoding subunit to generate N video intermediate frames; Decoding the second time sequence frame code by the decoding subunit to generate M video end frames; The first frame of the video, N middle frames of the video and M last frames of the video are spliced ​​to generate the special effects video.

9. The method according to claim 1, characterized in that: The calling of the video generation model to process the image to be processed and the target image to generate a special effects video includes: Acquire a second prompt word, where the second prompt word is used to describe the characteristics of the change process of the target effect in the special effect video; The video generation model is called to process the image to be processed and the target image based on the second prompt word to generate a special effects video having the characteristics of the change process.

10. A special effects video generation device, characterized in that: include: An acquisition module, used for acquiring an image to be processed; A first generating module is used to generate a target image based on the image to be processed, wherein the target image is an image generated after applying a target special effect to the target object in the image to be processed, and the target object in the target image has a target effect generated by the target special effect; The second generation module is used to call the video generation model to process the image to be processed and the target image to generate a special effects video, wherein the special effects video includes a first video frame, a last video frame, and at least one video middle frame located between the first video frame and the last video frame, the first video frame has the same image content as the image to be processed, the last video frame has the same image content as the target image, and the at least one video middle frame is used to characterize the continuous change process of generating the target effect.

11. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the special effect video generation method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the special effects video generation method as described in any one of claims 1 to 9 is implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the special effects video generation method according to any one of claims 1 to 9 is implemented.

Images