Film and television key frame generation method and device, equipment and medium
By performing global semantic encoding and detail feature extraction on the keyframe generation method for film and television, and using a multimodal diffusion model to process the keyframes, the problem that existing keyframes cannot meet the accuracy requirements of close-up shots in film and television is solved, and high-frequency texture details are preserved and composition accuracy is improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- MALANSHAN AUDIO & VIDEO LABORATORY
- Filing Date
- 2026-01-09
- Publication Date
- 2026-04-21
AI Technical Summary
Existing technologies cannot meet the precision requirements of close-up shots in film and television when generating keyframes, especially in terms of effectively preserving high-frequency texture details, such as moles at the corners of actors' eyes and the material textures of props.
By performing global semantic encoding and detail feature extraction on the target composite canvas, and using a multimodal diffusion model to process global semantics and detail features, keyframes for film and television are generated.
It improves the detail fidelity and composition accuracy of the generated images, ensuring the accurate preservation of actors' facial features and prop textures.
Smart Images
Figure CN121908083A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, device, and medium for generating keyframes in film and television. Background Technology
[0002] In professional AIGC film and television production workflows, the intermediate step from script to video generation—"keyframe design"—has extremely high requirements for continuity. For example, element consistency requirements: the main characters in the film (such as AI characters trained by specific actors) must maintain absolute consistency in facial features, hairstyles, and clothing textures in different shots.
[0003] Currently, the process of generating keyframes using a canvas only performs global semantic encoding on the canvas. This method loses high-frequency texture details (such as a mole near an actor's eye or the texture of a prop) during the conversion of the canvas image into a token, resulting in generated keyframes that cannot meet the precision requirements of close-up shots in film and television. Therefore, improving the detail and precision of generated keyframes has become a pressing technical problem. Summary of the Invention
[0004] In view of this, the purpose of this invention is to provide a method, apparatus, device, and medium for generating keyframes in film and television. By processing global semantics and detailed features, it achieves an understanding of the overall logic of the image and the preservation of detailed textures, thereby improving the detail fidelity and compositional accuracy of the generated image. The specific solution is as follows:
[0005] Firstly, this application provides a method for generating keyframes in film and television, applied on a server-side, including:
[0006] Obtain the target composite canvas and the corresponding entity reference image list sent by the client; wherein, the target composite canvas includes a scene layer, an entity layer and a control layer, the scene layer is the base of the target composite canvas, the entity layer is used to place entity reference images in the entity reference image list, and the scene layer is used to record the position mask of the entity in the target composite canvas;
[0007] The target composite canvas is subjected to global semantic encoding to obtain the corresponding global semantic feature sequence, and detailed features are extracted from each entity reference image in the entity reference image list to obtain pixel-level detailed feature groups.
[0008] The global semantic feature sequence and the detail feature group are processed using a target multimodal diffusion model to obtain the target video keyframes corresponding to the target composite canvas.
[0009] Optionally, the process of the client constructing the target composite canvas includes:
[0010] A preset scene image is determined as the scene layer. A target entity reference image selected by the target user from a preset entity asset library is received, and the target entity reference image is placed at a target position in the scene layer to obtain the entity layer; wherein, the target entity reference image is an image with the background removed.
[0011] The target position mask is automatically generated based on the placement position of each entity to obtain the control layer, and the target composite canvas is constructed based on the scene layer, the entity layer and the control layer.
[0012] Optionally, the step of performing global semantic encoding on the target composite canvas to obtain the corresponding global semantic feature sequence includes:
[0013] Construct target prompt words corresponding to the target composite canvas, and use a preset encoder and the target prompt words to perform global semantic encoding on the target composite canvas to obtain the global semantic feature sequence.
[0014] Optionally, before extracting detailed features from each entity reference image in the entity reference image list, the method further includes:
[0015] The model parameters of the preset image feature extractor are frozen to obtain the target image feature extractor corresponding to the target composite canvas.
[0016] Optionally, the step of extracting detailed features from each entity reference image in the entity reference image list to obtain pixel-level detailed feature groups includes:
[0017] The target image feature extractor is used to extract detail features from each of the entity reference images in the entity reference image list to obtain detail feature vectors corresponding to each entity reference image, and the detail feature group is constructed using each detail feature vector.
[0018] Optionally, the processing of the global semantic feature sequence and the detail feature group using the target multimodal diffusion model includes:
[0019] At each time step, the self-attention mechanism of the target multimodal diffusion model is used to perform self-attention calculation on the first noise latent variable in the target multimodal diffusion model to obtain the corresponding second noise latent variable;
[0020] The cross-attention mechanism of the target multimodal diffusion model is used to perform cross-attention calculation on the second noise latent variable and the global semantic feature sequence to obtain the corresponding third noise latent variable;
[0021] Based on the target location mask, the detailed features of each entity in the detailed feature group are injected into the corresponding target spatial location in the third noise latent variable to obtain the corresponding fourth noise latent variable;
[0022] The fourth noise latent variable and the preset background latent variable are mixed based on the target location mask; wherein, the preset background latent variable is the latent variable of the background image corresponding to the scene layer.
[0023] Optionally, the step of mixing the fourth noise latent variable with the preset background latent variable based on the target location mask includes:
[0024] Determine the first fusion coefficient corresponding to the fourth noise latent variable and the second fusion coefficient corresponding to the preset background latent variable; wherein, the first fusion coefficient is less than the second fusion coefficient;
[0025] The fourth noise latent variable and the preset background latent variable are latently mixed according to the first fusion coefficient, the second fusion coefficient and the target location mask.
[0026] Secondly, this application provides a keyframe generation device for film and television, applied on a server side, comprising:
[0027] The canvas acquisition module is used to acquire the target composite canvas and the corresponding entity reference image list sent by the client; wherein, the target composite canvas includes a scene layer, an entity layer and a control layer, the scene layer is the base of the target composite canvas, the entity layer is used to place entity reference images in the entity reference image list, and the scene layer is used to record the position mask of the entity in the target composite canvas;
[0028] The feature extraction module is used to perform global semantic encoding on the target composite canvas to obtain the corresponding global semantic feature sequence, and to extract detailed features from each entity reference image in the entity reference image list to obtain pixel-level detailed feature groups.
[0029] The keyframe acquisition module is used to process the global semantic feature sequence and the detail feature group using a target multimodal diffusion model to obtain the target video keyframes corresponding to the target composite canvas.
[0030] Thirdly, this application provides an electronic device, comprising:
[0031] Memory, used to store computer programs;
[0032] A processor is used to execute the computer program to implement the aforementioned method for generating keyframes in film and television.
[0033] Fourthly, this application provides a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the aforementioned method for generating keyframes in film and television.
[0034] In this application, the client first obtains the target composite canvas and the corresponding entity reference image list sent by the client. The target composite canvas includes a scene layer, an entity layer, and a control layer. The scene layer serves as the base of the target composite canvas, the entity layer is used to place entity reference images from the entity reference image list, and the scene layer records the position mask of the entities within the target composite canvas. Then, global semantic encoding is performed on the target composite canvas to obtain the corresponding global semantic feature sequence. Detail features are extracted from each entity reference image in the entity reference image list to obtain pixel-level detail feature groups. Finally, a target multimodal diffusion model is used to process the global semantic feature sequence and the detail feature groups to obtain the target video keyframes corresponding to the target composite canvas. Therefore, this application achieves an understanding of the overall logic of the image and the preservation of detailed textures by performing global semantic encoding on the target composite canvas; by extracting pixel-level detail features from the entity reference images to retain high-frequency textures and identity information; and finally, by using a multimodal diffusion model to process global semantics and detail features, thereby improving the detail fidelity and composition accuracy of the generated image. Attached Figure Description
[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0036] Figure 1 This is a schematic diagram of a method for generating keyframes in film and television disclosed in this application;
[0037] Figure 2 This application discloses a canvas generation flowchart;
[0038] Figure 3 This is a flowchart of a feature extraction process disclosed in this application;
[0039] Figure 4 This application discloses a flowchart of a keyframe generation process;
[0040] Figure 5 This is a schematic diagram of a film and television keyframe generation device disclosed in this application;
[0041] Figure 6This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation
[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0043] Current methods for generating keyframes in film and television suffer from the problem that the generated keyframes cannot meet the precision requirements of close-up shots. To address this, this application provides a method for generating keyframes in film and television that processes global semantics and detailed features to achieve an understanding of the overall logic of the image and the preservation of detailed textures, thereby improving the detail fidelity and compositional accuracy of the generated image.
[0044] See Figure 1 As shown, this embodiment of the invention discloses a method for generating keyframes in film and television, applied to a server, including:
[0045] Step S11: Obtain the target composite canvas and the corresponding entity reference image list sent by the client; wherein, the target composite canvas includes a scene layer, an entity layer and a control layer, the scene layer is the base of the target composite canvas, the entity layer is used to place entity reference images in the entity reference image list, and the scene layer is used to record the position mask of the entity in the target composite canvas.
[0046] In this embodiment, the server first needs to obtain the target composite canvas sent by the client. That is, the target composite canvas is generated by the client. Accordingly, the process of the client generating the target composite canvas includes: determining a preset scene image as the scene layer, receiving a target entity reference image selected by the target user from a preset entity asset library, and placing the target entity reference image at the target position in the scene layer to obtain the entity layer; wherein, the target entity reference image is an image with the background removed; automatically generating the corresponding target position mask according to the placement position of each entity to obtain the control layer, and constructing the target composite canvas based on the scene layer, entity layer and control layer.
[0047] The process of generating the canvas described above is as follows: Figure 2 As shown, the process includes specifying actions, dragging in characters and props, selecting a fixed scene template, automatically generating a position mask, and collaging the scene template and prop reference image into a canvas.
[0048] In this embodiment, the system is divided into three main processing stages: composite input construction (client), dual-stream feature extraction (server), and region-aware generation (server).
[0049] The process of constructing the above client-side composite input is as follows: the user operates on the client interface, generating a data packet containing multi-level information.
[0050] 1. Base Layer: The user selects a "fixed scene image" ( ( ) as the base plate.
[0051] 2. Entity Layer: Users drag and drop entity reference images from the asset library, including "role reference images" (…). ) and "Props Reference Diagram" ( After the image is cut out (that is, the target entity reference image is an image without the background), it is placed in the specified position on the base plate.
[0052] 3. Control Layer:
[0053] Mask / Box Generation: The system automatically records the position mask of each entity on the canvas. ).
[0054] Pose rendering: For characters, users can overlay OpenPose skeleton maps ( ( ) to specify the action.
[0055] 4. Composite Output: The client will composite the RGB canvas (...) (i.e., the target composite canvas) along with the original reference image list ( ) and corresponding mask list ( Send to the server.
[0056] Step S12: Perform global semantic encoding on the target composite canvas to obtain the corresponding global semantic feature sequence, and extract detailed features from each entity reference image in the entity reference image list to obtain pixel-level detailed feature groups.
[0057] This embodiment employs a dual-path encoding method. The original VLM semantic path is retained for understanding lighting and logic, while a new high-fidelity reference stream is added to directly extract high-definition features from the original footage without VLM compression, ensuring the preservation of skin textures and prop details. The specific process is as follows: Figure 3 As shown, the process includes: processing the original high-resolution reference image using an image feature extractor to obtain local detail feature groups; inputting the canvas and prompts into a VLM encoder to obtain a global semantic feature sequence; in addition, the server will encode the fixed scene image as a background latent variable.
[0058] Accordingly, in this embodiment, the process of performing global semantic encoding on the target composite canvas to obtain the corresponding global semantic feature sequence includes: constructing target prompt words corresponding to the target composite canvas, and using a preset encoder and target prompt words to perform global semantic encoding on the target composite canvas to obtain the global semantic feature sequence.
[0059] It should be noted that, before extracting detailed features from each entity reference image in the entity reference image list in this embodiment, the method further includes: freezing the model parameters of the preset image feature extractor to obtain the target image feature extractor corresponding to the target composite canvas.
[0060] In addition, the process of extracting detail features from each entity reference image in the entity reference image list to obtain pixel-level detail feature groups includes: using a target image feature extractor to extract detail features from each entity reference image in the entity reference image list to obtain detail feature vectors corresponding to each entity reference image, and using each detail feature vector to construct detail feature groups.
[0061] Specifically, after receiving the data, the server processes the image information in parallel through two encoders:
[0062] 1. Semantic Stream (inherited from the original paper's architecture):
[0063] Input: Composite canvas + Text prompt T (i.e., target prompt word).
[0064] Model: Visual Language Model Encoder (VLM Encoder, e.g., Qwen-VL).
[0065] Output: Global semantic token sequence (i.e., global semantic feature sequence).
[0066] Function: Responsible for understanding the overall logic of the picture (such as "a person is sitting on a sofa", "the light comes from the left", "character A is looking at character B").
[0067] 2. High-fidelity reference stream (Fidelity Stream - New module):
[0068] Input: A list of original high-resolution reference images, i.e., a list of entity reference images. .
[0069] Model: Image feature extractor (Image Encoder, such as CLIP Vision Model or ArcFace, with frozen parameters).
[0070] Output: Local detail feature set Each It corresponds to a specific character ID or item texture.
[0071] Function: Responsible for providing "pixel-level" texture and identity information to solve the problem of VLM losing details.
[0072] By performing global semantic understanding and detail feature extraction on the image, the accuracy of detail features under global constraints is ensured.
[0073] Step S13: Process the global semantic feature sequence and the detail feature group using the target multimodal diffusion model to obtain the target video keyframes corresponding to the target composite canvas.
[0074] In this embodiment, the process of processing the global semantic feature sequence and detail feature group using the target multimodal diffusion model includes: performing self-attention calculation on the first noise latent variable in the target multimodal diffusion model using the self-attention mechanism of the target multimodal diffusion model at each time step to obtain the corresponding second noise latent variable; performing cross-attention calculation on the second noise latent variable and the global semantic feature sequence using the cross-attention mechanism of the target multimodal diffusion model to obtain the corresponding third noise latent variable; injecting the detail features of each entity in the detail feature group into the corresponding target spatial position in the third noise latent variable based on the target position mask to obtain the corresponding fourth noise latent variable; and performing latent variable mixing on the fourth noise latent variable and the preset background latent variable based on the target position mask; wherein, the preset background latent variable is the latent variable of the background image corresponding to the scene layer.
[0075] The process of mixing the fourth noise latent variable with the preset background latent variable based on the target location mask includes: determining the first fusion coefficient corresponding to the fourth noise latent variable and the second fusion coefficient corresponding to the preset background latent variable; wherein the first fusion coefficient is less than the second fusion coefficient; and mixing the fourth noise latent variable with the preset background latent variable according to the first fusion coefficient, the second fusion coefficient and the target location mask.
[0076] The above process is as follows Figure 4 As shown, the process includes steps such as region-aware model prediction of latent variables and background latent variable anchoring and mixing, ultimately generating target video keyframes.
[0077] It should be noted that (self-attention → semantic cross-attention → injection of details) is not a strictly sequential order required in the solution. In actual Transformer Blocks, these attention layers are usually parallel or flexibly stacked, and there is not necessarily a strict dependency relationship between "second, third, and fourth" noise latent variables. The above description is merely to illustrate the logical flow.
[0078] Specifically, this stage uses the improved Region-Aware MM-DiT (Multimodal Diffusion Transformer) for image generation.
[0079] 1. Improved Transformer Block structure: In the original MM-DiT Block (the original structure only contains Self-Attention and Cross-Attention), a new sub-module is inserted: Masked ReferenceAttention.
[0080] 2. Detailed Description of Processing Flow: Assume the current diffusion time step is t, and the input noise latent variable is... .
[0081] Step A: Self-Attention: Self-attention calculation is performed on (i.e., noise latent variables) to handle spatial correlation within the image.
[0082] Step B: Semantic Cross-Attention: and Cross-Attention is performed using VLM semantic tokens. This step allows the model to understand the overall composition and lighting logic.
[0083] Step C: Masked Reference Attention (Added Step):
[0084] Objective: To Precise injection of features into The corresponding position.
[0085] Input: Query from Key / Value comes from .
[0086] Mechanism: Introducing a mask uploaded by the client { } as spatial bias.
[0087] Calculation logic: For each pixel (x, y) in the generated image feature map:
[0088] Determine which mask (x, y) falls under. Within the range.
[0089] If it falls If the point is within the eigenvector of Character A, then the eigenvector of that point is within the eigenvector of High weights are assigned to other reference images, while zero weights are assigned to others. (That is, this embodiment employs spatial routing. Utilizing positional information in the canvas, such as Mask / Box, high-resolution features extracted during the generation process are injected "point-to-point" into the corresponding regions of the diffusion model, forcing the model to call only the features of specific roles at specific coordinates.)
[0090] If it falls within the background area, skip this layer or focus only on global style features.
[0091] Effect: When drawing the face area, the model must refer to the ID features of a specific actor; when drawing the prop area, it must refer to the texture of a specific prop.
[0092] Step D: Background Anchoring: In the final output stage of each denoising step, latent blending is performed.
[0093] Pre-set the original fixed scene image Encoding as latent variables (i.e., background latent variables).
[0094] Union using foreground masks The generated latent variables and mix:
[0095] ;in A small blending coefficient (e.g., 0.1) allows for subtle changes in background lighting and shadows (such as projections) while keeping the object structure unchanged. (That is, a background latent variable anchoring technique is used. During the generation process, pixel-level latent variable backfilling is performed on non-occluded areas to ensure that the background matches the set parameters.) Figure 1 (Note: Only the blending of light and shadow is permitted.)
[0096] The final It contains all your expected content (fixed background, precise actors, accurate props), the clearest latent variables. However, it's still just a bunch of digital code. At this point, the system will call a decoder that works with the encoder to... Convert back to a standard pixel image to obtain the final video keyframes.
[0097] This embodiment uses the Masked Reference Attention module to physically bind "spatial location" with "identity features." This makes it possible for multiple specific actors and props to appear simultaneously in a single scene without interfering with each other.
[0098] Furthermore, by designing a background latent variable mixing mechanism, the generated foreground characters are allowed to cast realistic shadows and reflections on the background, while ensuring that the original background assets such as furniture and wallpaper are not redrawn or distorted by AI, perfectly meeting the stringent requirements of scene consistency in storyboarding.
[0099] As can be seen, this application obtains the overall composition and lighting logic by performing global semantic encoding on the target composite canvas; extracts pixel-level detail features from the entity reference image to retain high-frequency texture and identity information; and finally uses a multimodal diffusion model to process global semantics and detail features, thereby realizing the understanding of the overall logic of the picture and the preservation of detail texture, improving the detail fidelity and composition accuracy of the generated image.
[0100] See Figure 5 As shown, this embodiment of the invention discloses a keyframe generation device for film and television, applied to a server, comprising:
[0101] The canvas acquisition module 11 is used to acquire the target composite canvas and the corresponding entity reference image list sent by the client; wherein, the target composite canvas includes a scene layer, an entity layer and a control layer, the scene layer is the base of the target composite canvas, the entity layer is used to place entity reference images in the entity reference image list, and the scene layer is used to record the position mask of the entity in the target composite canvas;
[0102] The feature extraction module 12 is used to perform global semantic encoding on the target composite canvas to obtain the corresponding global semantic feature sequence, and to extract detailed features from each entity reference image in the entity reference image list to obtain pixel-level detailed feature groups.
[0103] The keyframe acquisition module 13 is used to process the global semantic feature sequence and the detail feature group using the target multimodal diffusion model to obtain the target video keyframes corresponding to the target composite canvas.
[0104] In some specific embodiments, the client may specifically include:
[0105] An image placement module is used to determine a preset scene image as the scene layer, receive a target entity reference image selected by a target user from a preset entity asset library, and place the target entity reference image at a target position in the scene layer to obtain the entity layer; wherein, the target entity reference image is an image with the background removed;
[0106] The canvas construction module is used to automatically generate corresponding target position masks based on the placement position of each entity in order to obtain the control layer, and to construct the target composite canvas based on the scene layer, the entity layer and the control layer.
[0107] In some specific embodiments, the feature extraction module 12 may specifically include:
[0108] The canvas encoding unit is used to construct the target prompt word corresponding to the target composite canvas, and to perform global semantic encoding on the target composite canvas using a preset encoder and the target prompt word to obtain the global semantic feature sequence.
[0109] In some specific embodiments, the feature extraction module 12 further includes:
[0110] The parameter freezing unit is used to freeze the model parameters of the preset image feature extractor in order to obtain the target image feature extractor corresponding to the target composite canvas.
[0111] In some specific embodiments, the feature extraction module 12 may specifically include:
[0112] The feature extraction unit is used to extract detailed features from each of the entity reference images in the entity reference image list using the target image feature extractor, so as to obtain detailed feature vectors corresponding to each entity reference image, and to construct the detailed feature group using each detailed feature vector.
[0113] In some specific embodiments, the keyframe acquisition module 13 may specifically include:
[0114] The first latent variable calculation unit is used to perform self-attention calculation on the first noise latent variable in the target multimodal diffusion model at each time step using the self-attention mechanism of the target multimodal diffusion model, so as to obtain the corresponding second noise latent variable;
[0115] The second latent variable calculation unit is used to perform cross-attention calculation on the second noise latent variable and the global semantic feature sequence using the cross-attention mechanism of the target multimodal diffusion model to obtain the corresponding third noise latent variable;
[0116] The feature injection unit is used to inject the detailed features of each entity in the detailed feature group into the corresponding target spatial position in the third noise latent variable based on the target position mask, so as to obtain the corresponding fourth noise latent variable.
[0117] The latent variable mixing submodule is used to perform latent variable mixing of the fourth noise latent variable and the preset background latent variable based on the target location mask; wherein, the preset background latent variable is the latent variable of the background image corresponding to the scene layer.
[0118] In some specific embodiments, the latent variable mixing submodule may specifically include:
[0119] A fusion coefficient determination unit is used to determine a first fusion coefficient corresponding to the fourth noise latent variable and a second fusion coefficient corresponding to the preset background latent variable; wherein, the first fusion coefficient is less than the second fusion coefficient;
[0120] The latent variable mixing unit is used to perform latent variable mixing on the fourth noise latent variable and the preset background latent variable according to the first fusion coefficient, the second fusion coefficient and the target position mask.
[0121] Furthermore, embodiments of this application also disclose an electronic device, Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.
[0122] Figure 6 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the video keyframe generation method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.
[0123] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.
[0124] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.
[0125] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the video keyframe generation method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.
[0126] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned method for generating keyframes in film and television. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.
[0127] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.
[0128] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0129] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0130] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0131] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A method for generating keyframes in film and television, characterized in that, Applied to the server side, including: Obtain the target composite canvas and the corresponding entity reference image list sent by the client; wherein, the target composite canvas includes a scene layer, an entity layer and a control layer, the scene layer is the base of the target composite canvas, the entity layer is used to place entity reference images in the entity reference image list, and the scene layer is used to record the position mask of the entity in the target composite canvas; The target composite canvas is subjected to global semantic encoding to obtain the corresponding global semantic feature sequence, and detailed features are extracted from each entity reference image in the entity reference image list to obtain pixel-level detailed feature groups. The global semantic feature sequence and the detail feature group are processed using a target multimodal diffusion model to obtain the target video keyframes corresponding to the target composite canvas.
2. The method for generating keyframes in film and television according to claim 1, characterized in that, The process by which the client constructs the target composite canvas includes: A preset scene image is determined as the scene layer. A target entity reference image selected by the target user from a preset entity asset library is received, and the target entity reference image is placed at a target position in the scene layer to obtain the entity layer; wherein, the target entity reference image is an image with the background removed. The target position mask is automatically generated based on the placement position of each entity to obtain the control layer, and the target composite canvas is constructed based on the scene layer, the entity layer and the control layer.
3. The method for generating keyframes in film and television according to claim 1, characterized in that, The step of performing global semantic encoding on the target composite canvas to obtain the corresponding global semantic feature sequence includes: Construct target prompt words corresponding to the target composite canvas, and use a preset encoder and the target prompt words to perform global semantic encoding on the target composite canvas to obtain the global semantic feature sequence.
4. The method for generating keyframes in film and television according to claim 1, characterized in that, Before performing detailed feature extraction on each entity reference image in the entity reference image list, the method further includes: The model parameters of the preset image feature extractor are frozen to obtain the target image feature extractor corresponding to the target composite canvas.
5. The method for generating keyframes in film and television according to claim 4, characterized in that, The step of extracting detailed features from each entity reference image in the entity reference image list to obtain pixel-level detailed feature groups includes: The target image feature extractor is used to extract detail features from each of the entity reference images in the entity reference image list to obtain detail feature vectors corresponding to each entity reference image, and the detail feature group is constructed using each detail feature vector.
6. The method for generating keyframes in film and television according to claim 2, characterized in that, The process of using a target multimodal diffusion model to process the global semantic feature sequence and the detail feature group includes: At each time step, the self-attention mechanism of the target multimodal diffusion model is used to perform self-attention calculation on the first noise latent variable in the target multimodal diffusion model to obtain the corresponding second noise latent variable; The cross-attention mechanism of the target multimodal diffusion model is used to perform cross-attention calculation on the second noise latent variable and the global semantic feature sequence to obtain the corresponding third noise latent variable; Based on the target location mask, the detailed features of each entity in the detailed feature group are injected into the corresponding target spatial location in the third noise latent variable to obtain the corresponding fourth noise latent variable; The fourth noise latent variable and the preset background latent variable are mixed based on the target location mask; wherein, the preset background latent variable is the latent variable of the background image corresponding to the scene layer.
7. The method for generating keyframes in film and television according to claim 6, characterized in that, The step of mixing the fourth noise latent variable with the preset background latent variable based on the target location mask includes: Determine the first fusion coefficient corresponding to the fourth noise latent variable and the second fusion coefficient corresponding to the preset background latent variable; wherein, the first fusion coefficient is less than the second fusion coefficient; The fourth noise latent variable and the preset background latent variable are latently mixed according to the first fusion coefficient, the second fusion coefficient and the target location mask.
8. A device for generating keyframes for film and television, characterized in that, Applied to the server side, including: The canvas acquisition module is used to acquire the target composite canvas and the corresponding entity reference image list sent by the client; wherein, the target composite canvas includes a scene layer, an entity layer and a control layer, the scene layer is the base of the target composite canvas, the entity layer is used to place entity reference images in the entity reference image list, and the scene layer is used to record the position mask of the entity in the target composite canvas; The feature extraction module is used to perform global semantic encoding on the target composite canvas to obtain the corresponding global semantic feature sequence, and to extract detailed features from each entity reference image in the entity reference image list to obtain pixel-level detailed feature groups. The keyframe acquisition module is used to process the global semantic feature sequence and the detail feature group using a target multimodal diffusion model to obtain the target video keyframes corresponding to the target composite canvas.
9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the video keyframe generation method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the video keyframe generation method as described in any one of claims 1 to 7.