Multi-lens consistency video generation method based on dynamic memory management

Abstract

The invention provides a multi-shot consistency video generation method based on dynamic memory management, and the method comprises the steps: carrying out the coding and feature extraction of a story script to obtain semantic information, and mapping the semantic information into a structured shot description; entity categories and attribute vectors are extracted according to the shot description; retrieving a matched target entity reference image in a dynamic memory library according to the entity category and the attribute vector; when the matched entity reference image does not exist in the dynamic memory bank, generating a new entity reference image according to the attribute vector and the historical image, and writing the new entity reference image into the dynamic memory bank; generating key frames according to the shot description and the target entity reference image, and synthesizing a video according to a plurality of key frames; the performance of role consistency, prop consistency and background consistency in long video generation can be remarkably improved, and the method has high robustness and expandability.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to a method for generating multi-lens consistent video based on dynamic memory management. Background Technology

[0002] Current text-to-video technology has achieved high-quality performance in generating short videos (single shot, about a few seconds), but it still has significant shortcomings in generating long videos and multi-shot narratives. Characters in the video need to maintain consistent identities at different times and from different perspectives, the appearance of key props needs to remain stable, and the background environment also needs to remain identifiable between shots. These are all basic requirements for generating long videos.

[0003] However, most existing models employ single-shot generation or plan-generate methods, which cannot achieve consistent video generation of characters, items, and backgrounds. Therefore, the lack of a mechanism capable of "long-term memory of entities, explicit retrieval, and updating" is the fundamental reason for the current inconsistency in multi-camera video generation. Summary of the Invention

[0004] The following is an overview of the topics described in detail in this article.

[0005] The purpose of this application is to at least partially solve one of the technical problems existing in the related technologies. The embodiments of this application provide a multi-camera consistent video generation method based on dynamic memory management, which can improve the consistency performance of long video generation.

[0006] An embodiment of this application provides a method for generating multi-camera consistent video based on dynamic memory management, comprising:

[0007] Obtain the story script and input the story script into the video generation network; The story script is encoded and features are extracted to obtain semantic information, and the semantic information is mapped into structured shot descriptions; Based on the lens description, entity categories and attribute vectors are extracted; Based on the entity category and the attribute vector, retrieve the matching target entity reference image from the dynamic memory; When the dynamic memory does not have an entity reference image that matches the entity category and the attribute vector, a new entity reference image is generated as the target entity reference image based on the historical image corresponding to the attribute vector and the entity category, and the new entity reference image is written into the dynamic memory. Keyframes are generated based on the shot description and the target entity reference image, and a video is synthesized based on multiple keyframes.

[0008] According to certain embodiments of this application, the video generation network includes a storyboard agent, which encodes and extracts features from the story script to obtain semantic information, and maps the semantic information into a structured shot description. The semantic information includes characters, props, backgrounds, and scene rhythms, and the shot description includes information on characters, props, backgrounds, and scene rhythms in a shot.

[0009] According to certain embodiments of this application, the video generation network includes a memory management agent, which is used to extract entity categories and attribute vectors based on the shot description, retrieve matching target entity reference images from a dynamic memory based on the entity categories and attribute vectors, and when there are no matching entity reference images in the dynamic memory, generate new entity reference images as target entity reference images based on historical images corresponding to the attribute vectors and entity categories, and write the new entity reference images into the dynamic memory.

[0010] According to certain embodiments of this application, the video generation network includes a visualization generation agent, which is used to generate keyframes based on the shot description and the target entity reference image, and to synthesize a video based on a plurality of the keyframes.

[0011] According to certain embodiments of this application, generating a new entity reference image as the target entity reference image based on the historical image corresponding to the attribute vector and the entity category includes: When the entity category is not present in the dynamic memory, a first generation condition is constructed based on the attribute vector, and a new entity reference image is generated by an image generation tool based on the first generation condition.

[0012] According to certain embodiments of this application, generating a new entity reference image as the target entity reference image based on the historical image corresponding to the attribute vector and the entity category includes: When the dynamic memory is stored in the entity category but the attribute vector is not present, a second generation condition is constructed based on the attribute vector, and a new entity reference image is generated by an image generation tool based on the second generation condition and the historical image corresponding to the entity category.

[0013] According to certain embodiments of this application, when a new entity reference image is written into the dynamic memory, the historical image of the entity category corresponding to the new entity reference image is retained.

[0014] According to certain embodiments of this application, the entity reference images of the dynamic memory are stored through explicit dynamic entities.

[0015] According to certain embodiments of this application, the dynamic entity includes entity category, attribute information, and entity reference image.

[0016] According to certain embodiments of this application, the entity reference images of the dynamic memory are stored in a categorized manner according to characters, props, and backgrounds.

[0017] The above scheme has at least the following beneficial effects: By constructing a searchable and updatable dynamic entity memory, the visual and semantic states of narrative entities such as characters, props, and backgrounds are stored explicitly, retrieved before each shot is generated, and updated after generation, thus ensuring the continuity of entity images across long time spans and multiple shots. A multi-agent collaborative framework is adopted, including a storyboard agent, a memory management agent, and a visualization generation agent. The three agents work collaboratively to complete steps such as script parsing, entity recognition, memory retrieval and updating, keyframe generation, and shot video generation. By introducing a dynamic memory mechanism, the consistency of characters, props, and backgrounds in long video generation can be significantly improved, and it possesses strong robustness and scalability. Attached Figure Description

[0018] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0019] Figure 1 This is a flowchart illustrating the steps of a multi-camera consistency video generation method based on dynamic memory management. Figure 2 This is a structural diagram of a video generation network; Figure 3 It is an image generated with long-term consistency corresponding to roles across age groups; Figure 4 It is an image that corresponds to the consistent appearance of props across different scenes; Figure 5 It is an image that maintains consistency with the corresponding background scene. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0021] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, or the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0022] Current text-to-video technology has achieved high-quality performance in generating short videos (single shot, about a few seconds), but it still has significant shortcomings in generating long videos and multi-shot narratives. Characters in the video need to maintain consistent identities at different times and from different perspectives, the appearance of key props needs to remain stable, and the background environment also needs to remain identifiable between shots. These are all basic requirements for generating long videos.

[0023] However, most existing models employ single-shot generation or plan-generate methods, which cannot achieve consistent video generation of characters, items, and backgrounds. Therefore, the lack of a mechanism capable of "long-term memory of entities, explicit retrieval, and updating" is the fundamental reason for the current inconsistency in multi-camera video generation.

[0024] Currently, the technical solutions are mainly divided into three types: one-pass method, plan-generate method, and feature-level memory method.

[0025] The one-pass generation method lacks explicit modeling of entities across shots, which can easily lead to problems such as character facial drift, object deformation, and changes in background layout.

[0026] The plan-generate method allows for plot planning, but it lacks visual memory, with each shot generated independently, resulting in poor overall entity consistency.

[0027] Feature-level memory methods store frame-level or local features rather than specific entities, making them unsuitable for retrieving an entity's image long after it has been removed from the memory.

[0028] The following shortcomings exist: It cannot store entity-level visual memory; most methods only maintain short-term feature caches, failing to guarantee that "the character will remain consistent even after reappearing 10 shots away." It lacks explicit entity classification management, resulting in mixed storage of characters, props, and backgrounds, making it impossible for the model to independently handle different types of entities.

[0029] The lack of a story-semantic driven entity update mechanism makes it unable to handle story logic, such as characters aging, props breaking, and changes in background lighting. The generation process cannot utilize scene context; the current system does not refer to "historical visual states" when generating each shot, lacking long-term constraints.

[0030] Traditional methods typically rely solely on textual cues or short-term historical frame features, resampling character appearance, prop shapes, and background environments when generating each shot, lacking explicit modeling and long-term memory of entities. This approach struggles to ensure the stability of a character's identity, age, and clothing across different shots. Furthermore, when props and backgrounds reappear after being absent from intermediate shots, the model often fails to reconstruct their original appearance, leading to issues such as "face swapping" of characters, "deformation" of props, and chaotic scene structures, severely impacting the coherence and usability of multi-shot narratives.

[0031] To address the above issues, embodiments of this application provide a method for generating multi-camera consistent video based on dynamic memory management.

[0032] The embodiments of this application will be further described below with reference to the accompanying drawings.

[0033] Reference Figure 1 A method for generating consistent multi-camera videos includes the following steps: Obtain the story script and input the story script into the video generation network; The story script is encoded and features are extracted to obtain semantic information, which is then mapped into structured shot descriptions. Entity categories and attribute vectors are extracted based on the shot description; Retrieve the matching target entity reference image from the dynamic memory based on the entity category and attribute vector; When the dynamic memory does not have an entity reference image that matches the entity category and attribute vector, a new entity reference image is generated based on the historical image corresponding to the attribute vector and entity category as the target entity reference image, and the new entity reference image is written into the dynamic memory. Keyframes are generated based on shot descriptions and target entity reference images, and video is synthesized based on multiple keyframes.

[0034] In the script-driven multi-shot video generation task, the user first provides a story script as input. A text analysis network with reasoning capabilities encodes, understands, and extracts features from the script, obtaining high-level semantic information such as characters, props, background, and scene rhythm. Next, this semantic information is mapped into a series of structured shot descriptions, each shot containing camera perspective, shot size, participating characters, key props, and background environment. Subsequently, keyframes are generated based on these shot descriptions, and the process is further expanded into a multi-shot video sequence using image-to-video generation, ultimately yielding a complete multi-shot video as output.

[0035] The multi-agent collaboration framework decouples script analysis, entity parsing, memory retrieval and updating, and visualization generation.

[0036] The entity reference images in the dynamic memory are stored through explicit dynamic entities, rather than relying solely on implicit temporal feature caching. Dynamic entities include entity categories, attribute information, and reference images. Entity reference images in the dynamic memory are categorized and stored according to characters, props, and backgrounds, facilitating direct retrieval and manual inspection. The memory supports online updates as the story progresses, naturally depicting changes such as a character's appearance from youth to middle age, props' condition from intact to worn, and scene environmental changes from day to night, ensuring stable entity identities while reflecting plot evolution. Entity reference images in the memory can be reused across multiple shots, providing strong constraints for keyframe generation, ensuring that the generated video maintains a high degree of consistency in character appearance, prop form, and scene layout across different scenes and perspectives.

[0037] Reference Figure 2 The video generation network includes a storyboard agent, a memory management agent, and a visualization generation agent.

[0038] The storyboarding agent encodes and extracts features from the story script to obtain semantic information, which is then mapped into structured shot descriptions.

[0039] Semantic information includes characters, props, background, and scene rhythm. Shot description includes information about characters, props, background, and scene rhythm within a shot.

[0040] Understandably, structured shot descriptions break down a story from a "coherent narrative" into a list of shots that can be executed one by one. It emphasizes "structure," meaning each shot is no longer just random text, but contains clear and resolvable information: what scene is being filmed, which characters are in the shot, the characters' positions and prop locations, the shot size and composition, and the sequence of actions within the shot. The purpose of this is to allow downstream modules to stably generate continuous shots and maintain consistency in characters, props, and timing.

[0041] Treat a "plot segment" as a beat / scene (narrative rhythm), and then break the beat / scene down into multiple shots: first, use a wide shot to establish spatial relationships (letting the audience know "where" and "who is in what position"), then use a medium shot or close-up to advance emotions and conflict (letting the audience see the characters' expressions and the tension of confrontation), and insert close-ups when necessary to emphasize key objects or action triggers (such as a hand gripping a sword hilt or a key falling to the ground). These shots, arranged in sequence, constitute a "multi-shot" narrative chain.

[0042] For example, the "bridge conflict" beat can be structured as follows: The first shot is a wide shot establishing the positional relationship between the stone bridge and the two people walking towards each other; the second shot is a medium shot from the side, showing the two stopping and facing each other, clearly indicating that A is in the front left of the frame and B is in the back right; the third shot is a close-up showing A's expression and lines, with the action sequence being "speaking coldly -> glancing at the other's weapon"; the fourth shot is a close-up of B's ​​hand gripping the hilt of his sword, the sword slightly unsheathed, as a visual signal of escalating conflict. You'll find that structured description is not about "making the story longer," but about clearly breaking down "the elements that should be given shots," ensuring that each shot is executed accurately and connects seamlessly with the preceding and following shots.

[0043] The memory management agent extracts entity categories and attribute vectors based on the lens description. It then retrieves matching target entity reference images from the dynamic memory based on the entity category and attribute vectors. If no matching entity reference image exists in the dynamic memory, a new entity reference image is generated based on the historical images corresponding to the attribute vectors and entity categories, and this new entity reference image is written into the dynamic memory.

[0044] On the one hand, when there is no entity category in the dynamic memory, the first generation condition is constructed based on the attribute vector; and on the other hand, a new entity reference image is generated by the image generation tool based on the first generation condition.

[0045] On the other hand, when the dynamic memory contains entity categories but no attribute vectors, a second generation condition is constructed based on the attribute vectors, and a new entity reference image is generated by the image generation tool based on the second generation condition and the historical images corresponding to the entity categories.

[0046] For example, the memory management agent parses entities such as characters, props, and backgrounds from each shot description and constructs attribute vectors containing information such as age, clothing, state, and time period. It then uses these attribute vectors and entity categories to search in a dynamic memory bank. If a matching entry exists in the memory bank, the entity reference image corresponding to the entity category is directly retrieved for the current shot generation. If no matching entry exists in the memory bank, a new reference image is generated online based on the current attributes and the historical images corresponding to the entity category.

[0047] The memory management agent parses key attributes (such as appearance features, clothing accessories, material textures, and distinctive elements) of entities from the current shot description and treats these attributes as the entity's "state representation" at the current moment. It then matches the parsed results with existing entity records in its memory bank to determine whether the entity is "appearing for the first time" or "has appeared but its state has been updated." "Appearing for the first time" means there is no identical entity category, attribute information, or reference image in the memory bank; "has appeared but its state has been updated" means there is an identical entity category in the memory bank, but the attribute information is different. If the entity has never appeared before (i.e., there is no corresponding entry in the memory bank), the agent constructs generation conditions based on the parsed attributes, directly calls the image generation tool to generate an initial reference image for the entity according to these conditions, and writes "reference image + attribute state" as the entity's first memory entry into the memory bank for consistency constraints in subsequent shots. If an entity has appeared before, but its attributes change in this shot (e.g., changes in clothing, damage, significant changes in expression / posture, or the addition of props), the memory management agent will not start from scratch. Instead, it will input the "previous version of the entity reference image" and the "updated attribute description" into the image generation tool to generate a new entity reference image. After generation, the entity's memory is updated with the new image and new attributes, thus achieving traceable consistency and gradual evolution of the entity across multiple shots. The new entity reference image is written to the dynamic memory bank for incremental updates. When writing the new entity reference image to the dynamic memory bank, the historical image of the entity category corresponding to the new entity reference image is retained.

[0048] The visualization-generating agent generates keyframes based on shot descriptions and target entity reference images, and then synthesizes multiple keyframes into a video using an image-generated video model.

[0049] Specifically, each shot has a structured description output by the storyboard agent. This description includes not only shot language information (such as shot size, composition, camera position / movement, duration, and action sequence), but also clarifies the scene setting, participating characters, and key events that the shot needs to present, thus defining "what to shoot and how to shoot it." Based on this, the memory management agent from the previous step provides entity reference images (reference images of characters, key props, or iconic scene elements) related to the shot, serving as visual constraints for cross-shot consistency. The system inputs both the "shot structured description (textual conditions)" and the "entity reference images (visual conditions)" into the image generation tool, controlling the generation process through the combined conditions of text and images to obtain the keyframes of the shot.

[0050] By repeatedly executing a "retrieve-use-update" memory loop across all shots, it is possible to maintain the consistency of the identity and appearance of characters, props, and backgrounds over a long timescale.

[0051] The innovation of this framework lies in: Dynamic entity memory bank construction and online management: In the script-to-multi-camera video generation task, three types of memory banks, namely, character, prop and background, are introduced. Through the attribute-driven retrieval and update mechanism, entity-level appearance information is stored in the form of images for a long time and reused across shots. Memory-driven keyframe generation strategy: When generating keyframes, both shot text descriptions and entity reference images are used as conditions, so that the model strictly follows the entity appearance stored in the memory bank while arranging the composition and actions, thereby significantly improving the consistency of characters, props and backgrounds across shots. Multi-agent collaborative script-to-video generation process: Through the division of labor and cooperation among storyboard agents, memory management agents, and visualization generation agents, a complete closed loop is achieved from script parsing, entity extraction, memory reading and writing to video synthesis. This makes the memory mechanism the unified constraint center in the entire video generation process, ensuring the narrative coherence and entity stability of long video generation at the system level.

[0052] This dynamic memory management method for generating consistent multi-camera videos can achieve consistent character appearance, stable prop shapes, and coherent background structures across shots and scenes in practical applications. Multiple examples demonstrate its application effects in different scenarios, and its performance is explained from both qualitative and quantitative perspectives.

[0053] Example 1: Long-term consistent generation of roles across age groups.

[0054] Reference Figure 3 In this example, the input is a script describing a character across multiple age groups (e.g., from 10 to 60 years old). The system first generates shot descriptions for different age groups, and then uses a dynamic memory database to generate a consistent identity reference map for the same character.

[0055] Because the memory bank stores the character's appearance features in their youth, the model can introduce natural age changes, such as wrinkles, hair color changes, and clothing style adjustments, when generating shots of middle-aged and elderly characters while maintaining "identity consistency".

[0056] In the final video sequence, the character maintained significant consistency in facial structural features across multiple shots, solving the problems of character "face swapping" and appearance drift in traditional methods.

[0057] Example 2: Consistent performance of props across different scenes.

[0058] Reference Figure 4 In this embodiment, the script includes an important prop that appears in multiple scenes (such as a book, a red kite, a diary, etc.).

[0059] Traditional methods often result in inconsistencies such as shape changes, color changes, or replacements when props appear across shots. This system uses a prop memory sub-library within a dynamic memory bank to record the prop's complete attributes (color, texture, category, status, etc.) upon its first appearance. The same entry is then retrieved from the memory bank and reused in subsequent shots, ensuring that props maintain a stable appearance under different lighting and environmental conditions.

[0060] In the generated video, the shape, texture, and signature red color of the "prop - kite" remained consistent across multiple shots, without being replaced by other objects by the generated model.

[0061] Example 3: Maintaining consistency of background scene.

[0062] Reference Figure 5 This scene employs a multi-camera storytelling approach, with the "main background scene" appearing frequently, such as the interior of a room, a forest area, and a garage environment. Traditional video generation methods are prone to structural distortion or changes in environmental layout when the background appears continuously, such as furniture shifting, changes in the number of objects, and changes in wall color.

[0063] The background memory sub-library saves the first environmental image as a reference and applies consistency constraints during subsequent shot generation, ensuring the coherence and stability of the entire background environment. Even with significant changes in perspective, the structure remains unchanged. For example, in the "Studio" example, the positions of objects, wall decorations, and lighting directions remain stable across multiple shots.

[0064] The beneficial effects of this multi-camera consistency video generation method based on dynamic memory management are presented from both qualitative and quantitative perspectives.

[0065] Qualitatively, several cases were randomly selected from different types of story scripts (character-driven, prop-driven, and background-driven) to evaluate the performance of this invention in multi-shot videos through visualization results. The invention demonstrated consistency in characters, props, scenes, and natural transitions over time. Character consistency was shown in the stability of facial structure, hairstyle, clothing, and identity features throughout long sequences of shots. Prop consistency was demonstrated in the consistency of shape, texture, and color of recurring key props across multiple shots, without significant shifts. Scene consistency was demonstrated in the stable background scene layout, with furniture or objects not randomly changing or disappearing. Natural transitions over time were demonstrated in the consistent narrative logic reflected in changes in character age and background lighting. In subjective visual evaluation, the generated video showed better overall coherence and readability.

[0066] From a quantitative perspective, evaluation metrics are constructed across three dimensions: character consistency, prop consistency, and background consistency. These are compared on a multi-shot benchmark set to demonstrate advantages in multi-shot consistency. The character consistency metric uses visual feature similarity, embedding and encoding the facial regions of the same character in different shots and calculating feature distances to reflect the consistency of character identity. The prop consistency metric uses object region feature similarity to evaluate the stability of prop appearance across multiple shots. The background consistency metric measures the characteristic changes in background structure between shots to assess whether unreasonable distortions occur in the environmental layout.

[0067] Experiments were conducted in three test scenarios: character-driven, prop-driven, and background-driven. The results are shown in Table 1, which is a comparison table of various video generation methods.

[0068] Table 1. Comparison of the effects of various video generation methods

[0069] The multi-shot consistency video generation method based on dynamic memory management is superior in these metrics, especially when the number of shots increases (e.g., from 4 shots to 12 shots), the relative improvement is more significant, demonstrating the key role of the dynamic memory mechanism in long-term sequence consistency.

[0070] Embodiments of this application provide an electronic device. The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the multi-lens consistent video generation method based on dynamic memory management as described above.

[0071] This electronic device can be any smart terminal, including computers.

[0072] In general, for the hardware structure of electronic devices, the processor can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, to execute relevant programs and implement the technical solutions provided in the embodiments of this application.

[0073] The memory can be implemented in the form of read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory and is called and executed by the processor.

[0074] Input / output interfaces are used to implement information input and output.

[0075] The communication interface is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0076] The bus transmits information between various components of a device, such as the processor, memory, input / output interfaces, and communication interfaces. The processor, memory, input / output interfaces, and communication interfaces communicate with each other within the device via the bus.

[0077] Embodiments of this application provide a computer storage medium. The computer storage medium stores computer-executable instructions for performing the multi-camera consistency video generation method based on dynamic memory management as described above.

[0078] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium. In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0079] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0080] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0081] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0082] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0083] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed between each other may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms. Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0084] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A method for generating multi-camera consistent video based on dynamic memory management, characterized in that, include: Obtain the story script and input the story script into the video generation network; The story script is encoded and features are extracted to obtain semantic information, and the semantic information is mapped into structured shot descriptions; Based on the lens description, entity categories and attribute vectors are extracted; Based on the entity category and the attribute vector, retrieve the matching target entity reference image from the dynamic memory; When the dynamic memory does not have an entity reference image that matches the entity category and the attribute vector, a new entity reference image is generated as the target entity reference image based on the historical image corresponding to the attribute vector and the entity category, and the new entity reference image is written into the dynamic memory. Keyframes are generated based on the shot description and the target entity reference image, and a video is synthesized based on multiple keyframes.

2. The multi-lens consistent video generation method based on dynamic memory management according to claim 1, characterized in that, The video generation network includes a storyboard agent, which encodes and extracts features from the story script to obtain semantic information, and maps the semantic information into a structured shot description. The semantic information includes characters, props, backgrounds, and scene rhythm. The shot description includes information on characters, props, backgrounds, and scene rhythm in a shot.

3. The multi-lens consistent video generation method based on dynamic memory management according to claim 1, characterized in that, The video generation network includes a memory management agent, which is used to extract entity categories and attribute vectors based on the shot description, retrieve matching target entity reference images from the dynamic memory based on the entity categories and attribute vectors, and when the dynamic memory does not contain entity reference images that match the entity categories and attribute vectors, generate new entity reference images as target entity reference images based on historical images corresponding to the attribute vectors and entity categories, and write the new entity reference images into the dynamic memory.

4. The multi-lens consistent video generation method based on dynamic memory management according to claim 1, characterized in that, The video generation network includes a visualization generation agent, which generates keyframes based on the shot description and the target entity reference image, and synthesizes a video based on multiple keyframes.

5. The multi-lens consistent video generation method based on dynamic memory management according to claim 1, characterized in that, The step of generating a new entity reference image as the target entity reference image based on the attribute vector and the historical images corresponding to the entity category includes: When the entity category is not present in the dynamic memory, a first generation condition is constructed based on the attribute vector, and a new entity reference image is generated by an image generation tool based on the first generation condition.

6. The multi-lens consistent video generation method based on dynamic memory management according to claim 1, characterized in that, The step of generating a new entity reference image as the target entity reference image based on the attribute vector and the historical images corresponding to the entity category includes: When the dynamic memory is stored in the entity category but the attribute vector is not present, a second generation condition is constructed based on the attribute vector, and a new entity reference image is generated by an image generation tool based on the second generation condition and the historical image corresponding to the entity category.

7. The multi-lens consistent video generation method based on dynamic memory management according to claim 6, characterized in that, When writing a new entity reference image into the dynamic memory, the historical images of the entity categories corresponding to the new entity reference image are retained.

8. The multi-lens consistent video generation method based on dynamic memory management according to claim 1, characterized in that, The entity reference images of the dynamic memory are stored through explicit dynamic entities.

9. The multi-lens consistent video generation method based on dynamic memory management according to claim 8, characterized in that, The dynamic entity includes entity category, attribute information, and entity reference image.

10. The multi-lens consistent video generation method based on dynamic memory management according to claim 1, characterized in that, The entity reference images in the dynamic memory are stored in categories according to characters, props, and backgrounds.

Images