Video generation method and device, equipment and storage medium

Abstract

The invention discloses a video generation method and apparatus, a device and a storage medium. The method comprises the steps of obtaining a reference video and an editing prompt word; analyzing the two to generate a structured multi-track scene video script; and generating a target video conforming to the editing prompt according to the script. The multi-track scene video script comprises a reference feature anchor point module which is used for recording a unique identifier of an entity in a video and description information of the entity so as to establish a subject identity benchmark; the event track module is composed of a plurality of event units and is used for associating the relationship between the audio event and the execution entity by referencing the entity identifier so as to describe the audio narrative logic; and the visual track module is composed of a plurality of lens units and is used for associating the split picture with the audio event required to be presented by quoting an event identifier so as to describe a visual presentation mode. The quality and controllability of the generated video in the aspects of subject consistency, shot language accuracy and sound and picture synchronization effect can be improved, and the method can be widely applied to the technical field of videos.

Description

Technical Field

[0001] This application relates to the field of video technology, and in particular to a video generation method, apparatus, device, and storage medium. Background Technology

[0002] As video content creation enters the era of intelligentization, automatically generating video content based on text descriptions using generative artificial intelligence models has become a cutting-edge area of ​​focus in the industry. Currently, related technologies generally involve a subject providing a reference video and editing instructions (such as editing prompts), then using a video understanding task to analyze and understand the reference video to obtain video description information, which is then combined with the content of the editing instructions to generate the target video. In this process, the quality and form of the video description information driving the model largely determine the accuracy and controllability of the target video.

[0003] However, in practical applications, it has been found that the video description information obtained through video understanding tasks is intended to summarize the understanding formed by humans after watching reference videos. The content description has problems of being singular and mixed. When applied to generate target videos, it is difficult to achieve accuracy and ambiguity, resulting in poor quality target videos that often fail to meet the actual creative intent. Summary of the Invention

[0004] This application provides a video generation method, apparatus, device, and storage medium that can improve the quality and controllability of generated videos in terms of subject consistency, accuracy of shot language, and audio-visual synchronization.

[0005] One aspect of this application provides a video generation method, the method comprising:

[0006] Get reference videos and editing tips; The reference video and the editing prompts are parsed to generate a structured multitrack scene video script; Based on the multi-track scene video script, a video is generated to obtain a target video that conforms to the editing prompts; The multi-track scene video script includes a reference feature anchor module, an event track module, and a visual track module. The reference feature anchor module records the entity identifier and description information of at least one entity. The event track module records at least one event unit describing an audio event in the target video and the event identifier of the event unit. The event unit associates the execution entity of the audio event by referencing the entity identifier. The visual track module records at least one shot unit describing a storyboard in the target video. The shot unit associates the audio event presented in the storyboard by referencing the event identifier.

[0007] On the other hand, embodiments of this application provide a video generation apparatus, the apparatus comprising: The acquisition unit is used to acquire reference videos and editing prompts; The parsing unit is used to parse the reference video and the editing prompts to generate a structured multitrack scene video script; An execution unit is used to generate video according to the multi-track scene video script to obtain a target video that conforms to the editing prompts. The multi-track scene video script includes a reference feature anchor module, an event track module, and a visual track module. The reference feature anchor module records the entity identifier and description information of at least one entity. The event track module records at least one event unit describing an audio event in the target video and the event identifier of the event unit. The event unit associates the execution entity of the audio event by referencing the entity identifier. The visual track module records at least one shot unit describing a storyboard in the target video. The shot unit associates the audio event presented in the storyboard by referencing the event identifier.

[0008] Optionally, in some embodiments, the parsing unit is specifically used for: The reference video is parsed to generate a structured initial scene video script; The editing prompt is parsed using natural language processing technology to determine the editing intent corresponding to the editing prompt; The initial scene video script is edited according to the editing intent to obtain the multi-track scene video script.

[0009] Optionally, in some embodiments, the parsing unit is specifically used for: Entity detection is performed on the reference video to identify at least one entity contained in the reference video and the corresponding descriptive information of the entity; Assign an entity identifier to the entity, and construct an initial reference feature anchor module based on the entity identifier and the description information; The reference video is subjected to audio event detection to identify at least one audio event contained in the reference video and the execution entity of the audio event; An event unit is constructed based on the audio event and the execution entity, and an initial event track module is constructed based on the event unit; Perform scene detection on the reference video to identify at least one scene contained in the reference video; Based on the storyboard images, a shot unit is constructed, and based on the shot unit, an initial visual track module is constructed; The initial reference feature anchor point module, the initial event track module, and the initial visual track module are combined to obtain the initial scene video script.

[0010] Optionally, in some embodiments, the parsing unit is specifically used for: The reference video is sampled frame by frame to obtain a sequence of multiple sampled video frames; Entity detection is performed on each of the sampled video frames using an entity detection model to obtain multiple entity detection boxes; Extract feature data of the image content within each entity detection box, and cluster the image content based on the feature data to obtain at least one cluster category; Image content belonging to the same cluster category is identified as an entity, and the descriptive information corresponding to the entity is determined based on the image content corresponding to the entity.

[0011] Optionally, in some embodiments, the parsing unit is specifically used for: Select any feature data as the cluster center to establish a cluster; Calculate the similarity between each feature data to be clustered and the cluster center; wherein, the feature data to be clustered are feature data that have not been added to the cluster. When the similarity between the feature data to be clustered and the cluster center is greater than or equal to a preset threshold, the image content corresponding to the feature data to be clustered is added to the cluster corresponding to the cluster center, and the cluster center position of the cluster is updated. When the similarity between the feature data to be clustered and any of the cluster centers is less than the preset threshold, a new cluster is established using the feature data to be clustered as the cluster center.

[0012] Optionally, in some embodiments, the parsing unit is specifically used for: The feature data of the image content corresponding to the entity is input into the image understanding model, and the image understanding model generates the description information corresponding to the entity. The image understanding model is trained through the following steps: Obtain a training dataset; wherein the training dataset includes multiple sample images and labels corresponding to the sample images, the labels being used to describe the image content of the sample images; Extract the sample feature data of the sample image, input the sample feature data into the image understanding model to be trained, and generate the predicted description information corresponding to the sample image through the image understanding model; The training loss value is determined based on the difference between the predicted description information and the label; The parameters of the image understanding model are updated based on the loss value to obtain a trained image understanding model.

[0013] Optionally, in some embodiments, the parsing unit is specifically used for: The reference video is segmented into audio segments to obtain at least one audio segment and the start and end times of the audio segment. Content recognition is performed on the audio segment to determine the audio event and execution entity corresponding to the audio segment; Extract the event information of the audio event, construct the event unit based on the audio start and end time, the event information, and the execution entity, and assign the event identifier to the event unit.

[0014] Optionally, in some embodiments, the parsing unit is specifically used for: Perform scene detection on the reference video to obtain at least one scene and the start and end times of the video corresponding to the scene. Based on the start and end times of the video and the start and end times of the audio, the target audio event presented in the storyboard is determined; Extract the scene information from the storyboard, construct the shot unit based on the video start and end times, the scene information, and the target audio event, and assign a shot identifier to the shot unit.

[0015] Optionally, in some embodiments, the parsing unit is specifically used for: The description information of the target entity is updated according to the modification instructions for the target entity in the editing intent; Alternatively, the lens unit corresponding to the target lens may be updated according to the modification instructions for the target lens in the editing intent; Alternatively, based on the modification instructions for the new content in the editing intent, a new event unit can be added to the event track module or a new shot unit can be added to the visual track module.

[0016] Optionally, in some embodiments, the parsing unit is specifically used for: Based on the modification instructions for the target entity in the editing intent, the entity identifier, target attribute field, and target attribute value corresponding to the target entity are extracted; Based on the entity identifier corresponding to the target entity, the description information of the target entity is obtained by querying the reference feature anchor module; Set the value of the target attribute field in the description information of the target entity to the target attribute value.

[0017] On the other hand, embodiments of this application provide an electronic device, including a processor and a memory; The memory is used to store computer programs; The processor executes the computer program to implement the aforementioned video generation method.

[0018] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program that is executed by a processor to implement the aforementioned video generation method.

[0019] On the other hand, embodiments of this application also provide a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the aforementioned video generation method.

[0020] The embodiments of this application include at least the following beneficial effects: This application provides a video generation method, apparatus, device, and storage medium, aiming to improve the problem of poor video generation effects in related technologies. This application first obtains a reference video and editing prompts; then, it parses both to generate a structured multi-track scene video script; based on this script, it generates a target video that conforms to the editing prompts. The multi-track scene video script includes three core modules: a reference feature anchor point module, used to record the unique identifiers and descriptive information of entities in the video to establish the subject's identity benchmark; an event track module, composed of several event units, which associates audio events with executing entities by referencing entity identifiers to describe the audio narrative logic; and a visual track module, composed of several shot units, which associates storyboard shots with the audio events to be presented by referencing event identifiers to describe the visual presentation method. By decoupling video content into independent yet interconnected structured data tracks, this application provides precise, unambiguous, and editable control instructions for the video generation model, significantly improving the quality and controllability of the generated video in terms of subject consistency, shot language accuracy, and audio-visual synchronization. Attached Figure Description

[0021] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0022] Figure 1 This is a schematic diagram illustrating the principle of automatically generating videos in related technologies; Figure 2 This is a schematic diagram of a reference video provided in an embodiment of this application; Figure 3 This is a schematic diagram of the implementation environment in which a video generation method provided in this application embodiment is applied; Figure 4 This is a schematic diagram of the business logic of a virtual try-on provided in the embodiments of this application; Figure 5 This is a flowchart illustrating a video generation method provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a multitrack scene video script provided in the embodiments of this application; Figure 7 This is a schematic diagram illustrating the principle of determining entities in a reference video according to an embodiment of this application; Figure 8 This is a schematic diagram illustrating the training principle of an image understanding model provided in an embodiment of this application; Figure 9 This is a schematic diagram of the content of a multi-track scene video script provided in the embodiments of this application; Figure 10 This is a schematic diagram illustrating the principle of generating a multitrack scene video script provided in the embodiments of this application; Figure 11 This is a structural block diagram of a video generation device provided in the embodiments of this application; Figure 12 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0024] It is understood that the terms “first,” “second,” etc., used in this application may be used to describe various concepts herein, but unless otherwise stated, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another.

[0025] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0026] Before providing a further detailed description of the embodiments of this application, the nouns and terms used in the embodiments of this application are explained, and the nouns and terms used in the embodiments of this application shall be interpreted as follows: 1) Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. AI technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, speech processing, natural language processing, machine learning / deep learning, autonomous driving, and intelligent transportation.

[0027] 2) Machine learning is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence. Its applications span all areas of artificial intelligence. Machine learning (deep learning) typically includes techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0028] 3) Computer vision technology is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes for target recognition and measurement, and further processes images to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0029] 4) The diffusion model is a mainstream generative artificial intelligence model. Its principle simulates the diffusion process in physics: First, in the forward process, noise is gradually added to the training data (such as images) to completely destroy it into random noise; then, in the backward process, the model learns how to start from pure noise and gradually recover and reconstruct new samples that conform to the original data distribution through iterative denoising. This model performs well in fields such as image, video, and audio generation.

[0030] 5) J-Cut and L-Cut are audio-visual separation transition techniques used to achieve smooth and natural scene transitions, enhancing the flow of the narrative. J-Cut refers to the audio of the next shot appearing before its visuals; that is, the audience hears the sound of the new scene before seeing its visuals. L-Cut, on the other hand, refers to the audio of the current shot continuing until the beginning of the next shot; that is, the visuals of the current scene have changed, but the sound continues to overlay the new visuals for a short period.

[0031] As video content creation enters the era of intelligence, automatically generating video content based on text descriptions using generative artificial intelligence models has become a cutting-edge area of ​​focus in the industry. Generative AI-based video generation technology aims to automatically create dynamic visual content through simple text commands. This not only signifies a lower barrier to content creation but also brings more possibilities to fields such as personalized media, dynamic advertising, and virtual simulation.

[0032] For example, please refer to Figure 1 , Figure 1 The diagram illustrates the principle of automatically generating video provided in related technologies, such as... Figure 1 As shown, this technical solution typically begins with the object providing a reference video and editing instructions (such as editing prompts). Then, a video understanding model analyzes and understands the reference video, extracting video description information that describes its visual content. This video description information is combined with the content of the editing instructions and input into a video generation model to drive the model to generate a target video that conforms to the object's intent.

[0033] In the aforementioned video generation process, the quality and form of video description information, serving as the core bridge connecting creative intent and the generation model, fundamentally determine the accuracy and controllability of the final generated target video. However, in practical applications, it has been found that the video description information obtained through video understanding tasks is intended to summarize the understanding formed by humans after watching reference videos. The content description suffers from being both singular and mixed, making it difficult to achieve accuracy and unambiguity when applied to generate target videos. This results in poor quality target videos that often fail to meet the actual creative intent.

[0034] For example, please refer to Figure 2 , Figure 2 This illustration shows a schematic diagram of a reference video provided in an embodiment of this application, assuming the current object wants to... Figure 2 The reference video shown is used for creation and editing. The editing instructions can be such as "replace the speaker's black shirt with a blue shirt and give him a pair of frameless glasses", "adjust the style of the entire video from a serious business meeting to a relaxed and enjoyable team discussion", "add a scene at the end of the video: the colleague wearing glasses on the left stands up and asks a question", etc. The specific editing instructions can be flexibly set according to the needs. The above are only for illustrative purposes.

[0035] In the actual generation of the target video, the relevant technologies first use video understanding models to understand and analyze the reference video to obtain video description information. For example, in the process of... Figure 2 When analyzing the reference video, the following video description might be obtained: "The video opens with a medium shot of a man with short brown hair, wearing a black long-sleeved shirt and blue jeans, standing in an office. The company logo is printed on the wall behind him, and next to him is a whiteboard with charts drawn on it. He is actively giving a presentation, using vivid gestures as he speaks. Three colleagues are sitting around a long wooden conference table, listening to the presentation. The man on the far left, wearing glasses and a gray hoodie, is frowning and typing on a laptop with a puzzled expression. In the middle, a woman with long black hair is listening attentively, and another man is sitting next to her."

[0036] However, this type of video description information, as flat natural language text, has inherent limitations and may not be effective in guiding video generation tasks. Analysis reveals that this type of video description information suffers from poor consistency, lack of editing instructions, and difficulty in reflecting audio-visual logic. Specifically, firstly, regarding consistency, it uses vague references such as "man" or "man wearing glasses," which easily leads to identity drift when the video generation model needs to generate target videos containing multiple shots or long narratives. For example, the "standing man" and "listening man" in the description may be rendered as drastically different individuals in different shots of the generated video, severely disrupting narrative coherence. Secondly, this descriptive method lacks effective editing instructions; its content focuses on a passive summary of events rather than providing controllable generation parameters. The video generation model cannot distinguish whether the camera or the subject is moving, resulting in an inability to accurately respond to detailed editing instructions. In addition, video description information is difficult to support flexible audio-visual editing. Its linear narrative mode tightly binds the video's visuals and audio, making it impossible to demonstrate advanced features such as audio-visual separation. For example, it is difficult to instruct the generation of an effect where "the visuals have switched to the listener's reaction, but the audio still retains the speaker's voice," thus limiting its practicality.

[0037] In view of this, this application provides a video generation method, apparatus, device, and storage medium, aiming to improve the problem of poor video generation effects in related technologies. This application first obtains a reference video and editing prompts; then, it parses both to generate a structured multi-track scene video script; based on this script, it generates a target video that conforms to the editing prompts. The multi-track scene video script includes three core modules: a reference feature anchor module, used to record the unique identifiers and descriptive information of entities in the video to establish the subject's identity benchmark; an event track module, composed of several event units, which associates audio events with execution entities by referencing entity identifiers to describe the audio narrative logic; and a visual track module, composed of several shot units, which associates storyboard shots with the audio events to be presented by referencing event identifiers to describe the visual presentation method. By decoupling video content into independent yet interconnected structured data tracks, this application provides precise, unambiguous, and editable control instructions for the video generation model, significantly improving the quality and controllability of the generated video in terms of subject consistency, shot language accuracy, and audio-visual synchronization.

[0038] System architecture and scenario description used in the embodiments of this application Please refer to Figure 3 , Figure 3 The illustration shows an implementation environment in which a video generation method provided in this application is applied, including a terminal device 340, the Internet 330, a gateway 320, a backend server 310, etc.

[0039] In this embodiment, the terminal device 340 may be of a type including, but not limited to, smartphones, tablets, personal computers, smart cameras, video conferencing terminals, and embedded IoT devices. Furthermore, it may be a single device or a collection of multiple devices. The terminal device 340 can communicate with the Internet 330 via wired or wireless means to exchange data.

[0040] A backend server 310 refers to a computer system that can provide certain services to terminal devices 340. Compared to ordinary terminal devices 340, backend servers 310 have higher requirements in terms of stability, security, and performance. A backend server 310 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines).

[0041] Gateway 320, also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator, enabling network interconnection at the transport layer. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent from terminal device 340 to backend server 310 are forwarded to the corresponding backend server 310 via gateway 320. Messages sent from backend server 310 to terminal device 340 are also forwarded to the corresponding terminal device 340 via gateway 320.

[0042] The backend server 310 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0043] The video generation method provided in this application can be executed independently on the backend server 310, independently on the terminal device 340, or based on data interaction between the terminal device 340 and the backend server 310. This application does not impose any restrictions on this.

[0044] For example, when the video generation method provided in this application embodiment is executed independently on the backend server 310, the backend server 310 can directly obtain or receive the reference video and editing prompts uploaded by the object. Utilizing its powerful computing capabilities and integrated high-performance algorithm model, it centrally executes the entire process from parsing and generating a structured multi-track scene video script to rendering the target video. This approach is suitable for scenarios with high requirements for computing resources and generation quality, requiring the processing of long videos or complex editing instructions. It can ensure high-quality generation results and centrally manage video assets.

[0045] For example, when the video generation method provided in this application embodiment is executed independently on the terminal device 340, the terminal device 340 (such as a personal computer or a high-performance mobile device) can utilize its built-in lightweight algorithm model to complete the parsing, script editing, and video generation of the reference video locally. This approach limits all data processing to the local device, eliminates the need for network transmission, and has the advantages of low response latency and protection of the privacy of the original video data. It is suitable for personal creation or offline application scenarios with high requirements for real-time performance and privacy.

[0046] For example, when the video generation method provided in this application embodiment is collaboratively executed based on data interaction between the terminal device 340 and the backend server 310, the terminal device 340 can be responsible for the initial data acquisition, preprocessing, and some lightweight calculations (such as initial entity detection or simple editing), and upload the processed data and editing instructions to the backend server 310. The backend server 310, with its powerful computing resources, performs computationally intensive core steps, such as high-quality structured script generation, complex cross-modal correlation, or high-resolution video rendering. After completing the final video generation, it can send the generated target video or intermediate script results to the terminal device 340 for presentation, storage, or further editing. This collaborative approach effectively balances computational load with real-time performance and quality with terminal resource limitations, making it suitable for hybrid application scenarios such as online video editing platforms and cloud-based collaborative creation.

[0047] Of course, it is understood that the implementation environment corresponding to the method in the embodiments of this application is not limited to that of the implementation environment. Figure 3 As shown, those skilled in the art can flexibly select the specific implementation environment according to actual needs, and this application does not impose any restrictions on this.

[0048] The video generation method provided in this application can be executed in various scenarios, and the following is an exemplary description of this method.

[0049] (a) Virtual try-on scenario The method provided in this application embodiment can be applied in virtual try-on scenarios.

[0050] For example, please refer to Figure 4 , Figure 4 This illustration shows a business logic diagram of a virtual try-on method provided in an embodiment of this application. Currently, in a virtual try-on scenario, the user wants to quickly preview the wearing effect of different outfits based on their own image. Traditional video editing or generation technologies often struggle to ensure a high degree of consistency in the user's identity across different outfits, and cannot achieve precise and independent editing of clothing attributes.

[0051] In a virtual try-on scenario, the video generation method provided in this application embodiment can be applied. The subject can upload a reference video containing themselves (such as a short body-shaping video) and editing prompts (such as changing the pink suit to a red dress). Based on the method provided in this application embodiment, the video content can be parsed and constructed into a structured multi-track scene video script. The subject's image is identified as an entity with a unique entity identifier, and their clothing is recorded as an attribute feature in the description information. Based on the editing prompts, the clothing attribute values ​​of the corresponding entity in the script are accurately located and updated. The video generation model generates the target video based on the edited script. In the video, the subject's facial features, posture, and other identity characteristics remain completely consistent; only the clothing is changed to a red dress as required.

[0052] (ii) Scenes for film and television special effects The method provided in this application embodiment can be applied to film and television special effects scenarios.

[0053] Currently, in film and television special effects production and virtual production workflows, it is often necessary to make complex content modifications, expansions, or stylistic adjustments to the already shot footage according to the script or director's intentions. For example, it may be necessary to replace actors' costumes, change the weather in a scene, add or remove specific props, or insert new character interaction shots in post-production. Traditional workflows heavily rely on manual frame-by-frame processing, which is costly and inefficient.

[0054] The method provided in this application offers an efficient solution to the aforementioned problems. In this scenario, the visual effects production team can use the original footage as a reference video and input the director's or art director's specific requirements (such as "replace the protagonist's modern jacket with a retro leather jacket" or "adjust the daytime scene to a rainy dusk") as editing prompts into the system. The system can then precisely edit the script. For example, it can update the clothing attributes in the protagonist's entity description information or modify the visual style parameters of a specific shot. The video generation model will directly generate a target video that meets the requirements based on the edited script. This simplifies the visual effects preview and shot modification process, reduces reshoot costs, and enables flexible and controllable digital post-production.

[0055] Of course, it is understood that the methods in the embodiments of this application are not limited to the application scenarios given in the above examples. The methods in the embodiments of this application can also be applied in other scenarios, and this application does not impose any restrictions on them.

[0056] General Description of Embodiments in this Application Please refer to Figure 5 , Figure 5 A flowchart illustrating a video generation method provided in an embodiment of this application is shown. Figure 5As shown, a video generation method according to one embodiment of this application includes, but is not limited to, the following steps: Step 510: Obtain reference videos and editing prompts; Step 520: Analyze the reference video and editing prompts to generate a structured multitrack scene video script; Step 530: Generate video based on the multi-track scene video script to obtain the target video that matches the editing prompts; The multi-track scene video script includes a reference feature anchor module, an event track module, and a visual track module. The reference feature anchor module is used to record the entity identifier and description information of at least one entity. The event track module is used to record at least one event unit describing an audio event in the target video and the event identifier of the event unit. The event unit is associated with the execution entity of the audio event by referencing the entity identifier. The visual track module is used to record at least one shot unit describing a storyboard in the target video. The shot unit is associated with the audio event presented in the storyboard by referencing the event identifier.

[0057] In this application embodiment, a video generation method based on structured scripts is provided. The main inventive concept is to construct a generation framework with multi-track scene video scripts as the control core, so as to improve the problems of poor consistency of the main body of the generated video, weak editing control and difficulty in expressing audio-visual logic caused by the use of flat and mixed natural language descriptions in related technologies.

[0058] like Figure 5 As shown, this method first acquires reference videos and editing prompts, and then parses them to generate a structured multi-track scene video script. This script establishes unique identifiers and feature descriptions for entities in the video through a reference feature anchor module, ensuring identity consistency; it organizes audio events into event units through an event track module, and uses entity identifiers to clearly define the executing subject; and it defines scene frames into shot units through a visual track module, and uses event identifiers to establish audio-visual relationships. In this way, the video content can be decoupled into independent yet precisely correlated structured data tracks, providing the video generation model with clear, unambiguous, and editable control instructions. By parsing and executing this script, the video generation model can generate high-quality target videos that conform to the editor's intent in terms of subject identity, camera language, and audio-visual synchronization. This application can significantly improve the controllability, accuracy, and professionalism of video generation.

[0059] The video generation method provided in the embodiments of this application will be introduced and explained below with specific steps.

[0060] In step 510, when performing the video generation task, it is first necessary to obtain the reference video as the basis for creation and the editing prompts that express the creative intent of the subject.

[0061] The reference video serves as the original source material for the video generation process, carrying rich spatiotemporal information such as scenes, entities, actions, audio, and camera language. Editing prompts are specific instructions from the object to the desired generated content, defining the editing requirements that the generated target video should meet. Here, "object" refers to the entity initiating the video generation task; it can be a natural person, organization, or automated program, etc., and this application does not impose any restrictions on this.

[0062] For example, in some embodiments, the reference video may originate from video clips shot by the subject using a smart terminal device, publicly available material downloaded from an online video platform, or video content pre-generated by other content generation models. The editing prompts may be a piece of natural language text, such as "Change the protagonist's black shirt to blue," "Adjust the video style from serious to lighthearted," or "Add a clapping shot at the end of the video," etc. In this application embodiment, there are no restrictions on the specific format, resolution, duration of the reference video, or the specific expression of the editing prompts.

[0063] In this embodiment, the reference video typically comprises one or more consecutive video frames within a spatiotemporal narrative, and these video frames may include related entities. These entities can be core subjects that require identification and consistency maintenance. For example, an entity can be a specific person or animal in the real world, an object with a specific function (such as a product or prop), or a specific scene background. The editing prompts may include referential content about the entity, such as a direct description of its appearance (e.g., "woman in red"), or a description of its role or identity in the reference video (e.g., "speaker," "the first person to appear"), etc.

[0064] Understandably, to ensure the accuracy of subsequent editing, the reference video should contain sufficiently clear and stable images of the entities, such as frontal or side views of a person, in order to accurately extract their features. Of course, this application does not impose specific limitations on the number of entities in the reference video, the complexity of the image, or the degree of environmental interference.

[0065] In this application embodiment, the editing prompt is functionally designed to modify or expand the content, form, etc., of the reference video. The type of editing indicated by the prompt can include various categories. For example, in some embodiments, the editing types indicated by the prompt can include the following: First, attribute editing, which can be used to indicate modifications to the characteristics of an entity itself, such as appearance, clothing, hairstyle, etc. (e.g., "Change the shirt to blue"); second, content editing, which can be used to indicate adding, deleting, or modifying events or actions in the video (e.g., "Add a clapping shot at the end"); third, style editing, which can be used to indicate changes to the overall atmosphere or visual style of the video (e.g., "Adjust the style to relaxed"); and fourth, camera language editing, which can be used to indicate changes to the shooting method of the scene, such as camera movement, shot size, angle, etc. (e.g., "Change the opening shot to a close-up"). In practical applications, an editing prompt can contain one or more editing types, and this application does not impose any limitations on this.

[0066] Having obtained the reference video and editing prompts, the data preparation for the video generation task is complete. The reference video provides the source material and context for generation, while the editing prompts provide the direction and specific requirements for generation. Subsequent steps will use these two types of input to analyze their deep semantics and relationships in a structured manner, driving the video generation model to create a target video that matches the creator's intent.

[0067] It should be specifically noted that, in the specific implementation of the technical solutions described in this application, all processing activities involving personal information (including but not limited to reference videos containing portraits or voices, video description information generated therefrom, and target videos), such as collection, storage, use, processing, transmission, provision, and disclosure, must strictly comply with the requirements of relevant national or regional laws and regulations, national standards, and industry norms. Before implementing processing activities, especially when the reference video involves a clearly identifiable individual, it should be ensured that clear and explicit informed consent has been obtained from the personal information subject, or authorization has been obtained in specific scenarios permitted by law.

[0068] In step 520, after obtaining the reference video and editing prompts, both are parsed to generate a structured multitrack scene video script.

[0069] This step is the core step of the video generation method provided in this application. Its purpose is to transform unstructured reference videos and ambiguous natural language editing intentions (i.e., editing prompts) into a precise, parsable, and editable structured intermediate representation, namely a multi-track scene video script. This transformation process is crucial for improving problems such as mixed descriptive information and poor instruction in related technologies. The parsing here can be a collaborative multimodal understanding and structured construction process, and its output is no longer a single natural language segment, but a script file composed of multiple interrelated data tracks.

[0070] Specifically, in this step, the reference video is parsed to extract discrete, identifiable elements from its image and audio streams. This process may include entity parsing, event parsing, and shot parsing within the reference video.

[0071] Entity resolution is used to identify core elements in the reference video that need to be independently defined and described, such as people and key objects. To this end, in this embodiment, computer vision technologies (such as object detection and face recognition) can be used to identify entities contained in the reference video. Then, a unique entity identifier (such as Person_1, Object_2, etc.) is assigned to each entity, and descriptive information corresponding to the entity is extracted or generated. Here, the descriptive information is used to describe the entity, that is, to explain who (or what) the entity is and what its characteristics are.

[0072] For example, in some embodiments, the descriptive information of an entity may include, but is not limited to, type information, contextual information, source information, and appearance information. Type information is the foundation of the descriptive information; it can be used to indicate whether the entity belongs to a category such as "person," "object," or "scene," defining the scope for subsequent identification and processing. Based on this, contextual information can further describe the role or function of the entity in the narrative of the reference video, for example, indicating that a certain person is a "speaker" rather than an "audience member," thus giving it specific meaning. Source information can be used to describe in which time period and where the entity appears in the reference video. Appearance information can be used to describe the entity's shape, such as face, posture, clothing, hairstyle, and accessories.

[0073] Of course, it is understandable that the descriptive information of an entity is not limited to the above content, and this application does not impose any restrictions on this. For each entity, its corresponding entity identifier and descriptive information together constitute the definition of the entity, which is the basis for ensuring the consistency of the identity of the subject in the video.

[0074] The purpose of event analysis is to identify audio events and key action narratives in a reference video. Specifically, in this embodiment, audio processing technology can be used to segment the continuous audio stream in the reference video into discrete audio events (such as a dialogue or background music), and a corresponding event unit can be established for each audio event. Each event unit is assigned a unique event identifier (such as Dialogue_1, Music_1), and the time range, type, and content text of the audio event are recorded. Simultaneously, the executing entity of the audio event is identified by referencing an entity identifier (e.g., the speaker of Dialogue_1 is Person_1). All event units are arranged chronologically to form an event track module, which clearly defines the auditory narrative line of the reference video.

[0075] The purpose of shot analysis is to identify the visual structure of the reference video. In this embodiment, the reference video can be segmented into continuous shots (storyboard frames) using shot boundary detection technology. A corresponding shot unit is established for each storyboard frame, and a unique shot identifier (such as Shot_1) is assigned to each shot unit. For each storyboard frame, its corresponding image information can be generated (or extracted), and its shot language parameters (such as shot size, angle, and motion mode) can be recorded. Furthermore, in this embodiment, by analyzing the timeline, it can be determined which audio events should be presented synchronously during playback for each shot, thereby establishing references to event identifiers in the shot unit (such as Shot_2 referencing Dialogue_1), achieving flexible configuration of audio and video. All shot units are arranged in chronological order to form a visual track module, which defines the visual presentation line of the video.

[0076] This embodiment also includes parsing the editing prompts, a process that can be implemented using natural language processing technology. Parsing the editing prompts allows for a clear understanding of the object's creative intent and maps it to a specific part of the structured script already constructed. For example, when the editing prompt instructs "change the speaker's black shirt to blue," the corresponding instruction will be parsed as: operation type is attribute editing, the target entity is the entity with the identifier Person_1 (determined through entity recognition and role inference), the attribute information to be modified is clothing, and the value of the target attribute is blue shirt. Subsequently, based on the above instructions, the descriptive information parsed from the reference video can be modified, thereby achieving personalized editing and creation.

[0077] After parsing the reference video and editing prompts, a structured multi-track scene video script can be generated. This script integrates information revised according to the editing intent and includes a reference feature anchor module, an event track module, and a visual track module. This script is no longer a summary of the reference video content, but rather provides the video generation model with information to perform accurate video generation tasks. The multi-track scene video script explicitly addresses three core questions the video generation model needs to know: Who to generate (defined by the reference feature anchor module), When to generate what (defined by the event track module), and How to visually represent it (defined by the visual track module). Furthermore, a referencing mechanism (entity identifiers, event identifiers) ensures strict synchronization and correlation between these three dimensions, providing a foundation for precise and complex video editing and control.

[0078] Specifically, please refer to Figure 6 , Figure 6 This illustration shows a structural diagram of a multitrack scene video script provided in an embodiment of this application, such as... Figure 6 As shown in this embodiment, the multi-track scene video script is a structured data entity comprising three interconnected modules: a reference feature anchor module, an event track module, and a visual track module. The reference feature anchor module, serving as the foundation of the script, records all entities in the video that require unique identification. For example, it assigns a unique "entity identifier 6101" to each entity and associates it with its corresponding "description information A1," thereby establishing a unified identity reference system for the entire script. The event track module organizes the narrative logic of the video chronologically, including multiple event units such as "event unit B1," each with a unique event identifier, such as event unit B1 corresponding to event identifier 6201. Furthermore, event units can explicitly specify the executor of the event by referencing the entity identifier in the reference feature anchor module, thus ensuring accurate association between audio events and executing entities. The visual track module defines the visual presentation sequence of the video, consisting of a series of shot units such as "shot unit C1." Each shot unit not only describes the scene content but also precisely specifies the audio events that need to be presented synchronously within that shot time segment by referencing the event identifier in the event track module.

[0079] In this embodiment, these modules can be tightly coupled through an identifier referencing mechanism, forming an organic whole. This multi-track structure successfully decouples mixed video information into clear, independent, and collaboratively controllable tracks, facilitating the generation of target videos.

[0080] In step 530, based on the obtained multi-track scene video script, a video generation operation can be performed to obtain a target video that conforms to the editing prompts. This step is used to accurately convert the structured, machine-readable multi-track scene video script generated in the previous steps into dynamic visual content that conforms to the editor's intent. Generally, this process can be implemented using a video generation model (such as diffusion models like Sora or Pika, or autoregressive models). The video generation model can achieve the rendering output of the target video by parsing and executing the structured instructions in the script.

[0081] Specifically, when generating a target video based on a multi-track scene video script, the video generation model can load and parse the overall structure of the script. For example, when parsing the reference feature anchor module, the model reads the entity identifier and associated descriptive information for each entity. Identity features in the descriptive information (such as facial features and appearance descriptions) can serve as strong conditions to anchor the core identity of the entity throughout the timeline of the generated target video, ensuring that the person (or other entity) represented by an entity identifier maintains a consistent appearance in all shots, fundamentally improving the identity drift problem. Simultaneously, attribute features in the descriptive information (such as clothing and hairstyle) can be modified using editing prompts, and the video generation model will render according to the final values ​​in the script to meet the creative requirements.

[0082] Furthermore, the video generation model can process the event track module and the visual track module in parallel and synchronize them according to the timeline. For each shot unit in the visual track module, its visual description can be parsed. In the visual description, entity identifiers can be used to define the executing entity (e.g., "[Entity identifier 6101] is giving a speech"), and the model can accurately call the loaded corresponding entity features to generate the image. At the same time, it can also parse the independently defined shot language parameters (such as shot size, camera movement, etc.) in the shot unit, thereby precisely controlling how to shoot, such as achieving a specified push-in shot effect and clearly distinguishing between camera movement and subject movement.

[0083] Furthermore, in this embodiment, the video generation model can retrieve the corresponding event unit from the event track module based on the event identifier referenced in each shot unit. Based on this, the shot and its referenced audio events (such as dialogue or music) can be generated synchronously. This identifier-based mechanism allows the video generation model to easily implement complex editing logic. For example, it can associate a shot showing audience reactions with and generate the speaker's dialogue audio, thereby achieving a J-Cut or L-Cut effect for audio-visual separation, giving the generated target video a superior audio-visual presentation experience.

[0084] In summary, the video generation model can analyze visual descriptions shot by shot, associate entity features, execute cinematic language, and synchronize corresponding audio events. It then combines all generated shot sequences and audio tracks chronologically to render a target video that meets the editor's prompts in terms of subject identity, visual presentation, audio narrative, and their synchronization. In this way, the structured multi-track scene video script accurately conveys the abstract editing intent of the object to the video generation model, achieving highly controllable and professional video generation.

[0085] It is understood that the video generation method provided in this application first obtains a reference video and editing prompts; then, it parses both to generate a structured multi-track scene video script; and finally, it generates a target video that conforms to the editing prompts based on the script. The multi-track scene video script comprises three core modules: a reference feature anchor module, used to record the unique identifiers and descriptive information of entities in the video to establish the subject's identity benchmark; an event track module, composed of several event units, which uses entity identifiers to associate audio events with the executing entities, describing the audio narrative logic; and a visual track module, composed of several shot units, which uses event identifiers to associate storyboard shots with the audio events to be presented, describing the visual presentation. This method, by decoupling video content into independent yet interconnected structured data tracks, provides the video generation model with precise, unambiguous, and editable control instructions, significantly improving the quality and controllability of the generated video in terms of subject consistency, shot language accuracy, and audio-visual synchronization.

[0086] Specifically, in some embodiments, the reference video and editing prompts are parsed to generate a structured multitrack scene video script, including: The reference video is analyzed to generate a structured initial scene video script; The editing prompts are analyzed using natural language processing technology to determine the editing intent corresponding to the prompts; Edit the initial scene video script according to the editing intent to obtain a multi-track scene video script.

[0087] This application provides a video script generation method based on intent parsing and structured editing. The core of this method lies in constructing a transformation architecture from unstructured input to structured processing and executable instructions. Through a cascaded process of "video parsing - intent understanding - script editing," the original reference video and the object's editing prompts are precisely integrated into a multi-track scene video script.

[0088] Specifically, when generating multi-track scene video scripts, an initial scene video script is first generated. Through a series of parallel multimodal analysis techniques (such as computer vision, audio signal processing, and shot analysis), core narrative and visual elements are extracted from the reference video. For example, this process may include: identifying and deduplicating entities such as people and objects in the reference video, creating profiles for them containing unique identifiers and descriptive information; segmenting and identifying audio events (such as dialogue and sound effects) in the video and associating them with corresponding execution entities; detecting shot boundaries and generating a record containing visual descriptions and shooting parameters for each shot. These elements are then systematically organized into an initial scene video script containing reference feature anchor point modules, event track modules, and visual track modules. Understandably, the initial scene video script accurately reflects what the reference video is (entities), what happened (events), and how it is presented (shots).

[0089] In this embodiment, semantic analysis and understanding of the editing prompts provided by the object can also be performed based on natural language processing technology. The goal is to eliminate the ambiguity of natural language, parsing instructions such as "change the speaker's black shirt to blue" or "add a clapping shot at the end" into machine-operable editing intentions. These intentions can be formalized as structured data objects, explicitly including the operation type (e.g., "attribute modification," "content addition"), the operation target (e.g., pointing to a specific entity or shot), and operation parameters (e.g., new attribute values ​​or descriptions of added content). Then, the parsed editing intentions can be used to programmatically modify the aforementioned initial scene video script. This editing operation can be performed directly on the structured data of the script; for example, based on the "attribute modification" editing intention, the description information of relevant entities in the reference feature anchor module can be located and updated. Through this data-driven approach, the object's editing requirements are seamlessly and accurately integrated into the script, resulting in a multi-track scene video script that integrates the reference video content and the object's creative intentions.

[0090] Specifically, in some embodiments, the reference video is parsed to generate a structured initial scene video script, including: Perform entity detection on the reference video to identify at least one entity contained in the reference video and the corresponding descriptive information of the entity; Assign entity identifiers to entities and construct initial reference feature anchor point modules based on entity identifiers and description information; Perform audio event detection on the reference video to identify at least one audio event contained in the reference video and the entity that executes the audio event; Event units are constructed based on audio events and execution entities, and initial event track modules are constructed based on event units; Perform scene detection on the reference video to identify at least one scene contained in the reference video; Based on the storyboard images, construct the shot units, and based on the shot units, construct the initial visual track module; The initial scene video script is obtained by combining the initial reference feature anchor point module, the initial event track module, and the initial visual track module.

[0091] This application provides a method for deep analysis of reference videos to generate structured initial scene video scripts. This method uses parallel multimodal analysis technology to decouple continuous, unstructured video data into multiple logically clear and interconnected independent modules, thereby laying a solid foundation for the accurate editing and generation of subsequent videos.

[0092] Specifically, in this embodiment, the parsing of the reference video can be carried out simultaneously from three dimensions: entities, events, and shots. First, at the entity level, entity detection can be performed on the reference video to identify and distinguish all important visual elements appearing in the video, such as different people and key objects. For each identified entity, not only is its appearance and type descriptive information extracted, but it is also assigned a globally unique entity identifier. This entity identifier becomes the identity information throughout the entire script, facilitating the subsequent differentiation of different entities to ensure the global consistency of the video content. The entity identifiers of all entities and their corresponding descriptive information together constitute the initial reference feature anchor module, which provides data support for solving the identity drift problem.

[0093] At the event dimension, audio event detection can be performed on the reference video to extract and structure the narrative clues. This process can identify discrete audio events from the audio stream, such as a dialogue, background music, or a specific sound effect. Each audio event is precisely labeled with its time range and, through audio-visual association techniques (such as lip-syncing) or speaker recognition, is associated with entities defined in the initial reference feature anchor module. For example, the entity identifier corresponding to the speaker of a dialogue is determined to be PERSON_1. Each audio event and its associated information are encapsulated into an event unit, and all event units arranged in chronological order together construct the initial event track module. This module clearly defines the "auditory timeline" of the video, that is, "when, who said what, or what sound occurred."

[0094] In the visual dimension, the reference video can be segmented into shots to analyze its visual presentation structure. This process uses shot boundary detection technology to divide the video into continuous shot frames (i.e., shots). For each shot, a visual description can be generated, referencing entity identifiers from the initial reference feature anchor module (e.g., "[PERSON_1] is explaining in front of the whiteboard"), ensuring the description is unambiguous. Simultaneously, the shot's language parameters, such as framing, angle, and motion, are recorded. This information for each shot is encapsulated into a shot unit, and all shot units, arranged chronologically, constitute the initial visual track module. This module defines the video's "visual timeline," i.e., "at what time, and how the scene is presented."

[0095] Ultimately, the three independently constructed modules can be integrated together, and precise reference relationships can be established between the modules through identifiers (entity identifiers, event identifiers), thereby forming an organic and structured whole, namely the initial scene video script.

[0096] Specifically, in some embodiments, entity detection is performed on the reference video to identify at least one entity contained in the reference video and its corresponding descriptive information, including: Frame sampling is performed on the reference video to obtain a sequence of multiple sampled video frames; Entity detection is performed on each sampled video frame using an entity detection model to obtain multiple entity detection boxes; Extract feature data of the image content within each entity detection box, and cluster the image content based on the feature data to obtain at least one cluster category; Image content belonging to the same cluster category is identified as an entity, and the descriptive information corresponding to the entity is determined based on the image content corresponding to the entity.

[0097] This application provides an implementation process for performing entity detection on a reference video to determine the entities therein and their corresponding descriptive information. For example, please refer to... Figure 7 , Figure 7 This illustration shows a schematic diagram of the principle for determining entities in a reference video according to an embodiment of this application.

[0098] Specifically, when performing entity detection on the reference video, frame sampling can be performed first to obtain a sequence of multiple sampled video frames. It's easy to understand that the reference video itself is a continuous image stream, and directly processing all frames would be computationally expensive and result in information redundancy. Therefore, in this embodiment, the content of the reference video can be downsampled while preserving the video's temporal information. For example, 1 to 5 frames can be extracted per second to convert the reference video into a representative image sequence, providing input for subsequent image analysis.

[0099] After obtaining the sampled sequence, entity detection can be performed on each sampled video frame to obtain multiple entity detection boxes. In this embodiment, a pre-trained detection model (such as RetinaFace for faces or YOLO for general object detection) can be used to process each sampled video frame. The task of this detection model is to locate and initially classify entities of interest (such as people or specific objects) in the frame, and mark the position of each entity in the form of bounding boxes. For each sampled video frame, the number of detected entity detection boxes can be zero (some sampled video frames may be transition frames and do not include entities), one, two, or more; this application does not impose any restrictions on this.

[0100] It is easy to understand that in the same video, many entities often appear in multiple video frames. Therefore, after detecting multiple entity detection boxes, it is also necessary to determine whether the image content within these entity detection boxes belongs to the same entity, thereby achieving deduplication in entity recognition. Specifically, in the embodiments of this application, feature extraction can be performed on the image content within each entity detection box to obtain corresponding feature data. For example, a feature extraction model (such as ArcFace) can be used to extract a high-dimensional vector as feature data from the image content captured by each entity detection box. This vector is a digital representation of the image content. The features corresponding to the image content of the same entity in different video frames should be very similar, while the features corresponding to different entities should have significant differences.

[0101] After obtaining the feature data of the image content, a clustering algorithm can be used to perform unsupervised clustering of the image content within all entity detection boxes based on the feature data. The clustering algorithm automatically groups image content with similar feature data into the same cluster, thus aggregating image content from multiple different sampled video frames into several cluster categories (i.e., clusters). Each cluster category essentially corresponds to an independent entity instance in the reference video. Therefore, image content belonging to the same cluster category can be identified as an entity, and its descriptive information can be determined based on the image content corresponding to that entity. Here, a unique entity identifier (such as PERSON_1) can be assigned to each cluster category, and the descriptive information of this entity can be generated based on the combined image content belonging to that category.

[0102] It is understood that in this embodiment of the application, a stable entity identity identifier across frames is established through a clustering algorithm, which provides a good data foundation for the video generation model to maintain subject consistency throughout the timeline.

[0103] Specifically, in some embodiments, clustering of image content based on feature data includes: Select any feature data as the cluster center to establish a cluster; Calculate the similarity between each feature data to be clustered and the cluster center; where the feature data to be clustered are the feature data that have not been added to the clusters. When the similarity between the feature data to be clustered and the cluster center is greater than or equal to a preset threshold, the image content corresponding to the feature data to be clustered is added to the cluster corresponding to the cluster center, and the cluster center position of the cluster is updated. When the similarity between the feature data to be clustered and any cluster center is less than a preset threshold, a new cluster is established using the feature data to be clustered as the cluster center.

[0104] In this embodiment, during cluster analysis, firstly, feature data corresponding to any image content can be selected as initial cluster centers to establish clusters. Then, for each image content whose cluster category is not determined, its corresponding feature data is recorded as the feature data to be clustered. Based on the feature data to be clustered, the similarity between the feature data to be clustered and the cluster centers can be determined. Here, similarity is a quantitative indicator used to measure the degree of proximity or similarity between two data in the feature space. Generally, the value of similarity can range from 0 to 1.

[0105] It is understood that, in this embodiment, the similarity between the feature data to be clustered and the cluster centers can reflect the similarity between the image content and the cluster. The greater the similarity, the closer the image content is to the corresponding category of the cluster. Therefore, in this embodiment, a similarity threshold can be preset, denoted as the preset threshold. The specific value of the preset threshold is not limited in this application; for example, it can be set to 0.8.

[0106] The similarity is compared with a preset threshold. If the similarity between the feature data to be clustered and a cluster center is greater than or equal to the preset threshold, it means that the image content corresponding to the feature data to be clustered is sufficiently similar to the image content in the cluster corresponding to the cluster center. Therefore, the image content corresponding to the feature data to be clustered can be added to the cluster corresponding to the cluster center, and the coordinate position of the cluster center is updated once. Conversely, if the similarity between the feature data to be clustered and all the established cluster centers is less than the preset threshold, it means that the image content corresponding to the feature data to be clustered is not similar to the image content in the clusters corresponding to each cluster center. In this case, the image content corresponding to the feature data to be clustered needs to be divided into an additional category, that is, a new cluster is established using the feature data to be clustered as the initial cluster center.

[0107] Through the above streaming clustering process, the image content corresponding to all entity detection boxes in the reference video can be automatically, efficiently and accurately grouped, facilitating the unification of entity identities.

[0108] Specifically, in some embodiments, the descriptive information corresponding to the entity is determined based on the image content corresponding to the entity, including: The feature data of the image content corresponding to the entity is input into the image understanding model, and the image understanding model generates the descriptive information corresponding to the entity. The image understanding model is trained through the following steps: Obtain the training dataset; the training dataset includes multiple sample images and their corresponding labels, whereby the labels describe the image content of the sample images. Extract sample feature data from the sample images, input the sample feature data into the image understanding model to be trained, and generate the predicted description information corresponding to the sample images through the image understanding model; The training loss value is determined based on the difference between the predicted description information and the label; The parameters of the image understanding model are updated based on the loss value to obtain the trained image understanding model.

[0109] In this application embodiment, a specific method is provided for generating high-quality, understandable descriptive information for identified entities. By utilizing a specially trained deep learning model—an image understanding model—the visual features of the entity are automatically converted into a structured text description.

[0110] Specifically, feature data representing the image content of the entity (e.g., an aggregation of features extracted from multiple entity detection boxes belonging to the entity, or feature data of a specific image content corresponding to the entity) can be added as input to a pre-trained image understanding model. This model acts as a vision-to-language translator; its complex internal neural network structure can understand the visual content represented by the input features (such as a person's appearance, clothing, and expression; the shape and color of an object), and generate a fluent and accurate natural language description as the entity's descriptive information. For example, for the entity of a person, the image understanding model might generate a description such as "a man of about 30 years old, wearing black-rimmed glasses, with short hair, and wearing a dark blue suit."

[0111] In this embodiment of the application, the specific model architecture of the image understanding model is not limited, and it can be flexibly set according to actual needs.

[0112] It should be noted that, in the embodiments of this application, the image understanding model needs to be trained before it is officially put into use in order to improve the performance of its output. For example, please refer to... Figure 8 , Figure 8 This illustration shows a schematic diagram of the training principle of an image understanding model provided in an embodiment of this application.

[0113] When training an image understanding model, a training dataset can be obtained, which includes multiple sample images and their corresponding labels. These labels, which can be manually annotated, describe the image content of the sample images, providing detailed and accurate descriptions of the key visual attributes within them. Each sample image and label can then serve as a pair of training data. First, sample feature data can be extracted from the sample images (e.g., using a convolutional neural network as a feature extractor). This sample feature data is then input into the image understanding model to be trained (e.g., a sequence generation model based on an architecture such as Transformer). The image understanding model generates a predicted descriptive message based on the input sample feature data. The core objective of training is to minimize the difference between the model's prediction and the true label. Therefore, the difference between the predicted descriptive message and the manually annotated label can be calculated; this difference is quantified as a loss value to guide the model's parameter updates.

[0114] In the field of artificial intelligence, the accuracy of a model's predictions can be measured by a loss function. A loss function is defined on a single training data point and measures the prediction error of that data point. Specifically, the loss value is determined by the label of that individual training data point and the model's prediction result for that data. However, in actual training, a training dataset contains many data points. Therefore, a cost function is generally used to measure the overall error of the training dataset. The cost function is defined on the entire training dataset and calculates the average prediction error of all training data points, providing a better measure of the model's prediction performance. For general models, the aforementioned cost function, plus a regularization term to measure model complexity, can serve as the training objective function. Based on this objective function, the loss value for the entire training dataset can be calculated. In this embodiment, any loss function can be selected to determine the loss value, thereby updating the model's parameters.

[0115] In this embodiment, the parameter updates of the image understanding model can be performed using an iterative approach. That is, after updating the model parameters once, the updated model is used for prediction to determine a new loss value, and then the model parameters are updated again. This process is repeated until a pre-set training termination condition is met, at which point training is considered complete, and a trained model is obtained.

[0116] The conditions for ending training can be flexibly set according to requirements. For example, in some embodiments, a target number of training iterations can be set as the condition for ending training. When the number of parameter updates in the model reaches the target number of iterations, training can be considered complete. In other embodiments, a threshold difference between the loss values ​​obtained in two adjacent training iterations can be set as the condition for ending training. After updating the model parameters in one iteration, the absolute value of the difference between the loss value obtained in the current training iteration and the loss value obtained in the previous training iteration is calculated. If the absolute value of the difference is greater than the set threshold, iterative training continues; if the absolute value of the difference is less than or equal to the set threshold, training can be considered complete. Of course, the above are only exemplary descriptions of some optional training end condition settings in the embodiments of this application and do not imply any limitation on the actual implementation.

[0117] Specifically, in some embodiments, audio event detection is performed on the reference video to identify at least one audio event contained in the reference video and the execution entity of the audio event; an event unit is constructed based on the audio event and the execution entity, including: Perform audio segmentation on the reference video to obtain at least one audio segment and the start and end times of the corresponding audio segment; Perform content recognition on audio segments to determine the audio events and execution entities corresponding to the audio segments; Extract event information from audio events, construct event units based on audio start and end times, event information, and the executing entity, and assign event identifiers to event units.

[0118] In this application embodiment, an implementation process is provided for structurally parsing the audio stream of a reference video to construct event units.

[0119] Specifically, when constructing event units, the audio of the reference video can be finely segmented and classified. This involves separating the audio track from the reference video and processing it using audio detection algorithms. This process can intelligently identify the natural boundaries and categories of sound content based on the acoustic characteristics of the audio signal (such as spectrum, energy, Mel-frequency cepstral coefficients, etc.). For example, the algorithm can automatically segment continuous audio data into multiple audio segments and determine whether each segment belongs to a category such as "human dialogue," "background music," "ambient sound effects," or "silence," while accurately recording the start and end times of each audio segment.

[0120] Next, for the identified audio segments (especially those containing human voices), deeper content understanding and entity association can be performed. This step is crucial for constructing event units. Through automatic speech recognition technology, speech waveforms can be converted into readable text content. Furthermore, speaker recognition technology can distinguish different speakers within the same dialogue, assigning the same temporary identifier to speech segments from the same speaker. Moreover, by combining the visuals of a reference video (e.g., through lip movement detection, matching the appearance time of a person with their speaking time, and other cross-modal analysis), the speaker identifiers in the audio domain can be corrected and associated with the entity identifiers (such as PERSON_1) already defined in the reference feature anchor module, thereby determining the executing entity of the audio event. For non-human voice audio events (such as music and sound effects), their executing entities may be labeled as "background" or associated with specific visual objects.

[0121] Then, the above information can be integrated and structured into event units. Each event unit is a structured data object, which may include, but is not limited to: the audio's time range (start and end times), the event type (such as DIALOGUE, MUSIC) and specific content (such as dialogue text, music description) obtained from content recognition, and the execution entity determined through cross-modal association (such as entity identifiers pointing to PERSON_1, etc.). In this embodiment, each such event unit can be assigned a unique event identifier (such as EVENT_001), and all event units arranged in chronological order together constitute the event track module.

[0122] Specifically, in some embodiments, a scene detection is performed on the reference video to identify at least one scene contained in the reference video; a shot unit is constructed based on the scene, including: Perform scene detection on the reference video to obtain at least one scene and the start and end times of the video corresponding to the scene. Based on the start and end times of the video and audio, determine the target audio events presented in the storyboard. Extract the image information from the storyboard, construct shot units based on the video start and end times, image information, and target audio events, and assign shot identifiers to shot units.

[0123] This application also provides an implementation process for structurally parsing the visual stream of a reference video to construct shot units. It not only identifies the visual content of the reference video, but more importantly, through timeline alignment technology, it establishes a precise correlation between visual data (shots) and auditory data (audio events), thereby realizing the structuring of audio-visual logic and laying the foundation for generating more expressive videos.

[0124] Specifically, in this embodiment, shot boundary detection technology (such as based on color histograms, edge changes, etc.) is used to process the reference video, automatically identifying the moments of shot transitions in the reference video, thereby segmenting the continuous video stream into a series of visually continuous split-screen shots. For each identified split-screen shot, its corresponding start and end times can be accurately recorded. In this way, the reference video can be transformed from a continuous sequence of frames into discrete narrative units with clear temporal boundaries, completing the basic structuring of visual data.

[0125] Subsequently, the start and end times of the video for each shot are compared with the start and end times of all audio events in the event track module. By calculating the overlap of the time windows, it is possible to accurately determine which audio events are occurring within the duration of the shot. These audio events determined to be occurring are identified as the target audio events to be presented in the shot. For example, a 5-second shot showing the audience listening attentively (video start and end times: 00:01:00-00:01:05) may fall entirely within the audio events of a speaker's dialogue (audio start and end times: 00:00:58-00:01:06), and this dialogue can be identified as the target audio event for the shot. This correlation mechanism is the core of the professional editing (such as audio-visual separation) supported by the embodiments of this application.

[0126] Based on the above process, visual and auditory information can be further integrated to construct structured shot units. Specifically, the image information of the storyboard can be extracted, including visual descriptions generated by the video understanding model that reference entity identifiers, as well as independently defined shot language parameters (such as shot size, angle, and motion). Subsequently, the video start and end times, image information, and event identifiers of the target audio events are encapsulated into a shot unit, and a unique shot identifier (such as SHOT_010) is assigned to this shot unit, thus completing the construction of the shot unit.

[0127] Specifically, in some embodiments, editing operations are performed on the initial scene video script according to the editing intent, including at least one of the following: Update the description information of the target entity according to the modification instructions for the target entity in the editing intent; Alternatively, based on the modification instructions for the target shot in the editing intent, update the lens unit corresponding to the target shot; Alternatively, based on the modification instructions for the new content in the editing intent, add a new event unit in the event track module or a new shot unit in the visual track module.

[0128] This application provides specific implementation methods for editing the initial scene video script according to editing intentions in some embodiments. Editing intentions in the natural language form of objects can be mapped to precise modification operations on specific data objects in the structured script, thereby achieving decoupled and refined control over different dimensions of the generated video content.

[0129] For example, in some embodiments, the type of editing operation can be entity attribute editing. In this case, the description information of the target entity can be updated according to the modification instructions for the target entity in the editing intent. This operation directly acts on the reference feature anchor module. For example, when the intent of the object is to modify the appearance attribute of an entity (such as "changing the speaker's black shirt to blue"), the system will parse the identifier of the target entity (such as PERSON_1), the attribute field to be modified (such as clothing), and the target value (such as blue shirt). Subsequently, the description information corresponding to the entity is located directly in the reference feature anchor module, and the corresponding fields in its attribute feature set are updated. This editing method achieves effects such as "changing clothes," which can change only the variable attributes of the entity while keeping its core identity identifier and identity features unchanged, thereby ensuring that the entity's identity consistency in the video is not affected.

[0130] For example, in some embodiments, the type of editing operation can be shot language editing. In this case, the shot unit corresponding to the target shot can be updated according to the modification instructions for the target shot in the editing intent. This operation operates on the visual track module. For example, when the object's intent is to change the shooting technique of the video (such as "changing the opening shot from a fixed shot to a slow push-in shot"), the system will parse the identifier of the target shot (such as SHOT_1), the shot parameters to be modified (such as movement), and the new parameter values. Subsequently, the corresponding shot unit is found in the visual track module, and its shot parameter set is updated.

[0131] For example, in some embodiments, the type of editing operation can be content expansion editing. In this case, new event units can be added to the event track module or new shot units can be added to the visual track module according to the modification instructions for adding new content in the editing intent. This operation can be used to expand the narrative content of the video. For example, when the object's intent is to add new plot or scenes (such as "add a clapping shot at the end of the video"), the system will not modify the existing data, but will create a new data unit at the specified time position of the corresponding module. Specifically, a new "clapping" sound effect event unit can be created at the end of the event track module, and a new shot unit can be created in the visual track module to describe the clapping scene, and the new scene and the new sound effect can be associated through an active event reference list. In this way, the extension of the video narrative line and the supplementation of the plot can be achieved.

[0132] Specifically, in some embodiments, updating the description information of the target entity according to the modification instructions for the target entity in the editing intent includes: Based on the modification instructions for the target entity in the editing intent, the entity identifier, target attribute field, and target attribute value corresponding to the target entity are extracted; Based on the entity identifier corresponding to the target entity, the description information of the target entity can be obtained by querying the reference feature anchor module; Set the value of the target attribute field in the description information of the target entity to the target attribute value.

[0133] This application provides a specific implementation method for fine-grained editing of entity attributes.

[0134] Specifically, firstly, based on natural language processing technology, deep semantic analysis can be performed on the modification instructions targeting the target entity in the editing intent. Three key operands can be extracted from the modification instructions: first, the entity identifier corresponding to the target entity to be modified (e.g., from "change the color of the speaker's shirt," its pointer to PERSON_1 can be determined through entity linking technology); second, the specific target attribute field to be modified (e.g., from "change the color of the speaker's shirt," the target attribute can be parsed as clothing or the more granular shirt_color); and third, the modified target attribute value (e.g., "blue"). This step transforms unstructured language into parameters that the machine can explicitly execute.

[0135] Next, using the entity identifier extracted in the previous step as a unique primary key, a precise query is performed in the "entity information database" of the reference feature anchor module. The reference feature anchor module is a structured data collection in which each entity is indexed by its unique entity identifier. Through querying, the complete descriptive information of the target entity—the data object that needs to be modified—can be quickly located and obtained.

[0136] After successfully locating the description information of the target entity, the editing operation is performed: the value of the target attribute field in the description information is set to the target attribute value. For example, it will find the clothing field under the attributes set in the description information and directly replace its value from "black shirt" to "blue shirt". Because it only modifies the specific attribute field specified by the instruction, while completely preserving the entity's core identity identifiers (such as PERSON_1) and its unchanging identity features (such as facial features), it can achieve the editing effect while avoiding the risk of identity drift caused by the modification.

[0137] The following describes and explains a video generation method provided in this application embodiment, with reference to specific application examples.

[0138] For example, please refer to Figure 9 , Figure 9 This illustration shows a schematic diagram of the content of a multi-track scene video script provided in an embodiment of this application, such as... Figure 9 As shown, the fields in a multitrack scene video script may include, but are not limited to: Scene_description: Key plot description; Global_style: Overall style description; Global_audio: Background music, ambient music; References: Reference feature anchor point module; Events: Event Track module; Shots: Visual Track Module.

[0139] The reference feature anchor module stores global core information as the cornerstone of consistency, and this information may include, but is not limited to, the following: base_info (basic information): includes ref_id (entity identifier, such as person_1) and type (type, such as person, object, scene, etc.); semantic_description: Semantic description, explaining who this is / what this is / where this is; appearance_anchor: appearance anchor, including source (absolute time and position in the video), identity features (id_features, storing features that are invariant to the subject, such as face, pose, etc., and also including detailed descriptions for adapting to special tasks), and attribute features (attributes, storing features that are variable to the subject, such as clothing, hairstyle, accessories, etc., allowing only the attributes to be modified during generation while maintaining the ID features).

[0140] The event track module is used to describe a series of content events, which may include, but are not limited to, the following information: Basic information includes event_id, absolute start and end time (time_range), and type (e.g., dialogue, action, sound effect).

[0141] Content information: (Visual content related) Dialogue (including narration): time range, speaker, ASR results, audio caption (sub-types such as dialogue / monologue / narration, speaking mood, dialect, speech rate, and key positions are all marked with timestamps); (Music related to visual content): time range, speaker, ASR results, music caption (genre, style, typical techniques and their timestamps, prominent emotions and their timestamps). (Sound effects related to visual content): time range, sound effect type, sound effect caption (sound effect description, usage description).

[0142] Reference mechanism: The event executor is identified by referencing the ref_id in the content field's speaker.

[0143] The visual track module is used to describe discrete camera shots, and it may include, but is not limited to, the following information: Basic information: includes shot_id and absolute start and end times time_range.

[0144] Visual Description: Provides an accurate and detailed description of the storyboard's content, including references (people, objects, scenery, etc.), key actions, character expressions, text information, lighting changes, etc., with timestamps displayed at key locations. It also references IDs from the references (e.g., person_1, object_1, scene_1) to instruct the model to generate a specific character in the current frame.

[0145] Camera language: Independently defined parameters, such as shot size (close-up / medium shot), angle (eye level / low-angle shot), and camera movement (push-pull-pan).

[0146] Event references (Active Events): References the IDs in the Event Track to inform the generative model which events are happening in the current shot (e.g., the screen shows the audience's reaction, but the background is playing the speaker's audio).

[0147] Reference list: Lists the ref_id (entity identifier) ​​that appears in References.

[0148] Please refer to Figure 10 , Figure 10This illustration shows a schematic diagram of the principle of generating multi-track scene video scripts according to an embodiment of this application. The entire generation process is a multi-module collaborative data processing pipeline, with reference videos and editing prompts as inputs and a structured multi-track scene video script as output. This process clearly demonstrates the "WHO-WHAT-HOW" collaborative analysis framework and result fusion mechanism proposed in this application.

[0149] Specifically, the input reference video is first processed in parallel, with three core analysis modules extracting information from different dimensions: WHO module (Protagonist Analysis and Understanding): This module is responsible for visual analysis of the video. Its input is a reference video. Its main task is to identify non-repeating key entities (such as people) in the video through entity detection, feature extraction and clustering (such as using RetinaFace, ArcFace, DBSCAN and other technologies). It generates structured data for each entity, including a unique ref_id and descriptive information (such as generated by a fine-tuned Qwen-VL model). Its output constitutes the cornerstone of the script - the References module.

[0150] The WHAT module (Event Localization and Understanding) is responsible for parsing the audio track of the video. Its input is the audio signal extracted from the reference video. It uses technologies such as audio segmentation (SED), speaker recognition (pyannote), and audio understanding (Qwen-Omni) to identify audio events (such as dialogue and music), determine their time range, content (ASR text), and associate them with the executing entity (speaker). Its output is organized chronologically, forming the narrative thread of the script—the Events module.

[0151] The HOW module (detailed description of visual information): This module is responsible for parsing the visual structure and content of the video, taking a reference video as input. It divides the video into shots using shot segmentation (PyScenedetect) and video understanding (a fine-tuned Qwen-VL model), generating a detailed visual description (ref_id from References) and independent shot language parameters for each shot. Its output constitutes the script's visual presentation plan—the visual track module (Shots).

[0152] Subsequently, the initial outputs from the three modules, along with the editing prompts provided by the object, are input into the result fusion module. This module is driven by a large language model, which performs key integration and editing operations, including: Information Association: The speaker ID output by the WHAT module is corrected and associated with the entity ref_id determined by the WHO module to ensure that the event executor is clear. Script Structuring: The data from the References, Events, and Shots modules are organized according to a predefined script format, and global information (such as scene_description, global_style) is supplemented. Intent Editing: Based on the parsing results of the editing prompts, the initial script is edited, such as updating the attributes of an entity in References, or adding units in Events and Shots. Thus, the result fusion module can output the final multi-track scene video script containing the edited intent.

[0153] The technical solutions provided in this application embodiment have at least the following technical effects: 1. Decoupling and Association: Successfully decouple identity (Who), event (What), and presentation (How), and then accurately associate them through identifiers.

[0154] 2. High-fidelity reproduction and consistency: Through the mechanism of entity identifiers, the problem of inconsistent appearance of characters and objects in different shots during video generation is fundamentally improved, meeting the needs of long video narrative.

[0155] 3. Precise and controllable editing capabilities: It achieves decoupled control over "subject appearance," "subject action," and "camera movement." It allows for independent costume changes, face swaps, dialogue modifications, and camera adjustments without interference.

[0156] 4. Professional-grade editing logic support: Through multi-track design, it supports video generation using professional film editing techniques such as reaction shot, L-Cut, and J-Cut, enhancing the artistic expression and narrative ability of the generated video.

[0157] 5. Structured information is unambiguous: Compared with natural language prompts, structured scripts eliminate the problem of ambiguous referents (such as who "he" refers to), and improve the accuracy of model understanding.

[0158] Reference Figure 11 In this embodiment of the application, a video generation apparatus is also provided, which includes: Acquisition unit 1110 is used to acquire reference videos and editing prompts; The parsing unit 1120 is used to parse the reference video and editing prompts to generate a structured multitrack scene video script; The execution unit 1130 is used to generate video based on the multi-track scene video script to obtain a target video that conforms to the editing prompts. The multi-track scene video script includes a reference feature anchor module, an event track module, and a visual track module. The reference feature anchor module is used to record the entity identifier and description information of at least one entity. The event track module is used to record at least one event unit describing an audio event in the target video and the event identifier of the event unit. The event unit is associated with the execution entity of the audio event by referencing the entity identifier. The visual track module is used to record at least one shot unit describing a storyboard in the target video. The shot unit is associated with the audio event presented in the storyboard by referencing the event identifier.

[0159] Optionally, in some embodiments, the parsing unit is specifically used for: The reference video is analyzed to generate a structured initial scene video script; The editing prompts are analyzed using natural language processing technology to determine the editing intent corresponding to the prompts; Edit the initial scene video script according to the editing intent to obtain a multi-track scene video script.

[0160] Optionally, in some embodiments, the parsing unit is specifically used for: Perform entity detection on the reference video to identify at least one entity contained in the reference video and the corresponding descriptive information of the entity; Assign entity identifiers to entities and construct initial reference feature anchor point modules based on entity identifiers and description information; Perform audio event detection on the reference video to identify at least one audio event contained in the reference video and the entity that executes the audio event; Event units are constructed based on audio events and execution entities, and initial event track modules are constructed based on event units; Perform scene detection on the reference video to identify at least one scene contained in the reference video; Based on the storyboard images, construct the shot units, and based on the shot units, construct the initial visual track module; The initial scene video script is obtained by combining the initial reference feature anchor point module, the initial event track module, and the initial visual track module.

[0161] Optionally, in some embodiments, the parsing unit is specifically used for: Frame sampling is performed on the reference video to obtain a sequence of multiple sampled video frames; Entity detection is performed on each sampled video frame using an entity detection model to obtain multiple entity detection boxes; Extract feature data of the image content within each entity detection box, and cluster the image content based on the feature data to obtain at least one cluster category; Image content belonging to the same cluster category is identified as an entity, and the descriptive information corresponding to the entity is determined based on the image content corresponding to the entity.

[0162] Optionally, in some embodiments, the parsing unit is specifically used for: Select any feature data as the cluster center to establish a cluster; Calculate the similarity between each feature data to be clustered and the cluster center; where the feature data to be clustered are the feature data that have not been added to the clusters. When the similarity between the feature data to be clustered and the cluster center is greater than or equal to a preset threshold, the image content corresponding to the feature data to be clustered is added to the cluster corresponding to the cluster center, and the cluster center position of the cluster is updated. When the similarity between the feature data to be clustered and any cluster center is less than a preset threshold, a new cluster is established using the feature data to be clustered as the cluster center.

[0163] Optionally, in some embodiments, the parsing unit is specifically used for: The feature data of the image content corresponding to the entity is input into the image understanding model, and the image understanding model generates the descriptive information corresponding to the entity. The image understanding model is trained through the following steps: Obtain the training dataset; the training dataset includes multiple sample images and their corresponding labels, whereby the labels describe the image content of the sample images. Extract sample feature data from the sample images, input the sample feature data into the image understanding model to be trained, and generate the predicted description information corresponding to the sample images through the image understanding model; The training loss value is determined based on the difference between the predicted description information and the label; The parameters of the image understanding model are updated based on the loss value to obtain the trained image understanding model.

[0164] Optionally, in some embodiments, the parsing unit is specifically used for: Perform audio segmentation on the reference video to obtain at least one audio segment and the start and end times of the corresponding audio segment; Perform content recognition on audio segments to determine the audio events and execution entities corresponding to the audio segments; Extract event information from audio events, construct event units based on audio start and end times, event information, and the executing entity, and assign event identifiers to event units.

[0165] Optionally, in some embodiments, the parsing unit is specifically used for: Perform scene detection on the reference video to obtain at least one scene and the start and end times of the video corresponding to the scene. Based on the start and end times of the video and audio, determine the target audio events presented in the storyboard. Extract the image information from the storyboard, construct shot units based on the video start and end times, image information, and target audio events, and assign shot identifiers to shot units.

[0166] Optionally, in some embodiments, the parsing unit is specifically used for: Update the description information of the target entity according to the modification instructions for the target entity in the editing intent; Alternatively, based on the modification instructions for the target shot in the editing intent, update the lens unit corresponding to the target shot; Alternatively, based on the modification instructions for the new content in the editing intent, add a new event unit in the event track module or a new shot unit in the visual track module.

[0167] Optionally, in some embodiments, the parsing unit is specifically used for: Based on the modification instructions for the target entity in the editing intent, the entity identifier, target attribute field, and target attribute value corresponding to the target entity are extracted; Based on the entity identifier corresponding to the target entity, the description information of the target entity can be obtained by querying the reference feature anchor module; Set the value of the target attribute field in the description information of the target entity to the target attribute value.

[0168] It is understandable that, such as Figure 5 The content of the video generation method embodiments shown is applicable to the video generation device embodiments. The specific functions implemented by the video generation device embodiments are the same as those shown in the examples. Figure 5 The video generation method shown in the embodiment is the same, and the beneficial effects achieved are the same as those described above. Figure 5 The beneficial effects achieved by the video generation method embodiments shown are also the same.

[0169] This application also discloses an electronic device, including: At least one processor; At least one memory for storing at least one program; When at least one program is executed by at least one processor, such that at least one processor implements as Figure 5 The video generation method embodiment shown is illustrated.

[0170] The electronic device in the embodiments of this application may be a terminal device, a computer device, or a server device.

[0171] For example, taking a server device as an example, refer to Figure 12The server device 1200 can vary significantly due to differences in configuration or performance. It may include one or more central processing units (CPUs) 1210 and memory 1260, and one or more storage media 1230 (e.g., one or more mass storage devices) for storing application programs 1233 or data 1232. The memory 1260 and storage media 1230 may be temporary or persistent storage. The program stored in the storage media 1230 may include one or more units or modules, each unit or module may include a series of operation instructions for the server device 1200. Furthermore, the CPU 1210 may be configured to communicate with the storage media 1230 and execute the series of operation instructions stored in the storage media 1230 on the server device 1200.

[0172] Server device 1200 may also include one or more power supplies 1220, one or more wired or wireless network interfaces 1240, one or more input / output interfaces 1250, and one or more operating systems 1231.

[0173] The central processing unit 1210 in server device 1200 can be used to perform, for example, Figure 5 The video generation method embodiment shown is illustrated.

[0174] This application also discloses a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to implement, for example... Figure 5 The video generation method embodiment shown is illustrated.

[0175] Understandable, Figure 5 The content of the video generation method embodiments shown is applicable to the embodiments of this computer-readable storage medium. The specific functions implemented by the embodiments of this computer-readable storage medium are the same as those shown in the embodiments. Figure 5 The video generation method shown in the embodiment is the same, and the beneficial effects achieved are the same. Figure 5 The beneficial effects achieved by the video generation method embodiments shown are also the same.

[0176] This application also discloses a computer program product or computer program, which includes computer instructions stored in the aforementioned computer-readable storage medium. Figure 12 The processor of the illustrated electronic device can read the computer instructions from the aforementioned computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform... Figure 5 The video generation method embodiment shown is illustrated.

[0177] Understandable, Figure 5 The content of the video generation method embodiments shown is applicable to this computer program product or computer program embodiment, and the specific functions implemented by this computer program product or computer program embodiment are the same as those shown. Figure 5 The video generation method shown in the embodiment is the same, and the beneficial effects achieved are the same. Figure 5 The beneficial effects achieved by the video generation method embodiments shown are also the same.

[0178] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this application are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0179] Furthermore, although this application is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding this application. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional technology for an engineer. Therefore, those skilled in the art can implement the application set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of this application, which is determined by the full scope of the appended claims and their equivalents.

[0180] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0181] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0182] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable storage medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0183] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0184] In the foregoing description of this specification, the references to terms such as "one embodiment," "another embodiment," or "some embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0185] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0186] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A video generation method, characterized in that, The method includes: Get reference videos and editing tips; The reference video and the editing prompts are parsed to generate a structured multitrack scene video script; Based on the multi-track scene video script, a video is generated to obtain a target video that conforms to the editing prompts; The multi-track scene video script includes a reference feature anchor module, an event track module, and a visual track module. The reference feature anchor module records the entity identifier and description information of at least one entity. The event track module records at least one event unit describing an audio event in the target video and the event identifier of the event unit. The event unit associates the execution entity of the audio event by referencing the entity identifier. The visual track module records at least one shot unit describing a storyboard in the target video. The shot unit associates the audio event presented in the storyboard by referencing the event identifier.

2. The video generation method according to claim 1, characterized in that, The step of parsing the reference video and the editing prompts to generate a structured multi-track scene video script includes: The reference video is parsed to generate a structured initial scene video script; The editing prompt is parsed using natural language processing technology to determine the editing intent corresponding to the editing prompt; The initial scene video script is edited according to the editing intent to obtain the multi-track scene video script.

3. The video generation method according to claim 2, characterized in that, The step of parsing the reference video to generate a structured initial scene video script includes: Entity detection is performed on the reference video to identify at least one entity contained in the reference video and the corresponding descriptive information of the entity; Assign an entity identifier to the entity, and construct an initial reference feature anchor module based on the entity identifier and the description information; The reference video is subjected to audio event detection to identify at least one audio event contained in the reference video and the execution entity of the audio event; An event unit is constructed based on the audio event and the execution entity, and an initial event track module is constructed based on the event unit; Perform scene detection on the reference video to identify at least one scene contained in the reference video; Based on the storyboard images, a shot unit is constructed, and based on the shot unit, an initial visual track module is constructed; The initial reference feature anchor point module, the initial event track module, and the initial visual track module are combined to obtain the initial scene video script.

4. The video generation method according to claim 3, characterized in that, The step of performing entity detection on the reference video to identify at least one entity contained in the reference video and the corresponding description information of the entity includes: The reference video is sampled frame by frame to obtain a sequence of multiple sampled video frames; Entity detection is performed on each of the sampled video frames using an entity detection model to obtain multiple entity detection boxes; Extract feature data of the image content within each entity detection box, and cluster the image content based on the feature data to obtain at least one cluster category; Image content belonging to the same cluster category is identified as an entity, and the descriptive information corresponding to the entity is determined based on the image content corresponding to the entity.

5. The video generation method according to claim 4, characterized in that, The step of clustering the image content based on the feature data includes: Select any feature data as the cluster center to establish a cluster; Calculate the similarity between each feature data to be clustered and the cluster center; wherein, the feature data to be clustered are feature data that have not been added to the cluster. When the similarity between the feature data to be clustered and the cluster center is greater than or equal to a preset threshold, the image content corresponding to the feature data to be clustered is added to the cluster corresponding to the cluster center, and the cluster center position of the cluster is updated. When the similarity between the feature data to be clustered and any of the cluster centers is less than the preset threshold, a new cluster is established using the feature data to be clustered as the cluster center.

6. The video generation method according to claim 5, characterized in that, Determining the descriptive information corresponding to the entity based on the image content corresponding to the entity includes: The feature data of the image content corresponding to the entity is input into the image understanding model, and the image understanding model generates the description information corresponding to the entity. The image understanding model is trained through the following steps: Obtain a training dataset; wherein the training dataset includes multiple sample images and labels corresponding to the sample images, the labels being used to describe the image content of the sample images; Extract the sample feature data of the sample image, input the sample feature data into the image understanding model to be trained, and generate the predicted description information corresponding to the sample image through the image understanding model; The training loss value is determined based on the difference between the predicted description information and the label; The parameters of the image understanding model are updated based on the loss value to obtain a trained image understanding model.

7. The video generation method according to claim 3, characterized in that, The step involves performing audio event detection on the reference video to identify at least one audio event contained in the reference video and the entity executing the audio event. Based on the audio event and the execution entity, an event unit is constructed, including: The reference video is segmented into audio segments to obtain at least one audio segment and the start and end times of the audio segment. Content recognition is performed on the audio segment to determine the audio event and execution entity corresponding to the audio segment; Extract the event information of the audio event, construct the event unit based on the audio start and end time, the event information, and the execution entity, and assign the event identifier to the event unit.

8. The video generation method according to claim 7, characterized in that, The step of performing scene detection on the reference video and identifying at least one scene frame contained in the reference video; constructing a shot unit based on the scene frame includes: Perform scene detection on the reference video to obtain at least one scene and the start and end times of the video corresponding to the scene. Based on the start and end times of the video and the start and end times of the audio, the target audio event presented in the storyboard is determined; Extract the scene information from the storyboard, construct the shot unit based on the video start and end times, the scene information, and the target audio event, and assign a shot identifier to the shot unit.

9. The video generation method according to claim 2, characterized in that, The editing operation of the initial scene video script according to the editing intention includes at least one of the following: The description information of the target entity is updated according to the modification instructions for the target entity in the editing intent; Alternatively, the lens unit corresponding to the target lens may be updated according to the modification instructions for the target lens in the editing intent; Alternatively, based on the modification instructions for the new content in the editing intent, a new event unit can be added to the event track module or a new shot unit can be added to the visual track module.

10. The video generation method according to claim 9, characterized in that, The step of updating the description information of the target entity according to the modification instructions for the target entity in the editing intent includes: Based on the modification instructions for the target entity in the editing intent, the entity identifier, target attribute field, and target attribute value corresponding to the target entity are extracted; Based on the entity identifier corresponding to the target entity, the description information of the target entity is obtained by querying the reference feature anchor module; Set the value of the target attribute field in the description information of the target entity to the target attribute value.

11. A video generation apparatus, characterized in that, The device includes: The acquisition unit is used to acquire reference videos and editing prompts; The parsing unit is used to parse the reference video and the editing prompts to generate a structured multitrack scene video script; An execution unit is used to generate video according to the multi-track scene video script to obtain a target video that conforms to the editing prompts. The multi-track scene video script includes a reference feature anchor module, an event track module, and a visual track module. The reference feature anchor module records the entity identifier and description information of at least one entity. The event track module records at least one event unit describing an audio event in the target video and the event identifier of the event unit. The event unit associates the execution entity of the audio event by referencing the entity identifier. The visual track module records at least one shot unit describing a storyboard in the target video. The shot unit associates the audio event presented in the storyboard by referencing the event identifier.

12. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the video generation method according to any one of claims 1 to 10.

13. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the video generation method according to any one of claims 1 to 10.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the video generation method according to any one of claims 1 to 10.

Images