A video generation method and system

Abstract

The application discloses a video generation method, comprising the following steps: S1, processing creative materials input by a user to output a structured script; S2, processing appearance descriptions of characters in the structured script to generate feature vectors of the corresponding characters, and extracting character information of the corresponding characters in the structured script; S3, generating a basic prompt word based on the structured script, injecting the character information into the basic prompt word to form a shot prompt word used for constraining the appearance descriptions of the characters, and generating a structured shot according to the shot prompt word; S4, generating corresponding video prompt words based on the structured shot, and combining the feature vectors of the corresponding characters and the video prompt words to generate corresponding video clips; and continuously injecting the feature vectors of the corresponding characters to maintain the constraint of the appearance of the characters in the process of generating the video clips according to the continuous structured shot. The appearance of the characters is anchored by the feature vectors, the characters are prevented from drifting in the continuous shots, and the video generation quality is improved.

Description

Technical Field

[0001] This invention relates to the field of multimodal content generation, and more particularly to a video generation method. Background Technology

[0002] Currently, mainstream AI video generation solutions in the industry generally adopt a fragmented operation mode where each production stage is isolated from the others. Typically, a large language model is used to interpret user ideas and generate corresponding scripts, and then storyboard design and single-shot video clip generation are completed one by one. In order to maintain a general uniformity in the appearance of characters, the existing methods mainly rely on repeatedly listing character features in the generation prompts or introducing a single reference image to assist in constraints. However, a unified character control mechanism that can run through the entire process has not been established. The video clips of each shot are generated independently, and there is no correlation or inheritance logic of character features between shots.

[0003] Existing technologies suffer from severe character drift issues in multi-shot generation scenarios. Because each video clip is generated independently, there are no standardized character features that can be reused across shots as generation constraints, nor are there post-verification processes for character visual consistency. When a video contains multiple consecutive shots, the visual features of the same character, such as appearance, color, and body proportions, will gradually shift, ultimately causing character distortion and significantly reducing the quality of the final video. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention discloses a video generation method, comprising: Step S1: Process the creative materials input by the user and output the corresponding structured script; Step S2: Process the character appearance description in the structured script to generate the feature vector of the corresponding character, and extract the character information of the corresponding character in the structured script; Step S3: Generate basic prompts based on the structured script, inject the character information into the basic prompts to form storyboard prompts for constraining the character's appearance description, and generate a structured storyboard based on the storyboard prompts; Step S4: Generate corresponding video prompts based on the structured storyboard, and generate corresponding video clips by combining the feature vectors of the corresponding characters with the video prompts; During the continuous generation of the video clips from the structured storyboard, the feature vectors of the corresponding characters are continuously combined to maintain the constraints on the characters' appearance.

[0005] Preferably, the character information includes a unique character identifier and a character style tag; The structured script includes the character name and character style for each role; Step S2 includes: Step S21: Identify each character and the corresponding character appearance description from the structured script; Step S22: Using a preset multimodal semantic coding model, extract the character image, character color, and character proportion from the character appearance description to construct the feature vector; construct the unique identifier of the corresponding character based on the character name; and construct the character style tag of the corresponding character based on the character style.

[0006] Preferably, step S3 includes: Step S31: Process the structured script using a preset first prompt word model and output the basic prompt words used to describe the storyboard content; Step S32: Concatenate the text corresponding to the character's unique identifier and the text corresponding to the character's style tag with the text corresponding to the basic prompt word to obtain the storyboard prompt word; Step S33: Process the storyboard prompts using a preset storyboard generation model and output the structured storyboard.

[0007] Preferably, the role information includes a role voice identifier; The structured script includes the voice timbre of the corresponding character; Step S4 further includes processing the character timbre identifier using a preset speech model to output the audio of the corresponding character; The process of generating the video clip also includes matching the audio of the corresponding character with the corresponding character in the video clip, and outputting the video clip containing the voice-over of the corresponding character.

[0008] Preferably, step S4 includes: Step S41: Process the structured scene using a preset second prompt word model and output the video prompt words; Step S42: Convert the video prompts into corresponding text vectors; Step S43: Input the text vector and the feature vector into a preset video generation model, and the video generation model outputs the video segment.

[0009] Preferably, after step S4, a step of verifying the video segment is further included: Step A1: Divide the video segment into multiple image frames, and use a preset visual encoder to extract the feature information of the corresponding character in each image frame; Step A2: Convert the feature information into a corresponding real-time character vector, process it to obtain the cosine similarity between the feature vector and the real-time character vector, and determine whether the cosine similarity is less than a preset similarity threshold. If so, mark the corresponding character as a drifting character, regenerate the video segment using the video generation model, and then return to step A1; If not, output the video clip.

[0010] Preferably, the video segment is regenerated using the following steps: Step B1: Process to obtain the dimensionality difference information between the feature vector of the drifting character and the feature information of the drifting character; Step B2: Adjust the generation parameters of the video segment according to the dimensional difference information; Step B3: The video generation model regenerates the corresponding video segment based on the adjusted generation parameters, and then returns to step A1.

[0011] Preferably, the visual encoder is a CLIP visual encoder.

[0012] This invention discloses a video generation system, which is applied to the video generation method; The video generation system includes: The script generation module processes user-input creative materials and outputs structured scripts. A vector generation module, connected to the script generation module, is used to process the character appearance description in the structured script to generate the feature vector of the corresponding character. The information extraction module, connected to the script generation module, is used to extract the role information of the corresponding role in the structured script; The storyboard generation module, connected to the information extraction module, is used to generate basic prompts based on the structured script, inject the character information into the basic prompts to form storyboard prompts for constraining the character's appearance description, and generate structured storyboards based on the storyboard prompts. The video generation module, which connects the vector generation module and the storyboard generation module, is used to generate corresponding video prompts based on the structured storyboard, and to generate corresponding video clips by combining the feature vectors of the corresponding characters and the video prompts.

[0013] The following beneficial effects can be obtained by using the present invention: By constructing a feature vector for each character, the feature vector of the corresponding character is used to constrain the video clip each time a video clip containing that character is generated. By continuously injecting feature vectors during the video clip generation process, character drift in multiple shots is prevented, thereby improving the quality of video generation. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating the video generation method of the present invention; Figure 2 This is a schematic diagram illustrating the process of constructing feature vectors and extracting character information in this invention; Figure 3 This is a schematic diagram of the process for generating structured storyboards in this invention; Figure 4 This is a schematic diagram of the process for generating video clips in this invention; Figure 5 This is a schematic diagram of the process for verifying video segments in this invention; Figure 6 This is a schematic diagram of the process of regenerating video segments in this invention; Figure 7 This is a flowchart illustrating the video generation system of the present invention.

[0015] In the attached diagram: 1. Script generation module, 2. Vector generation module, 3. Information extraction module, 4. Storyboard generation module, 5. Video generation module. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0018] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.

[0019] This invention discloses a video generation method, such as... Figure 1 As shown, it includes: Step S1: Process the creative materials input by the user and output the corresponding structured script; Step S2: Process the character appearance description in the structured script to generate the feature vector of the corresponding character, and extract the character information of the corresponding character in the structured script; Step S3: Generate basic prompts based on structured scripts, inject character information into basic prompts to form storyboard prompts to constrain the description of character appearance, and generate structured storyboards based on storyboard prompts; Step S4: Generate corresponding video prompts based on structured storyboards, and generate corresponding video clips by combining the feature vectors of the corresponding characters with the video prompts; During the continuous generation of video clips from structured storyboards, the feature vectors of the corresponding characters are continuously combined to maintain the constraints on the characters' appearance.

[0020] Specifically, LLM (Large Language Model) is an artificial intelligence model trained on massive amounts of text data and containing billions or even trillions of parameters, capable of understanding, generating, and processing natural language.

[0021] Based on a general LLM model, a model for generating structured scripts is trained on a script dataset. Users input creative materials corresponding to the video they want to generate through user terminals such as web pages, apps, or workbenches, such as a creative sentence: "An orange curly-haired puppy is flying in the air." The script generation LLM identifies and analyzes the corresponding creative materials, and expands the creative materials into structured scripts based on knowledge such as narrative structure, scene rules, and dialogue logic in the script dataset.

[0022] The type of creative material is not limited; it can include creative statements, images, audio, or other types of materials. To process a specific type of material, the corresponding processing layer needs to be built when training the model, and the corresponding type of data needs to be included in the script dataset for training.

[0023] A script is a text outline generated for a video, describing the visuals, sounds, actions, and dialogue to guide the creative execution of all stages, including shooting, editing, dubbing, and special effects. A structured script includes scene numbers, shot descriptions, narration / dialogue, sound effects / background music suggestions, and duration references.

[0024] A pre-defined multimodal semantic coding model is used to process structured scripts, extracting features related to the character's appearance description. These extracted features are then aggregated in the target semantic space to obtain the character's feature vector. The feature vector is then normalized for subsequent calculations.

[0025] Multimodal semantic coding model is a deep learning model that can process multiple different types of data simultaneously, such as text, images, audio, and video. It is used to map information from different modalities to the same semantic space, so that semantically similar content is close in distance within that space.

[0026] Similarly, a multimodal semantic coding model is used to extract character information such as style and voice, which is then saved in text format. A corresponding character archive is built for each character within the database used to store this information. The feature vectors and character information of the same character are stored in the corresponding character archive for subsequent extraction of those information.

[0027] Of course, multimodal semantic coding models can also handle scripts containing images. By preprocessing the image, cropping it to the required size, and then normalizing it, pixel values ​​are converted from [0,255] to [0,1]. The normalized image is then divided into several patches by an image encoder, and each patch is linearly mapped to a vector. These vectors are then processed by a Transformer layer to extract features related to the character's appearance. Based on these extracted features, a feature image is output. The feature image is then aggregated and mapped to the same semantic space to generate a feature vector, which is subsequently normalized.

[0028] Structured storyboarding transforms a structured script into executable visual instructions. Multiple consecutive video clips are generated based on the structured script, each clip corresponding to a structured storyboard. Each structured storyboard includes: 1) Shot number and duration; 2) Shot type, including wide shot, medium shot, close-up, and extreme close-up; 3) Camera movement, such as push / pull / pan / track / follow / circle; 4) Visual content, such as character movements, character appearance, scene setting, and lighting effects; 5) Transition methods; and 6) Corresponding dialogue and sound effects for each character.

[0029] Furthermore, corresponding video clips are generated based on the structured storyboards. Each video clip includes an unlimited number of characters. When generating each video clip, feature vectors are extracted from the corresponding character database based on the characters appearing in that clip and injected into the video generation model to constrain the appearance of the characters in the generated video clip.

[0030] In a preferred embodiment of the present invention, the character information includes a unique character identifier and a character style tag; The structured script includes each character's name and style; like Figure 2 As shown, step S2 includes: Step S21: Identify each character and the corresponding character appearance description from the structured script; Step S22: Use a preset multimodal semantic coding model to extract the character image, character color and character proportion from the character appearance description to construct a feature vector, construct a unique character identifier for the corresponding character based on the character name, and construct a character style label for the corresponding character based on the character style.

[0031] Specifically, a recognition model is trained on a character text-related dataset based on a general LLM model, which is used to identify characters and character appearance descriptions from text.

[0032] Furthermore, the character's appearance description includes the character's image, such as clothing and facial features; the character's color includes clothing color, skin color, and hair color; and the character's proportions include head-to-body ratio and the character's stretching ratio. All of the above are extraction objects of the multimodal semantic coding model. After extraction, each item is mapped to corresponding data and stored in a structured array as a feature vector.

[0033] In structured scripts, character descriptions include not only physical appearance but also character name and style. A character ID (Identifier) ​​is created for each first appearance of the character name. Character style includes the character's personality and emotions when performing a certain action. Personality traits and emotions are labeled with preset numbers, and all labeled numbers are unified and saved as character style tags.

[0034] The character ID, character style tag, and feature vector are all stored in the character's character archive.

[0035] In a preferred embodiment of the present invention, such as Figure 3 As shown, step S3 includes: Step S31: The structured script is processed using a preset first prompt word model to output basic prompt words used to describe the storyboard content; Step S32: Concatenate the text corresponding to the character's unique identifier and the text corresponding to the character's style tag with the text corresponding to the basic prompt words to obtain the storyboard prompt words; Step S33: Use a preset storyboard generation model to process storyboard prompts and output structured storyboards.

[0036] Specifically, a first cue word model is trained on a film and television-related dataset using a general LLM model as the base model. This model processes structured scripts and automatically supplements shot parameters and technical details based on film and television knowledge from the dataset, generating the basic cue words for the storyboard. For each character appearing in the storyboard, character style tags are retrieved from the corresponding character archive using their character IDs. The basic cue words, the retrieved character IDs, and the retrieved character style tags are concatenated to obtain the text corresponding to a storyboard cue word, including various detailed descriptions of the storyboard, camera language, and style descriptions for each character. The storyboard cue words are then input into the storyboard generation model to generate a structured storyboard.

[0037] Specifically, the storyboard generation model constrains the corresponding characters based on character style tags, locking in the character's style characteristics at the data structure level. This ensures that the style characteristics of the same character can be inherited across multiple shots, ultimately generating corresponding structured storyboards. Each structured storyboard is a professional cinematic language that includes elements such as character, shot size, camera movement, lighting, and color.

[0038] In a preferred embodiment of the present invention, the character information includes a character voice identifier; The structured script includes the character's voice timbre for the corresponding role; Step S4 also includes processing the character's voice identifier using a preset voice model to output the corresponding character's audio; The process of generating video clips also includes matching the audio of the corresponding character with the corresponding character in the video clip, and outputting a video clip containing the voice-over of the corresponding character.

[0039] Specifically, a character's voice timbre includes the character's pitch, tone, and manner of speaking. The relevant content of the character's voice timbre is extracted from the structured script using a multimodal semantic coding model and stored in the corresponding character's character archive using the character voice timbre ID.

[0040] When generating video clips, multiple generation models are scheduled through a preset multimodal scheduling engine, such as the video generation model mentioned above, and also include a TTS speech model.

[0041] Specifically, the TTS speech model first performs text normalization, converting non-standard words such as numbers, abbreviations, and symbols in each character's dialogue within the structured storyboard into their corresponding natural language pronunciations. It then performs word segmentation and pronunciation annotation, using a dictionary or machine learning model to label the pronunciation and part of speech for each word, while resolving ambiguity caused by polyphonic characters and selecting the correct pronunciation based on context. Finally, it performs prosodic prediction, automatically marking appropriate pause positions, stress distribution, and intonation trends by analyzing the sentence's grammatical structure and semantic information. For example, it inserts short pause markers within sentences and adds rising intonation markers at the end of interrogative sentences.

[0042] Subsequently, the character dialogue, after text normalization, is fed into an acoustic model for outputting the spectrum. The acoustic model typically employs an encoder-decoder architecture, where the encoder uses a combination of multi-layered convolutional neural networks and recurrent neural networks to extract deep semantic features from the dialogue, converting each phoneme into a high-dimensional vector representation. The decoder introduces an attention mechanism, dynamically determining which parts of the text to focus on when generating audio frames, ensuring precise alignment between phonemes and audio frames. The decoder outputs the Mel spectrum frame by frame as the spectrum of the corresponding dialogue. The horizontal axis of the Mel spectrum represents time, and the vertical axis represents frequency channels; the value in each cell represents the energy intensity of a specific frequency band at that moment.

[0043] When the encoder extracts deep semantic features of a character's dialogue, the TTS speech model retrieves the corresponding timbre ID from the character archive based on the character's character ID, thereby obtaining the character's personalized acoustic attributes such as timbre, pitch range, and formant distribution. These acoustic attributes are represented as vectors and used as input to the decoder. When the decoder aligns phonemes and audio frames, the vectors corresponding to the acoustic attributes serve as constraints, guiding the model to generate a Mel spectrum with the speaker's unique timbre. Simultaneously, the TTS speech model also supports control over the emotional tone of the generated speech. By adding the character's style tag to the decoder's input, the model can adjust prosody, fundamental frequency curve, and pronunciation duration, making the synthesized speech sound as if it is spoken with joy, sadness, or surprise.

[0044] Finally, the vocoder in the TTS speech model restores the Mel spectrum output by the acoustic model to the original audible audio waveform. Modern neural vocoders mostly adopt a generative adversarial network architecture, in which the generator is responsible for mapping the Mel spectrum to the original audio waveform, while the discriminator needs to distinguish between the generated audio and real human voice recordings.

[0045] By matching the audio output by the TTS voice model with the corresponding lines of the character in each video segment using the character ID, the consistency between the character's appearance and voice is improved.

[0046] In a preferred embodiment of the present invention, such as Figure 4 As shown, step S4 includes: Step S41: The structured scene is processed using a preset second prompt word model, and video prompt words are output; Step S42: Convert the video prompts into corresponding text vectors; Step S43: Input the text vector and feature vector into the preset video generation model, and the video generation model outputs a video clip.

[0047] Specifically, the second cue word model is trained using a dataset of cue words corresponding to the structured storyboard, trained on a basic LLM model.

[0048] Furthermore, the video prompts are mapped to a semantic space of the same dimension as the feature vectors to generate corresponding text vectors. These text vectors, along with the feature vectors of all characters appearing in the scene, are input into the video generation model as conditional vectors. By anchoring the appearance of each character within the entire video segment using their feature vector, the model is guided to generate character appearances that conform to the parameters in the feature vector, preventing appearance drift in consecutive shots.

[0049] Video generation models can employ either a diffusion model or a generative model built on the Transformer architecture.

[0050] Specifically, the diffusion model uses a built-in denoising network to predict and denoise the images of the start and end frames of a video clip. Simultaneously, the diffusion model introduces a temporal dimension as a channel to generate a denoised video stream containing multiple frames, ensuring temporal continuity between frames during the denoising process, ultimately outputting a dynamic and coherent video clip.

[0051] The generative model built on the Transformer architecture also incorporates a temporal dimension. The images of the start and end frames are used as the first and last tokens, respectively. Starting from the first token, the model predicts the video frame corresponding to the next token, generating consecutive tokens to create a complete video frame sequence. Text vectors and feature vectors are input into the feature fusion layer of the Transformer architecture. The Transformer's self-attention mechanism captures the temporal dependencies between any two tokens, ensuring the video frame sequence is continuous and logical. This prevents logical breaks that could lead to partial or complete distortion of the image, resulting in changes to the appearance of people or scene drift.

[0052] In a preferred embodiment of the present invention, such as Figure 5 As shown, after step S4, a step of verifying the video segment is also included: Step A1: Divide the video clip into multiple image frames and use a preset visual encoder to extract the feature information of the corresponding character in each image frame; Step A2: Convert the feature information into the corresponding real-time character vector, process it to obtain the cosine similarity between the feature vector and the real-time character vector, and determine whether the cosine similarity is less than a preset similarity threshold. If so, mark the corresponding character as a drifting character, regenerate the video clip using the video generation model, and then return to step A1; If not, output a video clip.

[0053] Specifically, the dimension of the feature information collected by the visual encoder is the same as the dimension of the feature vector, and it is used for subsequent calculations.

[0054] Furthermore, the feature information is converted into real-time character vectors, and cosine similarity is calculated with the feature vectors stored in the character archive. The cosine similarity is then compared with a preset similarity threshold.

[0055] The cosine similarity is calculated using the following formula:

[0056] in, Cosine similarity; A is the feature vector; B is the vector corresponding to the feature information.

[0057] In one embodiment, LPIPS (Learned Perceptual Image Patch Similarity) or FID (Fréchet Inception Distance) can be used.

[0058] LPIPS is used to calculate the similarity between two images, while FID is used to calculate the similarity between two sets of images.

[0059] In a preferred embodiment of the present invention, such as Figure 6 As shown, the video clip is regenerated using the following steps: Step B1: Process the dimensionality difference information between the feature vector of the drifting character and the feature information of the drifting character; Step B2: Adjust the generation parameters of the video clip based on the dimensional difference information; Step B3: The video generation model regenerates the corresponding video segment based on the adjusted generation parameters, and then returns to step A1.

[0060] Specifically, the visual encoder processes video clips containing drifting characters frame by frame, extracting feature information of the drifting characters in each frame. Inter-frame feature mean filtering is used to eliminate the influence of single-frame noise, resulting in an average feature vector to be verified for the drifting clip. This feature vector is then aligned dimension-by-dimensionally with the feature vector of the corresponding character, calculating the relative deviation rate between each dimension. The relative deviation rates for each dimension are stored in the same array, forming a difference vector. Subsequently, the difference vector is mapped to the same semantic space as the feature vector, and each semantic dimension is scored, outputting a differentiated score for each semantic dimension. Finally, the output includes dimensional difference information containing the difference vector, a list of semantic dimensions, and differentiated scores.

[0061] Furthermore, a mapping table is established based on the dimensional difference information and the generation parameters of each dimension in the video generation model. For each dimension, its generation parameters are adjusted according to the differential score. At the same time, the adjusted values ​​of the generation parameters are constrained by a preset adjustment upper limit to prevent over-adjustment from causing image distortion.

[0062] Furthermore, after all generation parameters have been adjusted, the video generation model is used to regenerate the video segment based on the corresponding structured storyboard, and the process returns to step A1 to re-perform the consistency check until the consistency check is passed, at which point the video segment is output and saved.

[0063] In a preferred embodiment of the present invention, the visual encoder is a CLIP visual encoder.

[0064] Specifically, the CLIP visual encoder is used to map text features and image features to the same semantic space, and to align the vectors of text features with the vectors of image features.

[0065] The CLIP visual encoder consists of a similarity calculation layer and a dimensionality difference calculation layer. The similarity calculation layer calculates the similarity between feature vectors and their corresponding vectors, using methods such as cosine similarity, LPIPS, and FID. The dimensionality difference calculation layer calculates and outputs the dimensionality difference information between the feature vectors and their corresponding vectors.

[0066] This invention discloses a video generation system applied to the above-mentioned video generation method; The system processes user-inputted creative statements using a pre-defined large language model and outputs structured scripts. like Figure 7 As shown, the video generation system includes: Script generation module 1 is used to process user-input creative materials and output structured scripts; Vector generation module 2 connects to script generation module 1 and is used to process the character appearance description in the structured script to generate the feature vector of the corresponding character. Information extraction module 3, connected to script generation module 1, is used to extract the role information of the corresponding role in the structured script; Storyboard generation module 4 is connected to information extraction module 3. It is used to generate basic prompt words based on structured scripts, inject character information into basic prompt words to form storyboard prompt words to constrain the character's appearance description, and generate structured storyboards based on storyboard prompt words. The video generation module 5 connects the vector generation module 2 and the storyboard generation module 4. It is used to generate corresponding video prompts based on the structured storyboard, and to generate corresponding video clips by combining the feature vectors of the corresponding characters and the video prompts.

[0067] The above description is merely a preferred embodiment of the present invention and does not limit the implementation and protection scope of the present invention. Those skilled in the art should realize that any equivalent substitutions and obvious changes made based on the description and illustrations of the present invention should be included within the protection scope of the present invention.

Claims

1. A video generation method, characterized in that, include: Step S1: Process the creative materials input by the user and output the corresponding structured script; Step S2: Process the character appearance description in the structured script to generate the feature vector of the corresponding character, and extract the character information of the corresponding character in the structured script; Step S3: Generate basic prompts based on the structured script, inject the character information into the basic prompts to form storyboard prompts for constraining the character's appearance description, and generate a structured storyboard based on the storyboard prompts; Step S4: Generate corresponding video prompts based on the structured storyboard, and generate corresponding video clips by combining the feature vectors of the corresponding characters with the video prompts; During the continuous generation of the video clips from the structured storyboard, the feature vectors of the corresponding characters are continuously combined to maintain the constraints on the characters' appearance.

2. The video generation method according to claim 1, characterized in that, The character information includes a unique character identifier and a character style tag; The structured script includes the character name and character style for each role; Step S2 includes: Step S21: Identify each character and the corresponding character appearance description from the structured script; Step S22: Using a preset multimodal semantic coding model, extract the character image, character color, and character proportion from the character appearance description to construct the feature vector; construct the unique identifier of the corresponding character based on the character name; and construct the character style tag of the corresponding character based on the character style.

3. The video generation method according to claim 2, characterized in that, Step S3 includes: Step S31: Process the structured script using a preset first prompt word model and output the basic prompt words used to describe the storyboard content; Step S32: Concatenate the text corresponding to the character's unique identifier and the text corresponding to the character's style tag with the text corresponding to the basic prompt word to obtain the storyboard prompt word; Step S33: Process the storyboard prompts using a preset storyboard generation model and output the structured storyboard.

4. The video generation method according to claim 2, characterized in that, The character information includes a character voice identifier; The structured script includes the voice timbre of the corresponding character; Step S4 further includes processing the character timbre identifier using a preset speech model to output the audio of the corresponding character; The process of generating the video clip also includes matching the audio of the corresponding character with the corresponding character in the video clip, and outputting the video clip containing the voice-over of the corresponding character.

5. The video generation method according to claim 1, characterized in that, Step S4 includes: Step S41: Process the structured scene using a preset second prompt word model and output the video prompt words; Step S42: Convert the video prompts into corresponding text vectors; Step S43: Input the text vector and the feature vector into a preset video generation model, and the video generation model outputs the video segment.

6. The video generation method according to claim 1, characterized in that, Following step S4, a step of verifying the video segment is also included: Step A1: Divide the video segment into multiple image frames, and use a preset visual encoder to extract the feature information of the corresponding character in each image frame; Step A2: Convert the feature information into a corresponding real-time character vector, process it to obtain the cosine similarity between the feature vector and the real-time character vector, and determine whether the cosine similarity is less than a preset similarity threshold. If so, mark the corresponding character as a drifting character, regenerate the video segment using the video generation model, and then return to step A1; If not, output the video clip.

7. The video generation method according to claim 6, characterized in that, The video clip was regenerated using the following steps: Step B1: Process to obtain the dimensionality difference information between the feature vector of the drifting character and the feature information of the drifting character; Step B2: Adjust the generation parameters of the video segment according to the dimensional difference information; Step B3: The video generation model regenerates the corresponding video segment based on the adjusted generation parameters, and then returns to step A1.

8. The video generation method according to claim 6, characterized in that, The visual encoder used is a CLIP visual encoder.

9. A video generation system, characterized in that, Applied to the video generation method as described in any one of claims 1-8; The video generation system includes: The script generation module processes user-input creative materials and outputs structured scripts. A vector generation module, connected to the script generation module, is used to process the character appearance description in the structured script to generate the feature vector of the corresponding character. The information extraction module, connected to the script generation module, is used to extract the role information of the corresponding role in the structured script; The storyboard generation module, connected to the information extraction module, is used to generate basic prompts based on the structured script, inject the character information into the basic prompts to form storyboard prompts for constraining the character's appearance description, and generate structured storyboards based on the storyboard prompts. The video generation module, which connects the vector generation module and the storyboard generation module, is used to generate corresponding video prompts based on the structured storyboard, and to generate corresponding video clips by combining the feature vectors of the corresponding characters and the video prompts.

Images