A relatively controllable video generation system based on AIGC large model

Abstract

The present invention provides a relatively controllable video generation system based on an AIGC large model, which relates to the field of video generation technology and includes: a user input module, the user input module is used to receive the user input of the required video demand; a language construction module, the language construction module is used to generate a video text script based on a large language model according to the user input demand; a graphic generation module, the graphic generation module generates a static image of a video scene according to the video text script; a picture-generated video module, the picture-generated video module is used to convert the static image into a dynamic video shot; a video generation module, the video generation module is used to combine multiple dynamic video shots into a completed video. The relatively controllable video generation system based on the AIGC large model provided by the present invention realizes the rapid generation of videos based on a large language model (AIGC), improves the efficiency of video production, and does not require multiple software platforms to be switched and used.

Description

Technical Field

[0001] The present invention relates to the technical field of video generation, and in particular to a relatively controllable video generation system based on an AIGC large model. Background Art

[0002] Existing video generation technologies often rely on a lot of manual design and editing, especially for complex video content such as commercials, promotional videos or film and television clips, which requires creators to invest a lot of time and energy in script writing, scene design, image generation and other links. Although this method can generate high-quality videos, it is inefficient and costly, especially in scenarios with more personalized needs, and it is difficult to respond quickly. Automated video generation technology based on large language generation models (AIGC) has gradually emerged in recent years.

[0003] However, the video generation technology based on the large language generation model (AIGC) in the existing technology usually requires the collaboration of multiple software platforms, such as a large language model platform, an image processing platform, a video processing platform, etc. During the video production process, it is necessary to switch back and forth between the various software platforms, which is cumbersome to operate. There is a lack of different modules set up on the same platform, which can sequentially complete the generation of video text scripts, generate pictures according to the text scripts, convert the generated pictures into dynamic shots, and then combine multiple dynamic shots into a video generation system for complete video output.

[0004] Therefore, it is necessary to provide a relatively controllable video generation system based on the AIGC large model to solve the above technical problems. Summary of the invention

[0005] The present invention provides a relatively controllable video generation system based on the AIGC large model, which solves the problem that the video generation technology in the prior art usually requires the collaboration of multiple software platforms, and during the video production process, it is necessary to switch back and forth between the various software platforms, which is cumbersome to operate.

[0006] In order to solve the above technical problems, the present invention provides a relatively controllable video generation system based on an AIGC large model, comprising:

[0007] A user input module, the user input module is used to receive a user's input of a required video requirement;

[0008] A language construction module, the language construction module is used to generate a video text script based on a large language model according to user input requirements;

[0009] A graphics generation module, wherein the graphics generation module generates a static image of a video scene according to the video text script;

[0010] A picture-to-video module, the picture-to-video module is used to convert the static image into a dynamic video shot;

[0011] A video generation module, the video generation module is used to combine multiple dynamic video shots into a completed video;

[0012] The construction of the image-generated video module specifically includes the following steps:

[0013] S31, collecting static images and corresponding dynamic videos as training data;

[0014] S32, constructing a dynamic video generation model, wherein the dynamic video generation model includes a generator network structure and a discriminator network structure, wherein the generator network structure is used to receive a static image as input and then output a dynamic image, and the discriminator network structure is used to receive a dynamic image as input and then determine whether the dynamic image sequence is real or generated by the generator network structure;

[0015] S33, training a dynamic video generation model;

[0016] S34. Evaluate, optimize and deploy dynamic video generation models.

[0017] Preferably, the relatively controllable video generation system based on the AIGC large model also includes an audio processing module, and the audio processing module is used to add sound information to the video generated by the video generation module.

[0018] Preferably, the construction of the language construction module specifically includes the following steps:

[0019] S11. Determine the application scenario of the language model;

[0020] S12, collecting text data and preprocessing the collected data;

[0021] S13, selecting a neural network language model;

[0022] S14, training the neural network language model, including dividing the data set, selecting an optimal algorithm and a loss function;

[0023] S15. Evaluate, optimize and deploy the trained neural network language model.

[0024] Preferably, the construction of the graphic generation module specifically includes the following steps:

[0025] S21, collecting data pairs of images and corresponding text descriptions;

[0026] S22, pre-processing the image and text description data respectively;

[0027] S23, selecting a graph generation model;

[0028] S24, inputting the preprocessed image and text description data into the graphics generation model for training;

[0029] S25. Evaluate, optimize, and deploy the trained graph generation model architecture.

[0030] Preferably, the graphics generation module generates one or more static images corresponding to each video scene.

[0031] Preferably, the video generation method of the relatively controllable video generation system based on the AIGC large model comprises the following steps:

[0032] S01, the user input module inputs the required video requirements;

[0033] S02, generating a video text script based on a large language model according to the requirements of the required video, and splitting the video text script into multiple scene segments;

[0034] S03, generating a corresponding static image according to each scene segment;

[0035] S04, converting the static image corresponding to each scene segment into a dynamic video shot;

[0036] S05. Combining the dynamic video shots corresponding to each scene segment into a complete video output.

[0037] Preferably, the relatively controllable video generation system based on the AIGC large model further includes a setting module, the setting module includes a mode setting unit and a role setting unit, the mode setting unit is used to set the type and duration of the required video, when the duration of the set required video exceeds a preset value, the set required video is divided into a plurality of sub-videos, and the relatively controllable video generation system based on the AIGC large model generates one sub-video each time;

[0038] The role setting unit is used to actively set the role information in the required video.

[0039] Preferably, the relatively controllable video generation system based on the AIGC large model further includes a modeling module, and the modeling module is used to construct a role model to provide a role model for the role setting unit.

[0040] Preferably, the relatively controllable video generation system based on the AIGC large model further includes a marking module, which includes a text marking module, a picture marking module and a dynamic image marking module. The text marking module is used to mark the video text script generated by the language construction module, the picture marking module is used to mark the static image generated by the graphic generation module, and the dynamic image marking module is used to mark the dynamic video shot generated by the graphic generation module.

[0041] Wherein, when the user input module inputs similar required video requirements, the marking module reminds the user whether to use the previously marked video text script, and when the user selects no, the language construction module generates a video text script different from the previously marked video text script, and when the user selects yes, the user can select any one of the previously marked video text scripts, or merge multiple previously marked video text scripts to generate a new video text script;

[0042] When the video text script generated by the language construction module has the same theme as the previously marked video text script, or uses the previously marked video text script, the marking module reminds the user whether to use the previously marked static image, and when the user selects no, the graphic generation module generates a new static image, and when the user selects yes, the user can select the previously marked static image, or merge multiple marked static images into a new static image;

[0043] When fusing multiple marked static images, one of the marked static images is selected as the main image, and the other marked static images are selected as auxiliary images, and the elements in the auxiliary image that are different from the main image are added to the main image;

[0044] When the user chooses to use the marked static picture, the marking module prompts whether to use the marked dynamic video shot. When the user chooses no, the image-to-video module generates a different dynamic video shot.

[0045] Compared with the related art, the relatively controllable video generation system based on the AIGC large model provided by the present invention has the following beneficial effects:

[0046] The present invention provides a relatively controllable video generation system based on an AIGC large model. When a video needs to be constructed, some keywords of the video, i.e., requirements for the video, are input into a user input module according to the content of the required video. A language construction module calls the keywords to generate a text script of the video based on the keywords and a large language model (AIGC), and the video text script is divided into a plurality of lens scenes. A graphic generation module calls the video text script to generate corresponding static images corresponding to different lens scenes. Then, a graphic video module retrieves the static image and dynamically processes it to generate a 3-5 second video lens. A plurality of video lenses are combined into a complete video through a video generation module, thereby realizing rapid generation of videos based on a large language model (AIGC), improving the efficiency of video production, and not requiring multiple software platforms to be switched and used. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1A method step diagram of a relatively controllable video generation system based on an AIGC large model provided by the present invention;

[0048] Figure 2 A flowchart of the steps for constructing the language construction module provided by the present invention;

[0049] Figure 3 Provide a flowchart of the steps for constructing a graphics generation module for the present invention;

[0050] Figure 4 A flowchart of the steps for constructing the image-generated video module provided by the present invention;

[0051] Figure 5 A block diagram of the configuration module provided by the present invention;

[0052] Figure 6 A block diagram of the marking module provided by the present invention;

[0053] Figure 7 A flowchart of the steps used by the picture marking module provided by the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] The present invention provides a relatively controllable video generation system based on an AIGC large model.

[0056] Please refer to Figure 1 In one embodiment of the present invention, the relatively controllable video generation system based on the AIGC large model includes:

[0057] A user input module, the user input module is used to receive a user's input of a required video requirement;

[0058] A language construction module, the language construction module is used to generate a video text script based on a large language model according to user input requirements;

[0059] A graphics generation module, wherein the graphics generation module generates a static image of a video scene according to the video text script;

[0060] A picture-to-video module, the picture-to-video module is used to convert the static image into a dynamic video shot;

[0061] A video generation module is used to combine multiple dynamic video shots into a completed video.

[0062] When a video needs to be constructed, some keywords of the video are input into the user input module according to the content of the required video, that is, the requirements for the video. The language construction module calls the keywords to generate a text script of the video based on the keywords and the large language model (AIGC), and divides the video text script into multiple shot scenes. The graphics generation module calls the video text script to generate corresponding static images for different shot scenes, and then the graphic video module calls the static image to dynamically process it to generate a 3-5 second video shot. The video generation module combines multiple video shots into a complete video, thereby realizing rapid video generation based on the large language model (AIGC), improving the efficiency of video production, and eliminating the need to switch between multiple software platforms.

[0063] After the static image is generated, if you are not satisfied with the generated style, you can click to regenerate, or actively modify keywords or modify the video text script to make the image better meet the producer's requirements, thereby making the video generation controllable and better generating the required video.

[0064] In the user input module, the user inputs video requirements in natural language, including the theme, duration, style, scene description, shot description, action details and other information of the video;

[0065] The language construction module generates a video script based on the video requirements, and describes in detail the content arrangement, lens design, screen content, character action background music suggestions, and negative matters to be noted in each scene of the video;

[0066] The graphics generation module combines pre-trained image data to generate visual content that matches the scene description, including elements such as people, scenery, and buildings.

[0067] The image-to-video module automatically adds dynamic effects such as motion tracks, character movements, or camera push-pull to static images to generate dynamic shots.

[0068] As an optional method of this implementation, in actual application of the relatively controllable video generation system based on the AIGC large model, corresponding options are set for each module in the display main interface; when one of the module options is clicked, the page of the corresponding module is entered, and the option position of each module does not change at this time;

[0069] A button for generating the next module is provided in each module page, such as a button for generating a language building module is provided in the user input module;

[0070] When in use, after entering the requirements of the required video in the page corresponding to the user input module, click the generate button of the language construction module, the page will jump directly to the page of the language construction module, and display the generated video text script. The user can revise the video text script on this page. Similarly, when clicking the generate button of the graphic generation module on this page, it will jump to the page of the graphic generation module and automatically generate the corresponding static image. When you are not satisfied with the static image, you can enter the page of the language construction module to modify the video text script, or click to regenerate the corresponding static image; the subsequent graphic video module and video generation module can be operated in the same way.

[0071] In this embodiment, the data connection of multiple modules can be realized by using API calls. Different modules interact with each other through APIs. One module calls the API of another module to obtain data or trigger specific operations, and then performs page jumps or other processing according to the returned results.

[0072] Server-side redirection can also be used. After receiving the request, the server can make a judgment based on the parameters or conditions of the request, and then guide the user to the page of another module by redirecting.

[0073] For page jumps between front-end modules, hyperlinks or JavaScript event processing can be used.

[0074] The relatively controllable video generation system based on the AIGC large model also includes an audio processing module, which is used to add sound information to the video generated by the video generation module.

[0075] The audio processing module can add background music, sound effects, and voice to the video to generate a complete video.

[0076] See also Figure 2 As an optional method of this embodiment, the construction of the language construction module specifically includes the following steps:

[0077] S11. Determine the application scenario of the language model;

[0078] S12, collecting text data and preprocessing the collected data;

[0079] S13, selecting a neural network language model;

[0080] S14, training the neural network language model, including dividing the data set, selecting an optimal algorithm and a loss function;

[0081] S15. Evaluate, optimize and deploy the trained neural network language model.

[0082] In this embodiment, the application scenario of the language model is mainly text generation; the collected text data can come from books, news articles, blogs, social media, web pages and other text sources to ensure the diversity and representativeness of the data, covering different topics, styles and language variants;

[0083] Data preprocessing includes cleaning data, removing noise, errors and irrelevant content, and performing word segmentation, part-of-speech tagging, named entity recognition and other processes to better understand the structure and semantics of the text. Existing natural language processing toolkits such as NLTK can be used for preprocessing.

[0084] The neural network language model can choose Transformer architecture, recurrent neural network (RNN), etc.

[0085] In S14, the training set is used to update the model's parameters, the validation set is used to monitor the model's performance and adjust hyperparameters, and the test set is used to finally evaluate the model's generalization ability. Common optimization algorithms include stochastic gradient descent (SGD), Adam, RMSprop, etc.; the loss function can be cross entropy loss, mean square error loss, etc. depending on the task;

[0086] The specific steps of training a neural network language model include: initializing the parameters of the model; for each training sample, input it into the model, calculate the loss function, and update the parameters of the model through the back-propagation algorithm; batch training can be used to improve training efficiency; regularly evaluate the performance of the model on the validation set, and adjust the hyperparameters such as learning rate, number of layers, number of hidden units, etc. according to the performance;

[0087] In S15, the performance of the model can be evaluated on the test set, using pre-determined evaluation indicators; different model architectures, optimization algorithms, hyperparameter settings, etc. can be tried to improve the performance of the model. The trained model is deployed on the application platform of a relatively controllable video generation system based on the AIGC large model.

[0088] See also Figure 3 As an optional method of this embodiment, the construction of the graphic generation module specifically includes the following steps:

[0089] S21, collecting data pairs of images and corresponding text descriptions;

[0090] S22, pre-processing the image and text description data respectively;

[0091] S23, selecting a graph generation model;

[0092] S24, inputting the preprocessed image and text description data into the graphics generation model for training;

[0093] S25. Evaluate, optimize, and deploy the trained graph generation model architecture.

[0094] In this embodiment, the data pairs of collected images and corresponding text descriptions can be obtained from image datasets (such as ImageNet) and related text description resources;

[0095] In S22, the image is preprocessed, such as resizing and normalizing, so that the model can better process the image data. The text description is preprocessed, such as word segmentation, stop word removal, vectorization, etc., to convert the text into a format suitable for model input;

[0096] The graphics generation model in S23 can be selected from generative adversarial networks (GAN), variational autoencoders (VAE), etc.

[0097] In S24, the preprocessed image and text description data are input into the model for training.

[0098] For GAN, adversarial loss functions can be used to train the generator and discriminator. The goal of the generator is to generate images that are as realistic as possible, making it difficult for the discriminator to distinguish between real images and generated images; the goal of the discriminator is to accurately distinguish between real images and generated images, and by setting the discriminator, the generator can improve the authenticity of the images generated by the generator.

[0099] For VAE, the model can be trained using reconstruction loss and regularization loss. The reconstruction loss measures how similar the generated image is to the original image, while the regularization loss is used to constrain the distribution of the latent variables to have good properties.

[0100] During training, optimization algorithms such as stochastic gradient descent can be used to update the model’s parameters to minimize the loss function.

[0101] Some evaluation software can be used to evaluate the images generated by the model, such as Inception Score, etc.; the model can be adjusted and improved based on the evaluation results; finally, the trained model can be deployed to the application platform of a relatively controllable video generation system based on the AIGC large model.

[0102] See also Figure 4 As an optional method of this embodiment, the construction of the image-generated video module specifically includes the following steps:

[0103] S31, collecting static images and corresponding dynamic videos as training data;

[0104] S32, constructing a dynamic video generation model, wherein the dynamic video generation model includes a generator network structure and a discriminator network structure, wherein the generator network structure is used to receive a static image as input and then output a dynamic image, and the discriminator network structure is used to receive a dynamic image as input and then determine whether the dynamic image sequence is real or generated by the generator network structure;

[0105] S33, training a dynamic video generation model;

[0106] S34. Evaluate, optimize and deploy dynamic video generation models.

[0107] In this embodiment, the training data in S31 may come from video resources on the Internet, such as dynamic images, movies, TV series, animations, etc.;

[0108] The generator in S32 can use convolutional neural network (CNN), recurrent neural network (RNN) or other common structures in generative adversarial network, and the discriminator can also use CNN and other structures;

[0109] During the training process, the generator and the discriminator are constantly competing with each other. The generator tries to generate more realistic dynamic image sequences, while the discriminator strives to improve its ability to distinguish between real and generated images. By continuously adjusting the network parameters, the generator is eventually able to generate high-quality dynamic image sequences.

[0110] Users can evaluate the generated dynamic shots, and then make improvements to the model based on the evaluation, and finally deploy the trained model to the application platform of a relatively controllable video generation system based on the AIGC large model.

[0111] As an optional method of this embodiment, the implementation method of the video generation module specifically includes the following steps: 1. parsing the video file format, including parsing the video file structure and the video encoding format;

[0112] 2. Video decoding and processing, including video decoding, timeline alignment, and graphics and audio processing (such as image cropping, adjusting brightness, contrast, and color);

[0113] 3. Video encoding and output: the processed image and audio data are encoded again to generate the final composite video file and select the output format;

[0114] As an optional manner of this embodiment, the graphic generation module generates one or more static images corresponding to each video scene.

[0115] When a generated video script has many scenes, multiple static images can be generated for different scenes, which are then converted into dynamic shots to form a dynamic video shot. For example, if the shot to be generated is a building built on a mountain and the scenery of the mountain needs to be displayed, an image with the building as the core can be generated, and an image showing the mountain can be generated at the same time. After subsequent synthesis, the scene of the shot moving from the building to the mountain can be realized.

[0116] See also Figure 1 , in this embodiment, the video generation method of the relatively controllable video generation system based on the AIGC large model includes the following steps;

[0117] S01, the user input module inputs the requirements of the required video, including the theme, duration, style, scene description, shot description, action details and other information of the video;

[0118] S02, generating a video text script based on a large language model according to the requirements of the required video, and splitting the video text script into multiple scene segments;

[0119] S03, generating a corresponding static image according to each scene segment;

[0120] S04, converting the static image corresponding to each scene segment into a dynamic video shot;

[0121] S05. Combining the dynamic video shots corresponding to each scene segment into a complete video output.

[0122] In order to further understand the workflow of the video generation system, an example is given, such as generating a video with the theme of "Beijing City Promotional Video";

[0123] First, you can enter "Beijing city promotional video in the user input module. The content includes historical and cultural scenes such as the Forbidden City and the Great Wall, as well as modern buildings such as the CBD and China World Trade Center. It also needs to show the life scenes of citizens. The video style should be cinematic, combining modern and traditional elements, and the duration should be one minute."

[0124] Language building module: Generates scripts based on a large language model, including shot order and content description, for example:

[0125] Shot 1: A panoramic bird’s-eye view of the Forbidden City;

[0126] Shot 2: A distant view of the Great Wall through the morning mist;

[0127] Shot 3: Modern buildings in CBD;

[0128] Shot 4: Citizens doing morning exercises in the park;

[0129] According to the script, static images of relevant scenes are generated, such as the Forbidden City at sunrise, the Great Wall in the clouds, the busy CBD street scene, and citizens doing morning exercises in the park, etc.

[0130] Convert static images into dynamic shots and add corresponding camera movements (such as push-pull shots, drone perspectives) and visual effects;

[0131] Finally, all the shots are synthesized and background music is added (such as soft classical music for traditional scenes, modern electronic music for modern urban scenes) to generate a complete video.

[0132] See also Figure 5 As an optional method of this embodiment, the relatively controllable video generation system based on the AIGC large model further includes a setting module, the setting module includes a mode setting unit and a role setting unit, the mode setting unit is used to set the type and duration of the required video, when the duration of the set required video exceeds a preset value, the set required video is divided into a plurality of sub-videos, and the relatively controllable video generation system based on the AIGC large model generates one sub-video each time;

[0133] The role setting unit is used to actively set the role information in the required video.

[0134] In this embodiment, the mode setting unit is used to select the type of video to be generated, such as advertising, film and television production (such as short dramas, short videos, etc.). When film and television production is selected or when the set video duration is longer (such as 3-10 minutes or more, here when film and television production is selected in the mode setting unit, the default duration is longer than the time, that is, the preset value), the entire video is divided into multiple sub-videos, each sub-video is a short video, and multiple sub-videos constitute a complete series, and the duration of each sub-video is 1-5 minutes. Multiple sub-videos can also be combined later.

[0135] When it is set as a film and television production video, the language construction module will generate a complete story line, and then divide the story line into multiple sub-stories in sequence, and each sub-story corresponds to each sub-video;

[0136] The language construction module processes one sub-story at a time, that is, the sub-story is first divided into multiple shot scenes, the graphic generation module generates corresponding static images from the multiple shot scenes, the graphic video module generates corresponding dynamic shots from the static pictures, and finally the video generation module combines multiple dynamic shots into a completed sub-video;

[0137] Subsequently, other sub-videos are generated in sequence according to the story line, thus forming a multi-episode short play.

[0138] The required video duration can also be input through the user input module. When the video duration is input in this module, it cannot be set to multiple sub-videos.

[0139] Among them, the role setting unit is used to actively set the role information in the video, such as people, animals, etc., so that when the graphic generation module generates the role information, the actively set images of people or animals can be directly used.

[0140] As an optional embodiment of the present invention, the relatively controllable video generation system based on the AIGC large model further includes a modeling module, and the modeling module is used to construct a role model to provide a role model for the role setting unit.

[0141] By setting up the modeling module, you can model or collect different role models in advance, including people, animals, etc., and when the subsequent role setting unit needs to use the role model, you can choose it.

[0142] See also Figure 6 and Figure 7 As an optional method of this embodiment, the relatively controllable video generation system based on the AIGC large model also includes a marking module, which includes a text marking module, a picture marking module and a dynamic image marking module. The text marking module is used to mark the video text script generated by the language construction module, the picture marking module is used to mark the static image generated by the graphic generation module, and the dynamic image marking module is used to mark the dynamic video shot generated by the graphic video module;

[0143] Wherein, when the user input module inputs similar required video requirements, the marking module reminds the user whether to use the previously marked video text script, and when the user selects no, the language construction module generates a video text script different from the previously marked video text script, and when the user selects yes, the user can select any one of the previously marked video text scripts, or merge multiple previously marked video text scripts to generate a new video text script;

[0144] When the video text script generated by the language construction module has the same theme as the previously marked video text script, or uses the previously marked video text script, the marking module reminds the user whether to use the previously marked static image, and when the user selects no, the graphic generation module generates a new static image, and when the user selects yes, the user can select the previously marked static image, or merge multiple marked static images into a new static image;

[0145] When fusing multiple marked static images, one of the marked static images is selected as the main image, and the other marked static images are selected as auxiliary images, and the elements in the auxiliary image that are different from the main image are added to the main image;

[0146] When the user chooses to use the marked static picture, the marking module prompts whether to use the marked dynamic video shot. When the user chooses no, the image-to-video module generates a different dynamic video shot.

[0147] By setting a marking device, the previously generated video text scripts, static pictures and dynamic shots are marked and recorded, so as to avoid generating similar video text scripts, static pictures and dynamic shots to the previous ones and avoid repeated generation;

[0148] At the same time, users can also combine video text scripts and static images marked at different times according to the situation, or process previously marked images with different dynamic effects to form new dynamic image shots, thereby increasing the speed of video generation;

[0149] You can also choose to merge multiple marked video text scripts or multiple static images to form a new video text script or static image for use, thereby increasing the diversity of video text scripts and static images.

[0150] Among them, the marking device classifies and marks according to the type of video demand, so that when the same type of demand is input, the marking device can prompt the user whether to use the previous video text script solution, generated static pictures or dynamic shots; and different types of marks can be sorted by numbers according to time. During normal display, only the digital icon needs to be displayed. When the digital icon is clicked, the corresponding content is displayed.

[0151] In this embodiment, Python language can be used to create a "dictionary" or list to store the marked information. When new content is input, the "dictionary" or list is traversed to find information similar to the input content, and prompts are given.

[0152] Among them, when multiple marked static images are merged into a new image, such as the figure marked 1 is a picture of the Great Wall located in the mountains, the figure marked 2 includes the Great Wall, mountains, sunset, and snow; you can select mark 1 as the main picture and the other two as auxiliary pictures. The newly generated picture includes the Great Wall, mountains, sunset and snow in the main picture. Of course, the positions of the sunset and snow are adjusted according to the shapes of the Great Wall and mountains in the main picture, or a new picture with the elements of the Great Wall, mountains, sunset and snow is generated.

[0153] The fusion of video text scripts with different tags is based on the same principle as the fusion of static images with multiple tags.

[0154] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A relatively controllable video generation system based on AIGC large model, characterized in that: include: A user input module, the user input module is used to receive a user's input of a required video requirement; A language construction module, the language construction module is used to generate a video text script based on a large language model according to user input requirements; A graphics generation module, wherein the graphics generation module generates a static image of a video scene according to the video text script; A picture-to-video module, the picture-to-video module is used to convert the static image into a dynamic video shot; A video generation module, the video generation module is used to combine multiple dynamic video shots into a completed video; The construction of the image-generated video module specifically includes the following steps: S31, collecting static images and corresponding dynamic videos as training data; S32, constructing a dynamic video generation model, wherein the dynamic video generation model includes a generator network structure and a discriminator network structure, wherein the generator network structure is used to receive a static image as input and then output a dynamic image, and the discriminator network structure is used to receive a dynamic image as input and then determine whether the dynamic image sequence is real or generated by the generator network structure; S33, training a dynamic video generation model; S34. Evaluate, optimize and deploy dynamic video generation models; a marking module, the marking module comprising a text marking module, a picture marking module and a dynamic picture marking module, the text marking module is used to mark the video text script generated by the language construction module, the picture marking module is used to mark the static image generated by the graphic generation module, and the dynamic picture marking module is used to mark the dynamic video shot generated by the graphic generation module; When the video text script generated by the language construction module has the same theme as the previously marked video text script, or uses the previously marked video text script, the marking module reminds the user whether to use the previously marked static image. When the user selects no, the graphic generation module generates a new static image. When the user selects yes, the user selects the previously marked static image, or merges multiple marked static images into a new static image. When a plurality of marked static images are fused, one of the marked static images is selected as the main image, and the other marked static images are selected as auxiliary images, and the elements in the auxiliary image that are different from the main image are added to the main image.

2. The relatively controllable video generation system based on the AIGC large model according to claim 1 is characterized in that: The relatively controllable video generation system based on the AIGC large model also includes an audio processing module, which is used to add sound information to the video generated by the video generation module.

3. The relatively controllable video generation system based on the AIGC large model according to claim 1 is characterized in that: The construction of the language construction module specifically includes the following steps: S11. Determine the application scenario of the language model; S12, collecting text data and preprocessing the collected data; S13, selecting a neural network language model; S14, training the neural network language model, including dividing the data set, selecting an optimal algorithm and a loss function; S15. Evaluate, optimize and deploy the trained neural network language model.

4. The relatively controllable video generation system based on the AIGC large model according to claim 1 is characterized in that: The construction of the graphics generation module specifically includes the following steps: S21, collecting data pairs of images and corresponding text descriptions; S22, pre-processing the image and text description data respectively; S23, selecting a graph generation model; S24, inputting the preprocessed image and text description data into the graphics generation model for training; S25. Evaluate, optimize, and deploy the trained graph generation model architecture.

5. The relatively controllable video generation system based on the AIGC large model according to claim 1 is characterized in that: The graphics generation module generates one or more static images corresponding to each video scene.

6. The relatively controllable video generation system based on the AIGC large model according to claim 1 is characterized in that: The video generation method of the relatively controllable video generation system based on the AIGC large model comprises the following steps: S01, the user input module inputs the required video requirements; S02, generating a video text script based on a large language model according to the requirements of the required video, and splitting the video text script into multiple scene segments; S03, generating a corresponding static image according to each scene segment; S04, converting the static image corresponding to each scene segment into a dynamic video shot; S05. Combining the dynamic video shots corresponding to each scene segment into a complete video output.

7. The relatively controllable video generation system based on the AIGC large model according to claim 1 is characterized in that: The relatively controllable video generation system based on the AIGC large model also includes a setting module, the setting module includes a mode setting unit and a role setting unit, the mode setting unit is used to set the type and duration of the required video, when the duration of the set required video exceeds a preset value, the set required video is divided into a plurality of sub-videos, and the relatively controllable video generation system based on the AIGC large model generates one of the sub-videos each time; The role setting unit is used to actively set the role information in the required video.

8. The relatively controllable video generation system based on the AIGC large model according to claim 7 is characterized in that: The relatively controllable video generation system based on the AIGC large model also includes a modeling module, which is used to construct a role model and provide a role model for the role setting unit.

9. The relatively controllable video generation system based on the AIGC large model according to claim 1 is characterized in that: When the user input module inputs similar required video requirements, the marking module reminds the user whether to use the previously marked video text script, and when the user selects no, the language construction module generates a video text script different from the previously marked video text script, and when the user selects yes, the user selects any one of the previously marked video text scripts, or merges multiple previously marked video text scripts to generate a new video text script; When the user chooses to use the marked static picture, the marking module prompts whether to use the marked dynamic video shot. When the user chooses no, the image-to-video module generates a different dynamic video shot.

Images