Video generation system and method based on intelligent agent

Abstract

The invention discloses a video generation system and method based on an intelligent agent, and relates to the field of video content creation and editing. According to the invention, various artificial intelligence algorithms are called to process and fuse image information through a plurality of intelligent agent processing units, and the image information is accurately applied to a final video, so that multi-agent cooperation in a general video generation task becomes possible; wide video generation tasks are completed through the intelligent agent, and diversified video creation requirements of users can be met.

Description

Technical Field

[0001] The present invention relates to the field of video content creation and editing, and in particular to a video generation system and method based on intelligent agent. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.

[0003] With the development and application of generative artificial intelligence technology, artificial intelligence video generation algorithms and related artificial intelligence models have become an indispensable and efficient creative workflow in the process of film and television content creation; users can obtain generative artificial intelligence images and video data with corresponding meanings through text descriptions.

[0004] However, a single generative AI model technology can only generate short videos of a few seconds. Even if the model is capable of generating long video data, the generated content is usually unpredictable because the outside world cannot intervene in the generation process. Moreover, the longer the video, the more uncontrollable it is, which has a counterproductive effect. In addition, most of the current generative AI models are based on statistical generation algorithms, and the resulting videos are difficult to accurately express the user's wishes. On the one hand, it is difficult for users to accurately describe the lens in natural language, and on the other hand, the video output results are affected by statistical randomness. It is extremely difficult to generate a final video that meets user needs under a single AI model.

[0005] Therefore, the present invention proposes a video generation technology based on an intelligent agent unit calling a variety of algorithm synthesis. This technology can progressively generate corresponding text, image, and video data at each stage according to user description, thereby enhancing the matching and fidelity of the generated final video. In addition, this technology has wide applicability while ensuring that the generated final video meets user needs. Summary of the invention

[0006] The purpose of the present invention is to: aim at the current single artificial intelligence video generation algorithm and model cannot be directly applied to the generation of long film and television videos, aim at the uncontrollable / intervention of the generation algorithm and the statistical randomness of the output results, provide a video generation system and method based on intelligent agent, by combining a variety of conventional algorithms and artificial intelligence model scheduling, the creative description language text and reference image input by the user are analyzed and reprocessed, and the description information is converted into one or more long videos while accurately expressing the meaning of the user's description; in the generation process, the intelligent agent module automatically introduces various pre-designed conditions, iteratively adjusts and corrects the generated intermediate data, thereby avoiding the randomness and uncontrollable generation results of a single artificial intelligence algorithm. It is more conducive to improving work efficiency in film and video production work, and is conducive to using intelligent computing power for visual creative work.

[0007] The technical solution of the present invention is as follows: A video generation system based on intelligent agent, comprising: A description enhancement proxy processing unit is used to perform various calculation tasks on the shot description information of the target video to obtain a shot description script with richer content; A description generation image agent processing unit generates an image satisfying the storyboard information based on the storyboard description script and serves as a rough sketch of the storyboard, i.e., a storyboard sketch image; An image generation image proxy processing unit is used to intelligently adjust and edit the storyboard sketch image so that the generated storyboard image meets the accuracy of the storyboard and the accuracy of the content; The image generation video proxy processing unit uses the split-shot image as a guide frame to generate a split-shot candidate video clip of a certain length; The video connection / transformation proxy processing unit is used to perform effect transformation processing on the alternative storyboard video clips and connect multiple alternative storyboard video clips into one long video.

[0008] Furthermore, the computing tasks include: text enhancement, style script generation, and reference image preprocessing.

[0009] The present invention also proposes a video generation method based on an intelligent agent, based on the above-mentioned video generation system based on an intelligent agent, comprising: Step S1: Create shot description information of the target video; Step S2: performing multiple computing tasks on the shot description information of the target video to obtain a shot description script with richer content; Step S3: generating an image satisfying the storyboard information based on the storyboard description script and using it as a rough sketch of the storyboard, i.e., a storyboard sketch image; Step S4: intelligently adjusting and editing the storyboard sketch image so that the generated storyboard image meets the accuracy of the storyboard and the accuracy of the content; Step S5: using the storyboard image as a guide frame to generate a storyboard candidate video clip of a certain length; Step S6: performing effect transformation processing on the alternative storyboard video clips and connecting multiple alternative storyboard video clips into one long video.

[0010] Furthermore, the step S2 comprises: Enhance, polish or rewrite the text portion of the storyboard description information to obtain detailed description text suitable for machine learning and artificial intelligence understanding; Identify the style of the storyboard description information and generate a style formatting script; Preprocess the reference image information of the storyboard description information.

[0011] Furthermore, the step S3 comprises: Based on the storyboard description script, an image or image sequence that satisfies the description meaning is generated by describing a generation algorithm automatically selected by an image generation proxy processing unit, which is also called a storyboard sketch image; The process of generating the image proxy processing unit in the scheduling algorithm includes: multiple scheduling of different neural network models to generate a series of image sequences that meet the story sequence and consistent algorithm scheduling for the role; The characters include but are not limited to human or anthropomorphic images, and scene object images.

[0012] Furthermore, the step S4 comprises: Through the image generation image proxy module, the storyboard sketch image is further intelligently edited; through the scheduling of the image generation large model, portrait synthesis algorithm, text synthesis algorithm and various basic image processing algorithms, the storyboard sketch image is further processed to obtain a high-precision and high-quality storyboard starting image.

[0013] Furthermore, the step S5 comprises: Based on the starting image of the storyboard, the image-generated video proxy module is used to analyze and generate a series of video frame images. In this process, the image-generated video proxy module schedules different intelligent algorithms to continuously simulate the body and movements of the characters in the image, identify and simulate the physical phenomena of the scene in the image, and continuously transform the overall style of the image, thereby obtaining storyboard alternative video clips of a certain length that are suitable for the storyboard description.

[0014] Furthermore, the step S6 comprises: Based on the alternative video clips of the storyboard, various audio, video and image data editing algorithms are scheduled through the video connection / transformation proxy processing unit to merge and process the alternative video clips of the storyboard, and automatic adaptation is performed during the storyboard merging process to obtain film and television content that meets the storyboard description and output the final video.

[0015] Compared with the prior art, the present invention has the following beneficial effects: In the process of generating, each intelligent agent module automatically introduces various pre-designed conditions, and iteratively adjusts and corrects the generated intermediate data, thereby avoiding the randomness and uncontrollable generation results of a single artificial intelligence algorithm. It is more conducive to improving work efficiency in video production and using intelligent computing power for visual creative work. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1It is a flowchart of a long video generation method based on intelligent agent; Figure 2a A schematic diagram for generating an initial storyboard image; Figure 2b A schematic diagram for generating the second initial image of the storyboard; Figure 2c Schematic diagram for generating the initial image of the storyboard; Figure 2d Four schematic diagrams for generating the initial image of the storyboard; Figure 3 This is a schematic diagram of the adjusted and enhanced storyboard image; Figure 4 Generate short video schematics for storyboard images; Figure 5 This is a schematic diagram of storyboard video style transfer; Figure 6 A schematic diagram of the result is shown in FIG. DETAILED DESCRIPTION

[0017] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0018] The features and performance of the present invention are further described in detail below in conjunction with the embodiments.

[0019] Embodiment 1 See also Figure 1 , a video generation system based on intelligent agent, specifically including the following units: A description enhancement proxy processing unit is used to perform computing tasks such as text enhancement, style script generation, and reference image preprocessing on the shot description information of the target video to obtain a shot description script with richer content; The description-generated image proxy processing unit generates an image that satisfies the storyboard information based on the storyboard description script, and serves as a rough sketch of the storyboard, i.e., a storyboard sketch image. It should be noted that the description-generated image proxy processing unit is a multimodal model architecture innovatively designed based on the text-generated image model in the image generation model algorithm. An image generation image proxy processing unit is used to intelligently adjust and edit the storyboard sketch image so that the generated storyboard image meets the accuracy of the storyboard and the accuracy of the content; The image generation video proxy processing unit uses the split-shot image as a guide frame to generate a split-shot candidate video clip of a certain length; The video connection / transformation proxy processing unit is used to perform effect transformation processing on the alternative storyboard video clips and connect multiple alternative storyboard video clips into one long video.

[0020] Based on the above-mentioned video generation system based on intelligent agent, this embodiment further proposes a video generation method based on intelligent agent, which specifically includes the following steps: Step S1: Create shot description information of the target video; Step S2: Performing computing tasks such as text enhancement, style script generation, and reference image preprocessing on the shot description information of the target video to obtain a shot description script with richer content, so that the description information is more suitable for machine and algorithm processing; Step S3: generating an image satisfying the storyboard information based on the storyboard description script and using it as a rough sketch of the storyboard, i.e., a storyboard sketch image; Step S4: intelligently adjusting and editing the storyboard sketch image so that the generated storyboard image meets the accuracy of the storyboard and the accuracy of the content; Step S5: using the storyboard image as a guide frame to generate a storyboard candidate video clip of a certain length; Step S6: performing effect transformation processing on the alternative storyboard video clips and connecting multiple alternative storyboard video clips into one long video.

[0021] In this embodiment, specifically, step S2 includes: Enhance, polish or rewrite the text portion of the storyboard description information to obtain detailed description text suitable for machine learning and artificial intelligence understanding; Identify the style of the storyboard description information and generate a style formatting script; Preprocess the reference image information of the storyboard description information.

[0022] In this embodiment, specifically, step S3 includes: Based on the storyboard description script, an image or image sequence that satisfies the description meaning is generated by describing a generation algorithm automatically selected by an image generation proxy processing unit, which is also called a storyboard sketch image; The process of generating the image proxy processing unit in the scheduling algorithm includes: multiple scheduling of different neural network models to generate a series of image sequences that meet the story sequence and consistent algorithm scheduling for the role; The characters include but are not limited to human or anthropomorphic images, and scene object images.

[0023] In this embodiment, specifically, step S4 includes: Through the image generation image proxy module, the storyboard sketch image is further intelligently edited; through the scheduling of the image generation large model, portrait synthesis algorithm, text synthesis algorithm and various basic image processing algorithms, the storyboard sketch image is further processed to obtain a high-precision and high-quality storyboard starting image.

[0024] In this embodiment, specifically, step S5 includes: Based on the starting image of the storyboard, the image-generated video proxy module is used to analyze and generate a series of video frame images. In this process, the image-generated video proxy module continuously simulates the actions of the body, mouth shape, and expression of the characters in the image by scheduling different intelligent algorithms, identifies and simulates the physical phenomena of the scene in the image, and continuously transforms the overall style of the image, thereby obtaining storyboard alternative video clips of a certain length that are suitable for the storyboard description.

[0025] In this embodiment, specifically, step S6 includes: Based on the alternative video clips of the storyboard, various audio, video and image data editing algorithms are scheduled through the video connection / transformation proxy processing unit to merge and process the alternative video clips of the storyboard, and automatic adaptation is performed during the storyboard merging process to obtain film and television content such as video sound effects, subtitles and transition special effects that meet the storyboard description, and output the final video.

[0026] Embodiment 2 This embodiment is a specific application of a video generation method based on an intelligent agent proposed in Implementation 1, and specifically includes the following steps: Step S1: Create shot description information for the shot to be decomposed into a long video; in this embodiment, specifically: a beautiful model stands in front of the camera, she is a future warrior, standing in a future city. It should be noted that the shot description text can be a natural language text, or a formatted text information with special marks or codes, or a reference information with images. These marks and format information can be used to emphasize the degree of appearance of certain specific content elements or prohibit their appearance, and can also emphasize the specific image of the content elements in the shot (such as a human face), as well as the element movement information.

[0027] Step S2: Based on the split-shot description information, the description enhancement proxy processing unit analyzes the split-shot description information, makes intelligent judgments, and automatically schedules the underlying image algorithm or artificial intelligence model to process the information. For example, the description information is semantically enriched by the artificial intelligence language large model (LLM), or formatted to adapt to the content element generation strength description specifications in certain scenarios. In this embodiment, the split-shot content description formatted text is as follows: the positive description is as follows: close-up portrait of the model, hazy background, bright eyes, shiny lips, futuristic golden face war painting, artistic style, photo realism, clear urban background. The negative description is as follows: poor quality, poor anatomical structure, low resolution, extra fingers, blur, ugly, wrong proportions, watermarks, image artifacts, low resolution, deformation, noisy images. It should be noted that for the image or audio information involving external references in the description information, the description enhancement proxy processing unit will call different image algorithms and audio algorithms for recording and preprocessing, and add marks to the description formatted text.

[0028] Step S3: Generate an image proxy processing unit through description, intelligently schedule different text image artificial intelligence neural network models or algorithms according to different description information and formats, and generate an image with initial accuracy according to the text description and related formatting markup information.

[0029] In this embodiment, specifically, Figure 2a-2d As shown, the step S3 includes: Load and initialize different text-generated image models, including but not limited to Generative Adversarial Networks (GAN), Diffusion Models, basic image preprocessing algorithms (edge ​​extraction, portrait recognition and cropping, image depth extraction), Control Net, multimodal models, etc. The intelligent agent may call multiple different algorithm models in different formats for the same description at the same time, forming multiple business tasks that are calculated simultaneously, and generating multiple initial images for the same storyboard description; It should be noted that the algorithms used in describing the generated image proxy processing unit are, for example: the split-shot initial image generation fusion model algorithm, the character reference generation model algorithm, etc.; Among them, the split-shot initial image generation fusion model algorithm is a model algorithm formed after customized training using the existing open source Wenshengtu big model; that is, based on the existing Wenshengtu big model, an innovative design is made to generate standardized format descriptions for different reference information (text, images, etc.), and at the same time, the Wenshengtu big model is used to reference multiple information and images. Fusion generative model algorithm; The character reference generation model algorithm is a control network that generates a Vincent image model algorithm by taking the character region data in the character image as input parameters, and controlling the Vincent image model to generate an image by integrating a character image that is consistent with the character features of the character region image while maintaining its original text understanding characteristics; the characters include but are not limited to human or anthropomorphic images, and scene object images. Taking the human image as an example, through training, while ensuring that the intensity of the face similarity exceeds 90%, the facial features such as expression, lighting, accessories, hairstyle, skin color, etc. of the generated portrait are adjusted, which can be controlled by describing the intensity or directly through the control bar of the application case.

[0030] Step S4: generating an image intelligent agent unit through the image, intelligently scheduling different graphic image processing algorithms or graphic image artificial intelligence neural network models according to different user operations, and generating high-precision and high-accuracy storyboard preparation images according to text descriptions and related formatting mark information; In this embodiment, specifically, Figure 3 As shown, the step S4 includes: Image Generation The image intelligent agent unit can load and initialize different image generation image models, including but not limited to Generative Adversarial Networks (GAN), Diffusion Models, basic image preprocessing algorithms (edge ​​extraction, portrait recognition and cropping, image depth extraction), Control Net, multimodal models, etc. Image generation The image algorithms scheduled by the image intelligent agent unit include but are not limited to basic graphic image processing algorithms (brightness, saturation, contrast, etc. adjustment), intelligent enhancement algorithms, super-resolution algorithms, high dynamic range (HDR) adjustment algorithms, etc.; The image generation image intelligent agent unit is based on the image generation image fusion algorithm and model. It may call multiple different algorithm models in different formats for the same description at the same time, forming multiple business tasks that are calculated at the same time, and generating multiple preliminary images for the same storyboard initial image; It should be noted that the image generation and image fusion algorithm and model is a model algorithm formed after customized training using the existing open source Wenshengtu model; that is, based on the existing Wenshengtu model, an innovative design is made to generate standardized format descriptions for different reference information (text, images, etc.), and at the same time, the Wenshengtu model is used to perform a fusion generative model algorithm for multiple information and image references; The image algorithm is based on modern image processing algorithms and is an innovative design of a new type of image processing intelligent algorithm for dynamic range adjustment and super-resolution content generation.

[0031] In this embodiment, specifically, the character reference generation model algorithm generates a control network of the Vincent image model algorithm by inputting the character region data in the character image as a parameter, and controls the Vincent image model to generate an image by integrating a character image consistent with the character characteristics of the character region image on the basis of maintaining its original text understanding characteristics, and the character includes but is not limited to a human or anthropomorphic image, and a scene object image. Taking the human image as an example, through training, while ensuring that the intensity of the face similarity exceeds 90%, the expression, lighting, accessories, hairstyle, skin color and other character features of the generated portrait are adjusted, which can be controlled by describing the intensity or directly through the control bar of the application case.

[0032] Step S5: generating a 3 to 8 second storyboard preparatory video by using an image generation video proxy processing unit for different storyboard preparatory images, intelligently scheduling different image generation video artificial intelligence neural network models for text descriptions and related formatting markup information; In this embodiment, specifically, Figure 4 As shown, the step S5 includes: The storyboard preparation image can be used as the first frame or the last frame of the storyboard video to be generated, or it can be used only as a reference for the video content; Load and initialize different image generation video models, including but not limited to Generative Adversarial Networks (GAN), Diffusion Models, basic image preprocessing algorithms (edge ​​extraction, portrait recognition and cropping, image depth extraction), Control Net, VIT models, multimodal models, etc. Based on the image-generated video proxy processing unit, the image-generated video proxy processing unit may simultaneously call multiple different algorithm models in different formats for the same storyboard, form multiple business tasks that are calculated simultaneously, and generate multiple preliminary short videos for the same storyboard initial image; In this embodiment, specifically, the image-generated video algorithm and model is a model algorithm formed after customized training using an existing open source image-generated video big model; that is, based on the existing image-generated video big model, an innovative design is used to generate standardized format descriptions for different reference information (text, images, etc.), and at the same time, a fusion generative model algorithm is used to reference multiple information and images using the image-generated video big model.

[0033] Step S6: Through the video connection / conversion proxy processing unit, for the multiple split-shot short videos produced in the above steps, different video processing algorithms and artificial intelligence neural network models are intelligently scheduled, and the split-shot videos are transformed according to the long video text description and related formatted script information, and linked to generate a long video of more than 1 minute; In this embodiment, specifically, Figure 5 As shown, the step S6 includes: For split-shot short videos, we can perform style conversion, video enhancement, subtitle generation and fusion, or audio generation and fusion based on the styles, transitions, and other information described in different long videos. Loading and initializing different video fusion transformation algorithms and models, including but not limited to High Dynamic Range Enhancement (HDR), Diffusion Models, Video Filter Image Processing Algorithms, Multimodal Models, etc. The video connection / conversion proxy processing unit refers to different styles and content scripts for different long video descriptions based on the video style transformation model, video enhancement algorithm, and video fusion algorithm, assembles and links the above-mentioned multiple split-shot short videos, and generates the final long video result; It should be noted that the video style transfer model is a fast video model algorithm formed by customized training using an existing open source big picture model. That is, based on the existing big picture model, an innovative design is made to generate standardized format descriptions for different reference information (text, images, etc.), and at the same time, a fusion generative model algorithm is used to reference multiple information and images using the big picture model. The video enhancement algorithm is an innovative design algorithm for intelligent enhancement of high dynamic range, which can quickly enhance the dynamic range (HDR) of the video to achieve better video brightness and shading effects; The video fusion algorithm is an intelligent agent that innovatively quotes long video scripts, splits the video description, intelligently schedules open source or closed source subtitle generation algorithms and models, voice generation algorithms and models, sound effect generation algorithms and models, matches the split-shot videos in time, fuses the generated or script-preset subtitles and audio, processes the transition effects at the beginning and end of the split-shots, and finally merges the links to generate a result long video that meets the original description and satisfies the user's artistic creativity, such as Figure 6 shown.

[0034] The above-mentioned embodiments only express the specific implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the protection scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the technical solution concept of the present application, and these all belong to the protection scope of the present application.

[0035] This background section is provided to generally present the context of the invention, and the work of the presently named inventors, the work to the extent described in this background section, and aspects of this section that did not constitute prior art at the time of application are neither explicitly nor implicitly admitted to be prior art to the present invention.

Claims

1. A video generation system based on intelligent agent, characterized in that: include: A description enhancement proxy processing unit is used to perform various calculation tasks on the shot description information of the target video to obtain a shot description script with richer content; A description generation image agent processing unit generates an image satisfying the storyboard information based on the storyboard description script and serves as a rough sketch of the storyboard, i.e., a storyboard sketch image; An image generation image proxy processing unit is used to intelligently adjust and edit the storyboard sketch image so that the generated storyboard image meets the accuracy of the storyboard and the accuracy of the content; The image generation video proxy processing unit uses the split-shot image as a guide frame to generate a split-shot candidate video clip of a specified duration; The video connection / transformation proxy processing unit is used to perform effect transformation processing on the alternative storyboard video clips and connect multiple alternative storyboard video clips into one long video.

2. The video generation system based on intelligent agent according to claim 1, characterized in that: The computing tasks include: text enhancement, style script generation, and reference image preprocessing.

3. A video generation method based on intelligent agent, characterized in that: A video generation system based on an intelligent agent according to claim 1 or 2, comprising: Step S1: Create shot description information of the target video; Step S2: performing multiple computing tasks on the shot description information of the target video to obtain a shot description script with richer content; Step S3: generating an image satisfying the storyboard information based on the storyboard description script and using it as a rough sketch of the storyboard, i.e., a storyboard sketch image; Step S4: intelligently adjusting and editing the storyboard sketch image so that the generated storyboard image meets the accuracy of the storyboard and the accuracy of the content; Step S5: using the storyboard image as a guide frame to generate a storyboard candidate video clip of a specified duration; Step S6: performing effect transformation processing on the alternative storyboard video clips and connecting multiple alternative storyboard video clips into one long video.

4. The video generation method based on intelligent agent according to claim 3, characterized in that: The step S2 comprises: Enhance, polish or rewrite the text of the storyboard description information to obtain detailed description text suitable for machine learning and artificial intelligence understanding; Identify the style of the storyboard description information and generate a style formatting script; Preprocess the reference image information of the storyboard description information.

5. The video generation method based on intelligent agent according to claim 3, characterized in that: The step S3 comprises: Based on the storyboard description script, an image or image sequence that satisfies the description meaning is generated by describing a generation algorithm automatically selected by an image generation proxy processing unit, which is also called a storyboard sketch image; The process of generating the image proxy processing unit in the scheduling algorithm includes: multiple scheduling of different neural network models to generate a series of image sequences that meet the story sequence and consistent algorithm scheduling for the role; The characters include but are not limited to human or anthropomorphic images, and scene object images.

6. The video generation method based on intelligent agent according to claim 3, characterized in that: The step S4 comprises: Through the image generation image proxy module, the storyboard sketch image is further intelligently edited; through the scheduling of the image generation large model, portrait synthesis algorithm, text synthesis algorithm and various basic image processing algorithms, the storyboard sketch image is further processed to obtain a high-precision and high-quality storyboard starting image.

7. The video generation method based on intelligent agent according to claim 6, characterized in that: The step S5 comprises: Based on the starting image of the storyboard, the image-generated video proxy module is used to analyze and generate a series of video frame images. In this process, the image-generated video proxy module schedules different intelligent algorithms to continuously simulate the body and movements of the characters in the image, identify and simulate the physical phenomena of the scene in the image, and continuously transform the overall style of the image, so as to obtain storyboard alternative video clips of specified length that are suitable for the storyboard description.

8. The video generation method based on intelligent agent according to claim 7, characterized in that: The step S6 comprises: Based on the alternative video clips of the storyboard, various audio, video and image data editing algorithms are scheduled through the video connection / transformation proxy processing unit to merge and process the alternative video clips of the storyboard, and automatic adaptation is performed during the storyboard merging process to obtain film and television content that meets the storyboard description and output the final video.

Images