Digital actor-based full-automatic production system for film and television videos

Abstract

The application discloses a digital actor-based full-automatic production system for film and television videos, belongs to the field of artificial intelligence, computer graphics and film and television production, and comprises a creative engine unit, a production unit, an intelligent shooting unit, a post-processing unit and a quality inspection and rendering unit.The application has the following advantages: 1. The production efficiency is greatly improved; the production cycle of traditional film and television production is usually 6-12 months, while the production cycle of the application can be shortened to 4 hours, and the efficiency is improved by about 1000 times; 2. The production cost is greatly reduced; the production cost of traditional film and television production is usually 5 million-200 million US dollars, while the production cost of the application is only the algorithm cost, and the cost is less than one hundredth of the traditional film and television production cost; and 3. The style consistency is guaranteed; the traditional film and television production depends on personal ability, and the style consistency is difficult to guarantee, while the application strictly guarantees the style consistency through an algorithm, and the style consistency is improved by 100%.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, computer graphics, and film and television production, specifically a fully automated film and television video production system based on digital actors. Background Technology

[0002] Traditional film and television production typically involves multiple stages, including scriptwriting, visual design, music design, filming, and post-production. Each stage requires the participation of numerous professionals, resulting in long production cycles and high costs. While the development of artificial intelligence technology has led to the emergence of some tools to assist in film and television production, most of these tools can only assist in a single stage and cannot achieve full-process automation.

[0003] While some AI-assisted film and television production tools exist, such as AI script generation, AI image generation, and AI music generation, these tools typically operate independently and lack the ability to collaborate, failing to automate the entire process from concept to final product. Furthermore, existing technologies are inadequate in areas such as creative quantification, autonomous decision-making, and quality assurance, making it difficult to meet the demands of professional-level film and television production.

[0004] This invention aims to solve the technical problems existing in the traditional film and television production process, such as long production cycle, high cost, difficulty in ensuring style consistency, and slow creative iteration, as well as the problems of existing AI-assisted film and television production tools that cannot achieve full-process automation, lack collaborative work ability, insufficient creative quantification, weak decision-making autonomy, and imperfect quality assurance mechanism. Summary of the Invention

[0005] This invention provides a fully automated film and television production system based on digital actors. Through a three-layer design of creative algorithmization, execution programization, and intelligent decision-making, it realizes a fully automated film factory from text input to professional-grade video output, thereby solving the shortcomings of existing technologies.

[0006] This invention is achieved through the following technical solution:

[0007] A fully automated film and television production system based on digital actors, including a creative engine unit, a production unit, an intelligent shooting unit, a post-processing unit, and a quality inspection and rendering unit;

[0008] The creative engine unit is responsible for transforming the user's input natural language requirements into structured creative content;

[0009] The aforementioned production unit is responsible for transforming ideas into executable production plans;

[0010] The intelligent shooting unit is responsible for performing shooting in the virtual environment;

[0011] The post-processing unit is responsible for post-processing the footage obtained from the shooting.

[0012] The quality inspection and rendering unit is responsible for performing quality checks and rendering output on the final video.

[0013] The fully automated film and television production system based on digital actors, as described above, includes a creative engine unit comprising a script generation module, a visual design module, and a music design module.

[0014] The script generation module uses a large language model to generate structured scripts and automatically embeds sentiment curve analysis through a sentiment computing model.

[0015] The specific operations of the script generation module are as follows: receiving natural language requirements input by the user, generating script text based on the user's requirements using a large language model; analyzing the script text through an emotion computing model to generate an emotion curve, which includes the emotion intensity and emotion type at multiple time points; integrating the script text and emotion curve into a structured script, outputting it in JSON format, which includes fields such as scene number, shot suggestions, and emotion tags;

[0016] The visual design module automatically matches the visual style based on the script content, generates reference images through AIGC tools, and outputs a style parameter package.

[0017] The specific operations of the visual design module are as follows: using a large model to analyze the script content and extract key information about the theme, emotions, and scenes; automatically matching visual styles based on key information, including color schemes, composition styles, and material styles; using AIGC tools to generate reference images; and generating style parameter packages based on the reference images and the matched visual styles, including color matrices, composition rules, and material libraries.

[0018] The music design module generates music style tags based on the script's emotional curve and generates corresponding music using AI.

[0019] The specific operations of the music design module are as follows: generate music style tags based on the script's emotional curve, the music style tags include music type, rhythm, and mood; create a music emotion timeline and align the music style tags with the timeline; generate corresponding music using AI based on the music emotion timeline and music style tags; output a MIDI file and a timbre library list, the MIDI file contains the music's structure and melody information, and the timbre library list contains the required timbre information.

[0020] The fully automated film and television production system based on digital actors, as described above, specifically operates the emotion computing model as follows: A pre-trained emotion analysis model is used to perform emotion analysis on the script text, obtaining the emotion vector for each paragraph or sentence; an emotion curve is constructed based on the emotion vector, the emotion curve including a time axis and an emotion intensity axis, with the emotion intensity ranging from [-1, 1], where negative values ​​represent negative emotions and positive values ​​represent positive emotions; the emotion curve is aligned with the script text, and emotion labels are assigned to each scene or shot.

[0021] The style parameter package includes a color matrix, composition rules, and a material library. The color matrix includes parameters such as primary color, secondary color, contrast, brightness, and saturation, represented in matrix form. The composition rules include rules for lens type, perspective, depth of field, and motion, represented in rule set form. The material library includes material parameters for various objects in the scene, including roughness, metallicity, and transparency, represented in material parameter set form.

[0022] The method for generating the music emotion timeline is as follows: mapping the script's emotion curve onto the music emotion space, which includes dimensions such as tension, activity, and pleasure; generating the music emotion timeline based on the emotion points in the music emotion space, which includes time points and corresponding music emotion tags; aligning the music emotion timeline with the script timeline to ensure consistency between the music emotion and the script emotion.

[0023] The fully automated film and television video production system based on digital actors, as described above, includes an automatic storyboard generation module, an automatic resource matching module, and a shooting plan optimization module in its production unit.

[0024] The automatic storyboard generation module automatically generates storyboards based on the script and visual rules. Specifically, it receives a structured script and visual rules from a style parameter package as input; uses the StoryboardAgent.generate algorithm to generate storyboards based on the script content and visual rules; the storyboards include scene numbers, shot types, shot parameters, composition, lighting, and performance content; and outputs a structured storyboard script to guide subsequent virtual filming.

[0025] The automatic resource matching module automatically matches resources based on storyboard scenes and asset libraries. Its specific implementation involves: receiving storyboard scenes and asset libraries as input; using the ResourceAgent.allocate algorithm to automatically match resources based on the needs of the storyboard scenes and the content of the asset libraries; resources include digital actors, scenes, props, and special effects, and these asset library resources are mainly reusable materials generated by AI; outputting resource matching results, including a list of resources required for each scene and resource parameters.

[0026] The shooting plan optimization module optimizes the shooting plan based on the storyboard scene and constraints. Specifically, it receives the storyboard scene and constraints as input, uses the ScheduleAgent.optimize algorithm to optimize the shooting plan based on the dependencies and constraints of the storyboard scene, and the shooting plan includes shooting order, shooting time, shooting location, etc. The optimized shooting plan is output to guide the execution of virtual shooting.

[0027] The StoryboardAgent.generate algorithm in the fully automated film and television production system based on digital actors, as described above, is specifically implemented as follows: natural language processing technology is used to analyze the script content and extract key information about scenes, actions, and dialogues; based on visual rules, appropriate shot types and shot parameters are assigned to each scene; computer vision technology is used to pre-generate the composition and lighting effects of each shot; and reinforcement learning is used to optimize the coherence and artistry of the storyboard.

[0028] The ResourceAgent.allocate algorithm is implemented as follows: semantic analysis is used to analyze the needs of the storyboard scene and extract the required resource types and features; based on the resource types and features, matching resources are searched in the asset library; a graph matching algorithm is used to evaluate the matching degree between the resources and the needs; the resource with the highest matching degree is selected and the resource parameters are adjusted to better meet the needs; for needs that cannot be matched, the material library can be directly generated according to the needs.

[0029] The ScheduleAgent.optimize algorithm is implemented as follows: the storyboard scenes are modeled as graph nodes, and the dependencies between scenes are modeled as edges; the constraints are modeled as constraints in the graph; the constraint satisfaction problem algorithm is used to solve for the optimal shooting order; and a detailed shooting schedule is generated based on the shooting order and constraints.

[0030] The fully automated film and television video production system based on digital actors, as described above, includes a virtual shooting environment module, an automated execution module, an intelligent lighting module, and a performance generation module in its intelligent shooting unit.

[0031] The virtual shooting environment module constructs a virtual shooting environment, including scenes, digital actors, and equipment. Specifically, it is implemented as follows: a scene generation model is used to generate virtual scenes. This model, based on AIGC technology, can generate realistic 3D scenes according to the script description. AI-driven digital actors are used, whose expressions and movements are controlled by AI, enabling them to automatically perform corresponding acts according to the script description. A virtual camera system is used, which controls motion parameters through AI, automatically setting camera parameters according to the storyboard.

[0032] The automated execution module automatically sets camera parameters and performs shooting based on the storyboard. Specifically, it receives the storyboard as input, automatically sets camera parameters, including focal length, aperture, and motion trajectory, based on the lens suggestions in the storyboard, performs virtual shooting to capture scene images, and outputs EXR sequences and depth channels for subsequent post-processing.

[0033] The intelligent lighting module automatically sets up lighting based on a visual style package and dynamically adjusts the lighting according to emotion tags. Specifically, it receives a visual style package as input; automatically sets up lighting based on the color matrix and composition rules in the visual style package, including HDRI ambient light and key light; dynamically adjusts the lighting according to emotion tags; renders lighting effects in real time and outputs high-quality images.

[0034] The performance generation module generates dialogue using LLM and matches performances using a motion capture library. Its specific implementation includes: receiving dialogue and action descriptions from the script as input; generating dialogue using LLM, including voice content and emotional expression; matching performances using a motion capture library and selecting the action sequence that best matches the action description; synthesizing facial expressions in real-time based on the LLM-generated text to control the facial expressions of digital actors; and outputting the performance data of digital actors for virtual filming.

[0035] The fully automated film and television video production system based on digital actors, as described above, specifically implements the scene generation model as follows: using text-to-3D model generation technology to generate the basic structure of a 3D scene based on the script description; using procedural content generation technology to add details and textures to the 3D scene; using physically based rendering technology to add lighting and material effects to the 3D scene; and using neural rendering technology to optimize the visual effects of the 3D scene.

[0036] The automatic setting method for camera parameters is as follows: using a rule engine to set basic camera parameters based on lens type; using reinforcement learning to optimize camera parameters based on scene content and sentiment tags; using a path planning algorithm to generate camera motion trajectory; and using a motion control algorithm to smooth camera motion and avoid jitter and abrupt changes.

[0037] The specific implementation of automatic lighting is as follows: using image analysis technology, the reference images in the visual style package are analyzed to extract lighting information; based on the lighting information, basic lighting settings are generated, including ambient light, main light source, auxiliary light source, etc.; using physical rendering technology, the interaction between lighting and materials is simulated to generate realistic lighting effects; using reinforcement learning, the lighting settings are optimized based on sentiment tags to enhance emotional expression.

[0038] The specific implementation of facial expression synthesis is as follows: using text sentiment analysis technology to analyze the text generated by LLM and extract sentiment information; generating facial expression parameters based on sentiment information; using a parameterized facial model to generate facial expressions according to the facial expression parameters; and using expression transition technology to smooth changes in facial expressions and avoid abrupt changes and unnaturalness.

[0039] The post-processing unit of the fully automated film and television video production system based on digital actors, as described above, includes an automatic editing module, an intelligent color grading module, an automatic audio mixing module, and an intelligent special effects module.

[0040] The automatic editing module uses multimodal AI to analyze video, audio, and script, and automatically performs editing. Its specific implementation includes: receiving video footage, audio footage, and script as input; using multimodal AI to analyze video, audio, and script to evaluate shot quality; optimizing video rhythm based on music beats and emotional curves; automatically selecting the best shot and generating a rough cut EDL file.

[0041] Output the edited video;

[0042] The intelligent color grading module uses a style transfer network and a color science model to automatically grade colors. Specifically, it receives video footage and a visual style package as input; uses a style transfer network to transfer the style from the visual style package to the video footage; uses a color science model to automatically match the visual style package and optimize color consistency between shots; generates a color correction LUT and applies it to the video footage; and outputs the color-graded video.

[0043] The automatic mixing module uses audio separation and an AI mixing model to automatically mix audio. Its specific implementation includes: receiving audio material as input; using audio separation technology to separate the mixed audio into tracks for dialogue, sound effects, and music; using an AI mixing model to automatically balance the proportions and spatial positions of dialogue, sound effects, and music; generating spatial audio based on scene 3D data to enhance immersion; and outputting a 5.1 mixed WAV file.

[0044] The intelligent special effects module uses computer vision and generative AI to automatically generate special effects. Its specific implementation includes: receiving video footage and special effects requirements as input; using computer vision technology to automatically identify the areas or objects that need to have special effects added; using generative AI to programmatically generate special effects; automatically removing unwanted objects or fixing continuity errors; and outputting a VFX compositing layer for compositing with the original video.

[0045] The fully automated film and television video production system based on digital actors, as described above, specifically implements the shot quality scoring as follows: using computer vision technology to analyze video frames and evaluate composition quality; using motion analysis technology to evaluate shot motion quality; using performance analysis technology to evaluate the performance quality of digital actors; and combining the above scores to generate a total shot quality score for shot selection.

[0046] The specific implementation of the style transfer network is as follows: a deep learning model is used to transfer the style of the reference image to the video frame; temporal consistency technology is used to maintain style consistency between video frames and avoid flickering and abrupt changes; color space conversion technology is used to convert the style-transferred image back to the original color space; and color correction technology is used to fine-tune color parameters to ensure color accuracy and consistency.

[0047] The specific implementation of the AI ​​mixing model is as follows: using a deep learning model to learn the mixing strategies of professional mixing engineers; adjusting mixing parameters based on emotion curves and music emotion timelines; using spatial audio technology to generate 3D audio effects; and using audio rendering technology to mix multi-track audio into the final 5.1 surround sound.

[0048] The specific implementation of the procedural special effects generation is as follows: using a particle system to generate natural phenomenon special effects such as smoke, flames, and water flow; using physical simulation to simulate the physical effects of object movement, collision, and deformation; using neural network rendering to generate realistic lighting and material effects; and using image compositing technology to seamlessly integrate the special effects with the original video.

[0049] The fully automated film and television production system based on digital actors, as described above, includes an automated quality inspection process module and a rendering output module in its quality inspection and rendering unit.

[0050] The automated quality inspection process module performs technical indicator testing and artistic consistency testing on the final video, and automatically corrects problems. Its specific implementation includes: receiving the final video as input; performing technical indicator testing to check whether the technical parameters of resolution, frame rate, and color space meet the requirements; performing artistic consistency testing to verify whether the visual style and emotional curve are consistent with the original design; if problems are detected, using an automatic correction mechanism to fix them; and outputting a quality inspection report and the corrected video.

[0051] The specific implementation of the artistic consistency detection is as follows: using computer vision technology to analyze the visual features of video frames; comparing the visual features with the original style parameter package to calculate the deviation; using sentiment analysis technology to analyze the emotional expression of the video and comparing it with the original sentiment curve; combining visual deviation and sentiment deviation to calculate the artistic consistency score; if the score is lower than the threshold, automatic correction is triggered.

[0052] The rendering output module renders the final video into different formats and generates accompanying files. Its specific implementation includes: receiving the final video as input; selecting appropriate rendering parameters according to output requirements; performing rendering to generate video files in different formats; generating accompanying files, including SRT subtitles, XML metadata, JPG style reference atlas, PDF emotion curve charts, and TXT technical parameter reports; and outputting a complete digital asset package.

[0053] The specific implementation of the rendering output is as follows: using distributed rendering technology to allocate rendering tasks to multiple rendering nodes to improve rendering efficiency; using color management technology to ensure color consistency across different output formats; using metadata management technology to record all decisions and parameters during the production process and generate a traceable production report; and using packaging technology to package all output files into a digital asset package for easy delivery and archiving.

[0054] The advantages of this invention are:

[0055] 1. Significantly improves production efficiency: Traditional film and television production cycles typically take 6-12 months, while this invention can shorten the production cycle to 4 hours, increasing efficiency by approximately 1000 times;

[0056] 2. Significantly reduced production costs: The production costs of traditional films and other film and television productions are typically between $5 million and $200 million, while this invention only requires computing power costs, which is less than one percent of the cost of traditional film and television productions;

[0057] 3. Ensure stylistic consistency: Traditional film and television production relies on individual skills, making it difficult to guarantee stylistic consistency. This invention, however, strictly ensures stylistic consistency through algorithms, improving consistency by 100%.

[0058] 4. Accelerate creative iteration: Traditional film and television production creative iteration takes several weeks, while this invention can achieve real-time (minute-level) creative iteration, improving efficiency by about 1000 times;

[0059] 5. Achieve full-process automation: From user input to professional-grade video output, the entire process requires no human intervention, achieving truly fully automated film and television production;

[0060] 6. Provides traceable digital asset packages: In addition to outputting video files, it also includes digital asset packages with complete traceability of the production process, bringing efficiency improvements to the film and television industry. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a diagram of the architecture of the present invention. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] like Figure 1 As shown, the fully automated film and television video production system based on digital actors is characterized by including a creative engine unit, a production unit, an intelligent shooting unit, a post-processing unit, and a quality inspection and rendering unit.

[0065] Preferably, the creative engine unit described in this embodiment is responsible for transforming the user's input natural language requirements into structured creative content.

[0066] Preferably, the production unit described in this embodiment is responsible for transforming ideas into executable production plans.

[0067] Preferably, the intelligent shooting unit described in this embodiment is responsible for performing shooting in a virtual environment.

[0068] Preferably, the post-processing unit described in this embodiment is responsible for post-processing the captured footage.

[0069] Preferably, the quality inspection and rendering unit described in this embodiment is responsible for performing quality inspection and rendering output on the final video.

[0070] Preferably, the creative engine unit described in this embodiment includes a script generation module, a visual design module, and a music design module.

[0071] Preferably, the script generation module described in this embodiment uses large-scale language models (such as GPT-4, DeepSeek, Zhipu, Doubao, etc.) to generate structured scripts, and automatically embeds sentiment curve analysis through sentiment computing models.

[0072] Preferably, the specific operation of the script generation module described in this embodiment is as follows: receiving natural language input from the user, such as "cyberpunk-style romance film, 5 minutes long"; generating script text based on the user's needs using a large language model; analyzing the script text through an emotion computing model to generate an emotion curve, which includes the emotion intensity and emotion type at multiple time points; integrating the script text and the emotion curve into a structured script, outputting it in JSON format, which includes fields such as scene number, shot suggestions, and emotion tags.

[0073] Preferably, the visual design module described in this embodiment automatically matches the visual style based on the script content, generates reference images through the AIGC tool, and outputs a style parameter package.

[0074] Preferably, the specific operation of the visual design module described in this embodiment is as follows: using a large model to analyze the script content and extract key information such as theme, emotion, and scene; automatically matching visual styles based on key information, including color scheme, composition style, and material style; using AIGC tools (such as Kling / Midjourney / vidu, etc.) to generate reference images; and generating a style parameter package based on the reference images and the matched visual styles, including color matrix, composition rules, and material library.

[0075] Preferably, the music design module described in this embodiment generates music style tags based on the script's emotional curve and generates corresponding music using AI.

[0076] Preferably, the specific operations of the music design module described in this embodiment are as follows: generating music style tags based on the script's emotional curve, the music style tags including music type, rhythm, and mood; creating a music emotional timeline, aligning the music style tags with the timeline, such as "0:00-0:30 → tension, 0:31-1:20 → romance"; generating corresponding music using AI based on the music emotional timeline and music style tags; and outputting a MIDI file and a sound library list, the MIDI file containing the music's structure and melody information, and the sound library list containing the required sound information.

[0077] Preferably, the specific operation of the sentiment computing model described in this embodiment is as follows: using a pre-trained sentiment analysis model to perform sentiment analysis on the script text to obtain the sentiment vector of each paragraph or sentence; constructing a sentiment curve based on the sentiment vector, the sentiment curve including a time axis and a sentiment intensity axis, the sentiment intensity range being [-1,1], negative values ​​representing negative sentiment, and positive values ​​representing positive sentiment; aligning the sentiment curve with the script text, and assigning sentiment labels to each scene or shot;

[0078] Preferably, the style parameter package described in this embodiment includes a color matrix, composition rules, and a material library. The color matrix includes parameters such as primary color tone, secondary color tone, contrast, brightness, and saturation, represented in matrix form. The composition rules include rules for lens type, perspective, depth of field, and motion, represented in rule set form. The material library includes material parameters for various objects in the scene, including roughness, metallicity, and transparency, represented in material parameter set form.

[0079] Preferably, the method for generating the music emotion timeline in this embodiment is as follows: mapping the script emotion curve onto the music emotion space, which includes dimensions such as tension, activity, and pleasure; generating the music emotion timeline based on the emotion points in the music emotion space, which includes time points and corresponding music emotion tags; aligning the music emotion timeline with the script timeline to ensure that the music emotion is consistent with the script emotion.

[0080] Preferably, the production unit described in this embodiment includes an automatic storyboard generation module, an automatic resource matching module, and a shooting plan optimization module.

[0081] Preferably, the automatic storyboard generation module described in this embodiment automatically generates storyboards based on the script and visual rules; its specific implementation is as follows: receiving the structured script and visual rules in the style parameter package as input; using the StoryboardAgent.generate algorithm to generate storyboards based on the script content and visual rules; the storyboards include scene number, shot type, shot parameters, composition, lighting, and performance content; and outputting a structured storyboard script to guide subsequent virtual shooting.

[0082] Preferably, the automatic resource matching module described in this embodiment automatically matches resources based on storyboard scenes and asset libraries; its specific implementation is as follows: receiving storyboard scenes and asset libraries (such as UnrealEngine_Metahuman) as input; using the ResourceAgent.allocate algorithm to automatically match resources based on the needs of the storyboard scenes and the content of the asset library; resources include digital actors, scenes, props, and special effects, and these asset library resources are also reusable materials generated by AI; outputting resource matching results, including a list of resources required for each scene and resource parameters.

[0083] Preferably, the shooting plan optimization module described in this embodiment optimizes the shooting plan based on the storyboard scene and constraints. Specifically, it receives the storyboard scene and constraints (such as "daylight_cycle" and "actor_availability") as input; uses the ScheduleAgent.optimize algorithm to optimize the shooting plan based on the dependencies and constraints of the storyboard scene; the shooting plan includes shooting order, shooting time, shooting location, etc.; and outputs the optimized shooting plan to guide the execution of virtual shooting.

[0084] Preferably, the StoryboardAgent.generate algorithm described in this embodiment is implemented as follows: natural language processing technology is used to analyze the script content and extract key information about scenes, actions, and dialogues; based on visual rules, appropriate shot types and shot parameters are assigned to each scene; computer vision technology is used to pre-generate the composition and lighting effects of each shot; and reinforcement learning is used to optimize the coherence and artistry of the storyboard.

[0085] Preferably, the ResourceAgent.allocate algorithm described in this embodiment is implemented as follows: semantic analysis technology is used to analyze the needs of the storyboard scene and extract the required resource types and features; based on the resource types and features, matching resources are searched in the asset library; a graph matching algorithm is used to evaluate the matching degree between the resources and the needs; the resource with the highest matching degree is selected, and the resource parameters are adjusted to better meet the needs; for needs that cannot achieve the matching degree, the corresponding materials can be directly generated by AI.

[0086] Preferably, the ScheduleAgent.optimize algorithm described in this embodiment is implemented as follows: the storyboard scene is modeled as graph nodes, and the dependencies between scenes are modeled as edges; the constraints are modeled as constraints in the graph; the constraint satisfaction problem (CSP) algorithm is used to solve for the optimal shooting order; and a detailed shooting schedule is generated based on the shooting order and constraints.

[0087] Preferably, the intelligent shooting unit described in this embodiment includes a virtual shooting environment module, an automated execution module, an intelligent lighting module, and a performance generation module.

[0088] Preferably, the virtual shooting environment module described in this embodiment constructs a virtual shooting environment, including scenes, digital actors, and equipment; its specific implementation is as follows: a scene generation model is used to generate a virtual scene, which is based on AIGC technology and can generate realistic 3D scenes according to the script description; AI-driven digital actors are used, whose expressions and movements are controlled by AI and can automatically perform corresponding performances according to the script description; a virtual camera system is used, which controls motion parameters through AI and can automatically set camera parameters according to the storyboard.

[0089] Preferably, the automated execution module described in this embodiment automatically sets camera parameters and performs shooting according to the storyboard script; its specific implementation method is as follows: receiving the storyboard script as input; automatically setting camera parameters, including focal length, aperture, and motion trajectory, according to the lens suggestions in the storyboard script; performing virtual shooting to capture scene images; and outputting EXR sequence and depth channel for subsequent post-processing.

[0090] Preferably, the intelligent lighting module described in this embodiment automatically sets up lighting based on a visual style package and dynamically adjusts the lighting according to emotional tags. Specifically, it receives a visual style package as input; automatically sets up lighting based on the color matrix and composition rules in the visual style package, including HDRI ambient light and key light; dynamically adjusts the lighting according to emotional tags, such as "adding a strobe effect to tense scenes"; renders the lighting effects in real time and outputs high-quality images.

[0091] Preferably, the performance generation module described in this embodiment generates dialogue through LLM and matches performances using a motion capture library. Specifically, it includes: receiving dialogue and action descriptions from the script as input; generating dialogue using LLM, including voice content and emotional expression; matching performances using a motion capture library and selecting the action sequence that best matches the action description; synthesizing facial expressions in real time based on the text generated by LLM to control the facial expressions of digital actors; and outputting the performance data of digital actors for virtual filming.

[0092] Preferably, the specific implementation of the scene-based generation model described in this embodiment is as follows: using text-to-3D model generation technology to generate the basic structure of the 3D scene based on the script description; using procedural content generation technology to add details and textures to the 3D scene; using physically based rendering technology to add lighting and material effects to the 3D scene; and using neural rendering technology to optimize the visual effects of the 3D scene.

[0093] Preferably, the automatic setting method for camera parameters described in this embodiment is as follows: using a rule engine to set basic camera parameters based on shot type (such as close-up, medium shot, long shot); using reinforcement learning to optimize camera parameters based on scene content and sentiment tags; using a path planning algorithm to generate camera motion trajectory; and using a motion control algorithm to smooth camera motion and avoid jitter and abrupt changes.

[0094] Preferably, the automatic lighting described in this embodiment is implemented as follows: using image analysis technology, the reference image in the visual style package is analyzed to extract lighting information; based on the lighting information, basic lighting settings are generated, including ambient light, main light source, auxiliary light source, etc.; using physical rendering technology, the interaction between lighting and materials is simulated to generate realistic lighting effects; and using reinforcement learning, the lighting settings are optimized based on emotion tags to enhance emotional expression.

[0095] Preferably, the specific implementation of facial expression synthesis in this embodiment is as follows: using text sentiment analysis technology to analyze the text generated by LLM and extract sentiment information; based on the sentiment information, generating facial expression parameters, including motion parameters of parts such as eyebrows, eyes, and mouth; using a parametric facial model to generate facial expressions according to the facial expression parameters; and using expression transition technology to smooth changes in facial expressions and avoid abrupt changes and unnaturalness.

[0096] Preferably, the post-processing unit described in this embodiment includes an automatic editing module, an intelligent color grading module, an automatic audio mixing module, and an intelligent special effects module.

[0097] Preferably, the automatic editing module described in this embodiment uses multimodal AI to analyze video, audio, and script, and automatically performs editing. Its specific implementation includes: receiving video material, audio material, and script as input; using multimodal AI to analyze video, audio, and script, and evaluating shot quality, including composition, movement, and performance; optimizing video rhythm based on music beats and emotional curves; automatically selecting the best shot and generating a rough cut EDL file; and outputting the edited video.

[0098] Preferably, the intelligent color grading module described in this embodiment uses a style transfer network and a color science model to automatically perform color grading. Specifically, it is implemented as follows: receiving video footage and a visual style package as input; using a style transfer network to transfer the style from the visual style package to the video footage; using a color science model to automatically match the visual style package and optimize color consistency between shots; generating a color correction LUT and applying it to the video footage; and outputting the color-graded video.

[0099] Preferably, the automatic mixing module described in this embodiment uses audio separation and an AI mixing model to automatically perform audio mixing; its specific implementation includes: receiving audio materials (including dialogue, sound effects, and music) as input; using audio separation technology to separate the mixed audio into tracks for dialogue, sound effects, and music; using an AI mixing model to automatically balance the proportion and spatial position of dialogue, sound effects, and music; generating spatial audio based on scene 3D data to enhance immersion; and outputting a 5.1 mixed WAV file.

[0100] Preferably, the intelligent special effects module described in this embodiment uses computer vision and generative AI to automatically generate special effects. Its specific implementation includes: receiving video footage and special effects requirements as input; using computer vision technology to automatically identify the areas or objects for which special effects need to be added; using generative AI (such as AIGC VFX) to programmatically generate special effects; automatically removing unwanted objects or fixing continuity errors; and outputting a VFX compositing layer for compositing with the original video.

[0101] Preferably, the specific implementation of the shot quality scoring in this embodiment is as follows: using computer vision technology to analyze video frames and evaluate composition quality, such as the rule of thirds, symmetry, and visual guidance; using motion analysis technology to evaluate shot motion quality, such as smoothness, stability, and dynamic range; using performance analysis technology to evaluate the performance quality of digital actors, such as the naturalness of facial expressions and the continuity of movements; and combining the above scores to generate a total shot quality score for shot selection.

[0102] Preferably, the style transfer network described in this embodiment is implemented as follows: using deep learning models, such as AdaIN and NeuralStyleTransfer, to transfer the style of the reference image to the video frame; using temporal consistency technology to maintain style consistency between video frames and avoid flickering and abrupt changes; using color space conversion technology to convert the style-transferred image back to the original color space; and using color correction technology to fine-tune color parameters to ensure color accuracy and consistency.

[0103] Preferably, the AI ​​mixing model described in this embodiment is implemented as follows: using deep learning models, such as DNN and CNN, to learn the mixing strategies of professional mixing engineers; adjusting mixing parameters, such as volume, equalization, and compression, based on the emotion curve and the music emotion timeline; using spatial audio technologies, such as Ambisonics and HRTF, to generate 3D audio effects; and using audio rendering technology to mix multi-track audio into the final 5.1 surround sound.

[0104] Preferably, the specific implementation of the procedural special effects generation described in this embodiment is as follows: using a particle system to generate natural phenomenon special effects such as smoke, flames, and water flow; using physical simulation to simulate the physical effects of object movement, collision, and deformation; using neural network rendering to generate realistic lighting and material effects; and using image compositing technology to seamlessly integrate the special effects with the original video.

[0105] Preferably, the quality inspection and rendering unit described in this embodiment includes an automated quality inspection process module and a rendering output module.

[0106] Preferably, the automated quality inspection process module described in this embodiment performs technical indicator testing and artistic consistency testing on the final video, and automatically corrects problems. Its specific implementation includes: receiving the final video as input; performing technical indicator testing to check whether the technical parameters of resolution, frame rate, and color space meet the requirements; performing artistic consistency testing to verify whether the visual style and emotional curve are consistent with the original design; if problems are detected, using an automatic correction mechanism to fix the problems; and outputting a quality inspection report and the corrected video.

[0107] Preferably, the specific implementation of the artistic consistency detection described in this embodiment is as follows: using computer vision technology to analyze the visual features of video frames, such as color distribution, texture features, composition features, etc.; comparing the visual features with the original style parameter package to calculate the deviation; using sentiment analysis technology to analyze the emotional expression of the video and compare it with the original sentiment curve; combining visual deviation and sentiment deviation to calculate the artistic consistency score; if the score is lower than the threshold, automatic correction is triggered.

[0108] The rendering output module renders the final video in different formats and generates accompanying files. Its specific implementation includes: receiving the final video as input; selecting appropriate rendering parameters, such as resolution, frame rate, and encoding format, according to output requirements; performing rendering to generate video files in different formats, such as 4K DCI, 1080p, and 265MP4; generating accompanying files, including SRT subtitles, XML metadata, JPG style reference atlas, PDF sentiment curve charts, and TXT technical parameter reports; and outputting a complete digital asset package.

[0109] Preferably, the specific implementation of the rendering output described in this embodiment is as follows: using distributed rendering technology to allocate rendering tasks to multiple rendering nodes to improve rendering efficiency; using color management technology to ensure color consistency across different output formats; using metadata management technology to record all decisions and parameters during the production process and generate a traceable production report; and using packaging technology to package all output files into a digital asset package for easy delivery and archiving.

[0110] Preferably, the technical breakthroughs of this embodiment are mainly reflected in three aspects: creativity quantification, autonomous decision-making, and quality assurance.

[0111] Quantifying Creativity:

[0112] The artistic style is transformed into computable parameters, and the emotions in the script are mapped to audiovisual parameters through an emotion computing model; the specific implementation methods include:

[0113] a. Establish a parametric model of art style, and convert style tags such as "cyberpunk" into computable parameters, such as hue [0.2, 0.4, 0.8], contrast 1.8, granularity 0.6, etc.;

[0114] b. Establish an emotional computing model to map the emotions in the script to audiovisual parameters, such as mapping "tension" to fast-paced music, cool-toned lighting, and rapid editing;

[0115] c. Use parametric models to guide idea generation and production execution, ensuring the consistency and controllability of ideas.

[0116] Decision-making autonomy:

[0117] Creative conflicts are resolved through a multi-agent game theory algorithm, and the production path is optimized through reinforcement learning; its specific implementation methods include:

[0118] a. Establish a multi-agent system, with each agent responsible for a specific production stage, such as a director agent, an editor agent, etc.;

[0119] b. Use multi-agent game theory algorithms such as Q-learning to resolve the creative conflict among agents and achieve the optimal solution;

[0120] c. Use reinforcement learning to train a production path optimization model based on historical successful cases to improve production efficiency and quality;

[0121] d. Achieve the system's autonomous decision-making capability and reduce human intervention.

[0122] quality assurance:

[0123] The self-evolution of multimodal models and the automatic correction of technical defects through anomaly detection algorithms are specifically implemented through the following methods:

[0124] a. Establish a multimodal evaluation model to conduct multi-dimensional quality assessments of the generated videos, including technical quality, artistic quality, and emotional expression;

[0125] b. Based on the evaluation results, use a generation-evaluation loop mechanism to continuously optimize the generation model and achieve self-evolution;

[0126] c. Establish an anomaly detection algorithm to automatically identify and correct technical defects, such as continuity errors, color inconsistencies, and audio desynchronization.

[0127] d. Implement the system's self-correction capability to ensure the professional quality of the output video.

[0128] Example 1: System Architecture

[0129] The fully automated film and television video AI agent production system provided by this invention includes five core modules: AICreative Engine, AIGC Virtual Production Hub, Autonomous Cinematography, Automated Post-Production, and QC & Rendering.

[0130] The system workflow is as follows: User-provided natural language interaction requirements serve as input. First, the system enters the AICreative Engine to generate structured creative content, including scripts, visual designs, and music designs. Then, it enters the AIGC Virtual Production Hub to transform the creative ideas into executable production plans, including storyboards, resource matching, and shooting schedules. Next, it enters Autonomous Cinematography for shooting in a virtual environment, generating raw footage. Then, it enters the Automated Post-Production center for editing, color grading, audio mixing, and special effects processing. Finally, it enters QC & Rendering for quality inspection and rendering output, generating complete video files and related digital asset packages.

[0131] Example 2: End-to-End Automation Process Example

[0132] An example of an end-to-end automation process of this invention is as follows:

[0133] Input: The user requested "a post-apocalyptic action movie, 3 minutes long, with a female protagonist";

[0134] 1. Creative Stage (0.5 hours):

[0135] Agent script generation includes a script with 5 action scenes;

[0136] Agent visually generates style parameters for "yellowish-brown tone + dilapidated building";

[0137] Agent Music creates an emotional axis of "tense electronic music + epic percussion";

[0138] 2. Virtual production (1 hour):

[0139] AIGC programmatically generates ruin scenes (raw images, multi-image references, previous and next frames, video-driven).

[0140] Digital actors automatically perform parkour / fighting actions as described in the motion description;

[0141] Virtual cameras automatically execute dynamic shots (such as drone dive tracking).

[0142] 3. Post-processing (2 hours):

[0143] Agent automatically edits and selects the best shot (based on motion continuity score);

[0144] Add explosion effects and particle effects to Agent effects;

[0145] Agent color grading applies a stylized LUT (to enhance the sandstorm effect);

[0146] Agent mixes to generate 5.1 surround sound (automatic spatialization of bullet trajectory sound effects).

[0147] 4. Quality inspection rendering (0.5 hours):

[0148] Automatic detection of technical parameters (4K / 24fps / color space);

[0149] Artistic consistency verification (action scene rhythm matches music beat);

[0150] Output a DCP format movie package + MP4 preview version;

[0151] Final deliverables:

[0152] 4K DCI (4096×2160) 24fps ProRes 4444 main video;

[0153] 1080p H.265 MP4 preview version;

[0154] SRT subtitle file (automatically generates dialogue subtitles);

[0155] XML Agent metadata report (including all Agent decision logs);

[0156] Attached documents:

[0157] JPG style reference image collection;

[0158] PDF sentiment curve chart;

[0159] TXT Technical Specifications Report.

[0160] Example 3: Parametric Model for Quantifying Creativity

[0161] Parametric models for quantifying creativity transform artistic styles into computable parameters. For example, the "cyberpunk" style can be parameterized as:

[0162] Hue: [0.2, 0.4, 0.8] (RGB values);

[0163] Contrast ratio: 1.8 (a multiple of standard contrast ratio);

[0164] Granularity: 0.6 (value in the range of 0-1);

[0165] Affective computing models map the emotions in a script to audiovisual parameters. For example, the emotion of "tension" can be mapped as:

[0166] Music: Fast tempo (120 BPM), dissonances, high pitch;

[0167] Lighting: Cool tones (primarily blue), low brightness, high contrast;

[0168] Editing: Fast transitions (<2 seconds / shot), dynamic angles;

[0169] Through this parametric model, ideas can be quantified into specific parameters to guide subsequent production and ensure the consistency and controllability of the ideas.

[0170] Example 4: Multi-Agent Decision-Making Autonomous System

[0171] The multi-agent decision-making autonomous system includes multiple professional agents such as director agent, editing agent, and sound effects agent, with each agent responsible for a production stage;

[0172] The system workflow is as follows:

[0173] 1. Each agent generates a decision-making plan based on its own expertise and objectives;

[0174] 2. When there is a conflict in decision-making among agents, multi-agent game algorithms such as Q-learning are used to reach a consensus through negotiation;

[0175] 3. Based on historical success stories, use reinforcement learning to optimize the production path, thereby improving production efficiency and quality;

[0176] 4. Each agent executes the final decision and completes the corresponding production task;

[0177] For example, the director agent might want to use a long take to express emotions, while the editing agent might think that fast cuts are more suitable for the rhythm. In this case, the system will use a multi-agent game algorithm to comprehensively consider the opinions of the two agents and reach the optimal solution, such as using a long take but containing fast cuts.

[0178] Example 5: Self-evolution mechanism of quality assurance

[0179] The self-evolution mechanism of quality assurance consists of three parts: generation module, evaluation module, and optimization module.

[0180] The system workflow is as follows:

[0181] 1. The generation module generates video content based on the current model;

[0182] 2. The evaluation module uses a multimodal evaluation model to assess the quality of the generated videos, including dimensions such as technical quality, artistic quality, and emotional expression;

[0183] 3. The optimization module optimizes the parameters of the generated model based on the evaluation results to improve the generation quality;

[0184] 4. Repeat the above steps to continuously optimize the generated model and achieve self-evolution.

[0185] In addition, the present invention also includes an anomaly detection algorithm to automatically identify and correct technical defects, such as continuity errors, color inconsistencies, and audio asynchrony, to ensure the professional quality of the output video.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fully automated film and television production system based on digital actors, characterized by: It includes a creative engine unit, a production unit, an intelligent shooting unit, a post-processing unit, and a quality inspection and rendering unit; The creative engine unit is responsible for transforming the user's input natural language requirements into structured creative content; The aforementioned production unit is responsible for transforming ideas into executable production plans; The intelligent shooting unit is responsible for performing shooting in the virtual environment; The post-processing unit is responsible for post-processing the footage obtained from the shooting. The quality inspection and rendering unit is responsible for performing quality checks and rendering output on the final video. The post-processing unit includes an automatic editing module, an intelligent color grading module, an automatic audio mixing module, and an intelligent special effects module; The automatic editing module uses multimodal AI to analyze video, audio, and scripts to automatically edit them; The specific implementation includes: receiving video footage, audio footage, and script as input; using multimodal AI to analyze the video, audio, and script to evaluate shot quality; optimizing video pacing based on music beats and emotional curves; automatically selecting the best shots to generate a rough cut EDL file; and outputting the edited video. The intelligent color grading module uses a style transfer network and a color science model to automatically grade colors. Specifically, it receives video footage and a visual style package as input; uses a style transfer network to transfer the style from the visual style package to the video footage; uses a color science model to automatically match the visual style package and optimize color consistency between shots; generates a color correction LUT and applies it to the video footage; and outputs the color-graded video. The automatic mixing module uses audio separation and an AI mixing model to automatically mix audio. Its specific implementation includes: receiving audio material as input; using audio separation technology to separate the mixed audio into tracks for dialogue, sound effects, and music; using an AI mixing model to automatically balance the proportions and spatial positions of dialogue, sound effects, and music; generating spatial audio based on scene 3D data to enhance immersion; and outputting a 5.1 mixed WAV file. The intelligent special effects module uses computer vision and generative AI to automatically generate special effects. Its specific implementation includes: receiving video footage and special effects requirements as input; using computer vision technology to automatically identify the areas or objects that need to have special effects added; using generative AI to programmatically generate special effects; automatically removing unwanted objects or fixing continuity errors; and outputting a VFX compositing layer for compositing with the original video. The specific implementation method of the shot quality scoring is as follows: computer vision technology is used to analyze video frames and evaluate composition quality; motion analysis technology is used to evaluate shot motion quality; performance analysis technology is used to evaluate the performance quality of digital actors; and the above scores are combined to generate a total shot quality score for shot selection. The specific implementation of the style transfer network is as follows: a deep learning model is used to transfer the style of the reference image to the video frame; temporal consistency technology is used to maintain style consistency between video frames and avoid flickering and abrupt changes; color space conversion technology is used to convert the style-transferred image back to the original color space; and color correction technology is used to fine-tune color parameters to ensure color accuracy and consistency. The specific implementation of the AI ​​mixing model is as follows: using a deep learning model to learn the mixing strategies of professional mixing engineers; adjusting mixing parameters based on emotion curves and music emotion timelines; using spatial audio technology to generate 3D audio effects; and using audio rendering technology to mix multi-track audio into the final 5.1 surround sound. The specific implementation of the procedural special effects generation is as follows: using a particle system to generate natural phenomenon special effects such as smoke, flames, and water flow; using physical simulation to simulate the physical effects of object movement, collision, and deformation; using neural network rendering to generate realistic lighting and material effects; and using image compositing technology to seamlessly integrate the special effects with the original video. The quality inspection and rendering unit includes an automated quality inspection process module and a rendering output module; The automated quality inspection process module performs technical indicator testing and artistic consistency testing on the final video, and automatically corrects problems. Its specific implementation includes: receiving the final video as input; performing technical indicator testing to check whether the technical parameters of resolution, frame rate, and color space meet the requirements; performing artistic consistency testing to verify whether the visual style and emotional curve are consistent with the original design; if problems are detected, using an automatic correction mechanism to fix them; and outputting a quality inspection report and the corrected video. The specific implementation of the artistic consistency detection is as follows: using computer vision technology to analyze the visual features of video frames; comparing the visual features with the original style parameter package to calculate the deviation; using sentiment analysis technology to analyze the emotional expression of the video and comparing it with the original sentiment curve; combining visual deviation and sentiment deviation to calculate the artistic consistency score; if the score is lower than the threshold, automatic correction is triggered. The rendering output module renders the final video into different formats and generates accompanying files. Its specific implementation includes: receiving the final video as input; selecting appropriate rendering parameters according to output requirements; performing rendering to generate video files in different formats; generating accompanying files, including SRT subtitles, XML metadata, JPG style reference atlas, PDF emotion curve charts, and TXT technical parameter reports; and outputting a complete digital asset package. The specific implementation of the rendering output is as follows: using distributed rendering technology to allocate rendering tasks to multiple rendering nodes to improve rendering efficiency; using color management technology to ensure color consistency across different output formats; using metadata management technology to record all decisions and parameters during the production process and generate a traceable production report; and using packaging technology to package all output files into a digital asset package for easy delivery and archiving.

2. The fully automated film and television video production system based on digital actors according to claim 1, characterized in that: The creative engine unit includes a script generation module, a visual design module, and a music design module; The script generation module uses a large language model to generate structured scripts and automatically embeds sentiment curve analysis through a sentiment computing model. The specific operation of the script generation module is as follows: receiving natural language requirements input by the user; generating script text based on the user requirements using a large language model; The script text is analyzed using an emotion computing model to generate an emotion curve, which includes the emotion intensity and emotion type at multiple time points. The script text and emotion curve are then integrated into a structured script, which is output in JSON format and includes fields such as scene number, shot suggestion, and emotion tag. The visual design module automatically matches the visual style based on the script content, generates reference images through AIGC tools, and outputs a style parameter package. The specific operations of the visual design module are as follows: analyze the script content using a large model to extract key information about the theme, emotions, and scenes; automatically match visual styles based on key information, including color schemes, composition styles, and material styles; and generate reference images using AIGC tools. Based on reference images and matched visual styles, a style parameter package is generated, including a color matrix, composition rules, and a material library. The music design module generates music style tags based on the script's emotional curve and generates corresponding music using AI. The specific operations of the music design module are as follows: generate music style tags based on the script's emotional curve, the music style tags include music type, rhythm, and mood; create a music emotion timeline and align the music style tags with the timeline; Based on the music emotion timeline and music style tags, AI is used to generate corresponding music; the output is a MIDI file and a sound library list. The MIDI file contains the structure and melody information of the music, and the sound library list contains the required sound information.

3. The fully automated film and television video production system based on digital actors according to claim 2, characterized in that: The specific operation of the sentiment computing model is as follows: use a pre-trained sentiment analysis model to perform sentiment analysis on the script text to obtain the sentiment vector of each paragraph or sentence; construct a sentiment curve based on the sentiment vector, the sentiment curve includes a time axis and a sentiment intensity axis, the sentiment intensity range is [-1,1], negative values ​​represent negative sentiment, and positive values ​​represent positive sentiment; align the sentiment curve with the script text, and assign sentiment labels to each scene or shot; The style parameter package includes a color matrix, composition rules, and a material library. The color matrix includes parameters such as primary color, secondary color, contrast, brightness, and saturation, represented in matrix form. The composition rules include rules for lens type, perspective, depth of field, and motion, represented in rule set form. The material library includes material parameters for various objects in the scene, including roughness, metallicity, and transparency, represented in material parameter set form. The method for generating the music emotion timeline is as follows: mapping the script's emotion curve onto the music emotion space, which includes dimensions of tension, activity, and pleasure; generating the music emotion timeline based on the emotion points in the music emotion space, which includes time points and corresponding music emotion tags; aligning the music emotion timeline with the script timeline to ensure consistency between the music emotion and the script emotion.

4. The fully automated film and television video production system based on digital actors according to claim 1, characterized in that: The production unit includes an automatic storyboard generation module, an automatic resource matching module, and a shooting plan optimization module; The automatic storyboard generation module automatically generates storyboards based on the script and visual rules. Specifically, it receives a structured script and visual rules from a style parameter package as input; uses the StoryboardAgent.generate algorithm to generate storyboards based on the script content and visual rules; the storyboards include scene numbers, shot types, shot parameters, composition, lighting, and performance content; and outputs a structured storyboard script to guide subsequent virtual filming. The automatic resource matching module automatically matches resources based on storyboard scenes and asset libraries. Its specific implementation involves: receiving storyboard scenes and asset libraries as input; using the ResourceAgent.allocate algorithm to automatically match resources based on the needs of the storyboard scenes and the content of the asset libraries; resources include digital actors, scenes, props, and special effects, and these asset library resources are mainly reusable materials generated by AI; outputting resource matching results, including a list of resources required for each scene and resource parameters. The shooting plan optimization module optimizes the shooting plan based on the storyboard scene and constraints. Specifically, it receives the storyboard scene and constraints as input and uses the ScheduleAgent.optimize algorithm to optimize the shooting plan based on the dependencies and constraints of the storyboard scene. The shooting plan includes the shooting order, shooting time, and shooting location. Output an optimized shooting plan to guide the execution of virtual shooting.

5. The fully automated film and television video production system based on digital actors according to claim 4, characterized in that: The specific implementation of the StoryboardAgent.generate algorithm is as follows: natural language processing technology is used to analyze the script content and extract key information about scenes, actions, and dialogues; based on visual rules, appropriate shot types and shot parameters are assigned to each scene; computer vision technology is used to pre-generate the composition and lighting effects of each shot; and reinforcement learning is used to optimize the coherence and artistry of the storyboard. The ResourceAgent.allocate algorithm is implemented as follows: semantic analysis is used to analyze the needs of the storyboard scene and extract the required resource types and features; based on the resource types and features, matching resources are searched in the asset library; a graph matching algorithm is used to evaluate the matching degree between the resources and the needs; the resource with the highest matching degree is selected and the resource parameters are adjusted to better meet the needs; for needs that cannot be matched, the material library can be directly generated according to the needs. The ScheduleAgent.optimize algorithm is implemented as follows: the storyboard scenes are modeled as graph nodes, and the dependencies between scenes are modeled as edges; the constraints are modeled as constraints in the graph; the constraint satisfaction problem algorithm is used to solve for the optimal shooting order; and a detailed shooting schedule is generated based on the shooting order and constraints.

6. The fully automated film and television video production system based on digital actors according to claim 1, characterized in that: The intelligent shooting unit includes a virtual shooting environment module, an automated execution module, an intelligent lighting module, and a performance generation module; The virtual shooting environment module constructs a virtual shooting environment, including scenes, digital actors, and equipment. Specifically, it is implemented as follows: a scene generation model is used to generate virtual scenes. This model, based on AIGC technology, can generate realistic 3D scenes according to the script description. AI-driven digital actors are used, whose expressions and movements are controlled by AI, enabling them to automatically perform corresponding acts according to the script description. A virtual camera system is used, which controls motion parameters through AI, automatically setting camera parameters according to the storyboard. The automated execution module automatically sets camera parameters and performs shooting based on the storyboard. Specifically, it receives the storyboard as input, automatically sets camera parameters, including focal length, aperture, and motion trajectory, based on the lens suggestions in the storyboard, performs virtual shooting to capture scene images, and outputs EXR sequences and depth channels for subsequent post-processing. The intelligent lighting module automatically sets up lighting based on a visual style package and dynamically adjusts the lighting according to emotion tags. Specifically, it receives a visual style package as input and automatically sets up lighting based on the color matrix and composition rules in the visual style package, including HDRI ambient light and key light. Dynamically adjust lighting based on emotion tags; render lighting effects in real time and output high-quality images; The performance generation module generates dialogue using LLM and matches performances using a motion capture library. Its specific implementation includes: receiving dialogue and action descriptions from the script as input; generating dialogue using LLM, including voice content and emotional expression; matching performances using a motion capture library and selecting the action sequence that best matches the action description; synthesizing facial expressions in real-time based on the LLM-generated text to control the facial expressions of digital actors; and outputting the performance data of digital actors for virtual filming.

7. The fully automated film and television video production system based on digital actors according to claim 6, characterized in that: The specific implementation of the scene-based generation model is as follows: using text-to-3D model generation technology to generate the basic structure of the 3D scene based on the script description; using procedural content generation technology to add details and textures to the 3D scene; using physically based rendering technology to add lighting and material effects to the 3D scene; and using neural rendering technology to optimize the visual effects of the 3D scene. The automatic setting method for camera parameters is as follows: using a rule engine, basic camera parameters are set based on lens type; Reinforcement learning is used to optimize camera parameters based on scene content and sentiment labels; path planning algorithms are used to generate camera motion trajectories; and motion control algorithms are used to smooth camera motion and avoid jitter and abrupt changes. The specific implementation of the automatic lighting is as follows: using image analysis technology, the reference image in the visual style package is analyzed to extract lighting information; based on the lighting information, basic lighting settings are generated, including ambient light, main light source, and auxiliary light source; Using physically based rendering technology, the interaction between light and materials is simulated to generate realistic lighting effects; reinforcement learning is used to optimize lighting settings based on emotion tags to enhance emotional expression. The specific implementation method of facial expression synthesis is as follows: using text sentiment analysis technology, analyzing the text generated by LLM, and extracting sentiment information; Based on emotional information, facial expression parameters are generated; a parameterized facial model is used to generate facial expressions based on the facial expression parameters. Use facial expression transition technology to smooth out changes in facial expressions and avoid abrupt and unnatural movements.

Images