Modular virtual production system

Abstract

This invention relates to the field of digital animation and discloses a modular virtual production system. The key technical features include: a performance capture module for acquiring motion data of target actors and generating a performance motion data stream; a scene construction module for generating a virtual environment; a director's console module providing a visual interactive page, receiving input commands from external control devices, and generating and outputting a motion data stream from a virtual camera; and a real-time rendering and compositing module for independently receiving the data stream, inputting the performance motion data stream and camera motion data stream into a Kalman filter algorithm, and decoupling the composite image motion into independent background motion layers and character motion layers using a reference frame transformation algorithm; locking a predetermined focus target in the decoupled character motion layer, maintaining focus depth when the camera motion data stream changes drastically based on an inertial prediction algorithm; and calculating motion blur based on the decoupled background motion layer and character motion layer respectively, and performing layered compositing.

Description

Technical Field

[0001] This invention relates to the field of digital animation, and more specifically, to a modular virtual production system. Background Technology

[0002] The production of animated characters is gradually shifting from traditional offline rendering to real-time virtual production. Existing modular virtual production systems typically include modules such as motion capture, scene building, real-time rendering, and director control, achieving preliminary real-time preview functionality through physical connections and basic data interaction.

[0003] However, in practical applications, especially when shooting dynamic shots involving complex camera movements, existing systems have the following technical shortcomings: Although the modules are physically connected, they are highly coupled in terms of data processing logic. In particular, when the director control module intervenes to adjust the virtual camera, the camera's motion data and the character's performance motion data are simply overlaid in the rendering module, resulting in chaotic image motion logic.

[0004] Existing autofocus algorithms are mostly based on center-of-view distance measurement or simple AI face recognition. When the virtual camera performs dynamic camera movements such as rapid panning and tracking shots, the camera's own violent movement interferes with the autofocus algorithm, making it unable to distinguish whether the camera is moving or the character is moving. This causes the focus to frequently switch between the character and the background, resulting in a focus hunting phenomenon. Furthermore, the computational load for motion blur increases dramatically, compromising real-time performance.

[0005] Due to the aforementioned issues, animation teams are often forced to abandon complex camera movements in real-time shooting and instead manually adjust focus and motion blur in post-production, which prevents the core advantages of modular virtual production from being fully utilized in high-quality animated works. Summary of the Invention

[0006] The purpose of this invention is to provide a modular virtual production system that solves the problems of focus disorder and low rendering efficiency caused by the mutual interference between virtual camera movement and character performance movement by designing a motion decoupling engine within the real-time rendering and compositing module.

[0007] The above-mentioned technical objective of the present invention is achieved through the following technical solution: a modular virtual film production system, comprising: The performance capture module is used to collect the motion data of the target actor in real time and generate a performance motion data stream of the digital character; The scene building module is used to generate virtual environments; The director console module provides a visual interactive interface for receiving input commands from external control devices and generating and outputting motion data streams from the virtual camera. The real-time rendering and compositing module includes: a motion data splitting interface for independently receiving the performance motion data stream and the camera motion data stream; a motion decoupling analysis unit for inputting the performance motion data stream and the camera motion data stream into a Kalman filter algorithm and decoupling the composite image motion into independent background motion layers and character motion layers through a reference frame transformation algorithm; an intelligent focus prediction unit for locking a predetermined focus target in the decoupled character motion layer and maintaining focus depth when the camera motion data stream changes drastically based on an inertial prediction algorithm; and a layered rendering execution unit for calculating motion blur based on the decoupled background motion layer and character motion layer respectively, performing layered compositing, and outputting a real-time composite image.

[0008] As a preferred embodiment of the present invention, the performance capture module includes an inertial motion capture unit, a facial capture unit, and an audio acquisition unit; the inertial motion capture unit is used to acquire the actor's skeletal motion data, the facial capture unit is used to acquire the actor's facial expression data and generate BlendShape driving parameters, and the audio acquisition unit is used to acquire the actor's voice data and synchronize it with the performance motion data stream in time.

[0009] As a preferred technical solution of the present invention, the motion decoupling analysis unit is provided with a reference system converter, which is used to convert the root bone displacement data in the performance motion data stream from the world coordinate system to a dynamic polar coordinate system with the optical center of the virtual camera as the origin.

[0010] As a preferred technical solution of the present invention, the inertial prediction algorithm is as follows: when the focus target moves out of the frame partially or completely due to camera movement, the current position is predicted based on historical trajectory data, and virtual focus distance information is generated.

[0011] As a preferred technical solution of the present invention, the layered rendering execution unit calculates motion blur for multiple independent character motion layers during operation, and synthesizes and outputs multiple character layers with the background layer.

[0012] As a preferred technical solution of the present invention, it also includes a data back-storage and post-processing bridging module, which is used to deconstruct the composite image output by the layered rendering execution unit according to the character layer mask, depth of field channel and motion vector channel, and generate an engineering file compatible with post-processing non-linear editing software.

[0013] As a preferred embodiment of the present invention, the motion decoupling analysis unit is configured with a motion confidence evaluation subunit, which is used to calculate the confidence weights of the performance motion data stream and the camera motion data stream; when a data stream experiences instantaneous packet loss or noise interference, the Kalman filtering algorithm dynamically adjusts the filtering parameters according to the confidence weights and selects the high-confidence data stream for motion decoupling.

[0014] As a preferred technical solution of the present invention, it also includes a virtual shooting preview unit, which is used to connect with the director console module, and while presenting the real-time composite image on an independent display device, it overlays and displays the position trajectory of the virtual camera, the current focus depth value, and the graphical identifier of the motion decoupling state.

[0015] In summary, the present invention has the following beneficial effects: This invention solves the problem of coupling interference between camera motion and character performance motion from the bottom layer of data processing by innovatively designing a motion decoupling engine within the real-time rendering and compositing module without changing the original modular system physical architecture.

[0016] By combining Kalman filtering with inertial prediction algorithms, immune focus tracking is achieved even with rapid camera movement. Even if the camera pans or circles rapidly, or the character briefly exits the frame, the focus remains stably locked on the target character.

[0017] The layered rendering mechanism breaks down the complex coupled motion blur calculation into independent, simple calculations, which can reduce the computational load in complex scenes with two characters and ensure real-time performance at high image quality.

[0018] The system supports multi-character and multi-motion layer processing capabilities, allowing directors to move the camera freely as if operating a real camera, while precisely controlling the focus of the narrative.

[0019] The data recovery function preserves the complete decoupled layered data, providing great flexibility for later fine-tuning. Attached Figure Description

[0020] Figure 1 This is a block diagram of the present invention. Detailed Implementation

[0021] It is readily understood that, based on the technical solution of this invention, various embodiments of the invention can be conceived by those skilled in the art without altering the essential spirit of the invention. Therefore, the following detailed embodiments and accompanying drawings are merely illustrative examples of the technical solution of this invention and should not be considered as the entirety of the invention or as limitations or restrictions on the technical solution of this invention. Rather, these embodiments are provided to enable those skilled in the art to gain a more thorough understanding of the invention. Preferred embodiments of the invention are described below in conjunction with the accompanying drawings, which form part of this application and, together with the embodiments of the invention, serve to illustrate the innovative concept of the invention. Example 1:

[0022] like Figure 1As shown, this embodiment demonstrates the real-time production process of a single character running and being followed by a virtual camera. The specific workflow of this invention is as follows: When creating a complex shot of a digital animated character running through ruins while a virtual camera rapidly follows and follows the character, the specific workflow of this invention is as follows: S1. System Initialization and Module Preparation: The operator starts the modular virtual film production system of this invention. The system performs a self-check of the communication status of each module.

[0023] The scene building module loads a pre-defined "post-apocalyptic ruins" 3D scene from the local asset library, including building models, ground textures, sky spheres, and lighting presets. The scene building module then packages the scene data into a format recognizable by the real-time rendering and compositing module, establishing the scene graph structure.

[0024] The performance capture module is calibrated. Actors wear inertial motion capture units containing 17 inertial sensors, head-mounted binocular cameras for facial capture, and lavalier microphones for audio acquisition. The system performs T-pose calibration on the motion capture units and expression base calibration on the facial capture units to ensure the accuracy of the acquired data.

[0025] The director's console module launches its visual interactive interface. A professional-grade virtual crane, i.e., an external control device, connects to the director's console module to complete communication handshake and coordinate system alignment.

[0026] The real-time rendering compositing module starts, the motion decoupling engine is initialized, the Kalman filter parameters are set to default values, and the layered rendering pipeline is ready.

[0027] The virtual shooting preview unit is activated on a separate director monitor, displaying the current virtual camera's field of view and overlaying system status information.

[0028] S2. Independent acquisition and transmission of dual-channel motion data: The actors began their running performance.

[0029] This step involves two parallel data acquisition and transmission processes: Performance sports data collection path: The inertial motion capture unit acquires the actor's skeletal motion data in real time at a sampling rate of 120Hz, including the position coordinates of the root bones in the world coordinate system, the root bone rotation quaternions, and the rotation angles of each joint. The facial capture unit acquires the actor's facial expression data at a sampling rate of 60Hz, generating 46 BlendShape driving weight parameters through a facial feature point tracking algorithm. The audio acquisition unit acquires the actor's speech data at a sampling rate of 48kHz.

[0030] The time synchronizer inside the performance capture module adds a unified timestamp to each frame of data, encapsulates the skeletal data, BlendShape data, and audio data into a performance motion data stream, and sends it to the motion data distribution interface of the real-time rendering and compositing module through a zero-latency fiber optic transmission protocol.

[0031] Camera motion data acquisition path: The director operates a virtual camera crane to perform surround shots. The crane's control parameters, such as panning angle, tilt angle, push-pull speed, and rotational acceleration, are converted into motion data for the virtual camera in real time, including the camera's position in the world coordinate system, the camera's Euler angles of rotation, the camera's velocity vector, and the camera's angular velocity.

[0032] The intelligent camera movement control subunit of the director's console module encapsulates these parameters into a camera motion data stream and sends it to the same motion data splitter interface through an independent control channel, such as using a real-time transmission protocol based on timestamp synchronization.

[0033] S3. Motion data splitting and preprocessing: After receiving two data streams, the motion data splitting interface of the real-time rendering compositing module performs the following operations: First, the two data streams are time-aligned based on the timestamps of the data packets. Since the performance motion data stream and the camera motion data stream come from different acquisition devices, there may be slight clock deviations. The split interface uses an interpolation algorithm to align the two data streams onto the same timeline.

[0034] Secondly, the raw data undergoes filtering preprocessing. For the skeletal data in the performance motion data stream, a low-pass filter is used to remove high-frequency noise; for the camera motion data stream, a moving average filter is used to smooth instantaneous jitter.

[0035] Finally, the two preprocessed data streams are stored in two separate buffers, awaiting retrieval by the motion decoupling analysis unit. This independent storage design ensures that the two data streams remain logically separate before entering the core algorithm, avoiding mutual interference.

[0036] S4. Motion decoupling analysis, which is also a core step of this invention: The motion decoupling analysis unit reads the aligned two data streams from the buffer and executes the core decoupling algorithm: S4.1 Reference Frame Transformation: The motion decoupling analysis unit calls the reference frame converter to transform the root bone displacement data in the performance motion data stream from the world coordinate system to a dynamic polar coordinate system with the virtual camera's optical center as the origin. The specific transformation algorithm is as follows: Let the position of the root skeleton in the world coordinate system be P. r (t)=(xr ,y r ,z r The virtual camera's position in the world coordinate system is P. c (t)=(x c ,y c ,z c The camera's orientation is represented by the orientation vector V. c (t).

[0037] Calculate the relative position vector: P rel (t)=P r (t)-P c (t) Transform the relative position vector to the camera's local coordinate system: P local (t)=R c (t) -1 ·P rel (t), where R is a t), c (t) is the rotation matrix of the camera.

[0038] Further converted to polar coordinates: radial distance ρ(t) = |P local (t)|,Azimuth θ(t)=atan2(P local .y,P local .x), elevation angle φ(t) = asin(P local .z / ρ(t)).

[0039] The significance of this conversion is that, after the conversion, the character's movement is represented as "movement relative to the camera's perspective," rather than "absolute movement in the world." This facilitates the subsequent separation of camera movement.

[0040] S4.2, Kalman filter separation; The reference frame transformed data is input into the Kalman filter along with the original camera motion data. The Kalman filter uses a standard state-space model, and the state vectors include: background motion vector, character motion vector, and camera motion vector. The observation vector is the optical flow field of the synthesized image.

[0041] The Kalman filter estimates the optimal value of the state vector in real time through prediction-update iteration. Its core principle is to use the known motion information in the camera motion data stream as the control input, that is, the speed and acceleration of the camera's active camera movement. This contribution is subtracted from the composite optical flow field, and the remaining part is the contribution of the character's motion.

[0042] After Kalman filtering, the motion decoupling analysis unit outputs two independent motion layers: Background motion layer: Contains optical flow information of the background scene caused by camera movement, which is represented by the overall movement direction and speed of the background texture in the image.

[0043] Character motion layer: Contains information on changes in the character's outline and displacement caused by the character's own performance movements, and is represented as a local motion vector field driven by the character's skeletal animation.

[0044] In this embodiment, the camera rotates at an angular velocity of 120° / second, and the background motion layer is represented by the ruins background flowing rapidly in the opposite direction; the character's running speed is 3m / s, and the character motion layer is represented by the periodic motion vectors of the character's limbs swinging. The two were successfully separated.

[0045] S5. Motion confidence assessment and fault tolerance handling: During data acquisition, momentary packet loss or noise caused by electromagnetic interference may occur. The motion confidence assessment subunit continuously monitors the signal quality of both data streams. For performance motion data streams, confidence assessment is based on the following indicators: signal strength of the motion capture unit, occlusion detection flags, and physical plausibility of the skeletal data. For camera motion data streams, confidence assessment is based on: the communication packet loss rate of the control device and whether the rate of change of motion parameters is within a reasonable range.

[0046] When an anomaly is detected in a data stream, the motion confidence assessment subunit calculates the performance data weights w in real time. p and camera data weights w c The weight range is 0-1, and w p +w c =1.

[0047] The Kalman filter algorithm dynamically adjusts the filter parameters based on confidence weights. For example, when a performance motion data stream experiences momentary packet loss due to a brief obstruction by an actor, w p Reduced to 0.3, w c The value increases to 0.7. At this point, the Kalman filter relies more on the camera motion data stream for prediction, using historical states and camera motion information to infer approximate character motion until the performance data is recovered. This fault-tolerant mechanism ensures stable operation of the system under non-ideal conditions.

[0048] S6, Intelligent Focus Prediction and Locking: The intelligent focus prediction unit receives the decoupled character motion layer and executes the focus locking algorithm: S6.1, Focus Target Identification; The system defaults to using the facial region in the character motion layer as the predetermined focus target. In this embodiment, the actor's facial capture data provides the locations of facial feature points, which are mapped onto the virtual character model to form a set of 3D tracking points. The intelligent focus prediction unit selects the tip of the nose as the primary tracking point.

[0049] S6.2, Execution of the inertial prediction algorithm: When the camera quickly moves behind the character, the character is briefly obscured, and the facial tracking points disappear from the frame. At this moment, the inertial predictor activates: Read historical trajectory data: Obtain the motion trajectory of the focus target within the past second, including the position sequence P. f (t-Δt),P f (t-2Δt),...

[0050] Calculate motion trend: Perform linear or quadratic fitting on the historical trajectory to obtain the motion velocity vector v. f and acceleration vector a f .

[0051] Predict current position: based on the kinematic formula P pred (t)=P f (t-Δt)+v f Δt+0.5 a f (Δt)^2, predicts the position of the focal target at the current moment.

[0052] Generate virtual focus distance: Calculate the distance d from the virtual camera position to the predicted position. pred =|P pred (t)-P c (t)|, this distance is sent to the rendering pipeline as virtual focus distance information.

[0053] S6.3 Focus switching logic: The intelligent focus prediction unit continuously monitors the camera's motion. When the camera motion stabilizes, for example, when the angular velocity is below 5° / second for 0.5 seconds, the system allows the director to manually trigger a focus switch via the director's console module. In this embodiment, since it is a single-character shot, there is no need to switch the focus; the focus remains locked on the running character.

[0054] After this step, even though the character was briefly obscured by foreground objects multiple times during the camera's orbit, the focus of the image remained stable, without the "bellows" focus search phenomenon commonly seen in traditional systems.

[0055] S7, Layered Rendering and Motion Blur Calculation: The layered rendering execution unit receives the decoupled background motion layer and character motion layer, and performs layered rendering: S7.1 Layer Assignment: The system allocates two independent rendering channels: Channel A is responsible for rendering the background layer, and Channel B is responsible for rendering the character layer. Each channel has its own independent frame buffer.

[0056] S7.2 Motion fuzzy calculation: Background layer motion blur calculation: Using velocity information from the camera's motion data stream, the motion vector of each pixel is calculated. Specifically, for each pixel in the background layer, based on the camera's speed and direction, the pixel's trajectory within the exposure time is calculated. Sampling and blending are then performed along this trajectory to generate a motion blur effect.

[0057] Motion blur calculation for the character layer: Using skeletal animation information from the character's performance motion data stream, the motion vector of each vertex is calculated. The specific algorithm is as follows: For each vertex on the character model, the position of the current frame is compared with the position of the previous frame to calculate the vertex displacement vector; the vertex displacement vector is interpolated into the pixel shader to generate a pixel-by-pixel motion vector field; sampling and blending are performed along the motion vector field to generate the motion blur effect of the character layer.

[0058] S7.3 Depth of field effect processing: The current focus depth information provided by the intelligent focus prediction unit is passed to the depth-of-field post-processing effect unit. The system generates a Gaussian blur kernel based on the focus depth, applying varying degrees of blur to areas in the image far from the focus point to simulate the shallow depth-of-field effect of a real camera. In this embodiment, the focus is always locked on the character, thus keeping the character sharp while the foreground and background ruins exhibit a natural blurring effect.

[0059] S7.4, Layer Composition: The layered rendering execution unit calls the compositor to perform alpha blending on the rendering results of the two layers. Since the two layers are rendered independently, the compositing process only requires a simple overlay operation, resulting in minimal computational overhead. The final composited image is then written to the display buffer.

[0060] The entire layered rendering process fully utilizes the parallel computing power of the GPU. At 4K resolution, the frame rate is stably maintained at 60fps, meeting the performance requirements of real-time virtual production.

[0061] S8, Virtual Shooting Preview and Real-time Monitoring: While the virtual shooting preview unit displays the composite image in real time on the director's monitor, it also overlays the following monitoring information at the edge of the image: Virtual camera position trajectory: Displays the camera's position and direction of movement in the scene in the form of a minimap.

[0062] Current focus depth value: Displays the distance to the current focus point in numerical form, in meters.

[0063] Focus target identifier: The location of the currently locked focus target is marked with a green box in the image.

[0064] Motion decoupling status: A graphical progress bar displays the separation degree between the background motion layer and the character motion layer. Green indicates good decoupling, yellow indicates slight coupling, and red indicates anomaly.

[0065] The director monitors the shooting effects in real time via a monitor and adjusts the shooting plan as needed. In this example, the director observed that the excessively fast rotation speed caused overly strong background motion blur, and immediately slowed down the rotation speed by using the camera crane, thus adjusting the image effect in real time.

[0066] S9, Data Back-to-Work and Post-Processing Bridging: After shooting is completed, the data recovery and post-production bridging module performs the following operations: S9.1, Layered Data Export: The system will deconstruct the original composite footage captured in this operation into layers according to the following channels: Character layer mask channel: Records the mask information of which character or background each pixel belongs to.

[0067] Depth of field channel: Records the depth value of each pixel, used for adjusting the depth of field effect in post-processing.

[0068] Motion Vector Channel: Records the motion vector of each pixel, used for later adjustment of motion blur effect.

[0069] Diffuse channel, specular channel, and shadow channel: These store the basic properties of the material for later relighting.

[0070] S9.2 Metadata Generation: The metadata generator packages director's instructions, timeline markers, and focus switching information during filming into an XML metadata file. The metadata content includes: keyframe data of camera movement, focus lock time intervals, and director's comments.

[0071] S9.3, Project File Generation: The post-production bridging module generates project files compatible with mainstream post-production non-linear editing software based on layered data and metadata. In the project files, each layer is automatically placed on different video tracks, markers in the metadata are converted into timeline markers, and director's annotations are converted into comment tracks.

[0072] In post-production, animators can make detailed adjustments based on these layered data without having to start compositing from scratch, greatly improving post-production efficiency.

[0073] S10. Output of Results: Through the steps S1-S9 described above, this embodiment successfully achieved real-time production of a single-character running, circling follow-shot. The final output includes: A real-time synthesized 4K resolution 60fps video file.

[0074] The data files of each channel after layered deconstruction.

[0075] Project files containing shooting metadata.

[0076] Keyframe marker files recorded during the director's monitoring process.

[0077] The entire shooting process is time-consuming, but the efficiency is significantly improved compared to traditional post-production compositing methods. Example 2:

[0078] This embodiment demonstrates the extended application of the present invention in creating complex shots of two digitally animated characters fighting on a mobile platform, focusing on focus switching and handling complex motion blur in scenes with two characters fighting. For the sake of brevity, the steps identical to those in Embodiment 1 will not be repeated; only the special features of this embodiment will be described.

[0079] S1. Multi-role data access: Two actors, A and B, each wearing a performance capture module, simultaneously perform a fight scene. The performance capture module generates an independent performance motion data stream for each character: Character A Data Stream: Includes actor A's skeletal motion data, facial expression data, and voice data.

[0080] Character B Data Stream: Includes actor B's skeletal motion data, facial expression data, and voice data.

[0081] Two data streams are sent to the motion data splitter interface of the real-time rendering compositing module via fiber optic transmission protocol, and the interface allocates an independent buffer for each character.

[0082] S2, Multi-role motion decoupling: The motion decoupling analysis unit receives three data streams: camera motion data stream, character A data stream, and character B data stream. A Kalman filter performs multidimensional decoupling, expanding the state vector to include four components: background motion, character A motion, character B motion, and possible inter-character interaction motion.

[0083] The decoupling result is: Background motion layer: The moving platform and its surrounding environment, affected by the slow, circular motion of the camera.

[0084] Character A's movement layer: Character A's punching and dodging actions include rapid arm movements and changes in torso center of gravity.

[0085] Character B's movement layer: Character B's offensive and balance adjustment actions, including jumping and landing impacts.

[0086] S3, Intelligent Focus Prediction and Active Switching: The director marks focus switching points on the timeline through the interactive interface of the director console module: the focus is locked on character A's face from 0 to 5 seconds, and then switches to character B's face after 5 seconds.

[0087] Phase 0-5 seconds: The intelligent focus prediction unit locks onto the facial tracking points in the motion layer of character A. Even though character A moves rapidly during combat, and even briefly moves to the edge of the frame, the focus prediction unit maintains a stable lock on character A. Although the moving platform in the background is also in motion, it belongs to the background layer after motion decoupling and does not affect focus calculation.

[0088] Switching point at 5 seconds: When the timeline reaches the 5th second, the director's console module sends a focus switching command. Upon receiving the command, the intelligent focus prediction unit executes a smooth switching algorithm: Complete the final focus depth calculation for character A in the current frame.

[0089] At the start of the next frame, switch the focus lock target to the facial tracking point in character B's motion layer.

[0090] During the switching process, the focus depth value is linearly interpolated to transition, avoiding abrupt changes.

[0091] The switching process is smooth and flicker-free because the motion layers of the two characters are always separate and stable. The focus prediction unit only needs to switch the tracking target and does not need to search for focus again.

[0092] S4, Multi-character layered rendering: The layered rendering execution unit calculates motion blur for the background layer, character A layer, and character B layer respectively: Background layer: Motion blur is calculated using the camera motion data stream. Due to the camera's slow rotation, the background layer motion blur appears as a slight circular trail.

[0093] Character A Layer: Motion blur is calculated using the performance motion data stream of Character A. Character A's punching motion is extremely fast, and the motion blur is manifested as a linear stretching of the arm trajectory.

[0094] Character B Layer: Motion blur is calculated using the performance motion data stream of Character B. Character B's jumping motion generates full-body motion blur, with the blur direction and intensity adaptively changing according to the movement speed of different parts of the body.

[0095] Compositing stage: The motion vector compositor overlays the motion vectors of the three layers at the pixel level to generate a composite motion vector field. When character A's arm moves past character B, the motion vectors of the two layers are overlaid at that pixel position, producing the correct interactive motion blur effect.

[0096] S5. Results Evaluation: This embodiment successfully achieved real-time shooting of complex fight scenes involving two characters. Compared with traditional methods, the technical advantages of this invention are as follows: Precise focus switching: Directors can preset focus switching points on the timeline just like editors, without worrying about camera movement interference.

[0097] Realistic motion blur: The motion blur of each character is calculated independently, so even if two people move in completely different directions and at completely different speeds, it can still be rendered correctly.

[0098] Superior real-time performance: The computational load of layered rendering with three-way data is far lower than that of traditional coupled rendering, maintaining a frame rate of 60fps throughout. Example 3:

[0099] This embodiment demonstrates how the fault-tolerant mechanism of the present invention ensures stable system operation when data packet loss occurs due to momentary signal obstruction in the performance capture module during actual shooting.

[0100] Scene setting: During a rapid turn, the actor's body briefly obscured some of the motion capture unit's sensors, causing a 0.2-second packet loss in the performance's motion data stream. Simultaneously, the director was operating a virtual camera to perform a rapid panning motion.

[0101] S1. Anomaly Detection: The motion confidence assessment subunit continuously monitors the data stream quality. When three consecutive frames of missing data are detected in the performance motion data stream, the confidence weight is immediately calculated. Performance sports data confidence weight w p It dropped sharply from the normal value of 0.9 to 0.2.

[0102] Camera motion data confidence weight w c Considering that there is also slight noise in the camera data, the value was adjusted from 0.9 to 0.8.

[0103] S2. Dynamic adjustment of Kalman filter parameters: The Kalman filter receives updated confidence weights and dynamically adjusts the measurement noise covariance matrix R and the process noise covariance matrix Q. The specific adjustment strategy is as follows: Increase the measurement noise covariance corresponding to the performance motion data, and reduce the contribution weight of this data to the state update.

[0104] Keep the measurement noise covariance corresponding to the camera motion data constant, and maintain its contribution weight to state updates.

[0105] Increasing the process noise covariance allows for some uncertainty in state prediction, in order to cope with possible sudden changes in motion.

[0106] S3. State Prediction and Data Filling: During the 0.2 seconds of missing performance data, approximately 12 frames, the Kalman filter utilizes camera motion data and historical state information to perform a pure prediction mode: Based on the estimated state vector from the previous frame and camera motion data, predict the state of the current frame.

[0107] The prediction results include estimates for the background motion layer and the character motion layer.

[0108] The predicted values ​​for the character motion layer are calculated based on a combination of historical character motion trends and the influence of camera motion.

[0109] S4. Data Recovery and Smooth Transition: After 0.2 seconds, the performance motion data stream returned to normal. The confidence assessment subunit detected the data recovery and gradually increased the confidence level. p The value increases linearly from 0.2 back to 0.9, with a transition time of 0.1 seconds (approximately 6 frames).

[0110] The Kalman filter employs a weighted fusion strategy during the transition period: Output value = w p Measurement update value + (1-w) p ) Predicted value. With w p As the value gradually increases, it smoothly transitions to being entirely based on the measurement data.

[0111] S5. Results Evaluation: Thanks to the fault-tolerance mechanism, despite the brief loss of performance data, almost no visible anomalies appeared in the final composite footage. The character's movement maintained its continuity during the packet loss, without any stuttering or jumps. The director and actors were completely unaware that the data loss had occurred.

[0112] This embodiment demonstrates the robustness and reliability of the invention under non-ideal shooting conditions.

[0113] The modular virtual film production system described in this invention can be widely applied in the following scenarios: Animated film production: Virtual shooting stage applicable to feature-length animated films, allowing directors to complete shot design and focus adjustment during the live-action shooting phase, greatly reducing rework in post-production.

[0114] Game cutscene creation: Suitable for real-time cutscene creation in AAA games, and can directly output in engine-compatible formats.

[0115] Virtual live streaming and concerts: Suitable for scenarios requiring real-time interaction, such as virtual idol live streaming and virtual concerts, ensuring image quality while reducing computational load.

[0116] Advertising and MV Production: Suitable for commercial short film production that requires rapid iteration of creative ideas, shortening the production cycle.

[0117] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this invention should be included within the protection scope of this invention.

[0118] It should be understood that, in order to simplify the present invention and help those skilled in the art understand its various aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes described in a single embodiment or with reference to a single figure. However, the present invention should not be construed as including all features in the exemplary embodiments as essential technical features of the claims of this patent.

[0119] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0120] It should be understood that the modules, units, components, etc., included in the device of one embodiment of the present invention can be adaptively changed to be placed in a device different from that embodiment. Different modules, units, or components included in the device of the embodiment can be combined into a single module, unit, or component, or they can be divided into multiple sub-modules, sub-units, or sub-components.

[0121] The modules, units, or components in the embodiments of the present invention can be implemented in hardware, in software running on one or more processors, or in a combination thereof. Those skilled in the art should understand that... In practice, microprocessors or digital signal processors (DSPs) can be used to implement embodiments of the invention. The invention can also be implemented on computer program products or computer-readable media for performing some or all of the methods described herein.

Claims

1. A modular virtual production system, characterized by: include: The performance capture module is used to collect the motion data of the target actor in real time and generate a performance motion data stream of the digital character; The scene building module is used to generate virtual environments; The director console module provides a visual interactive interface for receiving input commands from external control devices and generating and outputting motion data streams from the virtual camera. The real-time rendering and compositing module includes: a motion data splitting interface for independently receiving the performance motion data stream and the camera motion data stream; a motion decoupling analysis unit for inputting the performance motion data stream and the camera motion data stream into a Kalman filter algorithm and decoupling the composite image motion into independent background motion layers and character motion layers through a reference frame transformation algorithm; an intelligent focus prediction unit for locking a predetermined focus target in the decoupled character motion layer and maintaining focus depth when the camera motion data stream changes drastically based on an inertial prediction algorithm; and a layered rendering execution unit for calculating motion blur based on the decoupled background motion layer and character motion layer respectively, performing layered compositing, and outputting a real-time composite image.

2. The modular virtual film production system according to claim 1, characterized in that: The performance capture module includes an inertial motion capture unit, a facial capture unit, and an audio acquisition unit. The inertial motion capture unit is used to collect the actor's skeletal motion data, the facial capture unit is used to collect the actor's facial expression data and generate BlendShape driving parameters, and the audio acquisition unit is used to collect the actor's voice data and synchronize it with the performance motion data stream in time.

3. The modular virtual film production system according to claim 2, characterized in that: The motion decoupling analysis unit is equipped with a reference system converter, which is used to convert the root bone displacement data in the performance motion data stream from the world coordinate system to a dynamic polar coordinate system with the optical center of the virtual camera as the origin.

4. A modular virtual film production system according to claim 3, characterized in that: The inertial prediction algorithm is as follows: when the focus target partially or completely moves out of the frame due to camera movement, its current position is predicted based on historical trajectory data, and virtual focus distance information is generated.

5. A modular virtual film production system according to claim 4, characterized in that: During runtime, the layered rendering execution unit calculates motion blur for multiple independent character motion layers and then composites and outputs the multiple character layers with the background layer.

6. A modular virtual film production system according to claim 5, characterized in that: It also includes a data back-storage and post-production bridging module, which is used to deconstruct the composite image output by the layered rendering execution unit according to the character layer mask, depth of field channel, and motion vector channel, and generate a project file compatible with post-production non-linear editing software.

7. A modular virtual film production system according to claim 6, characterized in that: The motion decoupling analysis unit is equipped with a motion confidence evaluation subunit, which is used to calculate the confidence weights of the performance motion data stream and the camera motion data stream. When a data stream experiences instantaneous packet loss or noise interference, the Kalman filter algorithm dynamically adjusts the filtering parameters according to the confidence weights and selects the high-confidence data stream for motion decoupling.

8. A modular virtual film production system according to claim 7, characterized in that: It also includes a virtual shooting preview unit, which is used to connect to the director console module and present the real-time composite image on an independent display device while overlaying and displaying the position trajectory of the virtual camera, the current focus depth value, and the graphical indicators of the motion decoupling status.

Images