Collaborative construction and rendering method of film and television latent feature field and related device

Abstract

The application provides a kind of synergistic movie and television latent feature field construction and rendering method and related device, it is related to computer graphics field. Through the deployment synchronous camera equipment, extraction multiscale feature, multi-view sampling and alignment, time sequence smoothing and feature completion, construct the latent feature field of space-time coherence, and support arbitrary virtual perspective rendering, solve the space-time inconsistency, flicker and ghost problem in dynamic scene, have the advantages such as improving the space-time coherence of video rendering, eliminating flicker and ghost, support arbitrary perspective rendering, improve the efficiency and quality of film and television production.

Description

Technical Field

[0001] This application relates to the field of computer graphics, and in particular to a collaborative method and apparatus for constructing and rendering latent feature fields in film and television. Background Technology

[0002] Current film and animation generation technologies suffer from multiple limitations. Traditional 3D modeling and texturing methods heavily rely on manually setting keyframes and physical engine simulations, resulting in a cumbersome and lengthy production process. When handling dynamic scenes such as rapid actor movement or complex material interactions, they struggle to capture and render details in real time, leading to insufficient dynamic expressiveness. While generative video AI technology can generate high-resolution images, its generation mechanism lacks explicit constraints on the scene's 3D geometry. When the camera's perspective shifts or rotates significantly, the continuity between video frames is disrupted, resulting in spatiotemporal inconsistencies such as distorted object shapes and structural breaks—a phenomenon known as "spatiotemporal flicker," which severely impairs visual coherence. Furthermore, neural radiance field-based solutions, such as 3D Gaussian Splatting and Neural Radiance Fields, primarily optimize static scenes or short-term data, failing to establish a logical correlation model across time dimensions. In multi-camera collaborative shooting, when actors' limbs or props obstruct the viewpoint, the system cannot effectively infer the feature information of the obscured area, leading to a "ghosting" phenomenon with missing features and severe artifact interference, significantly reducing reconstruction quality. These technological bottlenecks collectively restrict the efficiency and realism of film and television content production.

[0003] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention

[0004] This application provides a collaborative method and apparatus for constructing and rendering latent feature fields in film and television, which has the advantages of improving the spatiotemporal coherence of video rendering, eliminating flicker and ghosting, supporting rendering from any virtual perspective, and improving the efficiency and quality of film and television production.

[0005] Firstly, the collaborative film and television latent feature field construction and rendering method provided in this application adopts the following technical solution: A collaborative method for constructing and rendering latent feature fields in film and television includes: Deploy synchronously triggered camera equipment in the shooting area, complete the calibration of the camera equipment's internal and external parameters and clock synchronization, and acquire multiple RGB image streams with globally unified timestamps; The multi-channel RGB image stream is input into a deep encoder, multi-scale feature maps are extracted through a convolutional network, and the pixels in the feature maps are transformed into high-dimensional latent vectors containing color, texture, and material attributes. The 3D space of the actors' activities is discretized into a set of point clouds. The spatial points are projected onto the imaging planes of each camera using the intrinsic and extrinsic parameter matrices of the camera equipment. The feature maps of each imaging plane are sampled from multiple perspectives to obtain a multi-dimensional feature set of the spatial points. The multidimensional feature set is input into the alignment network to complete the viewpoint disambiguation. The credibility score of each viewpoint feature is calculated by MLP, and the aligned spatial point latent vector is obtained by weighted fusion. The latent point cloud in 3D space is serialized and input into the Mamba module. Selective state transition is completed through the state space equation to achieve temporal smoothing of features and feature completion in occluded scenes, resulting in a spatiotemporally coherent latent feature field. The intrinsic and extrinsic parameters and pose parameters of the virtual camera are set. The spatiotemporally coherent latent feature field is resampled within the virtual camera's view frustum. The sampled latent feature map is input into the neural decoder to restore and output a high-quality RGB video stream.

[0006] Optionally, the camera equipment consists of N industrial cameras with synchronous triggering function, and the intrinsic and extrinsic parameter calibration is as follows: intrinsic parameter calibration obtains the focal length of each industrial camera. Principal point coordinates and radial distortion coefficient; Extrinsic parameter calibration determines the rotation matrix of each industrial camera in the world coordinate system through feature point alignment. Translation vector ; The clock synchronization is achieved using an external clock generator, ensuring that the exposure error of all industrial cameras on the time axis t is less than 10. -6 The time is specified in seconds, and a globally uniform timestamp label is assigned to each frame of the image.

[0007] Optionally, the convolutional network is a convolutional network with a feature pyramid (FPN) structure, and the extracted multi-scale feature maps have a dimension of [missing value]. ; The high-dimensional latent vector is a C-dimensional vector that encodes the color information, local texture gradient information, and material attribute feature information of the pixel.

[0008] Optionally, the steps of discretizing the 3D space of the actor's activities into a point cloud set, projecting the spatial points onto the imaging planes of each camera device using the intrinsic and extrinsic parameter matrices of the camera devices, and performing multi-view sampling on the feature maps of each imaging plane to obtain a multi-dimensional feature set of the spatial points include: The 3D space for actor movement is divided into a point cloud set V. For any point in the space... Using the calibrated intrinsic and extrinsic parameter matrices, through the formula Project it back onto the imaging plane of each industrial camera; The feature set is obtained by sampling the projection points from the N feature maps corresponding to N industrial cameras. This forms a multidimensional feature description of the spatial point; in : Represents a spatial point The projection coordinates on the imaging plane of the i-th camera device, Let (x, y, z) represent any discrete point in the 3D space where the actor is active, with its world coordinates represented as (x, y, z). This represents the intrinsic parameter matrix of the i-th camera device. This represents the rotation matrix of the i-th camera relative to the world coordinate system. Let represent the translation vector of the i-th camera relative to the world coordinate system.

[0009] Optionally, the steps of inputting the multidimensional feature set into the alignment network to complete viewpoint disambiguation, calculating the confidence score of each viewpoint feature through MLP, and obtaining the aligned spatial point latent vector through weighted fusion include: inputting the sampled feature set into the alignment network to complete viewpoint disambiguation; Calculate the credibility score of features from each perspective using MLP. When limb occlusion causes a conflict in feature logic, the credibility score of the corresponding occluded camera position. Approaching 0; Using the weighted summation formula Achieving consistent fusion of multi-view features yields the aligned spatial point latent vectors at time t. .

[0010] Optionally, the process of serializing the latent point cloud in 3D space and inputting it into the Mamba module, performing selective state transitions through state-space equations to achieve temporal smoothing of features and feature completion in occluded scenes, and obtaining a spatiotemporally coherent latent feature field includes: Hilbert curves are used to unfold the potential point cloud in 3D space into a one-dimensional token sequence according to spatial proximity. The one-dimensional token sequence is input into the Mamba module, and the state-space equation is used to process it. Complete selective state transitions, where Let be the hidden state at time t. The hidden state at time t-1 This represents the latent vector of spatial points in the original one-dimensional token sequence input to the Mamba module. This represents the spatiotemporally coherent latent vector output after temporal smoothing and feature completion by the Mamba module. This represents the state transition matrix, which controls the evolution logic of the hidden states. C represents the input control matrix, responsible for mapping the current time-state features to the hidden state space. C represents the output matrix, which remaps the updated hidden state back to the feature space. This represents a residual connection term, used to directly superimpose input features onto the output to ensure training stability and feature fidelity. By hiding state Memory feature history trajectory to eliminate visual flicker caused by inter-frame sampling differences, when all camera devices are occluded at time t, according to The motion logic is used to predict and complete the latent features at time t.

[0011] Optionally, the steps of specific viewpoint sampling and neural decoding rendering include: Determine the intrinsic parameter matrix of any virtual camera based on the requirements of film and television rendering. Rotation matrix Translation vector ; Within the view frustum of the virtual camera, the spatiotemporally coherent latent feature field output by the Mamba module is resampled. The sampled latent feature map is input into the neural decoder to restore and output a high-fidelity, flicker-free, high-quality RGB video stream. The latent feature field supports multiple resampling rendering from any virtual viewpoint.

[0012] Secondly, this application provides a collaborative film and television latent feature field construction and rendering system, including: The data acquisition module is used to deploy synchronously triggered camera equipment in the shooting area, complete the calibration of the camera equipment's internal and external parameters and clock synchronization, and acquire multiple RGB image streams with a globally unified timestamp; The data conversion module is used to input the multi-channel RGB image stream into the depth encoder, extract multi-scale feature maps through the convolutional network, and convert the pixels in the feature maps into high-dimensional latent vectors containing color, texture and material attributes. The set generation module is used to discretize the 3D space of the actor's activities into a set of point clouds. The spatial points are projected onto the imaging planes of each camera device using the intrinsic and extrinsic parameter matrices of the camera devices. The feature maps of each imaging plane are sampled from multiple perspectives to obtain a multi-dimensional feature set of the spatial points. The vector generation module is used to input the multidimensional feature set into the alignment network to complete the viewpoint disambiguation, calculate the confidence score of each viewpoint feature through MLP, and obtain the aligned spatial point latent vector through weighted fusion. The feature generation module is used to serialize the latent point cloud in 3D space and input it into the Mamba module. Selective state transition is completed through the state space equation to achieve temporal smoothing of features and feature completion in occluded scenes, resulting in a spatiotemporally coherent latent feature field. The output module is used to set the intrinsic and extrinsic parameters and pose parameters of the virtual camera, resample the spatiotemporally coherent latent feature field within the virtual camera's view frustum, input the sampled latent feature map into the neural decoder, and restore and output a high-quality RGB video stream.

[0013] Thirdly, this application provides a computer device, the device comprising: a memory and a processor, wherein the processor, when executing computer instructions stored in the memory, performs the method described above.

[0014] Fourthly, this application provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method described above.

[0015] In summary, this application constructs a spatiotemporally coherent latent feature field by deploying synchronized camera equipment, extracting multi-scale features, sampling and aligning from multiple perspectives, performing temporal smoothing and feature completion, and supporting rendering from any virtual perspective. This solves the problems of spatiotemporal inconsistency, flickering, and ghosting in dynamic scenes, and has the advantages of improving the spatiotemporal coherence of video rendering, eliminating flickering and ghosting, supporting rendering from any perspective, and improving the efficiency and quality of film and television production. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the computer device structure of the hardware operating environment involved in the embodiments of this application; Figure 2 This is a flowchart illustrating the first embodiment of the collaborative film and television latent feature field construction and rendering method of this application; Figure 3 This is a structural block diagram of the first embodiment of the collaborative film and television latent feature field construction and rendering system of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0018] Reference Figure 1 , Figure 1 This is a schematic diagram of the computer device structure of the hardware operating environment involved in the embodiments of this application.

[0019] like Figure 1As shown, the computer device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0020] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0021] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a collaborative film and television latent feature field construction and rendering program.

[0022] exist Figure 1 In the computer device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in this application can be set in the computer device, and the computer device calls the collaborative film and television latent feature field construction and rendering program stored in the memory 1005 through the processor 1001, and executes the collaborative film and television latent feature field construction and rendering method provided in the embodiment of this application.

[0023] Traditional methods for generating film and animation, such as 3D modeling and texturing, have long production cycles and struggle to handle complex dynamic details in real time. Generative video AI is prone to object deformation or spatiotemporal flicker when the camera moves over a wide range. Existing 3DGS / NeRF solutions lack long-term temporal logical memory and are prone to ghosting or feature loss when there is occlusion from multiple viewpoints.

[0024] To address this, this application provides a collaborative method for constructing and rendering latent feature fields in film and television, referring to... Figure 2, Figure 2 This is a flowchart illustrating the first embodiment of the collaborative film and television latent feature field construction and rendering method of this application.

[0025] For ease of understanding, the following explains some key terms in this embodiment: Deep encoders: These encoders typically consist of multiple layers of neural networks. Their function is to transform the raw input image data into a more abstract and compact feature representation. Deep encoders can extract semantic information at different levels from images, such as color, texture, and edges. This information is crucial for subsequent feature processing and reconstruction.

[0026] High-dimensional latent vector: This vector is a compact data representation output by the depth encoder that encodes the visual information of pixels in an image (such as color, texture, and material properties) in a high-dimensional space. This vector can capture the complex features of the image and provide rich semantic information for subsequent 3D reconstruction and rendering.

[0027] Alignment Network: This network aims to address the viewpoint differences and inconsistencies that may exist during feature extraction from multi-view images. Through the alignment network, features from different viewpoints can be corrected and fused, eliminating ambiguity between viewpoints and ensuring that spatial points have consistent feature representations across different viewpoints.

[0028] The Mamba module is a novel neural network architecture based on a state-space model. Its key feature is its ability to efficiently process long sequences of data and capture long-distance dependencies. In the construction of the latent feature field, the Mamba module uses a selective state transition mechanism to achieve temporal smoothing of features and feature completion under occlusion conditions, thereby ensuring the spatiotemporal coherence of the latent feature field.

[0029] Neural Decoder: This decoder is a neural network module corresponding to the depth encoder. Its function is to restore the feature representation in the latent space into a recognizable image or video. Through the neural decoder, the constructed latent feature field can be converted into a high-quality RGB video stream, achieving the final rendering output.

[0030] In this embodiment, the collaborative film and television latent feature field construction and rendering method includes the following steps: Step S10: Deploy synchronous trigger camera equipment in the shooting area, complete the calibration of the camera equipment's internal and external parameters and clock synchronization, and acquire multi-channel RGB image streams with globally unified timestamps.

[0031] It should be noted that the camera equipment consists of N industrial cameras with synchronous triggering function, and the intrinsic and extrinsic parameter calibration is as follows: intrinsic parameter calibration obtains the focal length of each industrial camera. Principal point coordinates and radial distortion coefficients; extrinsic parameter calibration determines the rotation matrix of each industrial camera in the world coordinate system through feature point alignment. Translation vector The clock synchronization is achieved using an external clock generator, ensuring that the exposure error of all industrial cameras on the time axis t is less than 10. -6 The time is specified in seconds, and a globally uniform timestamp label is assigned to each frame of the image.

[0032] Understandably, this embodiment abandons the traditional "geometric modeling-texturing" path and adopts the "neural representation field" logic. Multiple cameras are regarded as collaborative perception nodes. By constructing a four-dimensional (3D+Time) latent feature cloud and performing logical evolution by Mamba, high-fidelity, flicker-free rendering from any viewpoint is achieved.

[0033] By employing N industrial cameras with synchronized triggering capabilities as imaging devices and performing high-precision intrinsic and extrinsic parameter calibration, this embodiment ensures extremely high spatial geometric consistency of images acquired from different perspectives, effectively eliminating lens distortion and uncertainties in camera position and orientation. Simultaneously, an external clock generator synchronizes exposure errors of all industrial cameras at the microsecond level or even lower, and assigns a globally unified timestamp to each frame, significantly improving the temporal alignment accuracy of multi-view images. This high-precision spatiotemporal synchronization and geometric calibration provides high-quality, highly reliable input data for subsequent deep encoders to extract multi-scale feature maps, perform multi-view sampling, and for alignment networks to perform viewpoint disambiguation. It significantly reduces feature extraction errors and fusion inconsistencies caused by inaccurate data sources, enabling the Mamba module to more accurately achieve temporal smoothing of features and feature completion in occluded scenes. Ultimately, it constructs a spatiotemporally coherent latent feature field free of visual artifacts, laying a solid foundation for the neural decoder to reproduce and output high-fidelity, flicker-free RGB video streams, greatly enhancing the realism and stability of film and television rendering.

[0034] Step S20: Input the multi-channel RGB image stream into the depth encoder, extract multi-scale feature maps through the convolutional network, and transform the pixels in the feature maps into high-dimensional latent vectors containing color, texture, and material attributes.

[0035] It should be noted that the convolutional network is a convolutional network with a feature pyramid (FPN) structure, and the extracted multi-scale feature maps have a dimension of [missing information]. The high-dimensional latent vector is a C-dimensional vector that encodes the color information, local texture gradient information, and material attribute feature information of the pixel.

[0036] By employing the aforementioned technical solution and utilizing a convolutional network with a Feature Pyramid Network (FPN) structure for feature extraction, the problem of traditional convolutional networks struggling to simultaneously capture multi-scale information and fine details in complex scenes can be effectively solved. The FPN structure, by fusing features from different levels, ensures that the extracted multi-scale feature maps retain both high-level semantic information and low-level spatial details. This allows the C-dimensional high-dimensional latent vector derived from each pixel to more comprehensively and accurately encode its color information, local texture gradient information, and material attribute features. This rich latent vector representation provides a more solid foundation for the subsequent construction of a 3D spatial latent feature field, significantly improving its accuracy and detail representation. Finally, when the spatiotemporally coherent latent feature field is resampled within the virtual camera's view frustum and input into the neural decoder, a high-quality RGB video stream with higher fidelity, richer details, and stronger realism can be reproduced and output. This effectively avoids rendering blur or loss of detail due to insufficient feature information, thereby significantly improving the visual effects of film and television content.

[0037] Step S30: Discretize the 3D space of the actor's activities into a set of point clouds, project the spatial points onto the imaging plane of each camera using the intrinsic and extrinsic parameter matrices of the camera equipment, and perform multi-view sampling on the feature maps of each imaging plane to obtain a multi-dimensional feature set of the spatial points.

[0038] The steps of discretizing the 3D space of actor activity into a point cloud set, projecting spatial points onto the imaging planes of each camera using the intrinsic and extrinsic parameter matrices of the camera devices, and performing multi-view sampling on the feature maps of each imaging plane to obtain a multi-dimensional feature set of spatial points include: dividing the 3D space of actor activity into a point cloud set V, and sampling any point in the space... Using the calibrated intrinsic and extrinsic parameter matrices, through the formula Project the image back onto the imaging plane of each industrial camera; sample the projection points from the N feature maps corresponding to the N industrial cameras to obtain the feature set. This forms a multidimensional feature description of the spatial point; among which : Represents a spatial point The projection coordinates on the imaging plane of the i-th camera device, Let (x, y, z) represent any discrete point in the 3D space where the actor is active, with its world coordinates represented as (x, y, z). This represents the intrinsic parameter matrix of the i-th camera device. This represents the rotation matrix of the i-th camera relative to the world coordinate system. Let represent the translation vector of the i-th camera relative to the world coordinate system.

[0039] Through the above technical solution, this embodiment establishes a precise and efficient association mechanism between 3D spatial points and 2D image features. By accurately projecting discrete points in 3D space onto the imaging plane of each industrial camera and sampling from the corresponding feature maps, each 3D spatial point can obtain rich feature descriptions from multiple perspectives. This multidimensional feature description not only includes the appearance information of the 3D point under different perspectives but also ensures the spatial consistency of features through geometric projection. This effectively solves the challenge of how to elevate 2D image features to 3D space and construct 3D feature representations with spatial semantics, providing a solid and accurate feature foundation for the subsequent construction of a spatiotemporally coherent latent feature field, thereby significantly improving the spatial accuracy and detail representation of the latent feature field.

[0040] Step S40: Input the multidimensional feature set into the alignment network to complete the viewpoint disambiguation, calculate the credibility score of each viewpoint feature through MLP, and obtain the aligned spatial point latent vector through weighted fusion.

[0041] The steps of inputting the multidimensional feature set into the alignment network to complete viewpoint disambiguation, calculating the confidence score of each viewpoint feature through MLP, and obtaining the aligned spatial point latent vector through weighted fusion include: inputting the sampled feature set into the alignment network to complete viewpoint disambiguation; Calculate the credibility score of features from each perspective using MLP. When limb occlusion causes a conflict in feature logic, the credibility score of the corresponding occluded camera position. Approaching 0; through the weighted summation formula Achieving consistent fusion of multi-view features yields the aligned spatial point latent vectors at time t. .

[0042] Through the above technical solution, this embodiment can effectively identify and quantify the reliability of features from different perspectives, especially in scenarios where limb occlusion causes feature logic conflicts, by scoring the reliability of the corresponding occluded camera position. By setting the values ​​close to 0 and employing a weighted summation method for fusion, the system can intelligently suppress unreliable viewpoint information and highlight the contribution of reliable viewpoint features. This significantly improves the accuracy and consistency of the latent vectors of spatial points, avoids rendering artifacts caused by multi-view feature conflicts or occlusion, and lays a solid foundation for the subsequent construction of a spatiotemporally coherent latent feature field, ultimately outputting a high-fidelity, flicker-free RGB video stream.

[0043] Step S50: After serializing the latent point cloud in 3D space, input it into the Mamba module. Selective state transition is completed through the state space equation to achieve temporal smoothing of features and feature completion in occluded scenes, resulting in a spatiotemporally coherent latent feature field.

[0044] The process involves serializing the latent point cloud in 3D space and inputting it into the Mamba module. Selective state transitions are then performed using state-space equations to achieve temporal smoothing of features and feature completion in occluded scenes, resulting in a spatiotemporally coherent latent feature field. This includes: using Hilbert curves to unfold the latent point cloud in 3D space into a one-dimensional token sequence based on spatial proximity; inputting the one-dimensional token sequence into the Mamba module and performing selective state transitions using state-space equations. Complete selective state transitions, where Let be the hidden state at time t. The hidden state at time t-1 This represents the latent vector of spatial points in the original one-dimensional token sequence input to the Mamba module. This represents the spatiotemporally coherent latent vector output after temporal smoothing and feature completion by the Mamba module. This represents the state transition matrix, which controls the evolution logic of the hidden states. C represents the input control matrix, responsible for mapping the current time-state features to the hidden state space. C represents the output matrix, which remaps the updated hidden state back to the feature space. This represents the residual connection term, used to directly superimpose input features onto the output to ensure training stability and feature fidelity; through the hidden state... Memory feature history trajectory to eliminate visual flicker caused by inter-frame sampling differences, when all camera devices are occluded at time t, according to The motion logic is used to predict and complete the latent features at time t.

[0045] Through the above technical solution, this embodiment can effectively solve the problems of temporal discontinuity and feature loss that may occur in multi-view fusion schemes under scenarios with small differences in inter-frame sampling and complete occlusion. (Hidden state of the Mamba module) It can memorize the historical trajectory of latent features, thereby smoothing out subtle differences between frames and eliminating visual flicker. More importantly, even in extreme cases where all camera devices are obstructed, the Mamba module can rely on the previous hidden state. The inherent motion logic intelligently predicts and completes the latent features at the current moment, ensuring that the latent feature field can maintain a high degree of spatiotemporal coherence in any complex scene, which greatly improves the quality and robustness of the final rendered video.

[0046] Step S60: Set the intrinsic and extrinsic parameters and pose parameters of the virtual camera, resample the spatiotemporally coherent latent feature field within the virtual camera's view frustum, input the sampled latent feature map into the neural decoder, and restore and output a high-quality RGB video stream.

[0047] The steps of specific viewpoint sampling and neural decoding rendering include: Determine the intrinsic parameter matrix of any virtual camera based on the requirements of film and television rendering. Rotation matrix Translation vector ; Within the view frustum of the virtual camera, the spatiotemporally coherent latent feature field output by the Mamba module is resampled. The sampled latent feature map is input into the neural decoder to restore and output a high-fidelity, flicker-free, high-quality RGB video stream. The latent feature field supports multiple resampling rendering from any virtual viewpoint.

[0048] Through the above technical solution, this embodiment effectively solves the problem of how to flexibly and efficiently generate the final video stream from a constructed latent feature field in film and television production. This is achieved by precisely setting the intrinsic parameter matrix of the virtual camera according to the film and television rendering requirements. Rotation matrix Translation vector This technology enables fine-grained control over the rendering perspective, meeting the requirements of various complex shot designs and artistic compositions. Within the view frustum of the virtual camera, the spatiotemporally coherent latent feature field output by the Mamba module is resampled, ensuring that only valid feature data relevant to the current viewpoint is extracted, avoiding unnecessary computational overhead. Subsequently, the sampled latent feature map is input into a neural decoder, which efficiently restores the abstract latent features to high-fidelity RGB pixels, resulting in a video output with accurate colors and rich details. Because the latent feature field itself possesses spatiotemporal coherence, and the Mamba module has already implemented temporal smoothing of features and feature completion in occluded scenes during its construction, the video stream restored by the neural decoder effectively avoids the spatiotemporal flicker problem common in traditional rendering methods, ensuring video smoothness and stability. Furthermore, the latent feature field supports multiple resampling renderings from any virtual viewpoint. This means that once the latent feature field is constructed, it can serve as a reusable scene asset, allowing producers to render high-quality videos multiple times from different angles and with different parameters without recalculating complex calculations, greatly improving the flexibility and efficiency of film and television production and reducing post-production costs.

[0049] The following example will provide a more detailed explanation of the above technical solution: A film production studio plans to shoot a challenging action sequence involving actors moving quickly and interacting physically in complex scenes. They require post-production to be able to flexibly adjust the virtual camera's perspective while avoiding the lengthy process of traditional 3D modeling and the visual limitations of existing generative artificial intelligence.

[0050] First, N industrial cameras with synchronized triggering capabilities are deployed within the shooting area. These cameras undergo precise intrinsic and extrinsic parameter calibration before shooting. Intrinsic parameter calibration obtains the focal length, principal point coordinates, and radial distortion coefficient of each camera to ensure the accuracy of the image geometry. Extrinsic parameter calibration determines the rotation matrix and translation vector of each camera in the world coordinate system through feature point alignment, thereby accurately determining their relative position and orientation in three-dimensional space. Simultaneously, an external clock generator is used to synchronize the clocks of all industrial cameras, ensuring that their exposure error on the time axis is less than 10. -6 The system acquires multiple RGB image streams with globally unified timestamps, each frame of which is assigned a globally unified timestamp. This approach allows the system to capture spatiotemporal consistency data, laying the foundation for subsequent spatiotemporal consistency processing and avoiding data misalignment issues caused by time asynchrony between different devices, as seen in traditional methods.

[0051] Subsequently, these multi-channel RGB image streams with globally unified timestamps are input into a deep encoder. This encoder employs a convolutional network with a Feature Pyramid (FPN) structure, capable of extracting multi-scale feature maps from the images, with dimensions of [missing information]. Each pixel in these feature maps is transformed into a C-dimensional high-dimensional latent vector. This vector not only encodes the pixel's color information but also contains local texture gradient information and material property features. This multi-scale feature extraction and high-dimensional latent vector representation can capture the visual details of a scene more comprehensively and precisely than traditional pixel-based or simple feature-based methods.

[0052] Next, the 3D space of the actors' movements is discretized into a series of point cloud sets. For any point cloud in the space... Using the calibrated intrinsic and extrinsic parameter matrices, through the formula The projection is then precisely projected back onto the imaging plane of each industrial camera. Then, from the N feature maps corresponding to the N industrial cameras, these projected points are sampled from multiple perspectives to obtain a multi-dimensional feature set for that spatial point. This process enables the projection of two-dimensional image features back into three-dimensional spatial features, providing a foundation for constructing a three-dimensional latent feature field.

[0053] To address potential ambiguity and occlusion issues in multi-view data, the sampled feature set is input into an alignment network. This network uses a multilayer perceptron (MLP) to calculate the confidence score of features from each viewpoint. When an actor's limbs obstruct the view, preventing a camera from clearly capturing the features of a specific spatial point, the credibility score for the obstructing camera position is... It will approach 0. Then, using the weighted summation formula... Achieving consistent fusion of multi-view features yields the aligned spatial point latent vectors at time t. This perspective disambiguation and weighted fusion mechanism effectively avoids the "ghosting" or feature loss phenomena that occur in existing 3DGS / NeRF methods when there is multi-view occlusion.

[0054] To achieve temporal smoothing of features and feature completion in occluded scenarios, the latent point cloud in 3D space is serialized and input into the Mamba module. Specifically, Hilbert curves are used to unfold the latent point cloud in 3D space into a one-dimensional token sequence according to spatial proximity, thus preserving spatial structure information. The Mamba module then uses state-space equations... and Complete the selective state transition. Among them, Let be the hidden state at time t. This represents the hidden state at time t-1. (The sentence is incomplete and requires further context.) The system can memorize the historical trajectory of features, thereby eliminating visual flicker caused by minute differences in inter-frame sampling, and solving the "spatiotemporal flicker" problem that easily occurs in generative video AI when the camera moves over a wide range. More importantly, when all camera devices are occluded at time t, the Mamba module can... The motion logic predicts and completes the latent features at time t, ensuring the spatiotemporal coherence of the latent feature field and overcoming the deficiency of existing technologies in lacking long-term temporal logical memory.

[0055] Finally, based on the requirements of film and television rendering, the intrinsic parameter matrix, rotation matrix, and translation vector of any virtual camera are set. Within the view frustum of the virtual camera, the spatiotemporally coherent latent feature field output by the Mamba module is resampled to obtain a latent feature map from a specific viewpoint. This latent feature map is then input into the neural decoder to reconstruct and output a high-fidelity, flicker-free, high-quality RGB video stream. This latent feature field supports multiple resampling renderings from any virtual viewpoint, greatly improving the flexibility and efficiency of film and television production, avoiding the tedious keyframe adjustments and long production cycles of traditional 3D modeling, while providing stronger 3D geometric constraints and spatiotemporal consistency than generative video AI.

[0056] This embodiment constructs a spatiotemporally coherent latent feature field by deploying synchronous camera equipment, extracting multi-scale features, sampling and aligning from multiple perspectives, performing temporal smoothing and feature completion, and supporting rendering from any virtual perspective. It solves the problems of spatiotemporal inconsistency, flickering and ghosting in dynamic scenes, and has the advantages of improving the spatiotemporal coherence of video rendering, eliminating flickering and ghosting, supporting rendering from any perspective, and improving the efficiency and quality of film and television production.

[0057] Furthermore, embodiments of this application also propose a computer-readable storage medium storing a program for collaborative film and television latent feature field construction and rendering. When the program for collaborative film and television latent feature field construction and rendering is executed by a processor, it implements the steps of the method for collaborative film and television latent feature field construction and rendering as described above.

[0058] Reference Figure 3 , Figure 3 This is a structural block diagram of the first embodiment of the collaborative film and television latent feature field construction and rendering system of this application.

[0059] like Figure 3 As shown, the collaborative film and television latent feature field construction and rendering system proposed in this application includes: Data acquisition module 10 is used to deploy synchronous trigger camera equipment in the shooting area, complete the calibration of internal and external parameters of the camera equipment and clock synchronization, and acquire multi-channel RGB image streams with globally unified timestamps; The data conversion module 20 is used to input the multi-channel RGB image stream into the depth encoder, extract multi-scale feature maps through the convolutional network, and convert the pixels in the feature maps into high-dimensional latent vectors containing color, texture and material attributes. The set generation module 30 is used to discretize the 3D space of the actor's activities into a set of point clouds, and to project the spatial points onto the imaging plane of each camera device using the intrinsic and extrinsic parameter matrices of the camera devices. Multi-view sampling is performed on the feature maps of each imaging plane to obtain a multi-dimensional feature set of the spatial points. The vector generation module 40 is used to input the multidimensional feature set into the alignment network to complete the viewpoint disambiguation, calculate the credibility score of each viewpoint feature through MLP, and obtain the aligned spatial point latent vector through weighted fusion. The feature generation module 50 is used to serialize the latent point cloud in 3D space and input it into the Mamba module. It completes selective state transition through the state space equation, realizes the temporal smoothing of features and feature completion in occluded scenes, and obtains a spatiotemporally coherent latent feature field. The output module 60 is used to set the intrinsic and extrinsic parameters and pose parameters of the virtual camera, resample the spatiotemporally coherent latent feature field within the virtual camera's view frustum, input the sampled latent feature map into the neural decoder, and restore and output a high-quality RGB video stream.

[0060] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solution of this application. In specific applications, those skilled in the art can make settings as needed, and this application does not impose any restrictions on this.

[0061] This embodiment constructs a spatiotemporally coherent latent feature field by deploying synchronous camera equipment, extracting multi-scale features, sampling and aligning from multiple perspectives, performing temporal smoothing and feature completion, and supporting rendering from any virtual perspective. It solves the problems of spatiotemporal inconsistency, flickering and ghosting in dynamic scenes, and has the advantages of improving the spatiotemporal coherence of video rendering, eliminating flickering and ghosting, supporting rendering from any perspective, and improving the efficiency and quality of film and television production.

[0062] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this application. In practical applications, those skilled in the art can select some or all of it to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0063] In addition, for technical details not described in detail in this embodiment, please refer to the method for collaborative film and television latent feature field construction and rendering provided in any embodiment of this application, which will not be repeated here.

[0064] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0065] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0066] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application. The above are only preferred embodiments of this application and do not limit the patent scope of this application. All equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A collaborative method for constructing and rendering latent feature fields in film and television, characterized in that, include: Deploy synchronously triggered camera equipment in the shooting area, complete the calibration of the camera equipment's internal and external parameters and clock synchronization, and acquire multiple RGB image streams with globally unified timestamps; The multi-channel RGB image stream is input into a deep encoder, and multi-scale feature maps are extracted through a convolutional network. The pixels in the feature maps are then transformed into high-dimensional latent vectors containing color, texture, and material attributes. The 3D space of the actors' activities is discretized into a set of point clouds. The spatial points are projected onto the imaging planes of each camera using the intrinsic and extrinsic parameter matrices of the camera equipment. The feature maps of each imaging plane are sampled from multiple perspectives to obtain a multi-dimensional feature set of the spatial points. The multidimensional feature set is input into the alignment network to complete the viewpoint disambiguation. The credibility score of each viewpoint feature is calculated by MLP, and the aligned spatial point latent vector is obtained by weighted fusion. The latent point cloud in 3D space is serialized and input into the Mamba module. Selective state transition is completed through the state space equation to achieve temporal smoothing of features and feature completion in occluded scenes, resulting in a spatiotemporally coherent latent feature field. The intrinsic and extrinsic parameters and pose parameters of the virtual camera are set. The spatiotemporally coherent latent feature field is resampled within the virtual camera's view frustum. The sampled latent feature map is input into the neural decoder to restore and output a high-quality RGB video stream.

2. The collaborative film and television latent feature field construction and rendering method according to claim 1, characterized in that, The camera equipment consists of N industrial cameras with synchronous triggering function. The intrinsic and extrinsic parameter calibration is as follows: intrinsic parameter calibration obtains the focal length of each industrial camera. Principal point coordinates and radial distortion coefficient; Extrinsic parameter calibration determines the rotation matrix of each industrial camera in the world coordinate system through feature point alignment. Translation vector ; The clock synchronization is achieved using an external clock generator, ensuring that the exposure error of all industrial cameras on the time axis t is less than 10. -6 The time is specified in seconds, and a globally uniform timestamp label is assigned to each frame of the image.

3. The collaborative film and television latent feature field construction and rendering method according to claim 1, characterized in that, The convolutional network is a convolutional network with a feature pyramid (FPN) structure, and the extracted multi-scale feature maps have a dimension of [missing value]. ; The high-dimensional latent vector is a C-dimensional vector that encodes the color information, local texture gradient information, and material attribute feature information of the pixel.

4. The collaborative film and television latent feature field construction and rendering method according to claim 1, characterized in that, The steps of discretizing the 3D space of the actors' activities into a point cloud set, projecting the spatial points onto the imaging planes of each camera using the intrinsic and extrinsic parameter matrices of the camera equipment, and sampling the feature maps of each imaging plane from multiple perspectives to obtain a multi-dimensional feature set of the spatial points include: The 3D space for actor movement is divided into a point cloud set V. For any point in the space... Using the calibrated intrinsic and extrinsic parameter matrices, through the formula Project it back onto the imaging plane of each industrial camera; The feature set is obtained by sampling the projection points from the N feature maps corresponding to N industrial cameras. This forms a multidimensional feature description of the spatial point; in, : Represents a spatial point The projection coordinates on the imaging plane of the i-th camera device, Let (x, y, z) represent any discrete point in the 3D space where the actor is active, with its world coordinates represented as (x, y, z). This represents the intrinsic parameter matrix of the i-th camera device. This represents the rotation matrix of the i-th camera relative to the world coordinate system. Let represent the translation vector of the i-th camera relative to the world coordinate system.

5. The collaborative film and television latent feature field construction and rendering method according to claim 1, characterized in that, The steps of inputting the multidimensional feature set into the alignment network to complete viewpoint disambiguation, calculating the confidence score of each viewpoint feature through MLP, and obtaining the aligned spatial point latent vector through weighted fusion include: inputting the sampled feature set into the alignment network to complete viewpoint disambiguation; Calculate the credibility score of features from each perspective using MLP. When limb occlusion causes a conflict in feature logic, the credibility score of the corresponding occluded camera position. Approaching 0; Using the weighted summation formula Achieving consistent fusion of multi-view features yields the aligned spatial point latent vectors at time t. .

6. The collaborative film and television latent feature field construction and rendering method according to claim 1, characterized in that, The process involves serializing the latent point cloud in 3D space and inputting it into the Mamba module. Selective state transitions are performed using state-space equations to achieve temporal smoothing of features and feature completion in occluded scenes, resulting in a spatiotemporally coherent latent feature field. This includes the following steps: Hilbert curves are used to unfold the potential point cloud in 3D space into a one-dimensional token sequence according to spatial proximity. The one-dimensional token sequence is input into the Mamba module, and the state-space equation is used to process it. Complete selective state transitions, where Let be the hidden state at time t. The hidden state at time t-1 This represents the latent vector of spatial points in the original one-dimensional token sequence input to the Mamba module. This represents the spatiotemporally coherent latent vector output after temporal smoothing and feature completion by the Mamba module. This represents the state transition matrix, which controls the evolution logic of the hidden states. C represents the input control matrix, responsible for mapping the current time-state features to the hidden state space. C represents the output matrix, which remaps the updated hidden state back to the feature space. This represents a residual connection term, used to directly superimpose input features onto the output to ensure training stability and feature fidelity. By hiding state Memory feature history trajectory to eliminate visual flicker caused by inter-frame sampling differences, when all camera devices are occluded at time t, according to The motion logic is used to predict and complete the latent features at time t.

7. The collaborative film and television latent feature field construction and rendering method according to claim 1, characterized in that, The steps of specific viewpoint sampling and neural decoding rendering include: Determine the intrinsic parameter matrix of any virtual camera based on the requirements of film and television rendering. Rotation matrix Translation vector ; Within the view frustum of the virtual camera, the spatiotemporally coherent latent feature field output by the Mamba module is resampled. The sampled latent feature map is input into the neural decoder to restore and output a high-fidelity, flicker-free, high-quality RGB video stream. The latent feature field supports multiple resampling rendering from any virtual viewpoint.

8. A collaborative film and television latent feature field construction and rendering system, characterized in that, include: The data acquisition module is used to deploy synchronously triggered camera equipment in the shooting area, complete the calibration of the camera equipment's internal and external parameters and clock synchronization, and acquire multiple RGB image streams with a globally unified timestamp; The data conversion module is used to input the multi-channel RGB image stream into the depth encoder, extract multi-scale feature maps through the convolutional network, and convert the pixels in the feature maps into high-dimensional latent vectors containing color, texture and material attributes. The set generation module is used to discretize the 3D space of the actor's activities into a set of point clouds. The spatial points are projected onto the imaging planes of each camera device using the intrinsic and extrinsic parameter matrices of the camera devices. The feature maps of each imaging plane are sampled from multiple perspectives to obtain a multi-dimensional feature set of the spatial points. The vector generation module is used to input the multidimensional feature set into the alignment network to complete the viewpoint disambiguation, calculate the confidence score of each viewpoint feature through MLP, and obtain the aligned spatial point latent vector through weighted fusion. The feature generation module is used to serialize the latent point cloud in 3D space and input it into the Mamba module. Selective state transition is completed through the state space equation to achieve temporal smoothing of features and feature completion in occluded scenes, resulting in a spatiotemporally coherent latent feature field. The output module is used to set the intrinsic and extrinsic parameters and pose parameters of the virtual camera, resample the spatiotemporally coherent latent feature field within the virtual camera's view frustum, input the sampled latent feature map into the neural decoder, and restore and output a high-quality RGB video stream.

9. A computer device, characterized in that, The device includes a memory and a processor, wherein the processor, when executing computer instructions stored in the memory, performs the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 7.

Images