A configuration decision optimization system and method for film and television rendering

Abstract

The application relates to the field of film and television rendering, in particular to a configuration decision optimization system and method for film and television rendering, which comprises a shooting calibration module, an intra-frame rendering module, a scene reconstruction module, a video rendering module and a configuration compression module; the shooting calibration module is used for video de-jittering and adjusting a picture center; the intra-frame rendering module is used for area light rendering; the scene reconstruction module is used for generating a point cloud model; the video rendering module is used for merging a rendered video; and the configuration compression module is used for compressing a video stream; the application can ensure the definition and consistency of an output picture, reduce high-frequency artifacts and fuzzy sawteeth caused by shooting angle problems, reduce video rendering result jitter, improve image rendering quality, solve a sampling rate deficiency problem, improve the scale of multi-path video rendering, improve video reconstruction speed and rendering efficiency, and realize efficient video compression and transmission.

Description

Technical Field

[0001] This invention relates to the field of film and television rendering, specifically to a configuration decision optimization system and method for film and television rendering. Background Technology

[0002] Film and television rendering is a process in post-production that converts constructed 3D scene data into final visualized images. After scene filming, rendering software performs ray tracing, global illumination, and shading calculations to enrich the details of the video and achieve better visual effects. Common rendering processes use Monte Carlo path tracing algorithms for ray simulation and shader application for simple rendering. However, effective techniques are lacking for rendering high-precision dynamic images.

[0003] High-precision dynamic images are characterized by complex scenes, large data volumes, and high requirements for latency and smoothness. Conventional renderers render scenes by controlling shaders to access closed spatial structures, which cannot directly obtain the spatial geometric information of the scene. Furthermore, they do not consider the correlation between the projection camera pose and the scene, making it difficult for shader points to sample based on the geometric surface of the light source. This affects the lighting calculation results and can easily lead to problems such as inaccurate geometric estimation, artifacts and jagged edges, and high computational resource consumption under complex rendering tasks.

[0004] In addition, high-quality video rendering requires high sampling rates of pixels to reproduce scene models. However, due to the limitations of GPU processing power, scene reconstruction faces problems of high computational complexity and long rendering time, which cannot meet the requirements of high modeling accuracy while performing light field rendering. Summary of the Invention

[0005] The purpose of this invention is to provide a configuration decision optimization system and method for film and television rendering, so as to solve the problems mentioned in the background art.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a configuration decision optimization system for film and television rendering, comprising: a shooting calibration module, an intra-frame rendering module, a scene reconstruction module, a video rendering module, and a configuration compression module; The shooting calibration module is used to repair video frames based on camera motion path and smooth angle after shooting images, perform video shading processing, construct adjustment models using horizontal lines, vertical lines, line segments and horizontal circles in the visible area of ​​each frame as control points, calculate camera orientation through consistency constraints of the same area from different angles, cluster frame pixels, progressively correct the cluster center position based on camera orientation, and adjust the center and resolution of each frame image. The intra-frame rendering module is used to calculate the spatial coordinates of the pixels of each frame image based on the camera orientation, construct dense blocks to remove image noise and extract image features, use a channel attention mechanism to parse the image feature channels, obtain image material parameters, model the illumination as a function with the incident light field, material parameters and camera orientation as variables and the outgoing irradiance as the output, calculate the area illumination integral through MLP, and perform area lighting rendering according to the corresponding area illumination integral to obtain the rendered image. The scene reconstruction module is used to sample rendered images at different resolutions using a scale smoothing filter, generate a point cloud patch sequence, search for action vectors based on the point cloud patches, extract geometric structures from the patch group, extract the intersection points of adjacent pixels and pixel boundary points on the geometric structures as mesh vertices, perform triangulation, use the QEM algorithm to adjust the distribution of mesh vertices, generate a triangular mesh, perform color interpolation and uniform sampling on the mesh, and generate a point cloud model. The video rendering module is used to perform three-dimensional processing on point clouds. It voxels the point clouds by using axis-aligned bounding boxes. Based on the distance field within the bounding boxes, it uses variable-step ray tracing to determine the visibility of the point clouds. It builds a decoder and renderer on the GPU platform, dynamically renders the point clouds according to their visibility, outputs the rendered dynamic point clouds, establishes the correlation between video frames based on the dynamic point clouds, and merges them to obtain the rendered video. The configuration compression module is used to divide the frame image into dynamic and static regions when the camera focal length or distance changes during video rendering. These regions are independently encoded by different processing units. The dynamic regions are compressed using an encoder, while the static regions are skipped from compression. The module clusters three-dimensional Gaussian parameter vectors at different resolution scales, stores the index of the Gaussian code, compresses the video stream through pixel-domain encoding, and packages and outputs the compressed video stream.

[0007] Furthermore, the shooting calibration module includes: a perspective stabilization unit and an inter-frame alignment unit; The viewpoint stabilization unit is used to select the initial two frames, determine the initial relative pose through the essential matrix and homography matrix, add new frames one by one to obtain the continuous motion pose, obtain the three-dimensional motion path of the camera by fitting a polynomial curve based on the continuous motion pose of the camera, perform interpolation processing on the path vector, smooth the viewpoint orientation, and output the smoothed camera path and viewpoint. The inter-frame alignment unit is used to construct textures at different resolutions. The texture level is determined based on the distance between the main camera and the object. Continuous sampling is performed at the resolution. Based on the transformation relationship between the smooth pose and the original pose, perspective projection transformation and resampling are performed on each frame to generate stable frames. Stable frames are aligned and then the frame image stream is synthesized.

[0008] Furthermore, the intra-frame rendering module includes: a material analysis unit, a ray tracing unit, and an image rendering unit; The material analysis unit is used to pre-cache the 3D image scene to a frame, extract rectangular image blocks centered on each pixel, extract high-dimensional features from the denoised image blocks through a convolutional neural network, and obtain image material parameters, which include: anisotropic roughness, layered roughness, and basic color. The ray tracing unit is used to represent the incident light field of a light source with arbitrary complexity area using latent variable encoding. Based on the material parameters, it uses an illumination integral network to reduce the dimensionality of the incident light field. Taking the incident light field at the current point as input, it outputs the incident light field of a light source with arbitrary complexity area using latent variable encoding. The image rendering unit is used to process the noisy irradiance image through a denoising network, input the processed image into the renderer, represent the image quality through Gaussian distribution uncertainty, adjust the number of optical path samples according to the uncertainty, and output the rendered image sequence.

[0009] Furthermore, the scene reconstruction module includes: a point cloud processing unit and a mesh construction unit; The point cloud processing unit is used to control the rendering frequency, represent pixels in the image with a rendering frequency above a certain value as linear interpolations of pixels in the image below the certain value, reset the interpolated pixel index and weight, and use a scale smoothing filter to smoothly transition between different resolution layers and minimize interpolation error. The mesh building unit is used to establish a mapping relationship between point cloud patches in consecutive frames, calculate the position of each point cloud patch in the next frame, obtain the action vector, group the point cloud patches to represent local surfaces, use principal component analysis to extract local geometric structures, fit polygons to the boundaries of adjacent pixels to generate triangular meshes, and uniformly sample to obtain the point cloud model.

[0010] Furthermore, the video rendering module includes: a point cloud indexing unit, a dynamic rendering unit, and a surface reconstruction unit; The point cloud indexing unit is used to construct a point cloud index using a nested octree, with nodes storing subsets of the point cloud. The point cloud is then sampled hierarchically using a Poisson disk sampling algorithm. Based on the sampling results, the point cloud is segmented into hierarchical indexes, and the distance field of the point cloud is calculated. The dynamic rendering unit is used to track video objects using time-consistent pixels, obtain the dynamic coordinates of the moving object's surface, align the point cloud with the moving object, adjust the ray step size using the distance field value within the current voxel, record the point cloud visibility when the ray intersects the surface, and allocate rendering resources according to the point cloud visibility. The surface reconstruction unit is used to model moving objects using neural radiation fields, output density and color fields at each time step, extract surface meshes through density field isosurface sampling and Poisson reconstruction, map the fused textures onto the surface meshes, reuse the rendering results of each frame, and output the rendered video.

[0011] Furthermore, the configuration compression module includes: a region coding unit and a compression clustering unit; The region coding unit is used to cluster consecutive frames, and uses a shared buffer within the cluster to encode each frame in parallel. This allows static regions within the cluster to share a buffer, while dynamic regions are encoded independently for each frame. The rendered image with unknown degradation distribution is input into the generator, and the model is trained by summing the loss in each iteration. The rendered image frame with noise, blur and compression artifacts is reconstructed using the image features generated in the previous iteration. The compression clustering unit is used to extract three-dimensional Gaussian parameters at different resolution scales, cluster the Gaussian parameter vectors, and store all cluster centers as a codebook. The processed image sequence is input into the video encoder, the encoding parameters are adjusted to control the output video bitrate, and the compressed video is packaged into a standard format for output.

[0012] A configuration decision optimization method for film and television rendering includes the following steps: Step S1. After capturing the image, repair the video frames according to the camera motion path, perform video stabilization, calculate the camera orientation through consistency constraints, correct the cluster center position according to the camera orientation, adjust the center of each frame image, and align each frame image. Step S2. Calculate the spatial coordinates of pixels in each frame based on the camera orientation, construct dense blocks to extract image features, parse the image feature channels to obtain image material parameters, model the lighting as a function, calculate the area lighting integral through MLP, and perform area lighting rendering according to the area lighting integral to obtain the rendered image. Step S3. Sample the rendered images at different resolutions to generate a point cloud patch sequence. Extract the geometric structure from the patch group. Perform triangulation using the intersection of adjacent pixels and the boundary points of pixels as mesh vertices to generate a triangular mesh. Perform color interpolation and uniform sampling on the mesh to generate a point cloud model. Step S4. Voxelize the point cloud using axis-aligned bounding boxes, use variable step size rays to determine the visibility of the point cloud, build a decoder and renderer on the GPU platform, dynamically render the point cloud according to its visibility, establish the relationship between video frames based on the dynamic point cloud, and merge them to obtain the rendered video. Step S5. When the camera focal length or distance is changed, the frame image is divided into dynamic and static regions. The dynamic region is compressed using an encoder, while the static region is skipped from compression. The Gaussian parameter vectors at each resolution scale are clustered, the index of the Gaussian code is stored, and the video stream is compressed and output through pixel-domain encoding.

[0013] Furthermore, step S1 includes: Step S11. Select the initial two frames, determine the initial relative pose through the essential matrix and homography matrix, add new frames one by one to obtain the continuous motion pose, and obtain the camera's three-dimensional motion path by fitting a polynomial curve based on the camera's continuous motion pose. Interpolate the path vector to smooth the view orientation and output the smoothed camera path and view. Step S12. Construct textures at different resolutions, determine the texture level based on the distance between the main camera and the object, perform continuous sampling at the resolution, and perform perspective projection transformation and resampling on each frame based on the transformation relationship between the smooth pose and the original pose to generate stable frames. Align the stable frames and synthesize the frame image stream.

[0014] Furthermore, step S2 includes: Step S21. Pre-cache the 3D image scene to a frame, extract rectangular image blocks centered on each pixel, extract high-dimensional features from the denoised image blocks through a convolutional neural network, and obtain image material parameters, including: anisotropic roughness, layered roughness, and basic color. Step S22. Represent the incident light field of an arbitrary complexity area light source with latent variable encoding. Based on the material parameters, use an illumination integral network to reduce the dimensionality of the incident light field. Take the incident light field at the current point as input and output the incident light field of an arbitrary complexity area light source with latent variable encoding. Process the noisy illumination image through a denoising network. Input the processed image into the renderer. Represent the image quality with Gaussian uncertainty. Adjust the number of light path samples based on the uncertainty and output the rendered image sequence.

[0015] Furthermore, step S3 includes: Step S31. Control the rendering frequency, represent the pixels in the image with a rendering frequency above a certain value as linear interpolation of the pixels in the image below the certain value, reset the interpolated pixel index and weight, and use a scale smoothing filter to smoothly transition between different resolution layers and minimize the interpolation error. Step S32. Establish a mapping relationship between point cloud patches in consecutive frames, calculate the position of each point cloud patch in the next frame, obtain the motion vector, group the point cloud patches to represent local surfaces, use principal component analysis to extract local geometric structures, fit polygons to the boundaries of adjacent pixels to generate triangular meshes, and uniformly sample to obtain the point cloud model.

[0016] Furthermore, step S4 includes: Step S41. Construct a point cloud index using a nested octree, with nodes storing subsets of the point cloud. Perform hierarchical sampling of the point cloud using a Poisson disk sampling algorithm. Perform hierarchical index segmentation based on the sampling results and calculate the distance field of the point cloud. Use time-consistent pixel tracking to track video objects and obtain the dynamic coordinates of the moving object's surface. Align the point cloud with the moving object. Adjust the ray step size using the distance field value within the current voxel. Record the point cloud as visible when the ray intersects the surface. Allocate rendering resources according to the visibility of the point cloud. Step S42. Model the moving object using neural radiation field, output the density field and color field at each time step, extract the surface mesh through density field isosurface sampling and Poisson reconstruction, map the fused texture onto the surface mesh, reuse the rendering results of each frame, and output the rendered video.

[0017] Furthermore, step S5 includes: Step S51. Cluster the continuous frames and use the intra-cluster shared buffer to encode each frame in parallel. The static region within the cluster shares the buffer, while the dynamic region is encoded independently for each frame. Input the rendered image with unknown degradation distribution into the generator and train the model by summing the loss of each iteration. Reconstruct the rendered image frame with noise, blur and compression artifacts using the image features generated in the previous iteration. Step S52. Extract the three-dimensional Gaussian parameters at different resolution scales, cluster the Gaussian parameter vectors, store all cluster centers to form a codebook, input the processed image sequence into the video encoder, adjust the encoding parameters to control the output video bitrate, and encapsulate the compressed video into a standard format for output.

[0018] Compared with the prior art, the beneficial effects achieved by the present invention are: 1. This invention can obtain the camera motion path based on the continuous motion pose of the camera, perform video stabilization, adjust the center and resolution of each frame image, calculate the spatial coordinates of the pixels in each frame image, and analyze the material by combining three-dimensional geometric information to ensure the clarity and consistency of the output image. It reduces high-frequency artifacts and blurriness caused by shooting angle issues, reduces the jitter of video rendering results, and improves the quality of image rendering.

[0019] 2. This invention reduces the dimensionality of the incident light field, calculates the area illumination integral, inputs it into the renderer for area lighting rendering, adjusts the light path selection, obtains the rendered image, meshes the image, performs color interpolation and sampling, generates a point cloud model, completes high-quality real-time rendering of image frames, adaptively balances rendering quality and efficiency, solves the problem of insufficient sampling rate, increases the scale of multi-channel video rendering, improves video reconstruction speed and rendering efficiency, and achieves high-quality reconstruction of scene geometry.

[0020] 3. This invention processes point clouds in three dimensions, determines the visibility of point clouds, establishes the correlation between video frames, reuses the rendering results of each frame to complete video rendering, performs multi-frame parallel encoding when the focal length or distance changes, and compresses the video stream output. While maintaining rendering quality and visual effects, it achieves efficient video compression and transmission, improves rendering computation efficiency and cross-platform adaptability of the rendering process, and enhances the user's video viewing experience under limited bandwidth. Attached Figure Description

[0021] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the structure of a configuration decision optimization system for film and television rendering according to the present invention; Figure 2 This is a schematic diagram illustrating the steps of a configuration decision optimization method for film and television rendering according to the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Please see Figures 1 to 2 The present invention provides a technical solution: a configuration decision optimization system for film and television rendering, comprising: a shooting calibration module, an intra-frame rendering module, a scene reconstruction module, a video rendering module, and a configuration compression module; The shooting calibration module is used to repair video frames based on camera motion path and smooth angle after shooting images, perform video shading processing, construct adjustment models using horizontal lines, vertical lines, line segments and horizontal circles in the visible area of ​​each frame as control points, calculate camera orientation through consistency constraints of the same area from different angles, cluster frame pixels, progressively correct the cluster center position based on camera orientation, and adjust the center and resolution of each frame image. The shooting calibration module includes: a perspective stabilization unit and an inter-frame alignment unit; The viewpoint stabilization unit is used to select the initial two frames, determine the initial relative pose through the essential matrix and homography matrix, add new frames one by one to obtain the continuous motion pose, obtain the three-dimensional motion path of the camera by fitting a polynomial curve based on the continuous motion pose of the camera, perform interpolation processing on the path vector, smooth the viewpoint orientation, and output the smoothed camera path and viewpoint. The inter-frame alignment unit is used to construct textures at different resolutions. The texture level is determined based on the distance between the main camera and the object. Continuous sampling is performed at the resolution. Based on the transformation relationship between the smooth pose and the original pose, perspective projection transformation and resampling are performed on each frame to generate stable frames. Stable frames are aligned and then the frame image stream is synthesized.

[0024] The intra-frame rendering module is used to calculate the spatial coordinates of the pixels of each frame image based on the camera orientation, construct dense blocks to remove image noise and extract image features, use a channel attention mechanism to parse the image feature channels, obtain image material parameters, model the illumination as a function with the incident light field, material parameters and camera orientation as variables and the outgoing irradiance as the output, calculate the area illumination integral through MLP, and perform area lighting rendering according to the corresponding area illumination integral to obtain the rendered image. The intra-frame rendering module includes: a material analysis unit, a ray tracing unit, and an image rendering unit; The material analysis unit is used to pre-cache the 3D image scene to a frame, extract rectangular image blocks centered on each pixel, extract high-dimensional features from the denoised image blocks through a convolutional neural network, and obtain image material parameters, which include: anisotropic roughness, layered roughness, and basic color. The ray tracing unit is used to represent the incident light field of a light source with arbitrary complexity area using latent variable encoding. Based on the material parameters, it uses an illumination integral network to reduce the dimensionality of the incident light field. Taking the incident light field at the current point as input, it outputs the incident light field of a light source with arbitrary complexity area using latent variable encoding. The image rendering unit is used to process the noisy irradiance image through a denoising network, input the processed image into the renderer, represent the image quality through Gaussian distribution uncertainty, adjust the number of optical path samples according to the uncertainty, and output the rendered image sequence.

[0025] The scene reconstruction module is used to sample rendered images at different resolutions using a scale smoothing filter, generate a point cloud patch sequence, search for action vectors based on the point cloud patches, extract geometric structures from the patch group, extract the intersection points of adjacent pixels and pixel boundary points on the geometric structures as mesh vertices, perform triangulation, use the QEM algorithm to adjust the distribution of mesh vertices, generate a triangular mesh, perform color interpolation and uniform sampling on the mesh, and generate a point cloud model. The scene reconstruction module includes: a point cloud processing unit and a mesh construction unit; The point cloud processing unit is used to control the rendering frequency, represent pixels in the image with a rendering frequency above a certain value as linear interpolations of pixels in the image below the certain value, reset the interpolated pixel index and weight, and use a scale smoothing filter to smoothly transition between different resolution layers and minimize interpolation error. The mesh building unit is used to establish a mapping relationship between point cloud patches in consecutive frames, calculate the position of each point cloud patch in the next frame, obtain the action vector, group the point cloud patches to represent local surfaces, use principal component analysis to extract local geometric structures, fit polygons to the boundaries of adjacent pixels to generate triangular meshes, and uniformly sample to obtain the point cloud model.

[0026] The video rendering module is used to perform three-dimensional processing on point clouds. It voxels the point clouds by using axis-aligned bounding boxes. Based on the distance field within the bounding boxes, it uses variable-step ray tracing to determine the visibility of the point clouds. It builds a decoder and renderer on the GPU platform, dynamically renders the point clouds according to their visibility, outputs the rendered dynamic point clouds, establishes the correlation between video frames based on the dynamic point clouds, and merges them to obtain the rendered video. The video rendering module includes: a point cloud indexing unit, a dynamic rendering unit, and a surface reconstruction unit; The point cloud indexing unit is used to construct a point cloud index using a nested octree, with nodes storing subsets of the point cloud. The point cloud is then sampled hierarchically using a Poisson disk sampling algorithm. Based on the sampling results, the point cloud is segmented into hierarchical indexes, and the distance field of the point cloud is calculated. The dynamic rendering unit is used to track video objects using time-consistent pixels, obtain the dynamic coordinates of the moving object's surface, align the point cloud with the moving object, adjust the ray step size using the distance field value within the current voxel, record the point cloud visibility when the ray intersects the surface, and allocate rendering resources according to the point cloud visibility. The surface reconstruction unit is used to model moving objects using neural radiation fields, output density and color fields at each time step, extract surface meshes through density field isosurface sampling and Poisson reconstruction, map the fused textures onto the surface meshes, reuse the rendering results of each frame, and output the rendered video.

[0027] The configuration compression module is used to divide the frame image into dynamic and static regions when the camera focal length or distance changes during video rendering. These regions are independently encoded by different processing units. The dynamic regions are compressed using an encoder, while the static regions are skipped from compression. The module clusters three-dimensional Gaussian parameter vectors at different resolution scales, stores the index of the Gaussian code, compresses the video stream through pixel-domain encoding, and packages and outputs the compressed video stream.

[0028] The configuration compression module includes: a region coding unit and a compression clustering unit; The region coding unit is used to cluster consecutive frames, and uses a shared buffer within the cluster to encode each frame in parallel. This allows static regions within the cluster to share a buffer, while dynamic regions are encoded independently for each frame. The rendered image with unknown degradation distribution is input into the generator, and the model is trained by summing the loss in each iteration. The rendered image frame with noise, blur and compression artifacts is reconstructed using the image features generated in the previous iteration. The compression clustering unit is used to extract three-dimensional Gaussian parameters at different resolution scales, cluster the Gaussian parameter vectors, and store all cluster centers as a codebook. The processed image sequence is input into the video encoder, the encoding parameters are adjusted to control the output video bitrate, and the compressed video is packaged into a standard format for output.

[0029] A configuration decision optimization method for film and television rendering includes the following steps: Step S1. After capturing the image, repair the video frames according to the camera motion path, perform video stabilization, calculate the camera orientation through consistency constraints, correct the cluster center position according to the camera orientation, adjust the center of each frame image, and align each frame image. Step S1 includes: Step S11. Select the initial two frames, determine the initial relative pose through the essential matrix and homography matrix, add new frames one by one to obtain the continuous motion pose, and obtain the camera's three-dimensional motion path by fitting a polynomial curve based on the camera's continuous motion pose. Interpolate the path vector to smooth the view orientation and output the smoothed camera path and view. Step S12. Construct textures at different resolutions, determine the texture level based on the distance between the main camera and the object, perform continuous sampling at the resolution, and perform perspective projection transformation and resampling on each frame based on the transformation relationship between the smooth pose and the original pose to generate stable frames. Align the stable frames and synthesize the frame image stream.

[0030] Step S2. Calculate the spatial coordinates of pixels in each frame based on the camera orientation, construct dense blocks to extract image features, parse the image feature channels to obtain image material parameters, model the lighting as a function, calculate the area lighting integral through MLP, and perform area lighting rendering according to the area lighting integral to obtain the rendered image. Step S2 includes: Step S21. Pre-cache the 3D image scene to a frame, extract rectangular image blocks centered on each pixel, extract high-dimensional features from the denoised image blocks through a convolutional neural network, and obtain image material parameters, including: anisotropic roughness, layered roughness, and basic color. Step S22. Represent the incident light field of an arbitrary complexity area light source with latent variable encoding. Based on the material parameters, use an illumination integral network to reduce the dimensionality of the incident light field. Take the incident light field at the current point as input and output the incident light field of an arbitrary complexity area light source with latent variable encoding. Process the noisy illumination image through a denoising network. Input the processed image into the renderer. Represent the image quality with Gaussian uncertainty. Adjust the number of light path samples based on the uncertainty and output the rendered image sequence.

[0031] Step S3. Sample the rendered images at different resolutions to generate a point cloud patch sequence. Extract the geometric structure from the patch group. Perform triangulation using the intersection of adjacent pixels and the boundary points of pixels as mesh vertices to generate a triangular mesh. Perform color interpolation and uniform sampling on the mesh to generate a point cloud model. Step S3 includes: Step S31. Control the rendering frequency, represent the pixels in the image with a rendering frequency above a certain value as linear interpolation of the pixels in the image below the certain value, reset the interpolated pixel index and weight, and use a scale smoothing filter to smoothly transition between different resolution layers and minimize the interpolation error. Step S32. Establish a mapping relationship between point cloud patches in consecutive frames, calculate the position of each point cloud patch in the next frame, obtain the motion vector, group the point cloud patches to represent local surfaces, use principal component analysis to extract local geometric structures, fit polygons to the boundaries of adjacent pixels to generate triangular meshes, and uniformly sample to obtain the point cloud model.

[0032] Step S4. Voxelize the point cloud using axis-aligned bounding boxes, use variable step size rays to determine the visibility of the point cloud, build a decoder and renderer on the GPU platform, dynamically render the point cloud according to its visibility, establish the relationship between video frames based on the dynamic point cloud, and merge them to obtain the rendered video. Step S4 includes: Step S41. Construct a point cloud index using a nested octree, with nodes storing subsets of the point cloud. Perform hierarchical sampling of the point cloud using a Poisson disk sampling algorithm. Perform hierarchical index segmentation based on the sampling results and calculate the distance field of the point cloud. Use time-consistent pixel tracking to track video objects and obtain the dynamic coordinates of the moving object's surface. Align the point cloud with the moving object. Adjust the ray step size using the distance field value within the current voxel. Record the point cloud as visible when the ray intersects the surface. Allocate rendering resources according to the visibility of the point cloud. Step S42. Model the moving object using neural radiation field, output the density field and color field at each time step, extract the surface mesh through density field isosurface sampling and Poisson reconstruction, map the fused texture onto the surface mesh, reuse the rendering results of each frame, and output the rendered video.

[0033] Step S5. When the camera focal length or distance is changed, the frame image is divided into dynamic and static regions. The dynamic region is compressed using an encoder, while the static region is skipped from compression. The Gaussian parameter vectors at each resolution scale are clustered, the index of the Gaussian code is stored, and the video stream is compressed and output through pixel-domain encoding.

[0034] Step S5 includes: Step S51. Cluster the continuous frames and use the intra-cluster shared buffer to encode each frame in parallel. The static region within the cluster shares the buffer, while the dynamic region is encoded independently for each frame. Input the rendered image with unknown degradation distribution into the generator and train the model by summing the loss of each iteration. Reconstruct the rendered image frame with noise, blur and compression artifacts using the image features generated in the previous iteration. Step S52. Extract the three-dimensional Gaussian parameters at different resolution scales, cluster the Gaussian parameter vectors, store all cluster centers to form a codebook, input the processed image sequence into the video encoder, adjust the encoding parameters to control the output video bitrate, and encapsulate the compressed video into a standard format for output.

[0035] Example: Identify the continuous motion posture of the camera from the original jittery video, estimate the initial pose through motion recovery structure, perform global bundle adjustment and camera path fitting, detect geometric primitives, construct an adjustment model with geometric constraints, reconstruct dense point cloud, perform video stabilization based on 3D path, estimate material parameters, reconstruct neural lighting model, perform uncertainty-guided adaptive sampling, and output rendered frame image. Construct a multi-resolution pyramid, control the rendering frequency, reset the interpolation pixel index and weights, generate a point cloud patch sequence, search for action vectors based on the point cloud patches, extract the geometric structure and mesh vertices, triangulate, generate a point cloud model, construct a nested octree index, segment the hierarchical index, voxelize the model and calculate the distance field, determine the visibility of the point cloud, render the dynamic point cloud, and output the rendered video.

[0036] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0037] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A configuration decision optimization method for film and television rendering, characterized in that, The method includes the following steps: Step S1. After capturing the image, repair the video frames according to the camera motion path, perform video stabilization, calculate the camera orientation through consistency constraints, correct the cluster center position according to the camera orientation, adjust the center of each frame image, and align each frame image. Step S2. Calculate the spatial coordinates of pixels in each frame based on the camera orientation, construct dense blocks to extract image features, parse the image feature channels to obtain image material parameters, model the lighting as a function, calculate the area lighting integral through MLP, and perform area lighting rendering according to the area lighting integral to obtain the rendered image. Step S3. Sample the rendered images at different resolutions to generate a point cloud patch sequence. Extract the geometric structure from the patch group. Perform triangulation using the intersection of adjacent pixels and the boundary points of pixels as mesh vertices to generate a triangular mesh. Perform color interpolation and uniform sampling on the mesh to generate a point cloud model. Step S4. Voxelize the point cloud using axis-aligned bounding boxes, use variable step size rays to determine the visibility of the point cloud, build a decoder and renderer on the GPU platform, dynamically render the point cloud according to its visibility, establish the relationship between video frames based on the dynamic point cloud, and merge them to obtain the rendered video. Step S5. When the camera focal length or distance is changed, the frame image is divided into dynamic and static regions. The dynamic region is compressed using an encoder, while the static region is skipped from compression. The Gaussian parameter vectors at each resolution scale are clustered, the index of the Gaussian code is stored, and the video stream is compressed and output through pixel-domain encoding.

2. The configuration decision optimization method for film and television rendering according to claim 1, characterized in that: Step S1 includes: Step S11. Select the initial two frames, determine the initial relative pose through the essential matrix and homography matrix, add new frames one by one to obtain the continuous motion pose, and obtain the camera's three-dimensional motion path by fitting a polynomial curve based on the camera's continuous motion pose. Interpolate the path vector to smooth the view orientation and output the smoothed camera path and view. Step S12. Construct textures at different resolutions, determine the texture level based on the distance between the main camera and the object, perform continuous sampling at the resolution, and perform perspective projection transformation and resampling on each frame based on the transformation relationship between the smooth pose and the original pose to generate stable frames. Align the stable frames and synthesize the frame image stream.

3. The configuration decision optimization method for film and television rendering according to claim 2, characterized in that: Step S2 includes: Step S21. Pre-cache the 3D image scene to a frame, extract rectangular image blocks centered on each pixel, extract high-dimensional features from the denoised image blocks through a convolutional neural network, and obtain image material parameters, including: anisotropic roughness, layered roughness, and basic color. Step S22. Represent the incident light field of an arbitrary complexity area light source with latent variable encoding. Based on the material parameters, use an illumination integral network to reduce the dimensionality of the incident light field. Take the incident light field at the current point as input and output the incident light field of an arbitrary complexity area light source with latent variable encoding. Process the noisy illumination image through a denoising network. Input the processed image into the renderer. Represent the image quality with Gaussian uncertainty. Adjust the number of light path samples based on the uncertainty and output the rendered image sequence.

4. The configuration decision optimization method for film and television rendering according to claim 3, characterized in that: Step S3 includes: Step S31. Control the rendering frequency, represent the pixels in the image with a rendering frequency above a certain value as linear interpolation of the pixels in the image below the certain value, reset the interpolated pixel index and weight, and use a scale smoothing filter to smoothly transition between different resolution layers and minimize the interpolation error. Step S32. Establish a mapping relationship between point cloud patches in consecutive frames, calculate the position of each point cloud patch in the next frame, obtain the action vector, group the point cloud patches to represent local surfaces, use principal component analysis to extract local geometric structures, fit polygons to the boundaries of adjacent pixels to generate triangular meshes, and uniformly sample to obtain the point cloud model. Step S4 includes: Step S41. Construct a point cloud index using a nested octree, with nodes storing subsets of the point cloud. Perform hierarchical sampling of the point cloud using a Poisson disk sampling algorithm. Perform hierarchical index segmentation based on the sampling results and calculate the distance field of the point cloud. Use time-consistent pixel tracking to track video objects and obtain the dynamic coordinates of the moving object's surface. Align the point cloud with the moving object. Adjust the ray step size using the distance field value within the current voxel. Record the point cloud as visible when the ray intersects the surface. Allocate rendering resources according to the visibility of the point cloud. Step S42. Model the moving object using neural radiation field, output the density field and color field at each time step, extract the surface mesh through density field isosurface sampling and Poisson reconstruction, map the fused texture onto the surface mesh, reuse the rendering results of each frame, and output the rendered video.

5. The configuration decision optimization method for film and television rendering according to claim 4, characterized in that: Step S5 includes: Step S51. Cluster the continuous frames and use the intra-cluster shared buffer to encode each frame in parallel. The static region within the cluster shares the buffer, while the dynamic region is encoded independently for each frame. Input the rendered image with unknown degradation distribution into the generator and train the model by summing the loss of each iteration. Reconstruct the rendered image frame with noise, blur and compression artifacts using the image features generated in the previous iteration. Step S52. Extract the three-dimensional Gaussian parameters at different resolution scales, cluster the Gaussian parameter vectors, store all cluster centers to form a codebook, input the processed image sequence into the video encoder, adjust the encoding parameters to control the output video bitrate, and encapsulate the compressed video into a standard format for output.

6. A configuration decision optimization system for film and television rendering, characterized in that, The system includes the following modules: shooting calibration module, intra-frame rendering module, scene reconstruction module, video rendering module, and configuration compression module; The shooting calibration module is used to repair video frames based on camera motion path and smooth angle after shooting images, perform video shading processing, construct adjustment models using horizontal lines, vertical lines, line segments and horizontal circles in the visible area of ​​each frame as control points, calculate camera orientation through consistency constraints of the same area from different angles, cluster frame pixels, progressively correct the cluster center position based on camera orientation, and adjust the center and resolution of each frame image. The intra-frame rendering module is used to calculate the spatial coordinates of the pixels of each frame image based on the camera orientation, construct dense blocks to remove image noise and extract image features, use a channel attention mechanism to parse the image feature channels, obtain image material parameters, model the illumination as a function with the incident light field, material parameters and camera orientation as variables and the outgoing irradiance as the output, calculate the area illumination integral through MLP, and perform area lighting rendering according to the corresponding area illumination integral to obtain the rendered image. The scene reconstruction module is used to sample rendered images at different resolutions using a scale smoothing filter, generate a point cloud patch sequence, search for action vectors based on the point cloud patches, extract geometric structures from the patch group, extract the intersection points of adjacent pixels and pixel boundary points on the geometric structures as mesh vertices, perform triangulation, use the QEM algorithm to adjust the distribution of mesh vertices, generate a triangular mesh, perform color interpolation and uniform sampling on the mesh, and generate a point cloud model. The video rendering module is used to perform three-dimensional processing on point clouds. It voxels the point clouds by using axis-aligned bounding boxes. Based on the distance field within the bounding boxes, it uses variable-step ray tracing to determine the visibility of the point clouds. It builds a decoder and renderer on the GPU platform, dynamically renders the point clouds according to their visibility, outputs the rendered dynamic point clouds, establishes the correlation between video frames based on the dynamic point clouds, and merges them to obtain the rendered video. The configuration compression module is used to divide the frame image into dynamic and static regions when the camera focal length or distance changes during video rendering. These regions are independently encoded by different processing units. The dynamic regions are compressed using an encoder, while the static regions are skipped from compression. The module clusters three-dimensional Gaussian parameter vectors at different resolution scales, stores the index of the Gaussian code, compresses the video stream through pixel-domain encoding, and packages and outputs the compressed video stream.

7. The configuration decision optimization system for film and television rendering according to claim 6, characterized in that: The shooting calibration module includes: a perspective stabilization unit and an inter-frame alignment unit; The viewpoint stabilization unit is used to select the initial two frames, determine the initial relative pose through the essential matrix and homography matrix, add new frames one by one to obtain the continuous motion pose, obtain the three-dimensional motion path of the camera by fitting a polynomial curve based on the continuous motion pose of the camera, perform interpolation processing on the path vector, smooth the viewpoint orientation, and output the smoothed camera path and viewpoint. The inter-frame alignment unit is used to construct textures at different resolutions. The texture level is determined based on the distance between the main camera and the object. Continuous sampling is performed at the resolution. Based on the transformation relationship between the smooth pose and the original pose, perspective projection transformation and resampling are performed on each frame to generate stable frames. Stable frames are aligned and then the frame image stream is synthesized.

8. The configuration decision optimization system for film and television rendering according to claim 7, characterized in that: The intra-frame rendering module includes: a material analysis unit, a ray tracing unit, and an image rendering unit; The material analysis unit is used to pre-cache the 3D image scene to a frame, extract rectangular image blocks centered on each pixel, extract high-dimensional features from the denoised image blocks through a convolutional neural network, and obtain image material parameters, which include: anisotropic roughness, layered roughness, and basic color. The ray tracing unit is used to represent the incident light field of a light source with arbitrary complexity area using latent variable encoding. Based on the material parameters, it uses an illumination integral network to reduce the dimensionality of the incident light field. Taking the incident light field at the current point as input, it outputs the incident light field of a light source with arbitrary complexity area using latent variable encoding. The image rendering unit is used to process the noisy irradiance image through a denoising network, input the processed image into the renderer, represent the image quality through Gaussian distribution uncertainty, adjust the number of optical path samples according to the uncertainty, and output the rendered image sequence.

9. A configuration decision optimization system for film and television rendering according to claim 8, characterized in that: The scene reconstruction module includes: a point cloud processing unit and a mesh construction unit; The point cloud processing unit is used to control the rendering frequency, represent pixels in the image with a rendering frequency above a certain value as linear interpolations of pixels in the image below the certain value, reset the interpolated pixel index and weight, and use a scale smoothing filter to smoothly transition between different resolution layers and minimize interpolation error. The mesh building unit is used to establish a mapping relationship between point cloud patches in consecutive frames, calculate the position of each point cloud patch in the next frame, obtain the action vector, group the point cloud patches to represent local surfaces, use principal component analysis to extract local geometric structures, fit polygons to the boundaries of adjacent pixels to generate triangular meshes, and uniformly sample to obtain the point cloud model. The video rendering module includes: a point cloud indexing unit, a dynamic rendering unit, and a surface reconstruction unit; The point cloud indexing unit is used to construct a point cloud index using a nested octree, with nodes storing subsets of the point cloud. The point cloud is then sampled hierarchically using a Poisson disk sampling algorithm. Based on the sampling results, the point cloud is segmented into hierarchical indexes, and the distance field of the point cloud is calculated. The dynamic rendering unit is used to track video objects using time-consistent pixels, obtain the dynamic coordinates of the moving object's surface, align the point cloud with the moving object, adjust the ray step size using the distance field value within the current voxel, record the point cloud visibility when the ray intersects the surface, and allocate rendering resources according to the point cloud visibility. The surface reconstruction unit is used to model moving objects using neural radiation fields, output density and color fields at each time step, extract surface meshes through density field isosurface sampling and Poisson reconstruction, map the fused textures onto the surface meshes, reuse the rendering results of each frame, and output the rendered video.

10. A configuration decision optimization system for film and television rendering according to claim 9, characterized in that: The configuration compression module includes: a region coding unit and a compression clustering unit; The region coding unit is used to cluster consecutive frames, and uses a shared buffer within the cluster to encode each frame in parallel. This allows static regions within the cluster to share a buffer, while dynamic regions are encoded independently for each frame. The rendered image with unknown degradation distribution is input into the generator, and the model is trained by summing the loss in each iteration. The rendered image frame with noise, blur and compression artifacts is reconstructed using the image features generated in the previous iteration. The compression clustering unit is used to extract three-dimensional Gaussian parameters at different resolution scales, cluster the Gaussian parameter vectors, and store all cluster centers as a codebook. The processed image sequence is input into the video encoder, the encoding parameters are adjusted to control the output video bitrate, and the compressed video is packaged into a standard format for output.

Images