Compression rendering method and device for multi-view video dynamic neural Gaussian coding

Abstract

The invention discloses a compression rendering method and device for multi-view video dynamic neural Gaussian coding, and belongs to the technical field of video compression. Comprising the following steps: extracting a first frame image of different visual angles of a multi-visual-angle video, processing the first frame image to obtain point cloud data, constructing a voxel space based on the point cloud data, inputting static characteristics of each anchor point in the voxel space and an observation visual angle into a static MLP network for training to obtain the static characteristics of the anchor points, offset scaling information and the trained static MLP network, constructing a three-dimensional voxel representation; converting the three-dimensional voxel representation into four-dimensional voxel representation based on the trained dynamic features; inputting the four-dimensional voxel representation, the trained static MLP network and the trained dynamic MLP network into an HAC module for compression coding to obtain a compressed bit stream; and inputting the compressed bit stream into a decoder for decoding to obtain the four-dimensional voxel representation, and converting the four-dimensional voxel representation into the Gaussian point cloud representation for rendering to obtain the multi-view video, thereby improving the compression efficiency and the rendering precision.

Description

Technical Field

[0001] This application belongs to the field of video compression technology, and in particular relates to a compression rendering method and apparatus for multi-view video dynamic neural Gaussian coding. Background Technology

[0002] Traditional 3D video compression is mostly based on joint coding schemes using multi-viewpoint sequences and their depth data. These schemes primarily rely on parallax prediction and motion compensation mechanisms to extract compressible information from the temporal dimension and multi-viewpoint redundancy to improve overall coding efficiency. However, because their underlying encoding is still based on 2D image patches, it is difficult to fully capture the 3D geometric structure and fine texture information of real scenes, thus limiting the renderable field of view and failing to meet the requirements of immersive applications for high-quality arbitrary viewpoint synthesis. Furthermore, when the scene contains violent motion, rapid occlusion changes, or complex spatial structures, traditional methods are prone to artifacts such as blurring, ghosting, or edge tearing during the decoding and reconstruction stage, thereby reducing the realism and stability of 3D videos.

[0003] To overcome the limitations of two-dimensional encoding, neural field compression technology based on three-dimensional Gaussian has emerged in recent years. This technology reconstructs scenes into differentiable, renderable spatial Gaussian point clouds, expressing geometry and appearance as continuous functions. Especially in dynamic scene modeling, dynamic Gaussians can update their center position, scale, rotation, and color attributes over time, giving them natural spatiotemporal continuity. This allows for more accurate representation of motion trajectories, local structural changes, and fine texture dynamics, thus significantly improving the compressibility and reconstructability of dynamic scenes.

[0004] However, existing dynamic Gaussian compression methods still have significant bottlenecks: when scenes involve local high-speed motion, complex occlusion relationships, or objects appearing or disappearing over time, the encoding process often fails to fully capture the fine-grained changes in Gaussian parameters, leading to insufficient temporal consistency constraints. During rendering, these deficiencies manifest as local geometric misalignment, texture drift, dynamic edge blurring, and even ghosting, resulting in unstable output image quality, especially in extremely dynamic regions. These problems essentially stem from the insufficient granularity of parameter modeling in dynamic regions, affecting the reconstruction accuracy and compression efficiency of dynamic scenes. Existing technologies perform poorly in dynamic regions, struggle to finely depict local motion, suffer from issues such as anchor point dynamic feature coupling leading to insufficient expression of local motion changes by neural Gaussians, and high storage overhead due to parameter redundancy during the encoding stage. Summary of the Invention

[0005] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a compressed rendering method and apparatus for dynamic neural Gaussian coding of multi-view video. By constructing a three-dimensional voxel representation, dynamic features are trained and transformed into a four-dimensional voxel representation, which is then further encoded into a transmittable compressed bitstream. By introducing anchor point feature decomposition and a Gaussian-by-Gaussian independent modeling mechanism, each neural Gaussian can capture motion changes independently, thereby significantly improving the precision and compression efficiency of dynamic reconstruction. The anchor point-driven spatiotemporal modeling framework achieves a compact representation of dynamic scenes, improving compression efficiency and reducing storage space.

[0006] To address the aforementioned problems, according to a first aspect of the present invention, a compression rendering method for multi-view video dynamic neural Gaussian coding is provided, the method comprising: A multi-view video to be compressed is acquired, and the first frame image of different viewpoints in the multi-view video to be compressed is extracted and processed to obtain point cloud data. A voxel space is constructed based on the point cloud data. The voxel space includes multiple anchor points. The static features and viewing angle of each anchor point in the voxel space are input into a static MLP network for training to obtain the static features, offset scaling information and trained static MLP network of the anchor points. A three-dimensional voxel representation is constructed based on the dynamic features, static features, position, spatial mask and offset scaling information of the anchor points. The dynamic features and time in the three-dimensional voxel representation are input into the dynamic MLP network for training, resulting in trained dynamic features and a trained dynamic MLP network. Based on the trained dynamic features, the three-dimensional voxel representation is transformed into a four-dimensional voxel representation. The four-dimensional voxel representation, the trained static MLP network, and the trained dynamic MLP network are input into the HAC module for compression encoding to obtain a compressed bitstream; The compressed bitstream is input into the decoder for decoding, resulting in a decoded four-dimensional voxel representation, a trained static MLP network, and a trained dynamic MLP network. Based on the trained static and dynamic MLP networks, the decoded four-dimensional voxel representation is converted into a Gaussian point cloud representation. Based on the Gaussian point cloud representation, rendering is performed to obtain a multi-view video.

[0007] According to one embodiment of this application, the step of inputting the static features of each anchor point in voxel space and the viewing angle into a static MLP network for training, to obtain the static features of the anchor points, offset scaling information, and the trained static MLP network, includes: S21 inputs the static features of each anchor point in the voxel space and the viewing angle into the static MLP network to obtain the static properties of the neural Gaussian corresponding to each anchor point under the viewing angle. The static properties of the neural Gaussian include opacity, rotation, scaling and color. Based on the offset scaling information of each anchor point, the three-dimensional position of the neural Gaussian corresponding to each anchor point is obtained. Based on the static properties and three-dimensional position of each neural Gaussian, the neural Gaussian point cloud in the normal space is obtained. S22 inputs the neural Gaussian point cloud in the normal space into the renderer to obtain the rendered image from the viewing perspective. The rendered image from the viewing perspective is compared with the real image to obtain the first loss function value. S23 optimizes the static attributes of each neuron Gaussian point cloud, the static MLP network parameters, the offset scaling information of each anchor point, and the static features of each anchor point based on the first loss function value. After reselecting the observation viewpoint, S21 is executed again until the iteration termination condition is reached, and the static features of the anchor points, the offset scaling information, and the trained static MLP network are obtained.

[0008] According to one embodiment of this application, the step of inputting the dynamic features and time from the three-dimensional voxel representation into a dynamic MLP network for training to obtain trained dynamic features and a trained dynamic MLP network, and converting the three-dimensional voxel representation into a four-dimensional voxel representation based on the trained dynamic features, includes: S31 transforms the dynamic features in the three-dimensional voxel representation into the dynamic features of the neural Gaussian. The dynamic features of the neural Gaussian and time are input into the dynamic MLP network to obtain the dynamic attributes of the neural Gaussian corresponding to each anchor point under different observation perspectives at different time points. The dynamic attributes include position change coefficient, opacity change coefficient and rotation change coefficient. The static attributes, dynamic attributes and position of the neural Gaussian are projected by the time projection method to construct the neural Gaussian point cloud in deformation space. S32 inputs the neural Gaussian point cloud in the deformation space into the renderer to obtain the rendered image from the viewing perspective. The rendered image from the viewing perspective is compared with the real image to obtain the value of the second loss function. S33 optimizes the dynamic attributes of each neuron Gaussian in the neural Gaussian point cloud, the dynamic MLP network parameters, and the dynamic features of the anchor points based on the second loss function value. After reselecting the observation viewpoint, S31 is executed again until the iteration termination condition is reached, resulting in trained dynamic features and a trained dynamic MLP network. Based on the trained dynamic features, the three-dimensional voxel representation is transformed into a four-dimensional voxel representation.

[0009] According to one embodiment of this application, the transformation of dynamic features in a three-dimensional voxel representation into dynamic features of a neural Gaussian includes: The dynamic features of anchor points in the 3D voxel representation are split according to a preset dimension to obtain multiple local features; Multiple local features are assigned to the corresponding neural Gaussians of the anchor points, and the local features correspond one-to-one with the neural Gaussians. The static features of the anchor point are used as the shared features of each neuron Gaussian corresponding to the anchor point; The shared and local features of each neural Gaussian are concatenated to obtain the dynamic features of the neural Gaussian.

[0010] According to one embodiment of this application, the step of inputting the four-dimensional voxel representation, the trained static MLP network, and the trained dynamic MLP network into the HAC module for compression encoding to obtain a compressed bitstream includes: The four-dimensional voxel representation is input into the HAC module, and the position of the anchor point is hash-encoded to obtain the hash feature corresponding to each anchor point. The hash features corresponding to each anchor point are input into the context MLP network to obtain the quantization encoding parameters of the four-dimensional voxel representation; The dynamic features of anchor points corresponding to neural Gaussians that did not participate in rendering are removed by the invalid Gaussian feature removal mechanism to obtain the remaining dynamic features of the anchor points. Based on the quantization coding parameters, the static features, offset scaling information and remaining dynamic features of the anchor point are compressed and encoded to obtain the first compressed bit stream; The trained static MLP network and the trained dynamic MLP network are compressed and encoded to obtain a second compressed bitstream; Arithmetic encoding is performed on the spatial mask of the trained anchor points to obtain the spatial mask bitstream; The compressed bitstream is obtained based on the first compressed bitstream, the second compressed bitstream, and the space mask bitstream.

[0011] According to one embodiment of this application, the step of removing the dynamic features of anchor points corresponding to neural Gaussians that did not participate in rendering through an invalid Gaussian feature removal mechanism to obtain the remaining dynamic features of the anchor points includes: Initialize the spatial mask of the anchor point to obtain the initial spatial mask; The initial spatial mask is trained and optimized to obtain a trained spatial mask. Determine whether the trained spatial mask is 0, and remove the dynamic features of the anchor points corresponding to the Gaussian neural networks with spatial masks of 0 to obtain the remaining dynamic features of the anchor points.

[0012] According to one embodiment of this application, the process of inputting the compressed bitstream into a decoder for decoding to obtain a decoded four-dimensional voxel representation, a trained static MLP network, and a trained dynamic MLP network; converting the decoded four-dimensional voxel representation into a Gaussian point cloud representation based on the trained static MLP network and the trained dynamic MLP network; and rendering based on the Gaussian point cloud representation to obtain a multi-view video includes: The compressed bitstream is input into the decoder for decoding to obtain the static features, dynamic features, position, spatial mask, offset scaling information of the anchor point, the trained static MLP network, and the trained dynamic MLP network. Based on the spatial mask of the anchor point, the dynamic features of the anchor point are assigned to the corresponding effective position of the anchor point; The static, dynamic, and positional properties of the corresponding neural Gaussian nodes at each anchor point are projected using a time projection method, thereby converting the decoded four-dimensional voxel representation into a Gaussian point cloud representation. The Gaussian point cloud representation is input into the renderer to obtain rendered images from different times and perspectives. The images from different times and perspectives are then combined to obtain a multi-view video.

[0013] According to a second aspect of the present invention, a compression rendering apparatus for multi-view video dynamic neural Gaussian coding is provided, the apparatus comprising: The acquisition module is used to acquire the multi-view video to be compressed, extract the first frame image of different viewpoints of the multi-view video to be compressed, process it to obtain point cloud data, construct a voxel space based on the point cloud data, the voxel space includes multiple anchor points, input the static features and viewing angle of each anchor point in the voxel space into a static MLP network for training, and obtain the static features, offset scaling information and trained static MLP network of the anchor points. Based on the dynamic features, static features, position, spatial mask and offset scaling information of the anchor points, a three-dimensional voxel representation is constructed. The conversion module is used to input the dynamic features and time from the three-dimensional voxel representation into the dynamic MLP network for training, to obtain the trained dynamic features and the trained dynamic MLP network, and to convert the three-dimensional voxel representation into a four-dimensional voxel representation based on the trained dynamic features. The encoding module is used to input the four-dimensional voxel representation, the trained static MLP network, and the trained dynamic MLP network into the HAC module for compression encoding to obtain a compressed bitstream; The decoding module is used to input the compressed bitstream into the decoder for decoding, to obtain the decoded four-dimensional voxel representation, the trained static MLP network, and the trained dynamic MLP network. Based on the trained static MLP network and the trained dynamic MLP network, the decoded four-dimensional voxel representation is converted into a Gaussian point cloud representation. Based on the Gaussian point cloud representation, rendering is performed to obtain a multi-view video.

[0014] According to a third aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the compression rendering method for dynamic neural Gaussian coding of multi-view video as described in the first aspect above.

[0015] According to a fourth aspect of the present invention, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the compression rendering method for dynamic neural Gaussian coding of multi-view video as described in the first aspect above.

[0016] According to a fifth aspect of the present invention, a chip is provided, the chip including a processor and a communication interface coupled to the processor, the processor being used to run a program or instructions to implement the compression rendering method for multi-view video dynamic neural Gaussian coding as described in the first aspect.

[0017] According to a sixth aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the compression rendering method for dynamic neural Gaussian coding of multi-view video as described in the first aspect above.

[0018] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application.

[0019] The present invention provides a compression rendering method for multi-view video dynamic neural Gaussian coding, which has the following advantages over the prior art: (1) This invention introduces an anchor feature decomposition and dynamic decoding method for dynamic scene reconstruction, which decomposes the dynamic features of the anchor according to the lower neuro-Gaussian granularity, and uses Gaussian-by-Gaussian independent dynamic attribute decoding of the changing parameters. This can reduce the situation where the neuro-Gaussian motion expression ability is insufficient due to the high coupling degree of dynamic features, enhance the adaptability of neuro-Gaussian to complex dynamics, and reduce the number of encoding parameters.

[0020] (2) By setting an invalid Gaussian feature removal mechanism in the encoding stage, the present invention deletes the corresponding dynamic features of the neural Gaussian that are not involved in rendering, thereby reducing the amount of stored parameters and improving the encoding efficiency. While ensuring the accuracy and detail quality of dynamic scene reconstruction, it can significantly reduce the scale of anchor point features and improve the parameter utilization and encoding efficiency in dynamic scene modeling.

[0021] (3) This invention constructs a three-dimensional voxel representation by introducing the static features of the static MLP network training anchors and transforms the three-dimensional voxel representation into a four-dimensional voxel representation by introducing the dynamic features of the dynamic MLP network training anchors. The three-dimensional voxel representation is compressed and encoded by the HAC module to generate a compressed bitstream suitable for network transmission, thereby improving the compression efficiency. The compressed bitstream is decompressed to restore the voxel structure, and two-dimensional images or video sequences from any viewpoint can be rendered in real time, achieving high-quality spatiotemporal consistent reconstruction. While ensuring the accuracy and detail quality of dynamic scene reconstruction, the scale of anchor features is significantly reduced, thereby improving the parameter utilization and encoding efficiency in dynamic scene modeling. Attached Figure Description

[0022] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is one of the flowcharts of the compression rendering method for dynamic neural Gaussian coding of multi-view video provided in the embodiments of this application; Figure 2 This is the second flowchart of the compression rendering method for dynamic neural Gaussian coding of multi-view video provided in the embodiments of this application; Figure 3 This is a flowchart illustrating the dynamic feature splitting and allocation of anchor points provided in an embodiment of this application; Figure 4 This is a flowchart illustrating the invalid Gaussian feature removal mechanism provided in the embodiments of this application. Figure 5 This is a schematic diagram of the structure of the compression rendering device for multi-view video dynamic neural Gaussian coding provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0024] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0025] The following description, in conjunction with the accompanying drawings, details the compression rendering method, apparatus, electronic device, and readable storage medium for multi-view video dynamic neural Gaussian coding provided in this application, through specific embodiments and application scenarios.

[0026] Among them, the compression rendering method for dynamic neural Gaussian coding of multi-view video can be applied to the terminal, specifically by hardware or software in the terminal.

[0027] The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0028] The following embodiments describe a terminal including a display and a touch-sensitive surface. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and joystick.

[0029] The compression rendering method for dynamic neural Gaussian coding of multi-view video provided in this application embodiment can be executed by an electronic device or a functional module or entity in an electronic device that can implement the compression rendering method for dynamic neural Gaussian coding of multi-view video. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras and wearable devices. The compression rendering method for dynamic neural Gaussian coding of multi-view video provided in this application embodiment will be described below using an electronic device as the execution subject.

[0030] Figure 1 This is one of the flowcharts illustrating the compression rendering method for multi-view video dynamic neural Gaussian coding provided in this application embodiment, such as... Figure 1As shown, the compression rendering method for multi-view video dynamic neural Gaussian coding includes steps 110, 120, 130 and 140.

[0031] Step 110: Obtain the multi-view video to be compressed, extract the first frame image of different viewpoints of the multi-view video to be compressed, process it to obtain point cloud data, construct a voxel space based on the point cloud data, the voxel space includes multiple anchor points, input the static features and viewing angle of each anchor point in the voxel space into a static MLP network for training, obtain the static features, offset scaling information of the anchor points and the trained static MLP network, construct a three-dimensional voxel representation based on the dynamic features, static features, position, spatial mask and offset scaling information of the anchor points; Figure 2 This is the second flowchart illustrating the compression rendering method for multi-view video dynamic neural Gaussian coding provided in this application embodiment, as shown below. Figure 2 As shown, the process involves acquiring a multi-view video to be compressed, extracting the first frame image from different viewpoints of the video, generating sparse point cloud data for the corresponding scene based on the COLMAP algorithm, constructing a voxel space based on the point cloud data, and initializing the anchor point distribution in the voxel space.

[0032] In some embodiments, the step of inputting the static features of each anchor point in voxel space and the viewing angle into a static MLP network for training, to obtain the static features of the anchor points, offset scaling information, and the trained static MLP network, includes: S21 inputs the static features of each anchor point in the voxel space and the viewing angle into the static MLP network to obtain the static properties of the neural Gaussian corresponding to each anchor point under the viewing angle. The static properties of the neural Gaussian include opacity, rotation, scaling and color. Based on the offset scaling information of each anchor point, the three-dimensional position of the neural Gaussian corresponding to each anchor point is obtained. Based on the static properties and three-dimensional position of each neural Gaussian, the neural Gaussian point cloud in the normal space is obtained. It is easy to understand that each anchor point includes multiple neural Gaussians. The static features of each anchor point in voxel space and the viewing perspective are input into the static MLP network to obtain the static properties of the neural Gaussians corresponding to each anchor point under the viewing perspective. The static MLP network includes multiple fully connected layers.

[0033] The anchor point offset and scaling information includes position offset and scaling factor. Based on the anchor point's position offset and scaling factor, the three-dimensional position of the corresponding neural Gaussian is calculated. Combined with the static properties of the neural Gaussian, a neural Gaussian point cloud in normalized space is generated. Normalized space is a standardized three-dimensional coordinate system used to uniformly represent neural Gaussians from different viewpoints.

[0034] S22 inputs the neural Gaussian point cloud in the normal space into the renderer to obtain the rendered image from the viewing perspective. The rendered image from the viewing perspective is compared with the real image to obtain the first loss function value. Furthermore, the neural Gaussian point cloud in the normalized space is input into the renderer, each neural Gaussian is projected onto the image plane, and a rendered image from the viewing perspective is generated through hybrid computation. The renderer can be a differentiable renderer.

[0035] The rendered image is compared with the real image to obtain the value of the first loss function. For example, the first loss function includes the color loss function, the depth loss function, and the regularization loss function.

[0036] S23 optimizes the static attributes of each neuron Gaussian point cloud, the static MLP network parameters, the offset scaling information of each anchor point, and the static features of each anchor point based on the first loss function value. After reselecting the observation viewpoint, S21 is executed again until the iteration termination condition is reached, and the static features of the anchor points, the offset scaling information, and the trained static MLP network are obtained.

[0037] For example, based on the first loss function value, the static properties of each neuron Gaussian in the neural Gaussian point cloud, the static MLP network parameters, the offset scaling information of each anchor point, and the static features of each anchor point are optimized by the backpropagation algorithm. The optimizer adopts Adam or stochastic gradient descent. After each optimization, a new observation view is selected and returned to S21.

[0038] Optionally, the iteration termination condition includes at least one of the following: the value of the first loss function is lower than a preset threshold, or the maximum number of iterations is reached.

[0039] Step 120: Input the dynamic features and time in the three-dimensional voxel representation into the dynamic MLP network for training to obtain the trained dynamic features and the trained dynamic MLP network. Based on the trained dynamic features, transform the three-dimensional voxel representation into a four-dimensional voxel representation. In some embodiments, the step of inputting the dynamic features and time from the three-dimensional voxel representation into a dynamic MLP network for training to obtain trained dynamic features and a trained dynamic MLP network, and then converting the three-dimensional voxel representation into a four-dimensional voxel representation based on the trained dynamic features, includes: S31 transforms the dynamic features in the three-dimensional voxel representation into the dynamic features of the neural Gaussian. The dynamic features of the neural Gaussian and time are input into the dynamic MLP network to obtain the dynamic attributes of the neural Gaussian corresponding to each anchor point under different observation perspectives at different time points. The dynamic attributes include position change coefficient, opacity change coefficient and rotation change coefficient. The static attributes, dynamic attributes and position of the neural Gaussian are projected by the time projection method to construct the neural Gaussian point cloud in deformation space. In some embodiments, the transformation of dynamic features in the three-dimensional voxel representation into dynamic features of a neural Gaussian includes: The dynamic features of anchor points in the 3D voxel representation are split according to a preset dimension to obtain multiple local features; Multiple local features are assigned to the corresponding neural Gaussians of the anchor points, and the local features correspond one-to-one with the neural Gaussians. The static features of the anchor point are used as the shared features of each neuron Gaussian corresponding to the anchor point; The shared and local features of each neural Gaussian are concatenated to obtain the dynamic features of the neural Gaussian.

[0040] Figure 3 This is a flowchart illustrating the dynamic feature splitting and allocation of anchor points provided in an embodiment of this application, as shown below. Figure 3 As shown, during training, the original dynamic features of the anchor point are first decomposed according to a preset dimensional structure. The resulting local features are then assigned to multiple Gaussian neurons under the anchor point, forming a pairing relationship between a Gaussian neuron and a local feature. Simultaneously, the static features of the anchor point remain unchanged, serving as shared features for all Gaussian neurons, indicating that they belong to the same anchor point. Assume the static features of a certain anchor point are... The dynamic characteristics are The anchor point corresponds to Dynamic characteristics of a neural Gaussian The calculation formula is as follows: in, For the dynamic characteristics of the anchor point, For the local features of the k-th neuron Gaussian, For the dynamic characteristics of k Gaussian neurons, These are shared features of the neural Gaussian.

[0041] During the decoding phase of dynamic attributes, shared features will be used. Local features of the corresponding neural Gaussian By splicing together, dynamic features specific to the neural Gaussian model are formed. The input is fed into a unified dynamic MLP network. Since each Gaussian neuron has independent local features, by calling the same dynamic MLP multiple times, the variation coefficients of different Gaussian neurons are obtained, enabling them to learn their independent spatiotemporal variation patterns under a consistent network structure. This not only improves the model's ability to fit the local nonlinear motion of Gaussian particles, but also allows for a reduction in the number of Gaussians involved in decoding in the later stages of training, thereby effectively reducing computational costs.

[0042] It's easy to understand that the anchor point of each voxel can utilize its static features, offset, and scaling to decode the position, opacity, rotation, and color information of the Gaussian neural network in canonical space using different fully connected neural network MLPs. Assume that the opacity, position, and rotation information of the Gaussian neural network changes in the time domain. By repeatedly calling the dynamic MLP network, the changes in opacity, position, and rotation of different Gaussian neural networks in the time domain are obtained, allowing them to learn their independent spatiotemporal variation patterns under a consistent network structure. Furthermore, the number of Gaussians involved in decoding can be reduced as needed in the later stages of training. The dynamic feature decoding process is shown below: Suppose the neural Gaussian dynamic characteristics at a certain anchor point are as follows: ,So The neural Gaussian is obtained through a shallow MLP used to decode opacity variation information. Then, the formula for expressing the number is as follows: The time-domain variation of neural Gaussian opacity.

[0043] in, Let i represent the opacity of the Gaussian neuron numbered i at time t. This indicates the opacity of the neural Gaussian reference, obtained by decoding static features; Indicates the magnitude of change in opacity. t is the base time, and t is the time when the frame to be rendered is located.

[0044] Similarly, The Gaussian of each neuron at the anchor point is also obtained through a different shallow MLP used for decoding position change information. This is used to express the displacement of the neural Gaussian in the time domain. The formula is as follows: in, This indicates the position of the neural Gaussian at time t. These are polynomial coefficients. The base time is k, and k is the order of the polynomial. It is the upper limit of the order.

[0045] Similarly, The anchor point for each neuron Gaussian is also obtained through a different shallow MLP used to decode Gaussian rotational transformation information. This is used to express the rotational changes of a neural Gaussian over the time domain. The formula is as follows: in, This represents the quaternion of the neural Gaussian at time t (quaternions can be used to express the rotation of the Gaussian). These are polynomial coefficients. The base time is k, and k is the order of the polynomial. It is the upper limit of the order.

[0046] By projecting the static, dynamic, and positional properties of the neural Gaussian point cloud into deformation space using a time-projection method, a neural Gaussian point cloud is constructed.

[0047] In this embodiment, by introducing an anchor feature decomposition and dynamic decoding method for dynamic scene reconstruction, the dynamic features of the anchor are decomposed according to the granularity of its lower neural Gaussian, and the changing parameters are decoded by Gaussian-wise independent dynamic attributes. This can reduce the situation where the neural Gaussian motion expression ability is insufficient due to the high coupling degree of dynamic features, enhance the adaptability of the neural Gaussian to complex dynamics, and reduce the number of encoding parameters.

[0048] S32 inputs the neural Gaussian point cloud in the deformation space into the renderer to obtain the rendered image from the viewing perspective. The rendered image from the viewing perspective is compared with the real image to obtain the value of the second loss function. S33 optimizes the dynamic attributes of each neuron Gaussian in the neural Gaussian point cloud, the dynamic MLP network parameters, and the dynamic features of the anchor points based on the second loss function value. After reselecting the observation viewpoint, S31 is executed again until the iteration termination condition is reached, resulting in trained dynamic features and a trained dynamic MLP network. Based on the trained dynamic features, the three-dimensional voxel representation is transformed into a four-dimensional voxel representation.

[0049] For example, based on the second loss function value, the dynamic properties of each neuron Gaussian, the dynamic MLP network parameters, and the dynamic characteristics of the anchor point are optimized through the backpropagation algorithm. The optimizer uses Adam or stochastic gradient descent. After each optimization, a new observation perspective is selected and the process returns to S31.

[0050] Optionally, the iteration termination condition includes at least one of the following: the value of the second loss function is lower than a preset threshold, or the maximum number of iterations is reached.

[0051] Step 130: Input the four-dimensional voxel representation, the trained static MLP network, and the trained dynamic MLP network into the HAC module for compression encoding to obtain a compressed bitstream; In some embodiments, the step of inputting the four-dimensional voxel representation, the trained static MLP network, and the trained dynamic MLP network into the HAC module for compression encoding to obtain a compressed bitstream includes: The four-dimensional voxel representation is input into the HAC module, and the position of the anchor point is hash-encoded to obtain the hash feature corresponding to each anchor point. The hash features corresponding to each anchor point are input into the context MLP network to obtain the quantization encoding parameters of the four-dimensional voxel representation; The dynamic features of anchor points corresponding to neural Gaussians that did not participate in rendering are removed by the invalid Gaussian feature removal mechanism to obtain the remaining dynamic features of the anchor points. In some embodiments, the step of removing the dynamic features of anchor points corresponding to neural Gaussians that did not participate in rendering through an invalid Gaussian feature removal mechanism to obtain the remaining dynamic features of the anchor points includes: Initialize the spatial mask of the anchor point to obtain the initial spatial mask; The initial spatial mask is trained and optimized to obtain a trained spatial mask. Determine whether the trained spatial mask is 0, and remove the dynamic features of the anchor points corresponding to the Gaussian neural networks with spatial masks of 0 to obtain the remaining dynamic features of the anchor points.

[0052] It is easy to understand that, Figure 4 This is a flowchart illustrating the invalid Gaussian feature removal mechanism provided in the embodiments of this application, as follows: Figure 4 As shown, in the encoding stage, in order to further reduce storage and parameter size, an invalid Gaussian feature removal mechanism is introduced. For neural Gaussians that are judged to have low contribution, are invisible or do not participate in rendering after training, their corresponding local features are directly removed, thereby reducing the overall dynamic feature dimension that anchors need to save and realizing parameter compression in the encoding stage.

[0053] For example, the spatial mask of the anchor point is initialized. The number of masks corresponds to the number of Gaussian neurons under the anchor point, with values ​​of 0 or 1. It is initialized to 1. In each training iteration, the spatial mask of the anchor point is multiplied by the opacity of the corresponding Gaussian neuron. This is used to measure the contribution of the Gaussian neuron during rendering, i.e., whether the presence or absence of this Gaussian neuron affects the rendered image. By comparing the rendered image with the real image, the spatial mask is optimized to maximize the number of 0 values, thus obtaining the final anchor point spatial mask.

[0054] For a Gaussian neural network with a spatial mask of 0, the corresponding local features of the anchor points are ineffective during training and can therefore be removed during encoding to reduce the encoding parameters. When entropy encoding the dynamic features of the anchor points, the invalid parts are removed first, and then the remaining dynamic features are encoded to obtain a bitstream. At the same time, arithmetic encoding is performed on the final anchor point spatial mask to obtain a spatial mask bitstream, which is used as part of the final bitstream.

[0055] In this embodiment, by setting an invalid Gaussian feature culling mechanism in the encoding stage, the corresponding dynamic features of neural Gaussians that are not involved in rendering are deleted, thereby reducing the amount of stored parameters and improving encoding efficiency. While ensuring the accuracy and detail quality of dynamic scene reconstruction, it can significantly reduce the scale of anchor point features and improve the parameter utilization and encoding efficiency in dynamic scene modeling.

[0056] Based on the quantization coding parameters, the static features, offset scaling information and remaining dynamic features of the anchor point are compressed and encoded to obtain the first compressed bit stream; The trained static MLP network and the trained dynamic MLP network are compressed and encoded to obtain a second compressed bitstream; Arithmetic encoding is performed on the spatial mask of the trained anchor points to obtain the spatial mask bitstream; The compressed bitstream is obtained based on the first compressed bitstream, the second compressed bitstream, and the space mask bitstream.

[0057] For example, in each iteration of training, the position of the anchor point is hash-encoded to obtain the hash feature corresponding to each anchor point. The hash feature is input into the context MLP to obtain the quantization offset parameter of the attribute corresponding to the anchor point. Based on the quantization offset parameter, the anchor point attribute is entropy-encoded to obtain the bit stream of static features, dynamic features, and offset scaling information.

[0058] The static features, dynamic features, offset scaling information, and spatial mask bitstreams are summed and multiplied by a preset coefficient, and then used as part of the loss function for optimization. After training, the static features, dynamic features, offset scaling information, spatial mask bitstreams of all anchors, and all MLP networks after iterative optimization are used as the encoding compression result to obtain the compressed bitstream.

[0059] Step 140: Input the compressed bitstream into the decoder for decoding to obtain the decoded four-dimensional voxel representation, the trained static MLP network, and the trained dynamic MLP network. Based on the trained static MLP network and the trained dynamic MLP network, convert the decoded four-dimensional voxel representation into a Gaussian point cloud representation. Render the Gaussian point cloud representation to obtain a multi-view video.

[0060] In some embodiments, the process of inputting the compressed bitstream into a decoder for decoding to obtain a decoded four-dimensional voxel representation, a trained static MLP network, and a trained dynamic MLP network, and then converting the decoded four-dimensional voxel representation into a Gaussian point cloud representation based on the trained static MLP network and the trained dynamic MLP network, and finally rendering based on the Gaussian point cloud representation to obtain a multi-view video includes: The compressed bitstream is input into the decoder for decoding to obtain the static features, dynamic features, position, spatial mask, offset scaling information of the anchor point, the trained static MLP network, and the trained dynamic MLP network. Based on the spatial mask of the anchor point, the dynamic features of the anchor point are assigned to the corresponding effective position of the anchor point; The static, dynamic, and positional properties of the corresponding neural Gaussian nodes at each anchor point are projected using a time projection method, thereby converting the decoded four-dimensional voxel representation into a Gaussian point cloud representation. The Gaussian point cloud representation is input into the renderer to obtain rendered images from different times and perspectives. The images from different times and perspectives are then combined to obtain a multi-view video.

[0061] What's easy to understand is that when rendering an image at any point in time, the voxel anchor points are first decoded into a static Gaussian set in the normal space, then projected onto the corresponding deformation space according to the input time parameters, and finally the target frame is generated through the fast projection and rasterization process of 3DGS. In this embodiment, backpropagation is used to optimize the automatic updating of anchor point structure and dynamic attributes along the training path, thereby achieving high-quality spatiotemporal consistent reconstruction. This reduces the network bandwidth requirements for transmitting large amounts of video data and enables real-time video decoding from any perspective on the terminal, improving the user's viewing experience. It also supports the synthesis of new perspectives, and the decoding and rendering process is not limited by the original shooting perspective, thus improving rendering accuracy.

[0062] The compression rendering method for multi-view video dynamic neural Gaussian encoding provided in this application embodiment can be executed by a compression rendering device for multi-view video dynamic neural Gaussian encoding. This application embodiment uses the execution of the compression rendering method for multi-view video dynamic neural Gaussian encoding by a compression rendering device for multi-view video dynamic neural Gaussian encoding as an example to illustrate the compression rendering device for multi-view video dynamic neural Gaussian encoding provided in this application embodiment.

[0063] This application also provides a compression rendering apparatus for multi-view video dynamic neural Gaussian coding, such as... Figure 5 As shown, the compression rendering device for multi-view video dynamic neural Gaussian coding includes: an acquisition module 510, a detection module 520, an encoding module 530, and a decoding module 540.

[0064] The acquisition module 510 is used to acquire the multi-view video to be compressed, extract the first frame image of different views of the multi-view video to be compressed, process it to obtain point cloud data, construct a voxel space based on the point cloud data, the voxel space includes multiple anchor points, input the static features and viewing angle of each anchor point in the voxel space into a static MLP network for training, obtain the static features, offset scaling information of the anchor points and the trained static MLP network, and construct a three-dimensional voxel representation based on the dynamic features, static features, position, spatial mask and offset scaling information of the anchor points; The detection module 520 is used to input the dynamic features and time in the three-dimensional voxel representation into the dynamic MLP network for training, to obtain the trained dynamic features and the trained dynamic MLP network, and to convert the three-dimensional voxel representation into a four-dimensional voxel representation based on the trained dynamic features. The encoding module 530 is used to input the four-dimensional voxel representation, the trained static MLP network, and the trained dynamic MLP network into the HAC module for compression encoding to obtain a compressed bitstream; The decoding module 540 is used to input the compressed bitstream into the decoder for decoding, to obtain the decoded four-dimensional voxel representation, the trained static MLP network and the trained dynamic MLP network. Based on the trained static MLP network and the trained dynamic MLP network, the decoded four-dimensional voxel representation is converted into a Gaussian point cloud representation. Based on the Gaussian point cloud representation, rendering is performed to obtain a multi-view video.

[0065] The compression rendering method for dynamic neural Gaussian coding of multi-view video provided in this application constructs a three-dimensional voxel representation by introducing static features of static MLP network training anchors, and transforms the three-dimensional voxel representation into a four-dimensional voxel representation by introducing dynamic features of dynamic MLP network training anchors. The method is then compressed and encoded using the HAC module to generate a compressed bitstream suitable for network transmission, thereby improving compression efficiency. The compressed bitstream is decompressed to restore the voxel structure, enabling real-time rendering of two-dimensional images or video sequences from any viewpoint. This achieves high-quality spatiotemporal consistent reconstruction, significantly reducing the anchor feature size while ensuring the accuracy and detail quality of dynamic scene reconstruction, and improving parameter utilization and coding efficiency in dynamic scene modeling.

[0066] The compression rendering apparatus for multi-view video dynamic neural Gaussian coding provided in this application embodiment can achieve... Figures 1 to 4 The various processes implemented in the embodiment of the compression rendering method for multi-view video dynamic neural Gaussian coding will not be described in detail here to avoid repetition.

[0067] In some embodiments, such as Figure 6As shown, this application embodiment also provides an electronic device 600, including a processor 601, a memory 602, and a computer program stored in the memory 602 and executable on the processor 601. When the program is executed by the processor 601, it implements the various processes of the above-described compression rendering method embodiment for multi-view video dynamic neural Gaussian coding and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0068] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0069] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described embodiment of the compression rendering method for dynamic neural Gaussian coding of multi-view video and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0070] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0071] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described compression rendering method for multi-view video dynamic neural Gaussian coding.

[0072] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0073] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described embodiment of the compression rendering method for dynamic neural Gaussian coding of multi-view video, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0074] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a device-level chip, device chip, chip device, or on-chip device chip, etc.

[0075] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0076] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the compression rendering method of dynamic neural Gaussian coding for multi-view video according to the various embodiments of this application.

[0077] In the description of this application, "first feature" and "second feature" may include one or more of the features.

[0078] In the description of this application, "multiple" means two or more.

[0079] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0080] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0081] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A compression rendering method for multi-view video dynamic neural Gaussian coding, characterized in that, The method includes: A multi-view video to be compressed is acquired, and the first frame image of different viewpoints in the multi-view video to be compressed is extracted and processed to obtain point cloud data. A voxel space is constructed based on the point cloud data. The voxel space includes multiple anchor points. The static features and viewing angle of each anchor point in the voxel space are input into a static MLP network for training to obtain the static features, offset scaling information and trained static MLP network of the anchor points. A three-dimensional voxel representation is constructed based on the dynamic features, static features, position, spatial mask and offset scaling information of the anchor points. The dynamic features and time in the three-dimensional voxel representation are input into the dynamic MLP network for training, resulting in trained dynamic features and a trained dynamic MLP network. Based on the trained dynamic features, the three-dimensional voxel representation is transformed into a four-dimensional voxel representation. The four-dimensional voxel representation, the trained static MLP network, and the trained dynamic MLP network are input into the HAC module for compression encoding to obtain a compressed bitstream; The compressed bitstream is input into the decoder for decoding, resulting in a decoded four-dimensional voxel representation, a trained static MLP network, and a trained dynamic MLP network. Based on the trained static and dynamic MLP networks, the decoded four-dimensional voxel representation is converted into a Gaussian point cloud representation. Based on the Gaussian point cloud representation, rendering is performed to obtain a multi-view video.

2. The compression rendering method for multi-view video dynamic neural Gaussian coding according to claim 1, characterized in that, The process of inputting the static features of each anchor point in voxel space and the viewing angle into a static MLP network for training, to obtain the static features of the anchor points, offset scaling information, and the trained static MLP network, includes: S21 inputs the static features of each anchor point in the voxel space and the viewing angle into the static MLP network to obtain the static properties of the neural Gaussian corresponding to each anchor point under the viewing angle. The static properties of the neural Gaussian include opacity, rotation, scaling and color. Based on the offset scaling information of each anchor point, the three-dimensional position of the neural Gaussian corresponding to each anchor point is obtained. Based on the static properties and three-dimensional position of each neural Gaussian, the neural Gaussian point cloud in the normal space is obtained. S22 inputs the neural Gaussian point cloud in the normal space into the renderer to obtain the rendered image from the viewing perspective. The rendered image from the viewing perspective is compared with the real image to obtain the first loss function value. S23 optimizes the static attributes of each neuron Gaussian point cloud, the static MLP network parameters, the offset scaling information of each anchor point, and the static features of each anchor point based on the first loss function value. After reselecting the observation viewpoint, S21 is executed again until the iteration termination condition is reached, and the static features of the anchor points, the offset scaling information, and the trained static MLP network are obtained.

3. The compression rendering method for multi-view video dynamic neural Gaussian coding according to claim 2, characterized in that, The process of inputting dynamic features and time from the 3D voxel representation into a dynamic MLP network for training, obtaining trained dynamic features and a trained dynamic MLP network, and then converting the 3D voxel representation into a 4D voxel representation based on the trained dynamic features includes: S31 transforms the dynamic features in the three-dimensional voxel representation into the dynamic features of the neural Gaussian. The dynamic features of the neural Gaussian and time are input into the dynamic MLP network to obtain the dynamic attributes of the neural Gaussian corresponding to each anchor point under different observation perspectives at different time points. The dynamic attributes include position change coefficient, opacity change coefficient and rotation change coefficient. The static attributes, dynamic attributes and position of the neural Gaussian are projected by the time projection method to construct the neural Gaussian point cloud in deformation space. S32 inputs the neural Gaussian point cloud in the deformation space into the renderer to obtain the rendered image from the viewing perspective. The rendered image from the viewing perspective is compared with the real image to obtain the value of the second loss function. S33 optimizes the dynamic attributes of each neuron Gaussian in the neural Gaussian point cloud, the dynamic MLP network parameters, and the dynamic features of the anchor points based on the second loss function value. After reselecting the observation viewpoint, S31 is executed again until the iteration termination condition is reached, resulting in trained dynamic features and a trained dynamic MLP network. Based on the trained dynamic features, the three-dimensional voxel representation is transformed into a four-dimensional voxel representation.

4. The compression rendering method for multi-view video dynamic neural Gaussian coding according to claim 3, characterized in that, The process of converting dynamic features in a three-dimensional voxel representation into dynamic features in a neural Gaussian includes: The dynamic features of anchor points in the 3D voxel representation are split according to a preset dimension to obtain multiple local features; Multiple local features are assigned to the corresponding neural Gaussians of the anchor points, and the local features correspond one-to-one with the neural Gaussians. The static features of the anchor point are used as the shared features of each neuron Gaussian corresponding to the anchor point; The shared and local features of each neural Gaussian are concatenated to obtain the dynamic features of the neural Gaussian.

5. The compression rendering method for multi-view video dynamic neural Gaussian coding according to claim 4, characterized in that, The process of inputting the four-dimensional voxel representation, the trained static MLP network, and the trained dynamic MLP network into the HAC module for compression encoding to obtain a compressed bitstream includes: The four-dimensional voxel representation is input into the HAC module, and the position of the anchor point is hash-encoded to obtain the hash feature corresponding to each anchor point. The hash features corresponding to each anchor point are input into the context MLP network to obtain the quantization encoding parameters of the four-dimensional voxel representation; The dynamic features of anchor points corresponding to neural Gaussians that did not participate in rendering are removed by the invalid Gaussian feature removal mechanism to obtain the remaining dynamic features of the anchor points. Based on the quantization coding parameters, the static features, offset scaling information, and remaining dynamic features of the anchor point are compressed and encoded to obtain the first compressed bit stream; The trained static MLP network and the trained dynamic MLP network are compressed and encoded to obtain a second compressed bitstream; Arithmetic encoding is performed on the spatial mask of the trained anchor points to obtain the spatial mask bitstream; The compressed bitstream is obtained based on the first compressed bitstream, the second compressed bitstream, and the space mask bitstream.

6. The compression rendering method for multi-view video dynamic neural Gaussian coding according to claim 5, characterized in that, The method of removing the dynamic features of anchor points corresponding to neural Gaussians that did not participate in rendering through the invalid Gaussian feature removal mechanism, to obtain the remaining dynamic features of the anchor points, includes: Initialize the spatial mask of the anchor point to obtain the initial spatial mask; The initial spatial mask is trained and optimized to obtain a trained spatial mask. Determine if the trained spatial mask is 0, and remove the dynamic features of the anchor points corresponding to the Gaussian neural networks with spatial masks of 0 to obtain the remaining dynamic features of the anchor points.

7. The compression rendering method for multi-view video dynamic neural Gaussian coding according to claim 6, characterized in that, The process involves inputting the compressed bitstream into a decoder for decoding, obtaining a decoded four-dimensional voxel representation, a trained static MLP network, and a trained dynamic MLP network. Based on the trained static and dynamic MLP networks, the decoded four-dimensional voxel representation is converted into a Gaussian point cloud representation. Rendering is then performed based on the Gaussian point cloud representation to obtain a multi-view video, including: The compressed bitstream is input into the decoder for decoding to obtain the static features, dynamic features, position, spatial mask, offset scaling information of the anchor point, the trained static MLP network, and the trained dynamic MLP network. Based on the spatial mask of the anchor point, the dynamic features of the anchor point are assigned to the corresponding effective position of the anchor point; The static, dynamic, and positional properties of the corresponding neural Gaussian nodes at each anchor point are projected using a time projection method, thereby converting the decoded four-dimensional voxel representation into a Gaussian point cloud representation. The Gaussian point cloud representation is input into the renderer to obtain rendered images from different times and perspectives. The images from different times and perspectives are then combined to obtain a multi-view video.

8. A compression rendering apparatus for multi-view video dynamic neural Gaussian coding, implemented using the compression rendering method for multi-view video dynamic neural Gaussian coding as described in any one of claims 1 to 7, characterized in that, The device includes: The acquisition module is used to acquire the multi-view video to be compressed, extract the first frame image of different viewpoints of the multi-view video to be compressed, process it to obtain point cloud data, construct a voxel space based on the point cloud data, the voxel space includes multiple anchor points, input the static features and viewing angle of each anchor point in the voxel space into a static MLP network for training, and obtain the static features, offset scaling information and trained static MLP network of the anchor points. Based on the dynamic features, static features, position, spatial mask and offset scaling information of the anchor points, a three-dimensional voxel representation is constructed. The conversion module is used to input the dynamic features and time from the three-dimensional voxel representation into the dynamic MLP network for training, to obtain the trained dynamic features and the trained dynamic MLP network, and to convert the three-dimensional voxel representation into a four-dimensional voxel representation based on the trained dynamic features. The encoding module is used to input the four-dimensional voxel representation, the trained static MLP network, and the trained dynamic MLP network into the HAC module for compression encoding to obtain a compressed bitstream; The decoding module is used to input the compressed bitstream into the decoder for decoding, to obtain the decoded four-dimensional voxel representation, the trained static MLP network, and the trained dynamic MLP network. Based on the trained static MLP network and the trained dynamic MLP network, the decoded four-dimensional voxel representation is converted into a Gaussian point cloud representation. Based on the Gaussian point cloud representation, rendering is performed to obtain a multi-view video.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the compression rendering method for dynamic neural Gaussian coding of multi-view video as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the compression rendering method for dynamic neural Gaussian coding of multi-view video as described in any one of claims 1 to 7.

Images