Method for generating volumetric video, method for playing and electronic device

Abstract

The application discloses a volumetric video generation method, comprising: acquiring a reference video containing a to-be-modeled object; generating a digital human model corresponding to the to-be-modeled object based on the reference video, and determining attribute information of each Gaussian point in the digital human model corresponding to each frame; determining change rule information of attribute information of the Gaussian point corresponding to the time axis of the reference video according to the attribute information of the Gaussian point corresponding to each frame; and generating volumetric video information corresponding to the reference video based on the change rule information of each Gaussian point and the digital human model. The scheme of the application can reduce the data volume of the volumetric video, facilitate storage and transmission, reduce the computing resources occupied by playing, improve the playing fluency of the volumetric video, and improve the viewing experience of users. The application also provides a volumetric video playing method and device, an electronic device, and a computer readable storage medium.

Description

Technical Field

[0001] This application relates to the field of video playback technology, specifically to a method for generating and playing a volumetric video, an apparatus, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the rapid development of virtual reality, spatial computing devices, and other technologies, volumetric video, as an emerging media form capable of recording dynamic scenes in three-dimensional space, has received widespread attention. Volumetric video allows users to view dynamic content from any angle, providing an immersive viewing experience. However, the technologies used to generate volumetric video typically involve independently modeling each frame and storing the model information for each frame. This approach results in excessively large volumes of volumetric video data, which is detrimental to storage and transmission, and also consumes significant computing resources during playback, affecting playback smoothness. Summary of the Invention

[0003] This application provides a method for generating and playing a large-format video, an apparatus, an electronic device, and a computer-readable storage medium, which can reduce the data size of the large-format video, make it easier to store and transmit, reduce the computing resources occupied by playback, improve the playback smoothness of the large-format video, and improve the user's viewing experience.

[0004] The specific plan is as follows: In a first aspect, this application provides a method for generating volumetric videos, the method comprising: Obtain a reference video, which contains the object to be modeled; A digital human model corresponding to the object to be modeled is generated based on the reference video, and the attribute information of each Gaussian point in the digital human model in each frame is determined. Based on the attribute information of the Gaussian point in each frame, determine the change pattern information of the attribute information of the Gaussian point on the time axis of the reference video; Based on the variation pattern information corresponding to each Gaussian point and the digital human model, the volumetric video information corresponding to the reference video is generated.

[0005] Secondly, this application provides a method for generating volumetric videos, the method comprising: Obtain volumetric video information, wherein the volumetric video information is generated by the volumetric video generation method described in the first aspect; The digital human model and the variation pattern information corresponding to each Gaussian point of the digital human model are obtained from the volumetric video information. The attribute information of the Gaussian point is determined based on the change pattern information corresponding to the Gaussian point. The Gaussian points are rendered based on their attribute information to generate volumetric video frames.

[0006] Thirdly, this application provides a method for playing a video of a large size, the method comprising: Volumetric video information is generated based on the volumetric video generation method described in the first aspect; Determine the current camera pose for rendering; Based on the current camera pose, determine the Gaussian point in the volumetric video where the digital human model is currently active. Based on the volumetric video information, the attribute information of the Gaussian points in the active state is determined, and the Gaussian points in the active state are rendered according to the attribute information to generate and play the volumetric video image from the current perspective.

[0007] Fourthly, this application also provides an electronic device, comprising: a processor, a memory, and computer program instructions stored in the memory and executable on the processor; wherein the processor executes the computer program instructions to implement the method as described in any one of the first to third aspects.

[0008] Fifthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in any one of the first to third aspects.

[0009] Sixthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method as described in any one of the first to third aspects.

[0010] Compared with the prior art, this application has the following advantages: The volumetric video generation method provided in this application embodiment obtains a reference video, wherein the reference video contains an object to be modeled, and then generates a digital human model corresponding to the object to be modeled based on the reference video. The method determines the attribute information of each Gaussian point in the digital human model at each frame, and then determines the change pattern information of the attribute information of the Gaussian point on the time axis of the reference video based on the attribute information of the Gaussian point at each frame. This change pattern information can characterize the change pattern of the attribute information of the corresponding Gaussian point over time. Therefore, storing only the change pattern information is sufficient to obtain the attribute information corresponding to the Gaussian point at any given time. Subsequently, based on the change pattern information corresponding to each Gaussian point and the digital human model, volumetric video information corresponding to the reference video can be generated. This volumetric video information is used for attribute querying of the Gaussian point at any given time during volumetric video playback, thereby enabling video rendering and playback.

[0011] Compared to related technologies that require storing model information obtained from independent modeling for each frame, this application only stores the variation law of Gaussian point attribute information over time. There is no need to store the model information of each frame independently, which greatly reduces the data volume of volumetric video information and makes it easier to store and transmit volumetric video over the network. When playing volumetric video, it is only necessary to calculate the Gaussian point attribute information required at any time based on the variation law information, without loading the full multi-frame model data. This effectively reduces the computing resources occupied during playback, improves the playback smoothness of volumetric video, and enhances the user's immersive viewing experience. Attached Figure Description

[0012] Figure 1 This is a schematic diagram illustrating the application scenario of the solution provided in this application.

[0013] Figure 2 This is a flowchart illustrating an example of a method for generating volumetric videos provided in an embodiment of this application.

[0014] Figure 3 This is an example diagram of the overall process architecture of the solution provided in the embodiments of this application.

[0015] Figure 4 This is a schematic diagram of the block-based boundary compression process for volumetric video information in this application.

[0016] Figure 5 This is a schematic diagram of the projection process of the Gaussian point from three-dimensional space to two-dimensional screen space in this application.

[0017] Figure 6 This is a structural block diagram of the electronic device provided in this application. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solutions of this application, the application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. However, this application can be implemented in many other ways different from those described below. Therefore, based on the embodiments provided in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0019] It should be noted that the terms "first," "second," "third," etc., in the claims, specification, and drawings of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. Such data are interchangeable where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown or described in this application. Furthermore, the terms "comprising," "having," and their variations are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0020] It should be understood that in the embodiments of this application, "at least one" means one or more, and "more than one" means two or more. "And / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the related objects before and after it are in an "or" relationship. "Contains A, B and / or C" means containing any one, two, or three of A, B, and C.

[0021] It should be understood that in the embodiments of this application, "B corresponding to A", "B corresponding to A", "A corresponds to B" or "B corresponds to A" means that B is associated with A, and B can be determined based on A. Determining B based on A does not mean that B is determined solely based on A; B can also be determined based on A and / or other information.

[0022] To facilitate understanding of the various embodiments of this application, the application background of the embodiments will be explained.

[0023] With the rapid development of virtual reality, spatial computing devices, and other technologies, volumetric video, as an emerging media form capable of recording dynamic scenes in three-dimensional space, has received widespread attention. Volumetric video is a video technology that captures dynamic scenes in three-dimensional space (such as performances or sporting events). Unlike traditional planar video, volumetric video records the depth and geometric information of the scene, allowing users to move their viewpoint with six degrees of freedom during playback. This means users can freely walk within the scene and view dynamic content from any angle, providing an immersive viewing experience. Related technologies typically generate volumetric videos by independently modeling each frame and storing the model information for each frame independently. This method results in excessively large volumes of volumetric video data, which is detrimental to storage and transmission, and also consumes significant computing resources during playback, affecting playback smoothness.

[0024] To address the above issues, embodiments of this application provide a method for generating and playing large-format videos, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. The aim is to reduce the data size of large-format videos, facilitating storage and transmission, reducing the computational resources required for playback, improving the smoothness of playback, and enhancing the user's viewing experience.

[0025] The volumetric video generation method provided in this application can be applied to the generation of volumetric videos in various fields, specifically in scenarios such as object display, digital human content creation, virtual live streaming, and immersive performances. This solution can be deployed on local terminal devices or cloud servers to complete the volumetric video generation process according to actual needs. For example, in spatial computing scenarios, when users watch interactive digital human content through virtual reality devices, the solution in this application can reduce the storage bandwidth and computing resource consumption of volumetric videos while ensuring viewing quality, thus improving the smoothness of interactive viewing. This application does not specifically limit the specific application scenarios of the solution provided.

[0026] To facilitate understanding of the method embodiments of this application, their application scenarios are described. Please refer to... Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of the solution provided in the embodiments of this application. This application scenario is merely an illustrative example and is not intended to limit the specific application scenario. Figure 1 As shown, in this application scenario, a terminal device 101 and a server 102 are provided. The terminal device can be an electronic device with rendering and playback capabilities, such as a spatial computing device, VR / AR device, personal computer, or mobile phone. The server can be a cloud server or a local server. Reference videos can be uploaded to the server 102, which executes the volumetric video generation method provided in this application to generate corresponding volumetric video information. The volumetric video information is then sent to the terminal device 101, which executes the volumetric video generation or playback method provided in this application to generate a volumetric video frame or play the volumetric video.

[0027] In actual deployment, the generation of volumetric video information can also be completed on the local terminal device. After generation, the video image is directly rendered and played locally. This application does not impose specific restrictions on the deployment method.

[0028] Example 1 The method for generating volumetric video according to the first embodiment of this application will now be described in detail with reference to the accompanying drawings. Please refer to... Figure 2 , Figure 2This is a schematic flowchart of an example of a method for generating volumetric videos according to an embodiment of this application. The execution entity of the volumetric video generation method in this embodiment can be a server, a terminal device, or a combination of both. This embodiment uses a server as the execution entity for illustration. Figure 2 As shown, the method includes the following steps S110 to S140.

[0029] Step S110: Obtain a reference video, which contains the object to be modeled.

[0030] In this step, the reference video can be the original video of the object to be modeled, which can be any dynamic object such as a person or an item.

[0031] The aforementioned reference videos may include multiple original video clips captured from multiple angles. This multi-view acquisition method provides more comprehensive reference information for subsequent 3D modeling, improving the accuracy of the generated digital human model. Alternatively, the reference video may include a single video clip that covers the object to be modeled from multiple perspectives, which can also meet the modeling requirements. This application does not limit the specific method of acquiring the reference videos.

[0032] In this embodiment, the reference video can be a two-dimensional video, which can reduce the amount of data in the original acquisition. Alternatively, the reference video can also be an acquired three-dimensional depth video to improve the modeling accuracy and precision. Those skilled in the art can choose according to actual needs, and this application does not impose any limitations.

[0033] Step S120: Generate a digital human model corresponding to the object to be modeled based on the reference video, and determine the attribute information of each Gaussian point in the digital human model in each frame.

[0034] When generating a digital human model corresponding to the object to be modeled based on a reference video, a 3D Gaussian sputtering method can be used to transform the object in the reference video into a digital human model containing multiple Gaussian points, thereby completing the construction of the digital human model. Specifically, based on the shape of the object to be modeled in the reference video, an initial 3D model can be obtained through 3D reconstruction, and then the initial 3D model can be Gaussian discretized to decompose it into multiple discrete Gaussian points, thus obtaining a digital human model containing multiple Gaussian points. Alternatively, the skeletal motion information of the object to be modeled can be extracted from the reference video, combined with a preset basic digital human template, and the template model can be adjusted to obtain a target digital human model that matches the shape and motion of the object to be modeled. Then, the target digital human model can be Gaussian discretized to obtain a corresponding digital human model containing multiple Gaussian points. This application does not limit the specific generation method of the digital human model.

[0035] Optionally, Gaussian points in the digital human model can also be sampled from the mesh surface reconstructed by NeuS2, i.e., from the surface of the mesh model obtained by NeuS2 reconstruction. NeuS2 reconstruction is a neural implicit surface reconstruction method for 3D reconstruction, capable of reconstructing continuous 3D mesh models based on multi-view video data. Compared to initializing Gaussian points from scratch, sampling initial Gaussian points from the mesh model surface reduces the number of iterations required for subsequent optimization, speeds up modeling, and ensures that the distribution of Gaussian points matches the spatial morphology of the object being modeled, thus improving modeling accuracy. Alternatively, uniform spatial sampling can be used to generate the digital human model, i.e., Gaussian points are uniformly sampled throughout the entire modeling space at preset intervals. This method is simple to implement and can quickly achieve initial distribution even for irregularly shaped objects, adapting to more scene requirements. This application does not limit the specific initial generation method of Gaussian points; those skilled in the art can choose according to actual computing power and accuracy requirements. The number of point clouds (i.e. the number of Gaussian points) included in the digital human model can be less than 50,000, which allows for faster iterative optimization. The specific number of Gaussian points can be adjusted according to the complexity of the object to be modeled and the accuracy requirements. This application does not impose a specific limit on this number.

[0036] In this step, Gaussian points are used as the basic units for constructing the 3D model. The attribute information corresponding to each Gaussian point may include position, rotation, scale, opacity, color (spherical harmonic coefficient), displacement, etc. Those skilled in the art can determine the specific attributes included according to the actual rendering requirements. The above attribute information is used to indicate the display effect of the Gaussian points after rendering. By determining the attribute information of each Gaussian point in each frame, the shape changes of the object to be modeled in each frame can be characterized.

[0037] When determining the attribute information of Gaussian points in each frame, the attribute information of Gaussian points in the previous frame can be used as the initial attribute information for the current frame. The initial attribute information can be fine-tuned by combining it with the current frame of the reference video to obtain the attribute information of Gaussian points in the current frame. This method can improve the efficiency of attribute determination and ensure the smoothness of the change of Gaussian point attributes between adjacent frames, avoiding the problem of jump jitter during rendering.

[0038] In this embodiment, the attribute information of each Gaussian point in the current frame can be determined based on preset rigid constraints. These rigid constraints are used to ensure that the Gaussian points maintain their relative positional relationship with surrounding Gaussian points as much as possible during the movement of adjacent frames. This avoids relative positional shifts in Gaussian points within the same rigid component when adjusting attributes frame by frame, ensuring the rationality of the digital human model structure and improving the accuracy of the modeling results. When determining the attribute information of Gaussian points, the topological relationship of the Gaussian points in the digital human model can be kept fixed, and the attribute information of each Gaussian point can be adjusted frame by frame only based on rigid constraints. This adapts to the dynamic changes of the object being modeled while avoiding unreasonable relative displacements between Gaussian points, ensuring modeling quality.

[0039] Step S130: Based on the attribute information corresponding to the Gaussian point in each frame, determine the change pattern information of the attribute information of the Gaussian point on the time axis of the reference video.

[0040] In this step, the change pattern information is information that can characterize the attribute information of the Gaussian point as it changes over time. For example, the change pattern information can be obtained by fitting the attribute information of the Gaussian point in each frame, such as by using keyframe interpolation. Only the parameters corresponding to the change pattern information need to be stored, and there is no need to store the complete attribute information for each frame.

[0041] The aforementioned pattern of change can be represented by a pattern of change function. Each Gaussian point corresponds to its own pattern of change function, which takes time as the input variable and outputs the attribute value of the corresponding attribute at the input time point.

[0042] Alternatively, the aforementioned variation pattern information can also be represented by continuously changing splines. For example, B-splines, Bezier splines, cubic Hermit splines, etc., can be used to fit the attribute points of each frame to obtain continuously changing splines for the corresponding attributes. This can also characterize the variation pattern of attributes over time. In this case, the variation pattern information includes the splines and the control point parameters of the splines.

[0043] Alternatively, the variation pattern information can also be represented in the form of keyframes. That is, the variation pattern information includes the attribute information of Gaussian points corresponding to multiple keyframes and interpolation rules. The attribute information at any non-keyframe time can be calculated by interpolation between adjacent keyframes, thus obtaining the attribute information corresponding to the Gaussian point at any time. Furthermore, only the attribute information and interpolation rules of the keyframes need to be stored, eliminating the need to store the attribute information of all frames, effectively reducing the data volume. This application does not limit the specific representation form of the variation pattern information, as long as the attribute information corresponding to the Gaussian point at any time can be calculated from this information.

[0044] Among them, the continuous change law function can obtain smoother attribute values ​​at any time, making the rendered volumetric video dynamic effect more natural and smooth. The key frame representation is simple to implement and the interpolation calculation speed is faster, which can adapt to application scenarios with different computing power requirements. The spline representation can reduce the parameter storage while ensuring the smoothness of change, and adapt to more different needs. Those skilled in the art can choose the appropriate representation according to the actual situation.

[0045] By determining the variation pattern information, the original discrete Gaussian point attributes of multiple frames can be transformed into a continuous and differentiable parameterized representation. When the Gaussian point attribute corresponding to any time point is needed, the corresponding result can be obtained directly by interpolation calculation based on the variation pattern information. There is no need to store the attribute information of all frames, which greatly reduces the amount of data that needs to be stored in the video.

[0046] In this step, for each Gaussian point, the corresponding variation pattern information can be determined separately for different attributes. Each attribute is fitted with its variation pattern independently, which can improve the accuracy of the fitting results and better adapt to the variation characteristics of different attributes. Alternatively, multiple attributes of the same Gaussian point can be integrated to uniformly determine the variation pattern information, thereby improving fitting efficiency. This application does not limit this approach.

[0047] When determining the variation pattern information, if the reference video is a dynamic video segment of the object to be modeled, the variation pattern information within the corresponding time range of the reference video can be obtained by fitting the attribute information corresponding to the Gaussian points in each frame of the reference video. The variation pattern information can also be obtained by combining keyframes; that is, first determine the attribute information of each Gaussian point in multiple keyframes, and then interpolate the attribute value at any time based on the attribute information of the keyframes. The attribute information corresponding to the keyframe is also a form of variation pattern information, and this application does not limit the specific form of the variation pattern information. After obtaining the variation pattern information through fitting, only the parameters corresponding to the variation pattern information need to be stored, without needing to store the complete attribute information of the corresponding Gaussian points in each frame, significantly reducing the amount of data required for storage.

[0048] In one implementation, step S130 can be implemented by following steps S131 to S132.

[0049] Step S131: Using the attribute information corresponding to the Gaussian point in each frame as sampling points, determine the initial change pattern information corresponding to the attribute information of the Gaussian point on the time axis.

[0050] Specifically, multiple sampling points corresponding to the attribute of a Gaussian point can be extracted from each frame. Each sampling point corresponds to the time position of a frame on the time axis, and the value of the sampling point is the attribute value of the Gaussian point in that frame. Then, based on multiple sampling points, fitting is performed to obtain initial variation information covering the entire time range of the reference video. This initial variation information can be represented in a parameterized form to show the trend of attribute change over time throughout the entire time range.

[0051] Cubic Hermitian splines can be used for fitting. By setting the function values ​​and derivatives at the endpoints, cubic Hermitian splines can produce smooth interpolation results. By explicitly controlling the tangent, they can easily ensure the velocity continuity of adjacent curve segments at the connection point, eliminating motion jitter. Modifying a control point or tangent only affects adjacent curve segments, making the optimization during gradient backpropagation more stable and efficient. It only requires calculating simple polynomial combinations, without complex recursive solutions, making it suitable for parallel rendering of large-scale Gaussian points.

[0052] This application does not limit the specific fitting method, as long as it can obtain the initial change law information of the characterization attribute over time.

[0053] The above-mentioned initial change pattern information can reflect the initial change trend of Gaussian points, but there may still be fitting deviations and errors between it and the original sampling points of each frame. Therefore, it is necessary to optimize the initial change pattern information through step S132.

[0054] In one specific embodiment, step S131 can be implemented according to the following steps S131a~S131b.

[0055] Step S131a: Using the attribute information corresponding to the Gaussian point in each frame as sampling points, determine the discrete initial change trajectory of the attribute information of the Gaussian point on the time axis.

[0056] In this discrete initial change trajectory, each trajectory point corresponds to the attribute value of the Gaussian point in a video frame. The trajectory points are arranged in order along the time axis, thus obtaining the initial discrete change trajectory of the Gaussian point attribute.

[0057] Step S131b: Model the initial change trajectory and generate an initial change pattern spline corresponding to the initial change trajectory. The initial change pattern spline is used to represent the initial continuous change pattern of the attribute information of the Gaussian point on the time axis.

[0058] Specifically, initial variation pattern splines covering the entire time axis can be obtained by fitting each sampling point in the initial variation trajectory. For example, cubic Hermitian spline fitting or B-spline fitting can be performed to obtain the control point parameters corresponding to the initial variation pattern splines. The discrete sampling points are then transformed into a continuously differentiable parameterized representation, resulting in continuous initial variation pattern splines connecting all sampling points. The attribute values ​​corresponding to each sampling point, i.e., the Gaussian points in each frame, can be directly obtained at any given time by fitting the discrete points into a continuous spline form. This eliminates the need for additional inter-frame interpolation calculations, and the parameter storage requirements are lower than storing all frame attributes.

[0059] The aforementioned initial variation pattern splines are continuous parametric curves that can provide predicted values ​​for the corresponding attributes at any given time point, thus completing the conversion from discrete trajectories to continuous parametric representations. These initial variation pattern splines represent the aforementioned initial variation pattern information.

[0060] This embodiment converts the attribute information of discrete frames into a continuous parameterized representation, which greatly reduces the amount of stored data while ensuring the temporal continuity of attributes, thus facilitating the storage, transmission and rendering of subsequent volumetric videos.

[0061] In one specific embodiment, the attribute information of the Gaussian point may include a first type of attribute information applicable to Euclidean space interpolation and / or a second type of attribute information applicable to manifold space or tangent space interpolation. The first type of attribute information includes at least one of position information, color information, and transparency information. The second type of attribute information includes at least rotation information. Each attribute information of the Gaussian point corresponds to its own initial change trajectory.

[0062] Step S131b can be implemented by following steps A to B to achieve the initial variation law spline.

[0063] Step A: For the initial change trajectory corresponding to the first type of attribute information, perform spline interpolation on the initial change trajectory in Euclidean space to generate the initial change pattern spline corresponding to the first type of attribute information.

[0064] Step B: For the initial change trajectory corresponding to the second type of attribute information, the second type of attribute information is transformed into the tangent space to obtain the corresponding attribute vector. Spline interpolation is performed on the attribute vector in the tangent space to generate the initial change pattern spline corresponding to the second type of attribute information.

[0065] Since the first type of attribute information exhibits linear superposition in Euclidean space, accurate fitting results can be obtained directly through interpolation in Euclidean space. However, the second type of attribute information, such as rotation, does not satisfy the superposition rule of linear space, and direct interpolation in the original space can easily lead to erroneous intermediate results. By transforming it to tangent space, rotation can be converted into a linear vector in tangent space, and interpolation can then yield more accurate fitting results of the variation pattern, ensuring the reasonableness of the rotation attribute interpolation and improving the accuracy of dynamic rendering. Using corresponding spatial interpolation for attributes of different properties can adapt to the mathematical characteristics of different attributes, significantly improving the fitting accuracy of the variation pattern of each attribute.

[0066] For example, since the unit quaternion of the rotation attribute is located on a non-Euclidean hypersphere, direct linear spline interpolation of the quaternion can lead to normalization failure or non-uniform angular velocity. This application adopts a tangent space interpolation strategy. First, the quaternion is mapped to the Lie algebra space through logarithmic mapping to obtain the rotation vector. Then, cubic Hermitian spline interpolation is performed on the rotation vector in Euclidean space. Finally, the interpolation result is mapped back to the quaternion space through exponential mapping. In this way, the smoothness and physical rationality of the rotation change are guaranteed.

[0067] For example, a quaternion q can be transformed to the tangent space through a logarithmic mapping to obtain the corresponding rotation vector r. The logarithmic mapping maps the quaternion to the Lie algebra space (rotation vector space), and its mathematical expression is: (1) Then, perform cubic Hermitian spline interpolation on the rotation vector r in Euclidean space to obtain the continuous rotation vector trajectory r(t). Assume the current time t is within the interval [t] of the k-th control point. k ,t k+1 Within this range, defining the normalized time τ, the mathematical expression for cubic Hermitian spline interpolation is: (2) in, The rotation vector values ​​for the two endpoints of the interval are m and m respectively. k and m k+1 This is the corresponding tangent line (rate of change).

[0068] The interpolated rotation vector r(t) is then mapped back to the quaternion space via an exponent to obtain the continuous quaternion rotation trajectory q(t): (3) This tangent space interpolation method avoids complex quaternion interpolation calculations while ensuring the smoothness and physical rationality of rotational changes.

[0069] Step S132: Using the reference video as a reference, adjust the initial change pattern information to obtain the change pattern information corresponding to the attribute information of the Gaussian point on the time axis, so that the change pattern information is consistent with the state change of the object to be modeled in the reference video.

[0070] Specifically, the Gaussian point attributes calculated from the initial change pattern information at any given time can be used to render the predicted image for that time. The predicted image is then compared with the original image from the reference video at the corresponding time, and the fitting loss is calculated. Based on this fitting loss, backpropagation is performed to optimize the parameters of the initial change pattern information. The parameters are adjusted until the fitting loss meets the preset convergence condition, ultimately yielding accurate change pattern information. This optimization step reduces the deviation between the fitted change pattern and the actual state changes of the object to be modeled in the original reference video, improving the accuracy of the subsequently generated volumetric video.

[0071] In this embodiment of the application, the initial variation law information can be the above-mentioned fitted spline or polynomial, and the parameters of the variation law information are the control point parameters of the spline or the coefficients of the polynomial.

[0072] This implementation method first fits the initial change pattern and then optimizes and adjusts it. While ensuring the parameter compression effect, it can improve the matching degree between the change pattern and the original motion state of the object to be modeled, thus balancing storage performance and modeling accuracy.

[0073] In one specific embodiment, when the above-mentioned initial change pattern information is the above-mentioned initial change pattern spline, step S132 can be implemented according to the following steps S132a to S132b.

[0074] Step S132a: Determine multiple control points on the initial variation law spline.

[0075] Specifically, the acceleration at each time step on the initial variation pattern spline can be determined. Based on the principle that the control point distribution density is positively correlated with the acceleration, multiple control points are determined on the initial variation pattern spline. That is, when the acceleration of a certain attribute of a Gaussian point is greater between two time points, it indicates that the change of that attribute is more drastic during this period, and more control points are needed to accurately represent this change; while the smaller the acceleration, the more gradual the attribute change, and only fewer control points are needed to represent the change trend. By allocating control points in this way, the total number of control points can be further reduced while ensuring the accuracy of the variation pattern representation, thus compressing the amount of data required to store the variation pattern information.

[0076] Alternatively, multiple control points can be determined on the initial variation spline at equal time intervals. Setting control points at equal intervals is simple to implement and easy to calculate, eliminating the need to calculate the acceleration at each time step. This method is suitable for motion scenarios with relatively uniform trends and reduces computational complexity.

[0077] This application does not limit the specific distribution of control points; those skilled in the art can select an appropriate distribution method based on the characteristics of the motion changes.

[0078] Step S132b: Using the reference video as a reference, adjust each of the control points on the initial variation pattern spline to obtain the adjusted variation pattern spline, and determine the adjusted variation pattern spline as the variation pattern information corresponding to the attribute information of the Gaussian point on the time axis.

[0079] Specifically, the attribute values ​​of Gaussian points at any time can be calculated based on the initial variation pattern spline. Then, the predicted image at the corresponding time is rendered. The predicted image is compared with the original image at the corresponding time in the reference video to calculate the fitting loss. The parameters of each control point are adjusted through backpropagation until the fitting loss meets the convergence condition, and finally the adjusted variation pattern spline is obtained.

[0080] Adjusting each control point can include adjusting its position coordinates or attribute values. Different types of splines require different parameter adjustments. Ultimately, the error between the image rendered by the attribute values ​​calculated by the adjusted spline at any given time and the original image at the corresponding time in the reference video should be minimized, ensuring that the change pattern can accurately reproduce the actual motion changes of the object to be modeled.

[0081] By first determining the initial variation pattern splines and then adjusting the control points accordingly, we can accurately represent the variation pattern of attributes with fewer control points. This further reduces the amount of stored data and improves storage and transmission efficiency while ensuring modeling accuracy.

[0082] In a specific embodiment, step S132b can obtain the adjusted variation pattern spline by the following steps: based on the principle that the intensity of motion of the object to be modeled in the reference video is positively correlated with the distribution density of control points, the distribution position and number of each control point on the initial variation pattern spline are adjusted to obtain the preliminary adjusted variation pattern spline; with the reference video as a reference, the parameter values ​​of each control point on the preliminary adjusted variation pattern spline are adjusted to obtain the adjusted variation pattern spline.

[0083] Specifically, higher intensity of motion indicates more complex attribute changes, requiring more control points to accurately represent the details of the changes. Conversely, gentler motion allows for a reduction in the number of control points. By adjusting the distribution and number of control points in this way, the representation accuracy of complex changing regions can be prioritized within a limited number of control points, further balancing storage volume and representation accuracy to adapt to the needs of different motion scenarios. After adjusting the distribution and number of control points, backpropagation is used to optimize the parameter values ​​of each control point, ultimately yielding accurate splines illustrating the change patterns.

[0084] The intensity of motion of the object to be modeled in the reference video can be determined in the following ways: Calculate the change amplitude of the attribute corresponding to the same Gaussian point between adjacent frames in the reference video, take the average of the change amplitudes of each Gaussian point, and compare it with the preset amplitude threshold to obtain the overall intensity of motion; Alternatively, calculate the change amplitude of the attribute in the adjacent frames of each Gaussian point to obtain the local intensity of motion of each Gaussian point, and adjust the control point distribution density of a single Gaussian point accordingly, allocating more control points to Gaussian points with fast attribute changes to further improve the utilization efficiency of control points and obtain higher modeling accuracy when the total number of parameters is fixed.

[0085] In one implementation, step S130 can divide the reference video into multiple video segments; for each video segment, based on the attribute information corresponding to the Gaussian point in each frame of the video segment, determine the segmented change pattern information corresponding to the attribute information of the Gaussian point in the time interval corresponding to the video segment, so that the volume video information includes the segmented volume video information corresponding to each video segment respectively.

[0086] When dividing the video into segments, the reference video can be divided into multiple segments of equal length to simplify the segmentation process. Alternatively, the reference video can be divided into segments of varying lengths based on the intensity of motion changes of the object to be modeled. Slower motion intervals can be divided into longer segments to enhance temporal continuity, while more drastic motion intervals (such as waving or jumping) can be divided into shorter segments to reduce the accumulation of errors caused by non-rigid deformation. For each segment, the segmented change pattern information within the corresponding time interval is fitted using the aforementioned steps. Finally, the segmented change pattern information of each video segment is combined to obtain the complete Gaussian point attribute change pattern across the entire time axis. The length of each video segment can be between 60 and 200 frames, or within other frame length ranges, to suit specific scenario requirements. This application does not impose specific limitations on the segmentation method or segment length. By fitting the variation patterns in segments, the time span of a single variation pattern can be shortened, reducing the fitting error of non-rigid motion. Even complex motions with long time sequences can obtain accurate fitting results. At the same time, the computational cost of single-segment fitting is smaller, making it more suitable for parallel computing and improving the overall modeling efficiency.

[0087] In this embodiment, if the motion changes of the object to be modeled in the reference video vary significantly across different time periods—for example, the motion is gentle in the first half and intense in the second half—or different motion patterns exist across different time periods, determining the change pattern information for each time period allows for the adaptation of corresponding parameters to the change characteristics of each time period. This avoids the problem of a single-segment change pattern failing to accurately adapt to the change characteristics of different time periods, further improving fitting accuracy without adding excessive storage. Each segment's change pattern information is also stored in the form of corresponding parameters, with each segment corresponding to its own parameters. The final volumetric video information includes the volumetric video information corresponding to each segment. During rendering, the corresponding video segment is located based on the time to be played, and the segment's change pattern information is retrieved to calculate the attribute values, without increasing the computational complexity during rendering. By adapting the change pattern segmentally, the fitting accuracy of the change pattern can be further improved with a fixed parameter storage capacity, adapting to more complex dynamic scenes. Alternatively, this application can reconstruct the entire reference video without segmentation. In this case, the number of curve control points can be increased to cover long-term motion, and the specific configuration can be adaptively adjusted based on the video memory capacity of the deployment environment.

[0088] In one specific embodiment, the time interval in the segmented video information corresponding to each video segment can be a normalized time interval or the time interval of the original timeline of the corresponding reference video; this application does not impose any specific limitation. When multiple segments adopt normalized time intervals, each segment uses a unified time scale, which facilitates independent calculation of attribute values ​​for each segment and adapts to the flexible division of video segments of different lengths.

[0089] Step S140: Based on the variation pattern information corresponding to each Gaussian point and the digital human model, generate volumetric video information corresponding to the reference video.

[0090] After obtaining the volumetric video information in this step, subsequent volumetric video rendering and playback can be performed based on this information. When it is necessary to obtain the Gaussian point attributes at any time during playback, the attributes at the corresponding time can be calculated directly by combining the time information corresponding to that time with the change pattern information of each Gaussian point. There is no need to pre-store the complete attributes corresponding to each frame.

[0091] In this step, the variation information corresponding to each Gaussian point and the digital human model can be determined as the volumetric video information corresponding to the reference video; alternatively, the basic Gaussian point topology information, static attribute information, and variation information of each Gaussian point of the digital human model can be integrated to determine the volumetric video information corresponding to the reference video. The basic Gaussian point topology information is used to characterize the relative topological structure between Gaussian points, the static attribute information is the attribute content that does not change over time, and the dynamic variation information is the variation information corresponding to the aforementioned attributes that change over time. This classification and storage method can further simplify the storage content and reduce redundant storage. After the volumetric video information is generated, it can be directly stored on the server or sent to the corresponding terminal device according to the terminal's playback request, and the terminal device will complete the subsequent rendering and playback.

[0092] The volumetric video generation method provided in this application only stores the information on the change law of Gaussian point attributes over time, without storing the complete model data obtained by independent modeling of each frame. This significantly reduces the data volume of the volumetric video, making it easier to store and transmit. During playback, it is only necessary to calculate the Gaussian point attributes required at the current playback moment, without loading the full multi-frame data, which reduces the computational resource consumption during playback and improves playback smoothness.

[0093] Compared to related technologies that require storing model information obtained from independent modeling for each frame, this application only stores the variation law of Gaussian point attribute information over time, and only stores one set of Gaussian point topology (i.e., digital human model). It eliminates the need for independent storage of model information for each frame, significantly reducing the data volume of volumetric video information. This facilitates the storage and network transmission of volumetric videos. When playing volumetric videos, it is only necessary to calculate the required Gaussian point attribute information at any given time based on the variation law information, without loading the full multi-frame model data. This effectively reduces the computational resources consumed during playback, improves the smoothness of volumetric video playback, and enhances the user's immersive viewing experience. It also achieves a high compression ratio (10-100 times compression compared to frame-by-frame solutions) while ensuring the continuity of the point cloud on the time axis, eliminating inter-frame flicker, and making video playback smoother.

[0094] In this embodiment, 3D Gaussian sputtering can be used to generate digital human models. 3D Gaussian sputtering is an explicit 3D scene representation and rendering technique that uses millions of anisotropic three-dimensional Gaussian ellipsoids (i.e., Gaussian points) to represent the scene. Each Gaussian point carries attribute information such as position, rotation, scaling, opacity, and spherical harmonic color. This application introduces a time variable on the basis of 3D Gaussian sputtering, extending it to 4D Gaussian sputtering. It describes the movement and changes of Gaussian points over time through parameterized trajectories, aiming to solve the problems of high-fidelity reconstruction, storage compression, and real-time playback of dynamic scenes.

[0095] In one embodiment, step S140 can generate volumetric video information corresponding to the reference video according to the following steps S141~S142.

[0096] Step S141: Determine the hybrid deformation parameters corresponding to the digital human model based on the reference video.

[0097] Specifically, the blendshapes parameter for each frame can be calculated based on the image content at the corresponding moment in the reference video. This parameter is used to characterize the degree of offset of deformable parts such as facial expressions and limb shapes in the digital human model relative to the baseline shape, and can accurately restore the shape details of the digital human in the reference video at different moments.

[0098] The aforementioned hybrid deformation parameters can also be directly output by the motion capture device without additional calculation from the reference video, which can further improve the efficiency of parameter acquisition and is suitable for scenarios where motion capture devices collect data.

[0099] Hybrid deformation parameters can accurately characterize the dynamic changes in the overall shape of a digital human with relatively few parameters, further compressing the amount of data in the overall volumetric video information.

[0100] Step S142: Based on the hybrid deformation parameters, the variation law information corresponding to each Gaussian point, and the digital human model, generate the volumetric video information corresponding to the reference video.

[0101] Specifically, the aforementioned hybrid deformation parameters, the variation pattern information corresponding to each Gaussian point, and the digital human model can be used to determine the volumetric video information corresponding to the reference video. Alternatively, the hybrid deformation parameters and the variation pattern information of the Gaussian points can be integrated. The overall shape of the digital human model can be adjusted first using the hybrid deformation parameters, and then the variation pattern information of the Gaussian points can be combined to supplement detailed dynamic changes, thus obtaining the volumetric video information. This method of combining hybrid deformation parameters can further improve the accuracy of volumetric video dynamic reconstruction.

[0102] While cubic Hermitian splines and other variation patterns can effectively capture the macroscopic nonlinear motion (low-frequency components) of an object, their fitting ability for instantaneous high-frequency jitter or minute deformations (such as the rapid flipping of fabric wrinkles or facial micro-expressions) is limited by the density of control points. Higher control point density improves the fitting accuracy of high-frequency details, but also increases data storage and is prone to overfitting. Therefore, in this embodiment, a learnable hybrid shape can be introduced as a high-frequency residual compensation term.

[0103] For example, Gaussian point G i The position at time t can be modeled as the superposition of spline basis and dynamic residual, which can be determined by the following formula: (4) Among them, spline items Determined by Hermite spline curves, the main motion responsible for smoothing, i.e., the low-frequency component, is calculated using spline curves corresponding to Gaussian points, and the residual term is obtained. It is obtained by linear combination of a set of learnable orthogonal bases (mixed shapes), representing the instantaneous high-frequency deformation components. It only requires a small number of basis vectors to characterize complex high-frequency details without significantly increasing the amount of stored data. At the same time, it can effectively compensate for the high-frequency error of spline fitting and further improve the accuracy of dynamic scene restoration.

[0104] In the above formula, ,in, , where j is the j-th mixed shape basis, and represents the j-th typical high-frequency displacement mode corresponding to the ith Gaussian point; Let be the mixing coefficient of the j-th basis at time t, which can be predicted by a lightweight neural network; E is the number of basis points, which is usually much smaller than the number of control points K, so as to effectively cover the high-order variation space with a small number of parameters.

[0105] In the above embodiments, spline curves focus on modeling large-scale rigid displacements and low-frequency non-rigid deformations, ensuring the overall smoothness of the time series; hybrid shapes focus on capturing high-frequency details with strong temporal locality, eliminating the need to add control points across the entire time axis. This significantly improves the system's fitting accuracy for complex dynamic scenes while maintaining low memory usage. By superimposing spline bases with dynamic residuals, the system can still guarantee the accuracy of detail reproduction even with a limited number of control points and a small overall data volume, balancing data compression and modeling quality.

[0106] Because hybrid deformation parameters can accurately characterize the detailed deformations of a digital human form, the motion results obtained by combining hybrid deformation parameters with information on variation patterns can jointly reconstruct a complete dynamic digital human model that includes both overall motion and local detailed deformations. This ensures accurate reconstruction of detailed changes even with a limited number of control points and a relatively small overall data volume, balancing data compression and modeling quality. By combining hybrid deformation parameters with the variation pattern information of Gaussian point attributes, the storage volume of macroscopic motion parameters is compressed using these variation patterns, while high-frequency detailed deformations are supplemented and reconstructed using hybrid deformation parameters. This solves the problem of balancing storage volume and accuracy when relying solely on variation patterns to fit details.

[0107] In one embodiment, during the adjustment of the initial change pattern information, the above method may further include the following steps S150-S160.

[0108] Step S150: Determine the Gaussian points from the Gaussian points where the consistency error with the reference video exceeds a preset error threshold.

[0109] Step S160: Split the error Gaussian point into Gaussian points to obtain at least one split Gaussian point corresponding to the error Gaussian point, and determine the initial change law information corresponding to the split Gaussian point.

[0110] Specifically, if the rendering error of a Gaussian point relative to the corresponding position in the reference video still exceeds a preset threshold after the current optimization adjustment, it indicates that a single Gaussian point cannot accurately represent the detailed changes of the object to be modeled at that position. In this case, the Gaussian point is split into multiple smaller Gaussian points. After initializing the change law information for the newly split Gaussian points, they are then reintroduced into the subsequent optimization adjustment process. This can improve the ability to represent complex detailed changes, further optimize the modeling accuracy, and avoid the problem of insufficient restoration accuracy caused by excessive smoothing of motion details by large-sized Gaussian points.

[0111] After the split is completed, the new Gaussian points obtained from the split can be added to the subsequent parameter optimization process to continue adjusting the variation information parameters of all Gaussian points until the overall fitting error meets the convergence condition, and finally obtain more accurate volumetric video information.

[0112] This implementation method can adaptively adjust the number and distribution of Gaussian points by splitting the Gaussian points with large errors. More Gaussian points are allocated in areas with complex motion and rich details to improve the representation accuracy, while the original large Gaussian points are retained in flat areas to avoid wasting parameters. This can further improve the modeling and restoration accuracy of volumetric videos while controlling the overall data volume, and restore richer dynamic details.

[0113] When determining the initial variation law information corresponding to the split Gaussian point, the variation law information of the error Gaussian point corresponding to the split Gaussian point can be determined as the replication variation law information corresponding to the split Gaussian point; a perturbation is added to the replication variation law information to obtain the initial variation law information corresponding to the split Gaussian point.

[0114] Specifically, by adding small random perturbations to the information on the pattern of change, the new Gaussian points after splitting can quickly adapt to the detailed changes in the original position, shortening the convergence time of subsequent optimization and adjustment. It can also make the two split Gaussian points have certain differences in position and attributes, better splitting and covering the detailed regions that cannot be represented by the original single Gaussian point, thus improving the ability to represent details.

[0115] When splitting the error Gaussian points, the splitting can be based on the principle that the number of Gaussian points in the digital human model does not exceed a preset upper limit threshold. This avoids unlimited growth in the number of Gaussian points, controls the total data volume of the overall video information, and balances storage and transmission costs with modeling accuracy. The upper limit threshold can be determined based on the video memory capacity of the rendering device. When multiple video segments are included, the splitting can be based on the principle that the number of Gaussian points corresponding to a single video segment does not exceed a preset segmentation threshold. This ensures that the modeling accuracy of a single segment meets the requirements while controlling the overall data volume from excessive growth, adapting to the storage and computing capabilities of different devices. The segmentation threshold can be any value between 200k and 300k, or other values ​​within the same range. The specific value can be adjusted according to the deployment environment to ensure the adaptability of this solution to different hardware devices. Here, 200k refers to 200,000 Gaussian points.

[0116] The process of generating volumetric video according to the first embodiment of this application is illustrated below by way of an example. Figure 3 This example provides a diagram illustrating the overall workflow architecture of the method for generating volumetric videos. Figure 3As shown, this application employs a two-stage generation method: The first stage is coarse-grained topological consistency tracking, which uses topologically consistent 3D Gaussian primitives for frame-by-frame tracking to obtain the initial change trajectory of Gaussian points, providing reliable initialization for subsequent stages. This stage can obtain the initial change trajectory corresponding to each attribute information and initialize the discrete change trajectory into the parameterized curve to be optimized, i.e., the initial change law information obtained above. The first stage corresponds to the execution process of step S131 above. The second stage is refined parameter curve modeling, which can determine the initial control points on the parameterized curve to be optimized, obtaining sparse splines. By jointly optimizing the spatial position and timestamp of the curve control points, continuous modeling of attribute information such as displacement, rotation, color, and transparency of Gaussian points is achieved. In addition, Gaussian point splitting is performed through adaptive Gaussian optimization and splitting strategies, increasing the number of Gaussian points and obtaining dense splines. The number of Gaussian points in the second stage increases to approximately 250,000. Residual compensation based on hybrid shape is also introduced, significantly enhancing the ability to express complex clothing textures and high-dynamic movements. For details, please refer to the above, which will not be elaborated here.

[0117] Example 2 The second embodiment of this application also provides a method for generating volumetric videos. This method is applied to electronic devices, which may be servers, desktop computers, laptops, mobile phones, tablets, smartwatches, smart TVs, VR devices, in-vehicle devices, wearable devices, terminal devices, or other electronic devices with data processing capabilities. This embodiment mainly uses an electronic device as a terminal device as an example for explanation. The implementation principles of other devices are similar to those of the terminal device and will not be described in detail here.

[0118] The volumetric video generation method provided in this embodiment is mainly used to generate volumetric video playback content by rendering video based on the volumetric video information generated in the first embodiment. The volumetric video generation method provided in this embodiment may include the following steps S210 to S240.

[0119] Step S210: Obtain volumetric video information, which is generated by the volumetric video generation method described in any one of the first embodiments.

[0120] The volumetric video information includes the variation patterns of each Gaussian point and a digital human model, and may also include hybrid deformation parameters. The volumetric video information can also include segmented volumetric video information. The volumetric video information can be retrieved from local storage or from a server storing the volumetric video information; the retrieval method should be determined based on the actual application scenario of the current device.

[0121] The 4D Gaussian point cloud file in the aforementioned volumetric video information can be in PLY format, and the data of each Gaussian point can be stored in a structured manner. The 4D Gaussian point cloud can include a static basic attribute layer, a dynamically changing attribute layer, and an additional compensation data layer. The volumetric video information can be stored through compression encoding. The compression process can employ multi-level compression strategies, including boundary compression, block storage, and variable-length data encoding. The data quantization precision of the volumetric video information can be freely selected, and various vector compression formats (such as Float32, Float16, Norm11, and Norm6) can be used to meet the requirements of different quality levels.

[0122] For example, please refer to Figure 4 , Figure 4 This is a schematic diagram illustrating the block-based boundary compression process for volumetric video information in an embodiment of this application. Figure 4 As shown, the compression process includes the following steps: First, acquire the volumetric video information, including the variation patterns of each Gaussian point. Each Gaussian point corresponds to variation patterns of multi-dimensional attributes such as position, color, and opacity.

[0123] Secondly, spatial block partitioning is performed. Gaussian points are divided into multiple chunks based on their spatial location, with each chunk containing a set of spatially adjacent Gaussian points. This spatial block partitioning strategy not only serves the purpose of compression but also optimizes memory access patterns. By organizing spatially adjacent Gaussian points into the same chunk, it improves cache hit rate and reduces memory access latency.

[0124] Then, boundary calculation and normalization are performed for each block. For all Gaussian points within each block, the minimum value of each attribute is calculated as the lower boundary, and the maximum value as the upper boundary, forming boundary information. Then, the actual attribute value of each Gaussian point is normalized: normalized value = (actual value - minimum value) / (maximum value - minimum value), compressing the attribute value to a numerical range of 0 to 1. The normalized value requires only a small number of bits to store, thus significantly reducing storage space.

[0125] Finally, the compressed storage structure is generated. The compressed data consists of two parts: boundary information (storing the minimum and maximum values ​​of each attribute in each block) and normalized attribute values. During rendering, the normalized values ​​are restored to the original value range using the boundary information: restored value = minimum value + normalized value × (maximum value - minimum value). This compression technique can significantly reduce storage space while maintaining accuracy, and is especially suitable for batch storage scenarios with a large number of Gaussian point attributes.

[0126] This compression method can significantly reduce storage space while maintaining accuracy. For example, the original attribute value may require 32 bits of floating-point storage, but after normalization, only 16 bits or even 11 bits are needed to achieve acceptable accuracy, reducing storage space by more than 50%. At the same time, due to the spatial adjacency within blocks, the attribute values ​​of Gaussian points within the same block usually have similar ranges, which makes the boundary information more compact and further improves compression efficiency.

[0127] After boundary compression, the normalized attribute values ​​can be further stored using different quantization formats. For example, for scenarios requiring high precision, the normalized values ​​can be stored in a 16-bit half-precision floating-point format; for scenarios requiring medium precision, an 11-bit compression format can be used; and for scenarios requiring low precision, a 6-bit compression format can be used. By combining boundary compression with quantization precision selection, a flexible balance between storage efficiency and rendering quality is achieved.

[0128] Step S220: Obtain the digital human model and the variation law information corresponding to each Gaussian point of the digital human model from the volumetric video information.

[0129] Specifically, based on the current playback time, the corresponding time interval can be located, and then the variation pattern information of each Gaussian point within that interval can be retrieved. If the volumetric video information is segmented and stored in segments, the target video segment can be determined first based on the current playback time, and then the required variation pattern information can be retrieved from the target segmented volumetric video information.

[0130] Alternatively, based on the time information corresponding to the current playback moment, the Gaussian point attribute value corresponding to that time information can be retrieved from the uniformly stored change pattern information to obtain the complete Gaussian point attribute of the current playback moment. Combined with the topological structure of the digital human model, the rendering preparation for the current frame can be completed.

[0131] Step S230: Determine the attribute information of the Gaussian point based on the change pattern information corresponding to the Gaussian point.

[0132] Specifically, if the variation pattern information is a parameterized expression corresponding to different attributes (such as the function form mentioned above), the current time information can be substituted into the function expression to obtain the corresponding values ​​of each attribute of the Gaussian point at that moment. If the variation pattern information is a spline form based on control point interpolation, the Gaussian point attribute value at the current moment can be calculated by interpolation between the corresponding control points based on the current playback time information, thus quickly obtaining the Gaussian point attribute information corresponding to any moment. If the volumetric video information contains hybrid deformation parameters, the corresponding hybrid deformation parameters can be obtained first based on the current moment to adjust the overall shape of the digital human model, and then combined with the currently calculated Gaussian point attributes to complete the subsequent rendering, ensuring the accuracy of the restoration of the overall shape and local details.

[0133] When calculating the Gaussian point attribute value at the current moment through interpolation, a spline interpolation method with non-uniform time intervals can be used to determine the attribute value. First, a binary search is used in the control point time array to locate the interpolation interval containing the current normalized time. Then, local interpolation parameters are calculated for subsequent spline interpolation calculations. In other words, it eliminates the need to traverse and search all control points; binary search can quickly locate the interpolation interval, significantly improving attribute calculation speed, shortening single-frame rendering preparation time, and further improving playback frame rate and ensuring smooth playback. In this application, binary search is... When different attribute information of a Gaussian point corresponds to its respective change pattern information, the current time information can be substituted into the change pattern information of the corresponding attribute to calculate the value of each attribute at the current time of the Gaussian point. Different attributes can be independently fitted with change patterns, further improving the fitting accuracy of different types of dynamic attributes. After obtaining the attribute information of each Gaussian point at the current time, the Gaussian algorithm can be used to render the video content of the current frame volume based on this attribute information and the digital human model, obtaining the rendering result at the corresponding time for playback and display.

[0134] For example, the attribute information of Gaussian points can be obtained through composite calculations. First, the keypoint information for position, rotation, and color is decompressed separately, and then the base offset is obtained through Hermite spline interpolation. For position, the direction vector can be extracted from the Gaussian point attributes, and a dot product is performed with the global pose parameters to generate additional position compensation. The accurate spatial coordinates are obtained by adding the base position to the calculated offset. For rotation, the rotation vector is converted into a quaternion, and the rotation offset quaternion is multiplied by the base rotation quaternion to obtain the composite rotation. For color, the color offset is directly superimposed on the base color coefficient of the spherical harmonic function.

[0135] Step S240: Render the Gaussian points according to their attribute information to generate a volumetric video frame.

[0136] Once the volumetric video frame corresponding to the current playback moment is rendered, it can be directly output and played on the display device, or it can be transmitted to other display devices for playback and display.

[0137] In one embodiment, when the volumetric video information includes volumetric video information corresponding to multiple video segments respectively, step S240 can generate volumetric video frames according to the following steps S241~S242.

[0138] Step S241: For each non-end video segment of the volumetric video, based on the attribute information of each Gaussian point of the non-end video segment, render each frame of the non-end video segment except for the last frame, and generate the volumetric video frame corresponding to the non-end video segment.

[0139] Step S242: For the last video segment of the volumetric video, render each frame of the last video segment according to the attribute information of each Gaussian point of the last video segment to generate the volumetric video frame corresponding to the last video segment.

[0140] Specifically, the last frame of a non-end video segment will be connected to the first frame of the next video segment. By skipping the last frame of a non-end video segment, duplicate frames at the connection points of adjacent video segments can be avoided, ensuring a stable video playback frame rate and preventing issues such as stuttering or repetition.

[0141] The segmented rendering method described above allows terminal devices to load and render video segments one by one, without having to load the full video file at once. This reduces memory and video memory usage during playback, avoids playback startup delays caused by loading all data at once, effectively reduces playback stuttering, improves the startup and playback smoothness of large-format videos, and is compatible with terminal devices with limited network bandwidth or weak hardware performance.

[0142] In one implementation, step S240 can be implemented according to the following steps S243~S244.

[0143] Step S243: Based on the content to be displayed at the current moment, determine the Gaussian point that is currently active.

[0144] Step S244: Render the active Gaussian points according to the attribute information of each Gaussian point to generate the volumetric video frame at the current moment.

[0145] Specifically, the Gaussian points in the active state can include at least one of the following: Gaussian points that are currently within the screen's visible range, or Gaussian points located within the current visible area based on the camera's viewpoint or viewing distance. For Gaussian points that are outside the visible range, obscured and invisible, or too far from the camera beyond the preset viewing distance, there is no need to render these Gaussian points. Only the active Gaussian points will be rendered. There is no need to perform the rendering process on all Gaussian points, which can effectively reduce the amount of rendering calculations for the current frame, further reduce the computing power burden on the terminal device, and improve rendering speed and playback smoothness.

[0146] In one specific embodiment, step S243 can determine the field of view of the camera at the current moment; based on the position information of each Gaussian point, select Gaussian points within the field of view from the Gaussian points of the digital human model, and determine the Gaussian points that are currently active based on the Gaussian points within the field of view.

[0147] The view frustum refers to the spatial range that the camera can cover during shooting. Only objects falling within this range will be captured in the image; objects outside the range will not appear in the final rendered image. By determining whether a Gaussian point is within the view frustum and filtering for active Gaussian points, invisible Gaussian points can be accurately removed from the current image. This minimizes redundant calculations and improves rendering efficiency while ensuring the integrity of the final rendered image.

[0148] When determining the active Gaussian point based on the Gaussian points within the range, depth information can be used to remove Gaussian points occluded by other objects within the range, further reducing invalid rendering calculations and better adapting to volumetric video rendering and playback in complex scenes.

[0149] For example, when determining the Gaussian point that is currently active based on the Gaussian points within the range, Gaussian points that meet preset non-rendering conditions at the current moment can also be removed from the Gaussian points within the range to obtain the Gaussian point that is currently active. The non-rendering conditions include at least one of the following: located in the opposite direction of the camera lens, the distance between the point and the camera is greater than a preset distance, and the transparency is higher than a preset transparency.

[0150] For example, Gaussian points that are too far away and have extremely low transparency contribute very little to the final image. Removing them in advance can further reduce unnecessary rendering calculations without significantly affecting the final image quality, but can effectively improve rendering speed. By combining view frustum selection with multi-condition culling to filter and activate Gaussian points, the amount of computation per frame can be reduced as much as possible while ensuring the rendering quality of volumetric video images. This effectively improves the playback smoothness of volumetric videos on terminal devices and adapts to scenarios with limited computing power, such as mobile devices.

[0151] Specifically, a view frustum intersection test can be performed on each Gaussian point, and Gaussian points located within the view frustum area can be retained as active Gaussian points. Since the position information of each Gaussian point has already been calculated through the change pattern information, only simple spatial position judgment is needed to complete the filtering, resulting in low computational cost. This can quickly reduce the number of Gaussian points that need to be rendered, significantly improving the rendering efficiency of a single frame.

[0152] In one embodiment, when the volumetric video information includes segmented volumetric video information corresponding to multiple video segments, step S240 can generate a volumetric video frame according to the following steps: sequentially determining the video segment to be rendered corresponding to each of the multiple video segments according to the time intervals respectively; for the current time interval, storing the attribute information of each Gaussian point of the corresponding video segment to be rendered into a rendering buffer, and rendering each Gaussian point in the video segment to be rendered based on the attribute information of each Gaussian point of the video segment to be rendered stored in the rendering buffer, thereby generating a rendered frame corresponding to the video segment to be rendered.

[0153] Specifically, electronic devices can preload the segmented video information of the next time interval in advance according to the playback order. When playing the current video segment in the current time interval, the Gaussian point attribute information of the next video segment to be played can be stored in the rendering buffer in advance. When the playback time enters the corresponding time interval, the existing data can be retrieved directly from the rendering buffer to complete the rendering. This can avoid the stuttering caused by temporarily calculating data when playing the corresponding time period, achieve smooth connection of segmented playback, and further improve the playback experience.

[0154] Alternatively, the electronic device can store the attribute information of each Gaussian point of the corresponding video segment to be rendered into the rendering buffer when the current time interval is reached, and then directly retrieve the data from the buffer to complete the rendering.

[0155] For previously played video segments, the memory and video memory occupied by the corresponding data can be released in a timely manner, avoiding invalid data from continuously occupying hardware resources. This can further reduce the resource consumption of the terminal device while ensuring smooth playback.

[0156] In one embodiment, when the volumetric video information includes hybrid deformation parameters, step S240 can be implemented by the following steps: obtaining the corresponding hybrid deformation parameters and attribute change law information corresponding to each Gaussian point based on the above volumetric video information; rendering the Gaussian points according to the attribute information of each Gaussian point, the above hybrid deformation parameters, and the above digital human model to generate a volumetric video image.

[0157] Specifically, the overall basic shape of the digital human model can be adjusted by mixing deformation parameters, and then combined with the Gaussian point attribute information calculated through attribute change patterns to complete the rendering output of the current frame. Adjusting the overall shape first and then combining it with local Gaussian point details can simultaneously ensure the accuracy of the digital human's overall shape restoration, while preserving the dynamic changes of local details such as expressions and textures, thus balancing the overall and local rendering quality and adapting to playback scenarios of high-detail volumetric videos such as dynamic digital humans.

[0158] The volumetric video playback method provided in this paper stores the dynamic change information of each Gaussian point as a change law parameter, eliminating the need to store the full attribute data of each Gaussian point in every frame. This significantly reduces the storage size of the volumetric video. During playback, the method only needs to quickly calculate the attributes at the current moment based on the change law to complete the rendering. It does not require decompressing the full data before playback, which can significantly reduce the data loading time at the start of playback. At the same time, it reduces the memory and video memory usage during playback, effectively improving the playback smoothness of the volumetric video and making it more suitable for network transmission and terminal playback scenarios.

[0159] In this embodiment, during the covariance calculation and projection process, quaternion rotation can be transformed into a rotation matrix, which is then combined with a scaling matrix to form a rotation-scaling matrix. Two main eigenvectors of the covariance matrix are extracted. Subsequently, the three-dimensional covariance matrix is ​​projected onto a two-dimensional screen space using a model-view projection matrix, thereby obtaining the elliptical shape of the Gaussian points on the screen. The sorting stage employs an efficient GPU sorting algorithm (radix sort) to sort all visible Gaussian points by depth. The rendering stage rasterizes the sorted Gaussian points and draws them onto the rendering target. The aforementioned covariance calculation and projection process refers to the preprocessing steps in Gaussian point rendering. By projecting and transforming the three-dimensional Gaussian point information into a two-dimensional screen space, it provides a foundation for subsequent rasterization rendering. This process can be directly implemented using existing mature Gaussian point rendering calculation logic without requiring additional custom complex calculations. It is compatible with current mainstream GPU rendering pipelines, adapts to common graphics processing hardware, and ensures the execution efficiency of the rendering process.

[0160] For example, such as Figure 5 The diagram illustrates the projection process of a Gaussian point from three-dimensional space to two-dimensional screen space. This projection process is used to calculate the shape and size of the Gaussian point in screen space. Figure 5 As shown, the projection process includes the following steps: The first step is the calculation of the 3D covariance. The system first converts the quaternion rotation of the Gaussian point into a rotation matrix, which is then combined with a scaling matrix to obtain a rotation-scaling matrix M = R × S. This matrix describes the shape and orientation of the Gaussian point in 3D space. From the rotation-scaling matrix, the system extracts two main eigenvectors of the covariance matrix. These vectors define the main orientation and shape of the Gaussian point in 3D space. The formula for calculating the 3D covariance matrix is ​​Σ. 3D =M×M T That is, the product of the rotation-scaling matrix and its transpose.

[0161] The second step is to project the data into a two-dimensional screen space. The system uses a model-view projection matrix to transform the covariance information in three-dimensional space into the screen coordinate system. The projected two-dimensional covariance matrix is ​​Σ. 2D The projected two-dimensional covariance matrix describes the elliptical shape of the Gaussian point on the screen, specifically including the major axis, minor axis, and orientation angle of the ellipse.

[0162] The third step is to generate a screen-space ellipse for rasterization. Based on the two-dimensional covariance matrix obtained from the projection, the position, shape, and orientation of the ellipse on the screen are determined. This ellipse will serve as the basic unit for subsequent rasterization rendering.

[0163] Through the projection process described above, Gaussian points in three-dimensional space are converted into two-dimensional ellipses in screen space, avoiding the construction of complex triangular meshes and improving rendering efficiency.

[0164] This application can also perform parallel rendering of multiple assets, enabling the simultaneous rendering of multiple digital human models. In multi-asset rendering mode, the system creates a merged rendering target, then renders each Gaussian asset separately, outputs the rendering result of each asset to a temporary rendering target, and finally merges the rendering results of each asset into the merged rendering target through a dedicated merge shader.

[0165] This embodiment utilizes the change pattern information stored in the volumetric video information to complete the rendering by simply calculating the Gaussian point attribute corresponding to the current playback moment. It eliminates the need to load the full multi-frame volumetric video data, significantly reducing memory usage and computational overhead during terminal playback. Even on terminal devices with limited performance, volumetric videos can be played smoothly, effectively improving the playback experience.

[0166] For volumetric video information containing hybrid deformation parameters, the hybrid deformation parameters at the corresponding time are called to adjust the overall shape, and the current attributes of the Gaussian points are calculated to supplement details. This can ensure the detail restoration accuracy of the volumetric video image even with low data volume, thus improving the user's viewing experience.

[0167] Example 3 The third embodiment of this application also provides a volumetric video generation apparatus corresponding to the volumetric video generation method embodiment provided in the first embodiment. Since the apparatus embodiment is basically similar to the method embodiment, it is described simply. For details of the relevant technical features and their effects, please refer to the corresponding descriptions of the volumetric video generation method embodiments provided above. The volumetric video generation apparatus provided in this embodiment includes: A video acquisition unit is used to acquire a reference video, wherein the reference video contains the object to be modeled. The model generation unit is used to generate a digital human model corresponding to the object to be modeled based on the reference video, and to determine the attribute information of each Gaussian point in the digital human model in each frame. The pattern determination unit is used to determine the change pattern information of the attribute information of the Gaussian point on the time axis of the reference video based on the attribute information of the Gaussian point in each frame. The video generation unit is used to generate volumetric video information corresponding to the reference video based on the variation pattern information corresponding to each Gaussian point and the digital human model.

[0168] Example 4 The fourth embodiment of this application also provides an embodiment of an electronic device. The following description of the electronic device embodiment is merely illustrative. The electronic device embodiment is as follows: Please refer to Figure 6 Understanding the above electronic devices, Figure 6 This is a schematic diagram of an electronic device. The electronic device provided in this embodiment includes: a processor 1001, a memory 1002, a communication bus 1003, and a communication interface 1004; The memory 1002 is used to store computer instructions for data processing. When the computer instructions are read and executed by the processor 1001, they execute the method described in any one of the first to second embodiments.

[0169] Example 5 The fifth embodiment of this application also provides a computer-readable storage medium for implementing the methods of any one of the first to second embodiments. The embodiments of the computer-readable storage medium provided in this application are described in a relatively simple manner; relevant parts can be found in the corresponding descriptions of the above method embodiments. The embodiments described below are merely illustrative.

[0170] The computer-readable storage medium provided in this embodiment stores computer instructions, which, when executed by a processor, implement the steps described in any one of the first to second embodiments.

[0171] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0172] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0173] 1. Computer-readable media includes both permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, optical disc read-only memory, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined in this application, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0174] 2. Those skilled in the art will understand that embodiments of this application can provide methods, systems, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, optical storage, etc.) containing computer-usable program code.

[0175] 3. This application embodiment may involve the use of user data. In practical applications, user-specific personal data may be used within the scope permitted by applicable laws and regulations of the country in which the application is located (e.g., with the user's explicit consent and effective notification to the user, etc.). Furthermore, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0176] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

Claims

1. A method of generating a volumetric video, characterized in that, The method includes: Obtain a reference video, which contains the object to be modeled; A digital human model corresponding to the object to be modeled is generated based on the reference video, and the attribute information of each Gaussian point in the digital human model in each frame is determined. Based on the attribute information of the Gaussian point in each frame, determine the change pattern information of the attribute information of the Gaussian point on the time axis of the reference video; Based on the variation pattern information corresponding to each Gaussian point and the digital human model, the volumetric video information corresponding to the reference video is generated.

2. The method of claim 1, wherein, The step of determining the change pattern information of the attribute information of the Gaussian point on the time axis of the reference video according to the attribute information of the Gaussian point in each frame includes: Using the attribute information corresponding to the Gaussian point in each frame as sampling points, the initial change pattern information corresponding to the attribute information of the Gaussian point on the time axis is determined. Using the reference video as a reference, the initial change pattern information is adjusted to obtain the change pattern information corresponding to the attribute information of the Gaussian point on the time axis, so that the change pattern information is consistent with the state change of the object to be modeled in the reference video.

3. The method of claim 2, wherein, Using the attribute information corresponding to the Gaussian point in each frame as sampling points, the initial change pattern information corresponding to the attribute information of the Gaussian point on the time axis is determined, including: The discrete initial change trajectory of the attribute information of the Gaussian point on the time axis is determined by using the attribute information corresponding to the Gaussian point in each frame as sampling points; The initial change trajectory is modeled to generate an initial change pattern spline corresponding to the initial change trajectory. The initial change pattern spline is used to represent the initial continuous change pattern of the attribute information of the Gaussian point on the time axis.

4. The method for generating volumetric video according to claim 3, characterized in that, The step of adjusting the initial change pattern information with reference to the reference video to obtain the change pattern information corresponding to the attribute information of the Gaussian point on the time axis includes: Multiple control points are determined on the initial variation spline; Using the reference video as a reference, the control points on the initial variation pattern spline are adjusted to obtain the adjusted variation pattern spline, and the adjusted variation pattern spline is determined as the variation pattern information corresponding to the attribute information of the Gaussian point on the time axis.

5. The method for generating volumetric video according to claim 4, characterized in that, The determination of multiple control points on the initial variation spline includes: Determine the acceleration at each time step on the initial variation spline; Based on the principle that the distribution density of control points is positively correlated with acceleration, multiple control points are determined on the initial variation law spline. Alternatively, multiple control points can be determined on the initial variation pattern spline at equal time intervals.

6. The method for generating volumetric video according to claim 3, characterized in that, The attribute information of the Gaussian point includes a first type of attribute information applicable to Euclidean space interpolation and / or a second type of attribute information applicable to manifold space or tangent space interpolation. The step of modeling the initial change trajectory and generating the initial change pattern spline corresponding to the initial change trajectory includes: For the initial change trajectory corresponding to the first type of attribute information, spline interpolation is performed on the initial change trajectory in Euclidean space to generate the initial change pattern spline corresponding to the first type of attribute information; For the initial change trajectory corresponding to the second type of attribute information, the second type of attribute information is transformed into the tangent space to obtain the corresponding attribute vector. Spline interpolation is performed on the attribute vector in the tangent space to generate the initial change pattern spline corresponding to the second type of attribute information.

7. The method for generating volumetric video according to claim 4, characterized in that, The step of adjusting each control point on the initial variation pattern spline with reference to the reference video to obtain the adjusted variation pattern spline includes: Based on the principle that the intensity of motion of the object to be modeled in the reference video is positively correlated with the distribution density of control points, the distribution position and number of control points on the initial variation pattern spline are adjusted to obtain the preliminary adjusted variation pattern spline. Using the reference video as a reference, the parameter values ​​of each control point on the initially adjusted variation pattern spline are adjusted to obtain the adjusted variation pattern spline.

8. The method for generating volumetric video according to claim 2, characterized in that, Also includes: From each of the Gaussian points, identify the error Gaussian points whose consistency error with the reference video exceeds a preset error threshold; The error Gaussian point is split into Gaussian points to obtain at least one split Gaussian point corresponding to the error Gaussian point, and the initial change law information corresponding to the split Gaussian point is determined.

9. The method for generating volumetric video according to claim 8, characterized in that, The determination of the initial variation information corresponding to the split Gaussian point includes: The variation law information of the error Gaussian point corresponding to the split Gaussian point is determined as the replication variation law information corresponding to the split Gaussian point. By adding a perturbation to the replication change pattern information, the initial change pattern information corresponding to the split Gaussian point is obtained.

10. The method for generating volumetric video according to any one of claims 1 to 9, characterized in that, The step of determining the change pattern information of the attribute information of the Gaussian point on the time axis of the reference video according to the attribute information of the Gaussian point in each frame includes: The reference video is divided into multiple video segments; For each video segment, based on the attribute information corresponding to the Gaussian point in each frame of the video segment, the segmented change pattern information corresponding to the attribute information of the Gaussian point in the time interval corresponding to the video segment is determined, so that the volume video information includes the segmented volume video information corresponding to each video segment.

11. The method for generating volumetric video according to any one of claims 1 to 9, characterized in that, The generation of volumetric video information corresponding to the reference video based on the variation pattern information corresponding to each Gaussian point and the digital human model includes: Determine the hybrid deformation parameters corresponding to the digital human model based on the reference video; Based on the hybrid deformation parameters, the variation law information corresponding to each Gaussian point, and the digital human model, the volumetric video information corresponding to the reference video is generated.

12. A method for generating volumetric video, characterized in that, The method includes: Obtain volumetric video information, wherein the volumetric video information is generated by the volumetric video generation method according to any one of claims 1 to 11; The digital human model and the variation pattern information corresponding to each Gaussian point of the digital human model are obtained from the volumetric video information. The attribute information of the Gaussian point is determined based on the change pattern information corresponding to the Gaussian point. The Gaussian points are rendered based on their attribute information to generate volumetric video frames.

13. The method for generating volumetric video according to claim 12, characterized in that, When the volumetric video information includes volumetric video information corresponding to multiple video segments, the step of rendering the Gaussian points based on the attribute information of each Gaussian point to generate a volumetric video frame includes: For each non-last video segment of the volumetric video, based on the attribute information of each Gaussian point of the non-last video segment, render each frame of the non-last video segment except for the last frame to generate the volumetric video frame corresponding to the non-last video segment. For the last video segment of the volumetric video, each frame of the last video segment is rendered based on the attribute information of each Gaussian point in the last video segment to generate the corresponding volumetric video frame.

14. The method for generating volumetric video according to claim 12, characterized in that, The step of rendering the Gaussian points based on their attribute information to generate volumetric video frames includes: Based on the content to be displayed at the current moment, determine the Gaussian point that is currently active; Render the active Gaussian points based on their attribute information to generate the volumetric video frame for the current moment.

15. The method for generating volumetric video according to claim 14, characterized in that, The step of determining the active Gaussian point based on the content to be displayed at the current moment includes: Determine the field of view of the camera at the current moment; Based on the position information of each Gaussian point, Gaussian points within the range of the view frustum are selected from the Gaussian points of the digital human model, and the Gaussian points currently in an active state are determined based on the Gaussian points within the range.

16. The method for generating volumetric video according to claim 15, characterized in that, Determining the active Gaussian point at the current moment based on the Gaussian points within the range includes: Remove Gaussian points that meet the preset non-rendering conditions from the Gaussian points within the range to obtain Gaussian points that are currently active. The non-rendering conditions include at least one of the following: located in the opposite direction of the camera lens, at a distance greater than a preset distance from the camera, and at a transparency higher than a preset transparency.

17. The method for generating volumetric video according to claim 12, characterized in that, When the volumetric video information includes segmented volumetric video information corresponding to multiple video segments, the step of rendering the Gaussian points based on the attribute information of each Gaussian point to generate a volumetric video frame includes: The video segment to be rendered is determined sequentially according to the time intervals corresponding to the multiple video segments; For the current time interval, the attribute information of each Gaussian point of the corresponding video segment to be rendered is stored in the rendering buffer. Based on the attribute information of each Gaussian point of the video segment to be rendered stored in the rendering buffer, each Gaussian point in the video segment to be rendered is rendered to generate the rendered screen corresponding to the video segment to be rendered.

18. A method for playing a video of a certain size, characterized in that, The method includes: Volumetric video information is generated based on the volumetric video generation method according to any one of claims 1 to 11; Determine the current camera pose for rendering; Based on the current camera pose, determine the Gaussian point in the volumetric video where the digital human model is currently active. Based on the volumetric video information, the attribute information of the Gaussian points in the active state is determined, and the Gaussian points in the active state are rendered according to the attribute information to generate and play the volumetric video image from the current perspective.

19. An electronic device, characterized in that, include: Processor, memory, and computer program instructions stored in said memory and executable on the processor; When the processor executes the computer program instructions, it implements the method as described in any one of claims 1-18.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-18.

Images