Method, device and product for realizing four-dimensional scene interaction playing of browser

Abstract

The application provides a method, device and product for realizing four-dimensional scene interactive playing of a browser, which organizes dynamic display content into static base resources, dynamic base resources and Gaussian cell deformation attribute label maps corresponding to each video frame, and combines the label maps into a Gaussian rendering object after loading at the browser end; during playing, a time axis determines a current target frame, only the dynamic Gaussian cells are written back according to frame attributes, and the static Gaussian cells remain constant and are not repeatedly updated; a user can also rotate, zoom, translate and switch a view angle through a virtual camera. Thus, the dynamic process can be continuously played like a video, and the view angle can be freely viewed like a three-dimensional model, and the complete three-dimensional scene is avoided to be loaded frame by frame, the loading, memory and rendering pressure of the browser end is reduced, and the application is suitable for dynamic interactive display scenes such as commodity display, handicraft process playback, teaching demonstration and live playback.

Description

Technical Field

[0001] This application relates to the field of interactive graphics immersive technology, specifically to a method, computer device, and program product for realizing interactive playback of four-dimensional scenes in a browser. Background Technology

[0002] With the development of applications such as virtual displays, live replays, product displays, and educational demonstrations, users' viewing needs for video content are no longer limited to two-dimensional images from a fixed shooting perspective. Instead, they hope to be able to interactively view the dynamic display process of the target object on their terminal devices.

[0003] In existing technologies, a common approach is to record the display process of a target object using single-channel or multi-channel video. This method can present the object's changes over time, but the playback content is usually still a pre-captured sequence of two-dimensional images, and the viewable content is limited by the actual camera's shooting angle. When a user needs to observe the target object from different angles at a particular playback moment, existing video playback methods struggle to generate a display that corresponds to the user's interactive perspective.

[0004] Another approach involves constructing the target object or scene as 3D content and displaying it interactively on the terminal device. While this method offers some spatial perspective interaction capabilities, for display processes involving dynamic changes, directly loading and playing the corresponding dynamic 3D content in the browser's runtime environment typically faces issues such as large data volume, long loading times, high memory consumption, and heavy pressure on graphics processor resources, making it difficult to meet the real-time playback requirements of the browser.

[0005] Especially during dynamic displays, the state of the target object needs to continuously change over time, while users may perform interactive operations such as rotation, scaling, panning, or perspective switching at any point in the playback. Therefore, the browser not only needs to restore the corresponding state of the target object according to the playback time, but also needs to generate a corresponding display screen based on the user's current perspective. Existing methods struggle to balance the continuous playback efficiency of dynamic content with the ability to display interactive spatial perspectives within the browser runtime environment.

[0006] Therefore, the technical problem that needs to be solved is how to reduce the loading and playback burden of dynamically displayed content in the browser runtime environment, and enable the dynamic display process of the target object to restore the corresponding state according to the playback time and generate the corresponding view display screen in response to the user's perspective interaction. Summary of the Invention

[0007] One objective of this application is to solve the technical problem of how to reduce the loading and playback burden of dynamically displayed content in the browser runtime environment, and enable the dynamic display process of the target object to restore the corresponding state according to the playback time and generate the display screen of the corresponding view in response to the user's perspective interaction operation.

[0008] According to one aspect of the embodiments of this application, a method for implementing interactive playback of a four-dimensional scene in a browser is disclosed. The method is applied to the browser runtime environment of a terminal device, and the method includes:

[0009] Load interactive video resources, which include static base resources, dynamic base resources, and Gaussian deformation attribute label maps corresponding to each video frame. The interactive video resources are parsed to recover the static Gaussian elements represented by the static base resources, the dynamic Gaussian elements in the baseline state represented by the dynamic base resources, and the deformation data corresponding to each video frame represented by the Gaussian element deformation attribute label map. The static Gaussian primitives and the dynamic Gaussian primitives in the baseline state are merged into a Gaussian rendering object, and the dynamic index information of the dynamic Gaussian primitives in the Gaussian rendering object is determined. In response to the player's timeline pointing to the target video frame, the target deformation data corresponding to the target video frame is obtained from the deformation data corresponding to each video frame; Based on the target deformation data and the dynamic index information, while keeping the static Gaussian primitives in the Gaussian rendering object from being written back to their attributes frame by frame, the dynamic Gaussian primitives indicated by the dynamic index information are written back to their attributes to obtain the rendering data corresponding to the target video frame. Based on the rendering data and the virtual camera parameters from the current user's perspective, a four-dimensional scene interactive display screen is drawn in the browser runtime environment from the corresponding perspective of the target video frame; The player timeline is used to determine the time dimension of the four-dimensional scene interactive display screen, and the virtual camera parameters are used to determine the spatial observation angle of the four-dimensional scene interactive display screen. The virtual camera parameters change with the user's perspective interaction operation so that the four-dimensional scene interactive display screen switches to the corresponding perspective.

[0010] According to one aspect of the embodiments of this application, a computer device is disclosed, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method as described in any of the preceding claims.

[0011] According to one aspect of the embodiments of this application, a computer program product is disclosed, including a computer program that, when executed by a processor, implements the steps of the method as described in any of the preceding claims.

[0012] This application provides a solution for playing interactive four-dimensional scene video content in a browser. Its core is to organize the dynamically displayed content into static base resources, dynamic base resources, and Gaussian element deformation attribute label maps corresponding to each video frame. After loading on the browser side, these are merged into a single Gaussian rendering object. During playback, the timeline determines the current target frame, and only dynamic Gaussian elements are written back to their frame-by-frame attributes, while static Gaussian elements remain resident and are not repeatedly updated. Users can also rotate, zoom, pan, and switch viewpoints using a virtual camera. Thus, it can play dynamic processes continuously like a video, and can also be viewed freely from a 3D perspective like a 3D model, while avoiding loading the complete 3D scene frame by frame, reducing loading, memory, and rendering pressure on the browser side. It is suitable for dynamic interactive display scenarios such as product displays, replays of handicraft processes, teaching demonstrations, and live stream replays.

[0013] Specifically, this application embodiment organizes interactive video resources into static base resources, dynamic base resources, and Gaussian element deformation attribute label maps corresponding to each video frame. This eliminates the need for the browser to load the complete 3D scene frame by frame. Instead, it first restores the static Gaussian elements and the dynamic Gaussian elements in the baseline state, and then merges them into a single Gaussian rendering object. Since the static Gaussian elements are not repeatedly updated with the target video frame during playback, and the dynamic Gaussian elements are restored frame by frame through the Gaussian element deformation attribute label maps corresponding to each video frame, it can reduce redundant data loading, object reconstruction, and resource uploading, thereby reducing the browser's memory usage and rendering pressure.

[0014] Furthermore, in this embodiment, dynamic Gaussian primitives in the Gaussian rendering object are located using dynamic index information. When the player's timeline points to the target video frame, only the dynamic Gaussian primitives indicated by the dynamic index information are written back in terms of attributes, while static Gaussian primitives are left unwritten back in terms of attributes frame by frame. Thus, the range of data that the browser needs to process in each frame is limited to the dynamic portion, preventing static backgrounds or static objects from being repeatedly updated with each frame, thereby improving the response speed and playback stability of dynamic playback.

[0015] Meanwhile, this embodiment determines the target video frame through the player's timeline and the current user's perspective through virtual camera parameters, enabling the restoration of the target object's dynamic state and the user's spatial perspective interaction to work together. Thus, users can continuously view the dynamic display process of the target object as if watching a video, or observe the target object from different angles at any target video frame through operations such as rotation, scaling, panning, or perspective switching, thereby overcoming the limitation that ordinary two-dimensional videos can only be viewed from a fixed shooting perspective.

[0016] Therefore, the embodiments of this application can balance dynamic process playback, free-viewpoint interaction and low resource consumption in the browser runtime environment, making dynamic display content more suitable for lightweight and interactive four-dimensional scene playback in scenarios such as product display, handicraft process playback, teaching demonstration, and live broadcast playback.

[0017] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0018] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0019] The above and other objectives, features and advantages of this application will become more apparent from a detailed description of exemplary embodiments thereof with reference to the accompanying drawings.

[0020] Figure 1 This is a flowchart illustrating a method for implementing interactive playback of a four-dimensional scene in a browser, according to an exemplary embodiment.

[0021] Figure 2 It is based on Figure 1 The flowchart shown in the corresponding embodiment describes the steps of resolving interactive video resources to recover static Gaussian elements represented by static base resources, dynamic Gaussian elements in a baseline state represented by dynamic base resources, and deformation data corresponding to each video frame represented by a Gaussian element deformation attribute label map.

[0022] Figure 3 It is based on Figure 2 The flowchart described in the corresponding embodiment describes the steps of parsing interactive video resources to obtain a two-dimensional attribute map corresponding to a static base resource, a two-dimensional attribute map corresponding to a dynamic base resource, and a Gaussian deformation attribute label map corresponding to each video frame.

[0023] Figure 4 This is a flowchart describing a method for recovering static and dynamic Gaussian primitives and their quantization attribute values ​​in a baseline state by dequantizing the quantization parameters indicated by metadata in an interactive video resource, according to a specific implementation of an exemplary embodiment.

[0024] Figure 5 It is based on Figure 2 The flowchart shown in the corresponding embodiment describes the steps of decoding the Gaussian deformation attribute label map corresponding to each video frame as a data map carrying Gaussian attributes, and dequantizing the decoded quantization label to obtain the deformation data corresponding to each video frame.

[0025] Figure 6 It is based on Figure 1 The corresponding embodiment shows a flowchart describing the steps of merging static Gaussian primitives and dynamic Gaussian primitives in a baseline state into a Gaussian rendering object, and determining the dynamic index information of the dynamic Gaussian primitives in the Gaussian rendering object.

[0026] Figure 7 It is based on Figure 1 The corresponding embodiment shows a method flowchart describing the steps of obtaining the target deformation data corresponding to the target video frame from the deformation data corresponding to each video frame in response to the player's timeline pointing to the target video frame.

[0027] Figure 8 It is based on Figure 1 The corresponding embodiment shows a flowchart describing the steps of drawing a four-dimensional scene interactive display screen of the corresponding viewpoint under the target video frame in the browser runtime environment based on rendering data and virtual camera parameters under the current user's viewpoint.

[0028] Figure 9 It is a display screen shown from a specific perspective based on a particular example.

[0029] Figure 10 From Figure 9 The display screen from the shown observation perspective is switched to the display screen from another observation perspective. Detailed Implementation

[0030] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided to make the description of this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The drawings are merely illustrative of this application and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0031] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more exemplary embodiments. Numerous specific details are provided in the following description to give a full understanding of exemplary embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced with one or more of the specific details omitted, or other methods, components, steps, etc., can be employed. In other instances, well-known structures, methods, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0032] Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0033] See Figure 1 , Figure 1 This is a flowchart illustrating a method for implementing interactive playback of a four-dimensional scene in a browser, according to an exemplary embodiment.

[0034] This application provides a method for implementing interactive playback of four-dimensional scenes in a browser, applied to the browser runtime environment of a terminal device. The method includes: Step S110: Load interactive video resources, which include static base resources, dynamic base resources, and Gaussian deformation attribute label maps corresponding to each video frame. Step S120: parse the interactive video resources and recover the static Gaussian elements represented by the static basis, the dynamic Gaussian elements in the reference state represented by the dynamic basis, and the deformation data corresponding to each video frame represented by the Gaussian element deformation attribute label map. Step S130: Merge the static Gaussian primitives and the dynamic Gaussian primitives in the baseline state into a Gaussian rendering object, and determine the dynamic index information of the dynamic Gaussian primitives in the Gaussian rendering object. Step S140: In response to the player's timeline pointing to the target video frame, obtain the target deformation data corresponding to the target video frame from the deformation data corresponding to each video frame; Step S150: Based on the target deformation data and the dynamic index information, while keeping the static Gaussian primitives in the Gaussian rendering object from being written back to their attributes frame by frame, write back the attributes of the dynamic Gaussian primitives indicated by the dynamic index information to obtain the rendering data corresponding to the target video frame. Step S160: Based on the rendering data and the virtual camera parameters under the current user's perspective, draw the four-dimensional scene interactive display screen of the corresponding perspective under the target video frame in the browser runtime environment; The player timeline is used to determine the time dimension of the four-dimensional scene interactive display screen, and the virtual camera parameters are used to determine the spatial observation perspective of the four-dimensional scene interactive display screen. The virtual camera parameters change with the user's perspective interaction operation so that the four-dimensional scene interactive display screen switches to the corresponding perspective.

[0035] These steps are explained in detail below.

[0036] First, it should be noted that in this embodiment, the browser runtime environment refers to the web page execution environment provided by the browser in the terminal device. For example, this browser runtime environment includes at least one of the following: a JavaScript execution environment, a WebAssembly execution environment, an HTML page environment, a Canvas environment, a WebGL graphics rendering environment, a WebGPU graphics rendering environment, a browser image decoder, a browser video decoder, a network resource loading module, a caching module, and a data upload channel between the browser and the graphics processor. For ordinary video playback, the browser runtime environment typically only needs to decode the two-dimensional video stream and display the decoded pixel images on the page according to the video frame order; the video frames themselves are the display content during this type of playback, and it is usually not necessary to restore the attributes of the three-dimensional spatial primitives at each playback moment, nor is it necessary to convert the time frame data into a three-dimensional state that can be rendered by a virtual camera from different perspectives.

[0037] For dynamic 3D or 4D scene content, the browser faces a more complex issue than simply displaying 2D pixel frames. Existing conventional methods include loading or reconstructing the corresponding 3D content for each video frame. However, this involves downloading, parsing, creating objects, and uploading graphics processor resources for each frame, leading to high network traffic, memory consumption, GPU memory pressure, and unstable frame rates. Another approach is to display the dynamic content as a regular video frame, but this only displays pre-generated 2D images and cannot generate a viewpoint based on the user's current virtual camera perspective within the target video frame. A third approach is to apply a global pose transformation to the 3D objects or temporarily overlay the transformation in the shader. However, for scenes composed of numerous Gaussian elements with local dynamic changes, this method struggles to accurately recover the center position, scale, rotation, and other attributes of each dynamic Gaussian element in the target video frame, and it also fails to ensure that the sorting center, bounding box, graphics processor buffer, or texture resource state are consistent with the current frame's dynamic state.

[0038] Therefore, the difficulty in playing dynamic 3D or 4D scene content in a browser runtime environment lies not only in resource loading and data parsing itself, but also in how to organize the time-varying dynamic state restoration, rendering object internal attribute update, rendering auxiliary structure synchronization, and user-perspective interactive rendering into a real-time executable playback process under the browser's limited script execution, memory, video memory, and graphics processor resources. This application's embodiment solves the problem of inefficient playback and interactive display of dynamic 4D scenes in a browser runtime environment through subsequent steps such as interactive video resource parsing, merging static and dynamic Gaussian primitives, dynamic index information positioning, dynamic Gaussian primitive attribute writing back, and virtual camera rendering.

[0039] In step S110, the browser runtime environment of the terminal device displays the interactive video of the acquired four-dimensional dynamic scene by loading interactive video resources. The interactive video resources are resource-based data generated from multiple acquired videos of the target object after preprocessing, Gaussian training, and discrete quantization compression, suitable for loading and playback in the terminal device's browser runtime environment. For example, the target object can be handicraft materials, handicraft works, commodities, teaching objects, tools, industrial parts, live demonstration objects, or other objects that need to be displayed in a dynamic process. Taking a handicraft demonstration scene as an example, multiple cameras arranged around the handicraft operation area simultaneously acquire multiple video feeds of the hands, tools, materials, and the forming process of the finished product. Further, camera parameter estimation, region mask generation, scene initialization training, static / dynamic separation training, and dynamic deformation modeling are performed on the multiple video feeds. The trained static Gaussian units, dynamic Gaussian units, and frame-by-frame deformation data are then processed for resource-based and compressed processing, thereby generating interactive video resources.

[0040] Specifically, firstly, multiple video feeds of the target object are acquired, and then synchronized, camera parameter estimated, image distortion corrected, and region mask generated for each video feed to obtain camera parameters for each feed and region masks corresponding to video frames and camera viewpoints. Subsequently, scene initialization training is performed based on the multi-view images from the multiple video feeds and the camera parameters for each feed to obtain an initial Gaussian representation of the complete scene containing the target object. Then, based on the initial Gaussian representation, the camera parameters for each feed, and the region masks, motion-static separation training is performed, dividing the Gaussian elements in the initial Gaussian representation into a set of static Gaussian elements and a set of dynamic Gaussian elements. Further, deformation field training is performed on the set of dynamic Gaussian elements to obtain a deformation field shared by the set of dynamic Gaussian elements. This deformation field is used to determine the attribute changes of dynamic Gaussian elements in the corresponding video frames based on their positions and video frame times.

[0041] After completing the three-stage Gaussian training, the static Gaussian primitive set is converted into static basis resources, and the attributes of the dynamic Gaussian primitive set under the baseline state are converted into dynamic basis resources. A dynamic deformation stream is then generated based on the dynamic basis resources and the deformation field. The dynamic deformation stream characterizes the attribute changes of dynamic Gaussian primitives relative to the dynamic basis resources in each video frame. Furthermore, the deformation data corresponding to each video frame in the dynamic deformation stream is quantized, visualized, and encoded and compressed to generate Gaussian primitive deformation attribute label maps for each video frame. The static basis resources, dynamic basis resources, Gaussian primitive deformation attribute label maps, and metadata are then organized into interactive video resources. Thus, the dynamic basis resources in the interactive video resources and the Gaussian primitive deformation attribute label maps corresponding to each video frame maintain a correspondence at the dynamic Gaussian primitive index level, enabling the browser runtime environment of the terminal device to recover the dynamic Gaussian primitive attributes of the target video frame based on the player's timeline.

[0042] Furthermore, the three-stage Gaussian training employs multi-GPU parallel training. Specifically, different camera perspectives, video frames, training batches, or subsets of Gaussian elements are allocated to multiple GPUs, and at least one of the following is synchronized during training: initial Gaussian representation attributes, dynamic response values ​​from static / dynamic separation training, deformation field parameters, or dynamic Gaussian element attributes. Multi-GPU parallel training improves training efficiency in multi-video scenarios, enabling the generation of static base resources, dynamic base resources, and Gaussian element deformation attribute label maps for each video frame within a short time for the multi-view dynamic display content of the target object. It should be noted that multi-GPU parallel training, as one method of generating interactive video resources, does not restrict the specific playback process of loading, parsing, attribute writing back, and rendering the interactive video resources on the terminal device within the browser runtime environment.

[0043] In other words, unlike ordinary two-dimensional video resources, interactive video resources are not merely used to decode and display pre-captured two-dimensional video frames in a browser. Instead, they enable the browser's runtime environment on the terminal device to reconstruct the static Gaussian units during the display of the target object, the dynamic Gaussian units in the baseline state, and the deformation data corresponding to the video frames. In other words, the acquisition and training phases reconstruct multiple acquired video streams into four-dimensional scene resources that can be played on the browser. The browser then parses, reconstructs, and renders this interactive video resource in real time based on the player's timeline and the user's interactive perspective.

[0044] In an exemplary embodiment, the training phase yields a set of static Gaussian primitives, a set of dynamic Gaussian primitives under a reference state, and a deformation field. The static Gaussian primitive set characterizes relatively stable static regions during the display process; the dynamic Gaussian primitive set under the reference state characterizes the basic geometry and appearance of dynamic objects at a reference time or in a canonical state; the deformation field determines the attribute changes of the dynamic Gaussian primitives in the corresponding video frame based on their spatial location and time. For any given time... t The dynamic Gaussian primitive state at a given moment is obtained from the dynamic Gaussian primitive set and deformation field under the baseline state. This dynamic Gaussian primitive state, together with the static Gaussian primitive set, forms the Gaussian scene at that moment, thus providing users with Gaussian interaction. Given the user's viewpoint or virtual camera parameters, the corresponding viewpoint display can be rendered based on the Gaussian scene at that moment.

[0045] It should be noted that during the training or export phase, a dynamic deformation stream is generated based on dynamic Gaussian elements and a deformation field. The dynamic deformation stream is used to characterize the attribute changes of dynamic Gaussian elements across different video frames. Furthermore, in the process of generating browser-loadable interactive video resources, the deformation data corresponding to each video frame in the dynamic deformation stream is quantized, visualized, and encoded and compressed to generate Gaussian deformation attribute label maps corresponding to each video frame. Therefore, correspondingly, on the browser playback side, the interactive video resources include static base resources, dynamic base resources, Gaussian deformation attribute label maps corresponding to each video frame, and metadata. The Gaussian deformation attribute label map is a playback-side data map formed after quantization and compression of the dynamic deformation stream; it is not a video frame used for direct display.

[0046] In the browser runtime environment, as the interactive video resources are loaded in step S110, the dynamic playback and Gaussian interaction of the four-dimensional scene are initiated.

[0047] To further clarify, static base resources are used to carry the resource representation of static Gaussian elements. Static Gaussian elements correspond to relatively stable backgrounds, fixed props, workbenches, fixed display stands, or other long-term immobile areas during the dynamic display of the target object. Dynamic base resources are used to carry the resource representation of dynamic Gaussian elements in their baseline state. Dynamic Gaussian elements correspond to objects that undergo movement, posture changes, deformation changes, or position changes during the display process, such as hands, tools, materials, product parts, manipulated objects, or other dynamic areas. The Gaussian element deformation attribute label map is used to carry the attribute change information of dynamic Gaussian elements in the corresponding video frame, and is obtained by quantizing and image processing the deformation data of the corresponding video frame in the dynamic deformation stream.

[0048] In one exemplary embodiment, the static base resource includes a two-dimensional attribute map, texture map, Gaussian resource file, or binary attribute file corresponding to static Gaussian primitives; the dynamic base resource includes a two-dimensional attribute map, texture map, Gaussian resource file, or binary attribute file corresponding to dynamic Gaussian primitives in the baseline state. The two-dimensional attribute map includes imaged data used to carry attributes such as center position, scale, rotation, transparency, color, or spherical harmonic coefficients.

[0049] For example, static and dynamic base resources include low-bitmaps, high-bitmaps, scale maps, rotation maps, low-frequency color maps, and optional high-order color maps. The above resource formats are merely examples; any resource that can be requested, cached, parsed, and restored as Gaussian meta-attributes by the browser runtime environment can be used as a static or dynamic base resource in the embodiments of this application.

[0050] This will be illustrated with an example. Attributes in static and dynamic base resources are organized in a two-dimensional atlas. One-dimensional Gaussian element attribute sequences are mapped to a two-dimensional attribute graph, where each Gaussian element corresponds to a pixel location or attribute unit in the two-dimensional attribute graph; unused positions are padded with default values. The width, height, attribute channel arrangement, attribute file name, and quantization parameters of the two-dimensional attribute graph are recorded in metadata. To improve image compression efficiency and browser-side access locality, the Gaussian element arrangement in the two-dimensional attribute graph uses a spatial ordering method, such as Morton order, Z-order, PLAS order, or other spatially localized ordering methods.

[0051] For dynamic Gaussian elements, the order of Gaussian elements in the dynamic base resource should be consistent with the order of Gaussian elements in the Gaussian deformation attribute label map corresponding to each video frame. This ensures that the deformation data recovered from the Gaussian deformation attribute label map under the target video frame can be applied to the corresponding dynamic Gaussian elements in the dynamic base resource using the same index, preventing deformation data from being incorrectly applied to different dynamic Gaussian elements.

[0052] The Gaussian deformation attribute label map corresponding to each video frame can be stored in the form of WebP images, PNG images, binary frames, H.265 / HEVC video frames, WebCodecs-decodeable video frames, or other video attribute streams. For dynamic display content with a large number of consecutive frames, video encoding is used to reduce resource size by leveraging temporal correlation. For scenarios with high requirements for initial screen response or random access, frame-by-frame images or segmentation can be used for on-demand loading. The browser decodes the data using image decoders, video decoders, WebAssembly decoders, or other decoding modules, but the decoding result is not used as the final display image; instead, it serves as the quantized label data for subsequent recovery of dynamic Gaussian deformation data.

[0053] Interactive video resources also include metadata. Metadata describes the indexing relationships and recovery parameters between static base resources, dynamic base resources, and Gaussian deformable attribute label maps. For example, metadata may include at least one of the following: resource version, resource address, file identifier, frame rate, number of frames, correspondence between video frames and Gaussian deformable attribute label maps, number of static Gaussian elements, number of dynamic Gaussian elements, 2D attribute map size, atlas arrangement, attribute channel arrangement, quantization range, quantization bits, attribute recovery method, dynamic representation method, dynamic base reference relationship, quaternion encoding method, spherical harmonic order, sorting method, dynamic Gaussian element sorting consistency identifier, validity filtering information, and validation fields.

[0054] When loading interactive video resources, the terminal device verifies whether the number of dynamic Gaussian elements is consistent with the number of dynamic Gaussian elements in the dynamic base resource, verifies whether the number of frames is greater than zero, confirms whether the frame-by-frame resource or video stream resource declared in the metadata is available, and determines whether to use frame-by-frame image decoding, binary frame decoding or video stream decoding method according to the resource type.

[0055] In step S120, the loaded interactive video resources are parsed to recover the static Gaussian elements represented by the static basis, the dynamic Gaussian elements in the reference state represented by the dynamic basis, and the deformation data corresponding to each video frame represented by the Gaussian element deformation attribute label map.

[0056] After loading interactive video resources on the browser side, the terminal device first reads the metadata in the interactive video resources. This metadata is used to locate the two-dimensional attribute map corresponding to the static base resource, the two-dimensional attribute map corresponding to the dynamic base resource, and the Gaussian deformation attribute label map corresponding to each video frame within the interactive video resources. In addition, the metadata is also used to establish the correspondence between video frames and Gaussian deformation attribute label maps, so that when the player's timeline points to a target video frame, the browser can determine which Gaussian deformation attribute label map should be requested, read, or decoded.

[0057] For static base resources, the terminal device reads the quantized attribute values ​​of static Gaussian primitives from the corresponding 2D attribute map, texture map, Gaussian resource file, or binary attribute resource, and performs inverse quantization in conjunction with the quantization parameters indicated in the metadata to recover the attributes of the static Gaussian primitives. For dynamic base resources, the terminal device reads the quantized attribute values ​​of dynamic Gaussian primitives in the baseline state from the corresponding 2D attribute map, texture map, Gaussian resource file, or binary attribute resource, and performs inverse quantization in conjunction with the quantization parameters indicated in the metadata to recover the attributes of the dynamic Gaussian primitives in the baseline state. These attributes include at least one of the following: center position (the center position is the spatial position of the corresponding Gaussian primitive), scale, rotation, transparency, color, or spherical harmonic coefficients. The center position, scale, and rotation are used as key attributes for recovering the geometric state of the Gaussian primitives during dynamic playback.

[0058] Quantization parameters include at least one of attribute type, quantization bit depth, quantization range, and attribute recovery method. Attribute recovery methods include linear inverse quantization, inverse signed logarithmic transform, scale codebook lookup, quaternion recovery, rotation vector recovery, color codebook lookup, or other attribute recovery methods. For example, for the center position, the low-bit and high-bit labels are first read from the low-bit and high-bit position maps, combined to obtain a 16-bit quantization label, then inverse quantization is performed according to the quantization range, and a signed exponential inverse transform can be further performed to recover the center position. For scale, the scale label is read from the scale map, and the scale or logarithmic scale is recovered according to the scale quantization range or scale codebook. For rotation, the rotation label is read from the rotation map, and a quaternion or rotation vector is recovered.

[0059] For the Gaussian deformation attribute label map corresponding to each video frame, the terminal device decodes it as a data map carrying Gaussian attributes and dequantizes the decoded quantization tags to obtain the deformation data corresponding to each video frame. Specifically, the terminal device reads the pixel values ​​in the Gaussian deformation attribute label map as quantization tags, recovers the attribute changes of dynamic Gaussian elements in the corresponding video frame based on the quantization tags and the quantization parameters indicated in the metadata, and uses the recovered attribute changes as the deformation data of the corresponding video frame.

[0060] In a specific implementation of an exemplary embodiment, the Gaussian deformation attribute label map is organized using an atlas arrangement. For example, the low-position label, high-position label, scale label, and rotation label can be stored in different regions of the same data map, or they can be split into multiple data maps and stored separately. The browser reads the quantization labels corresponding to different attributes from the decoded data map based on the atlas size, tile layout, channel arrangement, and quantization range recorded in the metadata. Because the pixel values ​​in the Gaussian deformation attribute label map are used to recover the Gaussian attribute labels, rather than being directly output as display colors, they have different data meanings than ordinary two-dimensional video frames.

[0061] In addition, if the Gaussian metamorphic attribute label map uses WebP or frame-by-frame binary resources, the terminal device reads the low-bit map, high-bit map, scale map, and rotation vector map, and obtains pixel data or byte data through a browser image decoder, Canvas, ImageBitmap, or binary reading method. If the Gaussian metamorphic attribute label map uses H.265 / HEVC video stream format, the terminal device can decode the video stream into raw RGB data, raw gray data, or other raw byte data through WebCodecs, WebAssembly decoders, or other video decoding modules. Compared to using the decoding result as the display pixel output in ordinary video playback, this embodiment uses the decoding result as Gaussian metamorphic attribute labels for further interpretation and dequantization, thereby avoiding the impact of color space conversion, chroma sampling, or range mapping in ordinary display links on attribute recovery accuracy.

[0062] In a specific implementation of an exemplary embodiment, the deformation data corresponding to each video frame is represented differentially relative to a dynamic basis. Let the center position, logarithmic scale, and rotation of the dynamic Gaussian element in the reference state be respectively... , and Target video frame t The corresponding deformation data includes changes in the center position. Logarithmic scale change and rotational changes Then the target video frame t The dynamic Gaussian meta-attribute is represented as follows:

[0063]

[0064]

[0065] in, Indicates the dynamic Gaussian elements in the target video frame tThe center position below; Indicates the dynamic Gaussian elements in the target video frame t Logarithmic scale below Indicates the dynamic Gaussian elements in the target video frame t Rotate, This represents quaternion multiplication. In this way, interactive video resources do not need to save complete dynamic Gaussian primitive attributes frame by frame; instead, they can save attribute changes relative to the dynamic basis, thereby reducing the amount of data per frame.

[0066] In another exemplary implementation, the Gaussian element deformation attribute label map can also carry the absolute attribute labels of dynamic Gaussian elements in the corresponding video frame. This method can be used for debugging, fault tolerance, or applications with high precision requirements, but the data volume is usually larger than that of differential representation. Regardless of whether differential or absolute representation is used, as long as the browser can recover the attributes or attribute changes of dynamic Gaussian elements in the target video frame based on the Gaussian element deformation attribute label map, it can be applied to the embodiments of this application.

[0067] As the static Gaussian primitives, the dynamic Gaussian primitives in the reference state, and the deformation data corresponding to each video frame are recovered in step S120, the static Gaussian primitives and the dynamic Gaussian primitives in the reference state will be merged into a Gaussian rendering object in step S130, and the dynamic index information of the dynamic Gaussian primitives in the Gaussian rendering object will be determined.

[0068] A Gaussian render object refers to a collection of Gaussian primitives that can be rendered by the browser-side graphics rendering pipeline. This Gaussian render object can be a GSplat object, a GSplatData object, a GSplatResource object, or other data objects used to represent and render Gaussian primitives in WebGL, WebGPU, PlayCanvas, or other browser graphics rendering frameworks. A Gaussian render object includes an attribute storage area that stores the center position, scale, rotation, transparency, color, or other attributes of each Gaussian primitive.

[0069] In one exemplary embodiment, the terminal device writes static Gaussian primitives and dynamic Gaussian primitives in a baseline state into the attribute storage area of ​​the same Gaussian rendering object, enabling the static and dynamic Gaussian primitives to share the same sorter, material, rendering pipeline, or graphics processor resources. Compared to creating static and dynamic objects separately, this approach helps reduce browser-side object management overhead and allows the dynamic playback process to be completed within the same Gaussian rendering object.

[0070] Furthermore, static Gaussian primitives are arranged in the preorder interval of the attribute storage area of ​​the Gaussian rendering object, while dynamic Gaussian primitives in the baseline state are arranged in the postorder interval of the attribute storage area, following the preorder interval. The terminal device determines the dynamic starting index of the postorder interval based on the number of static Gaussian primitives corresponding to the preorder interval; it also determines the dynamic quantity based on the number of dynamic Gaussian primitives corresponding to the postorder interval; and uses the dynamic starting index and dynamic quantity as dynamic index information. This dynamic index information is used to locate the attribute storage position corresponding to the dynamic Gaussian primitives in the Gaussian rendering object during attribute write-back.

[0071] It should be noted that dynamic indexing information is not limited to dynamic starting index and dynamic quantity. Dynamic indexing information can also include dynamic Gaussian meta-index tables, dynamic Gaussian meta-identifier mappings, attribute write offset tables, or other data structures that can locate the storage location of dynamic Gaussian meta-attributes. Using a continuous interval representation with a dynamic starting index and dynamic quantity is a simple and efficient implementation method suitable for browser-side access.

[0072] In step S130, the static and dynamic base resources are parsed and restored to static Gaussian primitives and dynamic Gaussian primitives in the baseline state on the browser side, and further merged into a resident Gaussian rendering object. During playback, the terminal device does not need to recreate the complete Gaussian rendering object for each video frame, nor does it need to repeatedly load or write back the static Gaussian primitives. The dynamic Gaussian primitives are stably located through dynamic index information, thus providing a basis for writing back local attributes in subsequent target video frames.

[0073] In step S140, in response to the player's timeline pointing to the target video frame, the target deformation data corresponding to the target video frame is obtained from the deformation data corresponding to each video frame.

[0074] The player's timeline is used to represent the current playback progress, user-dragped progress, pause position, or playback control status. The target video frame is determined by the current playback position on the player's timeline, or can be determined by the user dragging the timeline, clicking the progress bar, jumping frame by frame, or automatic playback scheduling by the program. The terminal device locates the Gaussian deformation attribute label map corresponding to the target video frame in the interactive video resources based on the frame number or timestamp of the target video frame.

[0075] In a specific implementation of an exemplary embodiment, if the Gaussian deformation attribute label map corresponding to the target video frame has been decoded and the target deformation data has been stored in the cache, the terminal device directly reads the target deformation data from the cache.

[0076] If the Gaussian deformation attribute label map corresponding to the target video frame has not yet been decoded, the terminal device decodes and dequantizes the Gaussian deformation attribute label map to obtain the target deformation data. The cache can be a memory cache, frame cache, decoding result cache, texture cache, or other browser-side cache structures. Furthermore, a recent frame cache or LRU cache can be used to store several decoded frames to reduce repeated decoding during short-distance playback or timeline dragging.

[0077] In a specific implementation of an exemplary embodiment, the browser side sets up a prefetching mechanism. For example, when the player plays the video sequentially according to the timeline, it requests or decodes deformation data for several frames following the current target video frame in advance; when the user drags the timeline, it prioritizes decoding the target video frame corresponding to the user's final position or the latest requested position. The amount of prefetching can be dynamically adjusted according to network conditions, decoding speed, browser memory usage, device performance, or playback frame rate.

[0078] In addition, the terminal device can also handle asynchronous decoding and timeline request merging. Specifically, if a new player timeline request is received while the Gaussian deformation attribute label graph of the current target video frame is being decoded or the attribute write-back is not yet complete, the terminal device records the latest target video frame. After the decoding or attribute write-back of the current target video frame is completed, the terminal device skips intermediate expired video frames and decodes and dequantizes the Gaussian deformation attribute label graph corresponding to the latest target video frame. This method avoids the browser repeatedly decoding and writing back expired intermediate frames when the user quickly drags the timeline, thereby reducing invalid calculations, avoiding out-of-order asynchronous writes, and improving the timeline interaction response speed.

[0079] In step S150, based on the target deformation data and dynamic index information, while keeping the static Gaussian primitives in the Gaussian rendering object from being written back to their attributes frame by frame, the dynamic Gaussian primitives indicated by the dynamic index information are written back to obtain the rendering data corresponding to the target video frame.

[0080] In the exemplary embodiments of this application, attribute write-back refers to writing the attributes of dynamic Gaussian primitives in the target video frame into the attribute storage area of ​​the Gaussian rendering object corresponding to the positions of the dynamic Gaussian primitives. Attribute write-back differs from ordinary video frame display and also differs from applying only a global transformation matrix to a 3D object. Attribute write-back directly acts on the Gaussian primitive attributes within the Gaussian rendering object, giving the dynamic Gaussian primitives in the Gaussian rendering object corresponding center position, scale, rotation, or other attributes in the target video frame.

[0081] Specifically, the execution process of step S150 includes: determining the attribute writing range corresponding to the dynamic Gaussian primitives in the attribute storage area of ​​the Gaussian rendering object according to the dynamic index information; determining the attributes of the dynamic Gaussian primitives under the target video frame based on the target deformation data, and writing the attributes of the dynamic Gaussian primitives under the target video frame into the attribute writing range; after completing the attribute writing, synchronously updating the rendering auxiliary structure of the Gaussian rendering object, the rendering auxiliary structure including at least one of sorting center, bounding box, graphics processor buffer, and texture resource state; and obtaining the rendering data corresponding to the target video frame based on the updated dynamic Gaussian primitives and the updated rendering auxiliary structure.

[0082] In a specific implementation of an exemplary embodiment, the terminal device determines the attribute writing range corresponding to the dynamic Gaussian primitives in the attribute storage area of ​​the Gaussian rendering object based on dynamic index information. The attribute writing range can be determined by the dynamic start index and dynamic quantity, or by the dynamic Gaussian primitive index table or the attribute writing offset table. Then, the terminal device determines the attributes of the dynamic Gaussian primitives in the target video frame based on the target deformation data and writes the attributes into the range. At the same time, the attribute storage location corresponding to the static Gaussian primitives remains unchanged during playback without frame-by-frame attribute write-back.

[0083] The target deformation data includes the attribute changes of dynamic Gaussian primitives under the target video frame. The terminal device obtains these attribute changes from the target deformation data and synthesizes them with the attributes of the dynamic Gaussian primitives under the reference state to obtain the attributes of the dynamic Gaussian primitives under the target video frame. The attribute changes include at least one of center position changes, scale changes, and rotation changes; in other implementations, they may also include at least one of transparency changes, color changes, or spherical harmonic coefficient changes. The terminal device writes the attributes of the dynamic Gaussian primitives under the target video frame into the attribute writing range to update the dynamic Gaussian primitives in the Gaussian rendering object.

[0084] Specifically, rotation in the attribute is represented by a rotation vector. When the target deformation data includes a rotation vector, the terminal device can convert the rotation vector into a quaternion and perform quaternion order conversion and normalization according to the quaternion order convention in the metadata. For example, the export end can use the WXYZ order, the browser's internal calculation uses the XYZW order, and the field order of the Gaussian rendering object can use other conventions. When reading and writing rotation attributes, the terminal device explicitly performs order conversion according to the metadata and normalizes the quaternions to ensure the correctness of the rotation attributes in the target video frame.

[0085] Attribute writes occur in the browser's memory in Gaussian attribute arrays, TypedArrays, ArrayBuffers, texture data, WebGL buffers, WebGPU buffers, or other attribute storage structures. For WebGL implementations, the terminal device updates the corresponding buffer or texture, triggering subsequent graphics processor uploads or resource refreshes. For WebGPU implementations, the terminal device can write to a GPUBuffer or texture resource and submit the update via a command queue. For implementations based on PlayCanvas or other rendering frameworks, the terminal device can write attribute fields such as center position, scale, and rotation to GSplatData or equivalent data structures.

[0086] After the attributes are written, the terminal device synchronously updates the rendering auxiliary structure of the Gaussian rendering object based on the updated attributes of the dynamic Gaussian primitives. The rendering auxiliary structure refers to the data structure or resource state within the Gaussian rendering object used to assist in completing Gaussian rendering on the browser side; it differs from the Gaussian primitive's own attributes such as center position, scale, and rotation. The rendering auxiliary structure can include at least one of the following: sort center, bounding box, graphics processor buffer, and texture resource state.

[0087] Among them, the sorting center is used to determine the sorting basis of transparent Gaussian primitives during the rendering process; the bounding box is used to describe the spatial coverage of the updated dynamic Gaussian primitives or Gaussian rendering objects; the graphics processor buffer is used to provide the updated Gaussian primitive properties to the browser graphics rendering pipeline; the texture resource state includes at least one of the following: texture object for storing Gaussian primitive properties, texture data update marker, texture upload state, or binding state between texture and graphics processor rendering pipeline.

[0088] Since transparent Gaussian rendering typically relies on primitive centers or approximate depth ordering, if the center position of a dynamic Gaussian primitive changes, and the transformation is only temporarily superimposed in the shader without updating the ordering center in the Gaussian rendering object, the transparent blending order may become inconsistent with the dynamic state under the current target video frame. Similarly, if the bounding box is not updated after a change in scale or position, the browser may not be able to accurately describe the spatial coverage of the Gaussian rendering object in the current frame. If the graphics processor buffer or texture resource state is not updated after the attribute is written, the browser's graphics rendering pipeline may not be able to read the latest dynamic Gaussian primitive attributes under the target video frame. Therefore, this embodiment synchronously updates the rendering auxiliary structure after the attribute is written to ensure that the rendering data corresponding to the target video frame is consistent with the current dynamic Gaussian primitive state.

[0089] In a specific implementation of an exemplary embodiment, after loading interactive video resources, the terminal device first stops the current timeline playback and clears the old scene, then reads metadata to determine the dynamic resource type; subsequently, it imports static and dynamic base resources and merges them into the same Gaussian rendering object; after the base resource initialization is completed, it copies the reference center position, reference logarithmic scale, and reference rotation of the dynamic Gaussian primitive interval; then, it creates frame-by-frame resources or video stream resources according to the dynamic resource type, sets the frame rate and total number of frames on the player's timeline, and forces the decoding and application of the initial video frames. For video attribute stream resources that take a long time to decode initially, the terminal device hides the Gaussian rendering object that has not yet completed the application of dynamic frames before the application of the initial video frames is completed, and displays a loading progress prompt to prevent the user from seeing intermediate images where the dynamic state has not yet been applied.

[0090] Through step S150, the range of data that needs to be updated frame-by-frame for each target video frame on the browser side is limited to the attribute writing range corresponding to the dynamic Gaussian primitives, while static Gaussian primitives remain resident and are not written back to their attributes frame-by-frame. This method avoids repetitive data processing for static backgrounds or static objects, and also avoids rebuilding the complete Gaussian rendering object frame-by-frame, thereby reducing the burden of memory writing, resource uploading, and rendering updates on the browser side.

[0091] Step S160: Based on the rendering data and the virtual camera parameters under the current user's perspective, draw the four-dimensional scene interactive display screen of the corresponding perspective under the target video frame in the browser runtime environment.

[0092] The virtual camera parameters include its position, orientation, field of view, view matrix, projection matrix, target point, zoom distance, or other parameters used to determine the browser's viewing angle. The virtual camera differs from the real camera used during the acquisition phase; it is a software camera used by the browser to generate the current user's viewpoint during interactive display. User interaction operations can include rotation, zoom, panning, viewpoint switching, dragging, scroll wheel zoom, touch zoom, two-finger rotation, or other operations that can change the virtual camera parameters.

[0093] The terminal device responds to user-viewpoint interaction operations, updates the virtual camera parameters at the current user's perspective, and performs Gaussian rendering on the rendering data corresponding to the target video frame based on the updated virtual camera parameters, obtaining a four-dimensional scene interactive display screen at the corresponding perspective of the target video frame. As the player continuously plays the timeline, the terminal device continuously acquires target deformation data corresponding to different video frames, continuously writes back dynamic Gaussian primitive attributes, and draws a continuously changing four-dimensional scene interactive display screen under the current virtual camera parameters. When the player pauses or the user drags the timeline to a specific target video frame, the terminal device maintains the dynamic state corresponding to that target video frame and allows the user to interact and observe the target object around that target video frame.

[0094] Through steps S110 to S160 above, the embodiments of this application can realize four-dimensional scene interactive playback in the browser runtime environment. Specifically, static base resources and dynamic base resources enable the browser to recover static Gaussian primitives and dynamic Gaussian primitives in the baseline state, and merge the two into a resident Gaussian rendering object; the Gaussian primitive deformation attribute label map corresponding to each video frame enables the dynamic Gaussian primitives to recover the dynamic state under the target video frame along the player's timeline; dynamic index information enables the browser to locate the attribute storage location of the dynamic Gaussian primitives; attribute write-back enables the dynamic state under the target video frame to directly act on the Gaussian rendering object; static Gaussian primitives do not undergo frame-by-frame attribute write-back, which can reduce redundant processing; synchronous updates of the rendering auxiliary structure can improve the correctness of Gaussian rendering under the current target video frame; virtual camera parameters enable the rendering data under the target video frame to generate display images from different spatial viewing angles.

[0095] Therefore, the embodiments of this application are neither ordinary two-dimensional video playback nor a playback method that loads a complete three-dimensional model frame by frame. Instead, on the browser side, through the collaboration between static base resources, dynamic base resources, Gaussian element deformation attribute tag graphs, dynamic index information, and local attribute write-back, the dynamic display process of the target object is made lightweight, continuous, and interactive.

[0096] See Figure 2 , Figure 2 It is based on Figure 1 The flowchart shown in the corresponding embodiment describes the steps of resolving interactive video resources to recover static Gaussian elements represented by static base resources, dynamic Gaussian elements in a baseline state represented by dynamic base resources, and deformation data corresponding to each video frame represented by a Gaussian element deformation attribute label map.

[0097] The step S120 of parsing interactive video resources provided in this application embodiment, which recovers static Gaussian elements represented by static base resources, dynamic Gaussian elements in a reference state represented by dynamic base resources, and deformation data corresponding to each video frame represented by Gaussian element deformation attribute label maps, includes: Step S121: parse the interactive video resources to obtain the two-dimensional attribute map corresponding to the static base resource, the two-dimensional attribute map corresponding to the dynamic base resource, and the Gaussian deformation attribute label map corresponding to each video frame. Step S122: Dequantize the two-dimensional attribute graphs corresponding to the static base resource and the two-dimensional attribute graphs corresponding to the dynamic base resource to recover the static Gaussian elements and the dynamic Gaussian elements in the baseline state, respectively. Step S123: Decode the Gaussian deformation attribute label map corresponding to each video frame as a data map carrying Gaussian attributes, and dequantize the quantized label obtained by decoding to obtain the deformation data corresponding to each video frame.

[0098] These steps are explained in detail below.

[0099] In this embodiment, step S120 is a process of restoring the interactive video resources loaded in step S110 before playback. Since the static base resources, dynamic base resources, and Gaussian deformable attribute label graphs in the interactive video resources are all in a quantized and compressed resource form, they need to be restored to data that can be used by the browser to construct Gaussian rendering objects and perform dynamic attribute write-back.

[0100] In step S121, the terminal device reads the metadata in the interactive video resources and locates the two-dimensional attribute map corresponding to the static base resource, the two-dimensional attribute map corresponding to the dynamic base resource, and the Gaussian deformation attribute label map corresponding to each video frame based on the metadata. For details regarding the fields that may be included in the metadata, the attribute types of the two-dimensional attribute maps, and the resource format of the Gaussian deformation attribute label maps, please refer to the relevant descriptions in the foregoing embodiments, which will not be repeated here.

[0101] During the specific parsing process, the terminal device determines the attribute reading position of each Gaussian element based on the size of the two-dimensional attribute graph and the arrangement of attribute channels. For static base resources, the terminal device determines the attribute position of the static Gaussian elements in the corresponding two-dimensional attribute graph; for dynamic base resources, the terminal device determines the attribute position of the dynamic Gaussian elements in the baseline state in the corresponding two-dimensional attribute graph. Therefore, the quantized attribute values ​​can be subsequently read from the corresponding two-dimensional attribute graph according to the Gaussian element index.

[0102] For each video frame's corresponding Gaussian deformation attribute label map, the terminal device establishes a mapping between the video frame index and the label map resources based on the correspondence between the video frames and the Gaussian deformation attribute label maps. This mapping is used to quickly determine the Gaussian deformation attribute label map that needs to be decoded when the player's timeline points to the target video frame.

[0103] See also Figure 3 , Figure 3 It is based on Figure 2 The flowchart described in the corresponding embodiment describes the steps of parsing interactive video resources to obtain a two-dimensional attribute map corresponding to a static base resource, a two-dimensional attribute map corresponding to a dynamic base resource, and a Gaussian deformation attribute label map corresponding to each video frame.

[0104] The step S121 of parsing interactive video resources provided in this application embodiment to obtain a two-dimensional attribute map corresponding to a static base resource, a two-dimensional attribute map corresponding to a dynamic base resource, and a Gaussian deformation attribute label map corresponding to each video frame includes: Step S1211: Read the metadata from the interactive video resource; Step S1212: Locate the two-dimensional attribute map corresponding to the static base resource and the two-dimensional attribute map corresponding to the dynamic base resource in the interactive video resource based on the metadata; Step S1213: Locate the Gaussian deformation attribute label map corresponding to each video frame in the dynamic deformation stream in the interactive video resource based on the metadata, and establish the correspondence between the video frame and the Gaussian deformation attribute label map.

[0105] In a specific implementation of an exemplary embodiment, the dynamic Gaussian element arrangement order in the dynamic base resource is consistent with the dynamic Gaussian element arrangement order in the Gaussian element deformation attribute label map corresponding to each video frame. That is, the order of the Gaussian elements in the dynamic base resource is consistent with the order of the dynamic Gaussian elements in the Gaussian element deformation attribute label map corresponding to each video frame. i The dynamic Gaussian element, and the first Gaussian element deformation attribute label diagram of the corresponding video frame. i Each attribute label unit corresponds to a specific element. Through this index consistency, the subsequently recovered deformation data can be accurately applied to the corresponding dynamic Gaussian elements, avoiding mismatches between attribute changes and dynamic Gaussian elements.

[0106] In step S122, the terminal device reads the quantized attribute values ​​of static Gaussian elements from the two-dimensional attribute map corresponding to the static base resource, and reads the quantized attribute values ​​of dynamic Gaussian elements in the reference state from the two-dimensional attribute map corresponding to the dynamic base resource, according to the attribute reading location determined in step S121. Subsequently, the terminal device performs dequantization on the read quantized attribute values ​​based on the quantization parameters recorded in the metadata, recovering the static Gaussian elements and the dynamic Gaussian elements in the reference state. It should be noted that the read quantized attribute values ​​are the quantized attribute values ​​of the static Gaussian elements and the quantized attribute values ​​of the dynamic Gaussian elements in the reference state.

[0107] Furthermore, in one exemplary embodiment, see [reference] Figure 4 As shown, Figure 4 This is a flowchart describing a method for recovering static and dynamic Gaussian primitives and their quantization attribute values ​​in a baseline state by dequantizing the quantization parameters indicated by metadata in an interactive video resource, according to a specific implementation of an exemplary embodiment.

[0108] The steps provided in this application embodiment, which involve dequantizing the quantization attribute values ​​of static Gaussian elements and dynamic Gaussian elements in a baseline state based on quantization parameters indicated by metadata in interactive video resources, to recover the static Gaussian elements and the dynamic Gaussian elements in the baseline state, include: Step S201: Read the quantization parameters corresponding to different attributes in the metadata of the interactive video resource. The quantization parameters include at least one of attribute type, quantization bit depth, quantization range, and attribute recovery method. Step S202: Based on the attribute type, determine at least the quantization attribute values ​​corresponding to the center position, scale, and rotation from the quantization attribute values ​​of the static Gaussian elements and the quantization attribute values ​​of the dynamic Gaussian elements in the reference state. Step S203: According to the quantization bit depth, quantization range and attribute recovery method, the quantization attribute values ​​corresponding to different attributes are dequantized respectively to recover the attributes of the static Gaussian unit and the dynamic Gaussian unit in the reference state. The attributes are used to characterize the static Gaussian unit or the dynamic Gaussian unit in the reference state.

[0109] In step S201, the terminal device reads the metadata of the interactive video resource and obtains the quantization parameters corresponding to different attributes from the metadata. Different attributes can use different quantization methods; for example, the center position uses a higher bit quantization label, the scale uses logarithmic scaling or codebook quantization, and rotation uses quaternion encoding or rotation vector encoding. Therefore, by reading the quantization parameters corresponding to different attributes, the terminal device can determine which method should be used to recover various Gaussian metadata attributes.

[0110] In step S202, the terminal device determines, based on the attribute type, at least quantized attribute values ​​corresponding to the center position, scale, and rotation from the quantized attribute values ​​of the static Gaussian primitives and the quantized attribute values ​​of the dynamic Gaussian primitives in the reference state. The center position determines the position of the Gaussian primitive in three-dimensional space, i.e., its spatial position; the scale determines the spatial extent of the Gaussian primitive; and the rotation determines the orientation state of the Gaussian primitive. These attributes are the foundational attributes for subsequent construction of the Gaussian rendering object and for target frame attribute recovery of the dynamic Gaussian primitives.

[0111] In a specific implementation of an exemplary embodiment, for the center position, quantization labels from different two-dimensional attribute maps or different channels are combined into a quantization label with a higher bit width, and then inverse quantization is performed according to the position quantization range. For example, low-bit labels and high-bit labels are combined into a position quantization label to improve the accuracy of center position recovery. If the position coordinates undergo a non-linear transformation at the export end, the browser can perform a corresponding inverse transformation after inverse quantization to recover the center position.

[0112] In another exemplary implementation, for scale, the scale or logarithmic scale is recovered based on the scale quantization label and the scale quantization range; for rotation, the rotation vector or quaternion is recovered based on the rotation quantization label, and the recovered quaternion is normalized; for color, transparency, or spherical harmonic coefficients, the appearance attributes are recovered based on the corresponding quantization label and attribute recovery method.

[0113] The terminal device can employ corresponding inverse quantization methods for different attributes, rather than using a uniform recovery method for all attributes. This allows for the preservation of key attributes such as center position, scale, and rotation while compressing both static and dynamic base resources.

[0114] Through step S122, the terminal device can restore the static and dynamic base resources to Gaussian meta-attribute data. The static Gaussian meta-attributes are used to form the static portion of the subsequent Gaussian rendering object, while the dynamic Gaussian meta-attributes in the baseline state serve as the basis for frame-by-frame deformation data synthesis.

[0115] In step S123, after decoding the Gaussian deformation attribute tag map, the terminal device does not output the decoding result as the display screen. Instead, it reads the pixel values ​​or byte values ​​in the decoding result as quantization tags. Subsequently, the terminal device dequantizes the quantization tags into deformation data for the corresponding video frame according to the quantization parameters in the metadata.

[0116] In a specific implementation of an exemplary embodiment, if the Gaussian deformation attribute label map is in the form of frame-by-frame images, the terminal device decodes it into pixel data and reads the quantization label of the corresponding attribute from the pixel data; if the Gaussian deformation attribute label map is in the form of frame-by-frame binary packets, the terminal device reads the quantization label of the corresponding attribute according to the data offset and data length; if the Gaussian deformation attribute label maps corresponding to multiple video frames are encoded into a video attribute stream, the terminal device decodes the video attribute stream into raw byte data and parses the raw byte data as attribute label data.

[0117] Furthermore, the Gaussian element deformation attribute label map for a single video frame includes position labels, scale labels, and rotation labels. For position labels, the low-order and high-order labels are combined and then dequantized; for scale labels, dequantization is performed to obtain scale changes or logarithmic scale changes; for rotation labels, dequantization is performed to obtain rotation vectors or rotation change quaternions. Thus, the deformation data of the dynamic Gaussian elements for that video frame is obtained.

[0118] In a specific implementation of an exemplary embodiment, deformation data characterizes the property changes of dynamic Gaussian elements relative to a dynamic basis. Assume that the properties of the dynamic Gaussian elements in the baseline state include center position. Logarithmic scale and rotation Target video frame t The corresponding deformation data includes changes in the center position. Logarithmic scale change and rotational changes Then the target video frame t The dynamic Gaussian meta-attributes can be recovered according to the following relationship:

[0119]

[0120]

[0121] in, Indicates the dynamic Gaussian elements in the target video frame t The center position below; Indicates the dynamic Gaussian elements in the target video frame t Logarithmic scale below Indicates the dynamic Gaussian elements in the target video frame t Rotate, This represents quaternion multiplication. If the Gaussian quaternion deformation attribute label graph carries a rotation vector, the terminal device can first convert the rotation vector into a rotation transformation quaternion, and then synthesize it with the reference rotation to form the rotation of the target video frame.

[0122] In another exemplary implementation, deformation data also characterizes the absolute properties of dynamic Gaussian primitives in the corresponding video frame. In this case, the terminal device directly uses the center position, scale, and rotation obtained from inverse quantization as the dynamic Gaussian primitive properties in the corresponding video frame. Differential representation and absolute representation are distinguished by the dynamic representation method in the metadata.

[0123] In step S123, the terminal device can restore the Gaussian primitive deformation attribute label map corresponding to each video frame to the deformation data under the corresponding video frame. This deformation data can be synthesized with the dynamic Gaussian primitive attributes in the reference state in subsequent steps and used to write back the attributes of the dynamic Gaussian primitives in the Gaussian rendering object.

[0124] See Figure 5 , Figure 5 It is based on Figure 2 The flowchart shown in the corresponding embodiment describes the steps of decoding the Gaussian deformation attribute label map corresponding to each video frame as a data map carrying Gaussian attributes, and dequantizing the decoded quantization label to obtain the deformation data corresponding to each video frame.

[0125] The step S123, which involves decoding the Gaussian deformation attribute label map corresponding to each video frame as a data map carrying Gaussian attributes and dequantizing the decoded quantization labels to obtain the deformation data corresponding to each video frame, as shown in the embodiments of this application, includes: Step S1231: Read the pixel values ​​in the Gaussian element deformation attribute label map as quantization labels; Step S1232: Based on the quantization parameters indicated by the quantization tags and metadata in the interactive video resources, recover the attribute changes of the dynamic Gaussian elements in the corresponding video frames. Step S1233: The recovered attribute changes are used as the deformation data of the corresponding video frame.

[0126] These steps are described below.

[0127] In this embodiment, steps S1231 to S1233 are a further refinement of step S123. The key point is to convert the pixel values ​​or byte values ​​in the Gaussian pixel deformation attribute label map from ordinary display pixels into quantized labels used to recover dynamic Gaussian pixel attribute changes. Therefore, the decoding result of the Gaussian pixel deformation attribute label map on the browser side is not directly used as the playback output, but rather as the basis for recovering the frame-by-frame deformation data of the dynamic Gaussian pixels.

[0128] In an exemplary embodiment of step S1231, the terminal device decodes the Gaussian deformation attribute label map corresponding to the target video frame to obtain pixel data or raw byte data. The terminal device reads pixel values ​​from the decoding result according to the image layout, channel arrangement, and attribute type indicated by the metadata, and uses the read pixel values ​​as quantization labels.

[0129] For example, when using a 2×2 atlas for the Gaussian transistor deformation attribute label map, the low-order position label, high-order position label, scale label, and rotation label are read from different atlas regions. For position attributes, the low-order and high-order position labels are combined into a higher-bit-width position quantization label; for scale attributes, the scale label is read; and for rotation attributes, the rotation vector label or rotation encoding label is read. These pixel values ​​are not displayed colors, but rather quantization labels used to represent dynamic changes in Gaussian transistor attributes.

[0130] In another exemplary embodiment, if the Gaussian deformable attribute tag map is stored in the form of frame-by-frame binary packets or video attribute streams, the terminal device reads the quantization tags of the corresponding attributes from the decoded byte data. For video attribute streams, the terminal device decodes the video frames into raw byte data and then reads the quantization tags according to the attribute layout to reduce the impact of color space conversion or range mapping in the ordinary display link on the tag values.

[0131] In step S1232, the terminal device performs inverse quantization on the quantization tags read in step S1231 according to the quantization parameters corresponding to different attributes in the metadata. The quantization parameters include at least one of quantization range, quantization bit depth, attribute type, attribute recovery method, and dynamic representation method.

[0132] For center position changes, inverse quantization is performed based on the position quantization label and position quantization range to obtain the center position change of the dynamic Gaussian unit in the corresponding video frame. If the position change underwent a signed logarithmic transform before encoding, the terminal device performs the corresponding inverse transform after inverse quantization to recover the center position change in real space.

[0133] For scale variations, the scale variation or logarithmic scale variation of the dynamic Gaussian units in the corresponding video frame is recovered based on the scale label and scale quantization range. If the scale is expressed in logarithmic scale form, the recovered result is used as the logarithmic scale variation for subsequent synthesis with the baseline logarithmic scale in the dynamic basis.

[0134] For rotational transformations, the rotation vector or rotational transformation quaternion is recovered based on the rotation label. If the recovered result is a rotation vector, the terminal device converts the rotation vector into a rotational transformation quaternion; if the recovered result is a quaternion, it is further normalized to ensure that the rotation representation satisfies the unit quaternion constraint.

[0135] As mentioned earlier, the attribute changes of dynamic Gaussian elements in the corresponding video frame can be represented by differential or absolute representations, which will not be elaborated further here.

[0136] In step S1233, the recovered attribute changes are organized into deformation data for the corresponding video frame. The deformation data may include at least one of the following: changes in the center position, scale, and rotation of dynamic Gaussian elements in the corresponding video frame. It may also further include changes in transparency, color, or other attribute changes as needed.

[0137] As mentioned earlier, the terminal device establishes a cache mapping between the recovered deformation data and the video frame number or timestamp. When the player's timeline requests the same video frame again, the terminal device directly reads the deformation data corresponding to that video frame from the cache, without having to repeatedly decode the Gaussian deformation attribute label map.

[0138] Through steps S1231 to S1233, the terminal device can interpret the pixel values ​​or byte values ​​in the Gaussian element deformation attribute label map as quantization labels and further recover the deformation data under the corresponding video frame. This process enables the image or video decoding capability of the browser to be used not only for displaying two-dimensional images, but also for recovering the frame-by-frame attribute changes of dynamic Gaussian elements, thereby providing a data foundation for subsequent local attribute write-back based on dynamic index information.

[0139] See Figure 6 , Figure 6 It is based on Figure 1 The corresponding embodiment shows a flowchart describing the steps of merging static Gaussian primitives and dynamic Gaussian primitives in a baseline state into a Gaussian rendering object, and determining the dynamic index information of the dynamic Gaussian primitives in the Gaussian rendering object.

[0140] The step S130 provided in this application embodiment, which merges static Gaussian primitives and dynamic Gaussian primitives in a baseline state into a Gaussian rendering object and determines the dynamic index information of the dynamic Gaussian primitives in the Gaussian rendering object, includes: Step S131: Write the static Gaussian primitives and the dynamic Gaussian primitives in the baseline state into the attribute storage area of ​​the same Gaussian rendering object according to a preset arrangement. Step S132: Determine the dynamic index information corresponding to the dynamic high-level primitives based on the arrangement relationship between the static high-level primitives and the dynamic high-level primitives in the attribute storage area. The dynamic index information is used to locate the attribute storage location corresponding to the dynamic Gaussian primitive in the Gaussian rendering object when performing attribute write-back on the Gaussian rendering object.

[0141] Furthermore, in a specific implementation of an exemplary embodiment, the preset arrangement includes: arranging static Gaussian elements in the preceding interval of the attribute storage area, and arranging dynamic Gaussian elements in the baseline state in the following interval of the attribute storage area. Correspondingly, the execution process of step S132 includes: determining the dynamic starting index of the subsequent interval based on the number of static Gaussian elements corresponding to the preceding interval; determining the dynamic quantity based on the number of dynamic Gaussian elements corresponding to the subsequent interval; and using the dynamic starting index and the dynamic quantity as the dynamic index information.

[0142] In this exemplary embodiment, step S130 is used to organize the restored static Gaussian primitives and the dynamic Gaussian primitives in the baseline state into the same Gaussian rendering object on the browser side, and establish dynamic index information that can locate the dynamic Gaussian primitives. As a result, when the player switches video frames on the timeline, the terminal device does not need to recreate the complete Gaussian rendering object, but can locate the attribute storage location corresponding to the dynamic Gaussian primitive within the same Gaussian rendering object.

[0143] As mentioned earlier, the terminal device creates or initializes a Gaussian rendering object within the browser runtime environment, and the Gaussian rendering object includes an attribute storage area. Under the preset arrangement, the first part of the same Gaussian rendering object corresponds to static Gaussian primitives, and the second part corresponds to dynamic Gaussian primitives.

[0144] Under the control of the Gaussian rendering object, by merging static and dynamic Gaussian primitives, the terminal device does not need to create two separate rendering objects for static and dynamic Gaussian primitives. Static and dynamic Gaussian primitives can share the same sorter, materials, rendering pipeline, graphics processor buffer, or texture resources. This approach helps reduce browser-side object management overhead and avoids rendering results being affected by the sorting, depth blending, or inconsistent resource states of multiple rendering objects during playback.

[0145] The Gaussian meta-object is obtained through merging using a preset arrangement. Taking the attribute storage area of ​​the Gaussian meta-object as an example, if the number of static Gaussian meta-objects is `staticCount` and the number of dynamic Gaussian meta-objects is `dynamicBaseCount`, then the dynamic start index `dynamicStart` = `staticCount`, and the dynamic count `dynamicCount` = `dynamicBaseCount`. Therefore, the index range [`dynamicStart`, `dynamicStart + dynamicCount`] in the Gaussian rendering object corresponds to the attribute storage location of the dynamic Gaussian meta-objects.

[0146] Dynamic index information is used to locate the dynamic Gaussian primitive range in the Gaussian rendering object during subsequent attribute write-back. In other words, when the player's timeline points to the target video frame, the terminal device determines the attribute storage range that needs to be updated based on the dynamic index information, without scanning the entire Gaussian rendering object or re-determining whether each Gaussian primitive belongs to the dynamic part.

[0147] In another exemplary embodiment, if the dynamic Gaussian primitives are not arranged contiguously in the attribute storage area, the dynamic index information includes a dynamic Gaussian primitive index table, a dynamic Gaussian primitive identifier mapping table, or an attribute write offset table. This index table or mapping table is used to record the position of each dynamic Gaussian primitive in the attribute storage area of ​​the Gaussian rendering object. Compared to this alternative approach, using a dynamic starting index and a dynamic number of contiguous intervals is simpler to implement and more suitable for continuous memory writes, texture updates, or graphics processor buffer updates on the browser side.

[0148] In another exemplary embodiment, the terminal device verifies the consistency of the number of dynamic base resources and the Gaussian deformation attribute label maps corresponding to each video frame based on dynamic index information. For example, the dynamic number should be consistent with the number of dynamic Gaussian elements recovered from the dynamic base resources, and also consistent with the number of recoverable dynamic Gaussian deformation data in the Gaussian deformation attribute label map corresponding to the target video frame. If the numbers are inconsistent, the terminal device terminates playback, reloads the resources, or outputs an error message to avoid incorrectly writing deformation data to the attribute storage area.

[0149] Through the exemplary embodiment described above, the terminal device organizes static Gaussian primitives and dynamic Gaussian primitives in the baseline state into the same Gaussian rendering object, and determines the attribute storage location of the dynamic Gaussian primitives within the Gaussian rendering object through dynamic index information. This provides a positioning basis for the subsequent "only performing frame-by-frame attribute write-back on the attribute writing range located by the dynamic index information, while keeping static Gaussian primitives from being written back frame-by-frame," and also provides an implementation basis for the browser to avoid rebuilding the complete Gaussian rendering object frame-by-frame.

[0150] See Figure 7 , Figure 7 It is based on Figure 1 The corresponding embodiment shows a method flowchart describing the steps of obtaining the target deformation data corresponding to the target video frame from the deformation data corresponding to each video frame in response to the player's timeline pointing to the target video frame.

[0151] The step S140 provided in this application embodiment, which obtains the target deformation data corresponding to the target video frame from the deformation data corresponding to each video frame in response to the player's timeline pointing to the target video frame, includes: Step S141: In response to the current playback position of the player's timeline, determine the frame number or timestamp of the target video frame; Step S142: Based on the frame number or timestamp of the target video frame, locate the Gaussian deformation attribute label map corresponding to the target video frame in the interactive video resource. Step S143: With the Gaussian deformation attribute label map already decoded, read the target deformation data from the cache; Step S144: If the Gaussian deformation attribute label map is not decoded, decode and dequantize the Gaussian deformation attribute label map to obtain the target deformation data.

[0152] In this exemplary embodiment, step S140 is used to determine and obtain the target deformation data corresponding to the target video frame from the interactive video resources based on the current playback position of the player's timeline. Unlike decoding ordinary video frames sequentially, this embodiment supports obtaining deformation data frame by frame during normal playback, as well as obtaining deformation data as needed when pausing, dragging the timeline, jumping frames, or repeatedly viewing a specific part of the process.

[0153] In one exemplary embodiment, the player timeline determines the frame number of the target video frame based on the current playback time, playback frame rate, and start time. For example, when the current playback time on the player timeline is... t The frame rate corresponding to the interactive video resources is fps At that time, it can be based on t and fps Determine the frame number of the target video frame. If the interactive video resource uses timestamp indexing, determine the timestamp of the target video frame directly based on the current playback position.

[0154] In one exemplary embodiment, the target video frame can be generated by an automatic playback process, or by user operations such as dragging a progress bar, clicking the timeline, advancing frame by frame, rewinding frame by frame, or jumping to a specified moment. The terminal device converts the current playback position into the nearest frame number or timestamp, and uses this frame number or timestamp as the index for subsequent positioning of the Gaussian element deformation attribute label map.

[0155] In step S142, the terminal device uses the frame number or timestamp obtained in step S141 to query the frame index relationship recorded in the metadata of the interactive video resource to determine the Gaussian deformation attribute label map corresponding to the target video frame. The Gaussian deformation attribute label map can be a data block in a frame-by-frame image resource or a frame-by-frame binary resource, or a frame segment in a video attribute stream corresponding to the target video frame.

[0156] If the interactive video resource is in frame-by-frame format, the terminal device directly determines the corresponding resource address, file identifier, or data block offset based on the frame number. If the interactive video resource is in video-based attribute stream format, the terminal device determines the position of the corresponding frame in the video-based attribute stream based on the frame number or timestamp, and obtains the corresponding frame data by slicing from the decoded raw buffer, or requests the decoder to decode to the corresponding video frame.

[0157] In step S143, the terminal device maintains a decoded frame buffer. This buffer stores the deformation data that has been decoded and dequantized, indexed by the frame number or timestamp. When the player's timeline points to the target video frame, the terminal device first checks whether the target deformation data corresponding to the target video frame exists in the buffer; if it exists, it directly reads the target deformation data from the buffer.

[0158] Caching is suitable for short-range playback during dynamic playback of interactive video resources, situations where users drag the timeline and then return to adjacent frames, and repeated viewing of the same process segment while paused. By directly reading the target deformation data through cache hits, it avoids repeatedly decoding the Gaussian deformation attribute label map and reduces the overhead of image decoding, video decoding, and inverse quantization calculations on the browser side.

[0159] In step S144, if the target deformation data corresponding to the target video frame does not exist in the cache, the terminal device decodes the Gaussian deformation attribute tag map located in step S142, and dequantizes the decoded quantization tag according to the metadata of the interactive video resource to obtain the target deformation data corresponding to the target video frame.

[0160] If the Gaussian deformation attribute label map is in frame-by-frame image format, the terminal device uses the browser's image decoding capability to obtain pixel data and reads the quantization label from the pixel data. If it is in frame-by-frame binary resource format, the terminal device reads the quantization label from the corresponding data block. If it is in video attribute stream format, the terminal device obtains the raw byte data corresponding to the target video frame through the video decoding module and interprets the raw byte data as a quantization label. Subsequently, the terminal device recovers the target deformation data corresponding to the target video frame based on the quantization parameters.

[0161] At this point, after obtaining the target deformation data, the target deformation data is written into the cache and associated with the corresponding frame number or timestamp, so that when the player's timeline points to the same target video frame again, the target deformation data can be read directly from the cache through step S143.

[0162] In addition, the terminal device will prefetch or pre-decode several video frames near the target video frame based on the current playback direction and speed. For example, in normal playback, one or more video frames after the target video frame can be pre-decoded; when the user drags the timeline, the video frame corresponding to the latest pause position is decoded first. This prefetch or pre-decode process does not change the meaning of the target deformation data, but is used to reduce the waiting time during continuous playback or short-distance dragging.

[0163] Through this exemplary embodiment, the terminal device can locate the target video frame according to the player's timeline and directly read the target deformation data when the cache is hit, and decode the Gaussian deformation attribute label map on demand when the cache is not hit. This process eliminates the need for the browser to decode the deformation data corresponding to all video frames at once, and also eliminates the need to repeatedly decode already processed video frames with each timeline request, thereby reducing the decoding overhead during playback and dragging, and providing target frame data for subsequent dynamic Gaussian attribute write-back.

[0164] See Figure 8 , Figure 8 It is based on Figure 1 The corresponding embodiment shows a flowchart describing the steps of drawing a four-dimensional scene interactive display screen of the corresponding viewpoint under the target video frame in the browser runtime environment based on rendering data and virtual camera parameters under the current user's viewpoint.

[0165] The step S160 of this application embodiment, which involves drawing a four-dimensional scene interactive display screen corresponding to the viewpoint of the target video frame in the browser runtime environment based on rendering data and virtual camera parameters from the current user's perspective, includes: Step S161: In response to user-view interaction operations, update the virtual camera parameters under the current user view. Step S162: Based on the updated virtual camera parameters, Gaussian rendering is performed on the rendering data corresponding to the target video frame to obtain the four-dimensional scene interactive display screen of the corresponding viewpoint under the target video frame. The user-view interaction operations include at least one of rotation, scaling, panning, and viewpoint switching.

[0166] The following is a detailed explanation of these two steps.

[0167] In this embodiment, as the rendering data corresponding to the target video frame is generated, a four-dimensional scene interactive display screen is drawn based on the current user's perspective. Unlike the current process of restoring the dynamic state of the target video frame based on the player's timeline, this primarily handles changes in the spatial observation perspective. That is, the player's timeline is used to determine the dynamic state of the target video frame, and the user's perspective interaction is used to determine the observation perspective within that dynamic state.

[0168] In step S161, the current user viewpoint corresponds to the user's current observation state when observing the four-dimensional scene corresponding to the target video frame in the browser display interface. This current observation state can be represented by virtual camera parameters. For example, virtual camera parameters include at least one of the following: virtual camera position, orientation, field of view, observation target point, camera distance, view matrix, and projection matrix. The virtual camera is different from the real camera in the acquisition phase; it is a software camera used to generate the display image in the browser runtime environment.

[0169] In one exemplary embodiment, the terminal device maintains viewpoint state parameters, which are used to generate virtual camera parameters. These viewpoint state parameters include at least one of azimuth, pitch, zoom distance, translation offset, observation target point, and preset viewpoint identifier. The terminal device calculates the position and orientation of the virtual camera based on the viewpoint state parameters, and then generates a projection matrix based on the display area size, field of view, near clipping plane, and far clipping plane, thereby obtaining the virtual camera parameters from the current user's viewpoint.

[0170] When a user performs a rotation operation, the terminal device can update the azimuth and / or pitch angle in the maintained viewpoint state parameters based on the mouse drag distance, touch swipe distance, or gesture rotation amount. The viewpoint state parameters are updated as the virtual camera rotates around the observation target point. Thus, the virtual camera parameters under the current user's viewpoint are obtained based on the updated viewpoint state parameters, enabling the user to observe the target object from different sides, high positions, or oblique angles in the same target video frame.

[0171] When a user zooms in or out on the current browser display, the terminal device updates the distance between the virtual camera and the observed target point, or updates the virtual camera's field of view, based on input via the scroll wheel, two-finger zoom ratio, or zoom button, thus obtaining the virtual camera parameters from the user's current perspective. This allows the user to zoom in to view local details of a target object or zoom out to view the overall scene within the same target video frame.

[0172] When a user performs a panning operation, the terminal device updates the viewing angle parameters based on the dragging direction and distance, adjusting the position of the observed target point or virtual camera to move the currently displayed image horizontally or vertically. This operation can be used to view different local areas of a target object in zoomed-in mode.

[0173] When a user performs a viewpoint switching operation, the terminal device switches the virtual camera parameters to the camera parameters corresponding to the preset viewpoint. The preset viewpoint can include at least one of the following: front view, side view, top view, oblique view, initial view, or user-saved viewpoint. This method allows users to quickly switch to their preferred viewing angle without having to continuously drag and adjust.

[0174] To further explain, the terminal device performs event merging or throttling on user-view interaction operations. For example, when the user continuously drags or scrolls, the terminal device only retains the latest view state and updates the virtual camera parameters uniformly in the next browser rendering frame. This reduces the overhead of frequently calculating the view matrix and triggering rendering on the browser's main thread, improving the smoothness of view interaction.

[0175] In another exemplary implementation, the terminal device sets constraints on the virtual camera parameters, such as limiting the pitch angle range to prevent camera flipping; limiting the zoom distance or field of view range to prevent the target object from becoming invisible or being abnormally cropped; and limiting the translation range to keep the observation target point near the target object. In summary, these constraints improve the stability of user-view interaction.

[0176] In step S162, Gaussian rendering is performed on the rendering data corresponding to the target video frame based on the updated virtual camera parameters. The rendering data corresponding to the target video frame includes the attributes of static Gaussian primitives in the Gaussian rendering object and the attributes of dynamic Gaussian primitives that have been updated according to the target deformation data. After updating the virtual camera parameters, the terminal device performs Gaussian rendering on the Gaussian rendering object based on the updated virtual camera, thereby obtaining a four-dimensional scene interaction image from the corresponding perspective under the target video frame.

[0177] Specifically, the terminal device generates a view matrix and a projection matrix based on the virtual camera parameters, projects the Gaussian primitives in the Gaussian rendering object onto the browser display plane, and generates the display image based on the center position, scale, rotation, transparency, and appearance attributes of the Gaussian primitives. If the rendering pipeline needs to perform transparency sorting according to the current virtual camera viewpoint, the terminal device triggers a sorting update or re-rendering request corresponding to the current viewpoint after the virtual camera parameters change, so that the current display image is consistent with the new viewing viewpoint, such as... Figures 9 to 10 The perspective of observation is switched.

[0178] In a specific implementation of an exemplary embodiment, when the player's timeline remains at the same target video frame and the user only changes the viewpoint, the terminal device maintains the dynamic Gaussian meta-attributes of the target video frame unchanged, only updating the virtual camera parameters and re-rendering the current Gaussian rendering object. Thus, the user can freely rotate, scale, and pan to observe the target object in a paused state or at a specified video frame without needing to re-decode the Gaussian deformation attribute label map or re-restore the target deformation data.

[0179] In another exemplary implementation, when the player's timeline plays continuously and the user simultaneously performs viewpoint interaction operations, the terminal device processes the temporal dimension and spatial observation perspective separately: on the one hand, it continuously updates the target video frame according to the player's timeline and writes back the attributes of dynamic Gaussian primitives; on the other hand, it continuously updates the virtual camera parameters according to the user's viewpoint interaction operations. During each rendering, the browser generates the display screen based on the rendering data of the current target video frame and the virtual camera parameters of the current user's viewpoint. Thus, the dynamic playback process and the free-viewpoint interaction process can occur simultaneously.

[0180] In addition, the terminal device will also trigger screen rendering based on the animation frame scheduling generated by the browser. For example, when the target video frame changes, the target deformation data completes attribute write-back, the virtual camera parameters change, or the display area size changes, the terminal device requests the next rendering frame and uses the latest rendering data and the latest virtual camera parameters to generate a four-dimensional scene interactive display. This method can avoid repeated rendering multiple times in a short period of time, improving the rendering efficiency on the browser side.

[0181] If the user performs a viewpoint switching operation, the terminal device smoothly transitions the virtual camera parameters to those corresponding to the target preset viewpoint. During the smooth transition, the rendering data corresponding to the target video frame remains unchanged. The terminal device continuously updates the virtual camera parameters and re-renders the displayed image, giving the viewpoint switching a continuous visual effect. This method serves as an alternative to directly jumping to a different viewpoint.

[0182] Through steps S161 and S162, the terminal device can combine the dynamic state of the target video frame determined by the player's timeline with the spatial observation perspective determined by the user's interactive operation. Therefore, under the same target video frame, the user can observe the target object from different perspectives through operations such as rotation, scaling, panning, or perspective switching; during continuous playback, the user can also change the observation angle as the target object dynamically changes.

[0183] In summary, the dynamic state restoration and user-view rendering of the target video frame are decoupled. When the timeline changes, the terminal device updates the properties of the dynamic Gaussian primitives; when the viewpoint changes, the terminal device updates the virtual camera parameters and re-renders the current Gaussian rendering object. Therefore, the browser does not need to regenerate or download video frames for each user viewpoint, nor does it need to re-decode dynamic deformation data when only the viewpoint changes, thus enabling free-view interactive display of the target video frame with lower resource overhead.

[0184] In one exemplary embodiment, this application also provides a computer device including a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement the steps of the method as described above.

[0185] In one exemplary embodiment, this application also provides a computer program product including a computer program that, when executed by a processor, implements the steps of the method as described above.

[0186] In one exemplary embodiment, this application also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method as described above.

[0187] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the method according to the embodiments of this application.

[0188] In an exemplary embodiment of this application, a computer-readable storage medium is also provided, on which computer-readable instructions are stored, which, when executed by a computer's processor, cause the computer to perform the methods described in the above method embodiments.

[0189] According to one embodiment of this application, a program product for implementing the methods in the above-described method embodiments is also provided. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and can run on a terminal device, such as a personal computer. However, the program product of this invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0190] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0191] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0192] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, or any suitable combination thereof.

[0193] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0194] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0195] Furthermore, although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0196] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, and includes several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0197] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the appended claims.

Claims

1. A method for implementing interactive playback of a four-dimensional scene in a browser, characterized in that, The method is applied to the browser runtime environment of a terminal device, and the method includes: Load interactive video resources, which include static base resources, dynamic base resources, and Gaussian deformation attribute label maps corresponding to each video frame. The interactive video resources are parsed to recover the static Gaussian elements represented by the static base resources, the dynamic Gaussian elements in the baseline state represented by the dynamic base resources, and the deformation data corresponding to each video frame represented by the Gaussian element deformation attribute label map. The static Gaussian primitives and the dynamic Gaussian primitives in the baseline state are merged into a Gaussian rendering object, and the dynamic index information of the dynamic Gaussian primitives in the Gaussian rendering object is determined. In response to the player's timeline pointing to the target video frame, the target deformation data corresponding to the target video frame is obtained from the deformation data corresponding to each video frame; Based on the target deformation data and the dynamic index information, while keeping the static Gaussian primitives in the Gaussian rendering object from being written back to their attributes frame by frame, the dynamic Gaussian primitives indicated by the dynamic index information are written back to their attributes to obtain the rendering data corresponding to the target video frame. Based on the rendering data and the virtual camera parameters from the current user's perspective, a four-dimensional scene interactive display screen is drawn in the browser runtime environment from the corresponding perspective of the target video frame; The player timeline is used to determine the time dimension of the four-dimensional scene interactive display screen, and the virtual camera parameters are used to determine the spatial observation angle of the four-dimensional scene interactive display screen. The virtual camera parameters change with the user's perspective interaction operation so that the four-dimensional scene interactive display screen switches to the corresponding perspective.

2. The method according to claim 1, characterized in that, The process of parsing the interactive video resources to recover static Gaussian elements represented by the static base resources, dynamic Gaussian elements in a baseline state represented by the dynamic base resources, and deformation data corresponding to each video frame represented by the Gaussian element deformation attribute label map includes: The interactive video resources are analyzed to obtain a two-dimensional attribute map corresponding to the static base resource, a two-dimensional attribute map corresponding to the dynamic base resource, and a Gaussian deformation attribute label map corresponding to each video frame. The two-dimensional attribute graphs corresponding to the static base resource and the two-dimensional attribute graphs corresponding to the dynamic base resource are dequantized to recover the static Gaussian elements and the dynamic Gaussian elements in the baseline state, respectively. The Gaussian deformation attribute label map corresponding to each video frame is used as a data map carrying Gaussian attributes for decoding, and the quantized label obtained by decoding is dequantized to obtain the deformation data corresponding to each video frame.

3. The method according to claim 2, characterized in that, The process of parsing the interactive video resources to obtain a two-dimensional attribute map corresponding to the static base resource, a two-dimensional attribute map corresponding to the dynamic base resource, and a Gaussian deformation attribute label map corresponding to each video frame includes: Read the metadata from the interactive video resource; Based on the metadata, locate the two-dimensional attribute map corresponding to the static base resource and the two-dimensional attribute map corresponding to the dynamic base resource in the interactive video resource; Based on the metadata, locate the Gaussian deformation attribute label map corresponding to each video frame in the interactive video resource, and establish the correspondence between the video frame and the Gaussian deformation attribute label map.

4. The method according to claim 2, characterized in that, The step of inverse quantizing the two-dimensional attribute map corresponding to the static base resource and the two-dimensional attribute map corresponding to the dynamic base resource to recover the static Gaussian elements and the dynamic Gaussian elements in the baseline state, respectively, includes: The quantization attribute values ​​of static Gaussian elements are read from the two-dimensional attribute graph corresponding to the static base resource, and the quantization attribute values ​​of dynamic Gaussian elements in the baseline state are read from the two-dimensional attribute graph corresponding to the dynamic base resource. Based on the quantization parameters indicated by the metadata in the interactive video resource, the quantization attribute values ​​of the static Gaussian unit and the quantization attribute values ​​of the dynamic Gaussian unit in the baseline state are dequantized to recover the static Gaussian unit and the dynamic Gaussian unit in the baseline state.

5. The method according to claim 4, characterized in that, The step of dequantizing the quantization attribute values ​​of the static Gaussian elements and the quantization attribute values ​​of the dynamic Gaussian elements in the baseline state according to the quantization parameters indicated by the metadata in the interactive video resource, and recovering the static Gaussian elements and the dynamic Gaussian elements in the baseline state, includes: Read the quantization parameters corresponding to different attributes from the metadata of the interactive video resource. The quantization parameters include at least one of attribute type, quantization bit depth, quantization range, and attribute recovery method. Based on the attribute type, at least quantization attribute values ​​corresponding to the center position, scale, and rotation are determined from the quantization attribute values ​​of the static Gaussian element and the quantization attribute values ​​of the dynamic Gaussian element in the reference state. According to the quantization bit depth, the quantization range, and the attribute recovery method, the quantization attribute values ​​corresponding to different attributes are dequantized to recover the attributes of the static Gaussian unit and the dynamic Gaussian unit in the reference state. The attributes are used to characterize the static Gaussian unit or the dynamic Gaussian unit in the reference state.

6. The method according to claim 2, characterized in that, The step of decoding the Gaussian deformation attribute label map corresponding to each video frame as a data map carrying Gaussian attributes, and then dequantizing the decoded quantization labels to obtain the deformation data corresponding to each video frame includes: The pixel values ​​in the Gaussian element deformation attribute label map are read as quantization labels; Based on the quantization label and the quantization parameters indicated by the metadata in the interactive video resource, recover the attribute changes of the dynamic Gaussian primitives in the corresponding video frame; The recovered attribute changes are used as the deformation data for the corresponding video frames.

7. The method according to claim 1, characterized in that, The step of merging the static Gaussian primitives and the dynamic Gaussian primitives in the baseline state into a Gaussian rendering object, and determining the dynamic index information of the dynamic Gaussian primitives in the Gaussian rendering object, includes: The static Gaussian primitives and the dynamic Gaussian primitives in the baseline state are written into the attribute storage area of ​​the same Gaussian rendering object according to a preset arrangement. Based on the arrangement relationship between the static Gaussian elements and the dynamic Gaussian elements in the baseline state in the attribute storage area, determine the dynamic index information corresponding to the dynamic Gaussian elements; The dynamic index information is used to locate the attribute storage location corresponding to the dynamic Gaussian primitive in the Gaussian rendering object when performing attribute write-back on the Gaussian rendering object.

8. The method according to claim 7, characterized in that, The preset arrangement includes: arranging the static Gaussian elements in the preceding interval of the attribute storage area, and arranging the dynamic Gaussian elements in the baseline state in the following interval of the attribute storage area. The step of determining the dynamic index information corresponding to the dynamic Gaussian element based on the arrangement relationship between the static Gaussian element and the dynamic Gaussian element in the baseline state in the attribute storage area includes: The dynamic starting index of the subsequent interval is determined based on the number of static Gaussian elements corresponding to the preceding interval. The dynamic quantity is determined based on the number of dynamic Gaussian elements corresponding to the subsequent interval; The dynamic starting index and the dynamic quantity are used as the dynamic index information.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-8.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-8.

Images