Data encapsulation method, data rendering method and related products

Abstract

The invention discloses a data encapsulation method, a data rendering method and a related product. The data packaging method comprises the steps that a data file is sent, the data file comprises rendering space parameters and data description parameters, the rendering space parameters are used for describing attribute information of a rendering space, and the data description parameters are used for describing three-dimensional Gaussian splash 3DGS; the data description parameters and the rendering space parameters are used to render a digital object in the rendering space. The method is used for enriching the rendering effect and improving the rendering quality.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data encapsulation method, a data rendering method, and related products. Background Technology

[0002] The production and application of 3D interactive content has long been considered a major technological innovation following the popularization of images and videos. With the continuous evolution of bandwidth and terminal computing power, the dimensions of information expression and interaction are constantly expanding. With the development of 3D reconstruction technology, free-viewpoint and interactive content has broken through the limitations of traditional videos viewed from a fixed perspective, allowing users to freely switch their focus from any angle. The traditional 3D reconstruction process mainly includes core stages such as data acquisition, model reconstruction, and rendering output. The data acquisition stage collects 3D data, such as 3D Gaussian splatting (3DGS); the model reconstruction stage processes the 3D data to obtain a 3D model; and the core task of the rendering output stage is to simulate the physical laws of the real world, such as lighting, materials, and textures, based on the 3D model, thereby creating digital objects with realistic visual effects.

[0003] However, due to differences in scene data such as background and ambient light at different acquisition locations, the subsequent processing of different scene data during 3D rendering will vary significantly, resulting in a monotonous rendering effect and affecting rendering quality. For example, the rendering effect may lack realism, and the visual effects of multiple digital objects may lack consistency.

[0004] Therefore, enriching rendering effects and improving rendering quality are of great significance to the production and application of 3D interactive content. Summary of the Invention

[0005] This application provides a data encapsulation method, a data rendering method, and related products. The related products include a data encapsulation device, a data rendering device, an electronic device, and a computer-readable storage medium, which can enrich rendering effects and improve rendering quality.

[0006] Firstly, a data encapsulation method is provided, the method comprising: Send a data file, which includes rendering space parameters and data description parameters. The rendering space parameters are used to describe the attribute information of the rendering space, and the data description parameters are used to describe the 3D Gaussian splash 3DGS. The data description parameters and the rendering space parameters are used to render digital objects in the rendering space.

[0007] In any embodiment of this application, the rendering space parameters include at least one of the following: spatial background description information, spatial occlusion information, spatial color filter or spatial lighting attribute description information, wherein the spatial background description information is used to describe the background texture of the rendering space; the spatial occlusion information is used to describe the effective rendering area of ​​the rendering space; the spatial color filter is used to describe the 3DGS color filtering function in the rendering space; and the spatial lighting attribute description information is used to describe the lighting information of the rendering space.

[0008] In conjunction with any embodiment of this application, the spatial background description information includes at least one of the following: the rendering mode of the background image, the color of the background image, the identifier of the background image, the starting position coordinates of the background image, the coverage area size of the background image, the wrapping radius of the background image, or the hash mapping mode, wherein the rendering mode of the background image is used to indicate the type of the background image, and the hash mapping mode is used to indicate the hash algorithm used to generate and / or parse the background image.

[0009] In any embodiment of this application, the identifier of the background image includes at least one of the following: a resource link of the background image, a hash description of the background image, or the background image itself.

[0010] In any embodiment of this application, the identifier of the background image includes a hash description of the background image, and the spatial background description information further includes the hash mapping pattern.

[0011] In any embodiment of this application, the spatial occlusion information includes at least one of the following: spatial occlusion mode, spatial bounding box shape mode, center coordinate point of the spatial bounding box, radius information of the spatial bounding box, or rotation information of the spatial bounding box; the spatial occlusion mode is used to indicate whether occlusion exists.

[0012] In conjunction with any embodiment of this application, the spatial color filter includes at least one of the following: a color filtering mode for indicating a color filtering function, or the coefficients of the color filtering function.

[0013] In conjunction with any embodiment of this application, the spatial lighting attribute description information includes at least one of the following: light source mode, number of light sources or light source information, wherein the light source mode is used to indicate the overlay mode between at least two light sources, and the light source information includes at least one of the following: an index for indicating the rendering priority of the light source, light source type, initial light intensity of the light source, attenuation coefficient of the light source, center orientation of the light source, scattering radius of the light source or scattering angle of the light source.

[0014] In any embodiment of this application, the light source type includes a parallel light source, and the light source information further includes at least one of the following: the center orientation of the light source or the scattering radius of the light source.

[0015] In any embodiment of this application, the light source type includes a point light source, and the light source information also includes the scattering angle of the light source.

[0016] Secondly, a data rendering method is provided, the method comprising: Receive a data file, the data file including rendering space parameters and data description parameters, the rendering space parameters are used to describe the attribute information of the rendering space, the data description parameters are used to describe the 3D Gaussian splash 3DGS, the data description parameters and the rendering space parameters are used to render digital objects in the rendering space; Output the digital object.

[0017] In conjunction with any embodiment of this application, the method further includes: The digital object is obtained by rendering based on the rendering space parameters and the data description parameters.

[0018] In any embodiment of this application, the rendering space parameters include at least one of the following: spatial background description information, spatial occlusion information, spatial color filter or spatial lighting attribute description information, wherein the spatial background description information is used to describe the background texture of the rendering space; the spatial occlusion information is used to describe the effective rendering area of ​​the rendering space; the spatial color filter is used to describe the 3DGS color filtering function in the rendering space; and the spatial lighting attribute description information is used to describe the lighting information of the rendering space.

[0019] In conjunction with any embodiment of this application, the spatial background description information includes at least one of the following: the rendering mode of the background image, the color of the background image, the identifier of the background image, the starting position coordinates of the background image, the coverage area size of the background image, the wrapping radius of the background image, or the hash mapping mode, wherein the rendering mode of the background image is used to indicate the type of the background image, and the hash mapping mode is used to indicate the hash algorithm used to generate and / or parse the background image.

[0020] In any embodiment of this application, the identifier of the background image includes at least one of the following: a resource link of the background image, a hash description of the background image, or the background image itself.

[0021] In any embodiment of this application, the identifier of the background image includes a hash description of the background image, and the spatial background description information further includes the hash mapping pattern.

[0022] In any embodiment of this application, the spatial occlusion information includes at least one of the following: spatial occlusion mode, spatial bounding box shape mode, center coordinate point of the spatial bounding box, radius information of the spatial bounding box, or rotation information of the spatial bounding box; the spatial occlusion mode is used to indicate whether occlusion exists.

[0023] In conjunction with any embodiment of this application, the spatial color filter includes at least one of the following: a color filtering mode for indicating a color filtering function, or the coefficients of the color filtering function.

[0024] In conjunction with any embodiment of this application, the spatial lighting attribute description information includes at least one of the following: light source mode, number of light sources, or light source information, and the light source information includes at least one of the following: an index for indicating the rendering priority of the light source, light source type, initial light intensity of the light source, attenuation coefficient of the light source, center orientation of the light source, scattering radius of the light source, or scattering angle of the light source.

[0025] In any embodiment of this application, the light source type includes a parallel light source, and the light source information further includes at least one of the following: the center orientation of the light source or the scattering radius of the light source.

[0026] In any embodiment of this application, the light source type includes a point light source, and the light source information also includes the scattering angle of the light source.

[0027] Thirdly, a data encapsulation device is provided, the data encapsulation device comprising: A sending unit is used to send a data file, the data file including rendering space parameters and data description parameters. The rendering space parameters are used to describe the attribute information of the rendering space, and the data description parameters are used to describe the 3D Gaussian splash 3DGS. The data description parameters and the rendering space parameters are used to render digital objects in the rendering space.

[0028] In any embodiment of this application, the data encapsulation device further includes: an encapsulation unit for encapsulating data files.

[0029] In any embodiment of this application, the rendering space parameters include at least one of the following: spatial background description information, spatial occlusion information, spatial color filter or spatial lighting attribute description information, wherein the spatial background description information is used to describe the background texture of the rendering space; the spatial occlusion information is used to describe the effective rendering area of ​​the rendering space; the spatial color filter is used to describe the 3DGS color filtering function in the rendering space; and the spatial lighting attribute description information is used to describe the lighting information of the rendering space.

[0030] In conjunction with any embodiment of this application, the spatial background description information includes at least one of the following: the rendering mode of the background image, the color of the background image, the identifier of the background image, the starting position coordinates of the background image, the coverage area size of the background image, the wrapping radius of the background image, or the hash mapping mode, wherein the rendering mode of the background image is used to indicate the type of the background image, and the hash mapping mode is used to indicate the hash algorithm used to generate and / or parse the background image.

[0031] In any embodiment of this application, the identifier of the background image includes at least one of the following: a resource link of the background image, a hash description of the background image, or the background image itself.

[0032] In any embodiment of this application, the identifier of the background image includes a hash description of the background image, and the spatial background description information further includes the hash mapping pattern.

[0033] In any embodiment of this application, the spatial occlusion information includes at least one of the following: spatial occlusion mode, spatial bounding box shape mode, center coordinate point of the spatial bounding box, radius information of the spatial bounding box, or rotation information of the spatial bounding box; the spatial occlusion mode is used to indicate whether occlusion exists.

[0034] In conjunction with any embodiment of this application, the spatial color filter includes at least one of the following: a color filtering mode for indicating a color filtering function, or the coefficients of the color filtering function.

[0035] In conjunction with any embodiment of this application, the spatial lighting attribute description information includes at least one of the following: light source mode, number of light sources, or light source information, and the light source information includes at least one of the following: an index for indicating the rendering priority of the light source, light source type, initial light intensity of the light source, attenuation coefficient of the light source, center orientation of the light source, scattering radius of the light source, or scattering angle of the light source.

[0036] In any embodiment of this application, the light source type includes a parallel light source, and the light source information further includes at least one of the following: the center orientation of the light source or the scattering radius of the light source.

[0037] In any embodiment of this application, the light source type includes a point light source, and the light source information also includes the scattering angle of the light source.

[0038] Fourthly, a data rendering apparatus is provided, the data rendering apparatus comprising: A receiving unit is used to receive a data file, the data file including rendering space parameters and data description parameters, the rendering space parameters being used to describe the attribute information of the rendering space, the data description parameters being used to describe 3D Gaussian splash 3DGS, and the data description parameters and the rendering space parameters being used to render digital objects in the rendering space. An output unit is used to output the digital object.

[0039] In any embodiment of this application, the data rendering apparatus further includes a rendering unit, configured to render based on the rendering space parameters and the data description parameters to obtain the digital object.

[0040] In any embodiment of this application, the rendering space parameters include at least one of the following: spatial background description information, spatial occlusion information, spatial color filter or spatial lighting attribute description information, wherein the spatial background description information is used to describe the background texture of the rendering space; the spatial occlusion information is used to describe the effective rendering area of ​​the rendering space; the spatial color filter is used to describe the 3DGS color filtering function in the rendering space; and the spatial lighting attribute description information is used to describe the lighting information of the rendering space.

[0041] In conjunction with any embodiment of this application, the spatial background description information includes at least one of the following: the rendering mode of the background image, the color of the background image, the identifier of the background image, the starting position coordinates of the background image, the coverage area size of the background image, the wrapping radius of the background image, or the hash mapping mode, wherein the rendering mode of the background image is used to indicate the type of the background image, and the hash mapping mode is used to indicate the hash algorithm used to generate and / or parse the background image.

[0042] In any embodiment of this application, the identifier of the background image includes at least one of the following: a resource link of the background image, a hash description of the background image, or the background image itself.

[0043] In any embodiment of this application, the identifier of the background image includes a hash description of the background image, and the spatial background description information further includes the hash mapping pattern.

[0044] In any embodiment of this application, the spatial occlusion information includes at least one of the following: spatial occlusion mode, spatial bounding box shape mode, center coordinate point of the spatial bounding box, radius information of the spatial bounding box, or rotation information of the spatial bounding box; the spatial occlusion mode is used to indicate whether occlusion exists.

[0045] In conjunction with any embodiment of this application, the spatial color filter includes at least one of the following: a color filtering mode for indicating a color filtering function, or the coefficients of the color filtering function.

[0046] In conjunction with any embodiment of this application, the spatial lighting attribute description information includes at least one of the following: light source mode, number of light sources, or light source information, and the light source information includes at least one of the following: an index for indicating the rendering priority of the light source, light source type, initial light intensity of the light source, attenuation coefficient of the light source, center orientation of the light source, scattering radius of the light source, or scattering angle of the light source.

[0047] In any embodiment of this application, the light source type includes a parallel light source, and the light source information further includes at least one of the following: the center orientation of the light source or the scattering radius of the light source.

[0048] In any embodiment of this application, the light source type includes a point light source, and the light source information also includes the scattering angle of the light source.

[0049] Fifthly, an electronic device is provided, characterized in that it comprises: a processor and a memory, the memory being used to store computer program code, the computer program code including computer instructions; When the processor executes the computer instructions, the electronic device performs the first aspect and any of its embodiments as described above; when the processor executes the computer instructions, the electronic device also performs the second aspect and any of its embodiments as described above.

[0050] In a sixth aspect, another electronic device is provided, comprising: a processor, a transmitting device, an input device, an output device, and a memory, the memory being used to store computer program code, the computer program code including computer instructions; When the processor executes the computer instructions, the electronic device performs the first aspect and any of its embodiments as described above; when the processor executes the computer instructions, the electronic device also performs the second aspect and any of its embodiments as described above.

[0051] In a seventh aspect, a computer-readable storage medium is provided, wherein a computer program is stored therein, the computer program including program instructions; When the program instructions are executed by the processor, the processor is caused to perform the first aspect and any embodiment thereof as described above; or when the program instructions are executed by the processor, the processor is caused to perform the second aspect and any embodiment thereof as described above.

[0052] Eighthly, a computer program product is provided, the computer program product comprising a computer program or instructions; when the computer program or instructions are executed on a computer, causing the computer to perform the first aspect and any embodiment thereof described above; when the program instructions are executed by a processor, causing the processor to perform the second aspect and any embodiment thereof described above.

[0053] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application.

[0054] In this application, the data file sent by the computer device with encapsulation function includes not only parameters describing 3DGS (i.e., data description parameters) but also parameters describing the attribute information of the rendering space (i.e., rendering space parameters). The data description parameters and rendering space parameters are used to render digital objects in the rendering space. It can be understood that by encapsulating the rendering space parameters, unified data processing can be performed on the 3DGS within the rendering space, thereby reducing the impact of different acquisition methods on the same rendering space, enriching the rendering effect, and improving the rendering quality. For example, it can enhance the detail of the rendering, increase the realism of the rendering effect, or make the display effect of multiple digital objects within the same rendering space more consistent. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0056] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0057] Figure 1a This is a schematic diagram of a scenario supporting 3DoF interaction provided in an embodiment of this application; Figure 1b This is a schematic diagram of a scenario supporting 3DoF+ interaction provided in an embodiment of this application; Figure 1c This is a schematic diagram of a scenario supporting 6DoF interaction provided in an embodiment of this application; Figure 2 This is a schematic diagram of a network architecture provided in an embodiment of this application; Figure 3 This is a flowchart illustrating a data encapsulation method provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a data encapsulation device provided in an embodiment of this application; Figure 5This is a schematic diagram of the structure of a data rendering apparatus provided in an embodiment of this application; Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0058] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0059] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0060] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0061] It should be understood that this application provides a data encapsulation method applicable to 3D reconstruction scenarios, such as autonomous driving, smart cities, industrial inspection, multi-view live streaming, and immersive social interaction. In this application, the data encapsulation method can be applied to the field of artificial intelligence (AI). Artificial intelligence refers to the theories, methods, technologies, and application systems that utilize digital computers or computer-controlled computing to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine capable of reacting in a manner similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0062] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0063] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing, tracking, and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional (3D) object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), autonomous driving, intelligent transportation, and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0064] Before proceeding with the following explanation, let's define the technical terms that will appear in the text.

[0065] 1. Three-dimensional data 3D data is used to reconstruct digital objects and can include mesh data, point cloud data, and 3DGS. These will be discussed separately below: Mesh data is a collection of polygons consisting of vertices, edges, and faces. By using triangles or quadrilaterals as basic units, mesh data forms a continuous representation through connecting points, which can more intuitively identify the shape and topology of digital objects.

[0066] Point cloud data is a collection of data consisting of a large number of discrete points in three-dimensional space. Each point can contain attribute information such as three-dimensional position, color, or reflection intensity. Point cloud data has high precision, high resolution, and high-dimensional geometric information, and can intuitively represent the shape, surface, and texture of digital objects in space.

[0067] 3DGS can be understood as higher-order point cloud data. It's a technique that uses Gaussian distributions to model and render 3D scenes. Its core idea is to represent a scene as a set of 3D Gaussian distributions, which can effectively approximate the scene's geometry and appearance. Each point in 3DGS (called a Gaussian point) can contain parameters such as position (mean), covariance matrix (shape), transparency, and spherical harmonic coefficients (color). The spherical harmonic coefficients represent the color distribution of the Gaussian point under different viewpoints, capturing complex lighting and texture information. Compared to points in point cloud data, it has more dimensions.

[0068] Point clouds can flexibly and conveniently represent the spatial structure and surface properties of three-dimensional objects or scenes, and therefore have a wide range of applications, including virtual reality (VR) games, computer-aided design (CAD), geographic information systems (GIS), autonomous navigation systems (ANS), digital cultural heritage, free-viewpoint broadcasting, 3D immersive remote presentation, and 3D reconstruction of biological tissues and organs.

[0069] In this application, the three-dimensional model reconstructed based on 3DGS can be referred to as a 3DGS model.

[0070] 2. Degree of freedom (DoF) Degrees of freedom (DOF) refer to the freedom of movement and content interaction that a business object (such as a user) supports when viewing immersive media. Immersive media, also known as immersive content, refers to media content that enhances immersion, such as point cloud media and 3DGS media. DOF can include 3DoF (three degrees of freedom), 3DoF+, and 6DoF (six degrees of freedom). 3DoF refers to the three degrees of freedom for the business object's head to rotate around the x, y, and z axes. 3DoF+ adds a limited number of degrees of freedom for movement along the x, y, and z axes. 6DoF adds a limited number of degrees of freedom for movement along the x, y, and z axes.

[0071] For example, to facilitate understanding, please refer to Figure 1a , Figure 1a This is a schematic diagram illustrating a scenario supporting 3DoF interaction, as provided in an embodiment of this application. Figure 1a As shown, 3DoF refers to a situation where the consumer of immersive media is fixed at the center point in a three-dimensional space, and the head of the consumer rotates along the X, Y, and Z axes to view the images provided by the media content.

[0072] Please see Figure 1b , Figure 1b This is a schematic diagram illustrating a scenario supporting 3DoF+ interaction, as provided in an embodiment of this application. Figure 1b As shown, 3DoF+ refers to the ability of a business object's head to move within a limited space based on 3DoF to view the media content when the virtual scene provided by immersive media has a certain depth information.

[0073] Please see Figure 1c , Figure 1c This is a schematic diagram illustrating a scenario supporting 6DoF interaction provided in an embodiment of this application. For example... Figure 1c As shown, 6DoF is divided into window 6DoF, omnidirectional 6DoF, and 6DoF. Window 6DoF restricts the rotation and movement of a business object along the X and Y axes, as well as its translation along the Z axis; for example, a business object cannot see the view outside the window frame, and cannot pass through a window. Omnidirectional 6DoF restricts the rotation and movement of a business object along the X, Y, and Z axes; for example, a business object cannot freely move through a 360-degree VR content within a restricted movement area. 6DoF, building upon 3DoF, allows the business object to translate freely along the X, Y, and Z axes; for example, a business object can move freely within a 360-degree VR content.

[0074] Please see Figure 2 , Figure 2 This is a schematic diagram of a network architecture provided in an embodiment of this application. Figure 2 As shown, the network architecture specifically includes data production equipment and data consumption equipment.

[0075] The main functions of data production equipment include data acquisition, data encoding, and data encapsulation. In one possible implementation, these three functions can be deployed on different computer devices; for example, data acquisition and data encoding can be deployed on a data acquisition device, and data encapsulation on a data encapsulation device. In another possible implementation, these three functions can also be deployed on the same computer device (e.g., ...). Figure 2The device A shown is not limited here. The computer device here can be a terminal or a server. Terminals include personal computers (PCs), smart mobile devices (such as smartphones), etc. Servers here can be standalone servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0076] It is understood that data acquisition refers to acquiring multimedia data (including audio data and / or video data). This audio and video data are synchronized in both time and space. In one possible implementation, multimedia data is obtained by capturing sound-visual scenes from the real world through a data acquisition device. This data acquisition device can be a hardware component deployed within a data production device, or a hardware device connected to the data production device; the deployment method of the data acquisition device is not limited here. This data acquisition device can include audio devices (e.g., audio sensors or microphones), camera devices (e.g., cameras, ordinary cameras, stereo cameras, light field cameras, drone aerial photography), and sensing devices (e.g., laser devices, radar devices). For example, at least one data acquisition device can be deployed at specific locations in a real space to simultaneously acquire multimedia data within that space, such as multimedia data (e.g., 3D data) in a three-dimensional space used to provide a multi-degree-of-freedom (e.g., 6DoF) viewing experience.

[0077] The data encoding process refers to the process of encoding raw multimedia data (audio data and / or video data) into a format suitable for subsequent processing through an encoder, while retaining key information (such as geometric information and attribute information of point cloud data), thereby effectively reducing the data size and transmission bandwidth.

[0078] Data encapsulation refers to the process of encapsulating compressed multimedia data (including audio and / or video streams) according to a specified encapsulation format using an encapsulator to obtain a data file. For example, the file format of this data file can be Graphic Library Transmission Format (glTF).

[0079] Among them, data consumption devices (such as Figure 2The main functions of device B) shown include data decapsulation, data decoding, and data storage / presentation. This data consumption device can refer to a computer device used by a user of multimedia data (such as an object viewing point cloud media). This computer device can be a terminal, such as a PC, a smart mobile device (such as a smartphone), a VR device (such as a VR headset, VR glasses, etc.), a smart home appliance, an in-vehicle terminal, or an aircraft. This computer device integrates a client. The client here can be a client with the function of displaying data information such as text, images, audio, and video, including but not limited to multimedia clients (e.g., video clients), social clients (e.g., instant messaging clients), information applications (e.g., news clients), entertainment clients (e.g., game clients), shopping clients, in-vehicle clients, and browsers. This client can be a standalone client or an embedded sub-client integrated into a client (e.g., a social client); no limitation is made here.

[0080] It's understandable that data decapsulation is the reverse of data encapsulation; that is, data files are decapsulated according to the encapsulation format specifications to obtain compressed multimedia data. Data decoding is the reverse of data encoding, and it can restore the compressed multimedia data. Data storage refers to storing the decoded multimedia data (e.g., point cloud attribute information, point cloud coordinate data) according to the output format specifications. Data presentation refers to rendering the decoded multimedia data and outputting it to data consumption devices. For example, point cloud data can be rendered based on the current viewing position, viewing direction, or viewport, and displayed on the screen of a head-mounted display or other display device.

[0081] In the application scenario of 3DGS, the data file finally packaged by device A may include rendering space parameters (used to describe the attribute information of the rendering space) and data description parameters (used to describe 3DGS). The rendering space parameters and data description parameters are used to render digital objects in the rendering space.

[0082] Furthermore, device A can send data files. For example, in a streaming media scenario, device A can use segmented transmission. For instance, device A can send a stream to device B, which may include at least one segment. This effectively solves problems such as network interruptions and resource consumption during data transmission, thereby improving transmission efficiency and real-time performance. In addition, segmented transmission supports resume capability, so if an interruption occurs, it doesn't need to start from the beginning but can continue uploading from the last completed segment, thus reducing resource waste. For example, in a local playback scenario, device A does not need to segment the data file and can output the entire data file to the client.

[0083] It should be understood that the execution subject of the embodiments of this application is a first communication device, which can be a device for encapsulating 3DGS data (hereinafter referred to as a data encapsulation device, or encapsulation end). The first communication device can be any electronic device capable of executing the technical solutions disclosed in the embodiments of the method of this application. For example, the first communication device can be one of the following: a server or a terminal. It should be understood that the embodiments of the method of this application can also be implemented by a processor executing computer program code. The embodiments of this application will be described below with reference to the accompanying drawings.

[0084] Please see Figure 3 , Figure 3 This is a flowchart illustrating a data encapsulation method provided in an embodiment of this application. The method primarily describes how encapsulating rendering space parameters enables subsequent data rendering devices to perform unified data processing on the 3DGS within the rendering space, thereby enriching rendering effects and improving rendering quality. This method can be executed by a first communication device (i.e., the encapsulation end) and a second communication device (i.e., the rendering end, such as a data consumption device). The first communication device can be the one described above... Figure 2 The device A shown (such as a server) and the second communication device can be the one described above. Figure 2 Device B (such as a terminal) is shown.

[0085] like Figure 3 As shown, this data encapsulation method includes, but is not limited to, the following steps: In step S301, the first communication device sends a data file to the second communication device. The data file includes rendering space parameters and data description parameters. The rendering space parameters are used to describe the attribute information of the rendering space, and the data description parameters are used to describe 3DGS.

[0086] Correspondingly, the second communication device receives a data file from the first communication device. The data description parameters and rendering space parameters are used to render (or reconstruct) digital objects in the rendering space.

[0087] In one possible implementation, the above data description parameters may include at least one of the following: attribute information of the 3DGS model (such as distribution range, coordinate offset, field of view, model view clipping ratio, maximum rotation angle of the model, etc.) or attribute information of the 3DGS data (such as coordinates, normals, colors, spherical harmonics, transparency factors, rotation factors or scaling factors, etc.).

[0088] It is understood that the above rendering space parameters are optional components of 3D image format applications, mainly used to describe information such as lighting, occlusion, filters, or background color when rendering a 3D Gaussian sphere. For example, in glTF-based 3D image formats, the above rendering space parameters can also be used to describe global filter properties in the model's rendering space.

[0089] In one possible implementation, the rendering space parameters may include at least one of the following: spatial background description information, spatial occlusion information, spatial color filter, or spatial lighting attribute description information. For ease of understanding, please refer to Table 1, which is a structural schematic table of rendering space parameters provided in an embodiment of this application, as shown in Table 1: Table 1

[0090] Here, "string" indicates that the parameter type is string, "N / A" indicates that the parameter does not have a default value, and "O" indicates that the parameter is optional. The same expressions in the following text have the same meaning and will not be repeated.

[0091] The following sections will provide a detailed explanation of each of the four parameters shown in Table 1: I. Spatial Background Description Information Spatial background description information (or rendering space background image information) can be used to describe the background texture of the rendering space. Alternatively, it can be understood as: spatial background information indicates the background state of the initial model space of the 3DGS model, and is an extended description of undefined areas of the 3DGS model. The background texture is the environmental image located behind the 3DGS model in the rendering space, providing basic environmental information and a sense of depth for the entire rendering space. For example, placing detailed backgrounds (such as distant buildings, mountains, etc.) can effectively enhance the sense of depth in the image, making the rendering effect more realistic.

[0092] In one possible implementation, the spatial background description information includes at least one of the following: the rendering mode of the background image, the color of the background image, the identifier of the background image, the starting position coordinates of the background image, the size of the coverage area of ​​the background image, the wrapping radius of the background image, or the hash mapping mode.

[0093] The identifier of the background image may include at least one of the following: a resource link of the background image, a hash description of the background image, or the background image itself.

[0094] For ease of understanding, please refer to Table 2 as an example. Table 2 is a structural schematic table of spatial background description information provided in an embodiment of this application, as shown in Table 2: Table 2

[0095] In Table 2, “integer” can indicate that the parameter type is an integer; “array” can indicate that the parameter type is an array; “object” can indicate that the parameter type is an object type, which is a reference data type and can be used to store data in key-value pair form; “M” can indicate that the parameter usage is mandatory choice. The same expressions in the following text have the same meaning and will not be repeated.

[0096] The rendering mode of the background image can be used to indicate the type of the background image. For example, when the rendering mode is 0, it can indicate that the background image is a solid color background defined using RGB; when the rendering mode is 1, it can indicate that the background image is a two-dimensional image, that is, the background texture mode defined by a single two-dimensional image will be used for rendering; when the rendering mode is 2, it can indicate that the background image is a panoramic image, that is, the panoramic texture mode defined by a panoramic image will be used for rendering; when the rendering mode is 3, it can indicate that the background image is a hash image, that is, the hash-described blurred bitmap mode will be used for rendering.

[0097] The background image color can be represented by the RGB values ​​of the background color, with a default value of (0,0,0), which is black. It can be understood that the value range for each color is 0~255. For example, the RGB value for white is (255,255,255), for red it is (255,0,0), for green it is (0,255,0), and for blue it is (0,0,255).

[0098] For example, when the rendering mode of the background image indicates that the background image type is a solid color background, the spatial background description information may include the color of the background image (such as fixed RGB color values). During subsequent rendering, the starting position coordinates of the background image can be considered infinitely far, and the coverage area size of the background image can be considered infinitely large. Here, the color of the background image may be obtained by the first communication device in response to the user's input operation, or it may be obtained by the first communication device from other communication devices (such as a data acquisition terminal). The method of obtaining the color of the background image will not be limited here.

[0099] The hash description of the background image can refer to a string used to represent the background image, such as a string consisting of 8, 16, or 32 characters.

[0100] In one possible implementation, when the spatial background description information includes a hash description of the background image, the spatial background description information also includes a hash mapping pattern. This hash mapping pattern can be used to indicate the hash algorithm used to generate and / or parse the background image. For example, when the hash mapping pattern is "blur," it can represent the "blurHash" algorithm, such as dividing the background image into several color components, performing a discrete cosine transform on the color of each component, extracting the main color information, and encoding the result. When the hash mapping pattern is "thumb," it can represent the "thumbHash" algorithm, such as approximating the compression of the background image through a discrete cosine transform, encoding the image's brightness, chroma, and other information into a small amount of data, thereby improving the utilization of storage space. In other words, the hash description of the background image is a string obtained after hashing the background image based on the hash algorithm indicated by the hash mapping pattern.

[0101] Therefore, by encapsulating spatial background description information, this application can pre-set at least one of two-dimensional images, panoramic images (or spatial images), hash images, or solid color images to set background texture information for the initial space, so as to overlay and display it during rendering, thereby increasing the realism of the rendering effect.

[0102] II. Spatial Occlusion Information Spatial occlusion information (or rendering space occlusion information) can be used to describe the effective rendering area in the rendering space. Occlusion information, also known as masks or masks, can selectively control the rendering or compositing process.

[0103] In one possible implementation, the spatial occlusion information includes at least one of the following: a spatial occlusion pattern and at least one spatial bounding box, where the spatial bounding box can be used to indicate the effective rendering area.

[0104] For ease of understanding, please refer to Table 3a, which is a structural schematic table of spatial occlusion information provided in an embodiment of this application. As shown in Table 3a: Table 3a

[0105] The spatial occlusion mode is used to indicate whether occlusion exists. For example, when the spatial occlusion mode is 0, it can indicate no occlusion. When the spatial occlusion mode is 1, it can indicate that occlusion exists, that is, the effective rendering area of ​​the rendering space is the area indicated by the bounding box of the space.

[0106] In one possible implementation, the spatial bounding box may include at least one of the following: the shape pattern of the spatial bounding box, the center coordinates of the spatial bounding box, the radius information of the spatial bounding box, or the rotation information of the spatial bounding box. For ease of understanding, please refer to Table 3b, which is a structural schematic table of a spatial bounding box provided in an embodiment of this application. As shown in Table 3b: Table 3b

[0107] The shape mode of the bounding box can be used to indicate the shape of the bounding box. For example, when the shape mode of the bounding box is 0, it can indicate that the bounding box is a cube bounding box; when the shape mode of the bounding box is 1, it can indicate that the bounding box is an ellipsoid bounding box; when the shape mode of the bounding box is 2, it can indicate that the bounding box is a sphere bounding box; and when the shape mode of the bounding box is 3, it can indicate that the bounding box is a tetrahedral bounding box.

[0108] As can be seen, this application encapsulates spatial occlusion information, sets spatial bounding boxes and view occlusion, and achieves the truncation or occlusion of invalid rendering areas. In this way, various effects (such as textures, lighting, filters, etc.) can be selectively applied to specific areas (such as local areas inside the 3DGS model) without destroying the 3DGS model. This not only reduces rendering pressure and saves rendering resources, but also allows for a more refined display of rendering effects.

[0109] III. Spatial Color Filter Color space filters (or rendering space filter information) can be used to describe the color filtering function of 3DGS within the rendering space, where the color filtering function is used to represent the color mapping rules.

[0110] In one possible implementation, the spatial color filter includes at least one of the following: a color filtering mode, or coefficients of a color filtering function.

[0111] For ease of understanding, please refer to Table 4, which is a schematic diagram of the structure of a spatial color filter provided in an embodiment of this application. As shown in Table 4: Table 4

[0112] Among them, the color filter mode is used to indicate the color filter function. For example, when the color filter mode is 0, it can represent no color filter function; when the color filter mode is 1, it can represent color filter function 1. For example, color filter function 1 can satisfy the following formula (1): (1) Where R can represent the color value corresponding to the red channel, G can represent the color value corresponding to the green channel, and B can represent the color value corresponding to the blue color channel. R, G, and B are included in the data description parameters. , , Each represents the coefficient corresponding to each color channel; y can represent the mapped color value.

[0113] Optionally, the color filtering mode can also be used to indicate a color lookup table (LUT). This color lookup table can be understood as a three-dimensional lookup table used for color mapping, capable of mapping one set of color values ​​(such as RGB values) to another set of color values. For example, when the color filtering mode is 0, it can represent no color filtering function; when the color filtering mode is 1, it can represent color lookup table 1; and when the color filtering mode is 2, it can represent color lookup table 2. It can be understood that color lookup table 1 and color lookup table 2 are two different color lookup tables.

[0114] As can be seen, this application achieves a unified function mapping of the colors of all Gaussian points in the rendering space by encapsulating the spatial color filter. This allows for additional color mapping during 3DGS model rendering, supports overall color style transfer of the model, and enriches the rendering effect.

[0115] IV. Description of Spatial Illumination Attributes Spatial lighting attribute description information (or rendering space light field information) is used to describe the lighting information in the rendering space. The lighting information can be used to overlay the lighting and brightness of the model in the rendering space.

[0116] In one possible implementation, the spatial lighting attribute description information includes at least one of the following: light source mode, number of light sources, or light source information. The number of light source information items can be one or more, and this is not limited here. If the number of light source information items is multiple, these multiple light source information items can form a light source list. In other words, the spatial lighting attribute description information can also include a light source list, which includes at least one light source information item.

[0117] For ease of understanding, please refer to Table 5a, which is a structural schematic table of spatial illumination attribute description information provided in an embodiment of this application. As shown in Table 5a: Table 5a

[0118] Here, the light source mode can be used to indicate that there is no light source overlay, or to indicate the overlay mode between at least two light sources. For example, when the light source mode is 0, it can indicate that there is no light source overlay, that is, the 3DGS in the rendering space is rendered based on a single light source; when the light source mode is 1, it can indicate that there are at least two light sources in the rendering space, and the brightness of the at least two light sources is accumulated; when the light source mode is 2, it can indicate that there are at least two light sources in the rendering space, and the brightness of the at least two light sources is taken as the maximum value.

[0119] The light source information in the above light source list can be used to represent specific light source definitions. In one possible implementation, the light source information includes at least one of the following: an index for indicating the rendering priority of the light source, the light source type, the initial light intensity of the light source, the attenuation coefficient of the light source, the center orientation of the light source, the scattering radius of the light source, or the scattering angle of the light source.

[0120] For ease of understanding, please refer to Table 5b, which is a schematic diagram of the structure of light source information provided in an embodiment of this application. As shown in Table 5b: Table 5b

[0121] The index used to indicate the rendering priority of a light source is a unique identifier configured according to the order in which the light sources take effect; in other words, this index is used to specify the rendering order of the light sources. For example, the larger the index value, the later the rendering order. Generally, the base light intensity can be considered the light source with the highest priority for rendering.

[0122] The light source type can be used to represent the source of light. For example, when the light source type is 0, it can represent a point light source, which is a light source that emits light rays from a specific position in all directions. When the light source type is 1, it can represent a parallel light source, which is a light ray that enters from a position with no distance and at a specific angle. When the light source type is 2, it can represent the base light intensity, which is the original ambient light intensity of the rendering space is 0, and the superimposed light intensity during rendering is L.

[0123] In one possible implementation, when the light source type includes a parallel light source, the light source information also includes at least one of the following: the center orientation of the light source or the scattering radius of the light source.

[0124] In one possible implementation, when the light source type includes a point light source, the light source information also includes the scattering angle of the light source.

[0125] As can be seen, by encapsulating spatial lighting attribute description information, this application can pre-set light field and light source information, and then overlay them during rendering. This allows multiple 3DGS models in the same rendering space to be rendered under the same lighting, reducing the impact of light acquisition.

[0126] It should be understood that if the acquisition end and the encapsulation end in this application are deployed in different communication devices, then the first communication device in this application refers to the encapsulation end. In one possible implementation, the first communication device can obtain data (including data description parameters) sent by multiple acquisition ends respectively, and then encapsulate the data based on the received data description parameters and the rendering space parameters input by the user at the encapsulation end to obtain a data file.

[0127] In another implementation, if the data sent by the acquisition end includes data description parameters and rendering space parameters, then the first communication device needs to determine the final encapsulated rendering space parameters based on the acquired rendering space parameters, and encapsulate the data based on the determined rendering space parameters and the data description parameters sent by multiple acquisition ends to obtain a data file. For example, taking two acquisition ends as an example, if acquisition end a sends data including data description parameter 1 and rendering space parameter 1, and acquisition end b sends data including data description parameter 2 and rendering space parameter 2, where data description parameter 1 is used to render digital object a (e.g., an airplane) in the rendering space, and data description parameter 2 is used to render digital object 2 (e.g., a building) in the rendering space, then the first communication device can determine the final encapsulated rendering space parameters from rendering space parameter 1 and rendering space parameter 2. For example, the final encapsulated rendering space parameters can be the rendering space parameters received earliest by the first communication device (such as rendering space parameter 1). Furthermore, the first communication device can encapsulate the data based on rendering space parameter 1, data description parameter 1, and data description parameter 2 to obtain a data file, which can then be sent to the second communication device.

[0128] After step S301, Figure 3 The communication method shown may also include: In step S302, the second communication device renders the digital object based on the rendering space parameters and data description parameters.

[0129] For example, the second communication device can parse the received data file to obtain rendering space parameters and data description parameters, and then render the data based on the rendering space parameters and data description parameters to obtain a digital object.

[0130] The following will describe the different scenarios: Case a: Spatial rendering parameters include spatial background description information.

[0131] For example, if the background image is a solid color image, the spatial background description information includes the color of the background image (i.e., the target color). In this case, the second communication device can set the basic properties of the rendering ray (such as effective propagation distance and light energy intensity) to overlay the target color onto the color layer of the rendering space. Subsequently, based on the basic properties of the rendering ray, the color of the background image, and the data description parameters, the 3DGS within the rendering space is rendered to obtain a digital object. Here, the rendering ray is understood as a virtual ray emitted from the viewpoint (camera) into the rendering space, used to simulate the propagation path of real light. It can be understood that the rendering ray overlays the 3DGS model when it penetrates it; if the transparency overlay is less than 100%, it can be displayed at the bottom layer.

[0132] For example, if the background image is a hash image, the spatial background description information includes the hash description of the background image and the hash mapping pattern. In this case, the second communication device can parse the hash field based on the hash mapping pattern to obtain a two-dimensional image, and then instantiate the parsed two-dimensional image as the background texture of the rendering space. Subsequently, based on the background texture and data description parameters, the 3DGS in the rendering space is rendered to obtain a digital object.

[0133] For example, if the background image is a two-dimensional image, the spatial background description information includes the starting position coordinates of the background image and the size of the covered area of ​​the background image. In this case, the second communication device can decode the image format of the two-dimensional image, obtain the original RGB color values, and then scale the image based on the starting position coordinates and the size of the covered area of ​​the background image to fix the two-dimensional image in a specified area. Subsequently, based on the background texture and data description parameters, the 3DGS in the rendering space is rendered to obtain a digital object. The scaling method used during the scaling process can include any of the following: high-order interpolation (e.g., cubic interpolation), nearest-neighbor interpolation, or linear interpolation.

[0134] For example, if the background image is a spatial image, the spatial background description information may include the starting position coordinates of the background image and the surrounding coordinates of the background image. In this case, the second communication device can decode the image format of the spatial image, obtain the original RGB color values, and then calculate the size of the skybox based on the starting position coordinates of the background image and the surrounding radius of the background image. Then, the texture can be mapped onto the space and range corresponding to the skybox according to the inverse transformation method of the spatial map (generally spheres pieced together to form a rectangle). Subsequently, based on the background texture and data description parameters, the 3DGS in the rendering space is rendered to obtain a digital object.

[0135] Case b: Spatial rendering parameters include spatial occlusion information, which may include at least one spatial bounding box.

[0136] For example, the second communication device can filter data in the rendering space by selecting or removing 3DGS (Gaussian points) data within the bounding box.

[0137] For example, the second communication device may also treat the bounding box as an opaque or fixed-color model instance during the rendering process and perform ray truncation.

[0138] For example, the second communication device can also define the effective spatial range (i.e., the visible range) observable by the camera based on spatial occlusion information, and erase or blur 3DGS images that are outside the visible range. It is understood that this implementation can change the view output without affecting the 3DGS model rendering.

[0139] Case c: Spatial rendering parameters include spatial color filters.

[0140] For example, the second communication device can render the 3DGS data within the rendering space (such as the effective rendering space) based on the colors in the 3DGS data in the data description parameters to obtain a first color bitmap. Then, it can perform color mapping on the first color bitmap based on the color filtering function in the spatial color filter to obtain a second color bitmap (i.e., the filtered color bitmap). Further, the second communication device can crop the color values ​​in the second color bitmap based on the original color space range (0~255) to obtain a third color bitmap, and then display the third color bitmap. For example, color values ​​less than 0 can be cropped to 0, and color values ​​greater than 255 can be cropped to 255.

[0141] Case d: Spatial rendering parameters include spatial lighting attribute description information, which includes light source information of at least one light source (such as an index used to indicate the rendering priority of the light source).

[0142] For example, the second communication device can render sequentially based on an index used to indicate the rendering priority of the light source. It should be understood that the second communication device can convert the model's colors from a first color gamut (such as the RGB color gamut) to a second color gamut (such as the YCoCg color gamut), where the YCoCg color gamut is a luminance-chrominance separated color space, where Y represents the luminance component, reflecting lightness and darkness; Co represents the orange chrominance component, reflecting the contrast between orange and blue; and Cg represents the green chrominance component, reflecting the contrast between green and magenta.

[0143] For the base light intensity, the second communication device can enhance the brightness information of the 3DGS rendered colors within the rendering space and clip its effective value within a range (e.g., 0~255). Here, brightness enhancement refers to calculating the superposition value based on the original transparency of the Gaussian point and the attenuation coefficient of the light source, and then dispersing it towards the periphery according to a predefined or physically determined refractive index.

[0144] For point light sources or parallel light sources, the second communication device can calculate the effective angle and range of the light source within the rendering space, and improve the brightness attribute of the path 3DGS along the light trajectory.

[0145] In one possible implementation, if the rendering space parameters include four parameters: spatial background description information, spatial occlusion information, spatial color filter, and spatial lighting attribute description information, then the second communication device can perform rendering in a certain rendering order. For example, the second communication device can first render based on the spatial background description information to provide basic environmental information and depth for the entire rendering space; then render based on the spatial occlusion information; and finally render using the spatial color filter and spatial lighting attribute description information. This can not only improve rendering quality but also reduce rendering pressure and save rendering resources.

[0146] In step S303, the second communication device outputs a digital object.

[0147] For example, the second communication device can output a digital object to a screen or display device. It is understood that the digital object can refer to a 3D photograph of an object / scene generated through 3D reconstruction technology, and for example, the digital object can allow consumers to have an immersive experience from a 6DoF perspective.

[0148] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0149] The methods of the embodiments of this application have been described in detail above, and the apparatus of the embodiments of this application is provided below.

[0150] Please see Figure 4 , Figure 4 This is a schematic diagram of a data encapsulation device provided in an embodiment of this application. The data encapsulation device 4 includes an encapsulation unit 41 and a transmission unit 42. Specifically: The encapsulation unit 41 is used to encapsulate a data file, the data file including rendering space parameters and data description parameters. The rendering space parameters are used to describe the attribute information of the rendering space, and the data description parameters are used to describe the 3D Gaussian splash 3DGS. The data description parameters and the rendering space parameters are used to render digital objects in the rendering space. The sending unit 42 is used to send the data file.

[0151] In any embodiment of this application, the rendering space parameters include at least one of the following: spatial background description information, spatial occlusion information, spatial color filter or spatial lighting attribute description information, wherein the spatial background description information is used to describe the background texture of the rendering space; the spatial occlusion information is used to describe the effective rendering area of ​​the rendering space; the spatial color filter is used to describe the 3DGS color filtering function in the rendering space; and the spatial lighting attribute description information is used to describe the lighting information of the rendering space.

[0152] In conjunction with any embodiment of this application, the spatial background description information includes at least one of the following: the rendering mode of the background image, the color of the background image, the identifier of the background image, the starting position coordinates of the background image, the coverage area size of the background image, the wrapping radius of the background image, or the hash mapping mode, wherein the rendering mode of the background image is used to indicate the type of the background image, and the hash mapping mode is used to indicate the hash algorithm used to generate and / or parse the background image.

[0153] In any embodiment of this application, the identifier of the background image includes at least one of the following: a resource link of the background image, a hash description of the background image, or the background image itself.

[0154] In any embodiment of this application, the identifier of the background image includes a hash description of the background image, and the spatial background description information further includes the hash mapping pattern.

[0155] In any embodiment of this application, the spatial occlusion information includes at least one of the following: spatial occlusion mode, spatial bounding box shape mode, center coordinate point of the spatial bounding box, radius information of the spatial bounding box, or rotation information of the spatial bounding box; the spatial occlusion mode is used to indicate whether occlusion exists.

[0156] In conjunction with any embodiment of this application, the spatial color filter includes at least one of the following: a color filtering mode for indicating a color filtering function, or the coefficients of the color filtering function.

[0157] In conjunction with any embodiment of this application, the spatial lighting attribute description information includes at least one of the following: light source mode, number of light sources or light source information, wherein the light source mode is used to indicate the overlay mode between at least two light sources, and the light source information includes at least one of the following: an index for indicating the rendering priority of the light source, light source type, initial light intensity of the light source, attenuation coefficient of the light source, center orientation of the light source, scattering radius of the light source or scattering angle of the light source.

[0158] In any embodiment of this application, the light source type includes a parallel light source, and the light source information further includes at least one of the following: the center orientation of the light source or the scattering radius of the light source.

[0159] In any embodiment of this application, the light source type includes a point light source, and the light source information also includes the scattering angle of the light source.

[0160] In this application, the data file encapsulated by the data encapsulation device includes not only parameters describing 3DGS (i.e., description information) but also parameters describing the attribute information of the rendering space (i.e., rendering space parameters). The description information and rendering space parameters are used to render digital objects in the rendering space. It can be understood that by encapsulating the rendering space parameters, unified data processing can be performed on the 3DGS within the rendering space, thereby reducing the impact of different acquisition methods on the same rendering space, enriching the rendering effect, and improving the rendering quality. For example, it can enhance the detail of the rendering, increase the realism of the rendering effect, or make the display effect of multiple digital objects within the same rendering space more consistent.

[0161] Please see Figure 5 , Figure 5 This is a schematic diagram of a data rendering apparatus 5 provided in an embodiment of this application. The data rendering apparatus 5 includes a receiving unit 51 and an output unit 52. Optionally, the data rendering apparatus 5 further includes a rendering unit 53. Specifically: The receiving unit 51 is used to receive a data file, the data file including rendering space parameters and data description parameters, the rendering space parameters are used to describe the attribute information of the rendering space, the data description parameters are used to describe the 3D Gaussian splash 3DGS, and the data description parameters and the rendering space parameters are used to render digital objects in the rendering space. Output unit 52 is used to output the digital object.

[0162] In any embodiment of this application, the rendering unit 53 is used to render based on the rendering space parameters and the data description parameters to obtain the digital object.

[0163] In any embodiment of this application, the rendering space parameters include at least one of the following: spatial background description information, spatial occlusion information, spatial color filter or spatial lighting attribute description information, wherein the spatial background description information is used to describe the background texture of the rendering space; the spatial occlusion information is used to describe the effective rendering area of ​​the rendering space; the spatial color filter is used to describe the 3DGS color filtering function in the rendering space; and the spatial lighting attribute description information is used to describe the lighting information of the rendering space.

[0164] In conjunction with any embodiment of this application, the spatial background description information includes at least one of the following: the rendering mode of the background image, the color of the background image, the identifier of the background image, the starting position coordinates of the background image, the coverage area size of the background image, the wrapping radius of the background image, or the hash mapping mode, wherein the rendering mode of the background image is used to indicate the type of the background image, and the hash mapping mode is used to indicate the hash algorithm used to generate and / or parse the background image.

[0165] In any embodiment of this application, the identifier of the background image includes at least one of the following: a resource link of the background image, a hash description of the background image, or the background image itself.

[0166] In any embodiment of this application, the identifier of the background image includes a hash description of the background image, and the spatial background description information further includes the hash mapping pattern.

[0167] In any embodiment of this application, the spatial occlusion information includes at least one of the following: spatial occlusion mode, spatial bounding box shape mode, center coordinate point of the spatial bounding box, radius information of the spatial bounding box, or rotation information of the spatial bounding box; the spatial occlusion mode is used to indicate whether occlusion exists.

[0168] In conjunction with any embodiment of this application, the spatial color filter includes at least one of the following: a color filtering mode for indicating a color filtering function, or the coefficients of the color filtering function.

[0169] In conjunction with any embodiment of this application, the spatial lighting attribute description information includes at least one of the following: light source mode, number of light sources, or light source information, and the light source information includes at least one of the following: an index for indicating the rendering priority of the light source, light source type, initial light intensity of the light source, attenuation coefficient of the light source, center orientation of the light source, scattering radius of the light source, or scattering angle of the light source.

[0170] In any embodiment of this application, the light source type includes a parallel light source, and the light source information further includes at least one of the following: the center orientation of the light source or the scattering radius of the light source.

[0171] In any embodiment of this application, the light source type includes a point light source, and the light source information also includes the scattering angle of the light source.

[0172] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0173] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device 6 includes a processor 61 and a memory 62. Optionally, the electronic device 6 also includes an input device 63 and an output device 64. The processor 61, memory 62, input device 63, and output device 64 are coupled together via connectors, which include various interfaces, transmission lines, or buses, etc., and are not limited in this embodiment of the application. It should be understood that in the various embodiments of this application, coupling refers to mutual connection in a specific way, including direct connection or indirect connection through other devices, such as through various interfaces, transmission lines, buses, etc.

[0174] Processor 61 may include one or more processors, such as one or more central processing units (CPUs). If the processor is a CPU, it may be a single-core CPU or a multi-core CPU. Optionally, processor 61 may be a processor group consisting of multiple CPUs, with the multiple processors coupled to each other via one or more buses. Optionally, the processor may also be other types of processors, etc., which are not limited in this embodiment.

[0175] The memory 62 can be used to store computer program instructions, as well as various types of computer program code, including program code for executing the present application. Optionally, the memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), which is used for related instructions and data.

[0176] Input device 63 is used to input data and / or signals, and output device 64 is used to output data and / or signals. Input device 63 and output device 64 can be independent devices or an integrated device.

[0177] It is understood that in this embodiment of the application, the memory 62 can be used not only to store related instructions, but also to store related data. This embodiment of the application does not limit the specific data stored in the memory.

[0178] Understandable Figure 6 This is merely a simplified design of an electronic device. In practical applications, the electronic device may also include other necessary components, including, but not limited to, any number of input / output devices, processors, memories, etc., and all electronic devices that can implement the embodiments of this application are within the protection scope of this application.

[0179] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0180] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will also readily understand that the various embodiments of this application have different focuses, and for the sake of convenience and brevity, the same or similar parts may not be repeated in different embodiments. Therefore, parts not described or not described in detail in one embodiment can be referred to the descriptions in other embodiments.

[0181] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0182] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0183] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0184] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0185] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A data encapsulation method, characterized in that, The method includes: Send a data file, which includes rendering space parameters and data description parameters. The rendering space parameters are used to describe the attribute information of the rendering space, and the data description parameters are used to describe the 3D Gaussian splash 3DGS. The data description parameters and the rendering space parameters are used to render digital objects in the rendering space.

2. The method according to claim 1, characterized in that, The rendering space parameters include at least one of the following: spatial background description information, spatial occlusion information, spatial color filter or spatial lighting attribute description information, wherein the spatial background description information is used to describe the background texture of the rendering space; The spatial occlusion information is used to describe the effective rendering area of ​​the rendering space; The spatial color filter is used to describe the color filtering function of 3DGS within the rendering space; the spatial lighting attribute description information is used to describe the lighting information of the rendering space.

3. The method according to claim 2, characterized in that, The spatial background description information includes at least one of the following: the rendering mode of the background image, the color of the background image, the identifier of the background image, the starting position coordinates of the background image, the size of the coverage area of ​​the background image, the wrapping radius of the background image, or the hash mapping mode. The rendering mode of the background image is used to indicate the type of the background image, and the hash mapping mode is used to indicate the hash algorithm used to generate and / or parse the background image.

4. The method according to claim 3, characterized in that, The identifier of the background image includes at least one of the following: a resource link of the background image, a hash description of the background image, or the background image itself.

5. The method according to claim 4, characterized in that, The identifier of the background image includes a hash description of the background image, and the spatial background description information also includes the hash mapping pattern.

6. The method according to any one of claims 2-5, characterized in that, The spatial occlusion information includes at least one of the following: spatial occlusion mode, spatial bounding box shape mode, center coordinate point of the spatial bounding box, radius information of the spatial bounding box, or rotation information of the spatial bounding box; the spatial occlusion mode is used to indicate whether occlusion exists.

7. The method according to any one of claims 2-6, characterized in that, The spatial color filter includes at least one of the following: a color filtering mode for indicating a color filtering function, or the coefficients of the color filtering function.

8. The method according to any one of claims 2-7, characterized in that, The spatial lighting attribute description information includes at least one of the following: light source mode, number of light sources, or light source information. The light source mode is used to indicate the overlay mode between at least two light sources. The light source information includes at least one of the following: an index used to indicate the rendering priority of the light source, light source type, initial light intensity of the light source, attenuation coefficient of the light source, center orientation of the light source, scattering radius of the light source, or scattering angle of the light source.

9. The method according to claim 8, characterized in that, The light source type includes a parallel light source, and the light source information also includes at least one of the following: the center orientation of the light source or the scattering radius of the light source.

10. The method according to claim 8, characterized in that, The light source type includes point light sources, and the light source information also includes the scattering angle of the light source.

11. A data rendering method, characterized in that, The method includes: Receive a data file, the data file including rendering space parameters and data description parameters, the rendering space parameters are used to describe the attribute information of the rendering space, the data description parameters are used to describe the 3D Gaussian splash 3DGS, the data description parameters and the rendering space parameters are used to render digital objects in the rendering space; Output the digital object.

12. The method according to claim 11, characterized in that, The method further includes: The digital object is obtained by rendering based on the rendering space parameters and the data description parameters.

13. The method according to claim 11 or 12, characterized in that, The rendering space parameters include at least one of the following: spatial background description information, spatial occlusion information, spatial color filter or spatial lighting attribute description information, wherein the spatial background description information is used to describe the background texture of the rendering space; The spatial occlusion information is used to describe the effective rendering area of ​​the rendering space; The spatial color filter is used to describe the color filtering function of 3DGS within the rendering space; the spatial lighting attribute description information is used to describe the lighting information of the rendering space.

14. The method according to claim 13, characterized in that, The spatial background description information includes at least one of the following: the rendering mode of the background image, the color of the background image, the identifier of the background image, the starting position coordinates of the background image, the size of the coverage area of ​​the background image, the wrapping radius of the background image, or the hash mapping mode. The rendering mode of the background image is used to indicate the type of the background image, and the hash mapping mode is used to indicate the hash algorithm used to generate and / or parse the background image.

15. The method according to claim 14, characterized in that, The identifier of the background image includes at least one of the following: a resource link of the background image, a hash description of the background image, or the background image itself.

16. The method according to claim 15, characterized in that, The identifier of the background image includes a hash description of the background image, and the spatial background description information also includes the hash mapping pattern.

17. The method according to any one of claims 13-16, characterized in that, The spatial occlusion information includes at least one of the following: spatial occlusion mode, spatial bounding box shape mode, center coordinate point of the spatial bounding box, radius information of the spatial bounding box, or rotation information of the spatial bounding box; the spatial occlusion mode is used to indicate whether occlusion exists.

18. The method according to any one of claims 13-17, characterized in that, The spatial color filter includes at least one of the following: a color filtering mode for indicating a color filtering function, or the coefficients of the color filtering function.

19. The method according to any one of claims 13-17, characterized in that, The spatial lighting attribute description information includes at least one of the following: light source mode, number of light sources, or light source information. The light source information includes at least one of the following: an index for indicating the rendering priority of the light source, light source type, initial light intensity of the light source, attenuation coefficient of the light source, center orientation of the light source, scattering radius of the light source, or scattering angle of the light source.

20. The method according to claim 19, characterized in that, The light source type includes a parallel light source, and the light source information also includes at least one of the following: the center orientation of the light source or the scattering radius of the light source.

21. The method according to claim 19, characterized in that, The light source type includes point light sources, and the light source information also includes the scattering angle of the light source.

22. A data encapsulation device, characterized in that, The device includes: A sending unit is used to send a data file, the data file including rendering space parameters and data description parameters. The rendering space parameters are used to describe the attribute information of the rendering space, and the data description parameters are used to describe the 3D Gaussian splash 3DGS. The data description parameters and the rendering space parameters are used to render digital objects in the rendering space.

23. A data rendering apparatus, characterized in that, The device includes: A receiving unit is used to receive a data file, the data file including rendering space parameters and data description parameters, the rendering space parameters being used to describe the attribute information of the rendering space, the data description parameters being used to describe 3D Gaussian splash 3DGS, and the data description parameters and the rendering space parameters being used to render digital objects in the rendering space. An output unit is used to output the digital object.

24. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store computer program code, the computer program code including computer instructions; When the processor executes the computer instructions, the electronic device performs the method as described in any one of claims 1 to 10; When the processor executes the computer instructions, the electronic device may perform the method as described in any one of claims 11 to 21.

25. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions; When the program instructions are executed by the processor, the processor is caused to perform the method described in any one of claims 1 to 10; When the program instructions are executed by the processor, or when the processor is caused to perform the method described in any one of claims 11 to 21.

Images