Volumetric scene streaming transmission method, device and medium based on 3DGS

Abstract

The present invention discloses a volume scene streaming transmission method, device and medium based on 3DGS, which divides the volume scene into a number of non-overlapping cubic areas, and downsamples each cubic area to a data representation of a preset quality level; obtains the viewport information of the terminal device, and predicts the future viewport in combination with the historical prior information extracted from the acquired historical viewport sequence; determines the transmission representation of each transmission time slot according to each data representation, the future viewport and the acquired network quality information; transmits the transmission representation to the user's terminal device to reconstruct the 3DGS scene. The application scheme can realize the prediction of the user's future behavior, solve the spatial variability problem of the state space and the action space, and improve the streaming transmission performance.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a volume scene streaming transmission method, device and medium based on 3DGS. Background Art

[0002] Recently, 3DGS (3D Gaussian Splatting) has revolutionized the field of volumetric scene representation, achieving real-time high-fidelity rendering. Through segmentation-based rendering technology, 3DGS explicitly represents volumetric scenes with multiple 3D Gaussian elements, providing a new volumetric scene representation format. However, in terms of application, this advantage comes at the cost of a large amount of data. For example, using 3DGS to represent a 3×3m 2 The indoor scene requires nearly 1GB of space, which poses a major challenge to streaming under current network conditions.

[0003] Existing technologies cannot predict users' future behaviors, cannot solve the spatial variability problem of state space and action space, and have poor streaming performance. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a volumetric scene streaming transmission method, device and medium based on 3DGS, which can realize the prediction of user's future behavior, solve the spatial variability problem of state space and action space, and improve the streaming transmission performance.

[0005] An embodiment of the present invention provides a volume scene streaming transmission method based on 3DGS, the method comprising:

[0006] Divide the volumetric scene into a number of non-overlapping cubic regions, and downsample each cubic region to a data representation of a preset quality level;

[0007] Obtaining viewport information of the terminal device, and predicting the future viewport in combination with historical prior information extracted from the acquired historical viewport sequence;

[0008] Determine a transmission representation of each transmission time slot according to each data representation, the future viewport, and the acquired network quality information;

[0009] The transmission representation is transmitted to the user's terminal device to reconstruct the 3DGS scene.

[0010] Preferably, the volumetric scene is divided into a plurality of non-overlapping cubic regions, and each cubic region is downsampled to a data representation of a preset quality level, and the method further comprises:

[0011] Divide the volume scene into K non-overlapping cubic spaces as tiles;

[0012] Perform voxel grid downsampling on each tile to obtain data representations with L quality levels;

[0013] The data is represented as ; and Respectively represent k The first tile The number of bits and the number of Gaussians contained in the quality level representation, and Respectively represent two different quality levels.

[0014] Preferably, obtaining the viewport information of the terminal device and predicting the future viewport in combination with the historical prior information extracted from the obtained historical viewport sequence comprises:

[0015] Get User u Viewport information of the terminal device as a specific feature;

[0016] Extract historical prior information from the historical viewport sequence, use a preset attention model to learn the attention of each user in the historical prior information, output the obtained attention to the preset cross attention model, and use the corresponding user's embedding vector e u As a query, learn the user u Attention among all users;

[0017] The specific features and users u The attention among all users is used as historical prior information to decode the deep viewport features and connect them to two separate multi-layer perceptrons to predict the future unknown and future rotation angles of the user's virtual camera as the future viewport respectively;

[0018] in, , , and represents the coordinates of the position of the virtual camera of user u at time slot t, and , and denote the yaw, pitch and roll angles of the virtual camera of user u at time slot t respectively. The attention function of the attention model is , , Q , P and V are the queries corresponding to the attention functions ,key p Sum The matrix, d P represents the dimension of the matrix P, , and The corresponding matrices represent three learning weights respectively.

[0019] Preferably, determining the transmission representation of each transmission time slot according to each data representation, the future viewport and the acquired network quality information includes:

[0020] defining a state space according to a combination of a network bandwidth in the network quality information, data representations of the respective cubic regions, and the future viewport;

[0021] The features in the state space are respectively input into the action head and the value head in the pre-trained deep deterministic policy gradient network. In a certain state of the state space, an action that maximizes the reward is selected from the preset action space according to a preset reward function to determine the transmission representation of each transmission time slot.

[0022] Preferably, the state of time slot t in the state space ;

[0023] in, is the bandwidth throughput of time slot t, is the future viewport of user u at time slot t, H is the preset number of time slots, is the combination of data representations of each cubic region at time slot t, , K is the number of cubic areas in the volume scene, For time slot The eigenvector of the k-th cubic region below, , , and Respectively represent the k-th cubic area in the time slot The number of bits, Gaussian points and quality level transmitted in the previous time slot, Indicates the highest quality level, are the coordinates of the kth cubic region, represents the total number of Gaussian points that need to be transmitted in the k-th cubic area, Represents the total number of bits that need to be transmitted in the k-th cubic area.

[0024] Preferably, the reward function is ;

[0025] in, Status In Action The reward value, and are two hyperparameters, Indicates that the k-th cubic area will be The representation elements that appear in the user's field of view in a time slot, K is the number of cubic areas in the volume scene, Indicates the highest quality level, represents the total number of Gaussian points that need to be transmitted in the k-th cubic area, represents the total number of Gaussian points that need to be transmitted in the k-th cubic area at time slot t, Indicates that the kth cubic area is in the time slot The number of Gaussian points that have been transmitted in the previous time slots, Indicates time slot Select the transfer tile Level , is the time slot duration, is the delay time, , Indicates that the kth cubic area is in the time slot The number of bits transmitted in the previous time slot, represents the total number of bits that need to be transmitted in the k-th cubic area at time slot t, is the bandwidth throughput of time slot t.

[0026] Preferably, the deep deterministic policy gradient network training process specifically includes:

[0027] Construct a policy gradient network that interacts with the environment and a target network for auxiliary training;

[0028] Initialize the parameters of the two networks;

[0029] As the policy gradient network interacts with the environment, the cache is replayed. Synchronously store all records;

[0030] Each record is represented by , Is an application action In Status When the state transition function The conversion state of the sample in ;

[0031] Update the model parameters using the preset first parameter ;

[0032] Update the model parameters using the preset second parameter ;

[0033] Determine whether the current iteration step meets the preset termination condition;

[0034] If not, re-adopt the first parameter to update the model and update the parameters ; Use the second parameter to update the model update parameter ;

[0035] If so, the trained policy gradient network is output as the deep deterministic policy gradient network.

[0036] The embodiment of the present invention further provides a volume scene streaming transmission device based on 3DGS, the device comprising:

[0037] A preprocessing module, for dividing the volume scene into a number of non-overlapping cubic regions, and downsampling each cubic region into a data representation of a preset quality level;

[0038] A collaborative viewport prediction module, used to obtain viewport information of a terminal device and predict future viewports in combination with historical prior information extracted from the acquired historical viewport sequence;

[0039] an adaptive module, configured to determine a transmission representation of each transmission time slot according to each data representation, the future viewport, and the acquired network quality information;

[0040] The scene construction module is used to transmit the transmission representation to the user's terminal device to reconstruct the 3DGS scene.

[0041] An embodiment of the present invention also provides a volumetric scene streaming transmission device based on 3DGS, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements a volumetric scene streaming transmission method based on 3DGS as described in any of the above embodiments.

[0042] An embodiment of the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the 3DGS-based volumetric scene streaming method as described in any of the above embodiments.

[0043] The present invention provides a volume scene streaming transmission method, device and medium based on 3DGS, which divides the volume scene into a number of non-overlapping cubic areas, and downsamples each cubic area to a data representation of a preset quality level; obtains the viewport information of the terminal device, and predicts the future viewport in combination with the historical prior information extracted from the acquired historical viewport sequence; determines the transmission representation of each transmission time slot according to each data representation, the future viewport and the acquired network quality information; transmits the transmission representation to the user's terminal device to reconstruct the 3DGS scene. The application scheme can realize the prediction of the user's future behavior, solve the spatial variability problem of the state space and the action space, and improve the streaming transmission performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of a volume scene streaming transmission method based on 3DGS provided by an embodiment of the present invention;

[0045] Figure 2 is a structural schematic diagram of a volume scene streaming transmission device based on 3DGS provided by an embodiment of the present invention;

[0046] Figure 3 It is another structural schematic diagram of a volume scene streaming transmission device based on 3DGS provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] See also Figure 1 , is a flow chart of a volume scene streaming method based on 3DGS provided in an embodiment of the present invention, the method comprising steps S1 to S4;

[0049] S1, dividing the volume scene into a number of non-overlapping cubic regions, and downsampling each cubic region into a data representation of a preset quality level;

[0050] S2, obtaining viewport information of the terminal device, and predicting the future viewport in combination with historical prior information extracted from the obtained historical viewport sequence;

[0051] S3, determining a transmission representation of each transmission time slot according to each data representation, the future viewport and the acquired network quality information;

[0052] S4, transmitting the transmission representation to the user's terminal device to reconstruct the 3DGS scene.

[0053] In the specific implementation of this embodiment, the GSStream system proposed in the present invention can be decomposed into four independent parts when executing the 3DGS-based volume scene streaming method.

[0054] In the preprocessing stage, the volumetric scene is divided into non-overlapping cubic regions and downsampled into L representations of different quality levels.

[0055] GSStream predicts the user's future viewport based on the historical viewport sequence and user-specific feature embeddings. The predicted viewport is used to generate a tile quality level selection strategy. Then, the transmission representation of each transmission time slot is determined based on each data representation, the future viewport, and the obtained network quality information.

[0056] The server transmits according to the selected quality level to achieve progressive streaming.

[0057] When a user views a volumetric scene through a head-mounted display (HMD), the real-time viewport information will be concentrated in the Collaborative Viewport Prediction (CVP) module to construct a 3DGS scene.

[0058] This application proposes a volumetric scene streaming system based on 3DGS, which integrates the CVP module and the DBA module to better estimate the user's future behavior and solve the variability challenges of the state and action space in the bitrate adaptation problem.

[0059] In another embodiment of the present invention, the volumetric scene is divided into a plurality of non-overlapping cubic regions, and each cubic region is downsampled to a data representation of a preset quality level, and the method further includes:

[0060] Divide the volume scene into K non-overlapping cubic spaces as tiles;

[0061] Perform voxel grid downsampling on each tile to obtain data representations with L quality levels;

[0062] The data is represented as ; and Respectively represent k The first tile The number of bits and the number of Gaussians contained in the quality level representation, and Respectively represent two different quality levels.

[0063] In the specific implementation of this embodiment, in the preprocessing stage, the entire volume scene is divided into several non-overlapping cubic spaces (i.e., tiles), and then these tiles are downsampled to binary representations at multiple quality levels for further selective transmission. Due to the uneven distribution of 3D Gaussian bodies in 3D space, the number of points contained in each tile is variable, and the number of tiles that the volume scene can be divided into is also variable. Since each tile is essentially a 3D area, for the convenience of calculation, this article uses the center coordinates Represents tiles Assume that a volume scene is divided into a total of tiles, and perform voxel grid downsampling on each tile to obtain The quality level of the representation meets .

[0064] in, and Respectively represent tiles No. The number of bits and the number of Gaussians contained in the level representation. The preprocessed tile representation is selectively transmitted based on the future viewport sequence predicted by the CVP module and the output generated by the DBA module to ensure smooth transmission under bandwidth constraints while still providing a satisfying viewing experience for users.

[0065] In another embodiment provided by the present invention, obtaining viewport information of a terminal device and predicting a future viewport in combination with historical prior information extracted from the obtained historical viewport sequence include:

[0066] Get User u The viewport information of the terminal device as a specific feature;

[0067] Extract historical prior information from the historical viewport sequence, use a preset attention model to learn the attention of each user in the historical prior information, output the obtained attention to the preset cross attention model, and use the corresponding user's embedding vector e u As a query, learn the user u Attention among all users;

[0068] The specific features and users u The attention among all users is used as historical prior information to decode the deep viewport features and connect them to two separate multi-layer perceptrons to predict the future unknown and future rotation angles of the user's virtual camera as the future viewport respectively;

[0069] in, , , and represents the coordinates of the position of the virtual camera of user u at time slot t, and , and denote the yaw, pitch and roll angles of the virtual camera of user u at time slot t respectively. The attention function of the attention model is , , Q , P and Vare the queries corresponding to the attention functions ,key p Sum The matrix, d P represents the dimension of the matrix P, , and The corresponding matrices represent three learning weights respectively.

[0070] In the specific implementation of this embodiment, the proposed CVP module aims to improve the accuracy of viewport prediction by jointly utilizing user-specific features and historical information. Note that different users have different behavioral inertia when viewing volumetric scenes. However, current viewport prediction methods usually homogenize the viewport trajectories of different users, ignoring the connection between user-specific features and user behavior. To this end, the proposed network contains two main components: a collaborative prior information extraction (CPE) module that collaboratively learns user feature embeddings from multiple users, and a historical prior information extraction (HPE) module that learns from historical viewport sequences. The entire network is based on user ID and the corresponding historical viewport sequence As input. In symbolic representation, the user In time slot Viewport can be defined as a vector such as .in , and represents the position of the user's virtual camera, and , and Represent the yaw, pitch and roll angles of the user's virtual camera respectively. , the instant user viewport obtained from the HMD Focus on the client to form a historical viewport sequence, and then predict future viewports based on it.

[0071] Collaborative Prior Information Extraction Module: The CPE module aims to learn collaborative prior information from multiple users and contains two attention modules. The attention function of the attention module corresponds to the query ,key Sum ,like , , Q , P and V are the queries corresponding to the attention functions ,key p Sum The matrix, d P represents the dimension of the matrix P, , and The corresponding matrices represent three learning weights respectively.

[0072] Here we set up an embedding matrix , where each column Is a user A specific learnable feature embedding vector. The matrix First, it is fed into a criss-cross attention module as a query, with the historical viewport sequence as key and value, to learn the attention of each user in the time domain. The output is then connected to a second criss-cross attention module to embed the corresponding user As a query, learn the user Attention among all users.

[0073] Historical Prior Information Extraction Module: The HPE module aims to focus on the historical prior information extracted from the input historical viewport sequence, and contains a Multi-Layer Perceptron (MLP) and an iTransformer module. The MLP is used to project the viewport of each time slot into a higher-dimensional feature space for further learning. The iTransformer module has been shown to be state-of-the-art in the field of time series prediction. In the proposed network, iTransformer is used to predict future behavior information in the time domain.

[0074] Feature Fusion: Here, MLP is used to fuse the prior information learned from the CPE module and the HPE module. The outputs of the two modules are connected, and the deep viewport features are decoded using MLP and connected to two separate MLPs to predict the future position and future rotation angle of the user's virtual camera, respectively.

[0075] The proposed CVP module is trained on the collected user viewport trajectory dataset, and the loss function is the predicted viewport With the true value The mean absolute error (MAE) between .

[0076] According to a preset reward function, an action that maximizes the reward is selected from a preset action space, and a transmission representation of each transmission time slot is determined.

[0077] In another embodiment of the present invention, the state of time slot t in the state space is ;

[0078] in, is the bandwidth throughput of time slot t, is the future viewport of user u at time slot t, H is the preset number of time slots, is the combination of data representations of each cubic region at time slot t, , K is the number of cubic areas in the volume scene, For time slot The eigenvector of the k-th cubic region below, , , and Respectively represent the k-th cubic area in the time slot The number of bits, Gaussian points and quality level transmitted in the previous time slot, Indicates the highest quality level, are the coordinates of the kth cubic region, represents the total number of Gaussian points that need to be transmitted in the k-th cubic area, Represents the total number of bits that need to be transmitted in the k-th cubic area.

[0079] In the specific implementation of this embodiment, each time slot Observed state Defined as the current bandwidth throughput (expressed in the number of bits that can be transmitted per time slot), the predicted user viewport sequence And the characteristic information combination of tiles in the volume scene ,in represents the number of tiles in the volume scene, the state of time slot t in the state space .

[0080] in, It describes the time slot Lower tile For each tile , corresponding to Defined as: .

[0081] in, , and They represent the number of bits, number of Gaussian points and quality level that have been transmitted (or selected) in the previous time slot. Indicates the highest quality level.

[0082] The action space is defined as In each time slot, Indicates time slot Select the transfer tile Level GSStream greedily selects the action Select the representation to form a set, .

[0083] Each of these Indicates time slot Select the transfer tile Level , otherwise 0. Collection By greedily setting the highest Value is 1 until .

[0084] to ensure smooth progressive streaming. Also satisfied .

[0085] In another embodiment provided by the present invention, the reward function is ;

[0086] in, Status In Action The reward value, and are two hyperparameters, Indicates that the k-th cubic area will be The representation elements that appear in the user's field of view in a time slot, K is the number of cubic areas in the volume scene, Indicates the highest quality level, represents the total number of Gaussian points that need to be transmitted in the k-th cubic area, represents the total number of Gaussian points that need to be transmitted in the k-th cubic area at time slot t, Indicates that the kth cubic area is in the time slot The number of Gaussian points that have been transmitted in the previous time slots, Indicates time slot Select the transfer tile Level , is the time slot duration, is the delay time, , Indicates that the kth cubic area is in the time slot The number of bits transmitted in the previous time slot, represents the total number of bits that need to be transmitted in the k-th cubic area at time slot t, is the bandwidth throughput of time slot t.

[0087] In the specific implementation of this embodiment, the reward function is based on the state Evaluate each action In the design of the present invention, the DRL algorithm determines the set Indirectly participate in decision making. Therefore, the reward function With Collection Closely related, .

[0088] in, satisfy .

[0089] In the collection In the elements Represents the predicted viewport sequence , tiles In the future time slots, otherwise it will not appear in the user's field of view. and are two hyperparameters. In the reward function, the term It is a time slot The tiles to be incrementally transferred when The number of Gaussian points of is the number of remaining untransmitted Gaussian points. Therefore, The first term is the progressive transmission reward, which encourages the network to transmit tile representations that are closer to the user's field of view and have higher incremental content, while the second term is the latency penalty, where is the time slot duration, and .

[0090] Therefore, the state transition function can be expressed as .

[0091] In another embodiment provided by the present invention, the deep deterministic policy gradient network training process specifically includes:

[0092] Construct a policy gradient network that interacts with the environment and a target network for auxiliary training;

[0093] Initialize the parameters of the two networks;

[0094] As the policy gradient network interacts with the environment, the cache is replayed. Synchronously store all records;

[0095] Each record is represented by , Is an application action In Status When the state transition function The conversion state of the sample in ;

[0096] Update the model parameters using the preset first parameter ;

[0097] Update the model parameters using the preset second parameter ;

[0098] Determine whether the current iteration step meets the preset termination condition;

[0099] If not, re-adopt the first parameter to update the model and update the parameters ; Use the second parameter to update the model update parameter ;

[0100] If so, the trained policy gradient network is output as the deep deterministic policy gradient network.

[0101] In the specific implementation of this embodiment, the features extracted from the SFE module will be input into the Actor head and the Critic head respectively. The Actor head uses the SA and FP modules again to ensure that the Actor head estimates a tile-level preference score. , each Indicates time slot Transmission tiles Level The Critic head needs to estimate the action value associated with all tiles in the scene. Therefore, it only uses one SA module to perform global feature abstraction, and no longer uses FP modules to propagate features back to each tile.

[0102] The actual training process can be summarized as follows: two identical DDPG networks are constructed, represented as DDPG networks interacting with the environment and And the DDPG target network for auxiliary DRL training , and initialize the same parameters. When the DDPG network interacts with the environment, the replay cache All records are stored synchronously. Each record is represented by ,in Is an application action In Status Time The conversion state of the sampling in the sampling batch When training on The gradient is as follows: ;

[0103] in Indicates in status The following are the actions taken by the DDPG target network, represents the RL discount factor. At the same time, the policy gradient can also be written as .

[0104] To sum up, the pseudo code is as follows: is the soft update factor, is random Gaussian noise to encourage exploration.

[0105] enter: , , .

[0106] initialization ;

[0107] From the curtain cycle;

[0108] ;

[0109] From step cycle; ; ; ; End the loop;

[0110] Batch Sampling ;

[0111] according to renew ;

[0112] according to renew ;

[0113] ;

[0114] ;

[0115] The loop ends and outputs the trained policy gradient network as the deep deterministic policy gradient network.

[0116] In different volume scenes, the distribution of Gaussian points in 3D space is uneven, and their corresponding bounding boxes do not completely overlap, which is the reason for the variable number of tiles. This further leads to the variability of the state and action space in the MDP. In the DRL algorithm, the spatial variability problem is reflected in the fact that the input and output of the neural network are tensors of different dimensions.

[0117] One way to solve this problem is to use a neural network module that can handle tensors of different dimensions. For example, a straightforward approach is to use Transformer or attention modules to capture the features of each tile, since they However, there are still two problems. First, from the data structure point of view, each tile is represented as a vector , It is essentially a collection of unordered vectors of variable length. Therefore, The actual order of is a nuisance in feature extraction. Second, typical Transformer and attention modules cannot focus on the spatial relationship between tiles because they treat the spatial coordinates of each tile as the same as other attributes. Therefore, SA and FP modules are proposed to address the variability of the state space, and SFE module It is regarded as an unordered set of vectors, and the sampling and grouping layers of the SA module hierarchically aggregate features from the sampled tiles and their nearby tiles to form a high-order representation. Afterwards, the feature propagation mechanism ensures that these high-order features can be learned for each tile. The tile-level spatial feature is an unordered set, each item is a vector, representing the high-order embedding of the corresponding tile, which is a deep feature abstracted from its intrinsic features and its local and global relationships in the volumetric scene. Similarly, SA and FP modules are also used in the Actor head and Critic head to achieve better streaming performance.

[0118] This paper proposes for the first time a volumetric scene streaming system based on 3DGS, which integrates the CVP module and the DBA module to better estimate the user's future behavior and solve the challenge of variability in state and action space in the bitrate adaptation problem.

[0119] The present invention firstly constructs a user viewport trajectory dataset containing 15 diverse volumetric scenes by developing a user viewport data collection platform, and invites 32 subjects to watch each volumetric scene for 1 minute.

[0120] Experimental results demonstrate that the proposed GSStream system achieves significant improvements in both visual quality and network usage compared to existing state-of-the-art volumetric scene streaming systems.

[0121] In order to test the performance of the proposed GSStream system, the present invention constructed a user viewport trajectory collection platform for 3DGS streaming playback, and invited 32 subjects to collect user viewport trajectory data. Of the 15 volumetric scenes used, 8 were outdoor scenes and 7 were indoor scenes. Each subject was asked to watch each volumetric scene freely for 1 minute, resulting in a more enjoyable and realistic dataset with a total of more than 864,000 frames. Of the 15 3DGS scenes used, 13 were selected from the Tanks&Temples dataset, the Deep Blending dataset, and the Mip-NeRF 360 dataset. These scenes were trained according to the 3DGS scene reconstruction process provided by Kerbl et al. and watched by 32 subjects for 1 minute. The 32 subjects were between 18 and 25 years old, from multiple different industries, with 16 men and women. The size of the experimental area is set to m 2 In the experiment, the dataset is further split, with 12 scenes and their corresponding user viewport data constituting the training set and the rest constituting the test set.

[0122] In order to better simulate the visual performance of different volumetric scene streaming systems in actual transmission scenarios, bandwidth trajectory data captured in cellular networks are used. In addition, in order to evaluate the performance of each streaming system under different network conditions, 10 bandwidth trajectories are selected and each trajectory is linearly scaled to construct bandwidth conditions with average bandwidths of 40Mbps, 80Mbps, and 120Mbps. During the training phase, bandwidth conditions will be randomly selected to ensure the robustness of the streaming system under different network conditions.

[0123] The performance of the GSStream system is compared with the three existing state-of-the-art volumetric content streaming systems: the ViVo system, the Trans-RL system, and the CaV3 system, as shown in Table 1.

[0124] Table 1 Performance comparison table

[0125]

[0126] The GSStream system proposed in the present invention surpasses the three most advanced existing systems in most cases and achieves the best average viewport structural similarity (SSIM) value performance. The GSStream system proposed in the present invention can achieve higher temporal visual effects than the other three comparison systems under dynamic network conditions, and can make full use of redundant bandwidth to provide users with a higher visual quality experience.

[0127] In another embodiment provided by the present invention, see Figure 2 , is a schematic diagram of a volume scene streaming transmission device based on 3DGS provided by an embodiment of the present invention, the device comprising:

[0128] A preprocessing module, for dividing the volume scene into a number of non-overlapping cubic regions, and downsampling each cubic region into a data representation of a preset quality level;

[0129] A collaborative viewport prediction module, used to obtain viewport information of a terminal device and predict future viewports in combination with historical prior information extracted from the acquired historical viewport sequence;

[0130] an adaptive module, configured to determine a transmission representation of each transmission time slot according to each data representation, the future viewport, and the acquired network quality information;

[0131] The scene construction module is used to transmit the transmission representation to the user's terminal device to reconstruct the 3DGS scene.

[0132] The 3DGS-based volumetric scene streaming transmission device provided in this embodiment can execute all the steps and functions of the 3DGS-based volumetric scene streaming transmission method provided in any of the above embodiments, and the specific functions of the device are not described in detail here.

[0133] See also Figure 3 , is another structural diagram of a volumetric scene streaming transmission device based on 3DGS provided in an embodiment of the present invention. The volumetric scene streaming transmission device based on 3DGS includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, such as a volumetric scene streaming transmission program based on 3DGS. When the processor executes the computer program, the steps in each of the above-mentioned embodiments of a volumetric scene streaming transmission method based on 3DGS are implemented, such as Figure 1 Alternatively, the processor implements the functions of each module in the above-mentioned device embodiments when executing the computer program.

[0134] Exemplarily, the computer program may be divided into one or more modules, and the one or more modules are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the 3DGS-based volumetric scene streaming transmission device. For example, the computer program may be divided into various modules, and the specific functions of each module have been described in detail in a 3DGS-based volumetric scene streaming transmission method provided in any of the above embodiments, and the specific functions of the device will not be repeated here.

[0135] The volumetric scene streaming transmission device based on 3DGS can be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The volumetric scene streaming transmission device based on 3DGS can include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the schematic diagram is only an example of a volumetric scene streaming transmission device based on 3DGS, and does not constitute a limitation on a volumetric scene streaming transmission device based on 3DGS. It can include more or less components than shown in the figure, or combine certain components, or different components. For example, the volumetric scene streaming transmission device based on 3DGS can also include input and output devices, network access devices, buses, etc.

[0136] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the volumetric scene streaming transmission device based on 3DGS, and uses various interfaces and lines to connect the various parts of the entire volumetric scene streaming transmission device based on 3DGS.

[0137] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the volumetric scene streaming device based on 3DGS by running or executing the computer program and / or module stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0138] Wherein, if the module integrated in the volumetric scene streaming device based on 3DGS is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0139] It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A volume scene streaming method based on 3DGS, characterized in that: The method comprises: Divide the volumetric scene into a number of non-overlapping cubic regions, and downsample each cubic region to a data representation of a preset quality level; Obtaining viewport information of the terminal device, and predicting the future viewport in combination with historical prior information extracted from the acquired historical viewport sequence; Determine a transmission representation of each transmission time slot according to each data representation, the future viewport, and the acquired network quality information; Transmitting the transmission representation to a user's terminal device to reconstruct a 3DGS scene; Obtaining viewport information of the terminal device, and predicting the future viewport in combination with historical prior information extracted from the acquired historical viewport sequence, including; Get User u Viewport information of the terminal device as a specific feature; Extract historical prior information from the historical viewport sequence, use a preset attention model to learn the attention of each user in the historical prior information, output the obtained attention to the preset cross attention model, and use the corresponding user's embedding vector e u As a query, learn the user u Attention among all users; The specific features and users u The attention among all users is used as historical prior information to decode the deep viewport features and connect them to two separate multi-layer perceptrons to predict the future position and future rotation angle of the user's virtual camera as the future viewport respectively; in, , , and represents the coordinates of the position of the virtual camera of user u at time slot t, and , and denote the yaw, pitch and roll angles of the virtual camera of user u at time slot t respectively. The attention function of the attention model is , , Q , P and V are the queries corresponding to the attention functions ,key p Sum The matrix, d P represents the dimension of the matrix P, , and The corresponding matrices represent three learning weights respectively.

2. The volume scene streaming transmission method based on 3DGS according to claim 1, characterized in that: Dividing the volumetric scene into a plurality of non-overlapping cubic regions and downsampling each cubic region into a data representation of a preset quality level, the method further comprising: Divide the volume scene into K non-overlapping cubic spaces as tiles; Perform voxel grid downsampling on each tile to obtain data representations with L quality levels; The data is represented as ; and Respectively represent k The first tile The number of bits and the number of Gaussians contained in the quality level representation, and Respectively represent two different quality levels.

3. The volume scene streaming method based on 3DGS according to claim 1, characterized in that: Determining a transmission representation of each transmission time slot according to each data representation, the future viewport, and the acquired network quality information includes: defining a state space according to a combination of a network bandwidth in the network quality information, data representations of the respective cubic regions, and the future viewport; The features in the state space are respectively input into the action head and the value head in the pre-trained deep deterministic policy gradient network. In a certain state of the state space, an action that maximizes the reward is selected from the preset action space according to a preset reward function to determine the transmission representation of each transmission time slot.

4. The volume scene streaming transmission method based on 3DGS according to claim 3 is characterized in that: The state of time slot t in the state space ; in, is the bandwidth throughput of time slot t, is the future viewport of user u at time slot t, H is the preset number of time slots, is the combination of data representations of each cubic region at time slot t, , K is the number of cubic areas in the volume scene, For time slot The eigenvector of the k-th cubic region below, , , and Respectively represent the k-th cubic area in the time slot The number of bits, Gaussian points and quality level transmitted in the previous time slot, Indicates the highest quality level, are the coordinates of the kth cubic region, represents the total number of Gaussian points that need to be transmitted in the k-th cubic area, Represents the total number of bits that need to be transmitted in the k-th cubic area.

5. The volume scene streaming method based on 3DGS according to claim 3, characterized in that: The reward function is ; in, Status In Action The reward value, and are two hyperparameters, Indicates that the k-th cubic area will be The representation elements that appear in the user's field of view in a time slot, K is the number of cubic areas in the volume scene, Indicates the highest quality level, represents the total number of Gaussian points that need to be transmitted in the k-th cubic area, represents the total number of Gaussian points that need to be transmitted in the k-th cubic area at time slot t, Indicates that the kth cubic area is in the time slot The number of Gaussian points that have been transmitted in the previous time slots, Indicates time slot Select the transfer tile Level , is the time slot duration, is the delay time, , Indicates that the kth cubic area is in the time slot The number of bits transmitted in the previous time slot, represents the total number of bits that need to be transmitted in the k-th cubic area at time slot t, is the bandwidth throughput of time slot t.

6. The volume scene streaming transmission method based on 3DGS according to claim 3, characterized in that: The deep deterministic policy gradient network training process specifically includes: Construct a policy gradient network that interacts with the environment and a target network for auxiliary training; Initialize the parameters of the two networks; As the policy gradient network interacts with the environment, the cache is replayed. Synchronously store all records; Each record is represented by , Is an application action In Status When the state transition function The conversion state of the sample in Update the model parameters using the preset first parameter ; Update the model parameters using the preset second parameter ; Determine whether the current iteration step meets the preset termination condition; If not, re-adopt the first parameter to update the model and update the parameters ; Use the second parameter to update the model update parameter ; If so, the trained policy gradient network is output as the deep deterministic policy gradient network.

7. A volume scene streaming device based on 3DGS, characterized in that: The device comprises: A preprocessing module, for dividing the volume scene into a number of non-overlapping cubic regions, and downsampling each cubic region into a data representation of a preset quality level; A collaborative viewport prediction module, used to obtain viewport information of a terminal device and predict future viewports in combination with historical prior information extracted from the acquired historical viewport sequence; an adaptive module, configured to determine a transmission representation of each transmission time slot according to each data representation, the future viewport, and the acquired network quality information; A scene construction module, used for transmitting the transmission representation to a user's terminal device to reconstruct a 3DGS scene; The collaborative viewport prediction module is used for: Get User u Viewport information of the terminal device as a specific feature; Extract historical prior information from the historical viewport sequence, use a preset attention model to learn the attention of each user in the historical prior information, output the obtained attention to the preset cross attention model, and use the corresponding user's embedding vector e u As a query, learn the user u Attention among all users; The specific features and users u The attention among all users is used as historical prior information to decode the deep viewport features and connect them to two separate multi-layer perceptrons to predict the future position and future rotation angle of the user's virtual camera as the future viewport respectively; in, , , and represents the coordinates of the position of the virtual camera of user u at time slot t, and , and denote the yaw, pitch and roll angles of the virtual camera of user u at time slot t respectively. The attention function of the attention model is , , Q , P and V are the queries corresponding to the attention functions ,key p Sum The matrix, d P represents the dimension of the matrix P, , and The corresponding matrices represent three learning weights respectively.

8. A volume scene streaming device based on 3DGS, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the 3DGS-based volume scene streaming method as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the 3DGS-based volume scene streaming method according to any one of claims 1 to 6.

Images