Point cloud video transmission method and system

Abstract

The invention provides a point cloud video transmission method and system, and relates to the technical field of point cloud video semantic transmission, and the method comprises the steps: obtaining preprocessed 3D point cloud video data; a point cloud video semantic extraction and reconstruction backbone network is constructed, point cloud simplified semantic features in 3D point cloud video data are extracted at a sending end, the semantic features are expanded at a receiving end through a point cloud multi-branch semantic encoder, a point cloud semantic decoder and a branch exit judgment module after transmission, and a point cloud video is reconstructed. According to the method, geometric features of the point cloud are effectively extracted and reserved, higher point cloud reconstruction quality is achieved under the condition of the same compression ratio, and the optimal branch is adaptively selected for semantic transmission in combination with a branch network optimization model architecture, so that computing resources and bandwidth occupation are effectively saved, and the reconstruction efficiency is improved. The multi-model training and deployment cost and the model switching and reasoning delay are reduced, and the user experience is optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of semantic transmission of point cloud videos, and particularly to a method and system for transmitting point cloud videos. Background Art

[0002] As a new form of multimedia data, point cloud videos contain geometric information of three-dimensional coordinates in space and attribute information such as color, normal vector, and reflection intensity. Compared with traditional 2D videos and 360° panoramic videos, they can provide users with a 6-degree-of-freedom immersive experience. However, the high-dimensional data characteristics of point cloud videos also result in their data volume far exceeding that of traditional videos, which poses a severe challenge to the network channel transmission capacity under existing conditions. Therefore, research on key technologies such as compression coding and transmission optimization of point cloud videos has important theoretical significance and application value for providing high-quality point cloud video transmission services under existing network infrastructure conditions. Currently, this field has become a research hotspot in related technology research and standardization fields.

[0003] In recent years, the rise of artificial intelligence technology has provided a technical basis for communication systems to process semantic information. As a new type of intelligent communication paradigm, semantic communication can advance and transmit the semantic feature information of point cloud videos, achieving data compression while retaining video content. Users reconstruct and restore the original video at the receiving end based on the received semantic feature information, which can break through the performance bottleneck of traditional transmission methods.

[0004] Wang Jin et al. from Beijing University of Technology disclosed a point cloud geometry compression method based on density-constrained adaptive graph convolution (publication number: CN116437110A), which can achieve better rate-distortion optimization in point cloud data compression and obtain reconstructed point clouds with better quality in terms of geometric detail performance. This research mainly includes an encoder, an entropy encoder, and a decoder. First, an adaptive graph convolution module with density constraints is constructed at the encoding end to advance the point cloud geometric features, thereby effectively retaining the local geometric information of the point cloud. Then, an entropy encoder with a hyperprior is used to compress and encode the point cloud geometric features. Finally, the point cloud is reconstructed layer by layer at the decoding end to restore the geometric details of the point cloud. Adding local density loss constraints helps to maintain the local geometric consistency between the reconstructed point cloud and the original point cloud. This research only considers optimizing the reconstruction quality at a certain compression ratio for 3D point cloud compression, ignoring the computational efficiency and parameter size of the model.

[0005] Qiao Xiuquan et al. from Beijing University of Posts and Telecommunications proposed an AI-driven real-time point cloud video transmission method and system (Publication No.: CN113810736A). For point cloud video data, a hierarchical feature extraction module is used to determine the key point cloud features in the point cloud video frame. At the receiving end, a point cloud generation module based on a generative adversarial network is used to expand and reconstruct the feature matrix to obtain point cloud information similar to the original input point cloud. Multiple candidate models are trained to transmit feature matrices of different sizes to achieve different compression ratios. An adaptive matcher senses the current network condition and selects a suitable model for the transmission task to balance the transmission time and reconstruction quality. Although this transmission method incorporates bandwidth into the end-to-end system design, at the encoding end, the computationally intensive task of farthest point sampling (FPS) is used for downsampling, which causes a relatively high processing delay. Moreover, PointNet++ is used for feature extraction, which is an independent point-by-point processing method to capture local features of the point cloud, resulting in some loss of geometric information. In addition, the model lacks a simple and effective control over the semantic feature size, and network parameters need to be adjusted and retrained, which consumes a large amount of computing resources and storage space. Summary of the Invention

[0006] In view of the above-mentioned disadvantages of the prior art, the purpose of the present invention is to provide a point cloud video transmission method and system, which is used to solve the problem that in the prior art, the compression of 3D point clouds only considers optimizing the reconstruction quality at a certain compression ratio, while ignoring the computational efficiency and parameter size of the model.

[0007] To achieve the above purpose and other related purposes, the present invention provides a point cloud video transmission method, including the following steps: obtaining preprocessed 3D point cloud video data; constructing a backbone network for point cloud video semantic extraction and reconstruction, extracting the point cloud simplified semantic features in the 3D point cloud video data at the sending end, and after transmission, expanding the semantic features at the receiving end through a point cloud multi-branch semantic encoder, a point cloud semantic decoder, and a branch exit decision module to reconstruct the point cloud video; constructing a joint loss function L based on the weighted sum of the chamfer distances between the reconstructed point cloud videos output by all branches and the original input point cloud unity ; on the basis of the backbone network, combining the branch neural network architecture, constructing a point cloud video multi-branch semantic extraction and reconstruction network with variable feature sizes through a point cloud video semantic extraction and reconstruction module; using the point cloud video containing various 3D point cloud models as the training set for the constructed point cloud video multi-branch semantic extraction and reconstruction network with variable feature sizes, and according to the joint loss function L unityPerform end-to-end training on the branch network; construct a branch exit decision module, set a reward function and sense the current network bandwidth status, traverse all branch networks to calculate the reward function value according to the environmental parameter variables, and adaptively select the branch network with the largest reward function value to perform the transmission task; according to the transmission task, split the end-to-end trained point cloud video multi-branch semantic extraction and reconstruction transmission network into a multi-branch semantic extraction module and a semantic reconstruction module with corresponding complexity, and deploy them to the edge server near the sender and the user point cloud video viewing device as the receiver respectively. Sense the real-time network bandwidth, the amount of transmitted semantic feature data, and the model inference time through the branch exit decision module, calculate the reward function value of each branch, and select the branch with the largest reward function value for semantic feature transmission.

[0008] In an embodiment of the present invention, obtaining preprocessed 3D point cloud video data includes: randomly selecting a number of points in each point cloud model as seed coordinates; querying a number of points around them through the KNN algorithm, and defining the seed coordinates and the points queried around them through the KNN algorithm as a fragment; normalizing the coordinates of the points in the fragment into a unit sphere; using the fragment with a fixed number of points whose coordinates are normalized into a unit sphere as a sample of the training set to obtain preprocessed 3D point cloud video data.

[0009] In an embodiment of the present invention, constructing a point cloud video semantic extraction and reconstruction backbone network, extracting the point cloud concise semantic features in the 3D point cloud video data at the sender, and expanding the semantic features and reconstructing the point cloud video through the point cloud multi-branch semantic encoder, the point cloud semantic decoder and the branch exit decision module at the receiver, including: using a stacked multi-path dense connection graph convolutional module to extract point-by-point point cloud semantic features layer by layer; using an attention sampling module to downsample the extracted point cloud semantic features, adaptively learn the concise semantic features and attach them to the key points; repeating the above steps to gradually expand the receptive field and aggregate the point cloud multi-scale semantic features layer by layer, and finally extract the concise semantic feature matrix for transmission; using a semantic feature reconstruction module to expand and reconstruct the coordinates of the semantic features corresponding to the received concise semantic feature matrix to restore the original point cloud information.

[0010] In an embodiment of the present invention, constructing a point cloud video semantic extraction and reconstruction backbone network, extracting the point cloud concise semantic features in the 3D point cloud video data at the sender, and expanding the semantic features and reconstructing the point cloud video through the point cloud multi-branch semantic encoder, the point cloud semantic decoder and the branch exit decision module at the receiver, including: using a multi-path dense connection-based graph convolutional module (MDGCN) to extract point-by-point semantic features, with the input being the point cloud coordinates P and the feature F, and the output being the semantic features The extracted point-by-point semantic features are downsampled using the attention sampling module, key points and condensed semantic features are adaptively learned and appended to the key points to output a condensed semantic feature matrix Where r is the downsampling rate; repeat the above steps, regard each cycle as a semantic feature extraction layer, gradually expand the receptive field while aggregating the multi-scale semantic features of the point cloud layer by layer, and the semantic feature matrix F output by the l-th layer MDGCN module l ″ , with dimension N l ×C l (i.e. the final semantic feature), the input is the output N of the previous MDGCN layer l-1 ×C l-1 , to extract the semantic feature matrix for branch k for transmission Where K represents the number of branches; the semantic decoding module expands and reconstructs the coordinates of the received semantic feature matrix, first passing through a feature expansion module based on graph convolution point cloud upsampling to generate a more diverse number of point clouds and feature dimensions, so as to reconstruct the original point cloud video P through a coordinate reconstruction module.

[0011] In one embodiment of the present invention, a stacked multi-path densely connected graph convolution module is used to extract point cloud semantic features layer by layer, including: mapping the coordinates of the input point cloud (N, 3) to the potential feature space through a layer of graph convolution network, using the KNN algorithm to find the k nearest neighbor points of each point in the input point cloud, and constructing a local directed graph in and Represent the vertex set and edge set of the local directed graph respectively; i Considered as the center point, is the neighborhood point set in the local graph, point is the three-dimensional coordinates (x ij ,y ij ,z ij ), k is the number of neighboring points, and correspondingly, f ij is the feature of the jth neighbor in the local graph, and calculates the center point p i With neighboring point p ij The coordinate difference (p ij -p i ) or characteristic difference (f ij -f i ), get the local graph coordinates; put the center point p i and the coordinate difference p ij -p i A layer of MLP is used as the edge function to add the captured global features and local features, and the edge features of each point are obtained through the ReLU activation function layer and the maximum pooling aggregation: im =max j:(i,j)∈εReLU(θ m ·(p ij -p i )+φ m ·p i ); Embed the 3D coordinates of the input point cloud into the latent feature space F i with a dimension of (N × C i ), where C i represents the feature dimension embedded by the graph convolutional network; Pass the feature F i embedded by the graph convolutional network through a two-dimensional convolutional layer to compress the dimension to C' i . Pass the compressed feature through two parallel densely connected graph convolutional modules respectively. Each densely connected graph convolutional block consists of three layers of densely connected dilated graph convolutions, defined by the kernel size k, dilation rate d, and the number of output channels c. The kernel sizes of the two densely connected graph convolutional blocks are both set to 8. The dilation rate of the first densely connected graph convolutional block is 1, and the dilation rate of the second densely connected graph convolutional block is 2. The number of output channels is the same and is set to C d to calculate the local semantic features F d1 and F d2 under different receptive fields, and add a max pooling module to calculate the maximum value of each point feature dimension of the feature F i as the global semantic feature F g . Finally, concatenate F d1 , F d2 and F g to obtain the multi-scale semantic feature F'; Pass the multi-scale semantic feature F' extracted by the multi-path densely connected graph convolutional module through a fully connected layer to compress the feature dimension to obtain the feature F' m with a dimension of N l-1 × N l . After Softmax normalization operation and matrix transpose, obtain the final attention weight matrix F attention mask with a dimension of N l × N l-1 . Finally, multiply it with the semantic feature F' to obtain the output refined point cloud semantic feature matrix Repeat the steps of calculating the multi-scale semantic feature F' and the refined point cloud semantic feature matrix . By stacking multi-path densely connected graph convolutional modules and attention sampling modules with different numbers of layers, gradually expand the receptive field while aggregating the multi-scale semantic features of the point cloud layer by layer, and extract the semantic feature matrix corresponding to the refined point cloud features for transmission in the corresponding branch

[0012] In an embodiment of the present invention, based on the backbone network, combined with the branch neural network architecture, a point cloud video multi-branch semantic extraction and reconstruction network with variable feature sizes is constructed through the point cloud video semantic extraction and reconstruction module, including:

[0013] Combined with the branch neural network architecture, multi-path dense graph convolution modules and attention sampling modules with different numbers of layers are constructed through the point cloud video semantic extraction and reconstruction module, and a multi-branch structure with an entry point and multiple exit points is established;

[0014] Among them, each branch structure is a subset of the entire network, and contains a partial semantic extraction module that does not overlap with other branches to output a semantic feature matrix with variable sizes; each branch structure is provided with a semantic reconstruction module with corresponding complexity to reconstruct point cloud data of different qualities.

[0015] In an embodiment of the present invention, the weighted sum of the chamfer distances between the reconstructed point clouds output by all branches and the original input point cloud is: Where represents the point cloud reconstructed by the k-th branch, P represents the original input point cloud, θ represents the network parameters, ω k is the weight of the loss function of each branch; based on the joint loss function L of the training branch neural network unity is: Among them, L() represents the point-to-point chamfer distance loss between the reconstructed point cloud and the original point cloud, p and q are respectively and points in P, and this formula represents the sum of the minimum distances from any point p in to P, plus the sum of the minimum distances from any point q in P to the minimum distance sum of.

[0016] In an embodiment of the present invention, a branch exit decision module is constructed, a reward function is set and the current network bandwidth status is sensed, all branch networks are traversed to calculate the reward function value according to the environmental parameter variables, and the branch network with the largest reward function value is adaptively selected for the transmission task, including: constructing a branch exit decision module, constructing a reward function according to the semantic feature size, bandwidth status and semantic feature reconstruction time; sensing the real-time network bandwidth through the reward function to calculate the reward function, and selecting the branch with the largest reward function value for transmission; where the reward function is set as Where M k is the size of the semantic feature matrix of the k-th branch, B is the bandwidth status, T k is the reconstruction delay of the k-th branch, α 1 、α 2 、α 3 are the coefficients of each item respectively.

[0017] In an embodiment of the present invention, at the sending end, the point cloud concise semantic features in the 3D point cloud video data are extracted. After transmission, at the receiving end, the semantic features are extended through a point cloud multi-branch semantic encoder, a point cloud semantic decoder, and a branch exit decision module to reconstruct the point cloud video, including: the semantic feature matrix F corresponding to the received concise point cloud features ″ Perform graph feature extraction through one layer of GCN network processing, and extend the semantic features to obtain embedded graph features The embedded graph features F obtained by extending the semantic features e After a periodic rearrangement operation to rearrange the embedded graph features Re-arranged into features where N 0 = rN l , r is the downsampling factor, which encourages the model to learn the point distribution from the latent space;

[0018] The re-arranged features F' e Pass through a coordinate reconstruction unit composed of two layers of MLP to regress and fit the original coordinates of the point cloud to obtain the reconstructed point cloud video (N 0 ×3).

[0019] To achieve the above and other related purposes, the present invention also provides a point cloud video transmission system, an acquisition module for acquiring preprocessed 3D point cloud video data; a reconstruction module for constructing a point cloud video semantic extraction and reconstruction backbone network, extracting the point cloud concise semantic features in the 3D point cloud video data at the sending end, and after transmission, extending the semantic features through a point cloud multi-branch semantic encoder, a point cloud semantic decoder, and a branch exit decision module at the receiving end to reconstruct the point cloud video; a loss construction module for constructing a joint loss function L based on the weighted sum of the chamfer distances between the reconstructed point cloud videos output by all branches and the original input point cloud unity ; a network construction module for, on the basis of the backbone network, combining a branch neural network architecture, and constructing a point cloud video multi-branch semantic extraction and reconstruction network with variable feature sizes through a point cloud video semantic extraction and reconstruction module; a training module for using the constructed point cloud video multi-branch semantic extraction and reconstruction network with variable feature sizes, using a point cloud video containing various 3D point cloud models as a training set, according to the joint loss function L unityPerform end-to-end training on the branch network; a task transmission module, which is used to construct a branch exit decision module, set a reward function and sense the current network bandwidth status, traverse all branch networks to calculate the reward function value according to the environmental parameter variables, and adaptively select the branch network with the largest reward function value to perform the transmission task; and a selection transmission module, which is used to split the point cloud video multi-branch semantic extraction and reconstruction transmission network completed by end-to-end training into a multi-branch semantic extraction module and a semantic reconstruction module with corresponding complexity according to the transmission task, and deploy them to the edge server near the sender and the user point cloud video viewing device as the receiver respectively. The real-time network bandwidth, the amount of transmitted semantic feature data, and the model inference time are sensed through the branch exit decision module, and the reward function value of each branch is calculated, and the branch with the largest reward function value is selected for semantic feature transmission.

[0020] To achieve the above and other related purposes, the present invention further provides an electronic device, which includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device implements the aforementioned point cloud video transmission method.

[0021] To achieve the above and other related purposes, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by the processor of the computer, the computer executes the aforementioned point cloud video transmission method.

[0022] As described above, a point cloud video transmission method and system of the present invention have the following beneficial effects: by constructing a point cloud video semantic extraction and reconstruction network based on a stacked multi-path densely connected graph convolution module, and by transmitting the refined semantic features of the point cloud video, better reconstruction quality can be achieved under the same compression ratio compared with traditional methods. The present invention also considers the problem of excessive computational complexity caused by the stacked depth graph convolution network and the deployment of the model in a bandwidth-fluctuating environment. It adopts a branch neural network architecture to construct a point cloud video multi-branch semantic extraction and reconstruction network, reduces the number of model parameters for deployment on mobile devices, reduces the multi-model deployment cost and switching delay. And a QoE model based on semantic extraction and reconstruction transmission is constructed to provide an adaptive transmission control algorithm to optimize the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic flowchart of the point cloud video transmission method provided by the embodiment of the present invention.

[0024] Figure 2 It shows a flowchart of the point cloud video semantic transmission system provided by an embodiment of the present invention.

[0025] Figure 3It is shown as the structural diagram of the multi-branch semantic transmission network for point cloud video provided by an embodiment of the present invention.

[0026] Figure 4 It is shown as the structural block diagram of the point cloud video transmission system provided by an embodiment of the present invention.

[0027] Figure 5 It is shown as a schematic structural diagram of an electronic device provided by an embodiment of the present invention.

[0028] Description of component numbers

[0029] Electronic device 1; Point cloud video transmission system 11; Memory 12; Processor 13; Acquisition module 111; Reconstruction module 112; Loss construction module 113; Network construction module 114; Training module 115; Task transmission module 116; Selection transmission module 117. Detailed implementation manners

[0030] The following uses specific specific examples to illustrate the implementation manners of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0031] It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner. Therefore, only the components related to the present invention are shown in the diagrams, rather than being drawn according to the number, shape, and size of the components in actual implementation. The type, quantity, and ratio of each component in actual implementation can be arbitrarily changed, and the component layout type may also be more complex.

[0032] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0033] Please refer to Figure 1 , the present invention provides a point cloud video transmission method, including the following steps:

[0034] Step S10: Acquire preprocessed 3D point cloud video data;

[0035] Step S20: Construct a backbone network for point cloud video semantic extraction and reconstruction. Extract the point cloud concise semantic features from the 3D point cloud video data at the sending end. After transmission, expand the semantic features through the point cloud multi-branch semantic encoder, point cloud semantic decoder, and branch exit decision module at the receiving end, and reconstruct the point cloud video;

[0036] Step S30: Construct a joint loss function L based on the training branch neural network according to the weighted sum of the chamfer distances between the reconstructed point cloud videos output by all branches and the original input point cloud; unity ;

[0037] Step S40: Based on the backbone network, combined with the branch neural network architecture, construct a point cloud video multi-branch semantic extraction and reconstruction network with variable feature sizes through the point cloud video semantic extraction and reconstruction module;

[0038] Step S50: Use the point cloud video containing various 3D point cloud models as the training set for the constructed point cloud video multi-branch semantic extraction and reconstruction network with variable feature sizes. According to the joint loss function L unity Perform end-to-end training on the branch network;

[0039] Step S60: Construct a branch exit decision module, set a reward function, and sense the current network bandwidth status. Traverse all branch networks to calculate the reward function values according to the environmental parameter variables, and adaptively select the branch network with the largest reward function value to perform the transmission task;

[0040] Step S70: According to the transmission task, split the end-to-end trained point cloud video multi-branch semantic extraction and reconstruction transmission network into a multi-branch semantic extraction module and a semantic reconstruction module with corresponding complexity, and deploy them to the edge server near the sending end and the user point cloud video viewing device as the receiving end respectively. Sense the real-time network bandwidth, the amount of transmitted semantic feature data, and the model inference time through the branch exit decision module, calculate the reward function values of each branch, and select the branch with the largest reward function value for semantic feature transmission.

[0041] Figure 1 The flowchart of the point cloud video transmission method in an exemplary embodiment of the present application is shown, including steps S10 - S70. The technical solution of the present application will be elaborated in detail below in combination with Figure 1 and 2 ;

[0042] First, execute step S10 to obtain the preprocessed 3D point cloud video data.

[0043] The publicly available PU1K point cloud dataset is adopted, which contains 1,147 3D point cloud models, divided into 1,020 training models and 127 test models. In the training set, several points are randomly selected on the surface of each point cloud model as seed coordinates, and the KNN algorithm is used to query 256 points around the seed center point as a fragment. Then, the point set coordinates within the fragment are normalized into a unit sphere. In this embodiment, each fragment serves as a sample of the training set, that is, the input of the neural network can be represented as (256, 3).

[0044] In step S10, the preprocessed 3D point cloud video data is obtained, including:

[0045] Randomly select several points in each point cloud model as seed coordinates;

[0046] Query several points around it through the KNN algorithm,

[0047] Define the seed coordinates and several points queried around it through the KNN algorithm as a fragment;

[0048] Normalize the coordinates of the points within the fragment into a unit sphere;

[0049] Take the fragment with a fixed number of points whose coordinates are normalized into a unit sphere as a sample of the training set to obtain the preprocessed 3D point cloud video data.

[0050] Next, step S20 is executed to construct a backbone network for point cloud video semantic extraction and reconstruction. At the sending end, the point cloud concise semantic features in the 3D point cloud video data are extracted, and after transmission, at the receiving end, the semantic features are extended through a point cloud multi-branch semantic encoder, a point cloud semantic decoder, and a branch exit decision module to reconstruct the point cloud video.

[0051] In the process of constructing a backbone network for variable feature size point cloud video geometric feature extraction and reconstruction based on a hierarchical structure, the point cloud is represented as a set of 3D points where p i represents the three-dimensional coordinates (x i , y i , z i ) of the i-th point, and N is the number of points in the point cloud. Correspondingly, the point cloud feature is represented as where f i represents the feature of the i-th point, and C is the feature dimension.

[0052] In step S20, a backbone network for point cloud video semantic extraction and reconstruction is constructed. At the sending end, the point cloud concise semantic features in the 3D point cloud video data are extracted, and after transmission, at the receiving end, the semantic features are extended through a point cloud multi-branch semantic encoder, a point cloud semantic decoder, and a branch exit decision module to reconstruct the point cloud video, including:

[0053] Use a stacked multi - path dense - connected graph convolutional module to extract point - by - point point - cloud semantic features layer by layer;

[0054] Use an attention sampling module to downsample the extracted point - cloud semantic features, adaptively learn and refine semantic features and attach them to key points;

[0055] Repeat the above steps, gradually expand the receptive field while aggregating point - cloud multi - scale semantic features layer by layer, and finally extract a refined semantic feature matrix for transmission;

[0056] Use a semantic feature reconstruction module to expand and coordinate - reconstruct the semantic features corresponding to the received refined semantic feature matrix to restore the original point - cloud information.

[0057] During the execution of step S20, first, use a multi - path dense - connected graph convolutional module to extract point - cloud multi - scale semantic features. The multi - path dense - connected graph convolutional module first performs graph construction and feature embedding on the input through a layer of edge graph convolution, and then compresses it through a bottleneck layer (single - layer MLP) to reduce the computational amount of subsequent layers. The compressed features are passed to two parallel dense graph convolutional blocks with the same kernel size but different dilation rates, so as to extract geometric features with different receptive fields without losing resolution, and a parallel global pooling layer is added to extract global context information, thereby extracting multi - scale semantic information; in addition, each multi - path dense graph convolutional module also outputs the concatenation of two dense graph convolutional blocks and the global pooling layer, and finally extracts the multi - scale geometric features of the point cloud.

[0058] Next, use a learnable attention - based sampling structure to obtain a sampled subset and geometric features of the input point cloud, saving computational time; compress the captured point - cloud multi - scale geometric features through a layer of MLP, where the output feature dimension is an integer multiple of the downsampling of the number of point clouds. After matrix transposition and Softmax operation, obtain the normalized attention weights, perform matrix multiplication operations with the input point set and the feature matrix respectively, and finally concatenate them into a semantic feature matrix after attention sampling, and finally obtain a compact and accurate semantic feature representation of the input point cloud.

[0059] Repeat the above method, and use a residual architecture connection between different multi - path dense graph convolutional layers to extract increasingly compact point - cloud multi - scale semantic geometric features layer by layer, where the semantic feature matrix output by the last multi - path dense graph convolutional module is not concatenated with the sampled point set. Combine the branch - network architecture to construct a multi - branch encoder structure. Each branch outputs a feature matrix of different sizes by setting feature extraction layers with different numbers of layers, and this feature matrix is the data to be transmitted;

[0060] The point cloud semantic feature reconstruction module is used to expand and reconstruct the coordinates of the received semantic feature matrix. The feature expansion module first unifies the feature dimensions of the received semantic feature matrix through a bottleneck layer; then, through a graph convolutional network, the semantic features are channel-expanded, and after a periodic shuffling operation, the output of the channel-expanded features is rearranged, so as to decode the geometric space information from the semantic feature neighborhood, learn the geometric structure from the latent space, and finally, the expanded and rearranged semantic features are refitted into the original point cloud coordinates through a multi-layer perceptron, realizing the semantic feature extraction and reconstruction of point cloud data.

[0061] In step S20, a backbone network for point cloud video semantic extraction and reconstruction is constructed. The sparse semantic features of the 3D point cloud video data are extracted at the sending end, and after transmission, the semantic features are expanded at the receiving end through a point cloud multi-branch semantic encoder, a point cloud semantic decoder, and a branch exit decision module to reconstruct the point cloud video, which may further include:

[0062] Use a graph convolutional module based on multi-path dense connection (MDGCN) to extract per-point semantic features. The input is the point cloud coordinates P and the features F, and the output is the semantic features

[0063] Use an attention sampling module to downsample the extracted per-point semantic features, adaptively learn the key points and sparse semantic features and attach them to the key points to output a sparse semantic feature matrix where r is the downsampling rate;

[0064] Repeat the above steps, regard each cycle as a semantic feature extraction layer, gradually expand the receptive field while aggregating the multi-scale semantic features of the point cloud layer by layer. The semantic feature matrix F output by the l-th layer MDGCN module l ″ , with the dimension of N l ×C l (that is, the final semantic features), and the input is the output N of the previous layer MDGCN l-1 ×C l-1 to extract the semantic feature matrix for transmission in branch k where K represents the number of branches;

[0065] The semantic decoding module expands and reconstructs the received semantic feature matrix. First, through a feature expansion module based on graph convolutional point cloud upsampling, a more diverse number of point clouds and feature dimensions are generated to reconstruct the original point cloud video P through the coordinate reconstruction module.

[0066] In this embodiment, the point set is input into the multi-path dense graph convolution module for semantic feature extraction. The graph convolution networks adopted in this embodiment are all calculated by the following method: First, the input point set is subjected to graph feature embedding through a layer of graph convolution network. For the input point set P = (N 0 , 3), the k nearest neighbor points of each point are found using the KNN algorithm, and a local directed graph is constructed using the spatial neighborhood of each point where and represent the vertex set and edge set of the local directed graph respectively. Regarding p i as the center point, is the neighborhood point set in the local graph, the point is the three-dimensional coordinate (x ij , y ij , z ij ) of the j-th neighbor point in the local graph, and k is the number of neighbor points. Correspondingly, f ij is the feature of the j-th neighbor point in the local graph. Calculate the coordinate difference (p i -p ij ) or feature difference (f ij -f i ) between the center point p ij and the neighbor point p i ) to obtain the local graph coordinates. Pass the center point p i and the coordinate difference p ij -p i through the MLP respectively as edge functions to add the captured global features and local features, and obtain the edge features of each point through the ReLU activation function layer and max pooling aggregation, that is, p im = max j:(i,j)∈ε (ReLU(θ m ·(p ij -p i ) + φ m ·p i ))). Finally, the 3D coordinates of the input point cloud are embedded into the latent feature space F i , and its dimension is (N 0 ×C i ), where C i represents the feature dimension embedded by the graph convolution network.

[0067] In one embodiment, a stacked multi-path dense connection graph convolution module is used to extract the per-point point cloud semantic features layer by layer, including:

[0068] Map the coordinates of the input point cloud (N, 3) to the latent feature space through a layer of graph convolution network. Use the KNN algorithm to find the k nearest neighbor points of each point in the input point cloud and construct a local directed graph where and respectively represent the vertex set and edge set of the local directed graph;

[0069] Regarding p i as the center point, is the neighborhood point set in the local graph, and the point is the three-dimensional coordinate (x ij , y ij , z ij ) of the j-th neighboring point in the local graph. k is the number of neighboring points. Correspondingly, f ij is the feature of the j-th neighboring point in the local graph. Calculate the coordinate difference (p i -p ij ) or feature difference (f ij -f i ) between the center point p ij and the neighboring point p i ) to obtain the local graph coordinates;

[0070] Regarding the center point p i and the coordinate difference p ij -p i , respectively use a one-layer MLP as the edge function to add the captured global features and local features, and obtain the edge features of each point through the ReLU activation function layer and max pooling aggregation: p im = max j:(i,j)∈ε ReLU(θ m ·(p ij -p i ) + φ m ·p i );

[0071] Embed the 3D coordinates of the input point cloud into the latent feature space F i , whose dimension is (N×C i ), and C i represents the feature dimension embedded by the graph convolutional network;

[0072] Regarding the feature F i embedded by the graph convolutional network, compress its dimension to C' i through a one-layer two-dimensional convolutional layer. Regarding the compressed feature, pass it through two parallel densely connected graph convolutional modules respectively. Each densely connected graph convolutional block consists of three layers of densely connected dilated graph convolutions, which are defined by the kernel size k, dilation rate d, and the number of output channels c. The kernel sizes of the two densely connected graph convolutional blocks are both set to 8. The dilation rate of the first densely connected graph convolutional block is 1, and the dilation rate of the second densely connected graph convolutional block is 2. The number of output channels is the same and is set to C d , so as to calculate the local semantic features F d1 and F d2 under different receptive fields, and add a max pooling module to calculate the feature Fi The maximum value of each point feature dimension is used as the global semantic feature F g , and finally F d1 , F d2 and F g are concatenated to obtain the multi-scale semantic feature F';

[0073] The multi-scale semantic feature F' extracted by the multi-path densely connected graph convolution module is compressed in feature dimension through a fully connected layer to obtain the feature F' m , whose dimension is N l-1 × N l , and the final attention weight matrix F attention mask , whose dimension is N l × N l-1 is obtained through Softmax normalization operation and matrix transpose, and finally it is multiplied by the semantic feature F' to obtain the output refined point cloud semantic feature matrix

[0074] Repeat the steps of calculating the multi-scale semantic feature F' and the refined point cloud semantic feature matrix By stacking multi-path dense graph convolution modules and attention sampling modules with different numbers of layers, while gradually expanding the receptive field, layer by layer aggregating the multi-scale semantic features of the point cloud, and extracting the semantic feature matrix corresponding to the refined point cloud features for transmission in the corresponding branches

[0075] In this embodiment, by first passing the embedded feature F i through a bottleneck layer for dimension compression to reduce the subsequent calculation amount, the compressed feature F' i is obtained. Then, the compressed feature F' i is respectively passed through two parallel densely connected dilated graph convolution modules to calculate the local semantic features F d1 and F d2 under different receptive fields, where each densely connected graph convolution block consists of three layers of densely connected dilated graph convolutions, defined by the kernel size k, dilation rate d, and output channel number c. Then, the compressed feature F' i passes through a global pooling module to extract the global semantic feature F g . Finally, the semantic local features F d1 and F d2 with two different receptive fields are concatenated with the global semantic feature F g , and a residual connection is constructed with the input feature to obtain the semantic feature F' output by one layer of MDGCN block, with a dimension of N 0 × C 1 .

[0076] During the process of inputting the features F' extracted by the multi-path dense connection graph convolution module into the attention sampling module for sampling and compression, first, the number of channels of F' is compressed to N through a fully connected layer l to obtain the feature F' m with the dimension of N 0 ×N 1 After matrix transposition and Softmax normalization operations, the attention weight matrix F attention mask is obtained, with the dimension of N 1 ×N 0 Multiply it with the semantic feature F' through matrix multiplication to finally obtain the point cloud semantic feature matrix output by one layer of MDGCN

[0077] Next, by repeatedly calculating the dimension of N 0 ×C 1 and the point cloud semantic feature matrix By stacking multi-path dense graph convolution modules and attention sampling modules with different numbers of layers, the receptive field is gradually expanded and the multi-scale semantic features of the point cloud are aggregated. After that, the semantic feature matrix F l ″ output by the l-th layer MDGCN module, with the dimension of N l ×C l (i.e., the final semantic feature), the input is the output N l-1 ×C l-1 of the previous layer of MDGCN

[0078] Subsequently, a point cloud semantic decoding module is constructed, including a semantic feature expansion module and a coordinate reconstruction module. First, the semantic feature expansion module will receive the point cloud semantic feature matrix F ″ and obtain the embedded graph feature through one layer of GCN network processing to achieve semantic feature expansion

[0079] Then, the expanded embedded graph feature F e After the periodic rearrangement operation (Periodic Shuffle, PS), the embedded graph feature is rearranged into where N 0 = rN l , r is the downsampling multiple. Finally, the original coordinates of the point cloud are obtained through two-layer MLP regression fitting, and the reconstructed point cloud (N 0 ×3) is obtained

[0080] Next, execute steps S30 and S40, and construct the joint loss function L of the neural network based on the training branch according to the weighted sum of the chamfer distances between the reconstructed point cloud videos output by all branches and the original input point cloud unity; Based on the backbone network, combined with the branch neural network architecture, a point cloud video multi-branch semantic extraction and reconstruction network with variable feature sizes is constructed through the point cloud video semantic extraction and reconstruction module.

[0081] After dimension N 0 ×C 1 and the forward and backward propagation of the corresponding steps of the reconstructed point cloud (N 0 ×3), calculate the chamfer distance CD between the reconstructed point cloud and the original point cloud as the loss function, update the weights of the network, and finally complete the training of the point cloud video semantic extraction and reconstruction backbone network.

[0082] In step S40, based on the backbone network, combined with the branch neural network architecture, a point cloud video multi-branch semantic extraction and reconstruction network with variable feature sizes is constructed through the point cloud video semantic extraction and reconstruction module, including:

[0083] Combined with the branch neural network architecture, multi-path dense graph convolution modules and attention sampling modules with different numbers of layers are constructed through the point cloud video semantic extraction and reconstruction module, and a multi-branch structure with one entry point and multiple exit points is established;

[0084] Among them, each branch structure is a subset of the entire network and contains a partial semantic extraction module that does not overlap with other branches to output a semantic feature matrix with variable sizes; a semantic reconstruction module with corresponding complexity is set for each branch structure to reconstruct point cloud data with different qualities.

[0085] In this embodiment, based on the backbone network, multi-path dense graph convolution semantic encoding modules with different numbers of layers are stacked, and at the same time, a semantic feature reconstruction module with matching complexity is constructed, so as to establish a multi-branch point cloud video semantic extraction and reconstruction network in combination with the branch network architecture, realizing a network structure with one entry point and multiple exit points. In this embodiment, three branches are constructed, and the number of layers of the MDGCN module in each branch is set to 2, 3, and 4 respectively, so as to output three semantic feature matrices with different sizes, and for the feature matrices with different sizes, the semantic decoding module at the receiving end sets semantic reconstruction modules with different complexities, and finally obtains reconstructed point clouds with different reconstruction qualities.

[0086] Next, execute step S50, use the point cloud video containing various 3D point cloud models as the training set for the constructed point cloud video multi-branch semantic extraction and reconstruction network with variable feature sizes, and according to the joint loss function L unity Perform end-to-end training on the branch network.

[0087] Perform end-to-end training on the constructed multi-branch semantic extraction and reconstruction model according to the point cloud video semantic extraction and reconstruction backbone network based on stacked multi-path dense connection graph convolution and the joint loss function. As Figure 3As shown in the figure, in this embodiment, in order to obtain different compression ratios and reconstruction effects under each branch, the sizes of the semantically extracted feature matrices in the middle are set to (16, 16), (8, 10), and (4, 8) respectively, and the weights of each branch are 1, 0.8, and 0.3 respectively. The training is carried out on the NVIDIA 2060Ti GPU using the TensorFlow framework for 100 epochs, with a batch size of 64, and the Adam optimizer is used to optimize the network parameters.

[0088] In step S50, the weighted sum of the chamfer distances between the reconstructed point clouds output by all branches and the original input point cloud is: Where represents the point cloud reconstructed by the k-th branch, P represents the original input point cloud, θ represents the network parameters, and ω k is the weight of the loss function of each branch;

[0089] Based on the joint loss function L of the training branch neural network unity is: Where L() represents the point-to-point chamfer distance loss between the reconstructed point cloud and the original point cloud, p and q are respectively and points in P, and this formula represents the sum of the minimum distances from any point p in to P, plus the sum of the minimum distances from any point q in P to the minimum distance sum.

[0090] In this embodiment, a joint optimization loss function of the training branch neural network is constructed, that is, the weighted sum of the chamfer distances (CD) between the reconstructed point clouds output by all branches and the original input point cloud: Where represents the point cloud reconstructed by the k-th branch, P represents the original input point cloud, θ represents the network parameters, and ω k is the weight of the loss function of each branch, and L() represents the point-to-point chamfer distance loss between the reconstructed point cloud on branch k and the original point cloud. Its calculation method is as follows, Where p and q are respectively and points in P, and this formula represents the sum of the minimum distances from any point p in to P, plus the sum of the minimum distances from any point q in P to the minimum distance sum.

[0091] Next, step S60 is executed to construct a branch exit decision module, set a reward function and sense the current network bandwidth status, traverse all branch networks to calculate the reward function value according to the environmental parameter variables, and adaptively select the branch network with the largest reward function value to perform the transmission task.

[0092] By setting up a branch exit decision module, the bandwidth status when the current branch semantic extraction module outputs the semantic feature matrix can be predicted. According to the semantic feature size, bandwidth status, and end-to-end semantic extraction and reconstruction time, a reward function is constructed to evaluate the selection of the exit point and make an exit decision. The reward function is set as where M k is the size of the semantic feature matrix of the k-th branch, B is the bandwidth status, and T k is the end-to-end inference time of the k-th branch, and α 1 、α 2 、α 3 are the coefficients of each term respectively. When the point cloud video is input into the network for processing, by sensing the bandwidth status, the values of the reward function are calculated by traversing all branches, and the branch with the largest reward function value is selected for the transmission task of feature extraction and reconstruction.

[0093] In step S60, a branch exit decision module is constructed, the reward function is set, and the current network bandwidth status is sensed. By traversing all branch networks, the values of the reward function are calculated according to the environmental parameter variables, and the branch network with the largest reward function value is adaptively selected for the transmission task, including:

[0094] Construct a branch exit decision module, and construct a reward function according to the semantic feature size, bandwidth status, and semantic feature reconstruction time;

[0095] Perceive the real-time network bandwidth through the reward function, calculate the reward function, and select the branch with the largest reward function value for transmission;

[0096] where the reward function is set as where M k is the size of the semantic feature matrix of the k-th branch, B is the bandwidth status, and T k is the reconstruction delay of the k-th branch, and α 1 、α 2 、α 3 are the coefficients of each term respectively.

[0097] As shown in Figure XX, at the sending end, the point cloud refined semantic features in the 3D point cloud video data are extracted. After transmission, at the receiving end, the semantic features are extended through the point cloud multi-branch semantic encoder, point cloud semantic decoder, and branch exit decision module to reconstruct the point cloud video, including:

[0098] The semantic feature matrix F corresponding to the received refined point cloud features ″ is processed through a layer of GCN network for graph feature extraction, and the semantic features are extended to obtain the embedded graph features

[0099] The embedded graph features F obtained by extending the semantic features eAfter a periodic rearrangement operation to embed graph features rearranged into features where N 0 = rN l , r is the downsampling factor, encouraging the model to learn the point distribution from the latent space.

[0100] The rearranged feature F' e passes through a coordinate reconstruction unit composed of two layers of MLP to regress and fit the original coordinates of the point cloud to obtain the reconstructed point cloud video (N 0 ×3).

[0101] Step S70: According to the transmission task, split the end-to-end trained point cloud video multi-branch semantic extraction and reconstruction transmission network into a multi-branch semantic extraction module and a semantic reconstruction module with corresponding complexity, and deploy them to the edge server near the sender and the user point cloud video viewing device as the receiver respectively. Sense the real-time network bandwidth, the amount of transmitted semantic feature data, and the model inference time through the branch exit decision module, calculate the reward function value of each branch, and select the branch with the largest reward function value for semantic feature transmission.

[0102] The end-to-end trained multi-branch semantic extraction and reconstruction network can be split into a multi-branch semantic feature extraction module and a semantic reconstruction module with corresponding complexity, and deployed to the edge server near the sender and the user point cloud video viewing device as the receiver respectively. Sense the real-time network bandwidth through the branch exit decision module and calculate the reward function value of each branch, and select the branch with the largest reward function value for transmission.

[0103] In a preferred embodiment provided by the present invention, the thrust stage of point cloud video transmission may include the following steps:

[0104] Process the point cloud video to be transmitted frame by frame. When a frame of point cloud data is input into the multi-branch semantic extraction and reconstruction network, first calculate the number of points contained in the current frame of point cloud data, so as to select an appropriate number of points as seed coordinates, and use each seed coordinate as the center to query 1024 surrounding neighborhood point sets through KNN as a fragment, so that all the fragments can cover the entire point cloud model of a video frame;

[0105] Sense the bandwidth status when the semantic feature matrix extraction is completed, calculate the reward function value of each branch under this bandwidth condition, and select the branch with the largest reward for the transmission task;

[0106] After dividing a frame of point cloud video model into several fragments, input all the fragments into the selected branch with the optimal reward value one by one for forward inference calculation, and finally obtain the semantic feature matrix of each extracted fragment;

[0107] The extracted semantic feature matrix is sent to the receiving end through wireless transmission. The user terminal selects a semantic reconstruction module that matches the sending end to perform feature expansion and coordinate reconstruction on the received semantic feature matrix one by one. Through the forward inference calculation described in the training stage, the reconstruction data of each fragment is obtained;

[0108] Add the reconstruction data of all fragments of a frame of point cloud video, and use the farthest point sampling method to evenly select the same number of points as the original input point cloud. Finally, the reconstructed point cloud video frame is obtained.

[0109] Please refer to FIG. 4. The present invention also provides a point cloud video transmission system 11, including an acquisition module 111 for acquiring preprocessed 3D point cloud video data; a reconstruction module 112 for constructing a backbone network for point cloud video semantic extraction and reconstruction, extracting the point cloud simplified semantic features in the 3D point cloud video data at the sending end, and expanding the semantic features through a point cloud multi-branch semantic encoder, a point cloud semantic decoder, and a branch exit decision module at the receiving end after transmission to reconstruct the point cloud video; a loss construction module 113 for constructing a joint loss function L based on the weighted sum of the chamfer distances between the reconstructed point cloud videos output by all branches and the original input point cloud respectively; unity a network construction module 114 for constructing a point cloud video multi-branch semantic extraction and reconstruction network with variable feature sizes through a point cloud video semantic extraction and reconstruction module on the basis of the backbone network and in combination with a branch neural network architecture; a training module 115 for using a point cloud video containing various 3D point cloud models as a training set to perform end-to-end training on the constructed point cloud video multi-branch semantic extraction and reconstruction network with variable feature sizes according to the joint loss function L unity ; a task transmission module 116 for constructing a branch exit decision module, setting a reward function and sensing the current network bandwidth status, traversing all branch networks to calculate the reward function value according to the environmental parameter variables, and adaptively selecting the branch network with the largest reward function value to perform the transmission task; and a selection transmission module 117 for splitting the end-to-end trained point cloud video multi-branch semantic extraction and reconstruction transmission network into a multi-branch semantic extraction module and a semantic reconstruction module with corresponding complexity according to the transmission task, and respectively deploying them on an edge server near the sending end and a user point cloud video viewing device serving as the receiving end. The branch exit decision module senses the real-time network bandwidth, the amount of transmitted semantic feature data, and the model inference time, calculates the reward function value of each branch, and selects the branch with the largest reward function value for semantic feature transmission.

[0110] It should be noted that the point cloud video transmission system 11 provided in the above embodiments and the point cloud video transmission method provided in the above embodiments belong to the same concept. The specific ways in which each module and unit perform operations have been described in detail in the method embodiments and will not be elaborated here. In practical applications, the point cloud video transmission system 11 provided in the above embodiments can, according to needs, allocate the above functions to different functional modules, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above. This is not limited here either.

[0111] Please refer to Figure 5 , the electronic device 1 may include a memory 12, a processor 13, and a bus, and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a point cloud video transmission program.

[0112] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (such as: SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. The memory 12 can be an internal storage unit of the electronic device 1 in some embodiments, such as the mobile hard disk of the electronic device 1. The memory 12 can also be an external storage device of the electronic device 1 in other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device 1. Further, the memory 12 can also include both the internal storage unit and the external storage device of the electronic device 1. The memory 12 can be used not only to store application software installed on the electronic device 1 and various types of data, such as the code for point cloud video transmission, etc., but also to temporarily store data that has been output or will be output.

[0113] The processor 13 can be composed of integrated circuits in some embodiments. For example, it can be composed of a single packaged integrated circuit, or can be composed of multiple integrated circuits with the same or different functions, including a combination of one or more Central Processing Units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control core (Control Unit) of the electronic device 1, connecting various components of the entire electronic device 1 through various interfaces and lines, and by running or executing programs or modules stored in the memory 12 (such as the point cloud video transmission program, etc.), and calling data stored in the memory 12, to execute various functions of the electronic device 1 and process data.

[0114] The processor 13 executes the operating system of the electronic device 1 and various installed application programs. The processor 13 executes the application programs to implement the steps in the above point cloud video transmission method.

[0115] Exemplarily, the computer program may be divided into one or more modules, and the one or more modules are stored in the memory 12 and executed by the processor 13 to complete this application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into an acquisition module 111, a reconstruction module 112, a loss construction module 113, a network construction module 114, a training module 115, a task transmission module 116, and a selection transmission module 117.

[0116] The above integrated unit implemented in the form of a software functional module can be stored in a computer-readable storage medium, and the computer-readable storage medium may be non-volatile or volatile. The above software functional modules are stored in a storage medium, including several instructions to enable a computer device (which may be a personal computer, a computer device, or a network device, etc.) or a processor to execute some functions of the point cloud video transmission method described in various embodiments of this application.

[0117] In summary, a point cloud video transmission method and system disclosed in the present invention extracts point cloud semantic features based on a point cloud multi-branch semantic encoder with stacked multi-path dense connection graph convolution and attention sampling, reconstructs the point cloud video by a point cloud semantic decoder, and adaptively selects the optimal branch to perform the transmission task through a branch exit decision module to sense the network bandwidth, which can more accurately extract and retain the geometric information of the original point cloud data, and perform lightweight point cloud reconstruction, achieving a higher compression ratio while obtaining a higher quality reconstructed point cloud. And by combining with the branch network, deployment and efficient transmission can be completed through one training without fine-tuning or retraining. It reduces the memory cost and model switching delay of multi-model deployment, and adaptively balances the transmission delay and reconstruction quality, thereby optimizing the user experience. Therefore, the present invention effectively overcomes various disadvantages in the prior art and has high industrial utilization value.

[0118] The above embodiments only illustrate the principles and effects of the present invention and are not used to limit the present invention. Any person familiar with this technology can modify or change the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or changes completed by those with ordinary knowledge in the technical field without departing from the spirit and technical idea disclosed by the present invention should still be covered by the claims of the present invention.

Claims

1. A point cloud video transmission method, characterized in that: The following steps are involved: Obtain preprocessed 3D point cloud video data; Construct a point cloud video semantic extraction and reconstruction backbone network to extract the simplified semantic features of the point cloud in the 3D point cloud video data at the sending end. After transmission, the semantic features are expanded at the receiving end through the point cloud multi-branch semantic encoder, point cloud semantic decoder and branch exit judgment module to reconstruct the point cloud video. According to the weighted sum of the chamfer distances between the reconstructed point cloud video output by all branches and the original input point cloud, a joint loss function L based on the training branch neural network is constructed. unity ; Based on the backbone network, combined with the branch neural network architecture, a point cloud video multi-branch semantic extraction and reconstruction network with variable feature size is constructed through the point cloud video semantic extraction and reconstruction module; The constructed point cloud video multi-branch semantic extraction and reconstruction network with variable feature size is used as the training set, and the point cloud videos containing various 3D point cloud models are used as the training set. According to the joint loss function L unity Train the branch network end-to-end; Build a branch exit judgment module, set the reward function and sense the current network bandwidth status, traverse all branch networks to calculate the reward function value according to the environmental parameter variables, and adaptively select the branch network with the largest reward function value for the transmission task; According to the transmission task, the end-to-end trained point cloud video multi-branch semantic extraction and reconstruction transmission network is split into a multi-branch semantic extraction module and a semantic reconstruction module of corresponding complexity, which are respectively deployed on the edge server close to the sender and the user's point cloud video viewing device as the receiver. The branch exit judgment module perceives the real-time network bandwidth, the amount of transmitted semantic feature data, and the model inference time, calculates the reward function value of each branch, and selects the branch with the largest reward function value for semantic feature transmission.

2. The point cloud video transmission method according to claim 1, characterized in that: Get the preprocessed 3D point cloud video data, including: Randomly select several points in each point cloud model as seed coordinates; Use the KNN algorithm to query several points around it. The seed coordinates and several points around it queried by the KNN algorithm are defined as a fragment; Normalize the coordinates of points within the fragment to a unit sphere; Each patch with a fixed number of points whose coordinates are normalized into a unit sphere is taken as a sample of the training set to obtain the preprocessed 3D point cloud video data.

3. The point cloud video transmission method according to claim 1, characterized in that: Construct a point cloud video semantic extraction and reconstruction backbone network to extract the simplified semantic features of the point cloud in the 3D point cloud video data at the sending end. After transmission, the semantic features are expanded at the receiving end through the point cloud multi-branch semantic encoder, point cloud semantic decoder and branch exit judgment module to reconstruct the point cloud video, including: Use a stacked multi-path densely connected graph convolution module to extract point-by-point semantic features of point clouds layer by layer; Use the attention sampling module to downsample the extracted point cloud semantic features, adaptively learn the refined semantic features and attach them to the key points; Repeat the above steps to gradually expand the receptive field while aggregating the multi-scale semantic features of the point cloud layer by layer, and finally extract a simplified semantic feature matrix for transmission; The semantic feature reconstruction module is used to expand and reconstruct the coordinates of the semantic features corresponding to the received simplified semantic feature matrix to restore the original point cloud information.

4. The point cloud video transmission method according to claim 3, characterized in that: Construct a point cloud video semantic extraction and reconstruction backbone network to extract the simplified semantic features of the point cloud in the 3D point cloud video data at the sending end. After transmission, the semantic features are expanded at the receiving end through the point cloud multi-branch semantic encoder, point cloud semantic decoder and branch exit judgment module to reconstruct the point cloud video, including: Use the multi-path dense connection based graph convolution module (MDGCN) to extract point-by-point semantic features. The input is the point cloud coordinates P and features F, and the output is the semantic features. The extracted point-by-point semantic features are downsampled using the attention sampling module, key points and condensed semantic features are adaptively learned and appended to the key points to output a condensed semantic feature matrix Where r is the downsampling rate; Repeat the above steps, treating each cycle as a semantic feature extraction layer, gradually expanding the receptive field while aggregating the multi-scale semantic features of the point cloud layer by layer. The semantic feature matrix F output by the l-th layer MDGCN module is l ″ , with dimension N l ×C l (i.e. the final semantic feature), the input is the output N of the previous MDGCN layer l-1 ×C l-1 , to extract the semantic feature matrix for branch k for transmission Where K represents the number of branches; The semantic decoding module expands and reconstructs the coordinates of the received semantic feature matrix. First, it passes through the feature expansion module based on graph convolution point cloud upsampling to generate a more diverse number of point clouds and feature dimensions, so as to reconstruct the original point cloud video P through the coordinate reconstruction module.

5. The point cloud video transmission method according to claim 3 or 4, characterized in that: The point cloud semantic features are extracted layer by layer using a stacked multi-path densely connected graph convolution module, including: The input point cloud (N, 3) is mapped to the potential feature space through a layer of graph convolutional network. The KNN algorithm is used to find the k nearest neighbor points of each point in the input point cloud to construct a local directed graph. in and Respectively represent the vertex set and edge set of a local directed graph; The p i Considered as the center point, is the neighborhood point set in the local graph, point is the three-dimensional coordinate (x ij ,y ij ,z ij ), k is the number of neighboring points, and correspondingly, f ij is the feature of the jth neighbor in the local graph, and calculates the center point p i With neighboring point p ij The coordinate difference (p ij -p i ) or characteristic difference (f ij -f i ), get the local graph coordinates; The center point p i and the coordinate difference p ij -p i A layer of MLP is used as the edge function to add the captured global features and local features, and the edge features of each point are obtained through the ReLU activation function layer and the maximum pooling aggregation: im =max j:(i,j)∈ εReLU(θ m ·(p ij -p i )+φ m ·p i ); Embed the 3D coordinates of the input point cloud into the latent feature space F i The dimension is (N×C i ), C i Represents the feature dimension embedded by the graph convolutional network; The feature F embedded by the graph convolutional network i After a two-dimensional convolution layer, the dimension is compressed to C′ i , the compressed features are passed through two parallel densely connected graph convolution modules respectively. Each densely connected graph convolution block consists of three layers of densely connected dilated graph convolutions, defined by the kernel size k, dilation rate d and the number of output channels c. The kernel size of the two dense graph convolution blocks is set to 8, the dilation rate of the first dense graph convolution block is 1, and the dilation rate of the second dense graph convolution block is 2. The number of output channels is the same and is set to C d , thereby calculating the local semantic features F under different receptive fields d1 and F d2 , and add a maximum pooling module to calculate the feature F i The maximum value of each point feature dimension is taken as the global semantic feature F g Finally, F d1 、F d2 and F g After concatenation, we get the multi-scale semantic feature F′; The multi-scale semantic feature F′ extracted by the multi-path densely connected graph convolution module is compressed through a fully connected layer to obtain the feature F′ m , whose dimension is N l-1 ×N l , after Softmax normalization and matrix transposition, the final attention weight matrix F is obtained attentionmask , whose dimension is N l ×N l-1 Finally, it is matrix-multiplied with the semantic feature F′ to obtain the output simplified point cloud semantic feature matrix Repeated calculation of multi-scale semantic features F′ and simplified point cloud semantic feature matrix By stacking different layers of multi-path dense graph convolution modules and attention sampling modules, the receptive field is gradually expanded while the multi-scale semantic features of the point cloud are aggregated layer by layer to extract the semantic feature matrix corresponding to the simplified point cloud features for transmission in the corresponding branch.

6. The point cloud video transmission method according to claim 1, characterized in that: Based on the backbone network and combined with the branch neural network architecture, a point cloud video multi-branch semantic extraction and reconstruction network with variable feature size is constructed through the point cloud video semantic extraction and reconstruction module, including: Combined with the branch neural network architecture, the point cloud video semantic extraction and reconstruction module is used to construct multi-path dense graph convolution modules and attention sampling modules with different numbers of layers, and a multi-branch structure with an entry point and multiple exit points is established; Among them, each branch structure is a subset of the entire network and contains some semantic extraction modules that do not overlap with other branches to output a semantic feature matrix of variable size; each branch structure is set with a semantic reconstruction module of corresponding complexity to reconstruct point cloud data of different qualities.

7. The point cloud video transmission method according to claim 1, characterized in that: The weighted sum of the chamfer distances between the reconstructed point clouds output by all branches and the original input point clouds is: in represents the point cloud reconstructed by the kth branch, P represents the original input point cloud, θ represents the network parameters, ω k is the weight of the loss function of each branch; Joint loss function L based on training branch neural network unity for: Among them, L() represents the point-to-point chamfer distance loss between the reconstructed point cloud and the original point cloud, and p and q are and points in P, the formula represents The sum of the minimum distances from any point p in P to P, plus the distance from any point q in P to The sum of the minimum distances.

8. The point cloud video transmission method according to claim 1, characterized in that: Construct a branch exit judgment module, set the reward function and sense the current network bandwidth status, traverse all branch networks to calculate the reward function value according to the environmental parameter variables, and adaptively select the branch network with the largest reward function value for the transmission task, including: Build a branch exit decision module and construct a reward function based on the semantic feature size, bandwidth status, and semantic feature reconstruction time; The reward function is perceived by the real-time network bandwidth to calculate the reward function, and the branch with the largest reward function value is selected for transmission; The reward function is set as Among them, M k is the semantic feature matrix size of the kth branch, B is the bandwidth state, T k is the reconstruction delay of the kth branch, and α1, α2, and α3 are the coefficients of each term respectively.

9. The point cloud video transmission method according to claim 1, characterized in that: The simplified semantic features of the point cloud in the 3D point cloud video data are extracted at the sending end. After transmission, the semantic features are expanded at the receiving end through the point cloud multi-branch semantic encoder, point cloud semantic decoder and branch exit judgment module to reconstruct the point cloud video, including: The semantic feature matrix F corresponding to the received simplified point cloud features ″ After a layer of GCN network processing, graph features are extracted and semantic features are expanded to obtain embedded graph features. The embedded graph feature F obtained by expanding the semantic feature e After periodic rearrangement, the embedded graph features Rearrange to Features Where N0 = rN l , r is the downsampling multiple, which encourages the model to learn point distribution from the latent space; The rearranged features F′ e After a coordinate reconstruction unit consisting of two layers of MLP, the original coordinates of the fitted point cloud are regressed to obtain a reconstructed point cloud video (N0×3).

10. A point cloud video transmission system, characterized in that: An acquisition module is used to acquire pre-processed 3D point cloud video data; The reconstruction module is used to build a point cloud video semantic extraction and reconstruction backbone network. It extracts the simplified semantic features of the point cloud in the 3D point cloud video data at the sending end. After transmission, the semantic features are expanded at the receiving end through the point cloud multi-branch semantic encoder, point cloud semantic decoder and branch exit judgment module to reconstruct the point cloud video. The loss construction module is used to construct a joint loss function L based on the training branch neural network according to the weighted sum of the chamfer distances between the reconstructed point cloud videos output by all branches and the original input point cloud. unity ; A network construction module is used to build a point cloud video multi-branch semantic extraction and reconstruction network with variable feature size through the point cloud video semantic extraction and reconstruction module based on the backbone network and in combination with a branch neural network architecture; The training module is used to construct a point cloud video multi-branch semantic extraction and reconstruction network with variable feature sizes, using point cloud videos containing various 3D point cloud models as training sets, and according to the joint loss function L unity Train the branch network end-to-end; The task transmission module is used to build a branch exit judgment module, set the reward function and sense the current network bandwidth status, traverse all branch networks to calculate the reward function value according to the environmental parameter variables, and adaptively select the branch network with the largest reward function value to perform the transmission task; as well as A transmission module is selected to split the end-to-end trained point cloud video multi-branch semantic extraction and reconstruction transmission network into a multi-branch semantic extraction module and a semantic reconstruction module of corresponding complexity according to the transmission task, and deploy them on the edge server close to the sending end and the user's point cloud video viewing device as the receiving end, respectively. The branch exit judgment module perceives the real-time network bandwidth, the amount of transmitted semantic feature data and the model inference time, calculates the reward function value of each branch, and selects the branch with the largest reward function value for semantic feature transmission.

Images