A semantic communication method, system, storage medium, and program product
By employing inverse 3D Gaussian sputtering rasterization and hierarchical semantic coding techniques, the problems of incompatibility between traditional 3D model compression methods and semantic segmentation models, as well as information loss in low signal-to-noise ratio channel environments, are solved, achieving efficient 3D model compression and reconstruction.
Patent Information
- Application Number
- CN202510635444.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-16
- Publication Date
- 2026-11-17
AI Technical Summary
Traditional 3D model compression methods are incompatible with semantic segmentation models in semantic communication scenarios, and are prone to losing key information in low signal-to-noise ratio channel environments, which limits their application potential.
By employing reverse 3D Gaussian sputtering rasterization technology, and through multi-view projection parameter optimization and hierarchical semantic coding, the compatibility between the 3D model and the semantic segmentation model is achieved, and the impact of key information loss is reduced in low signal-to-noise ratio channel environments.
It achieves compatibility between 3D model compression and semantic segmentation models, improves the quality of reconstructed models under low signal-to-noise ratio channel conditions, avoids visual distortion and structural incompleteness, reduces the amount of transmitted data, and improves transmission efficiency.
Smart Images

Figure CN123123584A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of semantic communication technology, and in particular to a semantic communication method, system, storage medium, and program product. Background Technology
[0002] While traditional 3D model compression methods have achieved some success in compression efficiency, their compression methods limit their application potential in semantic communication scenarios, especially in the application of emerging semantic slice model propagation (SeSM) technology.
[0003] On the one hand, the semantic segmentation model in SeSM relies on clear and accurate semantic boundaries to perform its tasks. However, compressed data files generated by traditional 3D model compression algorithms often strip away the resolvable semantic structure, making them incompatible with the semantic segmentation model. This prevents the compressed data from being further compressed by the semantic segmentation model in SeSM, hindering the semantic segmentation model's ability to complete its tasks and limiting transmission performance. On the other hand, when dealing with low signal-to-noise ratio channel environments, the non-semantic-oriented compression strategies of traditional 3D model compression methods often lead to the non-selective loss of key information. This lost key information may include details crucial for model reconstruction, limiting the application potential in semantic communication scenarios. Summary of the Invention
[0004] To address the problems existing in the prior art, embodiments of the present invention provide a semantic communication method, system, storage medium, and program product, which can achieve compatibility between 3D model compression and semantic segmentation models, and reduce the impact of key information loss on the quality of reconstructed models in low signal-to-noise ratio channel environments.
[0005] In a first aspect, embodiments of the present invention provide a semantic communication method applied at a transmitting end, comprising:
[0006] Based on the initialized multi-view projection parameters, a differentiable grating operation is performed on the three-dimensional Gaussian splash model to be transmitted to obtain a feature map; wherein, the initialized multi-view projection parameters are determined based on camera parameters;
[0007] The 3D model is reconstructed based on the feature map to obtain the reconstructed model;
[0008] Based on the reconstruction model and the three-dimensional Gaussian splash model, the multi-view projection parameters are optimized, and a multi-channel feature map is obtained based on the optimized multi-view projection parameters.
[0009] The multi-channel feature map is subjected to hierarchical semantic encoding to obtain compressed data, and the compressed data is transmitted to the receiving end.
[0010] As an improvement to the above scheme, before performing a differentiable grating operation on the three-dimensional Gaussian splash model to be transmitted based on the initialized multi-view projection parameters to obtain the feature map, the method further includes:
[0011] Cluster the camera parameters corresponding to the initial reconstruction of the three-dimensional Gaussian splash model and select camera parameters from multiple camera perspectives.
[0012] The multi-view projection parameters are initialized based on the camera parameters selected from the multiple camera viewpoints; wherein, the multi-view projection parameters include the projection matrix corresponding to each camera viewpoint; the projection matrix is determined based on the camera parameters corresponding to the respective camera viewpoint.
[0013] As an improvement to the above scheme, the step of performing a differentiable grating operation on the three-dimensional Gaussian splash model to be transmitted based on the initialized multi-view projection parameters to obtain a feature map includes:
[0014] For each projection matrix in the multi-view projection parameters, the three-dimensional Gaussian sphere in the three-dimensional Gaussian splash model is subjected to mean projection through the corresponding projection matrix to obtain the two-dimensional coordinates of the corresponding three-dimensional Gaussian sphere.
[0015] By projecting the covariance matrix onto the three-dimensional Gaussian sphere in the three-dimensional Gaussian splash model using the corresponding projection matrix, the two-dimensional covariance matrix of the corresponding three-dimensional Gaussian sphere is obtained.
[0016] Based on the two-dimensional coordinates and two-dimensional covariance matrix of the three-dimensional Gaussian sphere, color mixing is performed on each pixel of the three-dimensional Gaussian splash model in two-dimensional space to obtain the feature map.
[0017] As an improvement to the above scheme, the step of optimizing the multi-view projection parameters based on the reconstructed model and the three-dimensional Gaussian splash model, and obtaining a multi-channel feature map based on the optimized multi-view projection parameters, includes:
[0018] Based on the geometric information difference between the reconstructed model and the 3D Gaussian splash model, a first loss function is constructed;
[0019] A second loss function is constructed based on the color information difference between the reconstructed model and the 3D Gaussian splash model.
[0020] Based on the first loss function and the second loss function, the multi-view projection parameters are iteratively trained to obtain optimized multi-view projection parameters;
[0021] Based on each projection matrix in the optimized multi-view projection parameters, the three-dimensional Gaussian splash model is re-performed with a differentiable grating to obtain the multi-channel feature map.
[0022] As an improvement to the above scheme, the step of performing hierarchical semantic encoding on the multi-channel feature map to obtain compressed data includes:
[0023] The semantic features of the multi-channel feature map are obtained by extracting semantic features from the multi-channel feature map using an encoder.
[0024] The semantic features are quantized using a quantizer to obtain quantized features;
[0025] Based on the quantization features, an image is reconstructed using a generator trained on an adversarial network to obtain the reconstructed image.
[0026] The reconstructed image is semantically masked using a pre-trained semantic segmentation model to generate a masked image;
[0027] Based on the residual between the reconstructed image and the multi-channel feature map and the mask image, extract the residual of the region of interest;
[0028] The residual of the region of interest is compressed to obtain the compressed residual.
[0029] The quantization features and the compression residuals are concatenated to generate the compressed data.
[0030] Secondly, embodiments of the present invention provide a semantic communication method applied at a receiving end, comprising:
[0031] Receive compressed data; wherein the compressed data is generated using the semantic communication method as described in any of the first aspects;
[0032] The quantization features and compression residuals are parsed from the compressed data;
[0033] Based on the quantization features and the compression residuals, the feature map is reconstructed;
[0034] The reconstructed feature map is restored using 3D Gaussian splashing, and a 3D Gaussian splashing model is reconstructed.
[0035] As an improvement to the above scheme, the step of reconstructing the feature map based on the quantization features and the compression residual includes:
[0036] Semantic recovery is performed on the quantized features to reconstruct a multi-channel feature map;
[0037] Super-resolution reconstruction is performed on the compressed residuals to obtain the enhanced residuals;
[0038] The enhanced residual is superimposed on the multi-channel feature map to generate a reconstructed feature map;
[0039] The reconstructed feature map is reconstructed using a three-dimensional Gaussian splash method to obtain the feature map.
[0040] Thirdly, embodiments of the present invention provide a semantic communication system, including: a transmitter and a receiver; wherein, the transmitter includes: a reverse three-dimensional Gaussian splash rasterization module, a model reconstruction module, a parameter optimization module, and a semantic encoding module; the receiver includes: a three-dimensional Gaussian splash model reconstruction module;
[0041] The reverse 3D Gaussian splash rasterization module is used to perform a differentiable raster operation on the 3D Gaussian splash model to be transmitted based on the initialized multi-view projection parameters to obtain a feature map; wherein, the initialized multi-view projection parameters are determined based on camera parameters.
[0042] The model reconstruction module is used to reconstruct the three-dimensional model based on the feature map to obtain the reconstructed model;
[0043] The parameter optimization module is used to optimize the multi-view projection parameters based on the reconstruction model and the three-dimensional Gaussian splash model, and to obtain a multi-channel feature map based on the optimized multi-view projection parameters.
[0044] The semantic encoding module is used to perform hierarchical semantic encoding on the multi-channel feature map to obtain compressed data, and transmit the compressed data to the receiving end;
[0045] The three-dimensional Gaussian splash model reconstruction module is used to recover the three-dimensional Gaussian splash model based on the compressed data.
[0046] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform a semantic communication method as described in any one of the first aspects or as described in any one of the second aspects.
[0047] Fifthly, embodiments of the present invention provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the semantic communication method as described in any one of the first aspects or the semantic communication method as described in any one of the second aspects.
[0048] Compared to existing technologies, the semantic communication method, system, storage medium, and program product provided by this invention have the following advantages: At the transmitting end, based on initialized multi-view projection parameters, a differentiable grating operation is performed on the three-dimensional Gaussian splash model to be transmitted to obtain a feature map; wherein, the initialized multi-view projection parameters are determined based on camera parameters; then, a three-dimensional model is reconstructed based on the feature map to obtain a reconstructed model; subsequently, based on the reconstructed model and the three-dimensional Gaussian splash model, the multi-view projection parameters are optimized, and a multi-channel feature map is obtained based on the optimized multi-view projection parameters; finally, the multi-channel feature map is subjected to hierarchical semantic encoding to obtain compressed data, and the compressed data is transmitted to the receiving end; at the receiving end, the three-dimensional Gaussian splash model is recovered based on the received compressed data; this invention can achieve compatibility between three-dimensional model compression and semantic segmentation models, and reduce the impact of key information loss on the quality of the reconstructed model in low signal-to-noise ratio channel environments. Attached Figure Description
[0049] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0050] Figure 1 This is a flowchart of a semantic communication method provided in an embodiment of the present invention;
[0051] Figure 2 This is an overall flowchart of the semantic communication system provided in the embodiments of the present invention;
[0052] Figure 3 This is a flowchart of the reverse three-dimensional Gaussian sputtering rasterization provided in an embodiment of the present invention;
[0053] Figure 4 This is a flowchart of semantic transmission provided in an embodiment of the present invention;
[0054] Figure 5 This is another flowchart of a semantic communication method provided in an embodiment of the present invention;
[0055] Figure 6 This is a structural block diagram of a semantic communication system provided in an embodiment of the present invention;
[0056] Figure 7 This is a structural block diagram of a semantic communication device provided in an embodiment of the present invention. Detailed Implementation
[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0058] It is understood that the various numerical designations used in the embodiments of this invention are merely for descriptive convenience and are not intended to limit the scope of this application. The order of the process numbers does not imply the order of execution; the execution order of each process should be determined by its function and internal logic.
[0059] In embodiments of the invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The term "a plurality or several" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.
[0060] Example 1
[0061] See Figure 1 , Figure 1 This is a flowchart of a semantic communication method provided in an embodiment of the present invention. The semantic communication method, applied at a transmitting end, includes:
[0062] S11: Based on the initialized multi-view projection parameters, perform a differentiable grating operation on the three-dimensional Gaussian splash model to be transmitted to obtain a feature map; wherein, the initialized multi-view projection parameters are determined based on camera parameters;
[0063] S12: Reconstruct the three-dimensional model based on the feature map to obtain the reconstructed model;
[0064] S13: Based on the reconstruction model and the three-dimensional Gaussian splash model, optimize the multi-view projection parameters, and obtain a multi-channel feature map based on the optimized multi-view projection parameters;
[0065] S14: Perform hierarchical semantic encoding on the multi-channel feature map to obtain compressed data, and transmit the compressed data to the receiving end.
[0066] In this embodiment of the invention, the semantic communication system includes a receiver and a transmitter, such as... Figure 2 As shown, at the receiving end, the original 3D Gaussian Splatting (3DGS) model to be transmitted (also known as the 3DGS 3D model) is first subjected to inverse 3DGS rasterization processing, including: determining the initialized multi-view projection parameters based on camera parameters; performing differentiable raster operations on the original 3D Gaussian Splatting model according to the initialized multi-view projection parameters to obtain feature maps; reconstructing the 3D model from the feature maps to obtain the reconstructed model; optimizing the multi-view projection parameters based on the reconstructed model and the original 3D Gaussian Splatting model to obtain multi-channel feature maps; then performing semantic encoding operations on the obtained multi-channel feature maps, including hierarchical semantic encoding of the multi-channel feature maps to obtain compressed data; subsequently, the transmitting end transmits the compressed data to the receiving end through a physical channel. At the receiving end, semantic decoding and 3DGS reconstruction operations are performed on the received compressed data to recover the original 3D Gaussian Splatting model. This invention achieves a reversible mapping from a 3D model to a 2D feature map through reverse 3DGS technology. This process not only preserves the semantic information of the model and achieves compatibility between 3D model compression and semantic segmentation models, but also enables subsequent data semantic transmission. Simultaneously, semantic communication is performed after data compression using semantic coding technology during transmission, which improves the overall quality of the reconstructed model under low signal-to-noise ratio (SNR) channel conditions, avoids introducing visual distortion and structural incompleteness, reduces the impact of key information loss on the quality of the reconstructed model in low SNR channel environments, and reduces the amount of data transmitted.
[0067] Furthermore, before performing a differentiable grating operation on the three-dimensional Gaussian splash model to be transmitted based on the initialized multi-view projection parameters to obtain the feature map, the method further includes:
[0068] Cluster the camera parameters corresponding to the initial reconstruction of the three-dimensional Gaussian splash model and select camera parameters from multiple camera perspectives.
[0069] The multi-view projection parameters are initialized based on the camera parameters selected from the multiple camera viewpoints; wherein, the multi-view projection parameters include the projection matrix corresponding to each camera viewpoint; the projection matrix is determined based on the camera parameters corresponding to the respective camera viewpoint.
[0070] In this embodiment of the invention, parameter initialization is performed before performing the differentiable grating operation; this includes setting hyperparameters; the hyperparameters include, but are not limited to:
[0071] The number of projection matrices is M; where M≥8, and each projection matrix corresponds to a camera vision, such as up, down, left, right, front, back, and oblique views.
[0072] Feature map resolution S×S;
[0073] The initial learning rate η = 0.01;
[0074] The convergence threshold ε = 1e-5.
[0075] Then, for the camera parameter set C = {c1, ..., cN} of the 3D Gaussian splash model during the initial reconstruction, M representative camera parameters from different camera views are selected using the k-means clustering algorithm to form the projection matrix parameters P = {P1, ..., PM}. The projection matrix parameters include M projection matrices, each composed of camera parameters (including intrinsic and extrinsic parameters) from the corresponding camera view, and each projection matrix has the greatest Euclidean distance to the cluster center. For example, each projection matrix Pi ∈ R... (3×4) =A·(R1, R2, T′); where A represents the intrinsic parameter matrix of the i-th projection matrix, (R1, R2, T′) represents the extrinsic parameter matrix of the i-th projection matrix, R1 and R2 represent the rotation matrices corresponding to different directions, the product of R1 and R2 represents the total rotation matrix, and T′ represents the translation vector.
[0076] In one optional embodiment, the step of performing a differentiable grating operation on the three-dimensional Gaussian splash model to be transmitted based on the initialized multi-view projection parameters to obtain a feature map includes:
[0077] For each projection matrix in the multi-view projection parameters, the three-dimensional Gaussian sphere in the three-dimensional Gaussian splash model is subjected to mean projection through the corresponding projection matrix to obtain the two-dimensional coordinates of the corresponding three-dimensional Gaussian sphere.
[0078] By projecting the covariance matrix onto the three-dimensional Gaussian sphere in the three-dimensional Gaussian splash model using the corresponding projection matrix, the two-dimensional covariance matrix of the corresponding three-dimensional Gaussian sphere is obtained.
[0079] Based on the two-dimensional coordinates and two-dimensional covariance matrix of the three-dimensional Gaussian sphere, color mixing is performed on each pixel of the three-dimensional Gaussian splash model in two-dimensional space to obtain the feature map.
[0080] In this embodiment of the invention, the following operations are performed for each projection matrix Pi:
[0081] (1) Mean Projection: The 3D Gaussian splash model obtained through the standard 3DGS process is composed of a large number of 3D Gaussian spheres. Under different camera viewpoints, the 3D Gaussian spheres appear in different colors. The center coordinates of the 3D Gaussian spheres are represented by the mean μ of the point coordinates of the 3D Gaussian spheres in 3D space. The center coordinates μ of all 3D Gaussian spheres within each camera's view frustum are transformed to screen space (i.e., 2D space) through the projection matrix Pi to obtain the 2D coordinates μ′ of the 3D Gaussian spheres in screen space.
[0082] For example, the two-dimensional coordinates μ′ of the projected three-dimensional Gaussian sphere in screen space can be obtained by multiplying the center coordinates μ of the three-dimensional Gaussian sphere by the projection matrix pi and then performing perspective division.
[0083] (2) Covariance Matrix Projection: The size and rotation information of the 3D Gaussian sphere are represented by the covariance matrix Σ (calculated based on the point coordinates of the 3D Gaussian sphere in 3D space). The covariance matrix Σ of all 3D Gaussian spheres within each camera's view frust is represented by a two-dimensional submatrix P of the Jacobian matrix V and the projection matrix Pi. 2D The projection is a two-dimensional covariance matrix Σ′ in screen space;
[0084] For example, this can be achieved by multiplying the Jacobian matrix V, the covariance matrix Σ, and the transpose of the Jacobian matrix V, and then multiplying the two-dimensional submatrix P. 2D The result of the above multiplication, the two-dimensional submatrix P 2D Multiplying by the transposes of the matrix yields the two-dimensional covariance matrix Σ′, and the specific calculation process is as follows:
[0085]
[0086] Here, the superscript T denotes transpose. The two-dimensional submatrix P of the projection matrix Pi. 2D It can be the content of the top left 2×4 part of the projection matrix Pi.
[0087] (3) Color Blending: Given the position of a pixel, the distance between that pixel and all overlapping 3D Gaussian spheres, i.e., the Gaussian depth of that pixel, can be calculated using the view transformation matrix W. The overlapping Gaussian spheres of that pixel are then sorted by depth to obtain a list of overlapping Gaussian spheres. Gaussian colors C are then blended according to the depth order of the overlapping Gaussian sphere list. i Superposition, based on the final opacity (Alpha)α′ of a three-dimensional Gaussian sphere i The value is alpha-blended to obtain the final color value C of the pixel. final The specific calculation process is as follows:
[0088]
[0089] Where, α′ iLet C be the final opacity of the i-th 3D Gaussian sphere. i Let be the color value of the i-th 3D Gaussian sphere, and N be the number of 3D Gaussian spheres. The final opacity α′ of the 3D Gaussian sphere is denoted by . i The specific calculation process is as follows:
[0090]
[0091] Where x′ and μ′ i These are the x-coordinate and center coordinates of the i-th 3D Gaussian sphere in screen space. It is the inverse of the two-dimensional covariance matrix of the i-th three-dimensional Gaussian sphere, α′ i It is the initial opacity of the i-th three-dimensional Gaussian sphere.
[0092] (4) Based on the final color value C of each pixel final This generates the feature map F of the original three-dimensional Gaussian splash model in two-dimensional space.
[0093] In this embodiment of the invention, the k-means clustering algorithm is used to select the most representative viewpoints from the original camera parameters, and the multi-view projection parameters are initialized based on these viewpoints. Subsequently, feature maps are generated through differentiable rasterization. Through the above-mentioned inverse 3DGS mapping process, the amount of information carried by the feature maps can be increased, and the loss of key information can be effectively reduced.
[0094] Furthermore, to improve computational efficiency, the following optimization strategies are adopted:
[0095] Tiling: Divide the screen space into multiple blocks (e.g., 16×16 pixel blocks), and each 3D Gaussian sphere is instantiated only within the block it covers.
[0096] GPU Sorting and Blending: Instances of 3D Gaussian spheres are sorted by depth using Radix Sort to ensure color blending from far to near, reducing redundant calculations.
[0097] In one optional embodiment, the step of optimizing the multi-view projection parameters based on the reconstructed model and the three-dimensional Gaussian splash model, and obtaining a multi-channel feature map based on the optimized multi-view projection parameters, includes:
[0098] Based on the geometric information difference between the reconstructed model and the 3D Gaussian splash model, a first loss function is constructed;
[0099] A second loss function is constructed based on the color information difference between the reconstructed model and the 3D Gaussian splash model.
[0100] Based on the first loss function and the second loss function, the multi-view projection parameters are iteratively trained to obtain optimized multi-view projection parameters;
[0101] Based on each projection matrix in the optimized multi-view projection parameters, the three-dimensional Gaussian splash model is re-performed with a differentiable grating to obtain the multi-channel feature map.
[0102] In this embodiment of the invention, the feature map F obtained above is input into a standard 3DGS reconstruction module to generate a reconstruction model Ω′. The final loss function L is then constructed. total Evaluate the quality of the reconstructed model Ω′.
[0103] L total =γ·L geometry +β·L appearance (4);
[0104] Among them, L geometry L represents the first loss function (also known as geometric loss), which uses the Chamfer distance to measure the surface similarity between the two models, i.e., the difference in geometric information between the two models; appearance The second loss function (also known as appearance loss) uses the SSIM (Structural Similarity Index) to measure the difference in color information between the two models; γ and β represent hyperparameters, which are determined based on experimental optimization.
[0105] Alignment optimization is performed based on the first loss function (also known as geometric loss) and the second loss function (also known as appearance loss) mentioned above, and the parameter update rules are as follows:
[0106]
[0107] Among them, P t Let P be the projection matrix before the update. t+1 For the updated projection matrix, L total Let η be the final loss function constructed above, representing the learning rate.
[0108] An adaptive moment estimator (Adam) is used for gradient updates, dynamically adjusting the learning rate η. The specific update process is as follows:
[0109]
[0110] Where τ = 0.1 represents the temperature coefficient, and ΔL is the loss function L after updating the projection matrix. total The change in ΔL. When ΔL < convergence threshold ε for multiple consecutive iterations, training terminates and the optimal feature map F is output. * .
[0111] In this embodiment of the invention, the quality of the feature map can be further optimized by iteratively training the projection matrix using a combination of geometric loss and appearance loss. For example... Figure 3 As shown, multi-resolution fusion technology was used throughout the reverse 3DGS process, namely, constructing a pyramid-shaped rendering architecture to generate feature maps of three scales, S×S, S / 2×S / 2, and S / 4×S / 4, for each projection matrix, which can improve the ability to preserve high-frequency details.
[0112] In one optional embodiment, the step of performing hierarchical semantic encoding on the multi-channel feature map to obtain compressed data includes:
[0113] The semantic features of the multi-channel feature map are obtained by extracting semantic features from the multi-channel feature map using an encoder.
[0114] The semantic features are quantized using a quantizer to obtain quantized features;
[0115] Based on the quantization features, an image is reconstructed using a generator trained on an adversarial network to obtain the reconstructed image.
[0116] The reconstructed image is semantically masked using a pre-trained semantic segmentation model to generate a masked image;
[0117] Based on the residual between the reconstructed image and the multi-channel feature map and the mask image, extract the residual of the region of interest;
[0118] The residual of the region of interest is compressed to obtain the compressed residual.
[0119] The quantization features and the compression residuals are concatenated to generate the compressed data.
[0120] For example, at the transmitting end, the basic model of a semantic communication system consists of an encoder, a quantizer, and a generator. The following section will combine... Figure 4 The specific semantic transmission process is as follows:
[0121] (1) Feature extraction and quantization: Input multi-channel feature map F∈R S×S×M Semantic features w = E(x) are extracted using an encoder, and then nearest-neighbor quantization is performed on the semantic features to obtain quantized features. The formula is:
[0122]
[0123] Where C = {c0, c1, ..., c} k c K} represents a predefined set of quantization centers, where K is the number of quantization centers, and c kLet w represent the k-th quantization center. r This represents the r-th semantic feature.
[0124] (2) Basic Reconstruction: Using a generator trained on an adversarial network, based on quantized features. Generate preliminary reconstructed images Here, G() represents the generator.
[0125] (3) Residual calculation: Calculate the residual between the original input multi-channel feature map F and the reconstructed image F′:
[0126] r'=FF′.
[0127] (4) Semantic mask control: Based on the reconstructed image F′, a semantic mask is generated using a pre-trained semantic segmentation model, resulting in a mask image Mask∈{0,1}. S×S =FNC(F′).
[0128] (5) Extract the residual r” = Mask × r' of the region of interest.
[0129] (6) Residual Compression: The mask-weighted residual Mask×r' is downsampled and compressed (e.g., JPEG2000 compression) to obtain the compressed residual rc=D s (Mask×r').
[0130] (7) Quantify the features of the basic model The compressed data is concatenated with the compressed residual rc of the enhanced model to obtain compressed data, which is then transmitted to the receiving end through the physical channel.
[0131] For example, compression encoding is performed before data transmission. For instance, to combat channel noise, this embodiment of the invention uses a hierarchical channel coding model. Assuming the channel noise feature fed back by the receiver is z, it is combined with the quantization feature... The feature w obtained by concatenating the compressed residual rc of the enhanced model q To splice: w f =Concat(w q ,z), the spliced features w f After channel coding is performed on the input hierarchical channel coding model, the signal is transmitted to the receiver through the physical channel. This achieves the decoupling of JSCC (Joint Source-Channel Coding) source coding and channel coding, facilitating migration under different channel conditions.
[0132] At the receiving end, basic semantic recovery is performed by a generator to recover the quantized features corresponding to the compressed data, and the enhanced residuals reconstructed by super-resolution are superimposed to synthesize the final feature map to recover the multi-channel feature map. The recovered multi-channel feature map is then input into a standard 3D Gaussian splashing module to reconstruct the three-dimensional Gaussian splashing model at the receiving end.
[0133] This invention implements a 3D model compression and transmission method based on reverse 3D Gaussian splashing. Through dynamic optimization of multi-view projection matrix and hierarchical semantic coding mechanism, it can significantly improve the compression efficiency and semantic compatibility of 3D data compared to the traditional 3D model compressed data file, which cannot generate a mask through semantic segmentation model. It is applicable to emerging scenarios such as metaverse and industrial digital twins.
[0134] This invention achieves a reversible mapping from a 3D model to a 2D feature map through inverse 3DGS technology, overcoming the geometric distortion bottleneck of traditional point cloud compression. During transmission, a semantic encoder is applied for data compression, employing semantic communication to improve the overall quality of the reconstructed model under low signal-to-noise ratio (SNR) channel conditions, avoiding visual distortion and structural incompleteness, while reducing the amount of transmitted data and improving transmission efficiency. This addresses the problem that existing 3D model compression methods strip away the resolvable semantic structure, making them incompatible with semantic segmentation models and prone to losing key information in low SNR channel environments, thus limiting their application potential in semantic communication scenarios. Furthermore, during the inverse 3DGS mapping process, a k-means clustering algorithm is used to select the most representative viewpoints from the original camera parameters, and multi-view projection parameters are initialized based on these viewpoints. Combined with differentiable rasterization operations to generate feature maps, this increases the information content carried by the feature maps. Further, iterative training of the projection matrix using joint geometric and appearance losses preserves high-frequency texture details while maintaining geometric accuracy, thus optimizing the quality of the feature maps.
[0135] Example 2
[0136] See Figure 5 , Figure 5 This is another flowchart of a semantic communication method provided in an embodiment of the present invention. The semantic communication method, applied at a receiving end, includes:
[0137] S21: Receive compressed data; wherein the compressed data is generated using the semantic communication method described in any one of the above embodiments;
[0138] S22: Parse the quantization features and compression residuals from the compressed data;
[0139] S23: Reconstruct the feature map based on the quantization features and the compression residual;
[0140] S24: Perform 3D Gaussian splash recovery on the reconstructed feature map to reconstruct the 3D Gaussian splash model.
[0141] In an optional embodiment, reconstructing the feature map based on the quantization features and the compression residual includes:
[0142] Semantic recovery is performed on the quantized features to reconstruct a multi-channel feature map;
[0143] Super-resolution reconstruction is performed on the compressed residuals to obtain the enhanced residuals;
[0144] The enhanced residual is superimposed on the multi-channel feature map to generate a reconstructed feature map;
[0145] The reconstructed feature map is reconstructed using a three-dimensional Gaussian splash method to obtain the feature map.
[0146] In this embodiment of the invention, after receiving the compressed data transmitted by the transmitter in Embodiment 1 above, the receiver performs semantic decoding and reconstruction to restore the original three-dimensional Gaussian splash model. The specific process is as follows:
[0147] (1) Channel decoding: Parsing the compressed data and extracting the quantization features. and compression residual
[0148] (2) Basic semantic recovery: Quantized features are recovered through a decoder. Reconstructed into multi-channel feature maps Here, G'() represents the decoder.
[0149] (3) Enhanced detail reconstruction: Perform super-resolution reconstruction on the compressed residual rc to obtain the enhanced residual rc′ = SuperResolution(rc). Wherein, SuperResolution() represents the super-resolution reconstruction operation.
[0150] (4) Feature stitching: The multi-channel feature map F' and the enhancement residual rc' obtained above are stitched together to obtain the reconstructed feature map F. final =F′+rc′.
[0151] (5) Standard 3DGS Reconstruction: After obtaining the reconstructed feature map F final Then, the reconstructed feature map F final The standard 3DGS process restores the original three-dimensional Gaussian splash model.
[0152] It should be noted that the working process of the semantic communication method described in Embodiment 2 of the present invention can refer to the working process of the semantic communication method described in Embodiment 1 above, and the technical effect achieved is the same as that of the semantic communication method described in Embodiment 1 above, so it will not be repeated here.
[0153] Example 3
[0154] See Figure 6 , Figure 6 This invention provides a structural block diagram of a semantic communication system, which includes a transmitter 1 and a receiver 2. The transmitter 1 includes a reverse 3D Gaussian sputtering rasterization module 11, a model reconstruction module 12, a parameter optimization module 13, and a semantic encoding module 14. The receiver 2 includes a 3D Gaussian sputtering model reconstruction module 21.
[0155] The reverse 3D Gaussian splash rasterization module 11 is used to perform a differentiable raster operation on the 3D Gaussian splash model to be transmitted according to the initialized multi-view projection parameters to obtain a feature map; wherein, the initialized multi-view projection parameters are determined based on camera parameters.
[0156] The model reconstruction module 12 is used to reconstruct a three-dimensional model based on the feature map to obtain a reconstructed model.
[0157] The parameter optimization module 13 is used to optimize the multi-view projection parameters according to the reconstruction model and the three-dimensional Gaussian splash model, and to obtain a multi-channel feature map according to the optimized multi-view projection parameters.
[0158] The semantic encoding module 14 is used to perform hierarchical semantic encoding on the multi-channel feature map to obtain compressed data, and transmit the compressed data to the receiving end.
[0159] The three-dimensional Gaussian splash model reconstruction module 21 is used to recover the three-dimensional Gaussian splash model based on the compressed data.
[0160] In an optional embodiment, the transmitter 1 further includes an initialization module, which includes:
[0161] Clustering unit is used to cluster the camera parameters corresponding to the initial reconstruction of the three-dimensional Gaussian splash model and select camera parameters from multiple camera perspectives.
[0162] The parameter initialization unit is used to initialize multi-view projection parameters according to the camera parameters selected from the multiple camera viewpoints; wherein, the multi-view projection parameters include a projection matrix corresponding to each camera viewpoint; the projection matrix is determined based on the camera parameters corresponding to the respective camera viewpoint.
[0163] It should be noted that, in the embodiments of the present invention, the model reconstruction module 12, the parameter optimization module 13, and the initialization module can be integrated into the reverse three-dimensional Gaussian sputtering rasterization module 11, or they can be set independently. No specific limitation is made in the embodiments of the present invention.
[0164] In an optional embodiment, the reverse three-dimensional Gaussian sputtering rasterization module 11 includes:
[0165] The mean projection unit is used to perform mean projection on the three-dimensional Gaussian sphere in the three-dimensional Gaussian splash model for each projection matrix in the multi-view projection parameters, and obtain the two-dimensional coordinates of the corresponding three-dimensional Gaussian sphere.
[0166] The covariance matrix projection unit is used to project the covariance matrix onto the three-dimensional Gaussian sphere in the three-dimensional Gaussian splash model through a corresponding projection matrix to obtain the two-dimensional covariance matrix of the corresponding three-dimensional Gaussian sphere.
[0167] The color mixing unit is used to perform color mixing on each pixel of the three-dimensional Gaussian splash model in two-dimensional space according to the two-dimensional coordinates and two-dimensional covariance matrix of the three-dimensional Gaussian sphere to obtain the feature map.
[0168] In an optional embodiment, the parameter optimization module 13 includes:
[0169] The first loss function construction unit is used to construct a first loss function based on the geometric information difference between the reconstructed model and the three-dimensional Gaussian splash model.
[0170] The second loss function construction unit is used to construct a second loss function based on the color information difference between the reconstructed model and the three-dimensional Gaussian splash model.
[0171] The parameter optimization unit is used to iteratively train the multi-view projection parameters according to the first loss function and the second loss function to obtain the optimized multi-view projection parameters.
[0172] The differentiable grating operation unit is used to perform a differentiable grating operation on the three-dimensional Gaussian splash model again according to each projection matrix in the optimized multi-view projection parameters to obtain the multi-channel feature map.
[0173] In an optional embodiment, the semantic encoding module 14 includes:
[0174] A semantic feature extraction unit is used to extract semantic features from the multi-channel feature map through an encoder to obtain the semantic features of the multi-channel feature map;
[0175] The feature quantization unit is used to quantize the semantic features through a quantizer to obtain quantized features;
[0176] An image reconstruction unit is used to reconstruct an image based on the quantized features using a generator trained on an adversarial network, thereby obtaining a reconstructed image.
[0177] A semantic masking unit is used to perform semantic masking on the reconstructed image using a pre-trained semantic segmentation model to generate a masked image.
[0178] The residual extraction unit is used to extract the residual of the region of interest based on the residual between the reconstructed image and the multi-channel feature map and the mask image;
[0179] The residual compression unit is used to compress the residual of the region of interest to obtain a compressed residual;
[0180] The splicing unit is used to splice the quantization features and the compression residual to generate the compressed data.
[0181] In an optional embodiment, the receiver 2 further includes: a semantic decoding module; the semantic decoding module includes:
[0182] The parsing unit is used to parse quantization features and compression residuals from the compressed data;
[0183] The feature map reconstruction unit is used to reconstruct a feature map based on the quantized features and the compression residual;
[0184] Furthermore, the three-dimensional Gaussian splash model reconstruction module 21 performs three-dimensional Gaussian splash recovery on the reconstructed feature map to reconstruct the three-dimensional Gaussian splash model.
[0185] In one optional embodiment, the feature map reconstruction unit includes:
[0186] A semantic recovery subunit is used to perform semantic recovery on the quantized features and reconstruct a multi-channel feature map.
[0187] The super-resolution reconstruction subunit is used to perform super-resolution reconstruction on the compressed residual to obtain the enhanced residual;
[0188] A feature overlay subunit is used to overlay the enhanced residual onto the multi-channel feature map to generate a reconstructed feature map;
[0189] A three-dimensional Gaussian splash reconstruction subunit is used to perform three-dimensional Gaussian splash reconstruction on the reconstructed feature map to obtain the feature map.
[0190] It should be noted that the working process of each module in the semantic communication device described in the embodiments of the present invention can refer to the working process of the semantic communication method described in Embodiments 1 and 2 above, and the technical effect achieved is the same as that of the semantic communication method described in Embodiments 1 and 2 above, which will not be repeated here.
[0191] Example 4
[0192] This invention also provides a computer-readable storage medium storing a computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the steps described in the various semantic communication method embodiments above.
[0193] Example 5
[0194] This invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the various semantic communication method embodiments described above.
[0195] by Figure 6 Taking the semantic communication device shown as an example, the above-mentioned computer-readable storage medium and / or computer-readable storage medium will be described. The semantic communication device includes a processor 31, a memory 32, and a computer program stored in the memory 32 and executable on the processor 31. When the processor 31 executes the computer program, it implements the steps in the various semantic communication method embodiments described above, such as steps S11-S14 or steps S21-S24.
[0196] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 32 and executed by the processor 31 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the semantic communication device.
[0197] The semantic communication device may include, but is not limited to, a processor 31 and a memory 32. Those skilled in the art will understand that the schematic diagram is merely an example of a semantic communication device and does not constitute a limitation on the semantic communication device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the semantic communication device may also include input / output devices, network access devices, buses, etc.
[0198] The processor 31 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 31 is the control center of the semantic communication device, connecting all parts of the semantic communication device via various interfaces and lines.
[0199] The memory 32 can be used to store the computer programs and / or modules. The processor 31 implements various functions of the semantic communication device by running or executing the computer programs and / or modules stored in the memory 32 and calling the data stored in the memory 32. The memory 32 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 32 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.
[0200] Wherein, if the modules / units integrated in the semantic communication device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor 31, it can implement the steps of the various method embodiments described above. Wherein, the computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.
[0201] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.
[0202] The above description represents the preferred embodiments of the present invention. It should be noted that, for those skilled in the art, various improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. A method of semantic communication, characterized by, Applications to the transmitting end include: Based on the initialized multi-view projection parameters, a differentiable grating operation is performed on the three-dimensional Gaussian splash model to be transmitted to obtain a feature map; wherein, the initialized multi-view projection parameters are determined based on camera parameters; The 3D model is reconstructed based on the feature map to obtain the reconstructed model; Based on the reconstruction model and the three-dimensional Gaussian splash model, the multi-view projection parameters are optimized, and a multi-channel feature map is obtained based on the optimized multi-view projection parameters. The multi-channel feature map is subjected to hierarchical semantic encoding to obtain compressed data, and the compressed data is transmitted to the receiving end.
2. The semantic communication method as described in claim 1, characterized in that, Before performing a differentiable grating operation on the 3D Gaussian splash model to be transmitted based on the initialized multi-view projection parameters to obtain the feature map, the method further includes: Cluster the camera parameters corresponding to the initial reconstruction of the three-dimensional Gaussian splash model and select camera parameters from multiple camera perspectives. The multi-view projection parameters are initialized based on the camera parameters selected from the multiple camera viewpoints; wherein, the multi-view projection parameters include the projection matrix corresponding to each camera viewpoint; the projection matrix is determined based on the camera parameters corresponding to the respective camera viewpoint.
3. The semantic communication method as described in claim 2, characterized in that, The step of performing a differentiable grating operation on the three-dimensional Gaussian splash model to be transmitted, based on the initialized multi-view projection parameters, to obtain a feature map includes: For each projection matrix in the multi-view projection parameters, the three-dimensional Gaussian sphere in the three-dimensional Gaussian splash model is subjected to mean projection through the corresponding projection matrix to obtain the two-dimensional coordinates of the corresponding three-dimensional Gaussian sphere. By projecting the covariance matrix onto the three-dimensional Gaussian sphere in the three-dimensional Gaussian splash model using the corresponding projection matrix, the two-dimensional covariance matrix of the corresponding three-dimensional Gaussian sphere is obtained. Based on the two-dimensional coordinates and two-dimensional covariance matrix of the three-dimensional Gaussian sphere, color mixing is performed on each pixel of the three-dimensional Gaussian splash model in two-dimensional space to obtain the feature map.
4. The semantic communication method as described in claim 2, characterized in that, The step involves optimizing the multi-view projection parameters based on the reconstructed model and the 3D Gaussian splash model, and obtaining a multi-channel feature map based on the optimized multi-view projection parameters, including: Based on the geometric information difference between the reconstructed model and the 3D Gaussian splash model, a first loss function is constructed; A second loss function is constructed based on the color information difference between the reconstructed model and the 3D Gaussian splash model. Based on the first loss function and the second loss function, the multi-view projection parameters are iteratively trained to obtain optimized multi-view projection parameters; Based on each projection matrix in the optimized multi-view projection parameters, the three-dimensional Gaussian splash model is re-performed with a differentiable grating to obtain the multi-channel feature map.
5. The semantic communication method as described in claim 1, characterized in that, The step of performing hierarchical semantic encoding on the multi-channel feature map to obtain compressed data includes: The semantic features of the multi-channel feature map are obtained by extracting semantic features from the multi-channel feature map using an encoder. The semantic features are quantized using a quantizer to obtain quantized features; Based on the quantization features, an image is reconstructed using a generator trained on an adversarial network to obtain the reconstructed image. The reconstructed image is semantically masked using a pre-trained semantic segmentation model to generate a masked image; Based on the residual between the reconstructed image and the multi-channel feature map and the mask image, extract the residual of the region of interest; The residual of the region of interest is compressed to obtain the compressed residual. The quantization features and the compression residuals are concatenated to generate the compressed data.
6. A semantic communication method, characterized in that, Applied to the receiving end, including: Receive compressed data; wherein the compressed data is generated using the semantic communication method as described in any one of claims 1-5; The quantization features and compression residuals are parsed from the compressed data; Based on the quantization features and the compression residuals, the feature map is reconstructed; The reconstructed feature map is restored using 3D Gaussian splashing, and a 3D Gaussian splashing model is reconstructed.
7. The semantic communication method as described in claim 6, characterized in that, The step of reconstructing the feature map based on the quantized features and the compressed residual includes: Semantic recovery is performed on the quantized features to reconstruct a multi-channel feature map; Super-resolution reconstruction is performed on the compressed residuals to obtain the enhanced residuals; The enhanced residual is superimposed on the multi-channel feature map to generate a reconstructed feature map; The reconstructed feature map is reconstructed using a three-dimensional Gaussian splash method to obtain the feature map.
8. A semantic communication system, characterized in that, include: The transmitter includes a transmitter module, a model reconstruction module, a parameter optimization module, and a semantic encoding module; the receiver includes a three-dimensional Gaussian splash model reconstruction module. The reverse 3D Gaussian splash rasterization module is used to perform a differentiable raster operation on the 3D Gaussian splash model to be transmitted based on the initialized multi-view projection parameters to obtain a feature map; wherein, the initialized multi-view projection parameters are determined based on camera parameters. The model reconstruction module is used to reconstruct the three-dimensional model based on the feature map to obtain the reconstructed model; The parameter optimization module is used to optimize the multi-view projection parameters based on the reconstruction model and the three-dimensional Gaussian splash model, and to obtain a multi-channel feature map based on the optimized multi-view projection parameters. The semantic encoding module is used to perform hierarchical semantic encoding on the multi-channel feature map to obtain compressed data, and transmit the compressed data to the receiving end; The three-dimensional Gaussian splash model reconstruction module is used to recover the three-dimensional Gaussian splash model based on the compressed data.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the semantic communication method as described in any one of claims 1 to 5 or the semantic communication method as described in any one of claims 6 to 7.
10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the semantic communication method as described in any one of claims 1 to 5 or the semantic communication method as described in any one of claims 6 to 7.
