Visual data transmission method, device, equipment, storage medium and program product

Abstract

The application relates to a visual data transmission method, device, equipment, storage medium and program product. The method comprises the following steps: obtaining multi-dimensional features corresponding to current visual data, fusing the multi-dimensional features to obtain current fusion features, taking the current fusion features as a query, taking the latest preset number of historical fusion features corresponding to historical latent representations as keys and values, obtaining current latent representations through cross-attention operation, encoding the current latent representations by using a probability distribution model based on time context prediction to obtain a binary code stream, and transmitting the binary code stream to a terminal. Since the obtained current latent representation is quantitative data representing the change characteristics of the current fusion features relative to the historical fusion features, the data amount of the current latent representation is relatively small, thereby reducing the data amount of subsequent encoding, improving the encoding efficiency, reducing the bandwidth occupation and shortening the transmission delay, and further improving the smoothness of terminal display.

Description

Technical Field

[0001] This application relates to the field of visual data transmission technology, and in particular to a visual data transmission method, apparatus, device, storage medium, and program product. Background Technology

[0002] With the rapid development of fields such as digital twins, metaverse, remote collaboration in industrial design, and autonomous driving, the real-time transmission of server-side rendered visualization data (3D models, images, etc.) to clients has become a core application scenario. To achieve real-time transmission of visualization data, the server first compresses the visualization data before transmitting the compressed data to the terminal.

[0003] Currently, with the rise of deep learning, image compression algorithms based on convolutional neural networks, such as COOL-CHIC and HiFiC, have emerged. However, these algorithms suffer from low encoding efficiency, affecting the transmission efficiency of visualized data and the smoothness of terminal visualized data display. Summary of the Invention

[0004] Therefore, it is necessary to provide a visualization data transmission method, apparatus, device, storage medium, and program product that can improve encoding efficiency, thereby improving the transmission efficiency of visualized data and the smoothness of terminal visualization data display, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a method for visualizing data transmission, applied to a server, the method comprising:

[0006] Obtain the multidimensional features corresponding to the current visualization data, and fuse the multidimensional features to obtain the current fused feature;

[0007] Using the current fused feature as the query, and the historical latent representations corresponding to the most recent preset number of historical fused features as the key and value, the current latent representation is obtained through cross-attention operation; the current latent representation represents the quantitative data of the change features of the current fused feature relative to the historical fused features.

[0008] The current latent representation is encoded using a probability distribution model based on time context prediction to obtain a binary code stream, which is then transmitted to the terminal.

[0009] In the above embodiments, multi-dimensional features corresponding to the current visualization data are obtained, and these multi-dimensional features are fused to obtain the current fused feature. Using the current fused feature as the query and the historical latent representations corresponding to the most recent preset number of historical fused features as the key and value, the current latent representation is obtained through cross-attention operation. The current latent representation is then encoded using a probability distribution model based on time context prediction to obtain a binary bitstream, which is then transmitted to the terminal. Since the obtained current latent representation is quantified data representing the change characteristics of the current fused feature relative to the historical fused features, the data volume of the current latent representation is relatively small, thereby reducing the amount of data to be encoded subsequently. Encoding results in a smaller binary data stream, improving encoding efficiency, reducing bandwidth usage, and shortening transmission latency, thereby improving the smoothness of terminal display.

[0010] In one embodiment, the current latent representation is obtained by using the current fused feature as the query and the historical latent representations corresponding to the most recent preset number of historical fused features as the key and value, through cross-attention operation, including:

[0011] Using the current fused feature as the query and the historical latent representation as the key and value, the current latent representation is obtained by performing cross-attention operation using the spatiotemporal attention module in the TransTIC encoder.

[0012] In the above embodiments, the current fused feature is used as the query, and the historical latent representation is used as the key and value. The current latent representation is obtained by performing cross-attention operation using the spatiotemporal attention module in the TransTIC encoder. This can eliminate temporal redundancy, mine and remove duplicate and redundant content between multiple visualization data, encode and represent only the dynamically changing features, reduce the amount of data for subsequent encoding, and improve the transmission efficiency of the binary code stream obtained after encoding.

[0013] In one embodiment, the current visualization data includes 3D visualization data, and the acquisition of multi-dimensional features corresponding to the current visualization data includes:

[0014] Extracting various data information from 3D visualization data; these data information include geometric information, structural information, and appearance information;

[0015] Multiple data types are vectorized to obtain information vectors;

[0016] Based on the data volume and information vector of the current visualization data, the feature extraction module in the TransTIC encoder is used to obtain the multidimensional features corresponding to the current visualization data.

[0017] In the above embodiments, the multidimensional features corresponding to the current visualization data are obtained by using the feature extraction module in the TransTIC encoder based on the data volume and information vector of the current visualization data. Since the larger the data volume of the current visualization data, the larger the information vector and the larger the data, the greater the computational load of the TransTIC encoder. Therefore, considering the size of the current visualization data to obtain the multidimensional features corresponding to the current visualization data can reduce the computational complexity of the subsequent TransTIC encoder and speed up the feature extraction.

[0018] In one embodiment, based on the data volume and information vector of the current visualization data, the feature extraction module in the TransTIC encoder is used to obtain the multidimensional features corresponding to the current visualization data, including:

[0019] When the amount of data is less than the preset amount of data, the feature extraction module is used to extract features from the information vector, and the extracted features are used as the multi-dimensional features corresponding to the current visualization data.

[0020] If the amount of data is not less than the preset amount of data, the information vector is divided into multiple vector blocks according to the semantic importance of the viewpoint prediction or 3D visualization data. For each vector block, the feature extraction module is used to extract features from the vector block, and the extracted features of the vector block are used as the multi-dimensional features corresponding to the current visualization data.

[0021] In the above embodiments, when the amount of data is not less than the preset amount of data, the information vector is divided into multiple vector blocks according to the semantic importance of the viewpoint prediction or 3D visualization data. For each vector block, the feature extraction module is used to extract features from a single vector block. The extracted features of the vector block are used as the multi-dimensional features corresponding to the current visualization data, which reduces the amount of data for feature extraction, speeds up the feature extraction speed, reduces the computational complexity of the subsequent TransTIC encoder, and further improves the encoding efficiency.

[0022] Secondly, this application also provides a method for visualizing data transmission, which is applied to a terminal and includes:

[0023] Receive the binary code stream transmitted by the server; the binary code stream is obtained by the server encoding the current latent representation using a probability distribution model based on time context prediction. The current latent representation is obtained by the server using the current fused feature as the query and the historical latent representations corresponding to the most recent preset number of historical fused features as the key and value, and then performing cross-attention operation. The current fused feature is obtained by fusing the multi-dimensional features corresponding to the current visualization data.

[0024] Thirdly, this application also provides a visual data transmission device for use on a server, the device comprising:

[0025] The acquisition module is used to acquire the multi-dimensional features corresponding to the current visualization data, and fuse the multi-dimensional features to obtain the current fused features;

[0026] The determination module is used to obtain the current latent representation by taking the current fused feature as the query and the historical latent representations corresponding to the most recent preset number of historical fused features as the key and value, and through cross-attention operation; the current latent representation represents the quantitative data of the change features of the current fused feature relative to the historical fused features.

[0027] The encoding module is used to encode the current latent representation using a probability distribution model based on time context prediction to obtain a binary code stream, and then transmit the binary code stream to the terminal.

[0028] Fourthly, this application also provides a visual data transmission device for use in a terminal, the device comprising:

[0029] The receiving module is used to receive the binary code stream transmitted by the server. The binary code stream is obtained by the server encoding the current latent representation using a probability distribution model based on time context prediction. The current latent representation is obtained by the server using the current fused feature as the query and the historical latent representations corresponding to the most recent preset number of historical fused features as the key and value, and then performing cross-attention operation. The current fused feature is obtained by fusing the multi-dimensional features corresponding to the current visualization data.

[0030] Fifthly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the above methods.

[0031] Sixthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above methods.

[0032] In a seventh aspect, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0033] The aforementioned visualization data transmission method, apparatus, device, storage medium, and program product acquire multi-dimensional features corresponding to the current visualization data, fuse these multi-dimensional features to obtain the current fused feature, use the current fused feature as the query, and use the historical latent representations corresponding to the most recent preset number of historical fused features as the key and value, respectively, to obtain the current latent representation through cross-attention operation. The current latent representation is then encoded using a probability distribution model based on time context prediction to obtain a binary code stream, which is then transmitted to the terminal. Since the obtained current latent representation is quantified data characterizing the change features of the current fused feature relative to the historical fused features, the data volume of the current latent representation is relatively small, thereby reducing the amount of data to be encoded subsequently. This results in a smaller binary data stream, improving encoding efficiency, reducing bandwidth usage, and shortening transmission latency, ultimately improving the smoothness of terminal display. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a diagram illustrating the application environment of a data transmission method in one embodiment.

[0036] Figure 2 This is a flowchart illustrating a data transmission method in one embodiment;

[0037] Figure 3 This is a flowchart illustrating a multidimensional feature acquisition method in one embodiment;

[0038] Figure 4 This is a flowchart illustrating a data reconstruction method in one embodiment;

[0039] Figure 5 This is a flowchart illustrating the data reconstruction method in another embodiment;

[0040] Figure 6 A flowchart illustrating the data transmission method in another embodiment;

[0041] Figure 7 This is a structural block diagram of a data transmission device in one embodiment;

[0042] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0044] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0045] With the rapid development of fields such as digital twins, metaverse, remote collaboration in industrial design, and autonomous driving, the real-time transmission of server-side rendered visualization data (3D models, images, etc.) to clients has become a core application scenario. To achieve real-time transmission of visualization data, the server first compresses the visualization data before transmitting the compressed data to the terminal.

[0046] Currently, with the rise of deep learning, image compression algorithms based on convolutional neural networks, such as COOL-CHIC and HiFiC, have emerged. However, these algorithms suffer from low compression efficiency, affecting the smoothness of data transmission and display on the terminal.

[0047] In view of this, this application proposes a visualization data transmission method applied to a server. The method acquires multi-dimensional features corresponding to the current visualization data, fuses these features to obtain the current fused feature, uses the current fused feature as the query, and uses the historical latent representations corresponding to the most recent preset number of historical fused features as the key and value. A cross-attention operation is then performed to obtain the current latent representation. The current latent representation represents the quantified data of the change characteristics of the current fused feature relative to the historical fused features. A probability distribution model based on time context prediction is used to encode the current latent representation to obtain a binary bitstream, which is then transmitted to the terminal. This approach achieves high compression efficiency, resulting in a low-bitrate binary bitstream with a smaller data volume, reducing the amount of data transmitted from the server to the terminal. This improves the smoothness of visualization data transmission and display on the terminal while maintaining high visual fidelity.

[0048] The visualization data transmission method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, server 102 communicates with terminal 104 via a network. Server 102 can obtain multi-dimensional features corresponding to the current visualization data, fuse the multi-dimensional features to obtain the current fused features, use the current fused features as the query, use the historical latent representations corresponding to the most recent preset number of historical fused features as the key and value, obtain the current latent representation through cross-attention operation, encode the current latent representation using a probability distribution model based on time context prediction to obtain a binary code stream, and transmit the binary code stream to terminal 104.

[0049] In one exemplary embodiment, such as Figure 2 As shown, a method for visualizing data transmission is provided, which can be applied to... Figure 1 Taking the server in the example, the explanation includes the following steps 201 to 203. Wherein:

[0050] Step 201: Obtain the multi-dimensional features corresponding to the current visualization data, and fuse the multi-dimensional features to obtain the current fused features.

[0051] The visualization data can be either 2D or 3D. 3D visualization data can include point cloud data, 3D models, triangular meshes, voxel meshes, etc. For 3D visualization data, the extracted multi-dimensional features can include multi-scale geometric features and appearance features. Multi-scale geometric features can be extracted from the original 3D data using a geometry encoding sub-network, such as a variant based on PointNet++ or a sparse 3D Convolutional Neural Network (3D CNN). Simultaneously, an appearance encoding sub-network is used to extract appearance features; for example, a 2D CNN can be used to process texture maps, or another sub-network can be used to process vertex colors.

[0052] Multi-scale geometric features and appearance features can be fused to obtain the current fused features, enabling the model to understand "which geometric structures correspond to which visual appearances", laying the foundation for subsequent perceptual optimization and compression.

[0053] Step 202: Using the current fusion feature as the query and the historical latent representations corresponding to the most recent preset number of historical fusion features as the key and value, the current latent representation is obtained through cross-attention operation; the current latent representation represents the quantitative data of the change features of the current fusion feature relative to the historical fusion features.

[0054] The historical latent representations corresponding to historical fusion features can be stored in the temporal context memory. Since the historical fusion features correspond to historical visualization data, the historical latent representations are also the historical latent representations corresponding to the historical visualization data.

[0055] Taking the current visualization data as the current frame 3D image I_t corresponding to the current time t as an example, the temporal context memory can store the historical latent representations corresponding to several previous historical frame 3D images. For example, it can store the historical latent representation z{t-1} corresponding to historical frame 3D images I{t-1}, the historical latent representation z{t-2} corresponding to historical frame 3D images I{t-2}, and the historical latent representation z{t-3} corresponding to historical frame 3D images I{t-3}. Here, I_{t-1} represents the historical frame 3D image corresponding to time t-1, I_{t-2} represents the historical frame 3D image corresponding to time t-2, and I{t-3} represents the historical frame 3D image corresponding to time t-3.

[0056] Using the current fused feature as the query and the historical latent representations corresponding to the most recent preset number of historical fused features as the key and value, a cross-attention operation is used to obtain the current latent representation. This cross-attention operation proactively analyzes the historical information most relevant to the current visualization data and encodes only inter-frame changes (motion) and newly added information with high precision, largely eliminating temporal redundancy. This yields quantitative data representing the changes in the current fused feature relative to historical fused features. Temporal redundancy refers to the presence of a large amount of static or similar repetitive scene information among multiple consecutive visualization data sets. Eliminating temporal redundancy involves mining and removing redundant content between multiple visualization data sets, encoding only dynamically changing features, thereby reducing information redundancy in the temporal dimension.

[0057] The Transformer's self-attention mechanism calculates the relationships between all feature blocks in a 3D scene. For example, it can understand that a window in the distance and a window nearby are structurally similar, thus enabling the encoding of repetitive structural patterns with fewer bits.

[0058] Since the amount of data showing changes in the visualization data between adjacent time points is relatively small, the amount of data obtained for the current latent representation is relatively small, thereby reducing the amount of data for subsequent encoding and improving encoding efficiency.

[0059] Traditional techniques using video coding standards such as H.264 / 265 or Audio Video Interleave (AV1) to compress rendered image sequences, while effective, are general-purpose designs and inefficient at compressing the sharp edges, uniform color blocks, and detailed textures unique to Computer-Generated Imagery (CGI). This often results in blurring and color banding, affecting the visual fidelity of the visualization data. In contrast, this application's embodiment achieves improved compression efficiency while maintaining the visual fidelity of the visualization data by obtaining a current latent representation of quantified data that characterizes the changes in current fusion features relative to historical fusion features.

[0060] Traditional AI compression algorithms have limitations: they suffer from temporal redundancy, and there is still room for improvement in encoding efficiency. However, this embodiment uses cross-attention computation to obtain the current latent representation, which eliminates temporal redundancy and improves compression efficiency.

[0061] Step 203: Encode the current latent representation using a probability distribution model based on time context prediction to obtain a binary code stream, and transmit the binary code stream to the terminal.

[0062] Traditional lossless or near-lossless compression formats (such as PNG) can guarantee quality, but their high bitrate puts enormous pressure on network bandwidth, making real-time interaction difficult. In this embodiment, however, a probability distribution model based on time context prediction is used to encode the current latent representation, resulting in a low-bitrate binary stream. Steps 202 and 203 achieve efficient compression encoding, removing redundant information from the visualization data and generating a smaller binary data stream. This reduces bandwidth usage, shortens transmission latency, and ensures smooth real-time display on the terminal.

[0063] In the above embodiments, multi-dimensional features corresponding to the current visualization data are obtained, and these multi-dimensional features are fused to obtain the current fused feature. Using the current fused feature as the query and the historical latent representations corresponding to the most recent preset number of historical fused features as the key and value, the current latent representation is obtained through cross-attention operation. The current latent representation is then encoded using a probability distribution model based on time context prediction to obtain a binary bitstream, which is then transmitted to the terminal. Since the obtained current latent representation is quantified data representing the change characteristics of the current fused feature relative to the historical fused features, the data volume of the current latent representation is relatively small, thereby reducing the amount of data to be encoded subsequently. Encoding results in a smaller binary data stream, improving encoding efficiency, reducing bandwidth usage, and shortening transmission latency, thereby improving the smoothness of terminal display.

[0064] In one embodiment, step 202, using the current fused feature as the query and the historical latent representations corresponding to the most recent preset number of historical fused features as keys and values, obtains the current latent representation through cross-attention operation, which can be implemented in the following way:

[0065] Using the current fused feature as the query and the historical latent representation as the key and value, the current latent representation is obtained by performing cross-attention operation using the spatiotemporal attention module in the TransTIC encoder.

[0066] The aforementioned time context memory can be the time context memory in the TransTIC encoder.

[0067] In the above embodiments, the current fused feature is used as the query, and the historical latent representation is used as the key and value. The current latent representation is obtained by performing cross-attention operation using the spatiotemporal attention module in the TransTIC encoder. This can eliminate temporal redundancy, mine and remove duplicate and redundant content between multiple visualization data, encode and represent only the dynamically changing features, reduce the amount of data for subsequent encoding, and improve the transmission efficiency of the binary code stream obtained after encoding.

[0068] In one embodiment, the current visualization data includes 3D visualization data, and the step 201 above, which involves obtaining the multidimensional features corresponding to the current visualization data, is as follows: Figure 3 As shown, it may include:

[0069] Step 301: Extract various data information from the 3D visualization data; these data information include geometric information, structural information, and appearance information.

[0070] Geometric information may include vertex coordinates and normal vectors; appearance information may include color, texture, and material; and structural information may include hierarchical relationships and semantic tags.

[0071] Step 302: Vectorize the various data information to obtain information vectors.

[0072] Different types and formats of data, such as geometric information, structural information, and appearance information, can be uniformly converted into a string of numerical vectors with fixed length and continuous values, thus obtaining a unified multimodal information vector, which facilitates subsequent feature extraction, attention calculation, encoding, and compression.

[0073] Step 303: Based on the data volume and information vector of the current visualization data, use the feature extraction module in the TransTIC encoder to obtain the multidimensional features corresponding to the current visualization data.

[0074] The feature extraction module in the TransTIC encoder can be a fusion of Transformer and geometric feature extractors. It employs an architecture that combines Transformer and geometric feature extractors to extract multi-dimensional features. This addresses the limitations of traditional 3D convolution in effectively handling sparse and irregular 3D data, such as point clouds, and its limited receptive field, which prevents the modeling of global context. Through extraction, the TransTIC encoder can intelligently determine which information is most critical to overall visual perception, thus achieving more efficient, content-aware compression. For complex but regular structures, encoding can be performed with minimal bits.

[0075] In the first scenario, when the amount of data is less than the preset amount of data, the feature extraction module is used to extract features from the information vector, and the extracted features are used as the multi-dimensional features corresponding to the current visualization data.

[0076] In the second scenario, assuming the data volume is not less than a preset amount, the information vector is divided into multiple vector blocks based on the semantic importance of the viewpoint prediction or 3D visualization data. For each vector block, a feature extraction module is used to extract features, and the extracted features are used as the multidimensional features corresponding to the current visualization data. In other words, the multidimensional features corresponding to the current visualization data can be features extracted from a single vector block.

[0077] Since the large amount of data in the current visualization data can affect the efficiency of subsequent encoding and compression, this embodiment divides the information vector into multiple vector blocks. The feature extraction module extracts features from each vector block to obtain multi-dimensional features. Based on these multi-dimensional features, the current fusion feature corresponding to the vector block is obtained. Subsequently, the current fusion feature is used as the query, and the historical latent representations corresponding to the most recent preset number of historical fusion features are used as the key and value. Cross-attention operation is used to obtain the current latent representation corresponding to the vector block. Then, the current latent representation is encoded using a probability distribution model based on time context prediction to obtain a binary bitstream. The binary bitstream corresponding to the vector block is then transmitted to the terminal, thereby improving the encoding efficiency.

[0078] In the above embodiments, the multidimensional features corresponding to the current visualization data are obtained by using the feature extraction module in the TransTIC encoder based on the data volume and information vector of the current visualization data. Since the larger the data volume of the current visualization data, the larger the information vector and the larger the data, the greater the computational load of the TransTIC encoder. Therefore, considering the size of the current visualization data to obtain the multidimensional features corresponding to the current visualization data can reduce the computational complexity of the subsequent TransTIC encoder and speed up the feature extraction.

[0079] In one embodiment, step 303, which involves obtaining multidimensional features corresponding to the current visualization data using the feature extraction module in the TransTIC encoder based on the data volume and information vector of the current visualization data, may include:

[0080] When the amount of data is less than the preset amount of data, the feature extraction module is used to extract features from the information vector, and the extracted features are used as the multi-dimensional features corresponding to the current visualization data.

[0081] If the amount of data is not less than the preset amount of data, the information vector is divided into multiple vector blocks according to the semantic importance of the viewpoint prediction or 3D visualization data. For each vector block, the feature extraction module is used to extract features from the vector block, and the extracted features of the vector block are used as the multi-dimensional features corresponding to the current visualization data.

[0082] When the data volume is less than the preset data volume, the feature extraction module extracts features from the information vectors and directly uses the extracted features as the multidimensional features corresponding to the current visualization data. When the data volume is not less than the preset data volume, the information vectors are divided into multiple vector blocks based on the viewpoint prediction or the semantic importance of the 3D visualization data. For each vector block, the feature extraction module extracts features from the vector block and uses the extracted features of one vector block as the multidimensional features corresponding to the current visualization data, that is, each vector block is processed separately. The information vectors can be divided into vector blocks according to an octree structure.

[0083] Given multiple vector blocks, the priority of each vector block can be determined, and the order of feature extraction for the vector blocks can be determined based on the priority, with higher priority vector blocks having their features extracted first.

[0084] In the above embodiments, when the amount of data is not less than the preset amount of data, the information vector is divided into multiple vector blocks according to the semantic importance of the viewpoint prediction or 3D visualization data. For each vector block, the feature extraction module is used to extract features from a single vector block. The extracted features of the vector block are used as the multi-dimensional features corresponding to the current visualization data, which reduces the amount of data for feature extraction, speeds up the feature extraction speed, reduces the computational complexity of the subsequent TransTIC encoder, and further improves the encoding efficiency.

[0085] In one embodiment, a method for visualizing data transmission is provided, which is applied to a terminal and includes the following steps:

[0086] Receive the binary code stream transmitted by the server; the binary code stream is obtained by the server encoding the current latent representation using a probability distribution model based on time context prediction. The current latent representation is obtained by the server using the current fused feature as the query and the historical latent representations corresponding to the most recent preset number of historical fused features as the key and value, and then performing cross-attention operation. The current fused feature is obtained by fusing the multi-dimensional features corresponding to the current visualization data.

[0087] In the above embodiments, the terminal receives the binary code stream transmitted by the server. Since the amount of data in the binary code stream obtained by the server through encoding is small, the bandwidth occupation of the binary code stream transmission can be reduced and the transmission latency can be shortened, thereby improving the smoothness of the terminal display.

[0088] In one embodiment, after receiving the binary code stream, the terminal can also perform a visualization data reconstruction operation based on the binary code stream to obtain reconstructed visualization data, and display the reconstructed visualization data as shown below. Figure 4 As shown, it may include:

[0089] Step 401: Decode the binary code stream to obtain the decoded latent representation.

[0090] Step 402: Perform visualization data reconstruction operation based on decoded latent representation and historical latent representation to obtain reconstructed visualization data, and display the reconstructed visualization data.

[0091] In the above embodiments, a decoded latent representation is obtained by decoding the binary bitstream, and a visualization data reconstruction operation is performed based on the decoded latent representation and the historical latent representation to obtain reconstructed visualization data. The reconstructed visualization data is then displayed, thus achieving high visual fidelity even with a low bitstream.

[0092] In one embodiment, step 402 involves performing a visualization data reconstruction operation based on the decoded latent representation and the historical latent representation to obtain reconstructed visualization data, and then displaying the reconstructed visualization data as shown below. Figure 5 As shown, it may include:

[0093] Step 501: Input the decoded latent representation and the historical latent representation into the spatiotemporal attention module in the TransTIC decoder to obtain the feature vector output by the spatiotemporal attention module.

[0094] Step 502: Input the feature vector into the visualization data reconstruction model to obtain the reconstructed visualization data.

[0095] Visual data reconstruction models can include, but are not limited to, Convolutional Neural Networks (CNN) models, Recurrent Neural Networks (RNN) models, and Fully Convolutional Neural Networks (FCN) models. For example, it could include the You Only Look Once Version 8 (YOLOv8) model, a single-stage real-time object detection algorithm.

[0096] The current visualization data can include 3D visualization data. The visualization data reconstruction model includes geometric reconstruction units and appearance reconstruction units. The feature vectors are input into the visualization data reconstruction model in the TransTIC decoder to obtain reconstructed visualization data, which can include:

[0097] The feature vectors are input into the geometric reconstruction unit and the appearance reconstruction unit to obtain the 3D structural information output by the geometric reconstruction unit and the surface appearance information corresponding to each geometric primitive in the 3D structural information output by the appearance reconstruction unit. The 3D structural information includes a set of point clouds or a set of mesh vertices, and the surface appearance information includes color values ​​and / or texture information. The reconstructed visualization data can include both 3D structural information and surface appearance information.

[0098] Among them, geometric primitives are the smallest geometric units that constitute a three-dimensional scene, including points, triangular facets, etc., and are used to represent the spatial morphology and structural features of the scene.

[0099] The terminal is no longer a simple decoder, but an intelligent reconstruction process with imagination. Traditional detail recovery based on prior knowledge is a deterministic process; details lost under high compression rates are permanently unrecoverable, resulting in coarse models and noise. The TransTIC decoder used in this application's embodiments learns the distribution prior of massive amounts of 3D data during training. Therefore, it can, like a 3D artist, generatively reconstruct the geometric details and textures lost during transmission based on the received incomplete and compressed features. Even if poor network conditions lead to highly compressed data, the terminal can still present visually complete and detailed visualizations, even achieving super-resolution reconstruction, far exceeding the effects of traditional interpolation algorithms, and significantly improving the robustness of user experience in weak network environments.

[0100] In the above embodiments, by inputting the decoded latent representation and the historical latent representation into the spatiotemporal attention module in the TransTIC decoder, the feature vector output by the spatiotemporal attention module is obtained. Then, the feature vector is input into the visualization data reconstruction model to obtain the reconstructed visualization data, thereby realizing intelligent utilization of time redundancy and maintaining the visual fidelity of the reconstructed visualization data at low bit rates.

[0101] In the embodiments of this application, such as Figure 6 As shown, a method for visualizing data transmission is provided, including:

[0102] The process involves acquiring the current visualization data, extracting multidimensional features using the feature extraction module, and fusing these features to obtain the current fused feature. Using the current fused feature as the query and the historical latent representations corresponding to the most recent preset number of historical fused features as the key and value, a cross-attention operation is performed using the spatiotemporal attention module in the TransTIC encoder to obtain the current latent representation. This latent representation is then encoded into a binary bitstream using a probability distribution model based on temporal context prediction. This probability distribution model can be understood as an arithmetic encoder, which outputs the binary bitstream. The binary bitstream is then transmitted to the terminal via the network.

[0103] The terminal receives a binary code stream, uses an arithmetic decoder to decode the binary code stream to obtain a decoded latent representation, inputs the decoded latent representation and the historical latent representation into the spatiotemporal attention module in the TransTIC decoder to obtain the feature vector output by the spatiotemporal attention module, inputs the feature vector into the visualization data reconstruction model to obtain reconstructed visualization data, and displays the reconstructed visualization data.

[0104] An initial TransTIC encoder and decoder can be trained end-to-end to obtain the TransTIC encoder and decoder. During training, an end-to-end bitrate distortion optimization model is employed, where distortion is measured not only by geometric error but, more importantly, by visual perception loss. For example, it renders the reconstructed 3D model as a 2D image and then compares it with the image rendered by the original model in the feature space of a pre-trained network such as VGG. The advantage is that the model learns how to compress to achieve the best final rendered image, rather than how to compress for the most accurate point coordinates. This represents a fundamental shift from digital precision to visual realism. Therefore, embodiments of this application optimize image fidelity for visual perception.

[0105] In the above embodiments, the server is responsible for the preprocessing, intelligent compression, and adaptive transmission scheduling of the visualization data. The terminal is used to receive and decode the binary bitstream and display the decoded data. The time context memory in the server is used to carry and interface with TransTIC intelligent encoding. The latent representation generated by the server is fed back to the terminal in a timely manner. As a historical latent representation, it is used to reconstruct the image with memory by combining features with the low bitrate binary bitstream sent by the server. The dynamic changes of the image are also smoother because of this memory, i.e., the historical latent representation. This effectively solves the high bandwidth usage problem faced by traditional image hard transmission, thereby eliminating time redundancy, maintaining high visual fidelity even with a low bitrate binary bitstream, and providing a smooth real-time interactive experience.

[0106] Based on the above analysis, the embodiments of this application have the following advantages:

[0107] Higher compression efficiency: Utilizing the spatiotemporal attention mechanism of Transformer, it intelligently and dynamically eliminates temporal redundancy, achieving excellent compression results for continuous motions commonly found in 3D visualization images, such as camera movement and model rotation. The bitrate can be reduced by more than 40% compared to H.265 / AV1, and by more than 60% compared to static WebP or AVIF.

[0108] Superior visual fidelity: Based on the nonlinear transformation capabilities of neural networks, sharp edges, detailed textures, and uniform color blocks of 3D images are better preserved, avoiding the blockiness and blurring problems of traditional encoding. Through perceptual loss training, the reconstructed image is nearly lossless in subjective vision.

[0109] Lower latency: The encoder relies only on the preceding frame and does not require the following frame, resulting in extremely low encoding latency, which meets the stringent requirements of real-time interactive applications.

[0110] Content Adaptability: The model automatically focuses on rapidly changing and static regions in the image through an attention mechanism, and assigns different bitrates to them, achieving intelligent bitrate control. Rapidly changing regions include, for example, interactive user interfaces and newly appearing objects, while static regions include, for example, the background.

[0111] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0112] Based on the same inventive concept, this application also provides a visual data transmission device for implementing the visual data transmission method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the visual data transmission device provided below can be found in the limitations of the visual data transmission method described above, and will not be repeated here.

[0113] In one exemplary embodiment, such as Figure 7 As shown, a visual data transmission device 700 is provided, applied to a server, including: an acquisition module 701, a determination module 702, and an encoding module 703, wherein:

[0114] The acquisition module 701 is used to acquire the multi-dimensional features corresponding to the current visualization data and fuse the multi-dimensional features to obtain the current fused features;

[0115] The determination module 702 is used to obtain the current latent representation by taking the current fused feature as the query and the historical latent representations corresponding to the most recent preset number of historical fused features as the key and value through cross-attention operation; the current latent representation represents the quantitative data of the change features of the current fused feature relative to the historical fused features.

[0116] The encoding module 703 is used to encode the current latent representation using a probability distribution model based on time context prediction to obtain a binary code stream, and then transmit the binary code stream to the terminal.

[0117] In one embodiment, the determining module 702 is specifically used to obtain the current latent representation by performing cross-attention operation using the spatiotemporal attention module in the TransTIC encoder, with the current fused feature as the query and the historical latent representation as the key and value.

[0118] In one embodiment, the current visualization data includes three-dimensional visualization data. The acquisition module 701 is specifically used to extract various data information from the three-dimensional visualization data. The various data information includes geometric information, structural information, and appearance information. The various data information is vectorized to obtain an information vector. Based on the data volume and information vector of the current visualization data, the feature extraction module in the TransTIC encoder is used to obtain the multi-dimensional features corresponding to the current visualization data.

[0119] In one embodiment, the acquisition module 701, when the data volume is less than a preset data volume, uses a feature extraction module to extract features from the information vector and uses the extracted features as the multidimensional features corresponding to the current visualization data; when the data volume is not less than the preset data volume, the information vector is divided into multiple vector blocks according to the viewpoint prediction or the semantic importance of the 3D visualization data, and for each vector block, the feature extraction module extracts features from the vector block and uses the extracted features of the vector block as the multidimensional features corresponding to the current visualization data.

[0120] In an exemplary embodiment, a visualization data transmission device is provided, applied to a terminal, including: a receiving module for receiving a binary code stream transmitted by a server; the binary code stream is obtained by the server encoding the current latent representation using a probability distribution model based on time context prediction, the current latent representation is obtained by the server using the current fused feature as the query and the historical latent representations corresponding to the most recent preset number of historical fused features as the key and value, through cross-attention operation, and the current fused feature is obtained by fusing the multi-dimensional features corresponding to the current visualization data.

[0121] In one embodiment, the device further includes:

[0122] The decoding module is used to decode the binary code stream to obtain the decoded latent representation;

[0123] The reconstruction module is used to perform visualization data reconstruction operations based on the decoded latent representation and historical latent representation, obtain reconstructed visualization data, and display the reconstructed visualization data.

[0124] In one embodiment, the reconstruction module is specifically used to input the decoded latent representation and the historical latent representation into the spatiotemporal attention module in the TransTIC decoder to obtain the feature vector output by the spatiotemporal attention module; and input the feature vector into the visualization data reconstruction model to obtain the reconstructed visualization data.

[0125] Each module in the aforementioned visual data transmission device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0126] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements an operator accuracy testing method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0127] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0128] In one exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of any of the method embodiments described above. The technical principles and effects are similar and will not be repeated here.

[0129] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the above method embodiments. The technical principles and effects are similar and will not be repeated here.

[0130] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of any of the method embodiments described above. Its technical principles and effects are similar and will not be repeated here.

[0131] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0132] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0133] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for visualizing data transmission, characterized in that, The method is applied to a server, and the method includes: Obtain the multidimensional features corresponding to the current visualization data, and fuse the multidimensional features to obtain the current fused features; Using the current fused feature as the query, and the historical latent representations corresponding to the most recent preset number of historical fused features as the key and value, the current latent representation is obtained through cross-attention operation; the current latent representation represents the quantitative data of the change features of the current fused feature relative to the historical fused features. The current latent representation is encoded using a probability distribution model based on time context prediction to obtain a binary code stream, which is then transmitted to the terminal.

2. The method according to claim 1, characterized in that, The process of obtaining the current latent representation by using the current fused feature as the query and the historical latent representations corresponding to the most recent preset number of historical fused features as keys and values ​​through cross-attention operation includes: Using the current fused feature as the query and the historical latent representation as the key and value, the current latent representation is obtained by performing cross-attention operation using the spatiotemporal attention module in the TransTIC encoder.

3. The method according to claim 1 or 2, characterized in that, The current visualization data includes 3D visualization data, and the step of obtaining the multi-dimensional features corresponding to the current visualization data includes: Extract various data information from the 3D visualization data; these various data information include geometric information, structural information, and appearance information; The various data information are vectorized to obtain an information vector; Based on the data volume of the current visualization data and the information vector, the feature extraction module in the TransTIC encoder is used to obtain the multidimensional features corresponding to the current visualization data.

4. The method according to claim 3, characterized in that, The step of obtaining multidimensional features corresponding to the current visualization data using the feature extraction module in the TransTIC encoder based on the data volume and information vector of the current visualization data includes: When the amount of data is less than the preset amount of data, the feature extraction module is used to extract features from the information vector, and the extracted features are used as the multi-dimensional features corresponding to the current visualization data. When the amount of data is not less than the preset amount of data, the information vector is divided into multiple vector blocks according to the viewpoint prediction or the semantic importance of the 3D visualization data. For each vector block, the feature extraction module is used to extract features from the vector block, and the extracted features of the vector block are used as the multidimensional features corresponding to the current visualization data.

5. A method for visualizing data transmission, characterized in that, The method is applied to a terminal, and the method includes: The server receives a binary code stream; the binary code stream is obtained by the server encoding the current latent representation using a probability distribution model based on time context prediction. The current latent representation is obtained by the server using the current fused feature as the query and the historical latent representations corresponding to the most recent preset number of historical fused features as the key and value, through cross-attention operation. The current fused feature is obtained by fusing the multi-dimensional features corresponding to the current visualization data.

6. The method according to claim 5, characterized in that, The method further includes: Decoding the binary code stream yields the decoded latent representation; Based on the decoded latent representation and the historical latent representation, a visualization data reconstruction operation is performed to obtain reconstructed visualization data, which is then displayed.

7. The method according to claim 6, characterized in that, The visualization data reconstruction operation based on the decoded latent representation and the historical latent representation to obtain reconstructed visualization data includes: The decoded latent representation and the historical latent representation are input into the spatiotemporal attention module in the TransTIC decoder to obtain the feature vector output by the spatiotemporal attention module. The feature vector is input into the visualization data reconstruction model to obtain the reconstructed visualization data.

8. A visual data transmission device, characterized in that, Applied to a server, the device includes: The acquisition module is used to acquire the multi-dimensional features corresponding to the current visualization data, and fuse the multi-dimensional features to obtain the current fused features. The determination module is used to obtain the current latent representation by taking the current fused feature as the query and the historical latent representations corresponding to the most recent preset number of historical fused features as the key and value, and by performing cross-attention operation; the current latent representation represents the quantitative data of the change features of the current fused feature relative to the historical fused features. The encoding module is used to encode the current latent representation using a probability distribution model based on time context prediction to obtain a binary code stream, and then transmit the binary code stream to the terminal.

9. A visual data transmission device, characterized in that, Applied to a terminal, the device includes: The receiving module is used to receive the binary code stream transmitted by the server. The binary code stream is obtained by the server encoding the current latent representation using a probability distribution model based on time context prediction. The current latent representation is obtained by the server using the current fused feature as the query and the historical latent representations corresponding to the most recent preset number of historical fused features as the key and value, and then performing cross-attention operation. The current fused feature is obtained by fusing the multi-dimensional features corresponding to the current visualization data.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images