A video communication method, device, software product, and storage medium
By extracting semantic features from multi-view videos and using joint source-channel coding technology, the problems of low efficiency and poor stability in traditional video transmission under low signal-to-noise ratio channels are solved, achieving efficient and robust multi-view video transmission that adapts to complex channel environments.
Patent Information
- Application Number
- CN202611168794.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-03
- Publication Date
- 2026-11-17
AI Technical Summary
Traditional video transmission technologies cannot function properly under low signal-to-noise ratio channel conditions, have weak adaptability, low transmission efficiency, poor transmission quality, and cannot effectively utilize the inherent correlation between multi-view video data, resulting in low compression efficiency, wasted bandwidth resources, insufficient adaptability in complex channel environments, and poor system stability.
By acquiring a set of feature information from multi-view videos, semantic feature information within frames, between frames, and between viewpoints is extracted using a multi-dimensional Swing Transformer model. This is combined with channel information to perform source-channel joint coding, generating a transmission data stream. Entropy model and Huffman coding are used to optimize coding efficiency, and coding strategies are dynamically adjusted to adapt to channel changes. The JSCC model is used to optimize the transmission process.
It achieves efficient transmission of multi-view video, reduces information redundancy, improves transmission robustness and system stability in low signal-to-noise ratio environments, adapts to dynamic channels, and improves transmission efficiency and quality.
Smart Images

Figure CN123123585A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of wireless communication technology, and in particular to a video communication method, device, program product, and storage medium. Background Technology
[0002] Traditional video transmission technologies primarily rely on pixel-based source coding (such as H.264 and H.265) and independent channel coding, requiring error-free communication. However, they cannot function properly under low signal-to-noise ratio (SNR) channel conditions. Traditional video communication methods exhibit weak adaptability in complex channel environments, resulting in low transmission efficiency and poor transmission quality. For multi-view video transmission, existing methods, besides their performance limitations under low SNR channels, also suffer from significant limitations in processing multiple video streams, leading to low compression efficiency and wasted bandwidth resources. Furthermore, existing multi-view video transmission methods lack adaptability to complex channel environments, failing to meet the system's flexibility and adaptability requirements when the SNR fluctuates significantly, resulting in poor system stability. Summary of the Invention
[0003] To address the aforementioned technical problems, embodiments of this application provide a video communication method, device, program product, and storage medium.
[0004] The video communication method provided in this application includes: Obtain a feature information set of an initial multi-view video, wherein the initial multi-view video includes initial videos from at least two perspectives, and the feature information set includes intra-frame semantic feature information of the initial video from each perspective, inter-frame semantic feature information of the initial video from each perspective, and inter-view semantic feature information of the initial multi-view video. The transmission data stream corresponding to the initial multi-view video is determined based on the feature information set of the initial multi-view video and the channel information of the first channel, wherein the first channel is the channel for transmitting the initial multi-view video. The initial multi-view video is transmitted through the first channel, and the multi-view synchronized video corresponding to the initial multi-view video is obtained based on the received transmission data stream.
[0005] The video communication device provided in this application includes a processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to execute the video communication method described above.
[0006] This application provides a computer program product, comprising: a computer program that, when executed by a processor, implements the aforementioned video communication method.
[0007] The computer-readable storage medium provided in this application is used to store a computer program that causes a computer to perform the above-described video communication method.
[0008] In the technical solution of this application, a feature information set of an initial multi-view video is obtained. The initial multi-view video includes initial videos from at least two perspectives. The feature information set includes intra-frame semantic feature information, inter-frame semantic feature information, and inter-view semantic feature information of the initial videos from each perspective. A transmission data stream corresponding to the initial multi-view video is determined based on the feature information set and channel information of a first channel, where the first channel is the channel for transmitting the initial multi-view video. The transmission data stream of the initial multi-view video is transmitted through the first channel, and a multi-view synchronized video corresponding to the initial multi-view video is obtained based on the received transmission data stream. Thus, multi-view video transmission is achieved. By extracting semantic features from the multi-view video to determine the transmission data stream, information redundancy between multiple perspectives is effectively reduced, achieving efficient transmission of the multi-view video. Determining the transmission data stream using semantic feature information and channel information can effectively improve transmission robustness in low signal-to-noise ratio environments and adapt to dynamic channels. Attached Figure Description
[0009] The accompanying drawings, which are provided to further illustrate this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application.
[0010] Figure 1 This is a flowchart illustrating the video communication method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the overall architecture of the multi-view video semantic communication system provided in the embodiments of this application; Figure 3 This is a schematic diagram of the framework for extracting semantic features provided in an embodiment of this application; Figure 4 This is a schematic diagram of a JSCC-based video end-to-end semantic communication framework provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structural composition of the video communication device provided in the embodiments of this application; Figure 6 This is a schematic structural diagram of a video communication device provided in an embodiment of this application; Figure 7 This is a schematic structural diagram of the chip according to an embodiment of this application. Detailed Implementation
[0011] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.
[0012] In the following description, the term "some embodiments" refers to a subset of all possible embodiments. However, it is understood that "some embodiments" can be the same or different subsets of all possible embodiments and can be combined with each other without conflict. It should also be noted that the terms "first," "second," and "third" used in the embodiments of this application are only used to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first," "second," and "third" can be interchanged in a specific order or sequence where permissible, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein. The term "and / or" in this document is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship. It should also be understood that the "instruction" mentioned in the embodiments of this application can be a direct instruction, an indirect instruction, or an indication of an association relationship. For example, A instructing B can mean that A directly instructs B, for example, B can obtain information through A; it can also mean that A indirectly instructs B, for example, A instructs C, and B can obtain information through C; it can also mean that there is an association between A and B. It should also be understood that the term "correspondence" mentioned in the embodiments of this application can mean that there is a direct or indirect correspondence between the two, or that there is an association between the two, or that there is an instruction and being instructed, configuration and being configured, etc.
[0013] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and they all fall within the protection scope of the embodiments of this application.
[0014] Traditional video transmission technologies primarily rely on pixel-based source coding (such as H.264 and H.265) and independent channel coding, requiring error-free communication. However, they cannot function properly under low signal-to-noise ratio (SNR) channel conditions. Traditional video communication methods exhibit weak adaptability in complex channel environments, resulting in low transmission efficiency and poor transmission quality. Therefore, a new video transmission paradigm is needed to improve the robustness and stability of communication systems in low SNR environments. For multi-view video transmission, existing methods, in addition to the aforementioned performance deficiencies under low SNR channels, also suffer from significant limitations in processing multiple video streams. Current technologies lack efficient collaborative compression and transmission mechanisms, failing to utilize the inherent correlations and redundancy between multi-view video data, leading to low compression efficiency and wasted bandwidth resources. Furthermore, in bandwidth-constrained scenarios, traditional communication methods, due to the high resource requirements and low efficiency in transmitting multi-view video data streams, struggle to achieve complete real-time transmission. In multi-view applications (such as sports broadcasts, virtual reality, or panoramic video), this inefficiency significantly reduces video smoothness and integrity, impacting user experience. Furthermore, existing multi-view video transmission methods lack adaptability in complex channel environments, failing to meet the system's flexibility and adaptability requirements when the signal-to-noise ratio fluctuates significantly, resulting in poor system stability. Therefore, there is an urgent need to develop more advanced multi-view video transmission systems to optimize the utilization of multi-view video data, reduce bandwidth consumption, and improve transmission efficiency and stability.
[0015] Therefore, how to maximize the efficiency and stability of multi-view video transmission while minimizing bandwidth consumption becomes a key consideration in video communication. To this end, the following technical solutions, as proposed in this application, are presented.
[0016] Figure 1 This is a flowchart illustrating the video communication method provided in an embodiment of this application, as shown below. Figure 1 As shown, the video communication method includes the following steps: Step 101: Obtain the initial set of feature information from the multi-view video.
[0017] The initial multi-view video includes initial videos from at least two perspectives, and the feature information set includes intra-frame semantic feature information of the initial video from each perspective, inter-frame semantic feature information of the initial video from each perspective, and inter-view semantic feature information of the initial multi-view video.
[0018] In some embodiments, the video communication method is applied to a video communication system, which includes a multi-view semantic feature extraction module for obtaining a set of feature information of an initial multi-view video input to the video communication system.
[0019] It is understandable that an initial multi-view video includes initial information from at least two perspectives, and here we extract the feature information set for each initial multi-view video.
[0020] In some implementations, obtaining an initial set of feature information from the multi-view video includes: The intra-frame semantic feature information of the initial video from each perspective in the initial multi-view video is obtained through the first model; The inter-frame semantic feature information of the initial video of each viewpoint in the initial multi-view video is determined by the second model and the intra-frame semantic feature information of the initial video of each viewpoint. The inter-view semantic feature information of the initial multi-view video is determined by the third model and the inter-frame semantic feature information of the initial video of each view.
[0021] In some implementations, the multi-view semantic feature extraction module includes a multi-dimensional Swing Transformer model, wherein the first model is a two-dimensional Swing Transformer, the second model is a three-dimensional Swing Transformer, and the third model is a four-dimensional Swing Transformer.
[0022] In some implementations, a two-dimensional Swing Transformer is used to process the intra-frame information of the initial video from each viewpoint in the initial multi-view video, and single-frame images from a single viewpoint are processed to extract intra-frame semantic feature information. Specifically, static elements of single-frame images are identified through edge detection and region segmentation, and features are extracted using local binary patterns and Gabor filtering.
[0023] In some implementations, the intra-frame semantic feature information of the initial video from each viewpoint is extracted in the manner shown in Equation (1): , (1) in, Represents a two-dimensional Swing Transformer. Indicates the first A perspective on time Frame images, This represents intra-frame features, also known as intra-frame semantic features.
[0024] In some implementations, a 3D Swing Transformer is used to process frame sequences from the same viewpoint, capturing semantic associations in the temporal dimension and generating inter-frame semantic features. Specifically, the initial video for each viewpoint is divided into multiple GOPs based on Group of Pictures (GOPs). Cosine similarity is used to measure invariant elements, and motion compensation is combined to predict dynamic changes. For each GOP, the complete features of keyframes are extracted, and difference features are calculated for the predicted frames to optimize coding efficiency.
[0025] In some implementations, the method for extracting inter-frame semantic feature information of the initial video from each viewpoint is as shown in Equation (2): , (2) in, These are inter-frame features, also known as inter-frame semantic features, which incorporate temporal dynamic information. Represents a 3D Swing Transformer; Representing a single perspective in a time series (from time...) arrive The set of intra-frame features for all frames; This represents the intra-frame features of a single frame at time point t. These features are typically obtained by feature extraction on each frame, capturing the spatial information of that frame. t=1,2,…,T: represents the index in the time dimension, where T is the total number of frames in a GOP.
[0026] In some implementations, a four-dimensional Swing Transformer is used to process inter-view information, jointly process the inter-frame features of all views, extract inter-view correlation information, which is also content relevance information, and measure it with feature similarity to generate joint features between views.
[0027] In some implementations, the semantic features between viewpoints of the initial multi-view video are extracted in the manner shown in Equation (3): , (3) in, This refers to the joint features between multiple perspectives, that is, the semantic feature information between perspectives. This represents a four-dimensional Swing Transformer. Indicates from the first perspective to the second. This is a set of all inter-frame features from each viewpoint. This set contains inter-frame features from all viewpoints, excluding inter-viewpoint correlation information, and serves as input to a four-dimensional SwinTransformer to further extract inter-viewpoint correlation information and generate joint inter-viewpoint features from multiple viewpoints. . This represents the inter-frame feature of the nth viewpoint. This feature is obtained by processing the frame sequence of the same viewpoint using a 3D Swing Transformer, capturing the dynamic information and semantic relationships of that viewpoint in the temporal dimension. n = 1, 2, ..., N represents the index of the viewpoint, and N is the total number of viewpoints.
[0028] In some implementations, during the extraction of semantic features between viewpoints, the weights of these features are dynamically adjusted using an attention mask to improve feature extraction efficiency. Specifically, when processing inter-viewpoint information in the four-dimensional Swin Transformer, an attention mask mechanism is introduced to dynamically adjust the inter-frame features from different viewpoints. The contribution weights are determined to optimize the efficiency and accuracy of feature extraction.
[0029] In some implementations, semantic feature information between viewpoints is calculated based on the content relevance between viewpoints. The contribution weights are then used to generate an attention mask matrix based on the content relevance between viewpoints and the contribution weights. .in Indicates the first Inter-frame features from the first perspective on the second Attention weights for each viewpoint are assigned, with mask values ranging from [0,1]. Larger values indicate a higher contribution of that viewpoint to the joint features. A multi-head self-attention mechanism is then used in the four-dimensional Swing Transformer to process these weights. Among them, the attention mask matrix It is incorporated into the self-attention calculation, and the specific formula is shown in equation (4): , (4) in, These are query, key, and value matrices, respectively. It is the dimension of the key. It is an attention mask used to dynamically adjust the interaction weights of features between viewpoints. During training or inference, the attention mask can be dynamically updated according to changes in the input video content.
[0030] In this way, by dynamically allocating weights through the attention mechanism, the feature extraction process is optimized, significantly improving the compression efficiency and accuracy of semantic information.
[0031] Step 102: Determine the transmission data stream corresponding to the initial multi-view video based on the feature information set of the initial multi-view video and the channel information of the first channel.
[0032] The first channel is the channel for transmitting the initial multi-view video.
[0033] In some implementations, the channel information of the first channel includes the channel noise characteristics of the first channel.
[0034] In some implementations, the transmission data stream corresponding to the initial multi-view video is determined based on the feature information set of the initial multi-view video and the channel information of the first channel, including: The semantic feature information between the viewpoints is encoded to obtain the first encoded feature; The transmission data stream of the initial multi-view video is determined based on the first encoding feature and the channel information of the first channel.
[0035] Here, the semantic feature information between viewpoints is jointly encoded by the source and channel to obtain the initial multi-view video transmission data stream.
[0036] In some implementations, encoding the semantic feature information between the viewpoints to obtain a first encoded feature includes: The semantic feature information between the viewpoints is quantized to obtain the quantized semantic feature information between the viewpoints; Obtain the information content of each symbol in the quantified semantic feature information between viewpoints; The quantized inter-view semantic feature information is hierarchically encoded based on the information content of each symbol to obtain basic layer features and enhancement layer features, wherein the first encoded feature includes basic layer features and enhancement layer features.
[0037] In some implementations, the extracted inter-view semantic features are first quantized to generate quantized inter-view semantic feature information, which is then used... express.
[0038] In some implementations, semantic feature information between viewpoints Perform nearest neighbor quantization to generate quantized semantic features. The specific method is shown in equation (5): , (5) in, For a predefined set of quantization centers, based on the representativeness of the feature distribution, channel capacity constraints, task relevance, and computational complexity, a data-driven method (VQ-VAE) and task-specific optimization are used to generate a set C of quantization centers, which is dynamically adjusted to adapt to changes in channel and content. This application does not specifically limit the method of obtaining this set. c is the set of quantization centers. A single quantization center in the feature space represents a discrete representative point, used to convert continuous features... Quantization into discrete symbols These symbols are discrete representations of the feature space, representing the semantic information of the video content. That is, Multi-perspective joint features The set of discrete symbols obtained after quantization.
[0039] In some implementations, an entropy model is used to obtain the information content of each symbol in the quantized inter-view semantic feature information. The specific method is shown in equation (6): , (6) in, For entropy, Let q represent the probability distribution of semantic symbols. Here, in the entropy model, q represents a single semantic symbol. Multi-perspective joint features The discrete symbol set obtained after quantization, where each q is A quantization symbol q corresponds to a center c in the set of quantization centers C. The probability distribution p(q) represents the probability of each semantic symbol q appearing.
[0040] In some implementations, p(q) is calculated by first statistically analyzing all training data obtained using the aforementioned method. The frequency of occurrence of each symbol in the text is analyzed to construct an empirical distribution, and a deep learning model is used to analyze it. We model the aforementioned empirical distribution and estimate p(q). Here, the deep learning model is a component of the entropy model.
[0041] In some implementations, each semantic symbol is calculated based on an entropy model. The information content is determined using Huffman coding, assigning different length codes to each semantic symbol. Here, the information content reflects the symbol's... Uncertainty or rarity, probability The lower the probability, the greater the information content, and the longer the encoding length should be; conversely, semantic symbols with high probability have less information content, so they should be assigned shorter encodings.
[0042] In some implementations, Huffman coding is used to encode the quantized inter-view semantic feature information based on the information content of each symbol, and then the encoded information is used... Hierarchical encoding is performed to obtain basic layer features and enhancement layer features.
[0043] Thus, based on the semantic features after Huffman coding The importance, information content, or contribution to reconstruction quality of the quantified features are categorized into a base layer and an enhancement layer to optimize transmission efficiency and receiver reconstruction quality. Information importance represents the contribution to the task, i.e., the amount of information contained in the information within the entropy model described above. Channel adaptability indicates that the base layer is designed to transmit and maintain basic reconstruction quality even in low-bandwidth or high-noise channels, potentially employing stronger error correction mechanisms or lower code rates during encoding. The enhancement layer, on the other hand, transmits under better channel conditions, supplementing details to improve reconstruction quality. Reconstruction quality and scalability indicate that the base layer provides a low-resolution or low-quality initial reconstruction sufficient to support basic semantic understanding or task requirements. The enhancement layer provides residual information for further refining the reconstruction results, enabling scalable coding. The layering standard ensures that the base layer is independently decodeable, while the enhancement layer relies on the base layer for incremental improvements.
[0044] In some implementations, after Huffman coding Input to the adversarial network, which then processes the data according to Huffman coding. Output base layer features Based on the base layer features and Huffman coding Determine the features of the enhancement layer.
[0045] In some implementations, by calculating the Huffman code... The residual features of the base layer features are used as enhancement layer features, as shown in equation (7): , (7) in, These are residual features, also known as enhancement layer features. These are the semantic features after Huffman coding. These are basic layer features.
[0046] In some implementations, determining the transmission data stream of the initial multi-view video based on the first coding feature and the channel information of the first channel includes: The first spliced feature is obtained by concatenating the base layer features, enhancement layer features, and channel noise features of the first channel. The first stitching feature is input into the first network model, and the first network model generates a transmission data stream of the initial multi-view video adapted to the first channel based on the first stitching feature.
[0047] In some implementations, the base layer features and the channel noise features of the first channel are concatenated to obtain a first concatenated sub-feature, and the enhancement layer features and the channel noise features of the first channel are concatenated to obtain a second concatenated feature. The first and second concatenated sub-features are input into a first network model, and the first network model generates the transmission data stream of the initial multi-view video based on the first and second concatenated sub-features. Here, the first concatenated feature includes both the first and second concatenated sub-features.
[0048] In some implementations, the first network model is a JSCC model.
[0049] In some implementations, the expression for obtaining the initial multi-view video transmission data stream is shown in equation (8): , (8) in, express This represents channel adaptation based on the features of the base layer and enhancement layer, respectively, to generate corresponding transmission data streams. Channel noise characteristics, For transmitting data streams.
[0050] In some embodiments, the method further includes: Obtain the channel state information of the first channel, and update the first network model based on the channel state information.
[0051] In some implementations, the parameters of the JSCC model are dynamically adjusted based on feedback from the channel state information (CSI) of the first channel. Specifically, the input, internal processing, and output of the JSCC model are optimized through a neural network-driven adaptive module. Here, the neural network-driven adaptive module is a module within the JSCC model.
[0052] Here, updating the JSCC model involves adjustments at three levels: parameter-level adjustments, policy-level adjustments, and resource-level adjustments. Specifically, parameter-level adjustments optimize the internal parameters of the JSCC model. Based on CSI feedback, network parameters are dynamically adjusted to minimize end-to-end loss. For example, online learning algorithms can be used to fine-tune the weights in the JSCC model based on CSI; for instance, when SNR is low, the weights of noise-robust layers in the network can be increased to strengthen the support for noise-robust layers. Suppression; at high SNR, these weights are reduced to improve detail coding accuracy. In policy-level adjustments, adaptive coding rules and priority scheduling are used, adjusting not only model parameters but also real-time switching of coding strategies to match channel dynamics. CSI feedback triggers the rule base, with adaptive code rate and modulation; at low SNR, the overall code rate is reduced, and priority coding is prioritized. For symbols of medium to high importance, use low-order modulation such as BPSK; for high SNR, increase the code rate and enable high-order modulation such as 64-QAM. To transmit residual details. Furthermore, policy and CSI adjustments can be made to address attention and masking mechanisms. In the JSCC Transformer module, attention mask weights in the JSCC model are adjusted based on CSI to dynamically increase the attention score of key symbols and suppress noise-sensitive symbols to adapt to the current channel state. In resource-level adjustments, channel and power allocation optimization involves physical layer resources for global efficiency. Based on CSI-based channel gain, higher power is allocated to bearers. The system utilizes subcarriers and implements a feedback closed loop. The receiver transmits CSI via the uplink channel, and the transmitter updates parameters every transmission cycle. A CSI prediction module is used to estimate future fading and adjust in advance to avoid sudden noise. End-to-end training support is employed. During training, a simulated CSI dataset is used to optimize JSCC, enabling the adjustment module to learn robust policies. During inference, parameter updates are achieved through few-sample fine-tuning, avoiding the overhead of full-network retraining.
[0053] Step 103: Transmit the initial multi-view video transmission data stream through the first channel, and obtain the multi-view synchronized video corresponding to the initial multi-view video based on the received transmission data stream.
[0054] In some implementations, the aforementioned transmitted data stream is input into a first channel, and the receiving end of the first channel obtains the multi-view synchronized video corresponding to the initial multi-view video based on the received transmitted data stream.
[0055] In some implementations, the received transmission data stream includes base layer data and enhancement layer data; obtaining the multi-view synchronized video corresponding to the initial multi-view video based on the received transmission data stream includes: A preliminary feature map is determined based on the base layer data; Determine the detailed feature map based on the enhancement layer data; The final feature map is determined based on the initial feature map and the detailed feature map; The multi-view synchronized video corresponding to the initial multi-view video is recovered based on the final feature map.
[0056] In some implementations, at the receiving end of the first channel, a preliminary feature map is obtained by decoding the base layer data in the received transmission data stream. A detailed feature map is obtained by decoding the received enhancement layer data. The preliminary feature map and the detailed feature map are superimposed to generate a final feature map. The generator then uses the final feature map to recover the multi-view synchronized video corresponding to the initial multi-view video.
[0057] In some embodiments, the method further includes: The video evaluation index is determined based on the initial multi-view video and the multi-view synchronized video. A loss function is determined based on the video evaluation metrics, and the loss function is used to evaluate the reconstruction quality of the multi-view synchronized video. The first network model and the second network model are updated according to the loss function, wherein the first network model is used to generate the transmission data stream of the initial multi-view video, and the second network model is used to obtain the feature information set of the initial multi-view video.
[0058] Here, the first network model is the JSCC model, and the second network model is the aforementioned multidimensional Swing Transformer model.
[0059] In some implementations, the video evaluation metrics include one or more of the following: geometric loss metrics. Appearance loss index and combat loss indicators .
[0060] In some implementations, the geometric loss metric is obtained as shown in equation (9): , (9) in, For multi-view synchronized video, This is the initial multi-view video. The coordinates of points in the feature point set representing a video frame in a multi-view synchronized video. This represents the nearest neighbor point of the video frame in the initial multi-view video. Here, the multi-view synchronized video can also be called the reconstructed video, and the initial multi-view video can also be called the real video. The feature point is the coordinate point corresponding to the key object in the video frame. For example, when a cup and a wall appear in the video frame, the cup can be used as a feature point for easy comparison. The physical meaning is to quantify the deviation of the reconstructed video in the spatiotemporal structure (object boundaries or motion paths) to ensure the capture of dynamic geometric information. It is obtained by comparing the model output with the labeled real... In comparison, the minimum sum of squared distances is calculated using nearest neighbor search, and this is computed in real time during training to optimize inter-frame compensation.
[0061] In some implementations, appearance loss index The method of obtaining is shown in equation (10): , (10) Among them, It is a pixel matrix of multi-view synchronized video. This is the initial multi-view video. SSIM measures the brightness, contrast, and structural similarity of video frames; physically, it evaluates the visual fidelity of the reconstructed frames, ensuring the video is natural and smooth. It is obtained by rendering features through a decoding module. Then, the input is used to calculate the SSIM function, which is combined with pixel loss in each iteration to improve the consistency of multi-view fusion.
[0062] In some implementations, the resistance loss index The method for obtaining it is shown in equation (11): , (11) in, It is a discriminator network that outputs multi-view synchronized video. The probability of being judged as real, in physical terms, drives the generator to produce reconstructed frames that are difficult to distinguish from real video, improving spatiotemporal distribution matching and dynamic realism, especially enhancing robustness in noisy transmission. It is obtained during the GAN loop, where the discriminator... and In binary classification training, the generator minimizes the loss, calculates the expected value through mini-batch sampling, and alternates with other losses for optimization. It can be understood that the GAN recurrent network here is a separate network.
[0063] In some implementations, a loss function is determined based on at least one of the video evaluation metrics.
[0064] In some implementations, the loss function is determined based on three video evaluation metrics as shown in Equation (12): , (12) Where is the loss function. Geometric loss metric As an indicator of appearance loss, To combat the loss indicator, , , The weight parameters for each indicator can be determined through experimental optimization, and can be specifically determined according to the actual situation. This application does not impose specific limitations on this.
[0065] It is understandable that the loss function is determined based on three video evaluation metrics, but one or two video evaluation metrics can also be selected to determine the loss function.
[0066] In some implementations, other dimensions can also be used to evaluate the reconstruction quality of multi-view synchronized video. Specifically, quantitative indicators such as peak signal-to-noise ratio (PSNR), SSIM, and MS-SSIM can be used for evaluation, and performance bottlenecks can be identified through heatmaps and reconstruction error curves.
[0067] In some implementations, TensorRT is used to accelerate the inference of the aforementioned multidimensional Swing Transformer model and JSCC model, reducing computational latency. Combined with model pruning techniques, the number of parameters in the multidimensional Swing Transformer model and JSCC model is reduced, thereby improving operational efficiency.
[0068] In some implementations, the Adam optimizer is used for extraction and updating, dynamically adjusting the learning rates of the multidimensional SwinTransformer model and the JSCC model, wherein the adjustment method is shown in Equation (13): , (13) in, Based on the previous learning rate, For the updated learning rate, For time, This is the temperature coefficient.
[0069] In some implementations, during the parameter updating of the multidimensional Swing Transformer model and the JSCC model, when the loss function of consecutive iterations falls below a first threshold... When the time is right, terminate the training of the model and output the optimal model parameters of the two models.
[0070] The technical solution of this application embodiment obtains a feature information set of an initial multi-view video, wherein the initial multi-view video includes initial videos from at least two perspectives, and the feature information set includes intra-frame semantic feature information, inter-frame semantic feature information, and inter-view semantic feature information of the initial videos from each perspective. Based on the feature information set of the initial multi-view video and the channel information of a first channel, a transmission data stream corresponding to the initial multi-view video is determined, where the first channel is the channel for transmitting the initial multi-view video. The transmission data stream of the initial multi-view video is transmitted through the first channel, and a multi-view synchronized video corresponding to the initial multi-view video is obtained based on the received transmission data stream. Thus, multi-view video transmission is realized. By extracting semantic features from the multi-view video to determine the transmission data stream, information redundancy between multiple perspectives is effectively reduced, achieving efficient transmission of multi-view video. Determining the transmission data stream through semantic feature information and channel information can effectively improve transmission robustness in low signal-to-noise ratio environments and adapt to dynamic channels.
[0071] Based on this, the video communication method provided in the embodiments of this application will be further described.
[0072] This application proposes a multi-view video semantic communication system, which achieves efficient and robust multi-view video transmission through multi-view semantic feature extraction, multi-dimensional Swing Transformer lightweight modeling, and source-channel joint coding technology.
[0073] Figure 2 This is a schematic diagram of the overall architecture of the multi-view video semantic communication system provided in the embodiments of this application, such as... Figure 2 As shown, the multi-view video input multi-view semantic feature extraction module acquires multi-view information and performs intra-frame semantic feature extraction, inter-frame semantic feature extraction, and inter-view semantic feature extraction to obtain multi-view joint features. After the multi-view joint features are input into the JSCC encoder, they are input into the physical channel, namely the aforementioned first channel. At the receiving end of the channel, the JSCC decoder is used to perform semantic decoding and reconstruction to obtain the reconstructed video. The performance of the system is evaluated based on the reconstructed video.
[0074] In some implementations, a multi-view video semantic communication system mainly includes a multi-view semantic feature extraction module, a source-channel joint coding module, a semantic decoding and reconstruction module, and a performance evaluation and optimization module. Specifically, the multi-view semantic feature extraction module uses an attention mechanism to extract intra-frame, inter-frame, and inter-view semantic features of synchronized multi-view video, and processes intra-frame spatial, inter-frame temporal, and inter-view correlation information to generate high-dimensional semantic features. The source-channel joint coding module performs source-channel joint coding on the processed high-dimensional semantic features to generate a transmission data stream adapted to the channel. The semantic decoding and reconstruction module reconstructs the synchronized multi-view video based on the received feature stream. The performance evaluation and optimization module evaluates system performance and optimizes the algorithm using multi-dimensional metrics.
[0075] The specific implementation process is as follows: Step 1: Parameter initialization.
[0076] Input multi-view synchronized video into a multi-view video semantic communication system, and set the hyperparameters of the multi-view video semantic communication system. Specifically, the parameters include the number of views. Frame sequence length, length of GOP corresponding to each frame sequence The number of layers and window size of the Swin Transformer (2D: 16×16, 3D: 8×8×4, 4D: 4×4×4×4), and the initial learning rate. Convergence threshold .
[0077] Based on the hyperparameters and channel state of the channel transmitting multi-view video, the multidimensional SwingTransformer model and JSCC encoder are initialized. Here, the JSCC encoder can also be called the JSCC model.
[0078] The channel state is incorporated into the initialization of the multidimensional Swin Transformer model and the JSCC encoder primarily because the goal of JSCC is to combine video coding with channel transmission optimization to achieve robust transmission performance under various channel conditions (noise, bandwidth limitations, fading, etc.). The channel state directly affects how the encoder maps video features to channel-acceptable signals and how the model adapts to channel characteristics in the early stages of training to optimize end-to-end performance. This is also the basis for "comprehensively considering the characteristics of the source and the wireless channel" in step 4.
[0079] Step 2: Multi-view semantic feature extraction and modeling.
[0080] Semantic features are extracted from multi-view synchronized videos using a multi-dimensional Swin Transformer. Figure 3 This is a schematic diagram of the framework for extracting semantic features provided in an embodiment of this application, such as... Figure 3 As shown, the intra-frame information of the video frames in a multi-view video is... A two-dimensional Swing Transformer is used as the intra-frame feature extraction module to extract intra-frame semantic features. The 3D Swing Transformer, as an inter-frame feature extraction module, extracts inter-frame semantic features based on intra-frame semantic features and inter-frame information. The four-dimensional Swin Transformer, as an inter-view feature extraction module, determines the joint semantic features of multiple views based on inter-frame semantic features and inter-view information. Specifically, the following operations are performed on multi-view synchronized videos to extract semantic features: Intra-frame information is processed using a two-dimensional Swing Transformer: a single-frame image from a single viewpoint is processed to extract intra-frame semantic features. Specifically, static elements are identified through edge detection and region segmentation, and features are extracted using local binary patterns and Gabor filtering, as shown in Equation (1).
[0081] Using a 3D Swing Transformer to process inter-frame information: A 3D Swing Transformer is used to process frame sequences from the same viewpoint, capturing semantic associations in the temporal dimension and generating inter-frame features. Specifically, it is based on GOP (Group of Pictures) partitioning, using cosine similarity to measure invariant elements, and combining motion compensation to predict dynamic changes. For each GOP, the complete features of keyframes are extracted, and the difference features of the predicted frames are calculated to optimize coding efficiency. As shown in Equation (2), where, Used for feature extraction between viewpoints and temporal dynamic modeling in 4D Swing Transformer.
[0082] The four-dimensional Swing Transformer is used to process inter-view information: the inter-frame features of all views are jointly processed to extract inter-view correlation information (i.e., content relevance information, measured by feature similarity), and joint inter-view features are generated. As shown in equation (3).
[0083] In the process of extracting inter-view correlation information and generating joint inter-view features, the weights of inter-view features are dynamically adjusted through an attention mask to improve feature extraction efficiency. Specifically, when the four-dimensional Swin Transformer processes inter-view information, an attention mask mechanism is introduced to dynamically adjust the inter-frame features of different views. The contribution weights are determined to optimize the efficiency and accuracy of feature extraction.
[0084] Attention mask generation: Based on the content relevance between viewpoints, calculate the final joint features for each viewpoint. The contribution weights are then determined. An attention mask matrix is generated based on the content relevance between viewpoints and the contribution weights. ,in Indicates the first Inter-frame features from the first perspective on the second Attention weights for each viewpoint are assigned, with mask values ranging from [0,1]. Larger values indicate a higher contribution of that viewpoint to the joint features. A multi-head self-attention mechanism is then used in the four-dimensional Swing Transformer to process these weights. Attention mask It is incorporated into the self-attention calculation, and the specific formula is shown in equation (4).
[0085] Step 3: Source-channel joint coding (JSCC).
[0086] Figure 4 This is a schematic diagram of a JSCC-based video end-to-end semantic communication framework provided in the embodiments of this application, as shown below. Figure 4 The diagram illustrates the semantic compression and channel adaptation process for a single-channel video stream. This application extends this framework to multi-channel video streams, optimizing coding by incorporating multi-view information. For single-channel video, considering the characteristics of both the source and the wireless channel, an efficient joint coding scheme is designed to map the video stream to a high-dimensional space through a nonlinear transformation (the object of this mapping operation is the video stream, i.e.,...). This refers to the raw video data or its low-level representation (such as pixel values or initial features), and this mapping operation occurs before JSCC semantic processing. Then, the JSCC encoder performs semantic compression to extract latent semantic information. Subsequently, an entropy model is applied to model the probability distribution of each semantic symbol, calculating its information content to measure its importance. Based on this information content estimation, a semantically aware variable-length coding strategy is adopted, adaptively adjusting bandwidth and transmission rate to prioritize the transmission of key semantic symbols, thereby reducing redundant information transmission. This design can dynamically adapt to channel changes, ensuring the transmission quality of key information and constructing an end-to-end communication system that optimizes data transmission performance.
[0087] An end-to-end semantic communication framework based on JSCC is used to extract semantic features. The specific process for joint encoding is as follows: Feature quantization: for Perform nearest neighbor quantization to generate quantized semantic features. As shown in equation (5).
[0088] Entropy Modeling and Semantic Aware Encoding: The information content of each semantic symbol is estimated using an entropy model, as shown in Equation (6). In the entropy model, q represents a single semantic symbol. Multi-perspective joint features The discrete symbol set obtained after quantization, where each q is A quantization symbol q corresponds to a center c in the set of quantization centers C. The probability distribution p(q) represents the probability of each semantic symbol q appearing, calculated as follows: First, statistical analysis is performed on the training data... The frequency of occurrence of each symbol in the text is analyzed to construct an empirical distribution, and a deep learning model is used to analyze it. The potential distribution is modeled to estimate p(q). In this architecture, p(q) is predicted by an entropy model, which combines the feature distribution in the training data with contextual information.
[0089] Each semantic symbol calculated based on the entropy model The information content is determined using Huffman coding, assigning codes of varying lengths. The information content reflects the symbol's... Uncertainty or rarity, probability The lower the probability, the greater the information content, and the longer the encoding length should be; conversely, semantic symbols with high probability have less information content, so they should be assigned shorter encodings.
[0090] Hierarchical encoding: based on semantic features after Huffman coding The importance, information content, or contribution to reconstruction quality are used to divide the quantified features into a base layer and an enhancement layer to optimize transmission efficiency and receiver reconstruction quality.
[0091] Information importance refers to its contribution to the task, i.e., the amount of information contained in the information in the entropy model described above. Channel adaptability is designed for the base layer to transmit and maintain basic reconstruction quality even in low-bandwidth or high-noise channels, potentially employing stronger error correction mechanisms or lower code rates during encoding. The enhancement layer transmits under better channel conditions, supplementing details to improve reconstruction quality. Reconstruction quality and scalability are determined by the base layer providing a low-resolution or low-quality initial reconstruction sufficient to support basic semantic understanding or task requirements. The enhancement layer provides residual information to further refine the reconstruction results, enabling scalable coding. The hierarchical standard ensures that the base layer is independently decodable, while the enhancement layer relies on the base layer for incremental improvements.
[0092] Base layer: By using semantic features The input is fed into an adversarial network, which generates a preliminary reconstructed feature map. (i.e., the base layer) and transmit.
[0093] Enhancement layer: Calculate residual features And transmit, as shown in equation (7). Channel adaptation: The encoded base layer features and enhancement layer features are concatenated with the channel noise features and input into the JSCC model to generate the transmission data stream, as shown in equation (8). Based on the feedback of Channel State Information (CSI), the coding parameters (parameters of the JSCC encoder) are dynamically adjusted. Specifically, the input, internal processing, and output of the JSCC model are optimized by an adaptive module driven by a neural network in the JSCC encoder.
[0094] The adjustments are divided into three levels: parameter level, strategy level, and resource level.
[0095] Parameter-level tuning optimizes the intrinsic parameters of the JSCC network model. Based on CSI feedback, network parameters are dynamically adjusted to minimize end-to-end loss. An online learning algorithm is used to fine-tune the encoder weights of JSCC based on CSI; for example, when SNR is low, the weights of noise-robust layers in the network are increased to strengthen the response to noise. Suppression; at high SNR, reduce these weights to improve detail coding accuracy.
[0096] In policy-level adjustments, adaptive coding rules and priority scheduling are used. Adjustments extend beyond model parameters to include real-time switching of coding strategies to match channel dynamics. CSI feedback triggers a rule base, with adaptive code rate and modulation. At low SNR, the overall code rate is reduced, prioritizing coding at lower speeds. For symbols of medium to high importance, use low-order modulation such as BPSK; for high SNR, increase the code rate and enable high-order modulation such as 64-QAM. To transmit residual details. In addition, policy and CSI adjustments can be made to the attention and masking mechanisms. In the Transformer module of JSCC, attention mask weights are adjusted based on CSI to dynamically increase the attention score of key symbols and suppress noise-sensitive symbols to adapt to the current channel conditions.
[0097] In resource-level adjustments, channel and power allocation optimization involves physical layer resources for global efficiency. Based on CSI-based channel gain, higher power is allocated to bearers. The system utilizes subcarriers. Simultaneous feedback loop implementation is achieved, with the receiver transmitting CSI via the uplink channel and the transmitter updating parameters every transmission cycle. A CSI prediction module is used to predict future fading and adjust in advance to avoid sudden noise. End-to-end training support is employed; during training, a simulated CSI dataset is used to optimize JSCC, enabling the adjustment module to learn a robust policy. During inference, parameter updates are achieved through few-sample fine-tuning, avoiding the overhead of full-network retraining. Through these adjustments, the system forms a closed-loop adaptive mechanism: CSI feedback, parameter / policy update, and JSCC regeneration. Transmission and monitoring.
[0098] Step 4: Semantic decoding and reconstruction.
[0099] Perform the following operations at the receiving end: Base layer decoding: based on the received base layer data Restore the initial feature map.
[0100] Enhancement layer decoding: decoding the enhancement layer residual data Perform super-resolution reconstruction to restore high-frequency details.
[0101] Feature fusion: The features of the base layer and the enhancement layer are superimposed to generate the final feature map, as shown in Equation (14): , (14) Video reconstruction: The multi-view synchronized video is restored by a generator, as shown in Equation (15): , (15) Step 5: Reconstruction, Verification, and Optimization.
[0102] The reconstruction quality is evaluated using the loss function in equation (12). The geometric loss is calculated as shown in equation (9), where This represents the coordinates of points in the feature point set of the reconstructed video frame. This represents the nearest neighbor of a ground truth video frame; physically, it quantifies the deviation of the reconstructed video from its spatiotemporal structure (object boundaries or motion paths), ensuring the capture of dynamic geometric information. It is obtained by comparing the model output with the labeled ground truth... In comparison, the minimum sum of squared distances is calculated using nearest neighbor search, and this is computed in real time during training to optimize inter-frame compensation. For appearance loss, based on the SSIM index, the calculation method is shown in equation (10), where It reconstructs the pixel matrix of the video frames. These are real frames. SSIM measures the brightness, contrast, and structural similarity of video frames; physically, it evaluates the visual fidelity of the reconstructed frames, ensuring the video appears natural and smooth. It is obtained by rendering features through a decoding module. Then, the input is used to calculate the SSIM function, which is combined with pixel loss in each iteration to improve the consistency of multi-view fusion. To counteract the loss, a generative adversarial network is used, and the calculation method is shown in equation (11), where It is a discriminator network that outputs a reconstructed frame sequence. The probability of being judged as real, in physical terms, drives the generator to produce reconstructed frames that are difficult to distinguish from real video, improving spatiotemporal distribution matching and dynamic realism, especially enhancing robustness in noisy transmission. It is obtained during the GAN loop, where the discriminator... and In binary classification training, the generator minimizes the loss, calculates the expected value through mini-batch sampling, and optimizes it alternately with other losses. Hyperparameters were determined through experimental optimization.
[0103] In addition, since the sender transmits the data stream based on semantic importance, the following methods can be used to avoid potential out-of-order issues at the receiver: First, at the encoding end, sequence identifiers are embedded in the data stream of each layer. RTP (Real-Time Transport Protocol) or custom JSCC header information is used to mark the priority and dependencies of base layer packets, ensuring that the receiver buffer is reassembled in order. Second, the receiver implements a layered decoding buffer, using a conditional decoding strategy. Enhancement layer residuals are only processed after the base layer has been fully decoded (verified by the discriminator of the adversarial network). (3) To ensure the integrity of the data stream; third, in conjunction with channel feedback, when the channel state is poor, priority is given to retransmitting the base layer data stream, while discarding or delaying the enhancement layer; finally, in the end-to-end optimization, a neural network decoder is incorporated, allowing partially out-of-order inputs to be adaptively sorted through an attention mechanism to estimate missing parts and minimize reconstruction loss. Multi-layered safeguards are used in the system to ensure decoding robustness, maintaining the orderly reconstruction of semantic information even under dynamic changes in the wireless channel.
[0104] In addition, multi-dimensional evaluation methods can be used to assess video reconstruction quality. These include quantification metrics such as PSNR (Peak Signal-to-Noise Ratio), SSIM, and MS-SSIM; and performance bottlenecks can be identified through heatmaps and reconstruction error curves for visualized analysis.
[0105] Optimization strategies can also be used to optimize the aforementioned multidimensional Swin Transformer and JSCC models, such as using TensorRT to accelerate neural network inference and reduce computational latency. Here, the neural network is the multidimensional Swin Transformer and JSCC models. Combined with model pruning techniques, the number of parameters is reduced, and the running efficiency is improved. Here, the parameters of the multidimensional Swin Transformer and JSCC models are reduced. Gradient updates are performed through the Adam optimizer to dynamically adjust the learning rate of the multidimensional Swin Transformer and JSCC models, as shown in Equation (13). When the loss change of consecutive iterations is lower than the threshold When the time is reached, training is terminated, and the optimal model parameters are output, which are the optimal model parameters for the multidimensional Swin Transformer model and the JSCC model.
[0106] The technical solution of this application addresses the problems of bandwidth resource waste, insufficient adaptability in complex channel environments, and poor system stability in existing multi-view video transmission technologies. It proposes a multi-view video semantic communication system. This system uses a two-dimensional Swing Transformer in a multi-dimensional Swing Transformer architecture to extract intra-frame semantics from the same frame image at different viewpoints, obtaining intra-frame semantic features for each viewpoint. A three-dimensional Swing Transformer performs inter-frame semantic association on frame sequences at different viewpoints, obtaining inter-frame semantic features for each viewpoint. A four-dimensional Swing Transformer jointly processes the inter-frame semantic features of all viewpoints, obtaining joint inter-view features. This achieves joint compression of multi-view video, effectively reducing information redundancy between viewpoints, enabling efficient transmission of multi-view video, reducing bandwidth consumption, and significantly lowering processing and transmission latency. During the generation of joint inter-view features, based on the content relevance between viewpoints, attention masks dynamically adjust the contribution weights of inter-frame semantic features from different viewpoints, thereby optimizing the efficiency and accuracy of feature extraction. After obtaining the joint features between multiple viewpoints, the joint features are then processed based on source-channel joint coding (JSCC). Quantization into semantic features And calculate semantic features based on the entropy model. The amount of information in each symbol is determined so that high-information-content and more important semantic features can be prioritized for subsequent transmission. The corresponding data stream. In addition, based on channel state information (CSI) feedback, a deep neural network is used to learn the optimal mapping between semantic features and channel noise. The channel resource allocation and modulation method are adaptively adjusted according to semantic importance, which significantly improves the transmission robustness and reconstruction quality in low signal-to-noise ratio environments, making the system more adaptable to dynamic channels.
[0107] Specifically, 1) Lightweight Multi-Dimensional Swin Transformer: Extending the Swin Transformer to a multi-dimensional architecture, using 2D, 3D, and 4D Swin Transformers to process intra-frame spatial information, inter-frame temporal information, and inter-viewpoint correlation information respectively, achieving computational decoupling of spatial, temporal, and viewpoint attention, reducing computational complexity, while preserving high-frequency semantic details, improving the transmission efficiency of multi-view video, and enabling real-time transmission of multi-view video. 2) Multi-View Semantic Feature Extraction and Attention Optimization: Extracting semantic features from multi-view synchronized videos through deep learning attention mechanisms, dynamically allocating weights through attention mechanisms, optimizing the feature extraction process, and significantly improving the compression efficiency and accuracy of semantic information. 3) Source-Channel Joint Coding Optimization: Using the Source-Channel Joint Coding (JSCC) optimization scheme, for multi-view video, by jointly optimizing source compression and channel error correction, directly mapping semantic features to channel input, and adaptively adjusting channel resource allocation and modulation methods according to semantic importance, significantly improving transmission robustness and reconstruction quality in low signal-to-noise ratio environments.
[0108] The multi-view video semantic communication system proposed in this application provides an innovative solution for the need for synchronous multi-view video transmission. With the surging demand for efficient and robust multi-view video transmission technology, this application significantly improves transmission efficiency and noise resistance through semantic information decomposition and JSCC technology, demonstrating broad commercial application prospects. The multi-dimensional Swin Transformer and JSCC modules can be integrated into existing video processing platforms, enhancing the product's competitiveness in immersive application scenarios. Through optimization of the multi-dimensional Swin Transformer and JSCC, this application breaks through the pixel-level compression bottleneck of traditional video communication, forming a unique technical path.
[0109] Figure 5 This is a schematic diagram of the structural composition of the video communication device provided in the embodiments of this application, as shown below. Figure 5 As shown, the video communication device includes: The acquisition unit 501 is used to acquire a feature information set of an initial multi-view video, wherein the initial multi-view video includes initial videos of at least two views, and the feature information set includes intra-frame semantic feature information of the initial video of each view, inter-frame semantic feature information of the initial video of each view, and inter-view semantic feature information of the initial multi-view video. The processing unit 502 is configured to determine the transmission data stream corresponding to the initial multi-view video based on the feature information set of the initial multi-view video and the channel information of the first channel, wherein the first channel is the channel for transmitting the initial multi-view video; transmit the transmission data stream of the initial multi-view video through the first channel; and obtain the multi-view synchronized video corresponding to the initial multi-view video based on the received transmission data stream.
[0110] In some embodiments, the acquisition unit 501 is configured to acquire intra-frame semantic feature information of the initial video of each viewpoint in the initial multi-view video through a first model; determine inter-frame semantic feature information of the initial video of each viewpoint in the initial multi-view video through a second model and the intra-frame semantic feature information of the initial video of each viewpoint; and determine inter-view semantic feature information of the initial multi-view video through a third model and the inter-frame semantic feature information of the initial video of each viewpoint.
[0111] In some embodiments, the processing unit 502 is configured to encode the semantic feature information between the viewpoints to obtain a first encoded feature; and to determine the transmission data stream of the initial multi-view video based on the first encoded feature and the channel information of the first channel.
[0112] In some embodiments, the processing unit 502 is configured to quantize the inter-view semantic feature information to obtain quantized inter-view semantic feature information; obtain the information content of each symbol in the quantized inter-view semantic feature information; and perform hierarchical encoding of the quantized inter-view semantic feature information according to the information content of each symbol to obtain basic layer features and enhanced layer features, wherein the first encoded feature includes basic layer features and enhanced layer features.
[0113] In some implementations, the processing unit 502 is configured to concatenate the base layer features, enhancement layer features, and channel noise features of the first channel to obtain a first concatenation feature; input the first concatenation feature into a first network model, and the first network model generates a transmission data stream of the initial multi-view video adapted to the first channel based on the first concatenation feature.
[0114] In some embodiments, the acquisition unit 501 is used to acquire channel state information of the first channel; the processing unit 502 is used to update the first network model according to the channel state information.
[0115] In some implementations, the received transmission data stream includes base layer data and enhancement layer data; the processing unit 502 is configured to determine a preliminary feature map based on the base layer data; determine a detailed feature map based on the enhancement layer data; determine the final feature map based on the initial feature map and the detailed feature map; and recover the multi-view synchronized video corresponding to the initial multi-view video based on the final feature map.
[0116] In some embodiments, the processing unit 502 is configured to determine a video evaluation index based on the initial multi-view video and the multi-view synchronized video; determine a loss function based on the video evaluation index, the loss function being used to evaluate the reconstruction quality of the multi-view synchronized video; and update a first network model and a second network model based on the loss function, wherein the first network model is used to generate the transmission data stream of the initial multi-view video, and the second network model is used to obtain a set of feature information of the initial multi-view video.
[0117] Those skilled in the art should understand that Figure 5 The functions of each unit in the video communication device shown can be understood by referring to the relevant descriptions of the aforementioned method. Figure 5 The functions of each unit in the video communication device shown can be implemented by a program running on a processor or by specific logic circuits.
[0118] Figure 6 This is a schematic structural diagram of a video communication device 600 provided in an embodiment of this application. Figure 6 The video communication device 600 shown includes a processor 610, which can call and run computer programs from memory to implement the methods in the embodiments of this application.
[0119] Optionally, such as Figure 6 As shown, the video communication device 600 may further include a memory 620. The processor 610 can retrieve and run computer programs from the memory 620 to implement the methods described in this embodiment.
[0120] The memory 620 can be a separate device independent of the processor 610, or it can be integrated into the processor 610.
[0121] Optionally, such as Figure 6As shown, the video communication device 600 may also include a transceiver 630, and the processor 610 may control the transceiver 630 to communicate with other devices. Specifically, it may send information or data to other devices or receive information or data sent by other devices.
[0122] The transceiver 630 may include a transmitter and a receiver. The transceiver 630 may further include antennas, and the number of antennas may be one or more.
[0123] The video communication device 600 can implement the corresponding processes implemented by the video communication device in the various methods of the embodiments of this application, which will not be described in detail here for the sake of brevity.
[0124] Figure 7 This is a schematic structural diagram of the chip according to an embodiment of this application. Figure 7 The chip 700 shown includes a processor 710, which can call and run computer programs from memory to implement the methods in the embodiments of this application.
[0125] Optionally, such as Figure 7 As shown, chip 700 may further include memory 720. Processor 710 can retrieve and run computer programs from memory 720 to implement the methods described in this embodiment.
[0126] The memory 720 can be a separate device independent of the processor 710, or it can be integrated into the processor 710.
[0127] Optionally, the chip 700 may also include an input interface 730. The processor 710 can control the input interface 730 to communicate with other devices or chips; specifically, it can acquire information or data sent by other devices or chips.
[0128] Optionally, the chip 700 may also include an output interface 740. The processor 710 can control the output interface 740 to communicate with other devices or chips, specifically, to output information or data to other devices or chips.
[0129] This chip can implement the corresponding processes implemented by the video communication device in the various methods of the embodiments of this application, which will not be described in detail here for the sake of brevity.
[0130] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.
[0131] It should be understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.
[0132] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.
[0133] It should be understood that the above-described memory is exemplary and not a limiting description. For example, the memory in the embodiments of this application may also be static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DR RAM), etc. That is to say, the memory in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.
[0134] This application also provides a computer program product, including a computer program.
[0135] When executed by a processor, the computer program implements the corresponding processes implemented by the video communication device in the various methods of the embodiments of this application, which will not be described in detail here for the sake of brevity.
[0136] This application also provides a computer-readable storage medium for storing computer programs.
[0137] The computer program causes the computer to execute the corresponding processes implemented by the video communication device in the various methods of the embodiments of this application, which will not be described in detail here for the sake of brevity.
[0138] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0139] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0140] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0141] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, the functional units in the various embodiments of this application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0142] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0143] The preferred embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this application, various simple modifications can be made to the technical solutions of this application, and these simple modifications all fall within the protection scope of this application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this application will not describe the various possible combinations separately. Furthermore, various different embodiments of this application can also be arbitrarily combined, as long as they do not violate the spirit of this application, they should also be considered as the content disclosed in this application. Moreover, without conflict, the various embodiments and / or the technical features in the various embodiments described in this application can be arbitrarily combined with the prior art, and the resulting technical solutions should also fall within the protection scope of this application.
[0144] It should be understood that in the various method embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The above description is only a specific implementation method of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.
Claims
1. A method of video communication, characterized by, The method includes: Obtain a feature information set of an initial multi-view video, wherein the initial multi-view video includes initial videos from at least two perspectives, and the feature information set includes intra-frame semantic feature information of the initial video from each perspective, inter-frame semantic feature information of the initial video from each perspective, and inter-view semantic feature information of the initial multi-view video. The transmission data stream corresponding to the initial multi-view video is determined based on the feature information set of the initial multi-view video and the channel information of the first channel, wherein the first channel is the channel for transmitting the initial multi-view video. The initial multi-view video is transmitted through the first channel, and the multi-view synchronized video corresponding to the initial multi-view video is obtained based on the received transmission data stream.
2. The method of claim 1, wherein, The acquisition of the initial multi-view video feature information set includes: The intra-frame semantic feature information of the initial video from each perspective in the initial multi-view video is obtained through the first model; The inter-frame semantic feature information of the initial video of each viewpoint in the initial multi-view video is determined by the second model and the intra-frame semantic feature information of the initial video of each viewpoint. The inter-view semantic feature information of the initial multi-view video is determined by the third model and the inter-frame semantic feature information of the initial video of each view.
3. The method of claim 2, wherein, The step of determining the transmission data stream corresponding to the initial multi-view video based on the feature information set of the initial multi-view video and the channel information of the first channel includes: The semantic feature information between the viewpoints is encoded to obtain the first encoded feature; The transmission data stream of the initial multi-view video is determined based on the first encoding feature and the channel information of the first channel.
4. The method of claim 3, wherein, The step of encoding the semantic feature information between the viewpoints to obtain the first encoded feature includes: The semantic feature information between the viewpoints is quantized to obtain the quantized semantic feature information between the viewpoints; Obtain the information content of each symbol in the quantified semantic feature information between viewpoints; The quantized inter-view semantic feature information is hierarchically encoded according to the information content of each symbol to obtain basic layer features and enhancement layer features, wherein the first encoded feature includes basic layer features and enhancement layer features; Determining the transmission data stream of the initial multi-view video based on the first encoding feature and the channel information of the first channel includes: The first spliced feature is obtained by concatenating the base layer features, enhancement layer features, and channel noise features of the first channel. The first stitching feature is input into the first network model, and the first network model generates a transmission data stream of the initial multi-view video adapted to the first channel based on the first stitching feature.
5. The method of claim 4, wherein, The method further includes: Obtain the channel state information of the first channel, and update the first network model based on the channel state information.
6. The method of claim 1, wherein, The received transmission data stream includes base layer data and enhancement layer data; obtaining the multi-view synchronized video corresponding to the initial multi-view video based on the received transmission data stream includes: A preliminary feature map is determined based on the base layer data; Determine the detailed feature map based on the enhancement layer data; The final feature map is determined based on the initial feature map and the detailed feature map; The multi-view synchronized video corresponding to the initial multi-view video is recovered based on the final feature map.
7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: The video evaluation index is determined based on the initial multi-view video and the multi-view synchronized video. A loss function is determined based on the video evaluation metrics, and the loss function is used to evaluate the reconstruction quality of the multi-view synchronized video. The first network model and the second network model are updated according to the loss function, wherein the first network model is used to generate the transmission data stream of the initial multi-view video, and the second network model is used to obtain the feature information set of the initial multi-view video.
8. A video communication device, characterized by include: A processor and a memory for storing a computer program, the processor for calling and running the computer program stored in the memory to perform the method as described in any one of claims 1 to 7.
9. A computer program product, characterised in that, include: A computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the method as described in any one of claims 1 to 7.
