A goal-oriented video semantic communication system based on a visual model

Abstract

This invention provides a target-oriented video semantic communication system based on a visual model, comprising: a semantic extractor, based on the SAM2 model, for processing the original video, generating a segmentation mask, and extracting semantic information; a ViMamba encoder for channel coding of the output of the semantic extractor; a channel adaptation module for optimizing the encoded sequence based on the signal-to-noise ratio information of the physical channel; a ViMamba decoder for performing channel decoding to obtain a feature sequence; and a semantic reconstructor for performing semantic reconstruction based on the feature sequence, recovering the data, and outputting the target video. This invention solves the problems of high interference from redundant semantic information in videos, insufficient deep semantic coding capabilities, and poor robustness in complex channel environments. While ensuring semantic integrity, it achieves efficient compression and robust transmission of video data, significantly improving the overall performance and adaptability of semantic communication.

Description

Technical Field

[0001] This invention relates to the field of semantic communication technology, and in particular to a goal-oriented video semantic communication system based on a visual model. Background Technology

[0002] With the widespread application of video data in scenarios such as autonomous driving, telemedicine, and intelligent security, traditional video communication systems are gradually revealing problems such as low transmission efficiency and high communication overhead in complex network environments with limited bandwidth, low latency, and high robustness. To alleviate this contradiction, semantic communication, as a cutting-edge technology integrating perception and communication, has received widespread attention in recent years. Semantic communication transmits key semantic information extracted from the source content, rather than reconstructing data pixel by pixel, thereby significantly compressing the data volume while meeting task accuracy requirements.

[0003] However, existing semantic communication systems mainly include: semantic communication systems based on convolutional neural networks (CNN) and autoencoders: CNN extracts image features and uses autoencoders to compress and reconstruct semantic information. However, it fails to effectively solve the redundancy problem at the pixel level, resulting in large bandwidth consumption; semantic communication systems based on visual Transformer (ViT): ViT uses attention mechanisms to model long-short-term dependencies in images, which is suitable for high resolution and complex scenes. However, the number of parameters in the ViT model is large and the inference speed is slow, making it difficult to meet the requirements of low-latency communication. Summary of the Invention

[0004] This invention addresses the aforementioned problems by providing a goal-oriented video semantic communication system based on a visual model. It solves the issues of high interference from redundant semantic information in videos, insufficient deep semantic coding capabilities, and poor robustness in complex channel environments by introducing a SAM2-based semantic extractor, designing channel coding, channel decoding, and channel adaptation methods based on the visual Mamba framework. While ensuring semantic integrity, it achieves efficient compression and robust transmission of video data, significantly improving the overall performance and adaptability of semantic communication. This system is suitable for goal-oriented application scenarios such as autonomous driving and intelligent monitoring.

[0005] Specifically, the first aspect of the present invention provides a goal-oriented video semantic communication system based on a visual model, comprising:

[0006] The semantic extractor, based on the SAM2 model, includes an image encoder, a cue encoder, a mask decoder, and a memory module, which processes the raw video, generates a segmentation mask, and extracts semantic information.

[0007] The ViMamba encoder combines the Vision Mamba architecture and a downsampling module to perform channel coding on the output of the semantic extractor and perform sender channel adaptation.

[0008] The channel adaptation module, including the transmitting end channel adaptation and the receiving end channel adaptation, is used to optimize the encoded sequence based on the signal-to-noise ratio information of the physical channel. It takes the output of the ViMamba encoder as input and outputs the semantic feature sequence to the ViMamba decoder.

[0009] The ViMamba decoder combines the Vision Mamba architecture and an upsampling module to perform channel decoding on the received semantic feature sequence after receiving-end channel adaptation, thereby obtaining the feature sequence.

[0010] A semantic reconstructor is used to perform semantic reconstruction based on feature sequences, recover data, and output the target video.

[0011] Furthermore, the image encoder functions to encode image features into multi-scale feature representations for a given original video sequence using a hierarchical Hierarchical Vision Transformer.

[0012] For a given video sequence ,in, For video frame rate, Let T be the image of frame T. The image encoder uses a hierarchical Hierarchical Vision Transformer to encode image features into multi-scale feature representations.

[0013] Specifically, the image encoder converts the input image into multi-scale feature maps. , Represents the features of the nth layer;

[0014] The feature fusion process can be represented as:

[0015] ;

[0016] in: The features of the fused image;

[0017] For feature fusion operations;

[0018] Multi-scale feature maps;

[0019] By fusing features at different levels, rich contextual information is provided to the mask decoder, enhancing the model's ability to extract and represent features of objects in images.

[0020] Furthermore, the prompt encoder is configured to encode the prompts input by the user into a feature vector.

[0021] The prompt encoder encodes user-input prompts (such as dots, boxes, or masks) into feature vectors, as shown in the following process:

[0022] ;

[0023] in: The feature vector generated by the prompt encoder represents the semantic embedding of the input prompt;

[0024] To prompt for encoding operations;

[0025] Prompts for user input;

[0026] The feature vector generated by the encoder will be prompted. Integrating into image features In the process, the final fusion characteristics are obtained. The formula is as follows:

[0027] ;

[0028] in: For the final fusion feature;

[0029] For feature fusion operations;

[0030] The features of the fused image;

[0031] This is to provide the feature vector generated by the encoder;

[0032] Furthermore, the memory module is used to store object information and previous segmentation results to track objects during video segmentation.

[0033] The memory update process is represented as:

[0034] ;

[0035] in: Information stored in the memory module for the (t+1)th frame;

[0036] For memory update operations;

[0037] Information stored in the memory module for the t-th frame;

[0038] This represents the segmentation result for frame t.

[0039] When processing subsequent frames, the model can use the information in memory to generate mask predictions, avoiding redundant calculations and improving efficiency.

[0040] Furthermore, the mask decoder is used to combine the features extracted by the image encoder, the prompting information provided by the prompt encoder, and the memory data in the memory module to generate the final segmentation mask, and to segment the moving object through the mask.

[0041] The mask generation process is represented as follows:

[0042] ;

[0043] in: This is the final segmentation mask;

[0044] For mask generation operation;

[0045] The features of the fused image;

[0046] This is to provide the feature vector generated by the encoder;

[0047] For memories within the memory module;

[0048] Through the above process, SAM 2 can generate accurate segmentation masks to achieve image and video segmentation tasks.

[0049] Furthermore, the generation of the segmentation mask and extraction of semantic information includes combining the features extracted by the image encoder, the prompt information provided by the prompt encoder, and the memory data in the memory module to generate the final segmentation mask. The mask is then multiplied by the pixels of the corresponding frame to obtain the moving object image. The same operation is used to obtain the moving object images in all video frames, thereby obtaining target semantic information containing only the moving object.

[0050] The process of segmenting moving objects using a mask is as follows:

[0051] ;

[0052] in: The image of the moving object in the i-th frame;

[0053] Let i be the image of the i-th frame;

[0054] This is a pixel-by-pixel multiplication operation;

[0055] This is the final segmentation mask;

[0056] Segmentation mask Regions with a median value of 1 will retain the pixels from the original image, while regions with a median value of 0 will be set to 0. Accordingly, images of moving objects in all video frames can be obtained, thereby achieving segmentation of moving objects.

[0057] Furthermore, the ViMamba encoder performs channel coding by including the following steps:

[0058] Step 1: The encoder receives the output of the semantic extractor and performs block processing on the target image, dividing it into non-overlapping image blocks, and linearly projects the image blocks into a fixed-dimensional vector;

[0059] The encoder receives the output of the semantic extractor. As output For the moving object image of frame T, first process the image... Perform block processing, where i is less than or equal to T, where... The height of the image. Divide the image into non-overlapping image blocks, given the width of the image. ,in, The size of the block. This represents the number of feature channels.

[0060] Each image block First, it is linearly projected onto a vector of fixed dimension. , The projected dimension is determined by a learnable projection matrix, expressed as follows:

[0061] ;

[0062] in: A vector of fixed dimensions;

[0063] For a learnable projection matrix, , The size of the block. The dimensions after projection;

[0064] Indicates the image block Vectorization;

[0065] Subsequently, to incorporate positional information, position embeddings are added to the projected vector. The final patch embedding is obtained, represented as:

[0066] ;

[0067] in: Embed for the final patch;

[0068] For the projected first One vector;

[0069] For the first projection Change it to a vector with added positional embedding;

[0070] The length of the sequence. , The height of the image. The width of the image. The size of the block;

[0071] Step 2: Using several stacked ViMamba modules and patch embedding layers, based on a bidirectional state space model, high-level semantic features are extracted by learning the semantic relationships between image patches;

[0072] The embedded tokens after patch processing are input into In a stack of ViMamba modules, these The stacked ViMamba modules and patch embedding layers are collectively referred to as the "encoding stage one." Each ViMamba module is based on a bidirectional state space model, extracting high-level semantic features by learning the semantic relationships between image patches. Internally, the ViMamba module captures global semantic information in the image by combining forward and backward state space models, while maintaining sensitivity to local features. Specifically, the processing procedure of the ViMamba module is as follows:

[0073] a) Forward and Backward State Space Models: The ViMamba module processes feature sequences using state space models in both forward and backward directions. The forward model processes the sequence from beginning to end, while the backward model processes it from end to beginning. This bidirectional processing approach captures the contextual information at each position in the sequence.

[0074] b) State update: At time step t, the forward state... and backward state Update using the following formulas respectively:

[0075] ;

[0076] ;

[0077] in: This represents the forward state at time step t;

[0078] This is the state transition matrix of the forward state-space model;

[0079] This represents the forward state at time step t-1;

[0080] This is the input matrix of the forward state-space model;

[0081] Let be the input feature vector at time step t;

[0082] This represents the backward state at time step t;

[0083] This is the state transition matrix of the backward state-space model;

[0084] This represents the backward state at time step t+1;

[0085] This is the input matrix for the backward state-space model;

[0086] c) Output Calculation: Final Output Features Calculated using the following formula:

[0087] ;

[0088] in: For the final output features;

[0089] This is the output matrix of the forward state-space model;

[0090] This represents the final state of the forward state-space model;

[0091] This is the output matrix of the backward state-space model;

[0092] This represents the final state of the backward state-space model.

[0093] Step 3: Merge adjacent image patch features using the downsampling module to generate a new feature sequence;

[0094] To gradually reduce the resolution of the feature sequence and increase the number of channels, we add a downsampling module, patch merging, after each stage. This module generates a new feature sequence by merging features from adjacent image patches, as detailed below:

[0095] ;

[0096] in: For new feature sequences;

[0097] This is a downsampling module operation, short for Residual Block with Downsampling, which helps the model capture the semantic information of images at different scales and gradually build high-level feature representations.

[0098] For the final output features;

[0099] Step 4: After multi-stage processing by the ViMamba module and downsampling module, multi-scale feature representations are obtained, and finally feature encoding is obtained.

[0100] The encoder consists of multiple stages, each including multiple ViMamba modules and a downsampling module. Specifically, the adjacent embeddings output from the first stage are merged in the second stage using a patch merging operation, resulting in a size of [size missing]. The connection embedding is reduced to a smaller size. Subsequently, , The height of the image. The width of the image. Each patch embedding tokens is input. In a ViMamba block.

[0101] Through each stage of processing, the resolution of the feature sequence gradually decreases while the number of channels gradually increases, eventually approaching a representation that is conducive to channel transmission. By employing a multi-stage processing approach, the proposed model significantly improves its capacity because it can capture long-range dependencies, utilize global information, and efficiently learn complex details in high-resolution images.

[0102] The encoder combines the efficient visual representation capabilities of Vision Mamba with the global semantic capture capabilities of the bidirectional state space model, while gradually constructing multi-scale feature representations through a downsampling module.

[0103] Furthermore, the transmitting end channel adaptation is a precoding network based on SNR feedback. By learning the mapping relationship of SNR parameters, the weights of the received feature-coded sequences are adjusted, and finally all channel weights are fused. Then, the weights are multiplied by the module input and the channels are shuffled to obtain the output.

[0104] The channel adaptation process at the transmitting end can be represented as:

[0105] ;

[0106] in: The adjusted encoded sequence;

[0107] For Encode Channel Adaptation operation, it represents a precoding network based on SNR feedback, which adjusts the weights of the encoded sequence by learning the mapping relationship of SNR parameters;

[0108] This is the output of the last stage of the encoder;

[0109] Signal-to-noise ratio (SNR) is a core indicator that measures the ratio of signal power to noise power.

[0110] The adjusted coded sequence will be transmitted to the receiving end as the channel input signal.

[0111] The channel adaptation module ensures that the coded sequence can be optimized according to the actual channel conditions, improving the system's adaptability and transmission efficiency. Details of the channel adaptation module are as follows:

[0112] The proposed channel adaptation module combines multi-scale feature fusion and channel attention mechanisms, considering channel features at the individual feature level while also incorporating spatial considerations at the sequence level. Specifically, it organizes the feature-encoded sequence into a feature map. ,in, These represent the outputs of the last stage of the encoder. The embedding dimensions, height, and width are then organized into a feature map.

[0113] First, the module convolves the feature map X with 1×1, 3×3, 5×5, and 7×7 convolution kernels, obtaining four tensors with different receptive fields. Their sizes are all , These represent the outputs of the last stage of the encoder. The embedding dimensions, height, and width are organized into feature maps, and then... Combined to obtain .

[0114] Next, each tensor is divided into two groups along the channel dimension, with each group having a size of [size missing]. The grouping operation process can be represented as:

[0115] ;

[0116] in: For grouping operations, this means dividing each tensor along the channel dimension. Divided into two groups, each group being of size . ;

[0117] , In order to Feature maps after grouping;

[0118] Then, two extraction modules with different kernel sizes [3×3, 5×5] are used to obtain the channel weights for each group. , Then, the two are combined along the channel dimension to obtain the weights of each tensor.

[0119] By analyzing the input feature map Perform global average pooling operation to obtain To avoid making the model too complex, it is compressed and swapped. , and then obtain Next, we use a one-dimensional convolutional network layer with a kernel size of k to achieve local cross-channel interaction to obtain... Obtained through the sigmoid activation function Finally, unzip and exchange. , and then obtain The extraction module formula is as follows:

[0120] ;

[0121] in: To The output after extraction;

[0122] To achieve adaptive average pooling, obtain ;

[0123] For compression and swapping operations, compression is achieved using a double fully connected layer structure consisting of a fully connected layer, a ReLU nonlinear activation layer, another fully connected layer, and a sigmoid function. Then, the spatial and channel dimension orders are swapped to obtain... ;

[0124] This is a convolution operation with a kernel of size k×k, where k∈[3,5];

[0125] For sigmoid operations;

[0126] For decompression and exchange operations, it is the reverse process of compression and exchange, which will... The shape is restored from [1,c] to [c,1,1] to facilitate subsequent processing;

[0127] Input feature map;

[0128] Subsequently, the channel weights of each group obtained by the extraction module are concatenated using the following formula:

[0129] ;

[0130] in: For the concatenation operation, it is represented as... , Concatenate along the channel dimension to obtain the feature map. , ;

[0131] To The output of the channel-coded signal after feature extraction;

[0132] To The output of the channel-coded signal after feature extraction;

[0133] right After extraction and concatenation, we can obtain 5 weighted reorganizations. .

[0134] Then we multiply each weight by Obtain the right to reorganize , The channel feature tensor obtained after passing through the SNR projection module is a feedforward neural network with three fully connected layers. This module aims to expand and map the input SNR value into a vector of the same dimension as the backbone network's feature vector. The first layer maps a single SNR value to a higher-dimensional space, then uses the LeakyReLU activation function. Compared to traditional ReLU, it has a non-zero gradient on the negative half-axis, effectively preventing neuron death and maintaining continuous information transmission. The second fully connected layer continues the nonlinear mapping in the expanded high-dimensional space, also using LeakyReLU activation, further enhancing the model's expressive power and robustness. The final fully connected layer maps the data back to the target dimension and uses the Sigmoid activation function to constrain the output between 0 and 1, allowing the generated vector to be used as weights in subsequent modulation operations. This design enables smooth, stable, and effective feature extraction from the input SNR signal, providing a normalized and nonlinearly transformed modulation factor. The SNR projection module is as follows:

[0135] ;

[0136] in: For sigmoid operations;

[0137] For LeakyReLU operations;

[0138] and These are the weights and biases of the Linear array, respectively.

[0139] Finally, the weights of all 5 channels are merged. The merged weights are then compared with the module input. Multiply by channel-by-channel multiplication, followed by channel shuffling. After processing, the final output of the channel adaptation module can be expressed as:

[0140] ;

[0141] in: This is the output of the channel adaptation module;

[0142] Shuffling operation for the channel;

[0143] This is a multiplication operation;

[0144] For the i-th weight reorganization;

[0145] This is the input for the channel adaptation module;

[0146] The channel adaptation module adjusts the coding based on the channel's signal-to-noise ratio (SNR) information to adapt to different channel conditions, thereby enhancing the system's robustness in dynamic channel environments, especially under low SNR conditions.

[0147] After channel adaptation at the transmitting end, transmission must be performed through a wireless physical channel. During this process, the signal will inevitably be affected by channel noise and attenuation. This process can be represented as follows:

[0148] ;

[0149] in: The symbols received by the final receiving end;

[0150] Channel gain;

[0151] The symbols encoded by the sending end;

[0152] It is additive white noise (AWGN);

[0153] Furthermore, the receiving end channel adaptation is implemented in the same way as the transmitting end channel adaptation, and its function is to optimize the coding sequence according to the signal-to-noise ratio information of the channel to adapt to different channel conditions.

[0154] The specific steps are as follows:

[0155] ;

[0156] in: The characteristic sequence of the channel output;

[0157] For Decode Channel Adaptation operation, it means that the decoding channel adaptation module based on SNR feedback adjusts the weights of the encoded sequence by learning the mapping relationship of SNR parameters;

[0158] The symbols received by the final receiving end;

[0159] Signal-to-noise ratio (SNR) is a core indicator that measures the ratio of signal power to noise power.

[0160] Furthermore, the ViMamba decoder performs channel decoding by including the following steps:

[0161] Step 1: Using several ViMamba modules based on a bidirectional state-space model, high-level semantic features are extracted by learning the semantic associations in the feature sequence;

[0162] Step 2: The upsampling module generates new feature sequences by segmenting and expanding the feature sequences;

[0163] Step 3: After multi-stage processing by the ViMamba module and downsampling module, a size and feature representation close to the original image is obtained.

[0164] The decoder, another crucial component of the semantic communication system, is responsible for restoring the received semantic feature sequence to the original image. The decoder design combines the advanced Vision Mamb architecture and upsampling modules to achieve efficient feature restoration and image reconstruction. The specific implementation process is similar to that of the encoder, as follows:

[0165] 1) ViMamba module processing: Adjusted feature sequences The input is first fed into the ViMamba module of the first stage of the decoder. Each ViMamba module is based on a bidirectional state space model and extracts high-level semantic features by learning the semantic associations in the feature sequence.

[0166] 2) Upsampling Module: To progressively increase the resolution of the feature sequence and reduce the number of channels, we add an upsampling module (patch division) at the end of each stage. The upsampling module generates new feature sequences by segmenting and expanding the feature sequences. Specifically, the upsampling module operates as follows:

[0167] ;

[0168] in: The feature sequence after processing by the upsampling module;

[0169] This is a Residual Block with Upsampling operation, belonging to the upsampling module. It achieves feature dimensionality increase and channel number reduction through convolutional layers and upsampling layers.

[0170] This refers to the feature sequence output by the ViMamba module in the decoder;

[0171] This process helps the model to restore the semantic information of the image at different scales and gradually recover high-level feature representations.

[0172] 3) Multi-stage processing: Similar to the encoder, the decoder also consists of multiple stages, each including multiple ViMamba modules and an upsampling module. Through the processing of each stage, the resolution of the feature sequence gradually increases, the number of channels gradually decreases, and it gradually approaches the size and feature representation of the original image.

[0173] Finally, the feature sequence after all stages of processing Video data of moving objects is recovered using a semantic reconstructor. The reconstructed video data of the moving object can be represented as:

[0174] ;

[0175] in: For recovering video data of moving objects;

[0176] For having a parameter set A semantic reconstructor;

[0177] These are the features ultimately generated after joint source and channel decoding. Attached Figure Description

[0178] To more clearly illustrate the technical solutions in the embodiments of this drawing or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this drawing. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0179] Figure 1 This is a structural diagram of the present invention;

[0180] The purpose, features, and advantages of this accompanying drawing will be further explained in conjunction with the embodiments and with reference to the accompanying drawing. Detailed Implementation

[0181] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments provided by this invention without inventive effort are within the scope of protection of this invention.

[0182] Obviously, the accompanying drawings described below are merely some examples or embodiments of the present invention. Those skilled in the art can apply the present invention to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this invention, modifications to design, manufacturing, or production based on the technical content disclosed in this invention are merely conventional technical means and should not be construed as insufficient disclosure of the present invention.

[0183] Unless otherwise specified, all embodiments and optional embodiments of the present invention can be combined with each other to form new technical solutions.

[0184] Unless otherwise specified, all technical features and optional technical features of this invention can be combined to form new technical solutions.

[0185] Unless otherwise specified, all steps of the present invention may be performed sequentially or randomly, preferably sequentially. For example, the method includes steps (a) and (b), indicating that the method may include steps (a) and (b) performed sequentially, or it may include steps (b) and (a) performed sequentially. For example, the mention that the method may also include step (c) indicates that step (c) may be added to the method in any order. For example, the method may include steps (a), (b), and (c), or it may include steps (a), (c), and (b), or it may include steps (c), (a), and (b), etc.

[0186] Unless otherwise specified, the terms "comprising" and "including" as used in this invention can be open-ended or closed-ended. For example, "comprising" and "including" can mean that other components not listed may also be included, or that only the listed components may be included.

[0187] Unless otherwise specified, the term "or" is inclusive in this invention. For example, the phrase "A or B" means "A, B, or both A and B". More specifically, the condition "A or B" is satisfied by any of the following conditions: A is true (or exists) and B is false (or does not exist); A is false (or does not exist) and B is true (or exists); or both A and B are true (or exist).

[0188] To better understand the solutions of the embodiments of the present invention, some related terms and concepts that may be involved in the embodiments of the present invention will be introduced below.

[0189] (1) ViMamba (Vision Mamba) architecture is a general-purpose visual backbone network based on a bidirectional state space model (SSM), designed specifically for efficient processing of visual data. Its core features include:

[0190] Input processing: The 2D image is segmented into non-overlapping blocks (e.g., 16×16 pixels), converted into a sequence vector through linear projection, and position embeddings are added to preserve spatial information.

[0191] Bidirectional scanning mechanism: Unlike traditional unidirectional Mamba, ViMamba blocks use forward and backward bidirectional SSM processing sequences, similar to bidirectional RNNs or Transformers, to capture the global spatial context.

[0192] Hardware optimization: Memory and computing efficiency are optimized through IO-aware algorithms, making it 2.8 times faster than DeiT when processing high-resolution images (such as 1248×1248) and saving 86.8% of GPU memory.

[0193] Application scenarios: It performs well in tasks such as ImageNet classification, COCO object detection, and ADE20k semantic segmentation, and has become a candidate architecture to replace Transformer.

[0194] (2) The bidirectional state space model is a core innovation of ViMamba, which solves the limitation of unidirectional modeling in traditional SSM:

[0195] Bidirectional processing: The input sequence is simultaneously scanned forward and backward, and modeled separately using two sets of SSM parameters (A, B, C, Δ). The final result is fused through a gating mechanism (z).

[0196] Advantages: Compared to unidirectional models, bidirectional SSM significantly improves the performance of dense prediction tasks (e.g., semantic segmentation mIoU increases from 32.3 to 35.9) while maintaining linear computational complexity.

[0197] Technical details: An inverse 1D convolution (Conv1d) is added to the SSM to enhance local feature extraction, forming a "Bidirectional SSM + Conv1d" strategy.

[0198] (3) SNR (Signal-to-Noise Ratio) SNR is a key indicator for measuring signal quality, defined as the ratio of the intensity of the useful signal to the background noise.

[0199] (4) SAM2 (Segment Anything Model 2) is a comprehensive upgrade of Meta's previous generation SAM model. It is the first to integrate image and video segmentation capabilities into a single model, supporting real-time processing of static images and dynamic videos. Its inference speed is 6 times faster than its predecessor, and the required human interaction is reduced by 3 times. It outperforms SAM in 23 zero-shot benchmark tests, and its video segmentation performance has reached new benchmarks on datasets such as DAVIS and MOSE.

[0200] In this embodiment, as Figure 1 As shown, a goal-oriented video semantic communication system based on a visual model includes:

[0201] The semantic extractor, based on the SAM2 model, includes an image encoder, a cue encoder, a mask decoder, and a memory module, which processes the raw video, generates a segmentation mask, and extracts semantic information.

[0202] The ViMamba encoder combines the Vision Mamba architecture and a downsampling module to perform channel coding on the output of the semantic extractor and perform sender channel adaptation.

[0203] The ViMamba decoder combines the Vision Mamba architecture and an upsampling module to perform channel decoding on the received semantic feature sequence after receiving-end channel adaptation, thereby obtaining the feature sequence.

[0204] A semantic reconstructor is used to perform semantic reconstruction based on feature sequences, recover data, and output the target video.

[0205] Furthermore, the image encoder functions to encode image features into multi-scale feature representations for a given raw video sequence using a hierarchical Hierarchical Vision Transformer.

[0206] For a given video sequence ,in, For video frame rate, Let T be the image of frame T. The image encoder uses a hierarchical Hierarchical Vision Transformer to encode image features into multi-scale feature representations.

[0207] Specifically, the image encoder converts the input image into multi-scale feature maps. , Represents the features of the nth layer;

[0208] The feature fusion process can be represented as:

[0209] ;

[0210] By fusing features at different levels, rich contextual information is provided to the mask decoder, enhancing the model's ability to extract and represent features of objects in images.

[0211] Furthermore, the prompt encoder is responsible for encoding the user-input prompts into feature vectors.

[0212] The prompt encoder encodes user-input prompts (such as dots, boxes, or masks) into feature vectors, as shown in the following process:

[0213] ;

[0214] The feature vector generated by the encoder will be prompted. Integrating into image features In the process, the final fusion characteristics are obtained. The formula is as follows:

[0215] ;

[0216] Furthermore, a memory module is used to store object information and previous segmentation results in order to track objects during video segmentation.

[0217] The memory update process is represented as:

[0218] ;

[0219] When processing subsequent frames, the model can use the information in memory to generate mask predictions, avoiding redundant calculations and improving efficiency.

[0220] Furthermore, the mask decoder combines the features extracted by the image encoder, the cue information provided by the cue encoder, and the memory data in the memory module to generate the final segmentation mask, and then segments the moving object using the mask.

[0221] The mask generation process is represented as follows:

[0222] ;

[0223] Through the above process, SAM2 can generate accurate segmentation masks to achieve image and video segmentation tasks.

[0224] Furthermore, a segmentation mask is generated and semantic information is extracted, including combining features extracted by the image encoder, cue information provided by the cue encoder, and memory data in the memory module to generate the final segmentation mask. The mask is then multiplied by the pixels of the corresponding frame to obtain the moving object image. The same operation is used to obtain the moving object images in all video frames, thus obtaining target semantic information containing only the moving object.

[0225] The process of segmenting moving objects using a mask is as follows:

[0226] ;

[0227] Segmentation mask Regions with a median value of 1 will retain the pixels from the original image, while regions with a median value of 0 will be set to 0. Accordingly, images of moving objects in all video frames can be obtained, thus enabling the segmentation of moving objects.

[0228] Furthermore, the ViMamba encoder performs channel coding by including the following steps:

[0229] Step 1: The encoder receives the output of the semantic extractor and performs block processing on the target image, dividing it into non-overlapping image blocks, and linearly projects the image blocks into a fixed-dimensional vector;

[0230] The encoder receives the output of the semantic extractor. As output For the moving object image of frame T, first process the image... Perform block processing, where i is less than or equal to T, where... The height of the image. Divide the image into non-overlapping image blocks, given the width of the image. ,in, The size of the block. This represents the number of feature channels.

[0231] Each image block First, it is linearly projected onto a vector of fixed dimension. , The projected dimension is determined by a learnable projection matrix, expressed as follows:

[0232] ;

[0233] Subsequently, to incorporate positional information, position embeddings are added to the projected vector. The final patch embedding is obtained, represented as:

[0234] ;

[0235] Step 2: Using several stacked ViMamba modules and patch embedding layers, based on a bidirectional state space model, high-level semantic features are extracted by learning the semantic relationships between image patches;

[0236] The embedded tokens after patch processing are input into In a stack of ViMamba modules, these The stacked ViMamba modules and patch embedding layers are collectively referred to as the "encoding stage one." Each ViMamba module is based on a bidirectional state space model, extracting high-level semantic features by learning the semantic relationships between image patches. Internally, the ViMamba module captures global semantic information in the image by combining forward and backward state space models, while maintaining sensitivity to local features. Specifically, the processing procedure of the ViMamba module is as follows:

[0237] a) Forward and Backward State Space Models: The ViMamba module processes feature sequences using state space models in both forward and backward directions. The forward model processes the sequence from beginning to end, while the backward model processes it from end to beginning. This bidirectional processing approach captures the contextual information at each position in the sequence.

[0238] b) State update: At time step t, the forward state... and backward state Update using the following formulas respectively:

[0239] ;

[0240] ;

[0241] c) Output Calculation: Final Output Features Calculated using the following formula:

[0242] ;

[0243] Step 3: Merge adjacent image patch features using the downsampling module to generate a new feature sequence;

[0244] To gradually reduce the resolution of the feature sequence and increase the number of channels, we add a downsampling module, patch merging, after each stage. This module generates a new feature sequence by merging features from adjacent image patches, as detailed below:

[0245] ;

[0246] Step 4: After multi-stage processing by the ViMamba module and downsampling module, multi-scale feature representations are obtained, and finally feature encoding is obtained.

[0247] The encoder consists of multiple stages, each including multiple ViMamba modules and a downsampling module. Specifically, the adjacent embeddings output from the first stage are merged in the second stage using a patch merging operation, resulting in a size of [size missing]. The connection embedding is reduced to a smaller size. Subsequently, , The height of the image. The width of the image. Each patch embedding tokens is input. In a ViMamba block.

[0248] Through each stage of processing, the resolution of the feature sequence gradually decreases while the number of channels gradually increases, eventually approaching a representation that is conducive to channel transmission. By employing a multi-stage processing approach, the proposed model significantly improves its capacity because it can capture long-range dependencies, utilize global information, and efficiently learn complex details in high-resolution images.

[0249] The encoder combines the efficient visual representation capabilities of Vision Mamba with the global semantic capture capabilities of the bidirectional state space model, while gradually constructing multi-scale feature representations through a downsampling module.

[0250] Furthermore, the transmitter channel adaptation is a precoding network based on SNR feedback. By learning the mapping relationship of SNR parameters, the weights of the received feature-coded sequences are adjusted. Finally, all channel weights are fused, and then the weights are multiplied by the module input and the channels are shuffled to obtain the output.

[0251] The channel adaptation process at the transmitting end can be represented as:

[0252] ;

[0253] The adjusted coded sequence will be transmitted to the receiver as the channel input signal. The channel adaptation module ensures that the coded sequence can be optimized according to the actual channel conditions, improving the system's adaptability and transmission efficiency. Details of the channel adaptation module are as follows:

[0254] The proposed channel adaptation module combines multi-scale feature fusion and channel attention mechanisms, considering channel features at the individual feature level while also incorporating spatial considerations at the sequence level. Specifically, it organizes the feature-encoded sequence into a feature map. ,in, These represent the outputs of the last stage of the encoder. The embedding dimensions, height, and width are then organized into a feature map.

[0255] First, the module convolves the feature map X with 1×1, 3×3, 5×5, and 7×7 convolution kernels, obtaining four tensors with different receptive fields. Their sizes are all , These represent the outputs of the last stage of the encoder. The embedding dimensions, height, and width are organized into feature maps, and then... Combined to obtain .

[0256] Next, each tensor is divided into two groups along the channel dimension, with each group having a size of [size missing]. The grouping operation process can be represented as:

[0257] ;

[0258] Then, two extraction modules with different kernel sizes [3×3, 5×5] are used to obtain the channel weights for each group. , Then, the two are combined along the channel dimension to obtain the weights of each tensor.

[0259] By analyzing the input feature map Perform global average pooling operation to obtain To avoid making the model too complex, it is compressed and swapped. , and then obtain Next, we use a one-dimensional convolutional network layer with a kernel size of k to achieve local cross-channel interaction to obtain... Obtained through the sigmoid activation function Finally, unzip and exchange. , and then obtain The extraction module formula is as follows:

[0260] ;

[0261] Subsequently, the channel weights of each group obtained by the extraction module are concatenated using the following formula:

[0262] ;

[0263] right After extraction and concatenation, we can obtain 5 weighted reorganizations. .

[0264] Then we multiply each weight by Obtain the right to reorganize , The channel feature tensor obtained after passing through the SNR projection module is a feedforward neural network with three fully connected layers. This module aims to expand and map the input SNR value into a vector of the same dimension as the backbone network's feature vector. The first layer maps a single SNR value to a higher-dimensional space, then uses the LeakyReLU activation function. Compared to traditional ReLU, it has a non-zero gradient on the negative half-axis, effectively preventing neuron death and maintaining continuous information transmission. The second fully connected layer continues the nonlinear mapping in the expanded high-dimensional space, also using LeakyReLU activation, further enhancing the model's expressive power and robustness. The final fully connected layer maps the data back to the target dimension and uses the Sigmoid activation function to constrain the output between 0 and 1, allowing the generated vector to be used as weights in subsequent modulation operations. This design enables smooth, stable, and effective feature extraction from the input SNR signal, providing a normalized and nonlinearly transformed modulation factor. The SNR projection module is as follows:

[0265] ;

[0266] Finally, all channel weights are combined, and then the weights are multiplied by the module input. The channel is shuffled to obtain the output, and the output of the channel adaptation module can be expressed as:

[0267] ;

[0268] The channel adaptation module adjusts the coding based on the channel's signal-to-noise ratio (SNR) information to adapt to different channel conditions, thereby enhancing the system's robustness in dynamic channel environments, especially under low SNR conditions.

[0269] After channel adaptation at the transmitting end, transmission must be performed through a wireless physical channel. During this process, the signal will inevitably be affected by channel noise and attenuation. This process can be represented as follows:

[0270] ;

[0271] Furthermore, the receiving end channel adaptation is implemented in the same way as the transmitting end channel adaptation, and its function is to optimize the coding sequence according to the signal-to-noise ratio information of the channel to adapt to different channel conditions.

[0272] The specific steps are as follows:

[0273] ;

[0274] Furthermore, the ViMamba decoder performs channel decoding by including the following steps:

[0275] Step 1: Using several ViMamba modules based on a bidirectional state-space model, high-level semantic features are extracted by learning the semantic associations in the feature sequence;

[0276] Step 2: The upsampling module generates new feature sequences by segmenting and expanding the feature sequences;

[0277] Step 3: After multi-stage processing by the ViMamba module and downsampling module, a size and feature representation close to the original image is obtained.

[0278] The decoder, another crucial component of the semantic communication system, is responsible for restoring the received semantic feature sequence to the original image. The decoder design combines the advanced Vision Mamba architecture and upsampling modules to achieve efficient feature restoration and image reconstruction. The specific implementation process is similar to that of the encoder, as follows:

[0279] 1) ViMamba module processing: Adjusted feature sequences The input is first fed into the ViMamba module of the first stage of the decoder. Each ViMamba module is based on a bidirectional state space model and extracts high-level semantic features by learning the semantic associations in the feature sequence.

[0280] 2) Upsampling Module: To progressively increase the resolution of the feature sequence and reduce the number of channels, we add an upsampling module (patch division) at the end of each stage. The upsampling module generates new feature sequences by segmenting and expanding the feature sequences. Specifically, the upsampling module operates as follows:

[0281] ;

[0282] This process helps the model to restore the semantic information of the image at different scales and gradually recover high-level feature representations.

[0283] 3) Multi-stage processing: Similar to the encoder, the decoder also consists of multiple stages, each including multiple ViMamba modules and an upsampling module. Through the processing of each stage, the resolution of the feature sequence gradually increases, the number of channels gradually decreases, and it gradually approaches the size and feature representation of the original image.

[0284] Finally, the feature sequence after all stages of processing Video data of moving objects is recovered using a semantic reconstructor. The reconstructed video data of the moving object can be represented as:

[0285] ;

[0286] In this embodiment, the specific process of performing video semantic communication on the original video sequence is as follows: the original video sequence is first processed by the semantic extractor, then processed and channel adapted by the four stages of the ViMamba encoder, and transmitted through the physical channel. After receiving the ViMamba decoder, it is also processed by the four stages of channel adaptation and output to the semantic reconstructor. Finally, the semantic reconstructor restores the video sequence.

[0287] It should be noted that the present invention is not limited to the above-described embodiments. The above embodiments are merely examples, and any embodiments that have the same structure and perform the same effects as the technical concept within the scope of the present invention are included within the scope of the present invention. Furthermore, various modifications that can be conceived by those skilled in the art to the embodiments, and other ways of constructing by combining some of the constituent elements of the embodiments, without departing from the spirit of the present invention, are also included within the scope of the present invention.

Claims

1. A goal-oriented video semantic communication system based on a visual model, characterized in that, include: The semantic extractor, based on the SAM2 model, includes an image encoder, a cue encoder, a mask decoder, and a memory module, which processes the raw video, generates a segmentation mask, and extracts semantic information. The ViMamba encoder combines the Vision Mamba architecture and a downsampling module to perform channel coding on the output of the semantic extractor. The channel adaptation module, including the transmitting end channel adaptation and the receiving end channel adaptation, is used to optimize the encoded sequence based on the signal-to-noise ratio information of the physical channel. It takes the output of the ViMamba encoder as input and outputs the semantic feature sequence to the ViMamba decoder. The ViMamba decoder, combining the Vision Mamba architecture and an upsampling module, is used to perform channel decoding on the received semantic feature sequence to obtain the feature sequence; it includes the following steps: Step 1: Using several ViMamba modules based on a bidirectional state-space model, high-level semantic features are extracted by learning the semantic associations in the feature sequence; Step 2: The upsampling module generates new feature sequences by segmenting and expanding the feature sequences; Step 3: After multi-stage processing by the ViMamba module and downsampling module, a size and feature representation close to the original image is obtained; A semantic reconstructor is used to perform semantic reconstruction based on feature sequences, recover data, and output the target video.

2. The goal-oriented video semantic communication system based on a visual model according to claim 1, characterized in that, The image encoder functions by using a hierarchical Hierarchical VisionTransformer to encode image features into multi-scale feature representations for a given original video sequence.

3. The goal-oriented video semantic communication system based on a visual model according to claim 1, characterized in that, The prompt encoder is used to encode user-input prompts into feature vectors.

4. The goal-oriented video semantic communication system based on a visual model according to claim 1, characterized in that, The memory module is used to store object information and previous segmentation results to track objects during video segmentation.

5. A goal-oriented video semantic communication system based on a visual model according to claim 1, characterized in that, The mask decoder is used to combine the features extracted by the image encoder, the prompt information provided by the prompt encoder, and the memory data in the memory module to generate the final segmentation mask, and to segment the moving object through the mask.

6. The goal-oriented video semantic communication system based on a visual model according to claim 1, characterized in that, The process of generating a segmentation mask and extracting semantic information includes combining features extracted by the image encoder, cue information provided by the cue encoder, and memory data in the memory module to generate the final segmentation mask. The mask is then multiplied by the pixels of the corresponding frame to obtain the moving object image. The same operation is used to obtain the moving object images in all video frames, thus obtaining target semantic information containing only the moving object.

7. A goal-oriented video semantic communication system based on a visual model according to claim 1, characterized in that, The ViMamba encoder performs channel coding by including the following steps: Step 1: The encoder receives the output of the semantic extractor and performs block processing on the target image, dividing it into non-overlapping image blocks, and linearly projects the image blocks into a fixed-dimensional vector; Step 2: Using several stacked ViMamba modules and patch embedding layers, based on a bidirectional state space model, high-level semantic features are extracted by learning the semantic relationships between image patches; Step 3: Merge adjacent image patch features using the downsampling module to generate a new feature sequence; Step 4: After multi-stage processing by the ViMamba module and downsampling module, multi-scale feature representations are obtained, and finally feature encoding is obtained.

8. A goal-oriented video semantic communication system based on a visual model according to claim 1, characterized in that, The transmitting end channel adaptation is a precoding network based on SNR feedback. By learning the mapping relationship of SNR parameters, the weights of the received feature-coded sequences are adjusted. Finally, all channel weights are fused, and then the weights are multiplied by the module input and the channels are shuffled to obtain the output.

9. A goal-oriented video semantic communication system based on a visual model according to claim 1, characterized in that, The receiver channel adaptation is implemented in the same way as the transmitter channel adaptation. Its function is to optimize the coding sequence according to the signal-to-noise ratio information of the channel to adapt to different channel conditions.

Images