Target-oriented video semantic communication system based on visual model

Abstract

The invention provides a target-oriented video semantic communication system based on a visual model, and the system comprises a semantic extractor which is based on an SAM2 model and is used for processing an original video, generating a segmentation mask and extracting semantic information; the ViMama encoder is used for carrying out channel encoding on the output of the semantic extractor; the channel adaptation module is used for optimizing a coding sequence ViMama decoder based on the signal-to-noise ratio information of a physical channel, and is used for carrying out channel decoding to obtain a feature sequence; and the semantic reconstruction device is used for performing semantic reconstruction based on the feature sequence, recovering data and outputting a target video. According to the method, the problems of large redundant semantic information interference, insufficient deep semantic coding capability and poor communication robustness in a complex channel environment in a video are solved, efficient compression and robust transmission of video data are realized on the premise of ensuring semantic integrity, and the overall performance and adaptability of semantic communication are greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of semantic communication technology, and in particular to a goal-oriented video semantic communication system based on a visual model. Background Art

[0002] With the widespread application of video data in scenarios such as autonomous driving, telemedicine, and intelligent security, traditional video communication systems are increasingly facing challenges with low transmission efficiency and high communication overhead in complex network environments requiring limited bandwidth, low latency, and high robustness. To alleviate this challenge, semantic communication, a cutting-edge technology that integrates perception and communication, has garnered widespread attention in recent years. Semantic communication extracts key semantic information from source content for transmission, rather than restoring data pixel by pixel. This significantly reduces data size while maintaining mission-critical accuracy.

[0003] However, existing semantic communication systems mainly include: semantic communication systems based on convolutional neural networks (CNNs) and autoencoders: CNNs are used to extract image features, and autoencoders are used to compress and reconstruct semantic information. However, this fails to effectively solve the redundancy problem at the pixel level, resulting in high bandwidth consumption; semantic communication systems based on visual transformers (ViTs): ViTs use attention mechanisms to model long-term and short-term dependencies in images, and are suitable for high-resolution and complex scenes. However, the ViT model has a large number of parameters and a slow inference speed, making it difficult to meet low-latency communication requirements. Summary of the Invention

[0004] The present invention is made in view of the above-mentioned problems, and its purpose is to provide a goal-oriented video semantic communication system based on a visual model. By introducing a semantic extractor based on SAM 2 and designing channel coding, channel decoding, and channel adaptation methods based on the visual Mamba framework, the problems of large interference from redundant semantic information in videos, insufficient deep semantic coding capabilities, and poor communication robustness in complex channel environments are solved. While ensuring semantic integrity, the system achieves efficient compression and robust transmission of video data, greatly improving the overall performance and adaptability of semantic communication. The system is suitable for target-oriented application scenarios such as autonomous driving and intelligent monitoring.

[0005] Specifically, a first aspect of the present invention provides a goal-oriented video semantic communication system based on a visual model, comprising: The semantic extractor, based on the SAM2 model, includes an image encoder, a hint encoder, a mask decoder, and a memory module to process raw video, generate segmentation masks, and extract semantic information; The ViMamba encoder combines the Vision Mamba architecture with a downsampling module to channel encode the output of the semantic extractor and perform sender-side channel adaptation; The channel adaptation module, including the transmitting end channel adaptation and the receiving end channel adaptation, is used to optimize the coding sequence based on the signal-to-noise ratio information of the physical channel, takes the output of the ViMamba encoder as input, and outputs the semantic feature sequence to the ViMamba decoder; The ViMamba decoder combines the Vision Mamba architecture with an upsampling module to perform channel decoding on the received semantic feature sequence after receiving-side channel adaptation to obtain a feature sequence; The semantic reconstructor is used to perform semantic reconstruction based on the feature sequence, restore the data and output the target video.

[0006] Furthermore, the image encoder uses a hierarchical Hierarchical Vision Transformer to encode image features into multi-scale feature representations for a given original video sequence.

[0007] For a given video sequence ,in, is the number of video frames, is the T-th frame image. The image encoder uses a layered Hierarchical Vision Transformer to encode image features into multi-scale feature representations.

[0008] Specifically, the image encoder converts the input image into a multi-scale feature map , Represents the features of the nth layer; The feature fusion process can be expressed as: ; in: is the fused image feature; It is the feature fusion operation; is a multi-scale feature map; By fusing features at different levels, rich contextual information is provided to the mask decoder, enhancing the model's ability to extract and represent features of objects in the image.

[0009] Furthermore, the prompt encoder is used to encode the prompt input by the user into a feature vector.

[0010] The prompt encoder encodes the user input prompt (such as a point, box, or mask) into a feature vector. The process is as follows: ; in: The feature vector generated by the prompt encoder represents the semantic embedding of the input prompt; To prompt the encoding operation; Prompts for user input; The feature vector generated by the prompt encoder Incorporating image features The final fusion feature is obtained , the formula is as follows: ; in: is the final fusion feature; It is the feature fusion operation; is the fused image feature; The feature vector generated for the prompt encoder; Furthermore, the memory module is used to store object information and previous segmentation results to track objects in video segmentation.

[0011] The memory update process is expressed as: ; in: The information stored in the memory module for the t+1th frame; For memory update operation; The information stored in the memory module for the tth frame; is the segmentation result of the t-th frame; When processing subsequent frames, the model can use the information in memory to generate mask predictions, avoiding repeated calculations and improving efficiency.

[0012] Furthermore, the mask decoder is used to combine the features extracted by the image encoder, the hint information provided by the hint encoder and the memory data in the memory module to generate a final segmentation mask, and segment the moving object through the mask.

[0013] The mask generation process is expressed as: ; in: is the final segmentation mask; Generate operations for masks; is the fused image feature; The feature vector generated for the prompt encoder; is the memory in the memory module; Through the above process, SAM 2 is able to generate accurate segmentation masks and achieve image and video segmentation tasks.

[0014] Furthermore, the generation of the segmentation mask and extraction of semantic information includes combining the features extracted by the image encoder, the prompt information provided by the prompt encoder, and the memory data in the memory module to generate a final segmentation mask, and then multiplying the mask with the pixels of the corresponding frame to obtain a moving object image. The moving object images in all video frames are obtained through the same operation to obtain target semantic information of only the moving object.

[0015] The process of segmenting moving objects through masks is as follows: ; in: is the moving object image of the i-th frame; is the i-th frame image; It is a pixel-by-pixel multiplication operation; is the final segmentation mask; Segmentation Mask The area with a median value of 1 will retain the pixels in the original image, and the area with a median value of 0 will be set to 0. Accordingly, the moving object images in all video frames can be obtained, thereby realizing the segmentation of the moving object.

[0016] Furthermore, the ViMamba encoder performs channel coding including the following steps: Step 1: The encoder receives the output of the semantic extractor and performs block processing on the target image, dividing it into non-overlapping image blocks and linearly projecting the image blocks into a vector of fixed dimension; The encoder receives the output of the semantic extractor As output, For the moving object image of the Tth frame, first Perform block processing, i is less than or equal to T, where, is the height of the image, The width of the image is divided into non-overlapping image blocks ,in, is the size of the block, is the number of feature channels.

[0017] Each image block First it is linearly projected into a vector of fixed dimension , is the dimension after projection. This process is completed through a learnable projection matrix, which is expressed as follows: ; in: is a vector of fixed dimension; is a learnable projection matrix, , is the size of the block, is the dimension after projection; Indicates that the image block Vectorization; Then, to introduce position information, position embedding is added to the projected vector , and the final patch embedding is obtained, which is expressed as: ; in: Embed the final patch; After projection vectors; After projection Change to position embedding added by a vector; is the length of the sequence, , is the height of the image, is the width of the image, is the size of the block; Step 2: Through several stacked ViMamba modules and patch embedding layers, based on the bidirectional state space model, high-level semantic features are extracted by learning the semantic associations between image patches; The embedded tokens after patch processing are input into In a stack of ViMamba modules, place these The stacked ViMamba modules and patch embedding layer are collectively referred to as the "first encoding stage." Each ViMamba module, based on a bidirectional state space model, extracts high-level semantic features by learning semantic associations between image patches. Internally, the ViMamba module combines forward and backward state space models to capture global semantic information in the image while maintaining sensitivity to local features. Specifically, the ViMamba module performs the following processing: a) Forward and Backward State-Space Models: The ViMamba module processes feature sequences using state-space models in both forward and backward directions. The forward model processes from the beginning to the end of the sequence, while the backward model processes from the end to the beginning. This bidirectional processing approach captures contextual information at every position in the sequence.

[0018] b) State update: At time step t, the forward state and backward state Updated by the following formulas respectively: ; ; in: is the forward state at time step t; is the state transfer matrix of the forward state space model; is the forward state at time step t-1; is the input matrix of the forward state space model; is the input feature vector at time step t; is the backward state at time step t; is the state transfer matrix of the backward state space model; is the backward state at time step t+1; is the input matrix of the backward state space model; c) Output calculation: final output features Calculated by the following formula: ; in: is the final output feature; is the output matrix of the forward state space model; is the final state of the forward state space model; is the output matrix of the backward state space model; is the final state of the backward state space model; Step 3: Merge the features of adjacent image blocks through the downsampling module to generate a new feature sequence; In order to gradually reduce the resolution of the feature sequence and increase the number of channels, we add a downsampling module patch merging after each stage to generate a new feature sequence by merging the features of adjacent image patches, as follows: ; in: is a new feature sequence; The downsampling module operation, also known as the Residual Block with Downsampling, helps the model capture the semantic information of the image at different scales and gradually build high-level feature representations; is the final output feature; Step 4: After multi-stage processing of ViMamba modules and downsampling modules, multi-scale feature representation is obtained, and finally feature encoding is obtained.

[0019] The encoder consists of multiple stages, each of which includes multiple ViMamba modules and a downsampling module. Specifically, the adjacent embeddings output by the first stage are merged in the second stage through a patch merging operation, resulting in a size of The concatenated embedding of is reduced to size . Then, , is the height of the image, is the width of the image, Patch embed tokens are input to in a ViMamba block.

[0020] After each stage of processing, the resolution of the feature sequence decreases, the number of channels gradually increases, and gradually approaches a representation that is conducive to channel transmission. Through multi-stage processing, the proposed model significantly improves model capacity because it can capture long-range dependencies, utilize global information, and efficiently learn complex details in high-resolution images.

[0021] The encoder combines the efficient visual representation capabilities of Vision Mamba and the global semantic capture capabilities of the bidirectional state-space model, while gradually building multi-scale feature representations through the downsampling module.

[0022] Furthermore, the transmitting end channel adaptation is a precoding network based on SNR feedback, which adjusts the weights of the received feature-encoded coding sequence by learning the mapping relationship of the SNR parameters, and finally fuses all channel weights, then multiplies the weights by the module input and performs channel shuffling to obtain the output.

[0023] The adjustment process of the channel adaptation at the transmitter can be expressed as: ; in: is the adjusted coding sequence; Encode Channel Adaptation operation, which represents the precoding network based on SNR feedback. It adjusts the weight of the encoding sequence by learning the mapping relationship of SNR parameters. is the output of the last stage of the encoder; The signal-to-noise ratio is the core indicator for measuring the ratio of signal power to noise power.

[0024] The adjusted coding sequence will be transmitted to the receiving end as the channel input signal.

[0025] The channel adaptation module ensures that the coding sequence can be optimized according to the actual channel conditions, improving the system's adaptability and transmission efficiency. The specific details of the channel adaptation module are as follows: The proposed channel adaptation module combines multi-scale feature fusion and channel attention mechanism, considering channel features at the level of individual features while also combining the space at the sequence level. Specifically, the feature encoding sequence is organized into a feature map ,in, Represents the output of the last stage of the encoder Embedding dimensions, height, and width after being organized into feature maps.

[0026] First, the module convolves the feature map X with 1×1, 3×3, 5×5, and 7×7 convolution kernels and obtains four tensors with different receptive fields. , their sizes are , Represents the output of the last stage of the encoder Arranged into the embedding dimensions, height and width of the feature map, and Add to obtain .

[0027] Afterwards, each tensor is divided into two groups in the channel dimension, and the size of each group is , the grouping operation process can be expressed as: ; in: It is a grouping operation, which is represented by dividing each tensor in the channel dimension Divide into two groups, each group is ; , , for Feature map after grouping; Then, two extraction modules with different convolution kernel sizes [3×3, 5×5] are used to obtain the channel weights of each group. , , and then combine the two groups along the channel dimension to get the weights for each tensor.

[0028] By inputting feature maps Perform global average pooling operation to obtain , in order to avoid the model being too complex, compress and exchange , and then obtain Afterwards, we use a one-dimensional convolutional network layer with kernel size k to achieve local cross-channel interaction to obtain , obtained through the sigmoid activation function , finally, unzip and swap , and then obtain , the extraction module formula is as follows: ; in: For Output after extraction; For the adaptive average pooling operation, we obtain ; For compression and exchange operations, the compression is achieved by a double fully connected layer structure of a fully connected layer, a RELU nonlinear activation layer, a fully connected layer, and a sigmoid layer, and then the spatial and channel dimension order is exchanged to obtain ; is a convolution operation with a kernel of k×k, k∈[3,5]; It is the sigmoid operation; For decompression and exchange operations, which are the reverse process of compression and exchange, The shape is restored from [1, c] to [c, 1, 1] for subsequent processing; is the input feature map; Subsequently, the channel weights of each group obtained by the extraction module are spliced. The splicing formula is as follows: ; in: is the splicing operation, which is represented by 、 Concatenate by channel dimension to obtain feature map , ; For Output of the signal after channel coding and feature extraction; For Output of the signal after channel coding and feature extraction; right After extraction and splicing operations, we can get 5 weight groups .

[0029] We then multiply each weight by , and get the weight group , The channel feature tensor obtained by the SNR projection module is a feedforward neural network with three fully connected layers. It is designed to expand the input SNR value and map it into a vector of the same dimension as the backbone network feature vector. The first layer of the network maps the single SNR value to a higher-dimensional space, and then uses the LeakyReLU activation function. Compared with the traditional ReLU, it has a non-zero gradient on the negative semi-axis, which effectively prevents neuron death and maintains the continuous transmission of information. The second fully connected layer continues to perform nonlinear mapping in the expanded high-dimensional space, and also uses LeakyReLU activation to further enhance the expressiveness and robustness of the model. The last fully connected layer maps the data back to the target dimension and limits the output to between 0 and 1 through the Sigmoid activation function, so that the generated vector can be used as a weight in subsequent modulation operations. Through this design, the entire module achieves smooth, stable and effective feature extraction of the input SNR signal, and provides a modulation factor after normalization and nonlinear transformation. The SNR projection module is as follows: ; in: It is the sigmoid operation; LeakyReLU operation; and are the weight and bias of Linear respectively; Finally, all 5 channel weights are fused , the fused weights are combined with the module input Through channel-by-channel multiplication, and then through channel shuffle operation Processing, the output of the final channel adaptation module can be expressed as: ; in: is the output of the channel adaptation module; Shuffle operations for channels; is the multiplication operation; is the i-th weight group; It is the input of the channel adaptation module; The channel adaptation module features adjust the coding according to the channel signal-to-noise ratio (SNR) information to adapt to different channel conditions, enhancing the robustness of the system in dynamic channel environments, especially low SNR conditions.

[0030] After the transmitter channel adapts, the signal needs to be transmitted through the wireless physical channel. During this process, it will inevitably be affected by channel noise and attenuation. This process can be expressed as: ; in: is the symbol received by the final receiving end; is the channel gain; It is the symbol encoded by the sender; is additive white noise (AWGN); Furthermore, the receiving end channel adaptation is implemented in the same manner as the transmitting end channel adaptation, and its function is to optimize the coding sequence according to the signal-to-noise ratio information of the channel to adapt to different channel conditions.

[0031] The specific operations are: ; in: is the characteristic sequence output by the channel; Decode Channel Adaptation operation, which represents the decoding channel adaptation module based on SNR feedback. It adjusts the weight of the coding sequence by learning the mapping relationship of SNR parameters. is the symbol received by the final receiving end; The signal-to-noise ratio is the core indicator for measuring the ratio of signal power to noise power.

[0032] Furthermore, the ViMamba decoder performs channel decoding including the following steps: Step 1: Extract high-level semantic features by learning semantic associations in feature sequences based on a bidirectional state-space model through several ViMamba modules; Step 2: Generate a new feature sequence by splitting and expanding the feature sequence through the upsampling module; Step 3: After multi-stage processing of ViMamba modules and downsampling modules, the size and feature representation close to the original image are obtained.

[0033] The decoder, another key component of the semantic communication system, is responsible for restoring the received semantic feature sequence to the original image. The decoder design combines the advanced Vision Mamb architecture and upsampling module to achieve efficient feature restoration and image reconstruction. The specific implementation process is similar to that of the encoder and is as follows: 1) ViMamba module processing: adjusted feature sequence , is first input into the ViMamba module of the first stage of the decoder. Each ViMamba module is based on the bidirectional state space model (Bidirectional State Space Model) and extracts high-level semantic features by learning the semantic associations in the feature sequence.

[0034] 2) Upsampling module: To gradually increase the resolution of the feature sequence and reduce the number of channels, we add an upsampling module (patch division) at the end of each stage. The upsampling module generates a new feature sequence by splitting and expanding the feature sequence. The specific operations are: ; in: It is the feature sequence processed by the upsampling module; Residual Block with Upsampling operation, which belongs to the upsampling module, achieves feature dimension increase and channel number reduction through convolution layer and upsampling layer; is the feature sequence output by the ViMamba module in the decoder; This process helps the model restore the semantic information of the image at different scales and gradually recover high-level feature representations.

[0035] 3) Multi-stage processing: Similar to the encoder, the decoder also consists of multiple stages, each of which includes multiple ViMamba modules and an upsampling module. With each stage of processing, the resolution of the feature sequence gradually increases and the number of channels gradually decreases, gradually approaching the size and feature representation of the original image.

[0036] Finally, the feature sequence after all stages of processing , recovering video data of moving objects through semantic reconstructor , the reconstructed video data of the moving object can be expressed as: ; in: Video data of the restored moving object; For a parameter set Semantic reconstructor of ; is the feature finally generated after joint source and channel decoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present drawings or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present drawings. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0038] Figure 1 It is a composition structure diagram of the present invention; The purpose, features and advantages of this drawing will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention. Based on the embodiments provided by the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0040] Obviously, the drawings described below are merely examples or embodiments of the present invention. Those skilled in the art can apply the present invention to other similar scenarios based on these drawings without inventive effort. Furthermore, it is understood that while the effort involved in such a development process may be complex and lengthy, for those skilled in the art related to the disclosure of the present invention, any design, manufacturing, or production changes based on the technical content disclosed in the present invention are merely conventional technical means and should not be construed as an inadequacy of the disclosure of the present invention.

[0041] Unless otherwise specified, all embodiments and optional embodiments of the present invention can be combined with each other to form new technical solutions.

[0042] Unless otherwise specified, all technical features and optional technical features of the present invention can be combined with each other to form a new technical solution.

[0043] Unless otherwise specified, all steps of the present invention may be performed sequentially or randomly, preferably sequentially. For example, the method includes steps (a) and (b), which means that the method may include steps (a) and (b) performed sequentially, or may include steps (b) and (a) performed sequentially. For example, the method may further include step (c), which means that step (c) may be added to the method in any order, for example, the method may include steps (a), (b) and (c), or may include steps (a), (c) and (b), or may include steps (c), (a) and (b), etc.

[0044] Unless otherwise specified, the terms "include" and "comprising" used in the present invention may be open-ended or closed-ended. For example, "include" and "comprising" may mean that other components not listed may also be included or that only the listed components are included.

[0045] Unless otherwise specified, the term "or" is inclusive in this disclosure. For example, the phrase "A or B" means "A, B, or both A and B." More specifically, the condition "A or B" is satisfied if any of the following conditions are met: A is true (or exists) and B is false (or does not exist); A is false (or does not exist) and B is true (or exists); or both A and B are true (or exist).

[0046] In order to better understand the solutions of the embodiments of the present invention, some relevant terms and concepts that may be involved in the embodiments of the present invention are first introduced below.

[0047] (1) The ViMamba (Vision Mamba) architecture is a general-purpose visual backbone network based on the Bidirectional State Space Model (SSM), designed for efficient visual data processing. Its core features include: Input processing: The 2D image is split into non-overlapping blocks (e.g., 16×16 pixels), converted into a sequence vector through linear projection, and position embeddings are added to preserve spatial information.

[0048] Bidirectional Scanning Mechanism: Different from the traditional unidirectional Mamba, the ViMamba block adopts forward and backward bidirectional SSM to process the sequence, similar to bidirectional RNN or Transformer, to capture the global spatial context.

[0049] Hardware optimization: Memory and computational efficiency are optimized through IO-aware algorithms, making it 2.8 times faster than DeiT when processing high-resolution images (such as 1248×1248) and saving 86.8% of GPU memory.

[0050] Application scenarios: It performs well in tasks such as ImageNet classification, COCO object detection, and ADE20k semantic segmentation, becoming a candidate architecture to replace Transformer.

[0051] (2) The Bidirectional State Space Model is ViMamba’s core innovation, which solves the unidirectional modeling limitations of traditional SSM: Bidirectional processing: The input sequence is scanned forward and backward simultaneously, modeled separately by two sets of SSM parameters (A, B, C, Δ), and the final results are fused through a gating mechanism (z).

[0052] Advantages: Compared with the unidirectional model, the bidirectional SSM significantly improves the performance of dense prediction tasks (for example, semantic segmentation mIoU increases from 32.3 to 35.9) while maintaining linear computational complexity.

[0053] Technical details: Inverse 1D convolution (Conv1d) is added to the SSM to enhance local feature extraction, forming a "Bidirectional SSM + Conv1d" strategy.

[0054] (3) SNR (Signal-to-Noise Ratio) SNR is a key indicator for measuring signal quality and is defined as the ratio of the intensity of the useful signal to the background noise.

[0055] (4) SAM2 (Segment Anything Model 2) is Meta's comprehensive upgrade of its predecessor, the SAM model. For the first time, it integrates image and video segmentation capabilities into a single model, supporting real-time processing of both static images and dynamic videos. It boasts six times faster inference speed than its predecessor, requires three times less human interaction, and outperforms SAM on 23 zero-shot benchmarks. Its video segmentation performance reaches new benchmarks on datasets such as DAVIS and MOSE.

[0056] In this embodiment, Figure 1 As shown, a goal-oriented video semantic communication system based on a visual model includes: The semantic extractor, based on the SAM2 model, includes an image encoder, a hint encoder, a mask decoder, and a memory module to process raw video, generate segmentation masks, and extract semantic information; The ViMamba encoder combines the Vision Mamba architecture with a downsampling module to channel encode the output of the semantic extractor and perform sender-side channel adaptation; The ViMamba decoder combines the Vision Mamba architecture with an upsampling module to perform channel decoding on the received semantic feature sequence after receiving-side channel adaptation to obtain a feature sequence; The semantic reconstructor is used to perform semantic reconstruction based on the feature sequence, restore the data and output the target video.

[0057] Furthermore, the image encoder uses a hierarchical Hierarchical Vision Transformer to encode image features into multi-scale feature representations for a given original video sequence.

[0058] For a given video sequence ,in, is the number of video frames, is the T-th frame image. The image encoder uses a layered Hierarchical Vision Transformer to encode image features into multi-scale feature representations.

[0059] Specifically, the image encoder converts the input image into a multi-scale feature map , Represents the features of the nth layer; The feature fusion process can be expressed as: ; By fusing features at different levels, rich contextual information is provided to the mask decoder, enhancing the model's ability to extract and represent features of objects in the image.

[0060] Furthermore, a prompt encoder is provided, which is used to encode the prompt input by the user into a feature vector.

[0061] The prompt encoder encodes the user input prompt (such as a point, box, or mask) into a feature vector. The process is as follows: ; The feature vector generated by the prompt encoder Incorporating image features The final fusion feature is obtained , the formula is as follows: ; Furthermore, a memory module is used to store object information and previous segmentation results to track objects in video segmentation.

[0062] The memory update process is expressed as: ; When processing subsequent frames, the model can use the information in memory to generate mask predictions, avoiding repeated calculations and improving efficiency.

[0063] Furthermore, the mask decoder is used to combine the features extracted by the image encoder, the hint information provided by the hint encoder, and the memory data in the memory module to generate a final segmentation mask, and segment the moving object through the mask.

[0064] The mask generation process is expressed as: ; Through the above process, SAM2 can generate accurate segmentation masks to achieve image and video segmentation tasks.

[0065] Furthermore, a segmentation mask is generated and semantic information is extracted, including combining the features extracted by the image encoder, the prompt information provided by the prompt encoder, and the memory data in the memory module to generate a final segmentation mask, and then multiplying the mask with the pixels of the corresponding frame to obtain a moving object image. The moving object image in all video frames is obtained through the same operation, and the target semantic information of only the moving object is obtained.

[0066] The process of segmenting moving objects through masks is as follows: ; Segmentation Mask The area with a median value of 1 will retain the pixels in the original image, and the area with a median value of 0 will be set to 0. Accordingly, the moving object images in all video frames can be obtained, thereby realizing the segmentation of the moving object.

[0067] Furthermore, the ViMamba encoder performs channel coding including the following steps: Step 1: The encoder receives the output of the semantic extractor and performs block processing on the target image, dividing it into non-overlapping image blocks and linearly projecting the image blocks into a vector of fixed dimension; The encoder receives the output of the semantic extractor As output, For the moving object image of the Tth frame, first Perform block processing, i is less than or equal to T, where, is the height of the image, The width of the image is divided into non-overlapping image blocks ,in, is the size of the block, is the number of feature channels.

[0068] Each image block First it is linearly projected into a vector of fixed dimension , is the dimension after projection. This process is completed through a learnable projection matrix, which is expressed as follows: ; Then, to introduce position information, position embedding is added to the projected vector , and the final patch embedding is obtained, which is expressed as: ; Step 2: Through several stacked ViMamba modules and patch embedding layers, based on the bidirectional state space model, high-level semantic features are extracted by learning the semantic associations between image patches; The embedded tokens after patch processing are input into In a stack of ViMamba modules, place these The stacked ViMamba modules and patch embedding layer are collectively referred to as the "first encoding stage." Each ViMamba module, based on a bidirectional state space model, extracts high-level semantic features by learning semantic associations between image patches. Internally, the ViMamba module combines forward and backward state space models to capture global semantic information in the image while maintaining sensitivity to local features. Specifically, the ViMamba module performs the following processing: a) Forward and Backward State-Space Models: The ViMamba module processes feature sequences using state-space models in both forward and backward directions. The forward model processes from the beginning to the end of the sequence, while the backward model processes from the end to the beginning. This bidirectional processing approach captures contextual information at every position in the sequence.

[0069] b) State update: At time step t, the forward state and backward state Updated by the following formulas respectively: ; ; c) Output calculation: final output features Calculated by the following formula: ; Step 3: Merge the features of adjacent image blocks through the downsampling module to generate a new feature sequence; In order to gradually reduce the resolution of the feature sequence and increase the number of channels, we add a downsampling module patch merging after each stage to generate a new feature sequence by merging the features of adjacent image patches, as follows: ; Step 4: After multi-stage processing of ViMamba modules and downsampling modules, multi-scale feature representation is obtained, and finally feature encoding is obtained.

[0070] The encoder consists of multiple stages, each of which includes multiple ViMamba modules and a downsampling module. Specifically, the adjacent embeddings output by the first stage are merged in the second stage through a patch merging operation, resulting in a size of The concatenated embedding of is reduced to size . Then, , is the height of the image, is the width of the image, Patch embed tokens are input to in a ViMamba block.

[0071] After each stage of processing, the resolution of the feature sequence decreases, the number of channels gradually increases, and gradually approaches a representation that is conducive to channel transmission. Through multi-stage processing, the proposed model significantly improves model capacity because it can capture long-range dependencies, utilize global information, and efficiently learn complex details in high-resolution images.

[0072] The encoder combines the efficient visual representation capabilities of Vision Mamba and the global semantic capture capabilities of the bidirectional state-space model, while gradually building multi-scale feature representations through the downsampling module.

[0073] Furthermore, the transmitting end channel adaptation is a precoding network based on SNR feedback. By learning the mapping relationship of SNR parameters, the coding sequence of the received feature code is weighted and finally all channel weights are fused. The weights are then multiplied by the module input and the channels are shuffled to obtain the output.

[0074] The adjustment process of the channel adaptation at the transmitter can be expressed as: ; The adjusted code sequence is then transmitted to the receiver as the channel input signal. The channel adaptation module ensures that the code sequence is optimized based on the actual channel conditions, improving the system's adaptability and transmission efficiency. The details of the channel adaptation module are as follows: The proposed channel adaptation module combines multi-scale feature fusion and channel attention mechanism, considering channel features at the level of individual features while also combining the space at the sequence level. Specifically, the feature encoding sequence is organized into a feature map ,in, Represents the output of the last stage of the encoder Embedding dimensions, height, and width after being organized into feature maps.

[0075] First, the module convolves the feature map X with 1×1, 3×3, 5×5, and 7×7 convolution kernels and obtains four tensors with different receptive fields. , their sizes are , Represents the output of the last stage of the encoder Arranged into the embedding dimensions, height and width of the feature map, and Add to obtain .

[0076] Afterwards, each tensor is divided into two groups in the channel dimension, and the size of each group is , the grouping operation process can be expressed as: ; Then, two extraction modules with different convolution kernel sizes [3×3, 5×5] are used to obtain the channel weights of each group. , , and then combine the two groups along the channel dimension to get the weights for each tensor.

[0077] By inputting feature maps Perform global average pooling operation to obtain , in order to avoid the model being too complex, compress and exchange , and then obtain Afterwards, we use a one-dimensional convolutional network layer with kernel size k to achieve local cross-channel interaction to obtain , obtained through the sigmoid activation function , finally, unzip and swap , and then obtain , the extraction module formula is as follows: ; Subsequently, the channel weights of each group obtained by the extraction module are spliced. The splicing formula is as follows: ; right After extraction and splicing operations, we can get 5 weight groups .

[0078] We then multiply each weight by , and get the weight group , The channel feature tensor obtained by the SNR projection module is a feedforward neural network with three fully connected layers. It is designed to expand the input SNR value and map it into a vector of the same dimension as the backbone network feature vector. The first layer of the network maps the single SNR value to a higher-dimensional space, and then uses the LeakyReLU activation function. Compared with the traditional ReLU, it has a non-zero gradient on the negative semi-axis, which effectively prevents neuron death and maintains the continuous transmission of information. The second fully connected layer continues to perform nonlinear mapping in the expanded high-dimensional space, and also uses LeakyReLU activation to further enhance the expressiveness and robustness of the model. The last fully connected layer maps the data back to the target dimension and limits the output to between 0 and 1 through the Sigmoid activation function, so that the generated vector can be used as a weight in subsequent modulation operations. Through this design, the entire module achieves smooth, stable and effective feature extraction of the input SNR signal, and provides a modulation factor after normalization and nonlinear transformation. The SNR projection module is as follows: ; Finally, all channel weights are fused and then multiplied by the module input , and perform channel shuffling to obtain the output. The output of the channel adaptation module can be expressed as: ; The channel adaptation module features adjust the coding according to the channel signal-to-noise ratio (SNR) information to adapt to different channel conditions, enhancing the robustness of the system in dynamic channel environments, especially low SNR conditions.

[0079] After the transmitter channel adapts, the signal needs to be transmitted through the wireless physical channel. During this process, it will inevitably be affected by channel noise and attenuation. This process can be expressed as: ; Furthermore, the receiving end channel adaptation is implemented in the same manner as the transmitting end channel adaptation, and its function is to optimize the coding sequence according to the signal-to-noise ratio information of the channel to adapt to different channel conditions.

[0080] The specific operations are: ; Furthermore, the ViMamba decoder performs channel decoding including the following steps: Step 1: Extract high-level semantic features by learning semantic associations in feature sequences based on a bidirectional state-space model through several ViMamba modules; Step 2: Generate a new feature sequence by splitting and expanding the feature sequence through the upsampling module; Step 3: After multi-stage processing of ViMamba modules and downsampling modules, the size and feature representation close to the original image are obtained.

[0081] The decoder, another key component of the semantic communication system, is responsible for restoring the received semantic feature sequence to the original image. The decoder design combines the advanced Vision Mamba architecture and upsampling modules to achieve efficient feature restoration and image reconstruction. The specific implementation process is similar to that of the encoder and is as follows: 1) ViMamba module processing: adjusted feature sequence , is first input into the ViMamba module of the first stage of the decoder. Each ViMamba module is based on the bidirectional state space model (Bidirectional State Space Model) and extracts high-level semantic features by learning the semantic associations in the feature sequence.

[0082] 2) Upsampling module: To gradually increase the resolution of the feature sequence and reduce the number of channels, we add an upsampling module (patch division) at the end of each stage. The upsampling module generates a new feature sequence by splitting and expanding the feature sequence. The specific operations are: ; This process helps the model restore the semantic information of the image at different scales and gradually recover high-level feature representations.

[0083] 3) Multi-stage processing: Similar to the encoder, the decoder also consists of multiple stages, each of which includes multiple ViMamba modules and an upsampling module. With each stage of processing, the resolution of the feature sequence gradually increases and the number of channels gradually decreases, gradually approaching the size and feature representation of the original image.

[0084] Finally, the feature sequence after all stages of processing , recovering video data of moving objects through semantic reconstructor , the reconstructed video data of the moving object can be expressed as: ; In this embodiment, the specific process of performing video semantic communication on the original video sequence is that the original video sequence is first processed by the semantic extractor, then undergoes four-stage processing and channel adaptation by the ViMamba encoder, and is transmitted through the physical channel. After receiving it, the ViMamba decoder also performs channel adaptation and four-stage processing and outputs it to the semantic reconstructor. Finally, the semantic reconstructor restores the video sequence.

[0085] It should be noted that the present invention is not limited to the above-mentioned embodiments. The above-mentioned embodiments are merely examples, and any embodiments having substantially the same structure and effect as the technical concept within the scope of the technical solution of the present invention are all included in the technical scope of the present invention. In addition, without departing from the scope of the present invention, other embodiments that can be conceived by those skilled in the art and that combine some of the constituent elements in the embodiments are also included in the scope of the present invention.

Claims

1. A goal-oriented video semantic communication system based on a visual model, characterized in that: include: The semantic extractor, based on the SAM2 model, includes an image encoder, a hint encoder, a mask decoder, and a memory module to process raw video, generate segmentation masks, and extract semantic information; The ViMamba encoder combines the Vision Mamba architecture with a downsampling module to channel encode the output of the semantic extractor; The channel adaptation module, including the transmitting-side channel adaptation and the receiving-side channel adaptation, is used to optimize the coding sequence based on the signal-to-noise ratio information of the physical channel, takes the output of the ViMamba encoder as input, and outputs the semantic feature sequence to the ViMamba decoder; The ViMamba decoder combines the Vision Mamba architecture and the upsampling module to perform channel decoding on the received semantic feature sequence to obtain a feature sequence; The semantic reconstructor is used to perform semantic reconstruction based on the feature sequence, restore the data and output the target video.

2. The goal-oriented video semantic communication system based on a visual model according to claim 1, characterized in that: The image encoder uses a hierarchical Hierarchical Vision Transformer to encode image features into multi-scale feature representations for a given original video sequence.

3. The goal-oriented video semantic communication system based on a visual model according to claim 1, characterized in that: The prompt encoder is used to encode the prompt input by the user into a feature vector.

4. The goal-oriented video semantic communication system based on a visual model according to claim 1, characterized in that: The memory module is used to store object information and previous segmentation results to track objects in video segmentation.

5. The goal-oriented video semantic communication system based on a visual model according to claim 1, characterized in that: The mask decoder is used to combine the features extracted by the image encoder, the prompt information provided by the prompt encoder and the memory data in the memory module to generate a final segmentation mask, and segment the moving object through the mask.

6. The goal-oriented video semantic communication system based on a visual model according to claim 1, characterized in that: The generation of the segmentation mask and the extraction of semantic information include combining the features extracted by the image encoder, the prompt information provided by the prompt encoder, and the memory data in the memory module to generate a final segmentation mask, and then multiplying the mask by the pixels of the corresponding frame to obtain a moving object image. The moving object images in all video frames are obtained through the same operation to obtain target semantic information of only the moving object.

7. The goal-oriented video semantic communication system based on a visual model according to claim 1, characterized in that: The ViMamba encoder performs channel coding including the following steps: Step 1: The encoder receives the output of the semantic extractor and performs block processing on the target image, dividing it into non-overlapping image blocks and linearly projecting the image blocks into a vector of fixed dimension; Step 2: Through several stacked ViMamba modules and patch embedding layers, based on the bidirectional state space model, high-level semantic features are extracted by learning the semantic associations between image patches; Step 3: Merge the features of adjacent image blocks through the downsampling module to generate a new feature sequence; Step 4: After multi-stage processing of ViMamba modules and downsampling modules, multi-scale feature representation is obtained, and finally feature encoding is obtained.

8. The goal-oriented video semantic communication system based on a visual model according to claim 1, characterized in that: The transmitting end channel adaptation is a precoding network based on SNR feedback. By learning the mapping relationship of SNR parameters, the weight of the received feature-coded coding sequence is adjusted, and finally all channel weights are integrated. The weights are then multiplied by the module input and the channels are shuffled to obtain the output.

9. The goal-oriented video semantic communication system based on a visual model according to claim 1, characterized in that: The receiving end channel adaptation is implemented in the same manner as the transmitting end channel adaptation, and is used to optimize the coding sequence according to the signal-to-noise ratio information of the channel to adapt to different channel conditions.

10. The goal-oriented video semantic communication system based on a visual model according to claim 1, characterized in that: The ViMamba decoder performs channel decoding comprising the following steps: Step 1: Extract high-level semantic features by learning semantic associations in feature sequences based on a bidirectional state-space model through several ViMamba modules; Step 2: Generate a new feature sequence by splitting and expanding the feature sequence through the upsampling module; Step 3: After multi-stage processing of ViMamba modules and downsampling modules, the size and feature representation close to the original image are obtained.

Images