An adaptive code rate transmission method and system based on multi-modal semantic importance

Abstract

The application discloses a kind of adaptive code rate transmission method and system based on multimodal semantic importance, belong to semantic communication technical field.The application collects video, audio, text, IoT multimodal data and pre-processes, extracts light-weight semantic features by CNN-Transformer fusion BERT, completes model pruning by means of differentiable mask, Markov priori;SIS model outputs semantic score in the interval 0~1, relies on LoRA to realize model online fine-tuning;Sliding window monitoring network is used to construct a code rate optimization model that maximizes semantic fidelity.Differentiate high and low semantic units with threshold, and use QPSK and LDPC to realize differential encoding;High semantic data is decoded first at the receiving end, and when bandwidth is insufficient, core semantics is transmitted in degraded mode, suitable for XR, Internet of Vehicles and other industrial scenarios.

Description

Technical Field

[0001] This invention relates to the field of semantic communication technology, and specifically to an adaptive code rate transmission method and system based on multimodal semantic importance. Background Technology

[0002] There are already many mature solutions for multimodal data adaptive bitrate transmission in current wireless networks. The existing implementation approaches mainly fall into three categories. The first category is the adaptation solution driven by network physical indicators, represented by MPEG-DASH and WebRTC GCC. This type of solution continuously monitors transmission parameters such as link throughput, transmission latency, packet loss rate, and jitter, and dynamically adjusts the bitrate only according to the network carrying capacity. It ensures stable link transmission through methods such as segmentation switching and bandwidth estimation, but does not distinguish the information value of the transmitted content itself. The second category is the single-modal content adaptation solution optimized for a single data stream. Typical methods include video ROI region of interest encoding and audio VAD voice activity detection. It only performs differentiated compression within the video and audio modalities, and only explores the information differences within a single modality. It cannot coordinate the information weights between visual, audio, and sensor text. The third category is the multimodal independent parallel transmission mechanism. In scenarios such as remote collaboration and industrial monitoring, video, audio, operation text, and sensor data are encoded and transmitted independently. Each modality adjusts its bitrate independently according to its own rules, and there is no collaborative scheduling logic between modalities.

[0003] All three types of existing technologies mentioned above have inherent defects that cannot be avoided. First, all solutions focus on bit fidelity as the core optimization goal, completely ignoring the semantic value differences of multimodal data. When network congestion or bandwidth contraction occurs, critical business data such as fault alarm text and obstacle videos occupy the same bandwidth as meaningless background images and environmental noise, resulting in synchronous distortion of core semantics and seriously affecting the execution reliability of safety-sensitive services such as autonomous driving and telemedicine. Second, existing optimization methods are limited to a single modal dimension, severing the semantic relationships between different modalities and failing to weigh the contribution of various heterogeneous data to downstream tasks from a global perspective, making it difficult to meet the overall service quality requirements of multimodal fusion services. Third, the multimodal parallel transmission mode lacks a unified cross-modal priority scheduling mechanism. When bandwidth resources are tight, all modalities simultaneously reduce their bit rate, which can easily cause critical instructions and early warning information to be damaged simultaneously. Fourth, traditional multimodal feature fusion models have high computational overhead and high feature redundancy, and have not been designed for lightweight edge computing devices such as vehicle terminals and XR headsets, making large-scale deployment difficult.

[0004] In summary, existing technologies lack a closed-loop transmission framework that is centered on semantic fidelity while also considering lightweight feature extraction, cross-modal semantic quantization, and global collaborative bandwidth allocation. This makes it impossible to prioritize the stable transmission of high semantic value information in resource-constrained scenarios. Therefore, there is an urgent need to propose a novel adaptive bitrate transmission scheme. Summary of the Invention

[0005] To achieve the above objectives, the present invention provides the following technical solution: An adaptive code rate transmission system based on multimodal semantic importance includes: The sending end acquisition module is used to acquire multimodal data of video, audio and text, and performs standardized preprocessing of video frame synchronization, audio noise reduction and text word segmentation, respectively, to achieve time and format alignment of multimodal data; The semantic processing node includes a multimodal semantic feature extraction submodule, a network state awareness submodule, a cross-modal semantic importance assessment model submodule, and a global bitrate optimization decision engine. The multimodal semantic feature extraction submodule tokenizes multimodal data. Video and audio tokens are input into the CNN-Transformer fusion model, and a differentiable mask is generated through Gumbel-Sigmoid. Markov uniform prior smoothing constraint is introduced to prune convolution parameters. Text tokens are used to extract features through BERT, outputting lightweight cross-modal semantic features, and mapped to a unified semantic vector space through contrastive learning. The network state awareness submodule uses a sliding window averaging method to collect available bandwidth, latency, packet loss rate, and jitter, and predicts network change trends. The cross-modal semantic importance evaluation model submodule (SIS Model) is trained end-to-end with MSE as the loss and outputs SIS values ​​in the range of 0 to 1; it receives feedback from the receiver and fine-tunes the model online through LoRA low-rank updates. The global bitrate optimization decision engine aims to maximize the total semantic fidelity, with constraints that the total bitrate ≤ bandwidth and the bitrate of each modality ≥ the minimum threshold, and solves the optimal bitrate allocation scheme. An adaptive encoder group is used to configure the QPSK modulation and LDPC channel coding rates according to the code rate scheme, and adopts an LDPC inequality error protection mechanism. The network transport layer is compatible with 5G / 6G / WiFi and schedules the transmission of encoded data according to the priority of the SIS value. The receiving module decodes according to the SIS value from high to low and uses semantic completion to repair lost semantic units; the acquisition task execution feedback is sent back to the semantic processing node; when the available bandwidth is lower than the sum of the minimum bit rates of all modalities, the semantic degradation mechanism is triggered to transmit only the high SIS core semantic data.

[0006] To achieve the above-mentioned objectives, the present invention also provides an adaptive code rate transmission method based on multimodal semantic importance, comprising the following steps: S1. Multimodal data acquisition, preprocessing, and link establishment: Acquire multimodal data including video stream, audio stream, and text data; perform frame synchronization and format standardization on the video stream; perform noise reduction and sampling rate unification on the audio stream; perform structured word segmentation on the text data to obtain a time-synchronized and format-compatible multimodal data stream; establish a wireless transmission link and start real-time monitoring of the link status to obtain network status parameters; S2. Lightweight Multimodal Semantic Feature Extraction: Tokenize the multimodal data stream by segmenting video frames into image blocks and obtaining a video token sequence through linear projection; convert audio signals into Mel spectrograms and obtain an audio token sequence through frame segmentation; and split text sentences into sub-word units to obtain a text token sequence. Input the video token sequence and audio token sequence into a CNN-Transformer fusion model to generate a query vector Q, a key vector K, and a value vector V. Generate a differentiable mask using the Gumbel-Sigmoid function. ,in , For temperature parameters, For learnable parameters, To eliminate noise, and based on Markov uniform prior constraints. Where b is Expected value These are strength parameters used to control uniformity. Pruning the convolution parameters yields lightweight semantic features for video, audio, and text. S3, Cross-modal semantic importance assessment: The lightweight semantic features of video, audio and text are mapped to a unified vector space through a contrastive learning algorithm to obtain multimodal semantic units. The multimodal semantic units include video semantic units, audio semantic units and text semantic units. The semantic importance score (SIS) of the multimodal semantic units is quantified through the cross-modal semantic importance assessment model. S4. Network Status Acquisition and Global Bitrate Decision: Acquire the network status parameters of the link status monitored in real time in step S1; predict the network status change trend based on the network status parameters; construct a bitrate allocation optimization model, with the objective function being to maximize the total semantic fidelity. , For the transmission quality of the i-th multimodal semantic unit, the constraints are: total bit rate ≤ current available bandwidth, and bit rate of each modality ≥ minimum transmission quality threshold; solve the optimization model to obtain the optimal transmission bit rate; S5. Differentiated coding and adaptive transmission: Set modulation and coding parameters according to the code rate allocation scheme, and adopt a low-density parity check (LDPC) unequal error protection mechanism: For multimodal semantic units with high SIS values, allocate a lower LDPC code rate to introduce more parity redundancy; for multimodal semantic units with low SIS values, allocate a higher LDPC code rate to introduce less parity redundancy; transmit the encoded multimodal semantic units through a wireless network. S6. Detect the multimodal semantic units encoded in step S5; when some semantic units are detected to be missing, infer and restore the missing semantics through semantic completion technology; collect task completion information, including task success rate, error type, and semantic understanding deviation, and generate feedback information for online fine-tuning of the cross-modal semantic importance assessment model and dynamic optimization of bitrate decision. S7. Dynamic Adaptation and Anomaly Handling: The semantic processing node receives network status updates and task execution feedback in real time and dynamically updates the bitrate allocation scheme; when the available network bandwidth is detected to be increased, the bitrate of low SIS multimodal semantic units is increased to improve the overall transmission quality. When a decrease in available network bandwidth is detected, priority is given to ensuring the bit rate of multimodal semantic units with high SIS values, while reducing the transmission quality of multimodal semantic units with low SIS values. If the available bandwidth is lower than the sum of the lowest bit rates of all modalities, a semantic degradation mechanism is triggered, and only core semantic data with high SIS values ​​is retained for transmission.

[0007] Furthermore, the Markov uniform prior constraint in step S2 is modified by a smoothing constraint. Implementation, where K is the number of output dimension channels.

[0008] Furthermore, the cross-modal semantic importance evaluation model in step S3 is trained using an end-to-end supervised learning framework: constructing an importance-labeled dataset of multimodal semantic units and generating a semantic importance score (SIS) for each multimodal semantic unit; with the training objective of minimizing the mean squared error (MSE) between the predicted SIS and the manually labeled true weights, the model parameters are iteratively updated through the backpropagation algorithm, enabling the model to learn to recognize the feature patterns corresponding to high semantic importance.

[0009] Furthermore, the setting of modulation and coding parameters in step S5 follows a mapping relationship. Where X is the modulation order, r is the optimal transmission code rate obtained in step S4, and B is the currently available bandwidth; the required LDPC code rate is obtained by back-calculating the channel transmission rate obtained from global optimization.

[0010] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Build a complete mathematical model to support lightweight semantic extraction and break through the traditional bit-fidelity transmission paradigm: achieve gradient-free blocking feature pruning through Gumbel-Sigmoid differentiable mask and Markov prior probability mathematical framework, take semantic importance weighted total fidelity as the core bandwidth scheduling basis, prioritize the complete transmission of high-value semantics such as fault instructions and operation signals in network congestion and bandwidth-limited scenarios, and greatly improve the reliability of safe business operations such as autonomous driving and telemedicine. 2. A mathematical framework for full-process quantitative semantic evaluation, adaptable to unified scoring of multiple heterogeneous modalities: The SIS model is trained under supervision based on MSE loss, and LoRA low-rank update formula is used to achieve low-cost online fine-tuning. The semantic contribution of video, audio and sensor text is uniformly quantified, which solves the defect of existing technologies that lack a unified cross-modal semantic evaluation standard. 3. Constrained global bitrate optimization model to achieve cross-modal collaborative bandwidth scheduling: Construct an objective function that maximizes the total semantic fidelity, superimposes bandwidth and minimum quality constraints, and globally coordinates the allocation of multimodal bitrates to solve the problems of bandwidth contention and core semantic synchronization distortion caused by traditional multimodal independent transmission, and significantly improves the task completion success rate of multimodal fusion services; 4. Channel rate mapping + LDPC differential protection significantly improves transmission robustness: relying on The formula automatically matches the redundancy of differentiated channel coding, increases error correction and check bits for high semantic data, saves bandwidth for low semantic data, integrates real-time network monitoring and bidirectional feedback of service execution, and automatically classifies and degrades transmission when bandwidth fluctuates or channel deteriorates. 5. Lightweight mathematical models lower the threshold for edge deployment and significantly improve bandwidth resource utilization: The mask pruning mathematical model greatly reduces the computational load of the multimodal fusion model and can be directly deployed on computing-constrained devices such as vehicle terminals, XR headsets, and industrial sensors; it also reduces the bit rate and channel verification redundancy for low semantic value background images and environmental noise, thereby reducing ineffective bandwidth consumption. 6. Strong versatility in business scenarios and broad expansion space: The basic mathematical framework is compatible with all multimodal wireless transmission scenarios such as vehicle networking, telemedicine, industrial metaverse, and intelligent security. It can be extended and integrated with derivative optimization solutions such as federated learning distributed training, edge computing, energy consumption sensing scheduling, and satellite heterogeneous network transmission. Attached Figure Description

[0011] Figure 1 Diagram of a lightweight semantic feature extraction architecture for multimodal applications; Figure 2 Here is a diagram of the SIS cross-modal semantic importance assessment model structure; Figure 3 Overall architecture diagram of the adaptive bit rate transmission system Figure 4A business process diagram for XR remote collaboration scenarios. Detailed Implementation

[0012] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0013] like Figure 1 As shown, an adaptive code rate transmission method based on multimodal semantic importance includes the following steps: S1: Multimodal data acquisition, preprocessing, and link establishment The sending end synchronously acquires multimodal raw data through the acquisition module, including: Video stream: Captures continuous video frames through a camera, such as environmental images in XR scenarios, equipment monitoring images in industrial scenarios, and images of roads, pedestrians, traffic signs, and surrounding vehicles in vehicle-to-everything (V2X) scenarios. Audio stream: Acquires audio signals such as voice and alarm sounds through a microphone, such as voice commands for remote collaboration and equipment fault alarm sounds; Text: Obtain text commands through the UI interface and physical quantity data through IoT sensors, such as operation command text and temperature / pressure sensor data.

[0014] The acquired multimodal data is preprocessed: video streams undergo frame synchronization and format standardization to unify resolution and frame rate; audio streams undergo noise reduction and sampling rate unification; and text undergoes structured processing to ensure time synchronization and format compatibility of multimodal data.

[0015] Establish a wireless transmission link and start real-time monitoring of the link status to obtain network status parameters.

[0016] S2: Lightweight Multimodal Semantic Feature Extraction The semantic processing node receives preprocessed multimodal data and processes it through the multimodal semantic feature extraction module, such as... Figure 1 As shown, video, audio, and text are tokenized separately. For the video modality, sampling is first performed along the temporal dimension to obtain a keyframe sequence. Each frame is segmented into fixed-size image blocks, which are then linearly projected into visual tokens. Simultaneously, spatial and temporal location encodings are fused to form a video token sequence with spatiotemporal information. For the audio modality, the original audio signal is first converted into a Mel spectrogram, then framed along the time axis and segmented along the frequency axis. Each spectrogram block is linearly projected to obtain an audio token sequence. For the text modality, sentences are split into sub-word units using a word segmenter, mapped to text tokens through a word embedding layer, and similarly supplemented with text location encoding. Finally, the data from all three modalities are uniformly transformed into a token sequence of the same dimension. The text sequence is then used for feature extraction using the BERT model.

[0017] The token sequences from the video and audio modalities are input into the CNN-Transformer fusion model. A convolutional layer is integrated before the multi-head attention module of the Transformer model, and a masking module is added to the output channel to achieve convolutional projection to generate attention parameters. In the calculation After attention scoring, a masking operation is performed. The masking modules follow a Bernoulli distribution. Direct sampling is non-differentiable and would block gradient backpropagation. Gumbel-Sigmoid provides a differentiable approximation method through reparameterization techniques. ,in , For temperature parameters, For learnable parameters, For noise, the obtained By incorporating the parameters from the previous convolutional layer, we obtain... After pruning one layer of the convolutional layer, in order to ensure the integrity of feature extraction, the mask part needs to be uniformly distributed. Therefore, Markov prior constraint mask generation process is introduced, Markov uniform prior. Core probability distribution: b is Expected value Strength parameters used to control uniformity Where: partition function K is the number of output dimension channels.

[0018] Smoothing constraints Based on the observed data and preset rules Find a binary sequence This maximizes the probability of its occurrence. Based on this, the optimal sequence is then used. A unified feature extraction is performed on complete modal data. At the same time, this prior is used to simplify the calculation logic and compress redundant operations, thereby effectively reducing the overall computational complexity while ensuring the integrity of feature representation.

[0019] S3: Cross-modal semantic importance assessment A contrastive learning algorithm maps the semantic features of video, audio, and text from different modalities to a unified vector space. Based on this unified semantic feature space, a cross-modal semantic importance assessment model (SIS Model) quantifies the semantic importance score (SIS value) of each modality's multimodal semantic unit. Figure 2 As shown: 1. Task-oriented semantic annotation: For specific downstream tasks, such as remote surgery and autonomous driving, construct a multimodal semantic importance annotation dataset to annotate the influence weight of different modal multimodal semantic units on task completion; 2. Model Training: A deep learning prediction model is employed. Lightweight semantic features, obtained in step 2, are used as input after being mapped to a unified vector space. In the feature fusion stage, a self-attention mechanism is first introduced to model long-range dependencies between elements within the sequence, capturing contextual semantic association patterns. Next, the fused high-level features undergo nonlinear transformation and dimensionality compression via a multilayer perceptron (MLP), mapping the features to a low-dimensional representation space. Finally, the output layer uses the sigmoid activation function to generate a semantic importance score (SIS) for each multimodal semantic unit, ranging from 0 to 1, with higher values ​​indicating higher semantic importance. The model's training objective is to minimize the mean squared error (MSE) between the predicted SIS value and the manually labeled true weights. Through backpropagation, the model parameters are iteratively updated, learning to effectively identify feature patterns corresponding to high semantic importance. This end-to-end supervised learning framework, which learns the mapping of SIS scores in the unified semantic feature space, enables the model to automatically extract task-related discriminative information from cross-modal semantic representations. 3. Dynamic Adjustment: Combining task execution feedback, such as the task completion status and error feedback at the receiving end, LoRA is used for low-rank updates to fine-tune the SIS model parameters online, ensuring the dynamic adaptability of semantic importance assessment. For example, in emergency task scenarios, the SIS value weight of alarm data is dynamically increased.

[0020] For example, in autonomous driving tasks, the SIS values ​​of video frame regions of obstacles ahead, collision warning audio, and braking command text are all close to 1, while the SIS values ​​of background scenery video and environmental noise are close to 0.1.

[0021] S4: Network Status Acquisition and Global Bitrate Decision 1. Network Status Awareness: The network status awareness module monitors key indicators of the transmission link in real time, including available bandwidth, transmission latency, packet loss rate, and jitter. It uses the sliding window averaging method to calculate the mean and trend of network status parameters and predict network status changes, such as bandwidth decline trends and the risk of sudden increases in latency. 2. Global Bitrate Optimization Decision: The global bitrate optimization decision integrates the SIS value and network state parameters to construct a bitrate allocation optimization model. Objective function: Maximize overall semantic fidelity. ,in For the first The transmission quality of each data unit is constrained by the following conditions: total bit rate ≤ current available bandwidth, minimum transmission quality of each modality ≥ threshold, to ensure that basic semantics are not lost. Decision output: For each modal multimodal semantic unit, under the constraints that the total bitrate does not exceed the current available bandwidth and each modality is not lower than the minimum transmission quality threshold, maximize the sum of the semantic importance weighted quality of all multimodal semantic units and output the bitrate allocation scheme.

[0022] S5: Differentiated Coding and Adaptive Transmission The adaptive encoder group encodes each modal multimodal semantic unit (MMS) based on the configuration parameters output by the global bit rate optimization decision. Employing an unequal error protection mechanism, when processing mixed-modal streams, the system first performs demultiplexing and priority classification, and determines the QPSK modulation scheme. Subsequently, based on the globally optimized mapping relationship between channel transmission rate, modulation scheme, and channel coding rate (channel transmission rate = bandwidth × log2 (modulation order) × channel coding rate), the channel coding rate is obtained. An unequal LDPC protection mechanism is used for encoding. For highly important MMS units, a lower LDPC bit rate is allocated, introducing more parity redundancy and forming stronger error correction capabilities, thus achieving "high-fidelity" recovery even when the channel is damaged, ensuring the integrity of the core semantics. For less important MMS units, a higher LDPC bit rate is allocated, introducing less parity redundancy, resulting in relatively weaker error correction capabilities, trading "low fidelity" for higher net information throughput.

[0023] In some embodiments, this step dynamically divides high-priority and low-priority semantic units based on the average SIS value of batch semantic units: the average SIS value of all multimodal semantic units to be transmitted is calculated, and semantic units with SIS values ​​greater than or equal to this average value are high-SIS semantic units, while the rest are low-SIS semantic units. Different LDPC code rates are configured for each to achieve hierarchical protection. Through the above mechanism, limited channel protection resources are preferentially allocated to data that plays a decisive role in the overall quality or downstream tasks, realizing a paradigm shift from "indiscriminate uniform protection" to "semantically aware differentiated protection," significantly improving the system's resource utilization efficiency and distortion resistance performance.

[0024] S6: Detect the multimodal semantic units encoded in step S5. 1. Semantic Reconstruction and Task Execution: After receiving the transmitted data, the receiving end processes it through the semantic reconstruction and task module: based on the encoding parameters and SIS values ​​of each modality, it prioritizes decoding data with high SIS values ​​and uses semantic completion technology to repair possible transmission distortions. Semantic completion is a technique for semantic communication: when channel transmission causes the loss of some semantic units, the receiving end uses a shared knowledge base to infer and recover the missing semantics from the incomplete data through a semantic decoder. This is a common method for data recovery in semantic communication systems. For example, context-based completion of lost key instruction fragments; subsequently, the reconstructed multimodal data is aligned, such as synchronizing video footage with voice commands and associating sensor data with the user interface; 2. Task Execution Feedback: After the receiving end executes the downstream task, it collects the task completion status through the task execution feedback module, such as task success rate, error type, and semantic understanding deviation. The feedback information is then sent back to the semantic processing node of the sending end for online fine-tuning of the SIS model and dynamic optimization of bit rate decision.

[0025] S7: Dynamic Adaptation and Exception Handling 1. Dynamic Adaptation: Semantic processing nodes receive real-time network status updates and task execution feedback, dynamically updating the bitrate allocation scheme. Specifically, multimodal semantic units with a semantic importance score (SIS) not lower than a predetermined threshold are defined as high SIS data, while other SIS values ​​below the predetermined threshold are classified as medium-low SIS data. When network bandwidth increases, the bitrate of medium-low SIS data is increased to improve overall transmission quality; when network bandwidth decreases, priority is given to ensuring the bitrate of high SIS data, while the transmission quality of medium-low SIS data is appropriately reduced. 2. Anomaly Handling: If the network condition continues to deteriorate, such as when the available bandwidth is lower than the sum of the lowest bit rates of all modalities, a semantic degradation mechanism is triggered, and only core semantic data with high SIS values ​​is retained for transmission; if the receiver reports that the semantic understanding error rate exceeds the threshold, the transmission of low SIS value data is suspended, and the bandwidth is concentrated to ensure the lossless transmission of high SIS value data.

[0026] The present invention also includes a system for an adaptive code rate transmission method based on multimodal semantic importance, comprising the following structure: 1. The hardware of the transmitting end acquisition module includes a high-definition camera array, an omnidirectional microphone, an IoT sensor acquisition interface, a human-computer interaction UI acquisition interface, a synchronous timing controller, and a data preprocessing unit. The timing controller uniformly triggers the synchronous acquisition of video, audio, and text sensor data, ensuring that the timestamps of all data streams are aligned. The preprocessing unit has built-in video standardization subunit, audio noise reduction sampling rate unification subunit, and text structured word segmentation subunit. It executes the preprocessing step S1 in the manual and outputs a preprocessed multimodal data stream with a unified format and synchronous timing, which is transmitted to the semantic processing node through a high-speed local bus.

[0027] 2. Semantic processing node, implemented using embedded edge computing chip, internally deployed with four sub-modules in parallel, with high-speed data flow between modules via memory: (1) Multimodal feature extraction sub-module: built-in video tokenization unit, audio tokenization unit, text BERT word embedding unit, CNN-Transformer fusion unit, Gumbel-Sigmoid mask operation unit, Markov prior probability solving unit; completes all S2 mathematical formulas, completes feature pruning, and outputs lightweight cross-modal unified semantic features; (2) Cross-modal semantic importance evaluation model sub-module (SIS) Model): Built-in contrastive learning feature alignment unit, multi-head self-attention calculation unit, MLP mapping unit, Sigmoid scoring output unit, MSE loss calculation unit, LoRA low-rank update operation unit; runs S3 full loss and weight update formula, outputs SIS value in the 0~1 interval, and only updates the low-rank matrix after receiving feedback to complete online fine-tuning; (3) Network state perception submodule: Built-in sliding window mean calculation unit, bandwidth delay real-time monitoring unit, network trend prediction unit, continuously collects wireless link physical indicators and outputs standardized network state parameters; (4) Global code rate optimization decision engine: Built-in constrained optimization solver, substitutes the total semantic fidelity objective function, bandwidth and minimum quality constraints, numerically solves the optimal code rate allocation scheme, and outputs the coding configuration parameters of each modality multimodal semantic unit.

[0028] 3. The adaptive encoder group includes a video hardware encoder, an audio encoder, and a text / sensor data compression unit. Each encoding unit integrates a QPSK modulation module, an LDPC unequal error protection coding submodule, and a channel rate formula calculation unit; the received bit rate allocation scheme and SIS classification identifier are based on... It automatically calculates and matches LDPC bitrates, distinguishes between high SIS and medium-low SIS data, configures differentiated check redundancy, and generates encoded transmission data streams.

[0029] 4. The network transmission layer is compatible with 5G / 6G and WiFi6 / 7 wireless communication modules, and has a built-in data stream priority scheduling queue to allocate higher transmission scheduling priority to high SIS encoded data streams, carrying the encoded data to the receiving end.

[0030] 5. The receiver module hardware includes a wireless receiver module, a multi-level priority decoder, a semantic completion operation unit, a service task execution unit, and a feedback transmission unit. The decoder decodes data in descending order of SIS score. The semantic completion unit has a built-in service shared knowledge base and repairs lost or distorted semantic units based on context. The task execution unit completes multimodal fusion service operations. The feedback transmission unit sends feedback information such as task error rate and task completion status back to the semantic processing node through a reverse wireless link to achieve closed-loop adjustment.

[0031] Example 1 Taking an XR remote collaboration scenario as an example, the system workflow is as follows: Figure 4 As shown: S1. Multimodal data acquisition, preprocessing, and link establishment: 1. The transmitting end acquires environmental video from the XR device. User voice commands Operation text instructions After preprocessing by standardizing the format and aligning the time, a multimodal data stream is obtained. ; 2. After preprocessing, the data is transmitted to the semantic processing node to extract cross-modal semantic features; The preprocessed multimodal data stream Each is tokenized to obtain .

[0032] video: ; Audio: Where M is the Mel spectrum; text: S2, Lightweight Multimodal Semantic Feature Extraction: The semantic processing node on the sending side receives the preprocessed data stream.

[0033] right Input CNN-Transformer: Introducing differentiable masks Will get The parameters of the previous convolutional layer are incorporated to obtain... To ensure the integrity of feature extraction, the mask portion needs to maintain a uniform distribution. Therefore, a Markov prior is introduced to constrain the mask generation process. The Markov uniform prior is as follows: Given observation data and preset rules Find the binary mask sequence with the highest probability. By applying the mask to projection and attention calculations, semantic features are obtained: , Lightweight semantic features extracted from videos. Lightweight semantic features extracted from audio. .

[0034] S3, Cross-modal semantic importance assessment: Receive the output of step S2 , and Contrastive learning is used to align multimodal features to a unified vector space S: Unified semantic features Self-attention + MLP + Sigmoid: The training objective is: Where N is the total number of training samples, This represents the true value of the nth manually labeled sample.

[0035] S4. Network Status Acquisition and Global Bitrate Decision: Obtain the network state parameters monitored in real time in step S1, and construct a bitrate allocation optimization model: The objective function of the model is: constraint: in The allocation of the i-th modality cannot be less than the minimum bitrate threshold required to avoid loss of basic semantics.

[0036] The optimal transmission rate can be obtained by solving the problem. .

[0037] S5, Differential Coding and Adaptive Transmission: Each modal multimodal semantic unit is encoded, and an unequal error protection mechanism is employed. When processing mixed-mode streams, the system first performs demultiplexing and priority classification, and determines the QPSK modulation scheme. Subsequently, based on the globally optimized mapping relationship between channel transmission rate, modulation scheme, and channel coding rate: Where X is the modulation order, r is the optimal transmission code rate obtained in step S4, and B is the currently available bandwidth.

[0038] The channel coding rate is obtained through the above mapping formula. Then, the LDPC inequality protection mechanism is used for coding to perform differentiated protection.

[0039] S6. Detect the multimodal semantic units encoded in step S5: After channel coding as described above, the signal enters the wireless channel for transmission. At the receiving end, the semantic features that were damaged or partially missing after transmission are then processed. Input the semantic completion module and output the fully recovered semantic features. This is used for subsequent multimodal fusion and task execution.

[0040] S7. Dynamic Adaptation and Exception Handling: The sending side provides online adaptive feedback on the task execution status, fine-tunes the SIS model, and adjusts the original pre-trained weight parameters of the SIS model. Perform a low-rank update and freeze the original weights. Only low-rank matrices A and B are trained, thus enabling efficient fine-tuning of the model. Both A and B are learnable low-rank matrices; A is responsible for upscaling low-dimensional features into a high-dimensional space, while B performs the reverse process.

[0041] Output the fine-tuned SIS value .

[0042] Glossary Downstream task: refers to the final business objective to be executed by the receiving end using the reconstructed multimodal semantic information. It is the only benchmark for SIS value labeling and transmission effect evaluation. Typical scenarios include "on-site equipment troubleshooting" in XR remote collaboration and "driving safety decision making" in vehicle networking.

[0043] Semantic completion refers to a technique where, when the receiving end detects a lost data unit, it does not request retransmission but infers and recovers the missing semantics based on the knowledge anchor state and cross-modal unified mapping rules. Its core basis is the pre-aligned semantic consensus between the sending and receiving parties, rather than an additionally constructed shared knowledge base.

[0044] End-to-end supervised learning framework: refers to the training logic of the cross-modal semantic importance assessment model (SIS model): taking multimodal raw data units as input and manually labeled "impact weights on downstream task completion" as supervision signals, the model automatically learns the mapping relationship from data to SIS values ​​without the need for manual design of intermediate feature extraction rules, and iteratively optimizes through backpropagation using MSE as the loss function.

[0045] Dynamic adaptation strategy: refers to the bandwidth adjustment mechanism when generating a bitrate allocation scheme. It includes three sub-strategies: increasing the bitrate of low SIS data units when bandwidth is sufficient, prioritizing the bitrate of high SIS data units when bandwidth decreases, and triggering semantic degradation to transmit only core data units when there is extreme congestion. It is the core mechanism for dealing with network fluctuations.

[0046] Low-Density Parity-Check (LDPC) coding is a linear block error-correcting code, belonging to the category of channel coding techniques. LDPC coding is the core implementation method for achieving differentiated transmission and semantic priority protection. By introducing specific check redundancy bits, the receiver can autonomously detect and repair data errors caused by channel noise or congestion during transmission without requesting retransmission.

[0047] Online fine-tuning: During the operation of the semantic transmission service, a lightweight parameter optimization mechanism is performed on the cross-modal semantic importance assessment model (SIS model) based on the actual business feedback returned by the receiving end. It is a specific application of incremental update of machine learning model and can complete the model performance iteration without interrupting the current transmission service.

Claims

1. An adaptive code rate transmission system based on multimodal semantic importance, characterized in that, include: The sending end acquisition module is used to acquire multimodal data of video, audio and text, and performs standardized preprocessing of video frame synchronization, audio noise reduction and text word segmentation, respectively, to achieve time and format alignment of multimodal data; The semantic processing node includes a multimodal semantic feature extraction submodule, a network state awareness submodule, a cross-modal semantic importance assessment model submodule, and a global bitrate optimization decision engine. The multimodal semantic feature extraction submodule tokenizes multimodal data. Video and audio tokens are input into the CNN-Transformer fusion model, and a differentiable mask is generated through Gumbel-Sigmoid. Markov uniform prior smoothing constraint is introduced to prune convolution parameters. Text tokens are used to extract features through BERT, outputting lightweight cross-modal semantic features, and mapped to a unified semantic vector space through contrastive learning. The network status awareness submodule uses a sliding window averaging method to collect available network bandwidth, latency, packet loss rate, and jitter, and predicts network change trends. The cross-modal semantic importance evaluation model submodule is trained with end-to-end supervision using MSE as the loss and outputs SIS values ​​in the range of 0 to 1. Receive feedback from the receiver and fine-tune the model online by updating the low-rank LoRA. The global bitrate optimization decision engine aims to maximize the total semantic fidelity, with constraints that the total bitrate ≤ bandwidth and the bitrate of each modality ≥ the minimum threshold, and solves the optimal bitrate allocation scheme. An adaptive encoder group is used to configure the QPSK modulation and LDPC channel coding rates according to the code rate scheme, and adopts an LDPC inequality error protection mechanism. The network transport layer is compatible with 5G / 6G / WiFi and schedules the transmission of encoded data according to the priority of the SIS value. The receiving module decodes according to the SIS value from high to low and uses semantic completion to repair lost semantic units; the acquisition task execution feedback is sent back to the semantic processing node; when the available bandwidth is lower than the sum of the minimum bit rates of all modalities, the semantic degradation mechanism is triggered to transmit only the high SIS core semantic data.

2. An adaptive code rate transmission method based on multimodal semantic importance, characterized in that, Includes the following steps: S1. Multimodal data acquisition, preprocessing, and link establishment: Acquire multimodal data including video stream, audio stream, and text data; perform frame synchronization and format standardization on the video stream; perform noise reduction and sampling rate unification on the audio stream; perform structured word segmentation on the text data to obtain a time-synchronized and format-compatible multimodal data stream; establish a wireless transmission link and start real-time monitoring of the link status to obtain network status parameters; S2. Lightweight Multimodal Semantic Feature Extraction: Tokenize the multimodal data stream by segmenting video frames into image blocks and obtaining a video token sequence through linear projection; convert audio signals into Mel spectrograms and obtain an audio token sequence through frame segmentation; and split text sentences into sub-word units to obtain a text token sequence. Input the video token sequence and audio token sequence into a CNN-Transformer fusion model to generate a query vector Q, a key vector K, and a value vector V. Generate a differentiable mask using the Gumbel-Sigmoid function. ,in , For temperature parameters, For learnable parameters, To eliminate noise, and based on Markov uniform prior constraints. Where b is Expected value The convolution parameters are pruned to control uniformity, resulting in lightweight semantic features for video, audio, and text. S3, Cross-modal semantic importance assessment: The lightweight semantic features of video, audio and text are mapped to a unified vector space through a contrastive learning algorithm to obtain multimodal semantic units. The multimodal semantic units include video semantic units, audio semantic units and text semantic units. The semantic importance score (SIS) of the multimodal semantic units is quantified through the cross-modal semantic importance assessment model. S4. Network Status Acquisition and Global Bitrate Decision: Acquire the network status parameters of the link status monitored in real time in step S1; predict the network status change trend based on the network status parameters; construct a bitrate allocation optimization model, with the objective function being to maximize the total semantic fidelity. Q i For the transmission quality of the i-th multimodal semantic unit, the constraints are: total bit rate ≤ current available bandwidth, and bit rate of each modality ≥ minimum transmission quality threshold; solve the optimization model to obtain the optimal bit rate allocation scheme for each modality of multimodal semantic unit; S5. Differentiated coding and adaptive transmission: Set modulation and coding parameters according to the code rate allocation scheme, and adopt a low-density parity check (LDPC) unequal error protection mechanism: For multimodal semantic units with high SIS values, allocate a lower LDPC code rate to introduce more parity redundancy; for multimodal semantic units with low SIS values, allocate a higher LDPC code rate to introduce less parity redundancy; transmit the encoded multimodal semantic units through a wireless network. S6. Detect the multimodal semantic units encoded in step S5; when some semantic units are detected to be missing, infer and restore the missing semantics through semantic completion technology; collect task completion information, including task success rate, error type, and semantic understanding deviation, and generate feedback information for online fine-tuning of the cross-modal semantic importance assessment model and dynamic optimization of bitrate decision. S7. Dynamic Adaptation and Anomaly Handling: The semantic processing node receives network status updates and task execution feedback in real time and dynamically updates the bitrate allocation scheme; when the available network bandwidth is detected to be increased, the bitrate of low SIS multimodal semantic units is increased to improve the overall transmission quality. When a decrease in available network bandwidth is detected, priority is given to ensuring the bit rate of multimodal semantic units with high SIS values, while reducing the transmission quality of multimodal semantic units with low SIS values. If the available bandwidth is lower than the sum of the lowest bit rates of all modalities, a semantic degradation mechanism is triggered, and only core semantic data with high SIS values ​​is retained for transmission.

3. The adaptive code rate transmission method based on multimodal semantic importance according to claim 2, characterized in that, The Markov uniform prior constraint in step S2 is passed through a smoothing constraint. Implementation, where K is the number of output dimension channels.

4. The adaptive code rate transmission method based on multimodal semantic importance according to claim 2, characterized in that, The cross-modal semantic importance evaluation model in step S3 is trained using an end-to-end supervised learning framework: an importance labeling dataset for multimodal semantic units is constructed, and a semantic importance score (SIS) value is generated for each multimodal semantic unit; with the training objective of minimizing the mean squared error (MSE) between the predicted SIS value and the manually labeled true weights, the model parameters are iteratively updated through the backpropagation algorithm, so that the model learns to recognize the feature patterns corresponding to high semantic importance.

5. The adaptive code rate transmission method based on multimodal semantic importance according to claim 2, characterized in that, The setting of modulation and coding parameters in step S5 follows a mapping relationship. Where X is the modulation order, r is the optimal transmission code rate obtained in step S4, and B is the currently available bandwidth; The required LDPC code rate is derived by working backward from the channel transmission rate obtained through global optimization.

Images