A speech semantic communication transmission method based on time-channel dual attention

Abstract

The application belongs to the technical field of wireless communication and intelligent communication, and particularly relates to a voice semantic communication transmission method based on time-channel double attention, which comprises the following steps: obtaining original single-channel voice time domain signals and preprocessing to obtain preprocessed voice samples; performing step-by-step semantic feature extraction and compression on the preprocessed voice samples through a multi-level multi-scale convolution compression module to obtain low-dimensional semantic feature representation; processing the low-dimensional semantic feature representation by using a time-channel double attention mechanism module to obtain time sequence channel features; processing the time sequence channel features through a CNN layer and a shaping layer to convert the time sequence channel features into a symbol sequence suitable for physical channel transmission, and completing transmission through a channel transmission module; inputting the symbol sequence after being transmitted through the channel transmission module into a semantic decoder after being de-shaped to obtain a reconstructed voice signal; and the application improves the compression transmission efficiency and reconstruction quality of voice semantic communication in a complex channel environment.

Description

Technical Field

[0001] This invention belongs to the field of wireless communication and intelligent communication technology, specifically relating to the field of semantic communication technology, and particularly to a speech semantic communication transmission method based on time-channel dual attention. Background Technology

[0002] With the development of sixth-generation (6G) communication technology, communication systems are evolving from traditional human-to-human information transmission to human-to-machine and machine-to-machine intelligent interaction. Their goals are also gradually expanding from precise bit-level transmission to efficient semantic-level transmission. Semantic communication, with its core focus on the effective extraction, compression, and transmission of semantic information, demonstrates significant application value in low-bandwidth, complex channel, and intelligent interaction scenarios.

[0003] Currently, deep learning and end-to-end modeling have become important technical paths for realizing semantic communication. As a key branch of single-modal semantic communication, existing research methods mainly include systems based on convolutional neural networks (CNN), Transformer, gated recurrent units (GRU), and their combinations, such as DeepSC-S, DSST, DeepSC-TS, DeepSC-ST, and SAC-ST. These methods have made a series of advances in terms of semantic extraction accuracy, compression efficiency, and robustness.

[0004] However, existing technologies still have several limitations. First, some methods employ single-scale convolution or fixed receptive field structures, making it difficult to simultaneously capture local high-frequency details and long-term low-frequency prosodic information in speech, resulting in insufficient multi-scale semantic representation capabilities. Second, most existing attention mechanisms primarily focus on temporal dimension modeling, failing to adequately consider feature differences across channels, making it difficult to effectively distinguish between key semantic channels and redundant interference channels, thus limiting compression efficiency and semantic extraction accuracy under limited channel resources. Furthermore, some methods rely on preprocessing steps such as Mel spectrograms and spectrograms during data processing. While these can extract some features, they introduce additional computational overhead, hindering the deployment of real-time speech-semantic communication systems. In complex channel environments, existing solutions also generally face the challenge of balancing semantic recovery quality and model complexity. To address these issues, there is an urgent need for a speech-semantic communication transmission method that can directly process speech time-domain signals while simultaneously achieving multi-scale feature extraction, dual-dimensional attention modeling, and robustness to complex channels. Summary of the Invention

[0005] To address the above problems, this invention provides a speech semantic communication transmission method based on time-channel dual attention, comprising the following steps:

[0006] S1. Acquire the original single-channel speech time-domain signal and perform preprocessing to obtain a preprocessed speech sample; the preprocessing includes at least waveform normalization.

[0007] S2. Input the preprocessed speech samples into the semantic encoder, including:

[0008] S21. Semantic features are extracted and compressed step by step through a multi-level, multi-scale convolutional compression module to obtain a low-dimensional semantic feature representation; the multi-level, multi-scale convolutional compression module is composed of multiple convolutional feature extraction units cascaded together.

[0009] S22. Employ a time-channel dual attention mechanism module to jointly model the time dimension and channel dimension of the low-dimensional semantic feature representation to obtain temporal channel features;

[0010] S23. The temporal channel features are processed through CNN layers and shaping layers to transform them into symbol sequences adapted for transmission through physical channels;

[0011] S3. The symbol sequence is transmitted through the channel transmission module;

[0012] S4. The symbol sequence transmitted through the channel transmission module is de-shaped and then input into the semantic decoder to obtain the reconstructed speech signal; the semantic decoder includes a time-channel dual attention mechanism module and a multi-level multi-scale deconvolution module;

[0013] S5. Calculate the loss function based on the error between the reconstructed speech signal and the original single-channel speech time-domain signal, and perform end-to-end joint optimization training on the parameters of the semantic encoder and semantic decoder based on the loss function to obtain a model for speech semantic compression and transmission in complex channel environments.

[0014] The beneficial effects of this invention are:

[0015] This application's method overcomes the limitations of traditional single-scale feature extraction and single-dimensional attention modeling by introducing a multi-scale convolutional semantic extraction mechanism and a time-channel dual-path attention joint modeling mechanism into an end-to-end semantic communication framework. Specifically, the transmitting end extracts multi-resolution semantic features of the speech signal through a multi-level multi-scale convolutional compression structure and constructs a hierarchical semantic representation. The receiving end recovers the semantic features step by step through a corresponding multi-level multi-scale deconvolutional structure, thereby reducing the transmission load while ensuring the quality of speech semantic recovery.

[0016] Meanwhile, this invention employs a temporal attention path and a channel attention path to jointly enhance features, which can simultaneously highlight key temporal semantic segments and important feature channels, suppress redundant and interference information, and improve semantic expression and recovery capabilities under limited channel resource conditions.

[0017] Furthermore, this application preferably uses the speech time-domain signal directly as input for semantic encoding and decoding. Compared to schemes that rely on intermediate representations such as spectrograms or Mel spectrograms, this reduces the computational overhead and information loss risk caused by additional feature preprocessing, which is beneficial for real-time deployment and engineering implementation. Simulation results show that the corresponding model in this application improves on multiple evaluation metrics. Although the number of parameters increases in terms of complexity, the floating-point operation cost (FLOPs) is reduced by approximately 27.8% compared to the DeepSC-S model, achieving a good balance between semantic representation capability and computational efficiency.

[0018] This application combines multi-scale convolutional semantic extraction, temporal-channel dual attention joint modeling, complex channel semantic transmission, and symmetric decoding recovery to improve the compression transmission efficiency and reconstruction quality of speech semantic communication in complex channel environments, providing a practical approach for the design of speech semantic communication systems for those skilled in the art. Attached Figure Description

[0019] Figure 1 This is a diagram illustrating the overall structure of a speech semantic communication transmission method based on time-channel dual attention according to the present invention.

[0020] Figure 2 This is a schematic diagram of the time-channel dual attention mechanism module structure of the present invention;

[0021] Figure 3 The diagram shows the AWGN channel performance of this invention under different signal-to-noise ratios.

[0022] Figure 4 This is a performance diagram of the present invention under AWGN channels;

[0023] Figure 5 This is a performance diagram of the present invention under Rayleigh fading channels. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] This invention provides a speech semantic communication transmission method based on time-channel dual attention. The method includes three main stages: semantic encoding at the sending end, channel transmission, and semantic decoding at the receiving end. It also obtains a speech semantic compression and transmission model suitable for complex channel environments through end-to-end joint optimization training.

[0026] Please see Figures 1-2The method in this application corresponds to a speech semantic communication model (MSTC-SC) based on multi-scale convolution and time-channel dual attention. Its core is to extract multi-resolution semantic features of speech through multi-scale convolution and realize long-range dependency modeling in the time dimension and dynamic selection of channel dimension features through dual-path attention mechanism.

[0027] In some embodiments, a speech semantic communication transmission method based on time-channel dual attention includes the following steps:

[0028] S1. Acquire the original single-channel speech time-domain signal and perform preprocessing to obtain preprocessed speech samples.

[0029] Preferably, the preprocessing includes at least waveform normalization to form an input speech sample sequence suitable for one-dimensional convolution processing.

[0030] In one specific embodiment, the preprocessing of the original single-channel speech time-domain signal includes the following steps:

[0031] Normalization: A batch statistics-based standardization method is used to eliminate energy differences in the original single-channel speech time-domain signal. Specifically, the mean and variance of the original single-channel speech time-domain signal are calculated using PyTorch tools, and corresponding normalization processing is performed while maintaining the original dimensionality of the data.

[0032] Silence segment filtering: An amplitude threshold-based segment selection strategy is employed to remove invalid silence signals. Specifically, the normalized original single-channel speech time-domain signal is divided into continuous segments of a fixed length (e.g., 16384 sampling points). The average absolute amplitude of each segment is calculated, and an amplitude threshold of 0.015 is set. Segments below this threshold are identified as silence segments and discarded, thus filtering out segments containing valid speech information.

[0033] S2. Input the preprocessed speech samples into the semantic encoder, including:

[0034] S21. Semantic features are extracted and compressed step by step through a multi-level, multi-scale convolutional compression module to obtain a low-dimensional semantic feature representation.

[0035] Preferably, the multi-level multi-scale convolutional compression module is composed of multiple convolutional feature extraction units cascaded together; each convolutional feature extraction unit includes 4 parallel one-dimensional convolutional blocks, and the kernel size of the 4 one-dimensional convolutional blocks is a different prime number; the output of the 4 one-dimensional convolutional blocks is concatenated in the channel dimension and used as the output of the convolutional feature extraction unit.

[0036] Preferably, each convolutional feature extraction unit uses adaptive padding to ensure that the output size of its four one-dimensional convolutional blocks is consistent, and to ensure that the sequence length is compressed by decreasing in multiples of 4 after downsampling.

[0037] Preferably, in the multi-level, multi-scale convolutional compression module, each convolutional feature extraction unit sequentially performs a 4x downsampling on the preprocessed speech samples.

[0038] In one specific embodiment, a three-level multi-scale compression module (i.e., including three convolutional feature extraction units) is used to extract and compress semantic features from preprocessed speech samples step by step. Within each convolutional feature extraction unit, convolutional kernels of 3, 7, 13, and 19 are used in parallel to extract speech features. Based on the uniqueness of prime factors, the size of each convolutional kernel cannot be decomposed into smaller integer factors, mathematically avoiding potential periodic overlap during feature extraction. The design of the selected prime convolutional kernel size is adapted to the semantic scale depth of the speech signal at an 8kHz sampling rate, ensuring that its receptive field can fully cover various semantic primitives from high-frequency details to low-frequency structures, thereby ensuring that each convolutional kernel can accurately capture semantic information at its corresponding level. Small-sized prime kernels are used to extract high-frequency detail features, while large-sized prime kernels are used to capture large-scale low-frequency features; the two are directly concatenated along the channel dimension. Since the spatial sampling periods of convolutional kernels of different scales are coprime, non-overlapping coverage is achieved during channel concatenation, thus ensuring the independence and complementarity of multi-scale features.

[0039] The three-level multi-scale compression module performs a 4x downsampling on the preprocessed speech samples in each convolutional feature extraction unit. The output sequence length of the third convolutional feature extraction unit is compressed to 1 / 64 of the preprocessed speech sample length.

[0040] Table 1 illustrates the size variations of the three-level multi-scale compression module and the different characteristics of each compression extraction. The first level is for semantic feature extraction, the second level is for semantic feature enhancement, which uses multiple different convolutional kernels to enhance cross-lexical semantic dependencies and abstract the semantic structure within sentences; the third level is for global semantic aggregation, compressing the temporal length to 256 << 16384. The output of each level's convolutional feature extraction unit serves as the input to the next level. Through the fusion of multi-scale features, complementary enhancement of local and global features is achieved, making the encoder more complete in representing everything from low-level acoustic features to high-level semantic information due to the unique design of prime kernels, while compressing the amount of data.

[0041] Table 1 Three-level multi-scale compression module

[0042]

[0043] Furthermore, the number of output channels of the third convolutional feature extraction unit is 128. In order to reduce the number of parameters, this invention further reduces the 128 channels to 4 channels through CNN and sends information through 4 channels. This compresses the amount of output information (the amount of information of the output features of the CNN layer) to 1 / 16 of the amount of input information (the amount of information of the preprocessed speech samples), thereby reducing the channel transmission load and preserving the core semantic information.

[0044] S22. A time-channel dual attention mechanism module is adopted to jointly model the time dimension and channel dimension of the low-dimensional semantic feature representation to obtain the temporal channel features.

[0045] Preferably, this invention introduces a time-channel dual attention mechanism module at the key dimension transformation node of the semantic encoder to jointly model the low-dimensional semantic features in both the time and channel dimensions, thereby filtering key semantic temporal segments and important feature channels. For example... Figure 2 As shown, the time-channel dual attention mechanism module processes the input in the following ways:

[0046] S231. The input passes through a channel attention path to obtain channel attention features; in addition, the input is transposed and then passed through a time attention path to obtain time attention features.

[0047] Specifically, in the channel attention path, maintain input The original dimensions are processed to obtain channel attention features. , can be represented as:

[0048] ,

[0049] The temporal attention path achieves cross-dimensional collaborative modeling through dimension swapping, that is, by transposing the channel dimension to the time dimension to obtain features. Features were obtained after processing. , can be represented as:

[0050] ,

[0051] Then the features Transpose back to the original dimension to obtain the temporal attention features. .

[0052] In the formula, B represents the batch size, L represents the current sequence length, C represents the number of channels, ChannelAttn1() represents the channel attention path, and TimeAttn1() represents the time attention path.

[0053] S232. Concatenate the channel attention features and temporal attention features along the channel dimension to expand the information capacity and obtain dual-path features. , can be represented as:

[0054]

[0055] In the formula, Concat() represents concatenation.

[0056] S233. Use 1×1 convolution to perform cross-channel interaction and dimensionality compression on dual-path features to obtain compressed interactive features. , can be represented as:

[0057]

[0058] In the formula, W conv This represents a 1×1 convolution.

[0059] S234. To avoid degradation of deep networks, residual connections are used to enhance feature propagation, and compressed interactive features are added to the input to obtain temporal channel features.

[0060] Preferably, the channel attention path includes two cascaded channel attention modules, and the temporal attention path includes two cascaded temporal attention modules; both the channel attention modules and the temporal attention modules are composed of a Transformer encoder. The Transformer encoder consists of a multi-head self-attention (MHSA) layer and a feedforward network (FFN) layer, employing layer normalization and using residual connections to enhance training stability. Its core is the multi-head self-attention mechanism, which maps input features to multiple subspaces, enabling each attention head to independently capture dependencies in different dimensions, and ultimately concatenates the outputs of each head into a fused feature.

[0061] Specifically, the temporal attention module generates the query matrix Q through a linear transformation. t Key matrix K t Value matrix V t And perform multi-head self-attention calculations, including:

[0062] ,

[0063] in The feature is a learnable weight matrix. The features are divided into h independent heads, each processing a C / h dimensional subspace, and the self-attention weights are calculated:

[0064]

[0065] in The scaling factor is used. The multi-head results are concatenated and then linearly transformed before being output, achieving semantic information fusion across subspaces.

[0066] Similarly, the temporal attention module will input features Generate query matrix Q through linear transformation c Key matrix K c Value matrix V c :

[0067]

[0068] in The feature is a learnable weight matrix. The features are divided into h independent heads, each processing an L / h dimensional subspace, and the self-attention weights are calculated:

[0069]

[0070] in This is the scaling factor.

[0071] Preferably, to improve the performance of the Transformer, this invention divides the input of the FFN into two parts and applies independent linear transformations to balance parameter efficiency and feature diversity. Specifically, the tokens output by the MHSA are split into two parts along the last dimension, denoted as... Subsequently, a linear projection is used to generate the FFN, as described below:

[0072]

[0073] Where GELU() represents the function layer, Represents the learnable parameters of a linear transformation.

[0074] S23. The temporal channel features are processed by CNN layers and shaping layers to transform them into symbol sequences adapted for transmission through physical channels.

[0075] S3. The symbol sequence is transmitted through the channel transmission module.

[0076] Preferably, the channel transmission module includes an additive white Gaussian noise channel or a Rayleigh fading channel to simulate noise interference and multipath fading effects in actual communication, thereby making the model training and testing process closer to the actual application environment.

[0077] S4. The receiving end de-shapes the symbol sequence transmitted by the channel transmission module and inputs it into the semantic decoder to obtain the reconstructed speech signal; the semantic decoder includes a time-channel dual attention mechanism module and a multi-level multi-scale deconvolution module.

[0078] Preferably, at the receiving end, a time-channel dual attention mechanism module with the same structure is used to realize the detailed restoration of the de-shaped symbol sequence. At the same time, the semantic decoder adopts a multi-level multi-scale deconvolution module with a structure symmetrical to the multi-level multi-scale convolution compression module, matches the prime kernel size of the semantic encoder, and restores semantic features step by step through the multi-level multi-scale deconvolution module, ensuring accurate restoration of features at each scale and avoiding feature misalignment problems caused by asymmetric kernels.

[0079] S5. Calculate the loss function based on the error between the reconstructed speech signal and the original single-channel speech time-domain signal, and perform end-to-end joint optimization training on the parameters of the semantic encoder and semantic decoder based on the loss function to obtain a model for speech semantic compression and transmission in complex channel environments.

[0080] In one specific embodiment, to verify the effectiveness of the method in complex channel environments, the Tsinghua University THCHS30 Chinese speech dataset was used as the benchmark dataset, which includes 10,000 training samples and 2,495 test samples. The speech sampling frequency was 16 kHz, downsampled to 8 kHz in the experiment, and the sampling precision was 16 bits. Considering the differences in the duration of different speech samples, the input signal length was uniformly fixed at 16384 in the simulation. During training, the channel signal-to-noise ratio was set in the range of -10 dB to 20 dB. The model training used the Adam optimizer, with a batch size of 32, 100 training epochs, an initial learning rate of 0.001, and a stepped learning rate decay strategy, which decayed the learning rate by a factor of 0.3 at the 30th, 50th, and 70th epochs.

[0081] Furthermore, six training scenarios were set up with SNR values ​​of -10dB, -5dB, 0dB, 8dB, 16dB, and 20dB. After the model training was completed, it was tested in an AWGN channel. The experimental results are as follows: Figure 3As shown in the figure. Simulation results indicate that models trained with a fixed SNR typically exhibit better local performance in test SNR scenarios similar to their training SNR, demonstrating a certain degree of environmental adaptability. Specifically, models trained under higher SNR conditions achieve higher peak performance in high SNR test scenarios; models trained under lower SNR conditions show some stability in low SNR test scenarios, but their overall peak performance is relatively low. In contrast, models trained with SNR=8dB exhibit better overall performance and stability over a wider range of test SNRs, maintaining superior levels across multiple evaluation metrics. This is presumably because, under medium SNR training conditions, the training samples simultaneously contain some noise perturbation and identifiable semantic information, which is beneficial for the model to learn a general feature extraction strategy that balances noise resistance and semantic representation capabilities, thereby reducing the tendency to overfit to a single channel environment. In contrast, models trained with SNR=8dB exhibit lower sensitivity and better robustness across both high and low test SNR ranges.

[0082] In AWGN channel verification, such as Figure 4 As shown, the MSTC-SC model proposed in this application is compared with existing speech-semantic communication models such as DeepSC-S, DeepSC-S2, and DeepSC-TS. Simulation results show that DeepSC-S2, due to the introduction of a multi-scale convolutional structure, improves semantic fidelity metrics compared to DeepSC-S, but its improvement in waveform recovery and mean square error is limited; DeepSC-TS further improves performance compared to the aforementioned models. Compared with the above models, the MSTC-SC model proposed in this application performs better in metrics such as SDR and PESQ, indicating that the synergistic design of the multi-scale convolutional structure and the time-channel dual attention mechanism adopted in this application can more effectively extract semantically relevant features and improve speech reconstruction quality in additive noise environments.

[0083] In Rayleigh fading channel verification, such as Figure 5As shown, compared with the AWGN channel, the overall performance of all models in the Rayleigh fading channel is reduced, indicating that multiplicative fading distortion has a more significant impact on semantic transmission and reconstruction. The MSTC-SC model proposed in this application still outperforms the comparative models such as DeepSC-S2, DeepSC-S, and DeepSC-TS. When the test SNR is greater than 4dB, the MSTC-SC model has low sensitivity to changes in channel SNR and its performance remains relatively stable; when the test SNR is less than 4dB, although the performance of the MSTC-SC model decreases with the decrease in SNR, its decrease is smaller than that of the comparative models, showing better robustness. This result shows that, through the collaborative design of multi-scale convolution and temporal-channel dual attention, this application can more effectively highlight key semantic features and suppress disturbed features in the fading environment, thereby reducing the impact of multiplicative distortion on speech reconstruction.

[0084] In one specific embodiment, to evaluate the computational efficiency of the MSTC-SC model described in this application, a statistical analysis is performed on the number of model parameters and computational complexity. The computational complexity is preferably represented by floating-point operations per second (FLOPs), and the model size is preferably represented by the number of parameters. For a one-dimensional convolutional network structure, its FLOPs and number of parameters can be calculated based on the number of input channels, the number of output channels, the input feature length, and the convolutional kernel size. The FLOPs calculation method is as follows:

[0085]

[0086] Where Cin and Cout represent the number of input and output channels, L represents the length of the input features of the one-dimensional convolutional neural network, and K is the kernel size. The number of parameters can be calculated using the following formula:

[0087]

[0088] For the MHSA and FFN modules in the Transformer encoder, the calculation of the FLOPs required for FFN can be expressed as:

[0089]

[0090] and These represent the input and output dimensions, respectively. MHSA uses three different weight matrices: , and Attention head parameters are ,in That's the head count. The output is also processed through a linear layer, the weight matrix of which has a dimension of... C outThis is the dimension of the feature vector corresponding to a single token in the Transformer encoder. The number of parameters in the self-attention mechanism can be represented as:

[0091]

[0092] N L This indicates the number of layers in a Transformer encoder.

[0093] The calculation of the number of parameters in FFN, where , It is the size of the hidden layer. This is the weight matrix of the first fully connected layer. Since there are four linear projections in the NL layer, the number of parameters becomes:

[0094]

[0095] Table 2 shows the number of parameters and FLOPs for the corresponding models. MSTC-SC has a higher number of parameters than DeepSC-S, mainly due to the additional parameters introduced by the multi-scale convolutional branches and dual-path attention mechanism. These parameters enhance the feature representation capability, especially the capture of multi-scale semantic information in speech signals. Despite the increased number of parameters, MSTC-SC's FLOPs are 27.8% lower than DeepSC-S, indicating that the proposed method is more efficient in terms of computational complexity. This result demonstrates that the structure described in this application effectively reduces redundant computational overhead while improving semantic representation capability and robustness to complex channels, thus possessing good engineering deployment value.

[0096] Table 2 Comparison of FLOPs and Parameter Quantities

[0097]

[0098] Furthermore, as shown in Table 2, the MSTC-SC model described in this application achieves a good balance between model size and computational efficiency, and is particularly suitable for edge devices, real-time voice interaction, and semantic communication scenarios with limited bandwidth and computing power. It achieves a significant reduction in FLOPs while increasing the number of parameters, demonstrating high computational efficiency and having more advantages in semantic communication under resource-constrained scenarios.

[0099] To verify the necessity of the multi-scale prime convolution kernel and time-channel dual attention mechanism in the core modules of the model proposed in this application, ablation experiments were conducted on different module combinations. The experiments were carried out under the condition of 8dB signal-to-noise ratio in AWGN and Rayleigh fading channels. Mean square error (MSE), signal-to-distortion ratio (SDR), and perceived speech quality (PESQ) were used as evaluation indicators, and the results are shown in Table 3.

[0100] Table 3 Comparison of ablation test performance of core modules (8dB SNR)

[0101]

[0102] Experimental results show that, compared to the complete MSTC-SC model presented in this paper, a single convolutional kernel group (3×1) reduces the MSE in the AWGN channel from 2.62×10⁻⁻⁶. 4 Increased to 9.16×10⁻ 4 (Improvement of more than 2 times) and a 4.5dB reduction in SDR confirm the necessity of multi-scale design for capturing multi-dimensional speech features. After replacing prime kernels with multi-scale convolutional kernel groups, the AWGN channel PESQ decreased by 0.1 and SDR decreased by 1.6, demonstrating the advantage of prime kernels in suppressing feature redundancy. At the same time, compared with the 3×1, 7×1, 13×1, and 19×1 convolutional kernels used in this paper, the 5×1, 7×1, 11×1, and 17×1 kernels are slightly inferior in terms of performance indicators, verifying the advantages of the prime convolutional kernels used in this paper. The SDR of the time / channel attention group alone decreased by 3.5~4.5 compared with the complete dual attention group, and the MSE also increased to a certain extent, indicating that the synergistic effect of dual attention can effectively optimize feature weight allocation, thereby improving the performance of the model.

[0103] This application effectively addresses the problem of efficient compression and transmission of speech and semantics in complex channel environments. Despite the increased parameter count due to the need for enhanced feature representation, it achieves a balance between parameter count and computational complexity by effectively reducing redundant computations through one-dimensional multi-scale convolution and optimizing the computational path through a dual-path attention mechanism. In AWGN and Rayleigh fading channels, the model significantly outperforms other models in both SDR and PESQ metrics, demonstrating strong robustness against additive and multiplicative noise. MSTC-SC overcomes the performance bottleneck of traditional models through structural innovation, exhibiting excellent computational efficiency and resistance to channel interference, providing an efficient and practical solution for speech and semantic transmission in resource-constrained scenarios such as edge devices and real-time communication.

[0104] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A speech-semantic communication transmission method based on time-channel dual attention, characterized in that, Includes the following steps: S1. Acquire the original single-channel speech time-domain signal and perform preprocessing to obtain preprocessed speech samples; The preprocessing includes at least waveform normalization; S2. Input the preprocessed speech samples into the semantic encoder, including: S21. Semantic features are extracted and compressed step by step through a multi-level, multi-scale convolutional compression module to obtain a low-dimensional semantic feature representation; the multi-level, multi-scale convolutional compression module is composed of multiple convolutional feature extraction units cascaded together. S22. Employ a time-channel dual attention mechanism module to jointly model the time dimension and channel dimension of the low-dimensional semantic feature representation to obtain temporal channel features; S23. The temporal channel features are processed through CNN layers and shaping layers to transform them into symbol sequences adapted for transmission through physical channels; S3. The symbol sequence is transmitted through the channel transmission module; S4. The symbol sequence transmitted through the channel transmission module is de-shaped and then input into the semantic decoder to obtain the reconstructed speech signal; the semantic decoder includes a time-channel dual attention mechanism module and a multi-level multi-scale deconvolution module; S5. Calculate the loss function based on the error between the reconstructed speech signal and the original single-channel speech time-domain signal, and perform end-to-end joint optimization training on the parameters of the semantic encoder and semantic decoder based on the loss function to obtain a model for speech semantic compression and transmission in complex channel environments.

2. The speech-semantic communication transmission method based on time-channel dual attention as described in claim 1, characterized in that, Each convolutional feature extraction unit consists of four parallel one-dimensional convolutional blocks. The four one-dimensional convolutional blocks use different prime numbers for their kernel sizes: 3, 7, 13, and 19. The outputs of the four one-dimensional convolutional blocks are concatenated along the channel dimension and used as the output of the convolutional feature extraction unit.

3. The speech-semantic communication transmission method based on time-channel dual attention according to claim 1, characterized in that, In the multi-level multi-scale convolutional compression module, each convolutional feature extraction unit performs a 4x downsampling on the preprocessed speech sample in sequence; when the multi-level multi-scale convolutional compression module uses 3 convolutional feature extraction units, the output sequence length of the 3rd convolutional feature extraction unit is compressed to 1 / 64 of the preprocessed speech sample.

4. The speech-semantic communication transmission method based on time-channel dual attention according to claim 1, characterized in that, By using a CNN layer to compress the number of channels to 4, information is transmitted through 4 channels, and the amount of output information is compressed to 1 / 16 of the original amount of input information.

5. The speech-semantic communication transmission method based on time-channel dual attention according to claim 1, characterized in that, The time-channel dual attention mechanism module processes the input in the following ways: S231. The input passes through a channel attention path to obtain channel attention features; additionally, the input is transposed and then passed through a temporal attention path to obtain temporal attention features. S232. Concatenate the channel attention features and the temporal attention features along the channel dimension to obtain dual-path features; S233. Use 1×1 convolution to perform cross-channel interaction and dimensionality compression on dual-path features to obtain compressed interactive features; S234. Add the compressed interactive features to the input to obtain the time-series channel features.

6. The speech-semantic communication transmission method based on time-channel dual attention according to claim 5, characterized in that, The channel attention path consists of two cascaded channel attention modules, and the temporal attention path consists of two cascaded temporal attention modules; both the channel attention modules and the temporal attention modules are composed of Transformer encoders.

7. The speech-semantic communication transmission method based on time-channel dual attention according to claim 6, characterized in that, In the Transformer encoder, the output of MHSA is split into two parts along the last dimension, and the processing of FFN is represented as follows: ， In the formula, m and n represent the two parts of the input obtained by splitting the output of MHSA along the last dimension, and GELU() represents the function layer. Represents the learnable parameters of a linear transformation.

8. The speech-semantic communication transmission method based on time-channel dual attention according to claim 1, characterized in that, The semantic decoder employs a multi-level, multi-scale deconvolution module that is symmetrical to the structure of the multi-level, multi-scale convolution compression module.

Images