A speech semantic signal transmission method based on Swin-Transformer

Abstract

The application discloses a voice semantic communication transmission method based on Swin-Trasformer, and comprises the following steps: 1, data preprocessing is carried out on a voice signal; 2, the voice signal after preprocessing is input into a semantic encoder, and semantic signals in the voice signal are automatically extracted; 3, the semantic signals are input into a physical channel to simulate noise interference in real life; 4, the semantic signals with noise interference are input into a semantic decoder; the signals with noise are respectively input into a convolution layer and a Transformer-based Decoder layer, shallow features and deep features of the signals are reconstructed, and finally, multi-dimensional semantic information is fused through a KeepfeatLayer; 5, signal reconstruction is carried out on the signals after the semantic decoder, so that the original voice signal is recovered. The application directly takes a time domain signal as a system input, and the cost of calculation is reduced, which is extremely important for an intelligent communication system that pays attention to real-time transmission.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of semantic communication, and particularly relates to a speech denoising decoding method based on a Swin-Transformer network structure. BACKGROUND

[0002] With the rapid increase of intelligent application requirements of wireless communication, future communication networks will change from the architecture of simply pursuing high transmission rate to a new architecture of wisdom connection of all things. Semantic communication is considered as a promising technology and is expected to become a new basic paradigm of future wisdom connection network of all things. Unlike traditional communication transmission of bit symbols, semantic communication transmits the meaning of information, its core is "intention communication" or "content communication", and it enables computers to understand deeper information in human signals, and is a truly artificial intelligent communication method.

[0003] Semantic communication has characteristics of high accuracy, low bandwidth, flexible and changeable application scenarios, and has become a research hotspot. The existing semantic transmission method mostly starts from the perspective of time domain or frequency domain, restores the original speech signal at the receiving end, but is strongly affected by noise and interference, and it is difficult to meet the actual demand. SUMMARY

[0004] The purpose of the application is to solve the above problems, and provide a speech semantic signal transmission method based on a Swin-Transformer network architecture.

[0005] In order to achieve the above purpose, the application adopts the following technical scheme:

[0006] A speech semantic communication transmission method based on Swin-Trasformer, the method comprises the following steps:

[0007] S1. Data preprocessing is performed on the speech signal; first, the input speech signal is subjected to data normalization operation, then after frame windowing and frame shifting, a plurality of equal-length speech signal frames are obtained, and then the data of each frame is spliced to obtain a two-dimensional matrix composed of speech frames as the input form of the entire network;

[0008] S2. The preprocessed speech signal is subjected to semantic encoding to automatically extract the semantic signal in the speech signal;

[0009] S3. The semantic signal is subjected to physical channel to simulate the noise interference in real life;

[0010] S4. The semantic signal with noise interference is passed through a semantic decoder; the signal with noise is passed through a convolution layer and a Transformer-based Decoder layer respectively, to reconstruct the shallow features and deep features thereof, and finally through a Keepfeat Layer to realize fusion of multi-dimensional semantic information;

[0011] S5. The signal passed through the semantic decoder is reconstructed to recover the original voice signal.

[0012] The beneficial effects of the present application are realized as follows:

[0013] The present application has the advantages that:

[0014] 1. The original voice signal does not need to be preprocessed, and the time domain signal is directly taken as the system input, thereby reducing the calculation cost, which is extremely important for an intelligent communication system that focuses on real-time transmission.

[0015] 2. By reconstructing the features of the time domain signal, the present application can realize accurate recovery of the original time domain signal at the receiving end without increasing the transmission data amount.

[0016] 3. The Swin-Transformer network is innovatively introduced at the receiving end, and the semantic features of the transmitted signal can be fully extracted by using the multi-head attention mechanism, so as to facilitate subsequent reconstruction.

[0017] 4. Compared with other semantic communication systems, the present system can be competent for more complex and variable channel environments, and has excellent noise resistance, and can still perform well at a very low signal-to-noise ratio. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 The overall structure of the voice semantic communication system;

[0019] Figure 2 The Transformer-based Decoder network of the present application;

[0020] Figure 3 Shifted-window mechanism diagram;

[0021] Figure 4 Upsample Layer network structure;

[0022] Figure 5 Performance of DeepSC-TS under an AWGN channel;

[0023] Figure 6 Performance of DeepSC-TS under a Rayleigh channel; DETAILED DESCRIPTION

[0024] The application will be further described below in conjunction with the accompanying drawings and examples.

[0025] As Figure 1 The overall structure of the voice semantic communication system of the application is shown, and the voice semantic communication system based on Swin-Transformer is named DeepSC-TS.

[0026] S1. Data preprocessing is performed on the voice signal; first, the input voice signal is subjected to data normalization operation, then subjected to frame windowing, and after frame shifting, a plurality of equal-length voice signal frames are obtained, and then the data of each frame is spliced to obtain a two-dimensional matrix composed of voice frames as the input form of the entire network;

[0027] S2. The preprocessed voice signal is passed through a semantic encoder to automatically extract the semantic signal in the voice signal;

[0028] S3. The semantic signal is passed through a physical channel to simulate the noise interference in real life;

[0029] S4. The semantic signal with noise interference is passed through a semantic decoder;

[0030] S5. The signal after the semantic decoder is subjected to signal reconstruction to recover the original voice signal.

[0031] Further, step S1 is implemented as follows:

[0032] First, a segment of original voice signal is sampled in the time domain to obtain a string of data points, and then the data points are subjected to normalization operation.

[0033] In the application, Z-score normalization method is adopted, specifically, the mean and variance of each sample are calculated first, and then the data distribution of the sample is converted to standard normal distribution S norm , which is realized by the following formula:

[0034]

[0035] Wherein, μ and σ represent the mean and variance of each sample respectively. S represents the original voice signal; since the length of each voice signal sample is inconsistent, in simulation, the input voice signal is first sampled at 8kHZ, then the length of the sampled signal is fixed as M, and each sample is composed of F frames, and the length of each frame is L, and the original one-dimensional voice signal is converted into a two-dimensional matrix tensor as the input of the entire semantic communication system.

[0036] Further, the specific implementation process of step S2 is described as follows:

[0037] S E =E α (s) (2)

[0038] Wherein, s represents the original input speech signal, E α (·) represents semantic encoder, S E The semantic signal after semantic feature extraction by semantic encoder.

[0039] In the application, the semantic encoder adopts two-dimensional convolutional neural network, and the correlation of sample point data features is obtained through matrix convolution, translation and other operations.

[0040] The specific structure of the two-dimensional convolutional neural network is as follows:

[0041] It is composed of two layers of two-dimensional convolutional neural network, the output channel number of the first layer is 32, the convolution kernel size is 3*3, the step is 2, and the padding is 1; the output channel number of the second layer is 1, the convolution kernel size is 1*1, the step is 1, and the padding is 0

[0042] Further, the specific implementation process of step S3 is described as follows:

[0043] S LQ =C(S E ) (3)

[0044] Wherein, S LQ Indicates the low-quality semantic signal after the physical channel, C(·) indicates the physical channel model, S E Is the semantic signal in step S2.

[0045] In the application, in order to be more close to the actual transmission environment, the signal after the semantic encoder is divided into IQ two ways in the channel transmission, and each way of signal adds noise.

[0046] Considering that there is additive noise and multiplicative noise in the actual environment, and the signal-to-noise ratio has a greater influence, therefore, for the noise of AWGN channel, a Gaussian random variable with mean 0 and variance Is used, wherein, p(s) represents the power of input signal; for Rayleigh fading channel, a Gaussian random variable with mean 0 and variance Is used, respectively multiplied by IQ two signals to simulate.

[0047] Further, the specific implementation process of step S4 is described as follows:

[0048] S D =D β (S C ) (4)

[0049] where S D represents the decoded signal, S C is the semantic encoded signal after the physical channel in step S3, D β represents the semantic decoder.

[0050] The semantic decoder includes a shallow feature extraction module and a deep feature extraction module, which will be described in detail below.

[0051] Shallow feature extraction module: given a low-quality speech signal S LQ after the physical channel, a 3x3 two-dimensional convolution SF CONV (·) is used to extract shallow features, which contain low-dimensional information, and can be specifically represented as:

[0052] F S = SF CONV (S LQ ) (5)

[0053] Deep feature extraction module: for speech signals, high-frequency information is often contained in deep semantic features; moreover, deep feature extraction means that the features learned by the network are more abstract, making the model have strong generalization ability. In the present application, the deep feature extraction module includes three parts: PatchEmbed, Transformer-based Decoder and Keepfeat Layer.

[0054] The role of the PatchEmbe layer is to represent each region in the input semantic signal as a vector, which is used to represent different regions in the semantic signal and input to the subsequent Transformer-based Decoder layer for processing.

[0055] Specifically, a 3x3 two-dimensional convolution P CONV (·) is used to change the original CxFxL semantic signal to P C x P F x P L , and finally, the three-dimensional matrix is flattened into a two-dimensional vector S PE as the input of the subsequent Transformer-based Decoder, where The above process is represented as:

[0056] S PE = F(P CONV (S LQ )) (6)

[0057] where S PEF represents the vector group obtained after PatchEmbed, F represents the flattening operation, S LQ represents the low-quality semantic signal after the physical channel.

[0058] The Transformer-based Decoder is the core architecture of the entire deep feature extraction network, which is mainly based on the Swin-Transformer network, and residual skip connection is adopted between each network block, and the overall structure is as shown in Figure 2 .

[0059] As shown in Figure 2 (a), the Transformer-based Decoder network is composed of a plurality of Residual-Transformer Layers; and as shown in Figure 2 (b), each Residual-Transformer Layer is composed of a plurality of Swin-Transformer Layers. Therefore, the signal after PatchEmbed is denoted as S PE , and the process of the semantic signal through the Transformer-based Decoder can be represented as:

[0060] F D(i,j) = RTL i (STL j (S PE )) (7)

[0061] Wherein, RTL i represents the i-th Residual-Transformer Layer in the Transformer-based Decoder; STL j represents the j-th Swin-Transformer Layer in the Residual-Transformer Layer; F D(i,j) represents the deep feature value extracted after the entire decoder.

[0062] After the input signal passes through the Transformer-based Decoder, sufficient semantic similarity information has been extracted. In order to further improve the correlation of these feature information, the Keepfeat Layer is designed, which is mainly composed of three two-dimensional convolutions, which is used to improve the integration ability of the model to the deep features, so as to optimize the overall performance of the network. Specifically, it can be represented by the formula:

[0063] F DR = KF CONV (FD ) (8)

[0064] The Keepfeat Layer is composed of three two-dimensional convolutional neural networks, with output channel numbers of 24, 24, and 96, respectively; convolution kernel sizes of 3x3, 1x1, and 3x3, respectively; step lengths of 2, 1, and 2, respectively; and padding of 1, 0, and 1, respectively

[0065] After the semantic feature extraction at the decoding end, S LQ Both the shallow features and the deep features are fully extracted, and finally added directly at the end, represented as:

[0066] F=F S +F DR (9)

[0067] where F S and F DR represent the shallow features and the deep features, respectively, and the final semantic signal contains multiple levels of information, with strong representation ability, laying a foundation for the subsequent semantic signal reconstruction work.

[0068] Unlike traditional Transformers, Swin-Transformer innovatively introduces W-MSA and SW-MSA, which makes the attention mechanism calculation no longer limited to text information, greatly expanding its application field.

[0069] W-MSA captures the mutual dependence between each feature by dividing the input features into windows of equal size and performing self-attention calculation within each window. For the speech semantic information in this paper, before inputting into the Swin-Transformer, the size of the input information is first changed from FxLxC to (M is the size of each window), that is, the original input is divided into multiple windows of the same size. For the features of each window Their respective Q, K, and V can be calculated as follows:

[0070] Q=W×W Q , K=W×W K , V=W×W V (10)

[0071] where W Q , W K , and W V are the weight matrices of each window.

[0072] Finally, similar to traditional Transformers, the self-attention within each window is calculated as follows:

[0073]

[0074] where d is the dimension of Q, K, V.

[0075] Since W-MSA calculates the self-attention of each window matrix, it only obtains the relationship between different positions in a local window, which will lose the connection between windows and limit the receptive field of the whole network, so it is difficult to pay attention to more global information. Therefore, SW-MSA is needed to improve it. As shown in Figure 3 (a), an information region is divided into four small windows, where the size of each window is M = 4. In order to obtain the interaction information between windows, the main strategy is to move the whole input information window down and right by one unit, and then repeat the W-MSA calculation of self-attention steps, which can obtain the connection between each window. But this will bring a problem that the original four windows have become nine, which will undoubtedly increase the calculation amount. The method to solve this problem is as shown in (b), the top and left blocks are moved down and right, and the number of windows is returned to four. In order to prevent information leakage and cross caused by attention mechanism when operating across non-adjacent sub-blocks, Swin-Transformer uses masks for shifted-window operation, which can make two different parts not participate in the calculation of attention, ensure that the semantic information of the original signal will not change, and improve the performance of the whole network. Figure 3

[0076] Further, the step S5 is specifically implemented as follows:

[0077]

[0078] wherein, is the reconstructed speech signal, R γ is a signal reconstructor, S D is the signal after semantic decoding in step S4.

[0079] The signal reconstructor R γ is composed of an Upsample Layer and a Last Layer. In the present method, the basic component unit of the Upsample Layer is as shown in Figure 4 , wherein the two two-dimensional convolutions before and after are to fix the features and increase the generalization of the whole network; and the up-sampling operation in the middle adopts Interpolate, which mainly controls the size of the target size by adjusting the size of the scale-factor. The above process is represented as:

[0080] ​S' = R CONV2 (R I (R CONV2 (F))) (13)

[0081] where R CONV1 , R CONV2 , R I represent convolution and up-sampling operation respectively.

[0082] The specific structure parameters of the two-dimensional convolution are as follows:

[0083] R CONV1 and R CONV2 have the same structure, and the number of output channels is 64, the size of the convolution kernel is 3x3, the step is 2, and the padding is 1

[0084] The Last Layer is composed of a two-dimensional convolutional neural network, and the main purpose is to reduce the channel dimension of the decoded signal, so as to achieve the same size as the original input signal.

[0085] The Last Layer is composed of three two-dimensional convolutional neural networks, and the number of output channels is 32, 32, and 1 respectively; the size of the convolution kernel is 3x3, 1x1, and 3x3 respectively; the step is 2, 1, and 2 respectively; and the padding is 1, 0, and 1 respectively

[0086] Then the reconstructed signal is compared with the original speech signal, and since the goal is to recover as much feature information as possible close to the original signal, it is expected that the value of each sampling point of the reconstructed signal is the same as the original signal, therefore, the loss function used can be specifically represented as:

[0087]

[0088] where C, F, and L represent the number of channels, the number of frames, and the frame length of the input speech signal respectively, and s and represent the original speech signal and the reconstructed speech signal, i, j, and k represent each time domain sampling point in the sample, and ε is a very small constant, which is 0.001 here. The purpose of increasing this constant is to keep the value of the loss function stable.

[0089] As shown in Tables 1 and 2, under the AWGN channel, our average value is 3.9723285575769234, and the comparison is 2.4422112586923075; under the Rayleigh channel, our is 2.1588110224999997, and the comparison is 1.198105006423077

[0090] Table 1

[0091]

[0092] Table 2

[0093]

[0094] In this embodiment, our goal is to train a semantic communication system with strong anti-noise ability to cope with harsh channel environment, so we choose to train in the environment of SNR = 1. We use adaptive moment estimation (Adam) algorithm as the optimizer of network training, and the initial learning rate is 0.01.

[0095] We simulate the proposed semantic communication system model in AWGN channel and Rayleigh channel environment respectively, and the results show that, as shown in Figure 5 and Figure 6 in the interval of SNR greater than or equal to -15dB and less than 10dB, the performance of DeepSC-TS proposed by us is better than DeepSC-S, especially in the extremely low SNR environment, it can still maintain excellent performance.

Claims

1. A speech semantic signal transmission method based on Swing-Transformer, characterized in that... Includes the following steps: S1. Perform data preprocessing on the speech signal; S2. The semantic signal in the preprocessed speech signal is automatically extracted by the semantic encoder; S3. Transmit semantic signals through physical channels to simulate noise interference in real life; S4. The noisy semantic signal is passed through a semantic decoder; the noisy signal is passed through a convolutional layer and a Transformer-based Decoder layer respectively to reconstruct its shallow and deep features, and finally through KeepfeatLayer to achieve the fusion of multi-dimensional semantic information; The specific implementation process is described as follows: S D =D β (S C ) (4) Among them, S D S represents the decoded signal. C D is the semantically encoded signal after passing through the physical channel in step S3. β This represents a semantic decoder; The semantic decoder includes a shallow feature extraction module and a deep feature extraction module; Shallow feature extraction module: Given a low-quality speech signal S after passing through a physical channel LQ Using 3×3 two-dimensional convolution SF CONV (·) is used to extract shallow features, which contain low-dimensional information, specifically represented as: F S =SF CONV (S LQ ) (5) The deep feature extraction module consists of three parts: PatchEmbed, Transformer-based Decoder, and Keepfeat Layer. The PatchEmbed layer represents each region in the input semantic signal as a vector. These vectors are used to represent different regions in the semantic signal and are then input into the subsequent Transformer-based Decoder layer for processing. S5. Reconstruct the signal from the semantic decoder to recover the original speech signal.

2. The speech semantic signal transmission method based on Swing-Transformer according to claim 1, characterized in that... The data preprocessing in step S1 is implemented as follows: 1-1. Sample a raw speech signal in the time domain to obtain a series of data points, and then normalize these data points; 1-2. The Z-score normalization method used first calculates the mean and variance of each sample, and then transforms the sample data distribution into a standard normal distribution S. norm Specifically, this is achieved through the following formula: Where μ and σ represent the mean and variance of each sample, respectively; S represents the original speech signal; since the length of each speech signal sample is inconsistent, in the simulation, the input speech signal is first sampled at 8kHz, and then the length of the sampled signal is fixed at M, and each sample consists of F frames, each frame having a length of L, and the original one-dimensional speech signal is converted into a two-dimensional matrix tensor as the input of the entire semantic communication system.

3. The speech semantic signal transmission method based on Swing-Transformer according to claim 2, characterized in that... The specific implementation process of step S2 is described as follows: S E =E α (s) (2) Where s represents the original input speech signal, E α (·) represents the semantic encoder, S E This represents the semantic signal after semantic feature extraction by the semantic encoder.

4. The speech semantic signal transmission method based on Swing-Transformer according to claim 3, characterized in that... The semantic encoder uses a two-dimensional convolutional neural network to obtain the correlation of sample point data features through matrix convolution and translation operations. The two-dimensional convolutional neural network consists of two layers. The first layer has 32 output channels, a kernel size of 3×3, a stride of 2, and padding of 1. The second layer has 1 output channel, a kernel size of 1×1, a stride of 1, and padding of 0.

5. The speech semantic signal transmission method based on Swing-Transformer according to claim 3, characterized in that... The specific implementation process of step S3 is described as follows: S LQ =C(S E ) (3) Among them, S LQ This represents the low-quality semantic signal after passing through the physical channel, where C(·) represents the physical channel model, and S... E This refers to the semantic signal in step S2; The signal after semantic encoding is split into two paths, I and Q, and transmitted in the channel. Noise is added to each path; the noise used in the AWGN channel has a mean of 0 and a variance of . The Gaussian random variable is given by p(s), where p(s) represents the power of the input signal; for Rayleigh fading channels, a mean of 0 and a variance of are used. The Gaussian random variable is simulated by multiplying it by the I and Q signals respectively.

6. The speech semantic signal transmission method based on Swing-Transformer according to claim 3, characterized in that... The deep feature extraction module uses a 3×3 two-dimensional convolution P CONV (·), transforming the original C×F×L semantic signal into P C ×P F ×P L Finally, the three-dimensional matrix is ​​flattened into a two-dimensional vector S. PE As input to the subsequent Transformer-based Decoder, where The above process can be represented as follows: S PE =F(P CONV (S LQ )) (6) Among them, S PE This represents the vector group obtained after PatchEmbed, where F represents the flattening operation, and S... LQ This indicates a low-quality semantic signal after passing through the physical channel; The Transformer-based Decoder network consists of several Residual-Transformer Layers; each Residual-Transformer Layer is composed of several Swin-Transformer Layers; therefore, the signal after PatchEmbed is denoted as S. PE The semantic signal is then represented by the Transformer-based Decoder as follows: F D(i,j) =RTL i (STL j (S PE )) (7) Among them, RTL i This represents the i-th Residual-Transformer Layer in the Transformer-based Decoder; S TL j This represents the j-th Swin-Transformer Layer in the Residual-Transformer Layer; F D(i,j) This represents the deep feature value extracted after passing through the entire decoder; After the input signal passes through the Transformer-based Decoder, sufficient semantic similarity information has been extracted. To further improve the relevance of these features, a Keepfeat Layer is designed, which mainly consists of three two-dimensional convolutions. This layer enhances the model's ability to integrate deep features, thereby optimizing the overall performance of the network. Specifically, it can be expressed by the following formula: F DR =KF CONV (F D ) (8) The Keepfeat Layer consists of three two-dimensional convolutional neural networks with output channels of 24, 24, and 96 respectively; kernel sizes of 3×3, 1×1, and 3×3 respectively; strides of 2, 1, and 2 respectively; and padding of 1, 0, and 1 respectively. After semantic feature extraction at the decoding end, S LQ Both shallow and deep features are fully extracted, and finally, they are directly added together at the end, as shown below: F=F S +F DR (9) Among them, F S and F DR These are represented as shallow features and deep features, respectively. The resulting semantic signal contains information at multiple levels and has a strong representational ability, laying the foundation for subsequent semantic signal reconstruction work.

7. A speech semantic signal transmission method based on Swing-Transformer according to claim 6, characterized in that... The specific implementation description of step S5 is as follows: in, For the reconstructed speech signal, R γ For signal reconstruction, S D This refers to the signal after semantic decoding in step S4.

8. A speech semantic signal transmission method based on Swing-Transformer according to claim 7, characterized in that... Signal Reconstructor R γ It consists of an Upsample Layer and a Last Layer. The basic unit of the Upsample Layer includes two 2D convolutions before and after, as well as an upsampling operation in between. The two 2D convolutions before and after are used to fix features and increase the generalization ability of the overall network. The upsampling operation in between uses Interpolate, which mainly controls the target size by adjusting the scale factor. The above process is represented as follows: S′=R CONV2 (R I (R CONV2 (F))) (13) Among them, R CONV1 ,R CONV2 ,R I These represent convolution and upsampling operations, respectively.

9. A speech semantic signal transmission method based on Swing-Transformer according to claim 8, characterized in that... The specific structural parameters of the two-dimensional convolution are as follows: R CONV1 and R CONV2 The structure is the same, with 64 output channels, a 3×3 kernel size, a stride of 2, and padding of 1. The Last Layer is composed of a two-dimensional convolutional neural network, which is used to reduce the channel dimension of the decoded signal, thereby achieving the same size as the original input signal. The Last Layer consists of three 2D convolutional neural networks with output channels of 32, 32, and 1 respectively; kernel sizes of 3×3, 1×1, and 3×3 respectively; strides of 2, 1, and 2 respectively; and padding of 1, 0, and 1 respectively. The reconstructed signal is then compared with the original speech signal. The loss function used can be specifically expressed as: Where C, F, and L represent the number of channels, the number of frames, and the frame length of the input speech signal, respectively, and s and Let i, j, k represent the original speech signal and the reconstructed speech signal, i, j, k represent each time-domain sampling point in the sample, and ε is a constant, taken as 0.

001. The purpose of increasing this constant is to keep the loss function numerically stable.

Images