A voice signal receiving, transmitting method and device
By extracting semantic features of speech signals and performing channel-adaptive encoding and decoding, the problems of scarce speech transmission resources and security are solved, and stability and security are improved in low signal-to-noise ratio environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV OF POSTS & TELECOMM
- Filing Date
- 2026-03-20
- Publication Date
- 2026-06-26
AI Technical Summary
With the convergence of AI, 5G/6G and IoT, voice signal transmission faces challenges in reliability and security under conditions of scarce channel resources and low signal-to-noise ratio, especially in densely populated areas and tall buildings that obstruct the view, leading to voice transmission stuttering, distortion and security risks.
By extracting semantic features from speech signals and performing channel-adaptive encoding and decoding, only core features are transmitted. Combined with adaptive signal-to-noise ratio adjustment and forgery detection, resource conservation and security assurance in speech transmission are achieved.
It significantly improves the stability and reliability of voice transmission in low signal-to-noise ratio environments, reduces communication resource consumption, and ensures the security of voice interaction through forgery detection.
Smart Images
Figure CN121907407B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of voice signal transmission technology, and in particular to a voice signal receiving and transmission method and apparatus. Background Technology
[0002] With the convergence of AI, 5G / 6G, and the Internet of Things, voice has become a core interactive medium in key scenarios such as remote financial verification and in-vehicle voice control. However, as voice interactions become more frequent, channel resources are becoming increasingly scarce: in financial scenarios, users often complete verification in densely populated areas such as subways and shopping malls, where signal interference from multiple devices leads to a decrease in the signal-to-noise ratio; in in-vehicle scenarios, when vehicles travel through densely populated urban areas with tall buildings or tunnels, signal obstruction and reflection cause fluctuations in the signal-to-noise ratio. Both of these pose challenges to the efficiency, reliability, and security of voice transmission.
[0003] Traditional voice transmission, based on waveform encoding, has significant drawbacks: firstly, it requires transmitting complete voice waveforms containing a large amount of redundant information, excessively consuming channel resources, which contradicts the current situation of scarce channel resources; secondly, it cannot dynamically adjust transmission strategies according to the signal-to-noise ratio (SNR), making it prone to voice stuttering and distortion in low SNR environments. Specifically, in financial remote verification scenarios, this directly affects the efficiency and accuracy of identity verification; in vehicle voice control scenarios, it may lead to invalid unlocking commands and loss of critical emergency call information, posing security risks to the application. Furthermore, deceptive voice commands can impersonate genuine entities to generate false content, causing serious security problems in critical scenarios. In financial remote verification scenarios, forged voice commands may impersonate users to complete identity verification, thereby committing fraudulent transfers, account theft, and other acts that harm users' property security; in vehicle voice control scenarios, false voice commands may lead to unauthorized vehicle unlocking, tampering with emergency call information, or even interference with vehicle control, threatening the personal and property safety of drivers and passengers. Summary of the Invention
[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a voice signal receiving and transmission method and apparatus that can improve the reliability and security of voice transmission tasks at low signal-to-noise ratios while saving communication resources.
[0005] To achieve the above objectives, the present invention is implemented using the following technical solution:
[0006] In a first aspect, the present invention provides a voice signal receiving method, applied at a signal receiving end, comprising:
[0007] The first fused semantic feature transmitted through the channel by the receiving signal transmitter is used to obtain the second fused semantic feature after being affected by noise interference in the channel.
[0008] The second fused semantic features are subjected to dimensionality recovery, denormalization, and splitting to obtain the first and second transmission semantic features.
[0009] The first transmission semantic feature is processed by the first channel decoder, and the statistical information of the channel noise itself is incorporated into the feature to obtain the first transmission semantic feature after feature processing.
[0010] The second channel decoder is used to adjust the dimensions of the second transmission semantic feature, and the channel dimensions of the first and second transmission semantic features are aligned to obtain the dimension-aligned second transmission semantic feature.
[0011] The first transmission semantic features after feature processing are decoded to obtain the reconstructed audio signal with the best evaluation of speech quality;
[0012] The second transmission semantic features after dimensional alignment are classified to obtain a prediction score, thereby determining the authenticity of the speech.
[0013] Based on the judgment result of speech authenticity and the reconstructed audio signal with optimal speech quality evaluation, the target reconstructed audio signal is selected.
[0014] Further, decoding the first transmission semantic feature after feature processing includes:
[0015] The DeepSC semantic decoder is controlled to decode the first transmitted semantic features after feature processing. The DeepSC semantic decoder includes an SE-ResNet sequence, a final convolutional layer, and an output processing layer.
[0016] The SE-ResNet sequence is used to perform deep reconstruction and information completion on the first transmission semantic features of the input to obtain multi-channel features;
[0017] The final convolutional layer is used to convert multi-channel features into single-channel features;
[0018] The output processing layer is used to restore the single-channel features to the reconstructed audio signal in the original format;
[0019] The classification of the second transmission semantic features after dimensional alignment, and the acquisition of prediction scores, to determine the authenticity of the speech, includes:
[0020] The control classifier classifies the second transport semantic features, and the classifier includes a Dropout layer and a linear layer;
[0021] The Dropout layer is used to suppress overfitting and improve the generalization ability of the model during classifier training.
[0022] The linear layer is used to project the feature dimensions of the second transport semantic features onto the category space;
[0023] The prediction score is obtained by multiplying the feature dimension and feature weight in the category space, and then adding the product to the feature bias term.
[0024] Speech signals with a prediction score greater than or equal to the prediction threshold are judged as real speech, while speech signals with a prediction score less than the prediction threshold are judged as fake speech, thus realizing the determination of the authenticity of speech.
[0025] Furthermore, obtaining the reconstructed audio signal with optimal evaluation of speech quality includes:
[0026] Construct a first detection task model that minimizes the mean squared error loss function. The training process of the first detection task model includes:
[0027] Obtain the training dataset for the audio signal;
[0028] Based on the training dataset of the audio signal, a mean squared error loss is constructed using the audio signal and the reconstructed audio signal;
[0029] Stochastic gradient descent is used, and the parameters are adjusted using the Adam optimizer until the mean squared error loss function is minimized, thus completing the training.
[0030] Furthermore, obtaining the prediction score includes:
[0031] Construct a second detection task model that minimizes the cross-entropy loss function. The training process of the second detection task model includes:
[0032] Obtain the training dataset for the audio signal;
[0033] Based on the training dataset of the audio signal, a cross-entropy loss function is constructed using the predicted score output by the classifier and the built-in label of the audio signal.
[0034] Stochastic gradient descent is used, and the parameters are adjusted using the Adam optimizer until the cross-entropy loss function is minimized, thus completing the training.
[0035] Furthermore, the first channel decoder includes two fully connected layers for feature processing of the first transmission semantic features, a residual connection layer for residual connection, and a convolutional layer for channel convolution.
[0036] The first transmission semantic feature is obtained by passing two fully connected layers for feature processing to obtain the first transmission semantic feature after preliminary processing. The first transmission semantic feature after preliminary processing and the first transmission semantic feature are then residually connected to obtain the first transmission semantic feature after residual connection. The first transmission semantic feature after residual connection is then convolved to obtain the first transmission semantic feature after feature processing.
[0037] The second channel decoder includes two convolutional layers for dimensionality adjustment of the second transport semantic features.
[0038] Furthermore, while performing feature processing on the first transmission semantic features, it also includes:
[0039] Obtain the signal-to-noise ratio of the channel;
[0040] The signal-to-noise ratio is converted to standard deviation, the standard deviation is multiplied by 255, and the result is then converted to an 8-bit binary representation. ;in, , Representing 8-bit binary form Median index Representing 8-bit binary form The first in Bits;
[0041] For 8-bit binary representation Each of them repeat Next, the 8-bit binary representation Expanded into a mask; where, , , array representing the number of repetitions The index for ranking by median value. The value of and The corresponding values are, array representing the number of repetitions The Middle Number of repetitions, Expressed as tolerance;
[0042] The first transmission semantic features, which have undergone preliminary processing, are multiplied by the mask to incorporate statistical information about the noise itself to adaptively adjust the feature representation, thereby achieving targeted countermeasures against the noise.
[0043] Secondly, the present invention provides a voice signal transmission method, applied at a signal transmitting end, comprising:
[0044] Acquire the audio signal to be transmitted;
[0045] The audio signal is subjected to feature extraction to obtain a first semantic feature for voice transmission and a second semantic feature for forgery detection;
[0046] The first semantic feature is processed by the first channel encoder, and the statistical information of the channel noise itself is incorporated into the feature to obtain the first semantic feature after feature processing.
[0047] The second semantic feature is dimensionally adjusted using a second channel encoder to align the channel dimensions of the first and second semantic features, thereby obtaining the dimensionally aligned second semantic feature.
[0048] The first semantic feature after feature processing and the second semantic feature after dimension alignment are spliced and fused, power normalized and dimension adjusted to obtain the first fused semantic feature.
[0049] Further, feature extraction is performed on the audio signal to obtain a first semantic feature for voice transmission and a second semantic feature for forgery detection, including:
[0050] The DeepSC semantic encoder is controlled to extract features from the audio signal to obtain the first semantic feature. The DeepSC semantic encoder includes a preprocessing layer, an initial convolutional layer and an SE-ResNet sequence connected in sequence.
[0051] The preprocessing layer is used to convert the original audio signal into a format suitable for neural network input, thereby obtaining the preprocessed audio signal.
[0052] The initial convolutional layer is used to extract shallow features from the preprocessed audio signal and output shallow features from multiple channels.
[0053] The SE-ResNet sequence is used to perform deep semantic mining and enhancement on shallow features to obtain the first semantic features;
[0054] The AASIST semantic encoder is controlled to extract features from the audio signal to obtain second semantic features. The AASIST semantic encoder includes a speech feature extraction module, an encoding module, and a graph attention feature enhancement module.
[0055] The speech feature extraction module is used to convert the original audio signal into preliminary extracted features with discriminative power.
[0056] The encoding module is used to mine and compress the initially extracted features, remove redundant information, enhance distinguishing features, and output the encoded enhanced features.
[0057] The graph attention feature enhancement module is used to construct a graph structure from the encoded enhanced features, capture the dependencies between feature nodes in the graph structure through the attention mechanism, obtain the modeled graph structure features, and then reduce the dimensions of the modeled graph structure features through graph pooling to obtain the second semantic features.
[0058] Furthermore, the first channel encoder includes two fully connected layers for feature processing of the first semantic features, a residual connection layer for residual connection, and a convolutional layer for channel convolution.
[0059] The first semantic feature is obtained by passing through two fully connected layers for feature processing to obtain the first semantic feature after preliminary processing. The first semantic feature after preliminary processing and the first semantic feature are then connected by residual connection to obtain the first semantic feature after residual connection. The first semantic feature after residual connection is then processed by convolution to obtain the first semantic feature after feature processing.
[0060] The second channel encoder includes two convolutional layers for dimensionality adjustment of the second semantic features.
[0061] Thirdly, the present invention provides a voice signal transmission and reception apparatus, comprising:
[0062] A voice signal transmission and reception device, characterized in that it includes a channel transmitter and a channel receiver;
[0063] The channel transmitter includes:
[0064] The signal acquisition module is used to acquire the audio signal to be transmitted.
[0065] The feature extraction module is used to extract features from the audio signal to obtain a first semantic feature for voice transmission and a second semantic feature for forgery detection.
[0066] The first feature processing module is used to perform feature processing on the first semantic feature using the first channel encoder, and to incorporate the statistical information of the channel noise itself into the feature to obtain the first semantic feature after feature processing.
[0067] The first dimension adjustment module is used to adjust the dimension of the second semantic feature using the second channel encoder, align the channel dimensions of the first semantic feature and the second semantic feature, and obtain the dimension-aligned second semantic feature.
[0068] The feature fusion module is used to concatenate and fuse the first semantic feature after feature processing and the second semantic feature after dimension alignment, perform power normalization and dimension adjustment, and obtain the first fused semantic feature.
[0069] The channel receiver includes:
[0070] The receiving module is used to receive the first fused semantic features transmitted by the signal transmitter through the channel, and to obtain the second fused semantic features after being affected by noise interference in the channel.
[0071] The feature splitting module is used to perform dimensionality recovery, denormalization and splitting processing on the second fused semantic features to obtain the first transmission semantic features and the second transmission semantic features.
[0072] The second feature processing module is used to perform feature processing on the first transmission semantic feature using the first channel decoder, and to incorporate the statistical information of the channel noise itself into the feature to obtain the first transmission semantic feature after feature processing.
[0073] The second dimension adjustment module is used to adjust the dimensions of the second transmission semantic feature using the second channel decoder, align the channel dimensions of the first transmission semantic feature and the second transmission semantic feature, and obtain the dimension-aligned second transmission semantic feature.
[0074] The feature decoding module is used to decode the first transmission semantic features after feature processing to obtain the reconstructed audio signal with the best evaluation of speech quality.
[0075] The feature classification module is used to classify the second transmission semantic features after dimensional alignment, obtain a prediction score, and determine the authenticity of the speech.
[0076] The feature output module is used to filter out the target reconstructed audio signal based on the judgment result of speech authenticity and the reconstructed audio signal with the best evaluation of speech quality.
[0077] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:
[0078] This invention proposes a method for voice signal transmission and reception. By pre-extracting the semantic features required for voice transmission and those required for forgery detection at the local transmitting end, only these two types of core features are transmitted, significantly reducing the amount of data transmitted, thereby shortening transmission time and saving communication resources. Simultaneously, the signal-to-noise ratio adaptive encoding and decoding method employed further reduces channel interference within the semantic communication framework, significantly improving the accuracy of voice transmission in low signal-to-noise ratio environments. Furthermore, because the semantic features used for forgery detection are transmitted synchronously, the receiving end can directly determine the authenticity of the received voice based on these features, effectively ensuring the security of voice transmission.
[0079] By extracting core semantic features of speech and discarding redundant complete speech waveform transmission, the amount of data can be significantly compressed, effectively reducing channel resource consumption. Simultaneously, with an adaptive signal-to-noise ratio (SNR) codec, it can dynamically adapt to channel fluctuations, significantly improving the stability and reliability of speech transmission in low SNR environments. Furthermore, by simultaneously extracting semantic features for forgery detection, a binary classification framework for speech authenticity can be built based on these extracted features, enabling accurate identification of deceptive speech and providing security for voice interaction. Attached Figure Description
[0080] Figure 1 This is a flowchart of a voice signal receiving method provided in the embodiment;
[0081] Figure 2 This is a flowchart of a voice signal transmission method provided in the embodiment;
[0082] Figure 3 This is a schematic diagram of the overall system framework provided in an embodiment of the present invention;
[0083] Figure 4 This is a schematic diagram of the signal-to-noise ratio adaptive coding process provided in an embodiment of the present invention. Detailed Implementation
[0084] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments and specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations thereof. In the absence of conflict, the embodiments and technical features in the embodiments can be combined with each other.
[0085] In this invention, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B together, or B alone. Additionally, in this invention, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.
[0086] Example 1:
[0087] Figure 1 This is a flowchart of the voice signal receiving method in Embodiment 1 of the present invention. This flowchart merely illustrates the logical order of the method described in this embodiment. Without conflict, in other possible embodiments of the present invention, different methods may be used. Figure 1 Complete the steps shown or described in the order indicated.
[0088] The voice signal receiving method provided in this embodiment can be applied to a terminal and can be executed by a mechanical equipment fault identification device. This device can be implemented in software and / or hardware and can be integrated into the terminal, such as any smartphone, tablet, or computer device with communication capabilities. The method in this embodiment is applied to a signal receiving end and specifically includes the following steps:
[0089] Step 1: Receive the first fused semantic features transmitted through the channel from the receiving signal transmitter, and obtain the second fused semantic features after interference from channel noise. The expression is as follows:
[0090] ,
[0091] in, For the second fusion semantic feature, For channel parameters, As the first semantic feature, Indicates Gaussian noise with variance.
[0092] Step 2: Perform dimensionality recovery, denormalization, and splitting on the second fused semantic features to obtain the first and second transmission semantic features;
[0093] Step 3: Use the first channel decoder to perform feature processing on the first transmission semantic features, incorporating the statistical information of the channel noise itself into the features, to obtain the feature-processed first transmission semantic features, including:
[0094] The first channel decoder includes two fully connected layers for feature processing of the first transmission semantic features, a residual connection layer for residual connection, and a convolutional layer for channel convolution.
[0095] Specifically, the first transmission semantic feature is obtained by passing two fully connected layers for feature processing to obtain the first transmission semantic feature after preliminary processing. The first transmission semantic feature after preliminary processing and the first transmission semantic feature are then residually connected to obtain the first transmission semantic feature after residual connection. The first transmission semantic feature after residual connection is then convolved to obtain the first transmission semantic feature after feature processing.
[0096] Step 4: Use the second channel decoder to adjust the dimensions of the second transmission semantic feature, align the channel dimensions of the first and second transmission semantic features, and obtain the dimension-aligned second transmission semantic feature.
[0097] The second channel decoder includes two convolutional layers for dimensionality adjustment of the second transport semantic features.
[0098] Step 5: Decode the first transmission semantic features after feature processing to obtain the reconstructed audio signal with the best evaluation of speech quality;
[0099] Decoding the first transmission semantic feature after feature processing includes:
[0100] The DeepSC semantic decoder is controlled to decode the first transmitted semantic features after feature processing. The DeepSC semantic decoder includes an SE-ResNet sequence, a final convolutional layer, and an output processing layer.
[0101] The SE-ResNet sequence is used to perform deep reconstruction and information completion on the first transmission semantic features of the input to obtain multi-channel features;
[0102] The final convolutional layer is used to convert multi-channel features into single-channel features;
[0103] The output processing layer is used to restore the single-channel features to the reconstructed audio signal in the original format;
[0104] The classification of the second transmission semantic features after dimensional alignment, and the acquisition of prediction scores, to determine the authenticity of the speech, includes:
[0105] The control classifier classifies the second transport semantic features, and the classifier includes a Dropout layer and a linear layer;
[0106] The Dropout layer is used to suppress overfitting and improve the generalization ability of the model during classifier training. The probability of Dropout can be set to 0.5 to prevent overfitting.
[0107] The linear layer is used to project the feature dimensions of the second transport semantic features onto the category space;
[0108] The prediction score is obtained by multiplying the feature dimension and feature weight in the category space, and then adding the product to the feature bias term.
[0109] The expression is as follows:
[0110] ,
[0111] ,
[0112] in, For the second fusion semantic feature, It is a DeepSC semantic decoder. For classifiers, For the first channel decoder, For the second channel decoder based on convolutional layers, To reconstruct the audio signal, This is to falsify the detection prediction results, i.e., the prediction score.
[0113] Speech signals with a prediction score greater than or equal to the prediction threshold are judged as real speech, while speech signals with a prediction score less than the prediction threshold are judged as fake speech, thus realizing the determination of the authenticity of speech.
[0114] The step of obtaining the reconstructed audio signal with optimal speech quality includes:
[0115] Construct a first detection task model that minimizes the mean squared error loss function. The training process of the first detection task model includes:
[0116] Obtain the training dataset for the audio signal;
[0117] Based on the training dataset of the audio signal, a mean squared error loss is constructed using the audio signal and the reconstructed audio signal;
[0118] Stochastic gradient descent is used, and the parameters are adjusted using the Adam optimizer until the mean squared error loss function is minimized, thus completing the training.
[0119] The process of obtaining the prediction score includes:
[0120] Construct a second detection task model that minimizes the cross-entropy loss function. The training process of the second detection task model includes:
[0121] Obtain the training dataset for the audio signal;
[0122] Based on the training dataset of the audio signal, a cross-entropy loss function is constructed using the predicted score output by the classifier and the built-in label of the audio signal.
[0123] Stochastic gradient descent is used, and the parameters are adjusted using the Adam optimizer until the cross-entropy loss function is minimized, thus completing the training.
[0124] In this embodiment, based on the training dataset of the audio signal, a mean squared error loss can be constructed using the audio signal and the reconstructed audio signal. A cross-entropy loss function can be constructed using the predicted score output by the classifier and the inherent label of the audio signal. A total loss function can be constructed based on the mean squared error loss and the cross-entropy loss. The total loss function is equivalent to the cross-entropy loss function of the second detection task model and the mean squared error loss function of the first detection task model, as expressed below:
[0125] ,
[0126] ,
[0127] ,
[0128] in, Represents the total loss function. Represents cross-entropy loss, The weights representing the cross-entropy loss, This represents the mean squared error loss. The weights representing the mean squared error loss are... Indicates the sample index within the batch. Indicates the number of samples in the batch. Represented as the first The true label of each sample Represented as the first The predicted score for each sample. The length of the audio. Represented as the index of the audio sample point, Represented as the first The sample at the th The audio signal at each sampling point Represented as the first The sample at the th The reconstructed audio signal from each sampling point.
[0129] Step 6: Classify the second transmission semantic features after dimensional alignment, obtain the prediction score, and determine the authenticity of the speech.
[0130] Step 7: Based on the judgment result of speech authenticity and the reconstructed audio signal with the best speech quality, select the target reconstructed audio signal.
[0131] The voice signal receiving method provided in this embodiment, while performing feature processing on the first transmission semantic features, also includes:
[0132] Obtain the signal-to-noise ratio of the channel;
[0133] The signal-to-noise ratio is converted to standard deviation, the standard deviation is multiplied by 255, and the result is then converted to an 8-bit binary representation. ;in, , Representing 8-bit binary form Median index Representing 8-bit binary form The first in Bits;
[0134] For 8-bit binary representation Each of them repeat Next, the 8-bit binary representation Expanded into a mask; where, , , array representing the number of repetitions The index for ranking by median value. The value of and The corresponding values are, array representing the number of repetitions The Middle Number of repetitions, Expressed as tolerance;
[0135] The first transmission semantic features, which have undergone preliminary processing, are multiplied by the mask to incorporate statistical information about the noise itself to adaptively adjust the feature representation, thereby achieving targeted countermeasures against the noise.
[0136] Example 2:
[0137] Figure 2 This is a flowchart of the voice signal transmission method in Embodiment 2 of the present invention. This flowchart only illustrates the logical order of the method described in this embodiment. Without conflict, in other possible embodiments of the present invention, different methods may be used. Figure 2 Complete the steps shown or described in the order indicated.
[0138] The voice signal transmission method provided in this embodiment can be applied to a terminal and can be executed by a mechanical equipment fault identification device. This device can be implemented in software and / or hardware and can be integrated into the terminal, such as any smartphone, tablet, or computer device with communication capabilities. The method in this embodiment is applied to a signal transmitting end and specifically includes the following steps:
[0139] Step 1: Obtain the raw audio signal, uniformly fill and trim the raw audio to about 4 seconds, and obtain the audio signal to be transmitted using the PyTorch toolkit;
[0140] Step 2: Perform feature extraction on the audio signal to obtain a first semantic feature for voice transmission and a second semantic feature for forgery detection, including:
[0141] The DeepSC semantic encoder is controlled to extract features from the audio signal to obtain the first semantic feature. The DeepSC semantic encoder includes a preprocessing layer, an initial convolutional layer and an SE-ResNet sequence connected in sequence.
[0142] The preprocessing layer is used to convert the original audio signal into a format suitable for neural network input, thereby obtaining the preprocessed audio signal.
[0143] The initial convolutional layer is used to extract shallow features from the preprocessed audio signal and output shallow features from multiple channels.
[0144] The SE-ResNet sequence is used to perform deep semantic mining and enhancement on shallow features to obtain the first semantic features;
[0145] The AASIST semantic encoder is controlled to extract features from the audio signal to obtain second semantic features. The AASIST semantic encoder includes a speech feature extraction module, an encoding module, and a graph attention feature enhancement module.
[0146] The speech feature extraction module is used to convert the original audio signal into preliminary extracted features with discriminative power.
[0147] The encoding module is used to mine and compress the initially extracted features, remove redundant information, enhance distinguishing features, and output the encoded enhanced features.
[0148] The graph attention feature enhancement module is used to construct a graph structure from the encoded enhanced features, capture the dependencies between feature nodes in the graph structure through the attention mechanism, obtain the modeled graph structure features, and then reduce the dimensions of the modeled graph structure features through graph pooling to obtain the second semantic features.
[0149] Step 3: Use the first channel encoder to perform feature processing on the first semantic feature, and incorporate the statistical information of the channel noise itself into the feature to obtain the first semantic feature after feature processing.
[0150] The first channel encoder includes two fully connected layers for feature processing of the first semantic features, a residual connection layer for residual connection, and a convolutional layer for channel convolution.
[0151] The first semantic feature is obtained by passing through two fully connected layers for feature processing to obtain the first semantic feature after preliminary processing. The first semantic feature after preliminary processing and the first semantic feature are then connected by residual connection to obtain the first semantic feature after residual connection. The first semantic feature after residual connection is then processed by convolution to obtain the first semantic feature after feature processing.
[0152] Step 4: Use the second channel encoder to adjust the dimensions of the second semantic feature, align the channel dimensions of the first and second semantic features, and obtain the dimension-aligned second semantic feature;
[0153] The second channel encoder includes two convolutional layers for dimensional adjustment of the second semantic features.
[0154] Step 5: Perform splicing and fusion, power normalization and dimension adjustment on the first semantic feature after feature processing and the second semantic feature after dimension alignment to obtain the first fused semantic feature.
[0155] The data dimension of the audio signal is B*64000, where B is the batch size.
[0156] The DeepSC semantic encoder extracts the first semantic feature for semantic transmission from the audio signal, with a dimension of B*32*160*400; the AASIST semantic encoder extracts the second semantic feature for forgery detection from the audio signal, with a dimension of B*1*160.
[0157] As attached Figure 4 As shown, the first semantic feature is processed by first passing it through a first linear layer to adjust the dimension to B*32*160*768, then passing it through another linear layer to adjust the dimension to B*32*160*400, then adding the input to the output feature of the second linear layer through a residual connection, and finally passing it through a convolutional layer to adjust the channel dimension to 8, that is, the dimension is B*8*160*400, which is the dimension of the first semantic feature after feature processing;
[0158] After two convolutional layers, the dimensions are adjusted to B*8*160.
[0159] The first semantic feature after feature processing and the second semantic feature after dimension alignment are concatenated and fused, power normalized, and dimension adjusted. Then, semantic feature dimension adjustment and normalization are performed to obtain the first fused semantic feature with a dimension of B*8*160*401. The overall expression is as follows:
[0160] ,
[0161] in, It is an audio signal. For DeepSC semantic encoder, For AASIST semantic encoder, As the first semantic feature, This is the first semantic feature after feature processing. For the first channel encoder, This is a second channel encoder based on convolutional layers. As a second semantic feature, The second semantic feature after dimension alignment. This is the first fused semantic feature.
[0162] The semantic feature dimension adjustment and power normalization are performed in the channel. Power normalization is performed first in the channel, then the dimension is multiplied by 2, and then one is alternately selected as the real part and the other as the imaginary part for transmission. After the transmission is completed, the dimension is restored at the other end and then power inverse normalization is performed.
[0163] In the speech signal transmission method disclosed in this embodiment, when performing feature processing on the first semantic feature, it may further include:
[0164] Obtain the signal-to-noise ratio of the channel;
[0165] The signal-to-noise ratio is converted to standard deviation, the standard deviation is multiplied by 255, and the result is then converted to an 8-bit binary representation. ;in, , Representing 8-bit binary form Median index Representing 8-bit binary form The first in Bits;
[0166] Using a non-uniform repetition strategy for the 8-bit binary representation Each of them repeat Next, the 8-bit binary representation Expanded into a mask; where, , , Array representing the number of repetitions The index for ranking by median value. The value of and The corresponding values are, Array representing the number of repetitions The Middle Number of repetitions, Expressed as tolerance;
[0167] The intermediate features obtained after processing by the first fully connected layer are multiplied by the mask to incorporate the statistical information of the noise itself to adaptively adjust the feature representation, thereby achieving targeted countermeasures against the noise.
[0168] In this embodiment, we applied masks to the last two dimensions, and the sizes of the last two dimensions are different. Therefore, we set the initial values and common differences of the arithmetic sequence for these two dimensions respectively. Specifically, for the last dimension, we keep the values of the first 256 points unchanged, and the arithmetic sequence parameters corresponding to this dimension are... and The values are 8 and 16 respectively; for the second-to-last dimension, the values of the first 32 points remain unchanged, and their corresponding arithmetic sequence parameters... and They are 2 and 4 respectively.
[0169] Based on the aforementioned Embodiment 1 and Embodiment 2, as follows Figure 3As shown, the DeepSC semantic encoder, DeepSC semantic decoder, AASIST semantic encoder, first channel encoder, first channel decoder, second channel encoder, second channel decoder, and classifier are implemented through a deep neural network. The parameters of the AASIST semantic encoder are frozen. The entire system is trained end-to-end with joint channel training, using cross-entropy loss and mean squared error loss as the loss function. Stochastic gradient descent is employed, and the parameters are adjusted using the Adam optimizer until the total loss function is minimized, thus completing the training.
[0170] The channel type is a Gaussian channel, and the signal-to-noise ratio of the Gaussian channel is in the range of 0~20dB.
[0171] After training on the training set, a validation set can be run: First, set five different signal-to-noise ratio (SNR) levels: 0, 5, 10, 15, and 20. Then, for each SNR condition, completely traverse the corresponding validation set data: read the validation set data in batches and input them into the model in batches. Calculate the model's prediction results for each batch of data, and evaluate the validation loss value of that batch based on the mean squared error loss and cross-entropy loss functions. After traversing all validation set data for a single SNR, calculate and record the single SNR validation loss corresponding to that SNR. After all SNR validation set traversals and single loss calculations are completed, take the arithmetic mean of all single SNR validation losses to obtain the average validation loss for the batch.
[0172] Save the best model: During training, continuously monitor the loss value on the validation set and save the model with the lowest loss on the validation set as the best model.
[0173] Run the test set: After saving the best model, evaluate it on the test set and calculate the performance metrics EER and PESQ. By analyzing the performance of these metrics, assess the model's accuracy and generalization ability, thereby confirming the model's effectiveness in real-world application scenarios.
[0174] The following describes a signal-to-noise ratio adaptive voice transmission and spoofing detection method based on semantic communication, using specific embodiments:
[0175] The sampling rate of the original speech signal used for training and testing is 16kHz. The traditional method is to transmit the speech signal first and then perform forgery detection and recognition.
[0176] In traditional schemes, the test spectrum is encoded using 8-bit and 16-bit Pulse-Code-Modulation (PCM), the modulation method is 64-Quadrature-Amplitude-Modulation (64QAM), the channel encoding and decoding method is polar code, and the channel type is the same as that of this invention.
[0177] To clearly demonstrate the performance differences between this invention and traditional voice transmission schemes and related comparative schemes under different channel signal-to-noise ratio (SNR) conditions, a comparative analysis is conducted from two dimensions: voice transmission quality and forgery detection accuracy. The specific calculation methods are as follows:
[0178] EER, or Equal-Error-Rate, is calculated by first determining the false acceptance rate and false rejection rate based on a preset threshold, then adjusting the decision threshold, and finding the result when the two values are equal. This result is the EER and is used to evaluate the performance of forgery detection.
[0179] PESQ, or Perceptual Evaluation of Speech Quality, is a metric for evaluating speech quality. It first aligns the original speech with the distorted speech, performs frequency domain filtering, then calculates the perceptual signal distortion, and finally outputs a score ranging from -0.5 to 4.5. A higher score indicates better quality speech transmission or reconstruction. The specific formula is as follows:
[0180] ,
[0181] in, For the index of the speech frame, For the first Frame of original audio For the first Frame-based speech reconstruction For the first Distortion of frame-by-frame audio Total number of frames This represents the speech quality score.
[0182] Voice transmission quality was evaluated using the PESQ index. The schemes included in the comparison were the traditional 8PCM scheme, the traditional 16PCM scheme, and the semantic communication comparison scheme (DeepSC-S). The PESQ index data of each scheme under different SNR are shown in Table 1.
[0183] Table 1
[0184]
[0185] As shown in Table 1, the present invention has significant advantages over traditional voice transmission schemes in terms of voice transmission quality, especially in low signal-to-noise ratio (SNR) scenarios: when SNR is 0, the PESQ value of the present invention is much higher than that of the traditional schemes 8PCM, 16PCM, and DeepSC-S; when SNR is increased to 5, the PESQ value of the present invention is still significantly higher than other comparative schemes; only in high SNR scenarios is the PESQ value of the present invention slightly lower than that of the traditional scheme 16PCM, but overall it still maintains an excellent level of voice transmission quality.
[0186] The accuracy of counterfeit detection was evaluated using the EER index. The schemes included in the comparison were the traditional scheme 8PCM, the traditional scheme 16PCM, and the counterfeit detection comparison scheme (AASIST). The EER index data of each scheme under different SNR are shown in Table 2.
[0187] Table 2
[0188]
[0189] As can be seen from the data in Table 2, this invention exhibits significant advantages in forgery detection performance: compared with traditional voice transmission schemes, the EER index of this invention is significantly reduced. In low signal-to-noise ratio scenarios, the EER of this invention is much lower than that of the traditional scheme (8 PCM and 16 PCM); as SNR increases, the EER of this invention remains at a low level and is still significantly better than the traditional scheme; at the same time, compared with the forgery detection comparison scheme AASIST, the EER index of Mask is still smaller, further demonstrating the performance advantage of this invention in forgery detection.
[0190] Example 3:
[0191] Embodiment 2 of the present invention provides a voice signal transmission and reception device, comprising:
[0192] Sending system and receiving system;
[0193] The transmitting end system includes:
[0194] The signal acquisition module is used to acquire audio signals;
[0195] The feature extraction module is used to extract features from the audio signal to obtain a first semantic feature for voice transmission and a second semantic feature for forgery detection.
[0196] The first feature processing module is used to perform feature processing on the first semantic feature and the second semantic feature to obtain the first semantic feature and the second semantic feature after feature processing.
[0197] The feature fusion module is used to concatenate and fuse the first semantic feature and the second semantic feature after feature processing, normalize the power and adjust the dimension to obtain the first fused semantic feature.
[0198] The receiving system includes:
[0199] The receiving module is used to receive the first fused semantic features transmitted through the channel and obtain the second fused semantic features;
[0200] The feature splitting module is used to perform dimensionality recovery, denormalization and splitting processing on the second fused semantic features to obtain the first transmission semantic features and the second transmission semantic features.
[0201] The second feature processing module is used to perform feature processing on the first transmission semantic feature and the second transmission semantic feature to obtain the first transmission semantic feature and the second transmission semantic feature after feature processing.
[0202] The feature decoding module is used to decode the first transmission semantic features after feature processing to obtain the reconstructed audio signal with the best evaluation of speech quality.
[0203] The feature classification module is used to classify the second transmission semantic features after dimensional alignment, obtain a prediction score, and determine the authenticity of the speech.
[0204] The voice transmission and forgery detection device provided in Embodiment 3 of the present invention can execute the method provided in Embodiment 1 or Embodiment 2 of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.
[0205] Example 4:
[0206] Embodiment 4 of the present invention also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the method described in Embodiment 1 or Embodiment 2, and has the corresponding functional modules and beneficial effects of the method.
[0207] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0208] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0209] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0210] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0211] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A method for receiving voice signals, applied at a signal receiving end, characterized in that, include: The receiving signal transmitter obtains the first fused semantic features transmitted through the channel, and then obtains the second fused semantic features after being affected by noise interference within the channel. The second fused semantic features are subjected to dimensionality recovery, denormalization, and splitting to obtain the first and second transmission semantic features. The first transmission semantic feature is processed by the first channel decoder, and the statistical information of the channel noise itself is incorporated into the feature to obtain the first transmission semantic feature after feature processing. The second channel decoder is used to adjust the dimensions of the second transmission semantic feature, and the channel dimensions of the first and second transmission semantic features are aligned to obtain the dimension-aligned second transmission semantic feature. The first transmission semantic features after feature processing are decoded to obtain the reconstructed audio signal with the best evaluation of speech quality; The second transmission semantic features after dimensional alignment are classified to obtain a prediction score, thereby determining the authenticity of the speech. Based on the judgment result of speech authenticity and the reconstructed audio signal with optimal speech quality, the target reconstructed audio signal is selected. Decoding the first transmission semantic feature after feature processing includes: The DeepSC semantic decoder is controlled to decode the first transmitted semantic features after feature processing. The DeepSC semantic decoder includes an SE-ResNet sequence, a final convolutional layer, and an output processing layer. The SE-ResNet sequence is used to perform deep reconstruction and information completion on the first transmission semantic features of the input to obtain multi-channel features; The final convolutional layer is used to convert multi-channel features into single-channel features; The output processing layer is used to restore the single-channel features to the reconstructed audio signal in the original format; The classification of the second transmission semantic features after dimensional alignment, and the acquisition of prediction scores, to determine the authenticity of the speech, includes: The control classifier classifies the second transport semantic features, and the classifier includes a Dropout layer and a linear layer; The Dropout layer is used to suppress overfitting and improve the generalization ability of the model during classifier training. The linear layer is used to project the feature dimensions of the second transport semantic features onto the category space; The prediction score is obtained by multiplying the feature dimension and feature weight in the category space, and then adding the product to the feature bias term. Speech signals with a prediction score greater than or equal to the prediction threshold are judged as real speech, while speech signals with a prediction score less than the prediction threshold are judged as fake speech, thus realizing the determination of the authenticity of speech.
2. The voice signal receiving method according to claim 1, characterized in that, The process of obtaining the reconstructed audio signal with optimal speech quality includes: Construct a first detection task model that minimizes the mean squared error loss function. The training process of the first detection task model includes: Obtain the training dataset for the audio signal; Based on the training dataset of the audio signal, a mean squared error loss is constructed using the audio signal and the reconstructed audio signal; Stochastic gradient descent is used, and the parameters are adjusted using the Adam optimizer until the mean squared error loss function is minimized, thus completing the training.
3. The voice signal receiving method according to claim 1, characterized in that, The process of obtaining the prediction score includes: Construct a second detection task model that minimizes the cross-entropy loss function. The training process of the second detection task model includes: Obtain the training dataset for the audio signal; Based on the training dataset of the audio signal, a cross-entropy loss function is constructed using the predicted score output by the classifier and the built-in label of the audio signal. Stochastic gradient descent is used, and the parameters are adjusted using the Adam optimizer until the cross-entropy loss function is minimized, thus completing the training.
4. The voice signal receiving method according to claim 1, characterized in that, The first channel decoder includes two fully connected layers for feature processing of the first transmission semantic features, a residual connection layer for residual connection, and a convolutional layer for channel convolution. The first transmission semantic feature is obtained by passing two fully connected layers for feature processing to obtain the first transmission semantic feature after preliminary processing. The first transmission semantic feature after preliminary processing and the first transmission semantic feature are then residually connected to obtain the first transmission semantic feature after residual connection. The first transmission semantic feature after residual connection is then convolved to obtain the first transmission semantic feature after feature processing. The second channel decoder includes two convolutional layers for dimensionality adjustment of the second transport semantic features.
5. The voice signal receiving method according to claim 4, characterized in that, In addition to feature processing of the first transmission semantic features, the process also includes: Obtain the signal-to-noise ratio of the channel; The signal-to-noise ratio is converted to standard deviation, the standard deviation is multiplied by 255, and the result is then converted to an 8-bit binary representation. ;in, , Representing 8-bit binary form Median index Representing 8-bit binary form The first in Bits; For 8-bit binary representation Each of them repeat Next, the 8-bit binary representation Expanded into a mask; where, , , array representing the number of repetitions The index for ranking by median value. The value of and The corresponding values are, array representing the number of repetitions The Middle Number of repetitions, Expressed as tolerance; The first transmission semantic features, which have undergone preliminary processing, are multiplied by the mask to incorporate statistical information about the noise itself to adaptively adjust the feature representation, thereby achieving targeted countermeasures against the noise.
6. A voice signal transmission method, applied at a signal transmitting end, characterized in that, include: Acquire the audio signal to be transmitted; The audio signal is subjected to feature extraction to obtain a first semantic feature for voice transmission and a second semantic feature for forgery detection; The first semantic feature is processed by the first channel encoder, and the statistical information of the channel noise itself is incorporated into the feature to obtain the first semantic feature after feature processing. The second semantic feature is dimensionally adjusted using a second channel encoder to align the channel dimensions of the first and second semantic features, thereby obtaining the dimensionally aligned second semantic feature. The first semantic feature after feature processing and the second semantic feature after dimension alignment are spliced and fused, power normalized and dimension adjusted to obtain the first fused semantic feature; Feature extraction is performed on the audio signal to obtain a first semantic feature for voice transmission and a second semantic feature for forgery detection, including: The DeepSC semantic encoder is controlled to extract features from the audio signal to obtain the first semantic feature. The DeepSC semantic encoder includes a preprocessing layer, an initial convolutional layer and an SE-ResNet sequence connected in sequence. The preprocessing layer is used to convert the original audio signal into a format suitable for neural network input, thereby obtaining the preprocessed audio signal. The initial convolutional layer is used to extract shallow features from the preprocessed audio signal and output shallow features from multiple channels. The SE-ResNet sequence is used to perform deep semantic mining and enhancement on shallow features to obtain the first semantic features; The AASIST semantic encoder is controlled to extract features from the audio signal to obtain second semantic features. The AASIST semantic encoder includes a speech feature extraction module, an encoding module, and a graph attention feature enhancement module. The speech feature extraction module is used to convert the original audio signal into preliminary extracted features with discriminative power. The encoding module is used to mine and compress the initially extracted features, remove redundant information, enhance distinguishing features, and output the encoded enhanced features. The graph attention feature enhancement module is used to construct a graph structure from the encoded enhanced features, capture the dependencies between feature nodes in the graph structure through the attention mechanism, obtain the modeled graph structure features, and then reduce the dimensions of the modeled graph structure features through graph pooling to obtain the second semantic features.
7. The voice signal transmission method according to claim 6, characterized in that, The first channel encoder includes two fully connected layers for feature processing of the first semantic features, a residual connection layer for residual connection, and a convolutional layer for channel convolution. The first semantic feature is obtained by passing through two fully connected layers for feature processing to obtain the first semantic feature after preliminary processing. The first semantic feature after preliminary processing and the first semantic feature are then connected by residual connection to obtain the first semantic feature after residual connection. The first semantic feature after residual connection is then processed by convolution to obtain the first semantic feature after feature processing. The second channel encoder includes two convolutional layers for dimensionality adjustment of the second semantic features.
8. A voice signal transmission and reception device, characterized in that, The method for performing the voice signal receiving method according to any one of claims 1-5 or the voice signal transmission method according to any one of claims 6-7 includes a channel transmitting end and a channel receiving end; The channel transmitter includes: The signal acquisition module is used to acquire the audio signal to be transmitted. The feature extraction module is used to extract features from the audio signal to obtain a first semantic feature for voice transmission and a second semantic feature for forgery detection. The first feature processing module is used to perform feature processing on the first semantic feature using the first channel encoder, and to incorporate the statistical information of the channel noise itself into the feature to obtain the first semantic feature after feature processing. The first dimension adjustment module is used to adjust the dimension of the second semantic feature using the second channel encoder, align the channel dimensions of the first semantic feature and the second semantic feature, and obtain the dimension-aligned second semantic feature. The feature fusion module is used to concatenate and fuse the first semantic feature after feature processing and the second semantic feature after dimension alignment, perform power normalization and dimension adjustment, and obtain the first fused semantic feature. The channel receiver includes: The receiving module is used to receive the first fused semantic features transmitted by the signal transmitter through the channel, and to obtain the second fused semantic features after being affected by noise interference in the channel. The feature splitting module is used to perform dimensionality recovery, denormalization and splitting processing on the second fused semantic features to obtain the first transmission semantic features and the second transmission semantic features. The second feature processing module is used to perform feature processing on the first transmission semantic feature using the first channel decoder, and to incorporate the statistical information of the channel noise itself into the feature to obtain the first transmission semantic feature after feature processing. The second dimension adjustment module is used to adjust the dimensions of the second transmission semantic feature using the second channel decoder, align the channel dimensions of the first transmission semantic feature and the second transmission semantic feature, and obtain the dimension-aligned second transmission semantic feature. The feature decoding module is used to decode the first transmission semantic features after feature processing to obtain the reconstructed audio signal with the best evaluation of speech quality. The feature classification module is used to classify the second transmission semantic features after dimensional alignment, obtain a prediction score, and determine the authenticity of the speech. The feature output module is used to filter out the target reconstructed audio signal based on the judgment result of speech authenticity and the reconstructed audio signal with the best evaluation of speech quality.