Method and system for voice communication based on importance ranking protection and mask recovery

Abstract

The application discloses a speech communication method and system based on importance grading protection and mask recovery. In the sending end, first, a multi-layer residual vector quantization speech encoder is used to encode a speech signal to be transmitted into discrete speech tokens, then importance evaluation is carried out, and according to a set importance level threshold, protection levels are assigned to the discrete speech tokens. Different unequal error protection parameters are configured for the discrete speech tokens of different protection levels, and are transmitted to the receiving end. For the discrete speech tokens that have errors, check failures, packet loss or erasure in the transmission process, the corresponding positions are set as preset mask identifiers, and in the receiving end, the original tokens corresponding to the mask positions are predicted by using the time context and the residual quantization inter-layer context of the undamaged discrete speech tokens. Finally, according to the current channel state information, the number of residual quantization layers participating in speech decoding is determined, and the speech signal is reconstructed. The method reduces the loss rate of high importance tokens and improves the reconstructed speech quality.

Description

Technical Field

[0001] This invention belongs to the field of communication technology, and relates to voice signal processing and communication transmission, specifically to a voice communication method and system based on importance-level protection and mask recovery. Background Technology

[0002] Traditional digital voice communication typically involves sampling, quantizing, and compressing the voice waveform, then transmitting the encoded bits to the receiver via channel coding, modulation, and a wireless channel. This type of method usually prioritizes waveform fidelity or bit error rate as its primary design objective, employing the same or pre-fixed protection methods for different voice information units.

[0003] With the development of deep learning speech coding and semantic communication technologies, existing techniques can utilize neural networks to map speech signals into continuous semantic features or discrete semantic symbols, and then reconstruct the speech at the receiving end using neural networks. For example, multilayer residual vector quantization encoders can quantize speech signals layer by layer into multiple discrete codeword sequences. The contributions of discrete speech tokens at different quantization layers and at different time positions to the final speech reconstruction result vary. Among them, low-level quantization codewords are usually used to describe the basic content and main acoustic structure of speech, while high-level quantization codewords are used to progressively compensate for quantization residuals and provide timbre, prosody, and detail information.

[0004] However, existing discrete speech transmission methods typically employ uniform channel coding, uniform modulation schemes, or fixed quantization layer truncation strategies. These unified protection methods fail to consider the varying importance of different discrete speech tokens, potentially causing tokens carrying critical speech information to have the same error probability as minor detail tokens. When critical tokens are erroneous or lost, the intelligibility and reconstruction quality of the received speech will significantly degrade. Fixed quantization layer truncation methods typically remove higher-level quantization tokens before they enter the channel. While this reduces the transmission rate, it mixes information loss caused by bandwidth compression with information loss caused by channel transmission. Even under favorable channel conditions, the pre-removed higher-level quantization tokens cannot be directly transmitted, forcing the receiver to predict a large number of acoustic detail tokens that could otherwise be reliably received, resulting in unnecessary reconstruction errors.

[0005] Furthermore, the error handling at the receiver also fails to fully utilize the contextual association of discrete voice tokens in the time dimension and the dependency between different residual quantization layers, making it difficult to accurately recover the original discrete tokens at the location of channel damage.

[0006] Therefore, it is necessary to provide a voice semantic communication method for multi-layer discrete voice tokens, so that the sender can implement differentiated protection based on the importance of discrete voice tokens, so that low-importance tokens can continue to participate in transmission with lower protection strength instead of being directly deleted before the channel, and the receiver can recover the tokens that have actually been damaged based on the temporal context and the quantization inter-layer context. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention proposes a voice communication method and system based on importance-level protection and mask recovery. The method encodes the voice signal into multi-layer discrete voice tokens, determines the importance level based on the contribution of each discrete voice token to voice content preservation and voice reconstruction, implements unequal error protection according to the importance level, and recovers the discrete voice tokens at the channel impairment location at the receiving end using temporal context and quantization inter-layer context. This solves the problem that existing technologies cannot simultaneously address acoustic reconstruction quality, voice intelligibility, and channel robustness.

[0008] A voice communication method based on importance-level protection and mask recovery includes the following steps:

[0009] Step S1: Multilayer Discrete Speech Coding

[0010] The voice signal to be transmitted is acquired, preprocessed, and then input into a multilayer residual vector quantization voice encoder to obtain a discrete voice token C containing multiple residual quantization layers.

[0011] C∈{0,1,…,V-1} Q×B×T

[0012] Where Q represents the number of residual quantization layers, B represents the batch dimension, T represents the duration of the discrete speech token, and V represents the codebook size corresponding to each residual quantization layer.

[0013] Preferably, the multilayer residual vector quantization speech encoder can be implemented using a neural speech encoder-decoder that includes an acoustic coding network, a residual vector quantization module, and a speech decoding network. Specifically, it can adopt the SpeechTokenizer structure, but is not limited to this structure.

[0014] Step S2: Importance Assessment of Discrete Voice Tokens

[0015] The discrete voice tokens are converted into tensor form C arranged in batch, quantization layer, and time order. BQT :

[0016] C BQT ∈{0,1,…,V-1} B×Q×T

[0017] And perform importance assessment to obtain the relationship with each discrete voice token C. BQT Importance score S for one-to-one correspondence:

[0018] S∈{0,1} B×Q×T

[0019] Preferably, the method for evaluating the importance of the discrete voice token is as follows:

[0020] First, a discrete voice token C is constructed through an embedding layer. BQT The input features. For the first The residual quantization layer, the first Discrete voice tokens at various time locations Its input features Represented as:

[0021]

[0022] in, This indicates the token embedding function. This represents the residual quantization layer embedding function. This represents the embedding function for time and location.

[0023] Then, a contextual modeling network is used to model the input features to obtain the first... The first audio message The residual quantization layer, the first Context feature vectors corresponding to each time position The context feature vector Input the score mapping network to obtain the importance score of the corresponding discrete speech token. :

[0024]

[0025] in, The weight matrix of the first linear mapping layer is used to map the context feature vector to the hidden feature space. This represents the bias vector of the first linear mapping layer; This represents the weight matrix of the second linear mapping layer, used to map hidden features to scalar score values; This represents the bias parameters of the second linear mapping layer; This represents the Gaussian error linear unit activation function, used to perform a nonlinear transformation on the output of the first linear mapping layer; This represents the Sigmoid function, used to map the scalar score value to the interval between 0 and 1.

[0026] As a preferred option, the above importance assessment method is trained using rule importance labels, reconstruction contribution importance labels, or a combination of both.

[0027] The rule importance label is determined based on the residual quantization level, and the value of the rule importance label decreases as the residual quantization level increases.

[0028] The reconstruction contribution importance label is determined based on the amount of reconstruction performance degradation caused by the occlusion location.

[0029] Step S3: Importance Perception Tiered Protection Decision

[0030] Let the high importance threshold be... The importance threshold is ,satisfy Each discrete voice token is assigned a protection level based on its importance score:

[0031]

[0032] in, Indicates the first The first audio message The residual quantization layer, the first Protection level of discrete voice tokens at each time location. =2 indicates strong protection. =1 indicates normal protection. =0 indicates low protection.

[0033] As a preferred option, to prevent the first residual quantization layer, which carries the basic speech content, from being incorrectly evaluated as low importance, the discrete speech token of the first residual quantization layer is forcibly set to a strong protection level, i.e. =2.

[0034] Step S4: Inequality error protection and communication channel transmission

[0035] Based on a pre-established mapping relationship between protection levels and unequal error protection parameters, different unequal error protection parameters are configured for discrete voice tokens with different protection levels. Higher protection levels result in stronger protection. During channel transmission, for discrete voice tokens that experience errors, verification failures, packet loss, or erasure, the corresponding location is set to a preset mask identifier and transmitted to the receiving end.

[0036] Preferably, the inequality error protection parameters are selected from one or more of the following: channel coding rate, channel coding redundancy length, modulation order, transmit power, number of repeated transmissions, number of automatic repeat requests, hybrid automatic repeat request priority, interleaving depth, resource block allocation location, subcarrier channel quality, and packet priority.

[0037] Preferably, the following parameters are used: lower channel coding rate, larger channel coding redundancy length, lower modulation order, higher transmit power, more retransmissions or maximum automatic repeat request (ATR) counts, higher hybrid ATR priority or group priority, larger interleaving depth, or allocation to resource blocks or subcarriers with higher channel quality. The specific level of each parameter is determined based on the current channel state and system resource constraints.

[0038] Step S5: Receiver masked discrete voice token recovery

[0039] The received discrete speech tokens containing mask locations are masked and recovered. The original tokens corresponding to the mask locations are predicted using the temporal context of the undamaged discrete speech tokens and the inter-layer context of residual quantization.

[0040] Preferably, the mask recovery method is as follows: constructing the receiver's first layer through an embedding layer. The residual quantization layer, the first Receive discrete voice tokens at each time location Input features :

[0041]

[0042] in, For mask identifier, For learnable mask embedding, This is an indicator function.

[0043] Then, the hidden state corresponding to each position is output using a context coding network, and the prediction score for each position and each codebook identifier is obtained through a token classification network. .

[0044] Finally, for the mask position, the codebook identifier with the highest prediction score is taken as the recovery result. For unmasked received discrete speech tokens, keep the received value unchanged and output the recovered discrete speech token. :

[0045]

[0046]

[0047] As a preferred approach, the mask recovery model is trained using a self-supervised method, calculating the token classification loss only at the occluded locations. :

[0048]

[0049] in, This represents the set of locations that are obscured during training. Represents a set Size, This represents the input token sequence obtained after masking the original discrete speech tokens; This indicates the input token sequence after a given masking. Under the condition that the mask recovery method will be the first The first audio message The residual quantization layer, the first Each time location is predicted as the original discrete speech token. The conditional probability.

[0050] As a preferred option, during the training of the mask recovery model, spectral reconstruction loss, waveform reconstruction loss, scale-invariant signal distortion ratio loss, or perceptual quality loss can be added to the token classification loss to improve the consistency between the recovered discrete speech tokens and the final speech quality index.

[0051] Step S6: Adaptively determine the number of decoding layers based on the channel state.

[0052] Obtain the channel state information of the current communication channel, and determine the number of residual quantization layers to participate in speech decoding based on the channel state information. , Restore the previous steps after step S5. The discrete speech token input of each residual quantization layer is combined with the speech decoder corresponding to the multi-layer residual vector quantization speech encoder to obtain the reconstructed speech signal.

[0053] The better the channel condition, the more residual quantization layers are involved in speech decoding. The larger the residual quantization layer, the more timbre, rhythm, and acoustic details are preserved; the worse the channel condition, the more residual quantization layers are involved in speech decoding. The smaller the value, the more likely it is to prioritize the basic voice content and intelligibility, and reduce the interference of erroneous high-level tokens on voice reconstruction.

[0054] Preferably, the channel state information is selected from one or more of the following: signal-to-noise ratio, bit error rate, packet error rate, token erasure rate, received signal strength, channel quality indicator, channel decoding confidence level, and number of retransmissions.

[0055] The voice communication system based on importance-level protection and mask recovery includes a discrete voice coding module, an importance assessment module, and an importance-aware hierarchical protection module at the transmitting end, and a mask recovery module and a voice decoding module at the receiving end.

[0056] The discrete speech coding module is used to encode the speech signal to be transmitted into a discrete speech token containing multiple residual quantization layers.

[0057] The importance assessment module calculates the importance score of each discrete voice token through a context modeling network.

[0058] The importance-aware hierarchical protection module assigns a protection level to each discrete voice token based on its importance score, implementing unequal error protection. The higher the importance score, the higher the protection level.

[0059] The mask recovery module uses the temporal context of the undamaged discrete speech token and the interlayer context of the residual quantization layer to predict the original token corresponding to the mask position.

[0060] The speech decoding module determines the number K of residual quantization layers participating in speech decoding based on the current communication channel status information, and inputs the discrete speech tokens of the first K residual quantization layers recovered by the mask recovery module into the speech decoder to obtain the reconstructed speech signal.

[0061] The present invention has the following beneficial effects:

[0062] Compared with the prior art, the present invention has at least the following beneficial effects.

[0063] First, the present invention assesses the importance of discrete voice tokens at the level of discrete voice tokens and implements unequal error protection for different tokens based on the importance results, so that limited channel protection resources are given priority to discrete voice tokens that have a greater impact on voice content, voice intelligibility and basic acoustic structure.

[0064] Second, this invention defines low-protection discrete voice tokens as tokens transmitted with weaker protection, rather than tokens that are directly deleted before channel transmission. Therefore, under good channel conditions, higher-layer residual quantization tokens can still be correctly received with a high probability, avoiding unnecessary predictions by the receiver of a large number of acoustic details that could be reliably transmitted.

[0065] Third, this invention separates the importance classification decision-making process from the channel impairment process. The importance classification module is responsible for determining the protection level, and the communication channel determines the actual error or erasure result based on the protection level and the current channel state, thereby avoiding the mixing of information loss caused by active code rate compression with information loss caused by channel errors.

[0066] Fourth, this invention utilizes the contextual relationships of discrete voice tokens in the time dimension and the dependencies between different residual quantization layers to predict the location where channel impairment actually occurs, while keeping the original received values ​​at unimpaired locations unchanged, thereby reducing erroneous modifications of reliable information by the recovery network.

[0067] Fifth, the present invention adaptively determines the number of residual quantization layers involved in speech decoding based on the channel state, preserving more acoustic details when the channel conditions are good, and prioritizing speech content and intelligibility when the channel conditions are poor.

[0068] Sixth, under the condition that the average erasure intensity is the same or similar, the present invention classifies the importance of discrete speech tokens and applies different levels of error protection to different protection levels, so that high-importance tokens have a lower actual loss probability, thereby improving the perceptual quality, intelligibility and channel robustness of reconstructed speech. Attached Figure Description

[0069] Figure 1 A schematic diagram of a multi-layer residual vector quantization discrete speech token structure;

[0070] Figure 2 A schematic diagram of importance-aware hierarchical protection and physical layer mapping;

[0071] Figure 3 This is a schematic diagram of mask recovery and adaptive decoding at the receiving end;

[0072] Figure 4 This is a schematic diagram of a voice communication system based on importance-based hierarchical protection and mask recovery. Detailed Implementation

[0073] The present invention will be further explained below with reference to the accompanying drawings;

[0074] A voice communication method based on importance-level protection and mask recovery includes the following steps:

[0075] Step S1: Multilayer Discrete Speech Coding

[0076] The voice data to be transmitted is acquired, converted into mono, and resampled to the sampling rate required by the multi-layer residual vector quantization voice encoder.

[0077] The processed speech signal is input into a pre-trained multilayer residual vector quantization speech encoder to obtain, as shown below. Figure 1 The discrete voice token C shown:

[0078] C∈{0,1,…,V-1} Q×B×T

[0079] In this embodiment, the number of residual quantization layers is set to Q=8, the codebook size of each layer is V=1024, B represents the batch dimension, and T represents the time length of the discrete voice token.

[0080] Step S2: Importance Assessment of Discrete Voice Tokens

[0081] To facilitate processing by the intermediate communication module, the discrete voice token C is... Convert to Tensor C of form BQT .

[0082] Using a Transformer encoder for CBQT Importance assessment was performed. The parameters of the Transformer encoder were set as follows: feature dimension of 256; number of multi-head self-attention heads of 4; number of Transformer encoder layers of 4; hidden dimension of feedforward network of 1024; dropout probability of 0.1; maximum token duration of 2000.

[0083] For each discrete speech token, a token embedding, a residual quantization layer embedding, and a temporal location embedding are constructed. These three embeddings are then summed and used as the input features, which are then fed into the Transformer encoder. The Transformer encoder is mapped via a two-layer fully connected fractional mapping network and a sigmoid function to obtain the... The first audio message The residual quantization layer, the first Discrete voice tokens at various time locations Output an importance score between 0 and 1 Complete the importance assessment

[0084] In this embodiment, the importance assessment method is trained in two stages, with the AdamW optimizer used and an initial learning rate of [missing information]. The weight decay coefficient is The gradient norm clipping threshold is 1.0. In the first training phase, rule importance labels are used for initial training, and importance scores are calculated. Importance of rules labels The mean squared error between them. Specifically, the rule importance label for the first residual quantization layer is set to 1.0, the rule importance label for the second to fourth residual quantization layers is set to 0.6, and the rule importance label for the fifth to eighth residual quantization layers is set to 0.2.

[0085] In the second training phase, construct reconstruction contribution importance labels. , and the importance of the rule label After combination, the importance assessment methods are jointly trained, and the combined label Y is:

[0086]

[0087] The importance label of the reconstruction contribution The construction method is as follows: For each speech segment, four random masking trials are conducted, with 12% of the target discrete speech tokens randomly masked in each trial. Speech decoding is performed on both the unmasked and masked discrete speech tokens, and the log-Mel spectrum L1 distance between the two reconstructed speech segments is calculated. A greater difference in the reconstructed segments after masking indicates a higher importance of the masked tokens. The difference in the reconstructed segments after masking is normalized and used as a label for the importance of the reconstruction contribution.

[0088] Step S3: Importance Perception Tiered Protection Decision

[0089] Set a high importance threshold The importance threshold is 0.75. The importance score is 0.35. When the importance score is not less than 0.75, the corresponding discrete speech token is set to the strong protection level; when the importance score is not less than 0.35 and less than 0.75, the corresponding discrete speech token is set to the normal protection level; when the importance score is less than 0.35, the corresponding discrete speech token is set to the low protection level. In particular, all discrete speech tokens in the first residual quantization layer are forcibly set to the strong protection level.

[0090] In typical output, strong protection discrete voice tokens account for approximately 12.5% ​​of the total tokens, normal protection discrete voice tokens account for approximately 37.5%, and low protection discrete voice tokens account for approximately 50%.

[0091] Each discrete voice token is grouped according to voice frame, residual quantization layer, and time position. Each token group includes at least: voice frame number, residual quantization layer number, time position number, discrete voice token identifier, protection level, group length, and cyclic redundancy check (CRC) code. The protection level can be directly written into the group header or implicitly represented through different logical channels, different queues, or different resource blocks.

[0092] Step S4: Inequality error protection and communication channel transmission

[0093] like Figure 2 As shown, different unequal error protection parameters are configured for different protection levels based on the protection level of each discrete voice token. The higher the protection level, the stronger the protection. High-protection discrete voice tokens are preferentially mapped to resource blocks with higher channel quality, while low-protection discrete voice tokens are mapped to the remaining resource blocks.

[0094] In this embodiment, a combination of channel coding, modulation scheme, and retransmission priority is used to implement unequal error protection. An example configuration is shown in Table 1:

[0095] Table 1

[0096]

[0097] Based on the channel estimation results, the signal-to-noise ratio, packet error rate, channel quality indicator, or decoding confidence level are obtained. The channel state is then classified into three levels—good, moderate, and poor—according to thresholds pre-determined through link calibration. The channel coding rate, modulation scheme, transmit power, and retransmission count corresponding to different protection levels can be adjusted based on the channel state.

[0098] Those skilled in the art can replace the channel coding with convolutional codes, Turbo codes, low-density parity-check codes, or polar codes, and replace the modulation scheme with other digital modulation schemes, without departing from the basic concept of the present invention, and adjust the resource allocation and retransmission mechanism according to the specific communication protocol.

[0099] This embodiment uses discrete erase channels to simulate the token loss result after inequality error protection during communication channel transmission.

[0100] The erasure probabilities of discrete voice tokens with different protection levels under different channel conditions are shown in Table 2:

[0101] Table 2

[0102]

[0103] Step S5: Receiver masked discrete voice token recovery

[0104] After demodulation and channel decoding are completed at the receiving end, the erroneous or lost discrete voice tokens are determined based on the cyclic redundancy check result, packet sequence number, and token position index. Figure 3 As shown, for token positions that fail verification or are not received, the corresponding discrete speech token is set as a mask identifier. A Transformer encoder is used to perform mask recovery on the received discrete speech tokens containing the mask positions. Its parameters are set as follows: token embedding dimension is 256; number of multi-head self-attention heads is 4; number of Transformer encoder layers is 6; feedforward network hidden dimension is 1024; dropout probability is 0.1; maximum time length is 2000; number of token classification categories is 1024.

[0105] The range of valid discrete voice token identifiers is 0 to 1023, and the mask identifier is set to 1024. Therefore, the token embedding table contains 1025 embedding vectors.

[0106] During training for mask recovery, the random occlusion probability of the first residual quantization layer was set to 0.02, the random occlusion probabilities of the second to fourth residual quantization layers were set to 0.10, and the random occlusion probabilities of the fifth to eighth residual quantization layers were set to 0.20. Cross-entropy loss was calculated only at occluded locations. Unoccluded locations were not included in the recovery loss calculation. The optimizer used was AdamW, with a learning rate of [missing information]. The weight decay coefficient is The gradient norm clipping threshold is 1.0.

[0107] During inference, token category scores are output for all positions, but only the mask positions in the received discrete speech tokens are replaced with the prediction results. Received discrete speech tokens that are not masked retain their original received values, and the recovered discrete speech tokens are output.

[0108] Step S6: Adaptively determine the number of decoding layers based on the channel state.

[0109] Obtain the channel state information of the current communication channel, and determine the number of residual quantization layers involved in decoding according to the definitions in Table 2. Restore the previous state after step S5. The discrete speech token input of each residual quantization layer is combined with the speech decoder corresponding to the multi-layer residual vector quantization speech encoder to obtain the reconstructed speech signal.

[0110] To verify the effectiveness of this method, it is compared with the following methods: an eight-layer residual quantization speech reconstruction method without channel impairment; a four-layer residual quantization speech reconstruction method without channel impairment; an eight-layer residual quantization method with unified erasure and no recovery; a four-layer residual quantization method with unified erasure and no recovery; a method using hierarchical protection but no recovery; and a method using hierarchical protection and neighbor token rule recovery. To ensure that the average channel impairment intensity is the same or similar during comparison, the unified erasure baseline uses the same unified erasure probability as the weighted average erasure rate of this embodiment for all discrete speech tokens. Based on the proportion of discrete speech tokens with different protection levels in the typical output of this method, the unified erasure probability is set to 0.0325 for good channel conditions, 0.13125 for medium channel conditions, and 0.03875 for poor channel conditions.

[0111] 824 speech samples were collected as test data. Evaluation metrics included mean square error (MSE), signal-to-distortion ratio (SDR), scale-invariant signal-to-distortion ratio (SI-SDR), speech perception quality assessment (PESQ), and short-time objective intelligibility (STOI).

[0112] Under good channel conditions, the test results of each representative method are shown in Table 3:

[0113] Table 3

[0114]

[0115] The measured average erasure rate of this method is 3.29%, while the measured average erasure rate of the unified erasure of the RVQ8 baseline is 3.26%. With essentially the same average erasure rate, compared to the unprotected unified erasure method, the method of this invention improves PESQ from 2.116 to 2.612, STOI from 0.890 to 0.919, and SI-SDR from 2.336 dB to 3.805 dB. This indicates that the performance improvement comes from the priority protection of critical discrete voice tokens, rather than from the lower average erasure rate.

[0116] The test results of each representative method under medium channel conditions are shown in Table 4:

[0117] Table 4

[0118]

[0119] Compared to the unified erase RVQ8 baseline, this method reduces MSE by approximately 39.2%, improves SDR by approximately 2.22 dB, SI-SDR by approximately 5.91 dB, PESQ by approximately 0.79, and STOI by approximately 0.107. Furthermore, this method also outperforms the unified erase RVQ4 baseline, indicating that the performance improvement is not solely achieved by reducing the number of quantization layers involved in decoding, but rather by enhancing the reliability of effective discrete speech tokens through hierarchical protection and context recovery.

[0120] The test results of each representative method under poor channel conditions are shown in Table 5:

[0121] Table 5

[0122]

[0123] Under poor channel conditions, this method uses four layers of residual quantization decoding. Therefore, the fifth to eighth residual quantization layers do not participate in the final speech decoding. The speech content and intelligibility are maintained by protecting the first to fourth residual quantization layers.

[0124] In traditional fixed quantization layer truncation methods, low-protection discrete voice tokens are pre-replaced with mask identifiers before entering the communication channel, resulting in approximately 50% of the fifth to eighth residual quantization layer tokens being deleted before channel transmission. In this method, importance-aware hierarchical protection ensures that low-protection discrete voice tokens retain their original identifiers when entering the channel, increasing the proportion of all discrete voice tokens entering the channel from 0.5 to 1.0.

[0125] Under good channel conditions, the results of this method and the fixed quantization layer truncation method are shown in Table 6:

[0126] Table 6

[0127]

[0128] Compared to the fixed quantization layer truncation method, this method reduces MSE by approximately 43.5%, improves SDR by approximately 2.49 dB, SI-SDR by approximately 4.04 dB, PESQ by approximately 0.71, and STOI by approximately 0.046. These results indicate that defining the low-protection discrete voice token as a weakly protected transmission token is more suitable for channel-robust communication scenarios than defining it as a non-transmission token.

[0129] like Figure 4 As shown, the voice communication system based on importance classification protection and mask recovery includes a discrete voice coding module, an importance assessment module, and an importance-aware classification protection module at the transmitting end, and a mask recovery module and a voice decoding module at the receiving end.

[0130] The discrete speech coding module is used to encode the speech signal to be transmitted into a discrete speech token containing multiple residual quantization layers.

[0131] The importance assessment module calculates the importance score of each discrete voice token through a context modeling network.

[0132] The importance-aware hierarchical protection module assigns a protection level to each discrete voice token based on its importance score, implementing unequal error protection. The higher the importance score, the higher the protection level.

[0133] The mask recovery module uses the temporal context of the undamaged discrete speech token and the interlayer context of the residual quantization layer to predict the original token corresponding to the mask position.

[0134] The speech decoding module determines the number K of residual quantization layers participating in speech decoding based on the current communication channel status information, and inputs the discrete speech tokens of the first K residual quantization layers recovered by the mask recovery module into the speech decoder to obtain the reconstructed speech signal.

Claims

1. A voice communication method based on importance-level protection and mask recovery, characterized in that: Includes the following steps: Step S1: Encode the speech signal to be transmitted into a discrete speech token C using a multilayer residual vector quantization speech encoder; Step S2: Evaluate the importance of the discrete speech tokens and calculate the importance score S; Step S3: Set the importance level threshold and assign a protection level to each discrete voice token according to the importance score; Step S4: Based on the pre-established mapping relationship between protection levels and unequal error protection parameters, configure different unequal error protection parameters for discrete voice tokens with different protection levels, and transmit them to the receiving end; the unequal error protection parameters are selected from one or more of the following: channel coding rate, channel coding redundancy length, modulation order, transmit power, number of retransmissions, number of automatic repeat requests, hybrid automatic repeat request priority, interleaving depth, resource block allocation location, subcarrier channel quality, and packet priority; Step S5: For discrete voice tokens that have encountered errors, failed verification, lost packets, or been erased during transmission, set the corresponding position as a preset mask identifier; at the receiving end, perform mask recovery on the discrete voice tokens containing the mask position, and use the temporal context of the undamaged discrete voice tokens and the inter-layer context of residual quantization to predict the original token corresponding to the mask position; Step S6: Determine the number K of residual quantization layers participating in speech decoding based on the current channel state information; input the discrete speech tokens of the first K residual quantization layers recovered in step S5 into the speech decoder corresponding to the multilayer residual vector quantization speech encoder to obtain the reconstructed speech signal.

2. The voice communication method based on importance-level protection and mask recovery as described in claim 1, characterized in that: The method for evaluating the importance of discrete voice tokens in step S2 is as follows: First, the discrete speech tokens are embedded using an embedding layer, performing token embedding, residual quantization layer embedding, and temporal location embedding to construct the input features of the discrete speech tokens. These features are then input into a context modeling network for modeling, resulting in the... The first audio message The residual quantization layer, the first Context feature vectors corresponding to each time position ; the context feature vector Input the score mapping network to obtain the importance score of the corresponding discrete speech token. .

3. The voice communication method based on importance-level protection and mask recovery as described in claim 2, characterized in that: The above importance assessment method is trained using rule importance labels, reconstruction contribution importance labels, or a combination of both; The rule importance label is determined based on the residual quantization level, and the value of the rule importance label decreases as the residual quantization level increases; the reconstruction contribution importance label is determined based on the amount of reconstruction performance degradation caused by the occlusion position.

4. The voice communication method based on importance-level protection and mask recovery as described in claim 1, characterized in that: In step S3, the highest protection level is assigned to the discrete speech token of the first residual quantization layer.

5. The voice communication method based on importance-level protection and mask recovery as described in claim 1, characterized in that: The higher the protection level, the lower the channel coding code rate, the larger the channel coding redundancy length, the lower the modulation order, the higher the transmit power, the more retransmissions or maximum automatic repeat request times, the higher the hybrid automatic repeat request priority or group priority, the larger the interleaving depth, or the higher the resource block or subcarrier with higher channel quality.

6. The voice communication method based on importance-level protection and mask recovery as described in claim 1, characterized in that: In step S5, the mask recovery method is as follows: the input features of the received discrete speech token are constructed through the embedding layer, then the hidden state corresponding to each position is output using the context coding network, and the prediction score for each position and each codebook identifier is obtained through the token classification network; finally, for the masked position, the codebook identifier with the largest prediction score is taken as the recovery result, and for the received discrete speech token that is not masked, its received value is kept unchanged, and the recovered discrete speech token is output. The mask recovery model is trained using a self-supervised approach, and the token classification loss is calculated only at the masked locations.

7. The voice communication method based on importance-level protection and mask recovery as described in claim 6, characterized in that: In the training of the masked reconstruction model, spectral reconstruction loss, waveform reconstruction loss, scale-invariant signal distortion ratio loss, or perceptual quality loss are added to the token classification loss to improve the consistency between the recovered discrete speech tokens and the final speech quality index.

8. The voice communication method based on importance-level protection and mask recovery as described in claim 1, characterized in that: The channel state information is selected from one or more of the following: signal-to-noise ratio, bit error rate, packet error rate, token erasure rate, received signal strength, channel quality indicator, channel decoding confidence level, and number of retransmissions; the better the channel state, the greater the number of residual quantization layers involved in voice decoding.

9. A voice communication system based on importance-level protection and mask recovery, characterized in that: It includes a discrete speech coding module, an importance assessment module, and an importance-aware hierarchical protection module at the transmitting end, and a mask recovery module and a speech decoding module at the receiving end; The discrete speech coding module is used to encode the speech signal to be transmitted into a discrete speech token containing multiple residual quantization layers. The importance assessment module calculates the importance score of each discrete voice token through a context modeling network; The importance-aware hierarchical protection module assigns protection levels to each discrete voice token based on its importance score, and implements unequal error protection. The higher the importance score, the higher the protection level; The mask recovery module uses the temporal context of the undamaged discrete speech token and the inter-layer context of residual quantization to predict the original token corresponding to the mask position. The speech decoding module determines the number K of residual quantization layers participating in speech decoding based on the current communication channel status information, and inputs the discrete speech tokens of the first K residual quantization layers recovered by the mask recovery module into the speech decoder to obtain the reconstructed speech signal.

Images