Voice transmission method, device, storage medium and program product

Abstract

The application discloses a voice transmission method and device, a storage medium and a program product, relates to the technical field of communication, and comprises the following steps: a client performs semantic-acoustic decoupling coding on a voice frame to be transmitted to obtain a semantic feature vector and an acoustic feature vector of the voice frame; at least according to current acquired network state information, a transmission strategy of the voice frame is determined; the transmission strategy at least comprises whether to discard the acoustic feature vector; and according to the transmission strategy, at least the semantic feature vector is sent to a server. The application reduces the probability of semantic illusion generated by the server AI model, and improves the robustness and availability of a voice interaction system based on a client-server architecture in a weak network environment.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a voice transmission method, device, storage medium, and program product. Background Technology

[0002] In voice-based human-computer interaction scenarios, a client-server system architecture is typically adopted. During the process of human-computer interaction based on the client, the client sends the collected voice to the server through the network. The server uses an AI model to process the received voice and obtain feedback information, which is then returned to the client.

[0003] However, the AI ​​models in the above system architecture are prone to semantic illusions, resulting in a poor voice interaction experience. Summary of the Invention

[0004] In view of the above problems, this application provides a voice transmission method, device, storage medium, and program product to reduce the probability of semantic illusion in AI models. The specific solution is as follows: The first aspect of this application provides a voice transmission method applied to a client, the method comprising: The speech frame to be transmitted is semantically and acoustically decoupled and encoded to obtain the semantic feature vector and acoustic feature vector of the speech frame. Based at least on the currently acquired network status information, a transmission strategy for the voice frame is determined; the transmission strategy includes at least: whether to discard the acoustic feature vector; According to the transmission strategy, at least the semantic feature vector shall be sent to the server.

[0005] In one possible implementation, semantic-acoustic decoupling coding of the speech frame includes: The speech frame is semantically and acoustically decoupled and encoded by a semantic-acoustic decoupling encoder to obtain the semantic feature vector and acoustic feature vector of the speech frame. The semantic-acoustic decoupling encoder is trained in the following way: The semantic-acoustic decoupling encoder is used to perform speech-acoustic decoupling encoding on each speech frame of the speech sample to obtain the semantic feature vector and acoustic feature vector of each speech frame. The semantic prediction result of the semantic sample is obtained by the parsing module based on the semantic feature vector of each speech frame; the reconstructed spectrogram of the speech sample is obtained by the reconstruction module based on the acoustic feature vector and semantic feature vector of each speech frame. The parameters of the semantic-acoustic decoupling encoder, the parsing module, and the reconstruction module are updated with the goal of making the semantic prediction result of the speech sample close to the semantic label of the speech sample, reconstructing the spectrogram close to the spectrogram of the speech sample, and making the semantic feature vector of each speech frame far away from the acoustic feature vector of the speech frame.

[0006] In one possible implementation, the method further includes: determining the semantic importance level of the speech frame based on the semantic feature vector;

[0007] Determining the transmission strategy for the voice frame based at least on the currently acquired network status information includes: Based on the currently acquired network status information and the semantic importance level, the transmission strategy for the voice frame is determined.

[0008] In one possible implementation, determining the semantic importance level of the speech frame based on the semantic feature vector includes: The semantic feature vectors are classified using a lightweight classification network to obtain the semantic importance level of the speech frames; The lightweight classification network takes the semantic feature vector of the speech frame sample as input, the semantic importance level of the speech frame sample as output, and aims to update the parameters of the lightweight classification network so that the semantic importance level output by the lightweight classification network approaches the semantic importance level label of the speech frame sample.

[0009] In one possible implementation, the semantic importance level labels of the speech frame samples are generated in the following manner: For each audio sample in the training corpus, the semantic feature vector of each speech frame of the audio sample is processed by the AI ​​model on the server side, and the error between the result of the AI ​​model processing the audio sample and the label of the audio sample is determined as the baseline error. The audio sample is perturbed multiple times. Each perturbation replaces the semantic feature vector of a speech frame in the audio sample with the perturbed frame to obtain a perturbed audio. The perturbed audio is then processed by the AI ​​model. The error between the result of the AI ​​model's processing of the perturbed audio and the label of the audio sample is determined as the perturbed error corresponding to that perturbation. The semantic importance score of each perturbation is determined based on the post-perturbation error corresponding to each perturbation. The percentile division method is used to divide all semantic importance scores obtained from the training corpus into multiple ranges, with different ranges corresponding to different semantic importance levels; the larger the maximum value in each range, the higher the corresponding semantic importance level.

[0010] In one possible implementation, the transmission strategy for the voice frame is determined based on the currently acquired network state information and the semantic importance level, including: According to the preset strategy lookup table, the transmission strategy corresponding to the currently acquired network status information and the semantic importance level is searched and used as the transmission strategy for the voice frame.

[0011] In one possible implementation, a transmission strategy corresponding to the currently acquired network state information and the semantic importance level is searched according to a preset strategy lookup table and used as the transmission strategy for the voice frame, including: The network state information is smoothed to obtain smoothed network state information; wherein, the smoothed network state information is the weighted sum of the currently acquired network state information and the previously acquired smoothed network state information; According to the preset strategy lookup table, the transmission strategy corresponding to the smoothed network state information and semantic importance level is found and used as the transmission strategy for the voice frame.

[0012] In one possible implementation, the transmission strategy further includes: a redundancy protection depth configured for the voice frames; different voice frames may have the same or different redundancy protection depths configured. The step of sending the semantic feature vector to the server at least according to the transmission strategy includes: At least the semantic feature vector and the semantic feature vector of the number of historical voice frames corresponding to the redundancy protection depth are sent to the server.

[0013] In one possible implementation, at least the semantic feature vector and the semantic feature vectors of a number of historical voice frames corresponding to the redundancy protection depth are sent to the server, including: If the transmission strategy includes discarding the acoustic feature vector and configuring a redundancy protection depth for the speech frame, then the first quantized feature obtained by first vector quantization encoding the semantic feature vector of the speech frame and the first quantized feature obtained by first vector quantization encoding the semantic features of the historical speech frame are encapsulated into a Real-Time Transport Protocol (RTP) payload packet and sent to the server. If the transmission strategy includes not discarding the acoustic feature vector and configuring a redundancy protection depth for the speech frame, then the first quantized feature obtained by performing a first vector quantization encoding on the semantic feature vector of the speech frame, the second quantized feature obtained by performing a second vector quantization encoding on the acoustic feature of the speech frame, and the first quantized feature obtained by performing a first vector quantization encoding on the semantic feature of the historical speech frame are encapsulated into an RTP payload packet and sent to the server; the data compression efficiency of the first vector quantization encoding is lower than the data compression efficiency of the second vector quantization encoding.

[0014] In one possible implementation, the load portion of the RTP payload includes: A header, the header encapsulating at least the following dynamic fields: a layer flag indicating whether the acoustic feature vector is included, a semantic flag indicating the semantic importance level, a redundancy count indicating the redundancy protection depth, and a redundancy encoding format.

[0015] In one possible implementation, the first quantization features of the number of historical voice frames corresponding to the redundancy protection depth are carried in the payload of the RTP payload packet in the form of a full copy or differential coding.

[0016] One possible implementation also includes: During the session establishment phase with the server, the parameters required for the first vector quantization encoding and the second vector quantization encoding are negotiated with the server through the extended fields of the Session Description Protocol (SDP).

[0017] A second aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the voice transmission method of the first aspect or any implementation thereof.

[0018] A third aspect of this application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the voice transmission method of the first aspect or any implementation thereof.

[0019] A fourth aspect of this application provides a computer storage medium carrying one or more computer programs that, when executed by an electronic device, enable the electronic device to perform the voice transmission method described in the first aspect or any implementation thereof.

[0020] By employing the above technical solutions, the voice transmission method, device, storage medium, and program products provided in this application involve the client performing semantic-acoustic decoupling encoding on the voice frame to be transmitted to obtain the semantic feature vector and acoustic feature vector of the voice frame; determining the transmission strategy of the voice frame based at least on the currently acquired network state information; the transmission strategy includes at least: whether to discard the acoustic feature vector; and sending the semantic feature vector to the server at least according to the transmission strategy. When encoding the voice frame, the client of this application decouples the semantic and acoustic information of the voice frame. Based on the network state, acoustic features can be discarded, and only the core features that ensure semantic understanding are transmitted. Based on this, semantic coherence can be maintained even in weak network environments, avoiding interruption of semantic information acquired by the server due to voice frame loss, thereby reducing the probability of semantic illusions generated by the server-side AI model and improving the robustness and availability of the client-server-based voice interaction system in weak network environments. Attached Figure Description

[0021] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0022] Figure 1 An example architecture diagram of the voice interaction system provided in this application; Figure 2 A flowchart illustrating an implementation of the voice transmission method provided in this application; Figure 3 An example diagram of the network structure provided in this application for training a semantic-acoustic decoupling encoder; Figure 4 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0023] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0024] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0025] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0026] like Figure 1 The diagram shown is an example architecture of a voice interaction system provided in this application. The voice interaction system includes a terminal 100 (also called a client) and a server 200 (also called a server). The server 200 may include one or more servers ( Figure 1 (This example uses a single server as an illustration). Server 200 can be a cloud server.

[0027] Terminal 100 and server 200 work together to enable voice interaction with the user. Terminal 100 can collect the user's voice, extract features from the voice, and send the extracted voice features to server 200. Server 200 calls a deployed AI model to process the voice features, generates a processing result (i.e., a response), and feeds the processing result back to terminal 100, which then outputs the processing result to the user. The AI ​​model can include, but is not limited to, at least one of the following: Automatic Speech Recognition (ASR) model, Large Language Model (LLM), Multimodal Large Models (MLLMs), etc.

[0028] Terminal 100 can be a mobile phone, tablet computer, teaching large screen, wearable device, vehicle-mounted device, conference terminal, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.

[0029] The inventors of this application have discovered that currently, voice interaction based on the aforementioned architecture often employs a cascaded approach of "audio encoding (e.g., Opus) → network transmission → ASR transcription → LLM inference." In this process, audio encoding (directly encoding the audio waveform of the speech frame) is waveform-guided encoding, which couples semantic information with acoustic details. This results in the inability to retain the smallest information units crucial to the downstream AI model in a semantically prioritized manner during network congestion or packet loss, leading to problems such as ASR / LLM errors (semantic illusions) caused by the loss of key semantics.

[0030] This application is filed to reduce the probability of semantic illusions in AI models.

[0031] like Figure 2 The diagram shown is a flowchart of one implementation of the voice transmission method provided in this application, which may include: Step S201: The client performs semantic-acoustic decoupling encoding on the voice frame to be transmitted to obtain the semantic feature vector and acoustic feature vector of the voice frame.

[0032] After a user inputs voice input on the client side, the client sends the voice to the server frame by frame.

[0033] For each speech frame, this application performs semantic-acoustic decoupling coding on the speech frame to obtain the semantic feature vector and acoustic feature vector of the speech frame.

[0034] The decoupled semantic and acoustic features of the speech frame can be obtained by encoding the speech frame using a pre-trained semantic-acoustic decoupling encoder.

[0035] Step S202: The client determines the transmission strategy of the voice frame based at least on the currently acquired network status information; the transmission strategy includes at least: whether to discard the acoustic feature vector.

[0036] The client can determine the transmission strategy for voice frames based solely on the currently acquired network status information, or it can determine the transmission strategy by combining the currently acquired network status information with the importance of the voice frames.

[0037] Network status information may include, but is not limited to: Packet Loss Rate (PLR), Round-Trip Time (RTT), etc.

[0038] The server periodically sends RTCP (Real-Time Transport Control Protocol) reports to the client, and the client can use the network status information recorded in the latest RTCP report as the current network status information.

[0039] The transmission strategy for a voice frame may only include whether to discard the acoustic features of the voice frame. Furthermore, the transmission strategy may also include a redundancy protection depth configured for the voice frame. The redundancy protection depth for each voice frame refers to the number of historical voice frames (i.e., voice frames already sent to the server) that need to be carried when sending the voice frame.

[0040] In one optional implementation, the redundancy protection depth is the same for different voice frames, meaning that the redundancy protection depth remains unchanged for different voice frames under different network conditions.

[0041] In another implementation, the redundancy protection depth corresponding to different voice frames may be the same or different; that is, the redundancy protection depth corresponding to different voice frames may change under different network conditions. In this application, the redundancy protection depth configured for the voice frame can be 0 (i.e., no historical voice frames are needed when sending the voice frame) or greater than 0 (i.e., the number of historical voice frames carried when sending the voice frame can be 1 or more).

[0042] Optionally, when the network state information indicates that the network is in a weak network environment, the acoustic feature vector can be discarded, that is, the semantic feature vector of the voice frame sent to the server is not sent to the server; if the network state information indicates that the network is in a normal network environment, the acoustic feature vector is not discarded, that is, the semantic feature vector and the acoustic feature vector of the voice frame are sent to the server.

[0043] Optionally, when the redundancy protection depth configured for the voice frame is variable, a smaller redundancy protection depth can be configured for the voice frame when the network status information indicates that the network is in a weak network environment. This avoids the problem of network congestion and packet loss caused by configuring a large redundancy protection depth for the voice frame in a weak network environment, which would lead to semantic interruption on the server side. When the network status information indicates that the network is in a normal network environment, configuring a larger redundancy protection depth for the voice frame can ensure that the server obtains the complete context of the voice frame, providing more contextual information for the semantic understanding of the voice frame and further improving the accuracy of semantic understanding.

[0044] Step S203: The client sends at least a semantic feature vector to the server according to the transmission strategy.

[0045] If the transmission strategy is to discard acoustic features, then only the semantic feature vector of the voice frame needs to be sent to the server.

[0046] If the transmission strategy is to not discard acoustic features, then the semantic feature vector and acoustic feature vector of the voice frame are sent to the server.

[0047] If the transmission strategy does not include redundancy protection depth configured for voice frames, the server can request the client to resend the lost voice frames when packet loss is detected. This is a post-transmission retransmission mechanism, suitable for voice interaction scenarios where real-time requirements are not high.

[0048] If the transmission strategy includes discarding acoustic features and the redundancy protection depth configured for the voice frame is 0, then only the semantic feature vector of the voice frame needs to be sent to the server.

[0049] If the transmission strategy is to not discard acoustic features and the redundancy protection depth configured for the voice frame is 0, then the semantic feature vector and acoustic feature vector of the voice frame are sent to the server.

[0050] If the transmission strategy includes discarding acoustic features and the redundancy protection depth configured for the voice frame is greater than 0, then the semantic feature vector of the voice frame and the semantic feature vector of the number of historical voice frames corresponding to the redundancy protection depth are sent to the server.

[0051] If the transmission strategy is to not discard acoustic features and the redundancy protection depth configured for the voice frame is greater than 0, then the semantic feature vector and acoustic feature vector of the voice frame, as well as the semantic feature vector of the number of historical voice frames corresponding to the redundancy protection depth, are sent to the server.

[0052] Including redundancy protection depth in the transmission strategy for voice frames is a proactive measure that allows the server to correct errors immediately when packet loss is detected. This is suitable for voice interaction scenarios with high real-time requirements.

[0053] The voice transmission method provided in this application decouples the semantic and acoustic information of the voice frame when the client encodes the voice frame. Depending on the network status, acoustic features can be discarded, and only the core features that ensure semantic understanding are transmitted. Based on this, the semantic coherence of the server can still be maintained in a weak network environment, avoiding the interruption of semantic information obtained by the server due to the loss of voice frames. This reduces the probability of semantic illusion generated by the server-side AI model and improves the robustness and availability of the voice interaction system based on the client-server architecture in a weak network environment.

[0054] In an optional embodiment, the semantic-acoustic decoupling encoder for semantic-acoustic decoupling encoding of speech frames described above can be trained based on semantic understanding tasks and speech reconstruction tasks. The training samples are speech samples, and the labels of the speech samples include: semantic labels of the speech samples and spectrograms of the speech samples (the spectrograms can be pre-generated and stored, and can be directly read when needed, or the speech samples can be processed to generate spectrograms when needed). The semantic labels are different for different semantic understanding tasks. For example, if the semantic understanding task is speech recognition, the corresponding semantic label can be speech-to-text; if the semantic understanding task is keyword extraction, the corresponding semantic label is keywords; if the semantic understanding task is intent recognition, the corresponding semantic label is intent.

[0055] Specifically, an example diagram of the network structure provided in this application for training a semantic-acoustic decoupling encoder is shown below. Figure 3 As shown, it may include: Semantic-acoustic decoupling encoder 301, parsing module 302 and reconstruction module 303; During training, the semantic-acoustic decoupling encoder 301 is used to perform semantic-acoustic decoupling encoding on each speech frame (or the acoustic features of the speech frame, such as Mel frequency cepstral coefficients MFCC, etc.) of the speech sample to obtain the semantic feature vector and acoustic feature vector of each speech frame.

[0056] As an example, the semantic-acoustic decoupling encoder 301 may include a shared encoding module 3011, a semantic encoding module 3012, and an acoustic encoding module 3013. The shared encoding module 3011 encodes the input speech frame to obtain an initial feature vector; the semantic encoding module 3012 encodes the initial feature vector to obtain a semantic feature vector; and the acoustic encoding module 3013 encodes the initial feature vector to obtain an acoustic feature vector.

[0057] The parsing module 302 is used to perform semantic prediction (including but not limited to at least one of the following: speech recognition, keyword extraction, intent recognition, etc.) based on the semantic feature vectors of each speech frame output by the semantic-acoustic decoupling encoder 301, and obtain the semantic prediction results of the speech samples.

[0058] The reconstruction module 303 is used to reconstruct speech based on the acoustic feature vectors and semantic feature vectors of each speech frame output by the semantic-acoustic decoupling encoder 301, and obtain the reconstructed spectrogram of the speech sample.

[0059] The parameters of the semantic-acoustic decoupling encoder 301, parsing module 302, and reconstruction module 303 are updated with the goal of making the semantic prediction result of the speech sample close to the semantic label of the speech sample, making the reconstructed spectrogram of the speech sample close to the spectrogram of the speech sample, and making the semantic feature vector of each speech frame far away from the acoustic feature vector of the speech frame.

[0060] Optionally, the first difference between the semantic prediction result and the semantic label can be calculated as the first loss (which can be denoted as semantic preservation loss), the second difference between the reconstructed spectrogram and the spectrogram of the speech sample can be calculated as the second loss (which can be denoted as reconstruction loss), and the sum of the differences between the semantic feature vector and the acoustic feature vector of each speech frame (for each speech frame, the difference between the speech feature vector and the acoustic feature vector of that speech frame is calculated, and the differences corresponding to each speech frame are summed) can be used as the third loss (which can be denoted as decoupling loss). With the goal of minimizing the first loss and the second loss and maximizing the third loss, the parameters of the semantic-acoustic decoupling encoder 301, the parsing module 302 and the reconstruction module 303 are updated.

[0061] Minimizing the first loss ensures that semantics are not lost; minimizing the second loss constrains the semantic and acoustic features to recover the main information of the input speech after fusion, thereby avoiding the loss of key content during the separation modeling process and ensuring that the overall information is not lost; maximizing the third loss ensures that the semantic and acoustic features are separated from each other.

[0062] Through the above training, the semantic-acoustic decoupling encoder can hierarchically identify the core semantic information and supplementary acoustic information in the input speech, enabling the voice interaction system to prioritize the processing and transmission of more critical semantic information in weak network environments, and flexibly combine acoustic information for compensation according to network conditions and business needs, thereby improving the robustness, real-time performance and recovery quality of voice interaction.

[0063] In an optional embodiment, to further improve the semantic coherence during voice transmission, the voice transmission method provided in this application may further include: The semantic importance level of a speech frame is determined based on its semantic feature vector.

[0064] Optionally, the semantic importance of a semantic frame can be divided into two levels: high and low; or, it can be divided into three levels: high, medium, and low.

[0065] Accordingly, at least one implementation of determining the transmission strategy for voice frames based on the currently acquired network state information can be: Based on the currently acquired network status information and the semantic importance level of the voice frame, the transmission strategy for the voice frame is determined.

[0066] Optionally, a transmission strategy corresponding to the currently acquired network status information and the semantic importance level of the voice frame can be found in a preset strategy lookup table and used as the transmission strategy for the voice frame.

[0067] Policy lookup tables can be obtained through offline simulation, historical playback testing, or statistical analysis of actual network data. Specifically, for different combinations of network states and semantic importance levels, the impact of different redundancy policies on semantic fidelity, interaction latency, and bandwidth overhead can be evaluated, and policy parameters that meet the target constraints can be selected to form a lookup table. The specific values ​​of the lookup table can vary for different application scenarios such as voice assistants, real-time conferencing, and in-vehicle interaction.

[0068] Table 1 shows an example of a strategy lookup table provided in an embodiment of this application.

[0069] Table 1

[0070]

[0071] The inventors of this application have discovered that in real-time voice interaction, network conditions (packet loss rate (PLR) and round-trip time (RTT)) often fluctuate drastically (i.e., so-called "jitter"), which can easily lead to sudden and sharp increases in bandwidth (for example, a momentary network outage causing a high packet loss rate, prompting the system to immediately switch to ultra-high redundancy mode, resulting in a sudden surge in bandwidth), or inconsistent policy changes leading to unstable voice quality. To overcome these problems, the network condition information can be smoothed first, and at least the transmission strategy for voice frames can be determined based on the smoothed network condition information.

[0072] One way to smooth the packet loss rate (PLR) is as follows: The current smoothed packet loss rate is obtained by weighted summing the current packet loss rate with the previously obtained smoothed packet loss rate.

[0073] The weight of the currently obtained packet loss rate is summed with the weight of the previously obtained smoothed packet loss rate, which is 1. Optionally, the weight of the currently obtained packet loss rate can be in the range of [0.1, 0.4].

[0074] One way to smooth the round-trip time (RTT) is as follows: The current smoothed round-trip time is obtained by weighted summing the current round-trip time with the smoothed round-trip time obtained previously.

[0075] The weight of the currently acquired round-trip delay is summed with the weight of the previously acquired smoothed round-trip delay, and the sum is 1. Optionally, the weight of the currently acquired round-trip delay can range from [0.1, 0.4]. The weight of the currently acquired round-trip delay can be the same as or different from the weight of the currently acquired packet loss rate.

[0076] When smoothing packet loss rate (PLR) and round-trip time (RTT), a transmission strategy corresponding to the smoothed network state information and semantic importance level is found according to a preset strategy lookup table.

[0077] In an optional embodiment, one way to determine the semantic importance level of a speech frame based on its semantic feature vector can be: A lightweight classification network is used to classify semantic feature vectors to obtain the semantic importance level of speech frames.

[0078] The lightweight classification network takes the semantic feature vector of the speech frame sample as input, the semantic importance level of the speech frame sample as output, and aims to make the semantic importance level output by the lightweight classification network approach the semantic importance level label of the speech frame sample. The parameters of the lightweight classification network are updated accordingly.

[0079] The number of parameters in a lightweight classification network can be controlled to within 10k, and the inference latency per frame is less than 0.5ms. Of course, this is only an illustrative example and does not constitute a limitation of this application.

[0080] Optionally, the semantic importance level labels of speech frame samples can be obtained through the following annotation method (which can be denoted as "task error reverse annotation"): For each audio sample in the training corpus, the semantic feature vector of each speech frame of the audio sample is processed by the AI ​​model on the server side. The error between the result of the AI ​​model processing the audio sample and the label of the audio sample is determined as the baseline error.

[0081] For any audio sample A={A1,A2,…,A1} in the training corpus, T}, A t (t=1,2,…,T) represents the t-th speech frame in audio sample A. The semantic feature vectors of each speech frame in audio sample A are input into the AI ​​model to obtain the processing result. The error between the processing result and the label of the audio sample is used as the baseline error corresponding to audio sample A.

[0082] If the AI ​​model is a speech recognition model, then the processing result is transcribed text, and the corresponding audio sample labels are text labels. The corresponding basic error can be the word error rate (WER). The calculation method for WER can use existing methods, which will not be detailed here.

[0083] If the AI ​​model is an intent recognition model, then the processing result of the AI ​​model is the recognized intent, and the corresponding audio sample is labeled with the intent category label. The corresponding basic error can be cross-entropy loss, etc.

[0084] If the AI ​​model is a question-answering model, then the processing result of the AI ​​model is the response text, and the corresponding audio sample labels are the text labels. The corresponding basic error can be cross-entropy loss, etc.

[0085] The audio sample is perturbed multiple times. Each perturbation replaces the semantic feature vector of one speech frame in the audio sample with the perturbed frame (the semantic feature vectors of other speech frames remain unchanged), resulting in a perturbed audio. The perturbed audio is then processed by an AI model. The error between the result of the AI ​​model's processing of the perturbed audio and the label of the audio sample is determined as the perturbed error corresponding to that perturbation.

[0086] For each time step t∈[1,T], the speech frame A t Replace with a perturbation frame (e.g., a silent frame or a zero vector) to obtain perturbation audio A. （-t） Disturb audio A （-t） Each speech frame is input into the AI ​​model to obtain the processing result. The error between the processing result and the label of the audio sample is taken as the speech frame A. t The corresponding error after the disturbance.

[0087] Each perturbation audio is obtained by replacing one speech frame in audio sample A with a perturbation frame, and different perturbation audios are obtained by replacing different speech frames in audio sample A with perturbation frames.

[0088] The semantic importance score of each perturbation is determined based on the post-perturbation error corresponding to each perturbation.

[0089] Optionally, if voice frame A t If the difference between the perturbation error and the reference error (or the error increment) is greater than zero, then the difference between the perturbation error and the reference error is defined as audio frame A. t The semantic importance score; otherwise, audio frame A t Its semantic importance score is 0.

[0090] In audio frame A t When the difference between the perturbation error and the reference error is greater than zero, speech frame At The larger the corresponding error increment, the more important the semantics of the speech frame A t is.

[0091] A percentile division method is adopted to divide all semantic importance scores obtained based on the training corpus into a plurality of ranges, and different ranges correspond to different semantic importance levels; the larger the maximum value within each range, the higher the corresponding semantic importance level.

[0092] As an example, the i-th percentile (denoted as P i ) and the j-th (i<j) percentile (denoted as P j ) are used as range division thresholds, that is, among all semantic importance scores obtained based on the training corpus, i% of the semantic importance scores are less than P i , and j% of the semantic importance scores are less than P j .

[0093] As an example, i=33 and j=75, that is, 33% of the semantic importance scores are less than the threshold P 33 , and 75% of the semantic importance scores are less than the threshold P 75 . On this basis, the semantic importance level of a speech frame whose semantic importance score is less than the threshold P 33 is marked as low, the semantic importance level of a speech frame whose semantic importance score is greater than or equal to the threshold P 33 and less than or equal to the threshold P 75 is marked as medium, and the semantic importance level of a speech frame whose semantic importance score is greater than the threshold P 75 is marked as high.

[0094] Optionally, the semantic importance level label of the speech frame sample can also be obtained through the following annotation method (which can be referred to as "keyword alignment annotation"): Based on the ASR forced alignment result, a frame containing a preset keyword or command word is marked as a high semantic importance level, otherwise it is marked as a low semantic importance level.

[0095] Optionally, the semantic importance level label of the speech frame sample can also be obtained through the following annotation method (which can be referred to as "attention weight distillation"): An AI model is used to process the semantic feature vector of each speech frame of the speech sample to obtain a processing result; attention weights are extracted from the AI model, and a speech frame whose attention weight is greater than a target weight is marked as a high semantic importance level, otherwise it is marked as a low semantic importance level.

[0096] In an optional embodiment, the transmission strategy further includes: configuring a redundancy protection depth for the speech frames, and the redundancy protection depths configured for different speech frames may be the same or different.

[0097] Accordingly, based on the transmission strategy, at least the semantic feature vector is sent to the server, including: At least the semantic feature vector and the semantic feature vector of the number of historical voice frames corresponding to the redundancy protection depth are sent to the server.

[0098] In an optional embodiment, whether it's a semantic feature vector or an acoustic feature vector, the elements in the vector are typically floating-point numbers (e.g., Float32), resulting in a large data volume. To reduce data transmission volume, the semantic and acoustic feature vectors to be transmitted can be quantized and encoded before transmission. In this application, the vector quantization methods for the semantic and acoustic feature vectors can be the same or different. Based on this: If the transmission strategy includes discarding acoustic feature vectors and configuring redundancy protection depth for voice frames, then the first quantized feature obtained by first vector quantization encoding the semantic feature vector of the voice frame and the first quantized feature obtained by first vector quantization encoding the semantic features of historical voice frames are encapsulated into a Real-Time Transport Protocol (RTP) payload packet and sent to the server.

[0099] If the redundancy protection depth configured for the speech frame is 0, the RTP payload packet does not carry the first quantization feature obtained by first vector quantization encoding the semantic features of the historical speech frame.

[0100] If the transmission strategy includes not discarding acoustic feature vectors and configuring redundancy protection depth for speech frames, then the first quantized feature obtained by first vector quantization encoding the semantic feature vector of the speech frame, the second quantized feature obtained by second vector quantization encoding the acoustic feature of the speech frame, and the first quantized feature obtained by first vector quantization encoding the semantic feature of historical speech frames are encapsulated into an RTP payload packet and sent to the server; the data compression efficiency of the first vector quantization encoding is less than that of the second vector quantization encoding.

[0101] Similarly, if the redundancy protection depth configured for the speech frame is 0, the RTP payload packet does not carry the first quantization feature obtained by first vector quantization encoding the semantic features of the historical speech frame.

[0102] The process of performing first vector quantization encoding on the semantic feature vector of the speech frame and the process of performing second vector quantization encoding on the acoustic feature vector of the speech frame can be performed before determining the transmission strategy of the speech frame based on the currently acquired network state information, or after determining the transmission strategy. This application does not make specific limitations on this.

[0103] Optionally, the first vector quantization encoding and the second vector quantization encoding can be the same quantization encoding method with different quantization encoding parameters. For example, the first vector quantization encoding and the second vector quantization encoding can both be product quantization (PQ) encoding, or both can be residual vector quantization (RVQ) encoding.

[0104] Optionally, the first vector quantization encoding and the second vector quantization encoding can be different quantization encoding methods. For example, the first vector quantization encoding can be product quantization encoding, and the second vector quantization encoding can be multi-level residual quantization encoding. By using a layered (semantic layer and acoustic layer) differentiated quantization strategy, an optimal balance can be achieved between compression efficiency and representational accuracy.

[0105] The following sections will explain the Product Quantization (PQ) coding method and the Multi-Level Residual Quantization (RVQ) coding method respectively.

[0106] Product quantization (PQ) encoding divides a high-dimensional vector into multiple low-dimensional sub-vectors, each of which is quantized independently, making it suitable for semantic feature compression.

[0107] Assume the length of the semantic feature vector is D. base , with length D base If the semantic feature vector is divided into M sub-vectors, then the length of each sub-vector is d = D. base / M, each subvector uses an independent codebook (assuming it contains...). For each d-dimensional vector (denoted as the centroid vector for ease of description and distinction), nearest neighbor quantization is performed. For each sub-vector, the nearest (e.g., Euclidean distance closest) centroid vector to it is found in its corresponding codebook, and the sub-vector is represented by the index of that centroid vector in the codebook. Thus, a sub-vector of length D... base The semantic feature vector is compressed into M indices (i.e., the first quantization feature corresponding to the voice frame to be transmitted has M indices), each index occupying B base bits. In other words, the size of the compressed semantic feature vector is: bytes.

[0108] For example, D base =128, M=16, d=8, B base If the value is 8, then the size of the semantic feature vector of each speech frame after vector quantization is 16 bytes.

[0109] Each centroid vector in the codebook represents a typical value within the vector space it covers. The codebook can be constructed as follows: A large-scale speech dataset was collected, and a large number of semantic feature vectors were extracted using the semantic-acoustic decoupling coding method proposed in this application.

[0110] Each semantic feature vector (of length D) base Each semantic feature vector is divided into M sub-vectors, each segment being a sub-vector of length d. Sub-vectors at the same position of different semantic feature vectors constitute a set of sub-vectors. For example, the first sub-vector of all semantic feature vectors constitutes the first set of sub-vectors, the second sub-vector of all semantic feature vectors constitutes the second set of sub-vectors, and the Mth sub-vector of all semantic feature vectors constitutes the Mth set of sub-vectors.

[0111] For the k-th (k=1, 2, 3, ..., M) sub-vector set, cluster the k-th sub-vector set to obtain _ ... These categories, take this The cluster centers of each category constitute the codebook for the k-th subvector. The index of each codebook is [0, ..., ... An integer within this range.

[0112] Multi-stage residual quantization (RVQ) coding is based on vector quantization (VQ). VQ is a classic lossy compression technique in signal processing, and its core idea is: A codebook is pre-trained. As an example, a large-scale speech dataset can be collected, and speech features from each speech data point can be extracted. All extracted speech features are clustered to obtain G clusters. The cluster centers of these G clusters form a codebook, with each codebook indexed as an integer within the range [0, G-1]. When quantizing and encoding the speech feature of a floating-point number, the codebook finds the centroid vector closest to that floating-point number's speech feature, and the index of this centroid vector represents the floating-point number's speech feature.

[0113] The RVQ encoding in this application is based on VQ multi-level quantization encoding. Each level of quantization encoding maintains an independent codebook. The process of establishing each level of codebook is as follows: A large-scale speech dataset is collected, and a large number of acoustic feature vectors are extracted using the semantic-acoustic decoupling coding method proposed in this application. All extracted acoustic feature vectors are then clustered to obtain... These categories, take this The cluster centers of each category constitute the first-level codebook. The index of each codebook is [0, ...]. An integer within this range.

[0114] The subsequent codebook training process for level L (L>1): For the corresponding g-th acoustic feature vector, obtain the centroid vector that is closest to the g-th acoustic feature vector in the codebook of each of the first L-1 levels. Calculate the cumulative residual corresponding to the g-th acoustic feature. The cumulative residual is calculated as follows: obtain the sum of the L-1 centroid vectors, and then subtract the g-th acoustic feature vector from the obtained sum to obtain the cumulative residual corresponding to the g-th acoustic feature.

[0115] Clustering is performed on all the obtained cumulative residuals to obtain These categories, take this The cluster centers of each category constitute the L-th level codebook. The index of each codebook is [0, ..., ... An integer within this range. The number of bits used for each index.

[0116] For example, the process of building the second-level codebook is as follows: For the g-th acoustic feature vector, find the centroid vector closest to the g-th acoustic feature vector in the first-level codebook, and calculate the difference between the g-th acoustic feature vector and the found nearest centroid vector to obtain the residual vector corresponding to the g-th acoustic feature vector; cluster all the obtained residual vectors to obtain... These categories, take this The cluster centers of each category constitute the second-level codebook. The index of each codebook is [0, ...]. An integer within this range.

[0117] The number of levels in RVQ encoding can be flexibly set as needed.

[0118] The RVQ encoding process is explained below: The acoustic feature vector of the speech frame to be transmitted is quantized using the first-level codebook: the centroid vector (denoted as H1) that is closest to the acoustic feature vector of the speech frame to be transmitted is searched in the first-level codebook, and the index of the centroid vector H1 in the first-level codebook is used as the first-level quantization result of the acoustic feature of the speech frame to be transmitted; the difference between the acoustic feature vector of the speech frame to be transmitted and the centroid vector H1 is obtained as the first-level residual vector. The first-level residual vector is quantized using the second-level codebook: the centroid vector (denoted as H2) that is closest to the first-level residual vector is searched in the second-level codebook, and the index of the centroid vector H2 in the second-level codebook is used as the second-level quantization result of the acoustic feature vector of the speech frame to be transmitted; the difference between the first-level residual vector and the centroid vector H2 is obtained as the second-level residual vector; The second-level residual vector is quantized using the third-level codebook: the centroid vector (denoted as H3) that is closest to the second-level residual vector is searched in the third-level codebook, and the index of the centroid vector H3 in the third-level codebook is used as the third-level quantization result of the acoustic feature vector of the speech frame to be transmitted; the difference between the second-level residual vector and the centroid vector H3 is obtained as the third-level residual vector. And so on, until N is obtained. levels The quantization results at each level will be used to quantize these N levels. levels The quantization result of each level is used as the quantization result of the acoustic feature vector of the speech frame to be transmitted (that is, the second quantization feature corresponding to the speech frame to be transmitted is N). levels (Index). The length of the compressed acoustic feature vector is: bytes.

[0119] For example, the length D of the acoustic feature vector enh =512, N levels =8, the index size is B enh =10 bits, so the size of the acoustic feature vector of each speech frame after vector quantization is 10 bytes.

[0120] RVQ coding reconstructs acoustic feature vectors through a stepwise approximation method, making it suitable for high-fidelity acoustic detail representation of acoustic features. Furthermore, it can dynamically discard higher levels based on network status, achieving scalable bitrate transmission.

[0121] Table 2 shows examples of data format and bit estimation based on PQ encoding and RVQ encoding provided in the embodiments of this application.

[0122] Table 2

[0123]

[0124] D b This indicates the length of the semantic feature vector. Since each Float32 value occupies 4 bytes, the size of the semantic feature vector is 128 × 4 B = 512 B.

[0125] D e This indicates the length of the acoustic feature vector. Since each Float32 value occupies 4 bytes, the size of the acoustic feature vector is 512 × 4 B = 2048 B.

[0126] Based on the example shown in Table 2, the bandwidth requirement is estimated (assuming the client samples, processes, and transmits the collected voice at a rate of 50 frames per second, i.e., each voice frame is 20ms in size): The original feature vector data size: (512+2048)B / frame×50fps=2560B / frame×50fps=128kB / s≈1024kbps; After quantization (Base + Enh): 26B / frame × 50fps = 1.3kB / s ≈ 10.4kbps; After quantization (Base only, weak network): 16B / frame × 50fps = 800B / s ≈ 6.4 kbps.

[0127] Compared to the original bandwidth, the hierarchical differentiated quantization strategy based on this application can greatly reduce the bandwidth requirement and improve network performance in weak network conditions.

[0128] When the client performs first vector quantization encoding on the semantic feature vector and second vector quantization encoding on the acoustic feature vector, the server needs to use the same codebook to decode the received RTP payload packet in order to recover the semantic and acoustic feature vectors.

[0129] After receiving the RTP payload packet, the server can proceed according to B. base Bit-width unpacking yields M vector indices (i.e. M first quantization features). Then, centroid vectors can be extracted from their corresponding codebooks. The extracted M centroid vectors are concatenated to obtain the recovered speech feature vector.

[0130] After receiving the RTP payload packet, the server can proceed according to B. enh Unpacking the bit width yields N levels The system uses a level-1 index, then queries the codebook level by level and accumulates the results to obtain the recovered acoustic feature vector.

[0131] During the session establishment phase, the server and client can negotiate the parameters for quantization encoding (including first vector quantization encoding and second vector quantization encoding) through the extended fields of the Session Description Protocol (SDP).

[0132] For the first vector quantization encoding, the parameters may include, but are not limited to: the number of subvectors M, and the bit width B of the subvector index. base The codebook version number, maximum redundancy depth, etc.

[0133] For the second vector quantization encoding, the parameters may include, but are not limited to: RVQ series N. levels The index width B at each level enh Version number of the codebook, etc.

[0134] After the quantization coding parameters are negotiated during the session establishment phase, the payload portion of subsequent RTP payload packets no longer needs to carry these information; it only needs to carry the dynamic information that changes frame by frame to minimize bandwidth overhead.

[0135] In an optional embodiment, the RTP payload packet includes two parts: an RTP header and an RTP payload. The payload includes: A payload header encapsulates at least the following dynamic fields: a layer flag indicating whether an acoustic feature vector is included, a semantic flag indicating the semantic importance level, a redundancy count indicating the redundancy protection depth, and a redundancy coding format. These are all dynamic information that changes frame by frame.

[0136] Redundancy coding format refers to the method of encoding the first quantization features of a number of historical speech frames corresponding to the redundancy protection depth. Redundancy coding format can include, but is not limited to, any of the following: Full Copy: This method carries a complete copy of the data in the RTP payload packet, specifically copying the first quantization features of historical speech frames into redundant blocks of the payload. This method is simple to implement and has low latency, but its bandwidth utilization is not high.

[0137] Differential XOR coding (DeltaXOR) + Run-Length Encoding (RLE): The first quantization feature of each historical speech frame is bitwise XORed with the first quantization feature of a reference frame (e.g., the speech frame to be transmitted, or an adjacent historical speech frame), and then the XOR result is run-length encoded. RLE is a classic lossless compression algorithm specifically designed to compress data sequences containing a large number of consecutive repeating values. It represents the data sequence in a (value, repetition count) format. For example, a sequence 0, 0, 0, 0, 5, 0, 0, 12, encoded using RLE, will result in (0, 4), (5, 1), (0, 2), (12, 1), effectively reducing the data volume. In this application, when the first quantization features of two speech frames are not significantly different, the XOR result will contain a large number of 0s and a small number of 1s, providing excellent conditions for RLE compression. Compared to full copy, difference-based coding, although computationally more complex, can significantly reduce the number of bytes occupied by redundant blocks, thereby saving more bandwidth, improving bandwidth utilization, and enhancing the overall efficiency of weak network countermeasures at the same redundancy depth.

[0138] Differential quantization + entropy coding: The difference between the first quantization feature of the historical speech frame and the first quantization feature of the speech frame to be transmitted is calculated to obtain the difference feature. This difference feature is then subjected to a second round of quantization (for example, coarser rounding is performed on the difference feature, such as quantizing [1,1,-1] to [1,1,0], which has a smaller value range and more concentrated distribution than [1,1,-1]) to reduce its numerical range and precision, resulting in the residual feature. The residual feature is then subjected to entropy coding (such as Huffman coding or arithmetic coding). Compared to "DeltaXOR + RLE", "differential quantization + entropy coding" can achieve a higher compression ratio and minimize the bandwidth occupied by redundant data.

[0139] "DeltaXOR + RLE" and "differential quantization + entropy coding" can be collectively referred to as differential coding.

[0140] Table 3 shows an example of the composition of the Payload Header provided in the embodiments of this application: Table 3

[0141]

[0142] The "Plaintext" section is a formatted ASCII illustration used to visually represent the bit layout of the RTP payload header.

[0143] The two rows of numbers in the diagram (0 1 2 3 ... and 0123456789 ...) represent the bit position indexes, used to precisely indicate the start and end positions of each field below in the 32-bit header.

[0144] Vertical lines and symbols: The vertical line |, plus sign +, and hyphen - in the diagram form a textual "table" or "diagram" that divides the 32-bit header space into several regions, corresponding to the actual fields such as V, L, S, R, K, and Reserved.

[0145] In the example shown in Table 3, the first row of numbers (0 1 2 3) and the second row of numbers (0123456789...) together illustrate that this is a space of 32 bits (4 bytes) from the 0th bit to the 31st bit.

[0146] |VV| represents the Version field, occupying positions 30-31 (2 bits). In this example, the version number is 01.

[0147] |L| represents the LayerFlag field, occupying the 29th bit (1 bit). In this example, LayerFlag=0 means sending the semantic feature vector but not the acoustic feature vector; LayerFlag=1 means sending both the semantic feature vector and the acoustic feature vector.

[0148] |SS| represents the SemFlag field, occupying positions 28-27 (2 bits). In this example, SemFlag=00 indicates a low semantic importance level, SemFlag=01 indicates a medium semantic importance level, SemFlag=10 indicates a high semantic importance level, and SemFlag=11 is a reserved extension.

[0149] |R| represents the RedFormat field, occupying the 26th bit (1 bit). In this example, RedFormat=0 indicates that the redundancy encoding method is FullCopy, and RedFormat=1 indicates that the redundancy encoding method is DeltaXOR + RLE.

[0150] |K| represents the RedundancyCount field, which occupies bits 25-18 (8 bits). In this example, the RedundancyCount value for redundancy depth ranges from [0,255], preferably [0,8].

[0151] |Reserved| indicates a reserved field, occupying bits 17-0 (18 bits). It can be used for validation or expansion.

[0152] As shown in Table 4, this is an example of an RTP payload packet with a redundancy depth of 3, a redundancy coding method of FullCopy, and LayerFlag=1 provided in the embodiments of this application.

[0153] Table 4

[0154]

[0155] Wherein, Base Payload represents the region carrying the first quantized feature of the semantic feature vector of the current speech frame, bit-packed means that the first quantized feature is packed in a continuous bit-packed format (i.e., the indices are arranged in order to form a continuous bit sequence, i.e., the index sequence), and big-endian means big-endian byte order, i.e., the most significant byte is stored at the lowest memory address (or appears first in the data stream), and the least significant byte is stored at the highest memory address (or appears last in the data stream). Enhancement Payload represents the region that carries the second quantization feature of the acoustic feature vector of the current speech frame; The redundancy entry represents the region carrying the first quantization feature (i.e., the Base index) of historical speech frames. Redundancy entry[0] corresponds to the speech frame with RTP sequence number "seq-1", and Redundancy entry[i] corresponds to the speech frame with RTP sequence number "seq-(i+1)". seq is the RTP sequence number corresponding to the current speech frame. The server can deduce the sequence number of historical speech frames based on the sequence number of the RTP payload packet of the current speech frame and the entry index i.

[0156] The server performs bit-unpacking based on the quantization parameters negotiated with the client's SDP, restores the index sequence, and then uses the index to look up the code table for decoding.

[0157] Optionally, if the RTP payload does not contain a second quantization feature carrying the acoustic feature vector of the current speech frame, the server can replace the second quantization feature with a specific vector (a predefined vector, a context-generated vector, or an all-zero vector) and mark the segment as missing when inputting it into the AI ​​model so that the AI ​​model can adjust its strategy (for example, an AI model for sentiment analysis may relax the sentiment analysis confidence).

[0158] Corresponding to the semantic feature vector, after decoding the individual sub-vectors, the server concatenates them sequentially to obtain the recovered semantic feature vector. A timestamp can also be added to the recovered semantic feature vector to represent the start and end times of the corresponding speech frame. This is crucial for aligning text and audio or processing instructions with time information. If the acoustic feature vector is decoded, the recovered semantic feature vector and the acoustic feature vector can be fused (e.g., added) to obtain a fused feature vector for the speech frame, which is then used as input to the AI ​​model. To enable the AI ​​model to distinguish between real and padded feature vectors, a mask can be generated to differentiate between them, preventing the AI ​​model from misinterpreting padded data.

[0159] Corresponding to the method embodiments, this application also provides an electronic device. (See reference...) Figure 4 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of this application. The electronic device in these embodiments may be a terminal device (e.g., an in-vehicle infotainment system, a large-screen device, a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.). Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0160] like Figure 4As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. When the electronic device is powered on, the RAM 403 also stores various programs and data required for the operation of the electronic device. The processing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0161] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, memory cards, hard drives, etc.; and communication devices 409. Communication device 409 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0162] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the voice transmission methods provided in this application.

[0163] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the voice transmission methods provided in this application.

[0164] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0165] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0166] In the above embodiments, the functionality can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented entirely or partially as a computer program product. Those skilled in the art can use different methods to implement the described functions for each specific solution, but such implementation should not be considered beyond the scope of this application.

[0167] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0168] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0169] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A voice transmission method, characterized in that, Applied to a client, the method includes: The speech frame to be transmitted is semantically and acoustically decoupled and encoded to obtain the semantic feature vector and acoustic feature vector of the speech frame. Based at least on the currently acquired network status information, the transmission strategy of the voice frame is determined; the transmission strategy includes: whether to discard the acoustic feature vector, and the redundancy protection depth configured for the voice frame; different voice frames may have the same or different redundancy protection depths configured. If the transmission strategy includes discarding the acoustic feature vector and configuring a redundancy protection depth for the speech frame, then the first quantization feature obtained by performing a first vector quantization encoding on the semantic feature vector of the speech frame and the first quantization feature obtained by performing a first vector quantization encoding on the semantic features of the historical speech frames corresponding to the redundancy protection depth are encapsulated into a Real-Time Transport Protocol (RTP) payload packet and sent to the server. If the transmission strategy includes not discarding the acoustic feature vector and configuring a redundancy protection depth for the speech frame, then the first quantized feature obtained by performing a first vector quantization encoding on the semantic feature vector of the speech frame, the second quantized feature obtained by performing a second vector quantization encoding on the acoustic feature of the speech frame, and the first quantized feature obtained by performing a first vector quantization encoding on the semantic feature of the historical speech frame are encapsulated into an RTP payload packet and sent to the server; the data compression efficiency of the first vector quantization encoding is lower than the data compression efficiency of the second vector quantization encoding.

2. The method according to claim 1, characterized in that, Performing semantic-acoustic decoupling coding on the speech frame includes: The speech frame is semantically and acoustically decoupled and encoded by a semantic-acoustic decoupling encoder to obtain the semantic feature vector and acoustic feature vector of the speech frame. The semantic-acoustic decoupling encoder is trained in the following way: The semantic-acoustic decoupling encoder is used to perform speech-acoustic decoupling encoding on each speech frame of the speech sample to obtain the semantic feature vector and acoustic feature vector of each speech frame. The semantic prediction result of the semantic sample is obtained by the parsing module based on the semantic feature vector of each speech frame; the reconstructed spectrogram of the speech sample is obtained by the reconstruction module based on the acoustic feature vector and semantic feature vector of each speech frame. The parameters of the semantic-acoustic decoupling encoder, the parsing module, and the reconstruction module are updated with the goal of making the semantic prediction result of the speech sample close to the semantic label of the speech sample, reconstructing the spectrogram close to the spectrogram of the speech sample, and making the semantic feature vector of each speech frame far away from the acoustic feature vector of the speech frame.

3. The method according to claim 1, characterized in that, Also includes: The semantic importance level of the speech frame is determined based on the semantic feature vector. Determining the transmission strategy for the voice frame based at least on the currently acquired network status information includes: Based on the currently acquired network status information and the semantic importance level, the transmission strategy for the voice frame is determined.

4. The method according to claim 3, characterized in that, Determining the semantic importance level of the speech frame based on the semantic feature vector includes: The semantic feature vectors are classified using a lightweight classification network to obtain the semantic importance level of the speech frames; The lightweight classification network takes the semantic feature vector of the speech frame sample as input, the semantic importance level of the speech frame sample as output, and aims to update the parameters of the lightweight classification network so that the semantic importance level output by the lightweight classification network approaches the semantic importance level label of the speech frame sample.

5. The method according to claim 4, characterized in that, The semantic importance level labels of the speech frame samples are generated in the following way: For each audio sample in the training corpus, the semantic feature vector of each speech frame of the audio sample is processed by the AI ​​model on the server side, and the error between the result of the AI ​​model processing the audio sample and the label of the audio sample is determined as the baseline error. The audio sample is perturbed multiple times. Each perturbation replaces the semantic feature vector of a speech frame in the audio sample with the perturbed frame to obtain a perturbed audio. The perturbed audio is then processed by the AI ​​model. The error between the result of the AI ​​model's processing of the perturbed audio and the label of the audio sample is determined as the perturbed error corresponding to that perturbation. The semantic importance score of each perturbation is determined based on the post-perturbation error corresponding to each perturbation. The percentile division method is used to divide all semantic importance scores obtained from the training corpus into multiple ranges, with different ranges corresponding to different semantic importance levels; the larger the maximum value in each range, the higher the corresponding semantic importance level.

6. The method according to claim 3, characterized in that, Based on the currently acquired network status information and the semantic importance level, the transmission strategy for the voice frame is determined, including: According to the preset strategy lookup table, the transmission strategy corresponding to the currently acquired network status information and the semantic importance level is searched and used as the transmission strategy for the voice frame.

7. The method according to claim 6, characterized in that, According to a preset strategy lookup table, the transmission strategy corresponding to the currently acquired network state information and the semantic importance level is searched and used as the transmission strategy for the voice frame, including: The network state information is smoothed to obtain smoothed network state information; wherein, the smoothed network state information is the weighted sum of the currently acquired network state information and the previously acquired smoothed network state information; According to the preset strategy lookup table, the transmission strategy corresponding to the smoothed network state information and the semantic importance level is used as the transmission strategy for the voice frame.

8. The method according to claim 1, characterized in that, The load portion of the RTP payload includes: A header, the header encapsulating at least the following dynamic fields: a layer flag indicating whether the acoustic feature vector is included, a semantic flag indicating the semantic importance level, a redundancy count indicating the redundancy protection depth, and a redundancy encoding format.

9. The method according to claim 8, characterized in that, In the RTP payload packet, the first quantization features of the number of historical voice frames corresponding to the redundancy protection depth are carried in the payload of the RTP payload packet in the form of a complete copy or differential coding.

10. The method according to claim 1, characterized in that, Also includes: During the session establishment phase with the server, the parameters required for the first vector quantization encoding and the second vector quantization encoding are negotiated with the server through the extended fields of the Session Description Protocol (SDP).

11. An electronic device, characterized in that, The electronic device includes at least one processor and a memory connected to the processor; wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the voice transmission method as described in any one of claims 1 to 10.

12. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the voice transmission method as described in any one of claims 1 to 10.

13. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the voice transmission method as described in any one of claims 1 to 10.

Images