Voice processing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a voice processing method and device, equipment and a storage medium. The method comprises the following steps: acquiring a voice feature sequence corresponding to a voice sample, wherein voice features in the voice feature sequence correspond to voice frames in the voice sample; and for a target voice feature in the voice feature sequence, generating one or more post-order voice marks corresponding to the one or more post-order voice features based on the target voice feature and the one or more pre-order voice features by using a voice coding model. A speech encoding model is trained based on the one or more post-order speech markers. The training mode can enable the speech coding model to learn high-quality speech representation.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and more particularly to methods, apparatuses, devices, and computer-readable storage media for speech processing. Background Technology

[0002] With the development of machine learning technology, machine learning models can now be used to perform tasks in various application environments. Speech models are used to process input speech information (e.g., speech recognition or speech editing). To improve the performance of speech models, pre-training is often required. However, current pre-training methods for speech models have some problems, resulting in poor performance of speech models in some speech processing tasks or scenarios, affecting the results of speech processing. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for speech processing is provided. The method includes: acquiring a speech feature sequence corresponding to a speech sample, wherein the speech features in the speech feature sequence correspond to speech frames in the speech sample; for a target speech feature in the speech feature sequence, using a speech coding model, generating one or more subsequent speech tags corresponding to one or more subsequent speech features, based on the target speech feature and one or more preceding speech features, wherein the one or more preceding speech features indicate speech features in the speech feature sequence preceding the target speech feature, and the one or more subsequent speech features indicate speech features in the speech feature sequence following the target speech feature; and training the speech coding model based on the one or more subsequent speech tags.

[0004] In a second aspect of this disclosure, an apparatus for speech processing is provided. The apparatus includes: an acquisition module configured to acquire a speech feature sequence corresponding to a speech sample, wherein the speech features in the speech feature sequence correspond to speech frames in the speech sample; a generation module configured to, for a target speech feature in the speech feature sequence, generate one or more subsequent speech tags corresponding to one or more subsequent speech features based on the target speech feature and one or more preceding speech features using a speech coding model, wherein the one or more preceding speech features indicate speech features in the speech feature sequence preceding the target speech feature, and the one or more subsequent speech features indicate speech features in the speech feature sequence following the target speech feature; and a training module configured to train the speech coding model based on the one or more subsequent speech tags.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0010] Figure 2 An architecture diagram of an example model training system according to some embodiments of the present disclosure is shown;

[0011] Figure 3 An architecture diagram of another example of a model training system according to some embodiments of the present disclosure is shown;

[0012] Figure 4 A flowchart of a speech processing procedure according to some embodiments of the present disclosure is shown;

[0013] Figure 5 A block diagram of an apparatus for speech processing according to some embodiments of the present disclosure is shown; and

[0014] Figure 6 A block diagram of an apparatus capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0015] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0016] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0017] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0018] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0019] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0020] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0021] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0022] In this document, unless explicitly stated otherwise, performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.

[0023] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0024] As used in this paper, the term "model" refers to a system that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. In this paper, "model" may also be referred to as a "machine learning model," a "machine learning network," or simply a "network," and these terms are used interchangeably. A model can also include different types of processing units or networks.

[0025] As used herein, a “unit,” “operation unit,” or “subunit” can consist of any suitable machine learning model or network. As used herein, a set of elements or similar expressions can include one or more such elements. For example, “a set of convolutional units” can include one or more convolutional units.

[0026] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. For example... Figure 1 As shown, model 130-1 with pre-training parameter values ​​and model 130-2 with post-training parameter values ​​can be collectively referred to as model 130 or individually. Model 130 can be implemented or included in electronic device 140 and / or electronic device 150.

[0027] exist Figure 1 In environment 100, it is desirable to train and use a machine learning model (i.e., model 130) configured for various application environments. For example, in the case of a speech recognition model, text corresponding to the speech can be generated based on the user's input speech to be processed.

[0028] like Figure 1 As shown, environment 100 includes electronic device 140 and electronic device 150. Electronic device 140 may contain a voice processing system, and electronic device 150 may contain a model application system. Figure 1The upper part illustrates the model training phase, and the lower part illustrates the model application phase. Before training, the parameter values ​​of model 130 can have initial values ​​or pre-trained parameter values ​​obtained through a pre-training process. Model 130-1 can be trained via forward and backward propagation, during which the parameter values ​​of model 130-1 can be updated and adjusted. After training, model 130-2 is obtained. Model training can further include pre-training and fine-tuning. After pre-training, model 130-1 has generalization ability, such as the ability to represent speech. Then, in the fine-tuning phase, the pre-trained model 130-1 is fine-tuned for the downstream speech processing task. At this point, the parameter values ​​of model 130-2 have been updated, and based on the updated parameter values, model 130-2 can be used to implement speech processing tasks, such as speech recognition tasks, in the model application phase.

[0029] During the fine-tuning phase of model training, model 130 can be trained using a model training system based on a training sample set 110 comprising multiple training samples 112. Each training sample 112 can involve a binary format. For example, for a speech recognition task, training sample 112 can include training input 120 and training output from the speech recognition task. The training input for the speech recognition task can, for example, include training audio and the corresponding text. Training sample 112, including model input 120 and model output 122, can be used to train model 130. Specifically, the training process can be performed iteratively using a large number of training samples. After training is complete, model 130 can include knowledge about the task to be processed. During the model application phase, model 130 (at this point, model 130 has the trained parameter values) can be used to perform the corresponding task. For example, model input 142 from the speech recognition task can be received, and a corresponding model output 144 can be output.

[0030] exist Figure 1 In this context, electronic devices 140 and 150 can include any computing system with computing capabilities, such as various computing devices / systems, terminal devices, servers, etc. Terminal devices can involve any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. Servers include, but are not limited to, mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0031] It should be understood that Figure 1The components and arrangements shown in environment 100 are merely examples, and a computing system suitable for implementing the exemplary implementations described in this disclosure may include one or more different components, other components, and / or different arrangements. Implementations of this disclosure are not limited in this respect.

[0032] As briefly mentioned earlier, speech models are used to process user-input speech information. In various speech processing tasks or scenarios, speech models are needed to encode and represent speech.

[0033] In some speech processing scenarios, such as streaming speech processing, causal speech models are required. Traditionally, speech models are trained primarily using self-supervised techniques based on masking prediction mechanisms. This training method requires utilizing the contextual content of the input speech for prediction. Speech models obtained in this way lack causality and therefore perform poorly in tasks such as streaming speech processing. Therefore, it is desirable to obtain causal speech models during the pre-training stage. Such causal speech models can be used in streaming speech processing, such as streaming speech recognition.

[0034] Therefore, embodiments of this disclosure provide a scheme for speech processing. According to various embodiments of this disclosure, a speech feature sequence corresponding to a speech sample is obtained, where the speech features in the speech feature sequence correspond to speech frames in the speech sample. For a target speech feature in the speech feature sequence, a speech coding model is used to generate one or more subsequent speech tags, each corresponding to one or more subsequent speech features, based on the target speech feature and one or more preceding speech features. The one or more preceding speech features indicate speech features in the speech feature sequence preceding the target speech feature, and the one or more subsequent speech features indicate speech features in the speech feature sequence following the target speech feature. The speech coding model is trained based on the one or more subsequent speech tags.

[0035] By determining the subsequent speech features corresponding to the target speech features based on the target speech features and their corresponding preceding speech features, a subsequent speech tag is determined. This training method enables the speech coding model to learn high-quality speech representations. This speech coding model can be used for various speech processing tasks, especially streaming speech processing tasks.

[0036] Figure 2 An architectural diagram of an example model training system according to some embodiments of the present disclosure is shown. Figure 2 As shown, the model training system can be implemented or included in the electronic device 140. The electronic device 140 is used to train the speech coding model 240 based on the speech samples 210 provided by the user, so as to update the parameters of the speech coding model 240.

[0037] In some embodiments, speech sample 210 is speech data containing human voice track information, and speech sample 210 is used to train speech coding model 240. The speech data can be existing data on the Internet that includes human voice track information and reverberation, or speech data that only includes human voice track information recorded in a high-standard recording environment.

[0038] If the voice data is data obtained from the Internet that includes reverberation and vocal track information, then voice sample 210 (i.e., vocal track information) needs to be extracted from the voice data. In some embodiments, a sound source separation module can be used to separate the voice data to obtain the vocal track information in the music data. The music sample may include multiple vocal track voice signals.

[0039] In some embodiments, the electronic device 140 acquires a speech feature sequence 230 corresponding to the speech sample 210. The speech feature sequence 230 includes multiple speech features, and each speech feature corresponds to multiple speech frames in the speech sample 210. In some embodiments, a speech frame indicates a segment of a speech signal in the speech sample 210.

[0040] In some embodiments, the speech sample 210 can be a time-domain signal. After acquiring the speech sample 210, the electronic device 140 first determines the time-frequency representation 220 corresponding to the speech sample 210. Subsequently, a sampling operation is performed on the obtained time-frequency representation 220 to generate a speech feature sequence 230. The speech feature sequence 230 includes speech features corresponding to multiple speech frames, respectively. Figure 2 As shown, the speech feature sequence 230 includes at least the first speech feature 231-1, the second speech feature 231-2, the third speech feature 231-3, the fourth speech feature 231-4, the fifth speech feature 231-5, the sixth speech feature 231-6, and the seventh speech feature 231-7, which can be referred to individually or collectively as speech feature 231. It should be noted that... Figure 2 The number of speech features 231 shown is exemplary. Speech feature sequence 230 may include any suitable number of speech features. Embodiments of this disclosure are not limited in this respect. The time-frequency representation 220 at least indicates the intensity of speech sample 210 changing over time at different frequencies. Exemplarily, the time-frequency representation 220 corresponding to speech sample 210 can be determined by signal processing methods (e.g., Mel-frequency cepstral coefficients).

[0041] In some embodiments, to reduce data volume and computational complexity, the sampling frequency of the time-frequency representation 220 can be reduced. A speech feature sequence 230 is generated by downsampling. For example, for a speech sample S, after time-frequency conversion, a time-frequency representation X = (x1, x2, x3, x4, ..., x6) corresponding to the speech sample S, comprising T speech frames, can be obtained.T Subsequently, the time-frequency representation 220 of the speech sample S is downsampled to obtain a speech feature sequence I = (i1, i2, i3, i4, ..., i...) of length L. L The length l of the speech feature sequence I is determined by the downsampling frequency, and the length L is no greater than the length T of the time-frequency representation 220.

[0042] For the target speech feature in the speech feature sequence 230, the electronic device 140 uses the speech coding model 240 to generate one or more subsequent speech tags corresponding to one or more subsequent speech features, based on the target speech feature and one or more preceding speech features. The one or more preceding speech features indicate speech features in the speech feature sequence 230 that precede the target speech feature, and the one or more subsequent speech features indicate speech features in the speech feature sequence 230 that follow the target speech feature.

[0043] In some embodiments, the speech feature sequences 230 corresponding to each speech frame in the speech sample 210 acquired by the electronic device 140 are provided to the speech coding model 240 to train the speech coding model 240. In some embodiments, the speech coding model 240 may be input with the complete speech feature sequence 230, or with a speech feature sequence 230 containing only some of the speech features corresponding to the speech frames, to achieve a downsampling effect.

[0044] The target speech feature can be any speech feature in the speech feature sequence 230. Taking streaming speech processing as an example, in a streaming speech processing scenario (e.g., streaming speech recognition), if a certain speech frame needs to be processed, the speech coding model can only encode the preceding speech frame. Therefore, the speech coding model needs to be causal. The pre-trained speech coding model according to embodiments of this disclosure possesses this causality and is thus adaptable to downstream streaming speech processing tasks, such as streaming speech recognition tasks. This training method enables the speech coding model to learn high-quality speech representations.

[0045] In some embodiments, the speech feature sequence 230 is processed using the speech coding model 240. First, the position of the target speech feature in the speech feature sequence 230 is determined. Based on the relative positions of each speech feature in the speech feature sequence 230, the preceding speech feature is determined. In some embodiments, determining the target speech feature can first involve determining the speech features corresponding to all speech frames in the speech sample 210, thereby constructing the speech feature sequence 230. The target speech feature is then determined from the constructed speech feature sequence 230. Alternatively or additionally, the target speech frame can be determined first, followed by the target feature sequence corresponding to the target speech frame. Based on the relative positions of other speech frames and the target speech frame, the preceding and following speech frames are determined. Subsequently, the preceding and following speech features corresponding to the preceding and following speech frames are determined, respectively.

[0046] In some embodiments, the number of preceding speech features can be one or more. When encoding the target speech feature to obtain the encoded representation corresponding to the target speech feature, it can be based on all preceding speech features of the target speech feature, or it can be based on a subset of the speech features of the target speech feature. For example, the encoded representation corresponding to the target speech feature can be obtained based on the top P speech features closest to the target speech feature, where P is a positive integer. In this way, computational resources can be saved when the speech coding model is used for downstream speech processing tasks.

[0047] In some embodiments, the electronic device 140 generates a training sequence from the obtained preceding speech features and target speech features. The training sequence is provided to a speech coding model 240 to obtain a speech coding representation generated by the speech coding model 240. The speech coding representation output by the speech coding model 240 is processed using prediction head modules to generate subsequent speech tokens (discrete tokens). The number of prediction head modules is consistent with the number of generated subsequent speech features. The model parameters of the prediction head modules can be updated during the training of the speech coding model 240. Figure 2 As shown, the prediction head module includes prediction head module 250-1, prediction head module 250-2, and prediction head module 250-3, which can be individually or collectively referred to as prediction head module 250. In some embodiments, prediction head modules 250 with different parameters are used to process the speech coding representations corresponding to different target speech features. Exemplarily, prediction head module 250 may include a softmax head and any other suitable processing layer (e.g., a mapping layer).

[0048] In some embodiments, the number of subsequent speech features can be one or more, and correspondingly, the number of predicted subsequent speech tags can be one or more. During the process of encoding and predicting subsequent speech tags corresponding to subsequent speech features using the speech coding model 240, due to the continuity and stationarity of the speech signal, it does not change drastically in a short period of time. This may result in a situation where the speech features of a subsequent speech frame have a high similarity to the speech features of the current speech frame. In this case, if only the speech tag of a single subsequent speech frame is predicted, the speech self-supervised learning task will be too simplistic, preventing the speech coding model from learning a high-quality speech representation. Considering the short-term stationarity of the speech signal at the microscopic level, in some embodiments, for each speech frame, speech tags for multiple subsequent speech frames are predicted to enable the model to learn a high-quality speech representation.

[0049] refer to Figure 2 Describe an example. Figure 2 The diagram shows the first speech marker 260-1, the second speech marker 260-2, the third speech marker 260-3, the fourth speech marker 260-4, the fifth speech marker 260-5, the sixth speech marker 260-6, and the seventh speech marker 260-7, which can be individually or collectively referred to as speech marker 260. It should be noted that... Figure 2 The number of speech markers 260 is exemplary, and the number of speech markers included in the speech marker sequence 260 is not limited here. For example, if speech feature 231-2 in the speech feature sequence 230 is determined as the target speech feature, the preceding speech feature is the first speech feature 231-1. The following speech features are the third speech feature 231-3 and the fourth speech feature 231-4. The speech coding model 240 generates a speech coding representation corresponding to the second speech feature 231-2 based on the first speech feature 231-1 and the second speech feature 231-2. The speech coding representation corresponding to the second speech feature 231-2 is provided to prediction head modules 250-1, 250-2, and 250-3 to generate subsequent speech markers 260-3 (corresponding to the third speech feature 231-3), 260-4 (corresponding to the fourth speech feature 231-4), and 260-5 (corresponding to the fourth speech feature 231-5). That is, in this example, for a certain speech feature, three subsequent speech tags are predicted. However, it should be understood that this is merely exemplary, and the number of predicted subsequent speech tags is not limited in the embodiments of this disclosure.

[0050] In some embodiments, the electronic device 140 trains a speech coding model 240 based on one or more subsequent speech tags.

[0051] In some embodiments, the electronic device 140 encodes speech samples 210 using a discrete encoder 290 to generate reference speech tags 280 corresponding to each subsequent speech feature. The parameters of the discrete encoder 290 can be pre-trained or updated during the training of the speech encoder. For example, a time-frequency representation 220 corresponding to the speech sample 210 is first determined. The speech encoder generates the reference speech tags 280 based on the time-frequency representation 220. Figure 2 As shown, the reference speech markers include the first reference speech marker 280-1, the second reference speech marker 280-2, the third reference speech marker 280-3, the fourth reference speech marker 280-4, the fifth reference speech marker 280-5, the sixth reference speech marker 280-6, and the seventh reference speech marker 280-7, which can be referred to individually or collectively as reference speech marker 280. It should be noted that... Figure 2 The number of reference speech markers 280 is exemplary, and the number of speech features included in the reference speech markers 280 is not limited here.

[0052] In some embodiments, the electronic device 140 determines training loss components by comparing one or more subsequent speech tags and one or more reference speech tags 280. Based on the training loss components, the model parameters of the speech coding model 240 are updated.

[0053] In some embodiments, the training loss component is a training loss 271 determined by the difference between a reference speech tag 280 generated from the speech feature sequence 230 of a speech frame in a speech sample 210 and subsequent speech tags. For example, for each subsequent speech tag, a corresponding subsequent speech feature is determined, and then a reference speech tag 280 corresponding to that subsequent speech feature is determined. The difference between each subsequent speech tag and its corresponding reference speech tag 280 is used as the training loss component.

[0054] In some embodiments, since the discrete encoder 290 typically includes a downsampling unit, the sequence length (i.e., the number of speech tags) of the speech tag sequence generated by the speech coding model 240 may differ from the sequence length of the reference speech tag sequence (i.e., the number of reference speech tags 280). Therefore, the sequence length of either the reference speech tag sequence or the sequence length of the speech tag sequence generated by the speech coding model 240 can be adjusted to ensure that their sequence lengths are consistent. For example, if the sequence length of the reference speech tag sequence is adjusted, the sequence length information of the corresponding subsequent speech tags is first determined. Based on the obtained sequence length information, the sequence length of the reference speech tags 280 is adjusted.

[0055] Figure 3An architectural diagram of another example of a model training system according to some embodiments of this disclosure is shown. Figure 3 As shown, the discrete encoder 290 can generate reference speech tags 280 based on the speech feature sequence 230, thereby ensuring that the sequence length of the reference speech tag sequence is consistent with the sequence length of the speech tag sequence generated by the speech coding model 240. The speech coding model 240 is trained based on the differences between one or more subsequent speech tags and one or more reference speech tags 280. In this embodiment, there is no need to additionally adjust the sequence length of the reference speech tags 280.

[0056] continue Figure 2 The electronic device 140 determines the training loss 271 corresponding to the speech feature sequence 230 (i.e., speech sample 210) based on the sum of the training loss components corresponding to each subsequent speech tag in the speech feature sequence 230. For example, the position of the target speech feature in the speech feature sequence 230 is l. The speech coding model 240 generates N subsequent speech tags {k}. l+1 ,k l+2 ,k l+3 ,k l+4 ,…,k l+n The training loss component at position l. The calculation formula is as follows:

[0057]

[0058] Where θ represents the parameters of speech coding model 240.

[0059] Training loss for 210 speech samples The calculation formula is as follows:

[0060]

[0061] The electronic device 140 updates the model parameters of the speech coding model 240 based on the obtained training loss 271. In some embodiments, the electronic device 140 may update the model parameters of the speech coding model 240 based on the training loss 271 corresponding to multiple speech samples 210 respectively.

[0062] This training method enables the speech coding model to learn high-quality speech representations. This speech coding model can be used for a variety of speech processing tasks, especially streaming speech processing tasks.

[0063] In some embodiments, the parameters of the trained speech coding model 240 and high-quality reference audio can be used to... Figure 2The model training system shown performs supervised fine-tuning of the speech coding model 240. This further improves the performance of the speech coding model 240. Compared to self-supervised training, there is no need to change the structure of the model training system and the speech coding model 240. Furthermore, since the model training system is unsupervised, there is no need to set text labels. This reduces the human resource costs during the training process.

[0064] Figure 4 A flowchart of a speech processing procedure 400 according to some embodiments of the present disclosure is shown. Procedure 400 may be implemented at an electronic device 140.

[0065] In box 410, the speech feature sequence corresponding to the speech sample is obtained, and the speech features in the speech feature sequence correspond to the speech frames in the speech sample.

[0066] In some embodiments, obtaining a speech feature sequence corresponding to a speech sample includes: determining a time-frequency representation corresponding to the speech sample, the time-frequency representation indicating at least the intensity of the speech sample changing over time at different frequencies; and generating a speech feature sequence by performing downsampling on the time-frequency representation.

[0067] In box 420, for the target speech feature in the speech feature sequence, a speech coding model is used to generate one or more subsequent speech tags corresponding to one or more subsequent speech features, based on the target speech feature and one or more preceding speech features. The one or more preceding speech features include speech features in the speech feature sequence that precede the target speech feature, and the one or more subsequent speech features include speech features in the speech feature sequence that follow the target speech feature.

[0068] In some embodiments, generating subsequent speech tags corresponding to one or more subsequent speech features includes: generating a speech coding representation based on the target speech features and one or more preceding speech features using a speech coding model; and generating one or more subsequent speech tags based on the speech coding representation.

[0069] In some embodiments, generating a speech coding representation using a speech coding model includes: generating a training sequence based on target speech features and one or more preceding speech features; and generating a speech coding representation based on the training sequence using a speech coding model.

[0070] In box 430, a speech coding model is trained based on one or more subsequent speech tags.

[0071] In some embodiments, training a speech coding model based on one or more subsequent speech tags includes: encoding speech samples using a discrete encoder to generate one or more reference speech tags corresponding to one or more subsequent speech features; determining training loss components by comparing one or more subsequent speech tags with one or more reference speech tags; and updating the model parameters of the speech coding model based on the training loss components.

[0072] In some embodiments, determining the training loss component includes: for a given subsequent speech tag in one or more subsequent speech tags, determining a given reference speech tag corresponding to the given subsequent speech tag from one or more reference speech tags; and determining the training loss component corresponding to the given subsequent speech tag based on the difference between the given subsequent speech tag and the given reference speech tag.

[0073] In some embodiments, process 400 further includes: determining sequence length information of a speech tag sequence corresponding to a speech feature sequence obtained using a speech coding model; and adjusting the sequence length of a reference speech tag sequence corresponding to a speech feature sequence obtained using a discrete encoder based on the sequence length information, wherein the adjusted reference speech tag sequence includes one or more reference speech tags.

[0074] In some embodiments, multiple speech features in the speech feature sequence are respectively used as target speech features to obtain training loss components, and updating the model parameters of the speech coding model based on the training loss includes: determining the training loss based on the sum of the training loss components obtained for the multiple speech features respectively; and updating the model parameters of the speech coding model based on the training loss.

[0075] In some embodiments, training a speech coding model based on one or more subsequent speech tags includes: encoding a sequence of speech features using a trained discrete encoder to generate one or more reference speech tags corresponding to one or more subsequent speech features; and training the speech coding model based on the differences between the one or more subsequent speech tags and the one or more reference speech tags.

[0076] Figure 5 A schematic structural block diagram of a speech processing apparatus 500 according to certain embodiments of the present disclosure is shown. The apparatus 500 may be implemented as or included in an electronic device. Various modules / components in the apparatus 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0077] like Figure 5As shown, the device 500 includes an acquisition module 510 configured to acquire a speech feature sequence corresponding to a speech sample, wherein the speech features in the speech feature sequence correspond to speech frames in the speech sample. The device 500 also includes a generation module 520 configured to, for a target speech feature in the speech feature sequence, use a speech coding model to generate one or more subsequent speech tags corresponding to one or more subsequent speech features, based on the target speech feature and one or more preceding speech features. The one or more preceding speech features indicate speech features in the speech feature sequence preceding the target speech feature, and the one or more subsequent speech features indicate speech features in the speech feature sequence following the target speech feature. The device 500 also includes a training module 530 configured to train the speech coding model based on the one or more subsequent speech tags.

[0078] In some embodiments, the acquisition module 510 is further configured to determine a time-frequency representation corresponding to a speech sample, the time-frequency representation indicating at least the intensity of the speech sample changing over time at different frequencies; and to generate a speech feature sequence by performing downsampling on the time-frequency representation.

[0079] In some embodiments, the generation module 520 is further configured to generate a speech coding representation based on the target speech features and one or more preceding speech features using a speech coding model; and to generate one or more subsequent speech tags based on the speech coding representation.

[0080] In some embodiments, the generation module 520 is further configured to generate a training sequence based on the target speech features and one or more preceding speech features; and to generate a speech coding representation based on the training sequence using a speech coding model.

[0081] In some embodiments, the training module 530 is further configured to encode speech samples using a discrete encoder to generate one or more reference speech tags corresponding to one or more subsequent speech features; determine training loss components by comparing one or more subsequent speech tags with one or more reference speech tags; and update the model parameters of the speech coding model based on the training loss components.

[0082] In some embodiments, the training module 530 is further configured to, for a given subsequent speech tag in one or more subsequent speech tags, determine a given reference speech tag corresponding to the given subsequent speech tag from one or more reference speech tags; and determine a training loss component corresponding to the given subsequent speech tag based on the difference between the given subsequent speech tag and the given reference speech tag.

[0083] In some embodiments, the apparatus 500 further includes an adjustment module configured to: determine sequence length information of a speech tag sequence corresponding to a speech feature sequence obtained using a speech coding model; and adjust the sequence length of a reference speech tag sequence corresponding to a speech feature sequence obtained using a discrete encoder based on the sequence length information, wherein the adjusted reference speech tag sequence includes one or more reference speech tags.

[0084] In some embodiments, the training module 530 is further configured to obtain training loss components by taking multiple speech features in the speech feature sequence as target speech features respectively, and to update the model parameters of the speech coding model based on the training loss, including: determining the training loss based on the sum of the training loss components obtained for the multiple speech features respectively; and updating the model parameters of the speech coding model based on the training loss.

[0085] In some embodiments, the training module 530 is further configured to encode a sequence of speech features using a trained discrete encoder to generate one or more reference speech tags corresponding to one or more subsequent speech features; and to train a speech coding model based on the differences between the one or more subsequent speech tags and the one or more reference speech tags.

[0086] Figure 6 A block diagram is shown illustrating an electronic device 600 in which one or more embodiments of the present disclosure may be implemented. It should be understood that... Figure 6 The electronic device 600 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 6 The electronic device 600 shown can be used to achieve Figure 1 Electronic devices 140 and 150.

[0087] like Figure 6 As shown, electronic device 600 is in the form of a general-purpose electronic device. Components of electronic device 600 may include, but are not limited to, one or more processors or processing units 610, memory 620, storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processing unit 610 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 620. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 600.

[0088] Electronic device 600 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 620 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 630 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 600.

[0089] Electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 6 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 620 may include computer program product 625 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0090] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 600 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 600 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0091] Input device 650 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 660 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 600 can also communicate with one or more external devices (not shown) via communication unit 640 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 600, or with any device that enables electronic device 600 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0092] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0093] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0094] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0095] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0096] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0097] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for speech processing, comprising: Obtain a speech feature sequence corresponding to a speech sample, wherein the speech features in the speech feature sequence correspond to the speech frames in the speech sample; For the target speech feature in the speech feature sequence, using a speech coding model, based on the target speech feature and one or more preceding speech features, one or more subsequent speech tags are generated, each corresponding to one or more subsequent speech features. The one or more preceding speech features include speech features in the speech feature sequence preceding the target speech feature, and the one or more subsequent speech features include speech features in the speech feature sequence following the target speech feature. The speech coding model is trained based on one or more subsequent speech tags.

2. The method according to claim 1, wherein generating subsequent speech tags corresponding to the one or more subsequent speech features comprises: Based on the target speech features and one or more preceding speech features, a speech coding representation is generated using the speech coding model; as well as Based on the speech encoding representation, one or more subsequent speech tags are generated.

3. The method according to claim 2, wherein generating a speech coding representation using the speech coding model comprises: A training sequence is generated based on the target speech features and one or more preceding speech features; as well as Based on the training sequence, the speech coding model is used to generate the speech coding representation.

4. The method according to claim 1, wherein training the speech coding model based on the one or more subsequent speech tags comprises: The speech samples are encoded using a discrete encoder to generate one or more reference speech tags corresponding to the one or more subsequent speech features, respectively. The training loss components are determined by comparing the one or more subsequent speech tags with the one or more reference speech tags; and The model parameters of the speech coding model are updated based on the training loss components.

5. The method of claim 4, wherein determining the training loss component comprises: For a given subsequent speech mark in one or more subsequent speech marks, a given reference speech mark corresponding to the given subsequent speech mark is determined from one or more reference speech marks; as well as Based on the difference between the given subsequent speech tag and the given reference speech tag, the training loss component corresponding to the given subsequent speech tag is determined.

6. The method according to claim 4, wherein the plurality of speech features in the speech feature sequence are respectively used as the target speech features to obtain the training loss components, and updating the model parameters of the speech coding model based on the training loss includes: The training loss is determined based on the sum of the training loss components obtained for each of the multiple speech features; as well as The model parameters of the speech coding model are updated based on the training loss.

7. The method of claim 4, further comprising: Determine the sequence length information of the speech marker sequence corresponding to the speech feature sequence obtained using the speech coding model; as well as Based on the sequence length information, the sequence length of the reference speech tag sequence corresponding to the speech feature sequence obtained by the discrete encoder is adjusted, wherein the adjusted reference speech tag sequence includes one or more reference speech tags.

8. The method according to claim 1, wherein obtaining the speech feature sequence corresponding to the speech sample comprises: Determine the time-frequency representation corresponding to the speech sample, wherein the time-frequency representation at least indicates the intensity of the speech sample changing over time at different frequencies; as well as The speech feature sequence is generated by downsampling the time-frequency representation.

9. The method of claim 1, wherein training the speech coding model based on the one or more subsequent speech tags comprises: The speech feature sequence is encoded using a trained discrete encoder to generate one or more reference speech tags, each corresponding to one or more subsequent speech features. The speech coding model is trained based on the differences between the one or more subsequent speech tags and the one or more reference speech tags.

10. An apparatus for speech processing, comprising: The acquisition module is configured to acquire a speech feature sequence corresponding to a speech sample, wherein the speech features in the speech feature sequence correspond to the speech frames in the speech sample; The generation module is configured to, for the target speech feature in the speech feature sequence, use a speech coding model to generate one or more subsequent speech tags corresponding to one or more subsequent speech features, based on the target speech feature and one or more preceding speech features, wherein the one or more preceding speech features indicate speech features in the speech feature sequence preceding the target speech feature, and the one or more subsequent speech features indicate speech features in the speech feature sequence following the target speech feature. as well as The training module is configured to train the speech coding model based on one or more subsequent speech tags.

11. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 9 when executed by the at least one processing unit.

12. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 9.

Images