Voice processing method and device, electronic equipment, chip and storage medium

Abstract

The invention relates to a voice processing method and device, electronic equipment, a chip and a storage medium, and belongs to the technical field of voice processing. The method comprises the following steps: acquiring first voice information of target voice processing to be executed; encoding the first voice information to obtain first encoding information; performing coding mapping associated with the target voice processing on the first coding information to obtain second coding information; and decoding the second coding information to obtain second voice information, and taking the second voice information as the voice information of which the target voice processing is completed. Therefore, the target voice processing can be performed on the first voice information in the submerged space to obtain the second voice information, the voice processing accuracy is improved, the quality of the voice information subjected to the target voice processing is further improved, and the method and the device are suitable for scenes such as noise reduction processing, dereverberation processing and bandwidth expansion processing.

Description

Technical Field

[0001] The present disclosure relates to the field of speech processing technology, and in particular to a speech processing method, device, electronic device, chip, and storage medium. Background Art

[0002] Currently, voice processing technology is widely used in scenarios such as phone calls, assisted driving, smart homes, smart offices, and business processing. For example, in phone calls, the original call voice can be processed to generate the target call voice. In smart home scenarios, for example, the original user voice can be processed to generate the target user voice, and control instructions for smart home devices can be obtained based on the target user voice. However, the voice processing methods in related technologies suffer from poor voice quality. Summary of the Invention

[0003] The present disclosure provides a speech processing method, apparatus, electronic device, chip, and storage medium to at least address the problem of poor speech quality in speech processing methods in related technologies. The technical solutions of the present disclosure are as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, a speech processing method is provided, comprising: acquiring first speech information on which target speech processing is to be performed; encoding the first speech information to obtain first encoding information; performing encoding mapping associated with the target speech processing on the first encoding information to obtain second encoding information; and decoding the second encoding information to obtain second speech information as speech information for completing the target speech processing.

[0005] According to a second aspect of an embodiment of the present disclosure, a method for training a speech processing model is provided, wherein the speech processing model includes a pre-trained audio encoder, a target encoder, and a vocoder, the method comprising: encoding sample first speech information by the audio encoder to obtain sample first coding information; training the target encoder and the vocoder based on the sample first coding information, the target encoder being used to perform coding mapping associated with target speech processing on the sample first coding information, and the vocoder being used to decode predicted coding information obtained by the coding mapping of the target encoder; wherein, during the process of training the target encoder and the vocoder, the parameters of the audio encoder are fixed.

[0006] According to a third aspect of an embodiment of the present disclosure, a speech processing device is provided, comprising: an acquisition module configured to acquire first speech information to be subjected to target speech processing; a first encoding module configured to encode the first speech information to obtain first encoding information; a second encoding module configured to perform encoding mapping associated with the target speech processing on the first encoding information to obtain second encoding information; and a decoding module configured to decode the second encoding information to obtain second speech information as speech information for completing the target speech processing.

[0007] According to a fourth aspect of an embodiment of the present disclosure, a training device for a speech processing model is provided, the speech processing model including a pre-trained audio encoder, a target encoder and a vocoder, the device including: a processing module configured to encode sample first speech information through the audio encoder to obtain sample first coding information; a training module configured to train the target encoder and the vocoder based on the sample first coding information, the target encoder being used to perform coding mapping associated with target speech processing on the sample first coding information, and the vocoder being used to decode predicted coding information obtained by the coding mapping of the target encoder; wherein, during the process of training the target encoder and the vocoder, the parameters of the audio encoder are fixed.

[0008] According to the fifth aspect of the embodiments of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, the steps of the speech processing method described in the first aspect of the embodiments of the present disclosure are implemented, and / or the steps of the training method of the speech processing model described in the second aspect of the embodiments of the present disclosure are implemented.

[0009] According to the sixth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the speech processing method described in the first aspect of the embodiments of the present disclosure are implemented, and / or the steps of the training method of the speech processing model described in the second aspect of the embodiments of the present disclosure are implemented.

[0010] According to the seventh aspect of the embodiments of the present disclosure, a chip is provided, which includes an interface circuit and a processing circuit coupled to each other, wherein the interface circuit is used to input or output signals, and the processing circuit is configured to implement the steps of the speech processing method described in the first aspect of the embodiments of the present disclosure, and / or implement the steps of the training method of the speech processing model described in the second aspect of the embodiments of the present disclosure.

[0011] According to the eighth aspect of the embodiments of the present disclosure, a computer program product is provided, comprising a computer program. When the computer program is executed by a processor, the computer program implements the steps of the speech processing method described in the first aspect of the embodiments of the present disclosure, and / or implements the steps of the speech processing model training method described in the second aspect of the embodiments of the present disclosure.

[0012] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects: obtaining first voice information to be subjected to target voice processing, encoding the first voice information to obtain first coded information, performing a coding mapping associated with the target voice processing on the first coded information to obtain second coded information, and decoding the second coded information to obtain second voice information as the voice information that has completed the target voice processing. Thus, the first coded information can be subjected to a coding mapping associated with the target voice processing to obtain second coded information, and the second coded information can be decoded to obtain second voice information, thereby performing the target voice processing on the first voice information in the latent space to obtain the second voice information. This improves the accuracy of the voice processing and, in turn, the quality of the voice information that has completed the target voice processing. The solution is suitable for scenarios such as noise reduction, dereverberation, and bandwidth extension.

[0013] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0015] Figure 1 The figure is a flowchart of a speech processing method according to an exemplary embodiment.

[0016] Figure 2 The figure is a flowchart of a speech processing method according to another exemplary embodiment.

[0017] Figure 3 The figure is a flowchart of a method for training a speech processing model according to an exemplary embodiment.

[0018] Figure 4 It is a flowchart of a method for training a speech processing model according to another exemplary embodiment.

[0019] Figure 5 The figure is a schematic diagram showing a speech processing method according to an exemplary embodiment.

[0020] Figure 6It is a schematic diagram of a method for training a speech processing model according to an exemplary embodiment.

[0021] Figure 7 The figure is a structural diagram of a speech processing device according to an exemplary embodiment.

[0022] Figure 8 The figure is a structural diagram of a training device for a speech processing model according to an exemplary embodiment.

[0023] Figure 9 The figure is a schematic structural diagram of an electronic device according to an exemplary embodiment.

[0024] Figure 10 The figure is a schematic structural diagram of a chip according to an exemplary embodiment. DETAILED DESCRIPTION

[0025] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0026] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0027] The following describes the speech processing method, air conditioning equipment, apparatus, electronic device, chip, and storage medium of the embodiments of the present disclosure with reference to the accompanying drawings.

[0028] Figure 1 FIG. 1 is a flow chart showing a method for voice processing according to an exemplary embodiment. Figure 1 As shown, the speech processing method of the embodiment of the present disclosure includes the following steps.

[0029] S101, obtaining first voice information to be subjected to target voice processing.

[0030] It should be noted that the execution entities of the speech processing methods of the embodiments of the present disclosure are electronic devices, such as terminal devices, vehicles, robots, smart home devices, smart office equipment, servers, chips, etc. Terminal devices may include mobile phones, wearable devices (such as smart watches and smart glasses), laptops, etc. Vehicles may include onboard terminals and onboard controllers, etc. Robots may include cleaning robots and service robots, etc. Service robots are used to provide services such as business processing, guiding users to service windows, and answering questions. Smart home devices may include televisions, speakers, refrigerators, washing machines, etc., and smart office equipment may include projectors, display screens, printers, etc.

[0031] The speech processing method of the embodiment of the present disclosure can be executed by the speech processing device of the embodiment of the present disclosure. The speech processing device of the embodiment of the present disclosure can be configured in any electronic device to execute the speech processing method of the embodiment of the present disclosure.

[0032] There are not too many restrictions on the target speech processing. For example, it may include noise reduction processing, dereverberation processing, bandwidth extension processing, etc. Noise reduction processing aims to reduce or remove noise information in speech information, thereby improving the clarity of speech information. Reverberation is caused by the reflection of speech information in the room, which makes the speech information sound unclear, that is, affects the clarity of speech information. De-dereverberation processing aims to reduce or remove reverberation information in speech signals. Bandwidth extension processing aims to reconstruct broadband speech information from narrowband speech information to improve the naturalness and clarity of speech information.

[0033] There are no excessive restrictions on the first voice information.

[0034] Optionally, the first voice information includes at least one of the following voice information:

[0035] Call voice information between the first terminal device and the second terminal device;

[0036] Interactive voice information between users and electronic devices.

[0037] Therefore, the voice processing method disclosed herein is applicable to call scenarios and / or voice interaction scenarios. In call scenarios, it can improve the accuracy of processing call voice information, thereby improving the quality of the processed call voice information. In voice interaction scenarios, it can improve the accuracy of processing interactive voice information, thereby improving the quality of the processed interactive voice information, thereby optimizing the user experience.

[0038] It is understandable that the call voice information between the first terminal device and the second terminal device may include at least one of the call voice information collected by the first terminal device and the call voice information collected by the second terminal device.

[0039] There are not too many restrictions on the communication method between the first terminal device and the second terminal device. For example, it can include VoIP (Voice over Internet Protocol, voice transmission based on Internet Protocol), voice transmission based on telephone network, etc., among which VoIP is also called Internet phone, Internet phone, and broadband phone.

[0040] The interactive voice information between the user and the electronic device may include at least one of interactive voice information sent by the user to the electronic device and interactive voice information sent by the electronic device to the user.

[0041] Optionally, obtaining the first voice information to be subjected to target voice processing includes collecting the first voice information, or receiving the first voice information. Thus, the first voice information can be collected or received to achieve the acquisition of the first voice information.

[0042] Optionally, before collecting the first voice information, the method further includes obtaining at least one of a call instruction and an instruction for starting a voice interaction function. Thus, the first voice information can be collected when at least one of the call instruction and the instruction for starting a voice interaction function is obtained, thereby satisfying the user's call and / or voice interaction needs.

[0043] For example, taking the execution subject as the first terminal device, the first terminal device can respond to the call instruction and collect call voice information 1 as the first voice information. It should be noted that in this embodiment, call voice information 1 refers to the call voice information collected by the first terminal device and not sent to the second terminal device.

[0044] For example, taking the execution subject as an electronic device, the electronic device can respond to the instruction for starting the voice interaction function and collect the interactive voice information 3 between the user and the electronic device as the first voice information.

[0045] Optionally, the call instruction is triggered based on the operation of at least one of the first terminal device and the second terminal device, and the instruction for starting the voice interaction function is triggered based on the operation of the electronic device. There are not many restrictions on the operations for triggering the call instruction and the instruction for starting the voice interaction function, for example, they may include mechanical operation, touch operation, voice input operation, etc.

[0046] Optionally, receiving the first voice information includes receiving the first voice information sent by the terminal device.

[0047] For example, taking the execution subject as the second terminal device, the second terminal device can receive call voice information 2 sent by the first terminal device as the first voice information. It should be noted that in this embodiment, call voice information 2 refers to the call voice information sent by the first terminal device to the second terminal device. For example, call voice information 2 includes call voice information collected by the first terminal device and sent to the second terminal device.

[0048] For example, taking the execution subject as a server, the server can receive the call voice information 1 sent by the first terminal device as the first voice information.

[0049] For example, taking the execution subject as a server, the server can receive the call voice information 2 sent by the second terminal device as the first voice information.

[0050] Optionally, receiving the first voice information includes receiving the first voice information sent by an electronic device.

[0051] For example, taking the execution subject as a server, the server can receive the interactive voice information 3 sent by the electronic device as the first voice information.

[0052] S102: Encode the first voice information to obtain first encoded information.

[0053] It should be noted that encoding the first voice information refers to mapping the first voice information to a latent space, which is also called an implicit space. Encoding the first voice information can be achieved using any audio encoding method in the relevant art, which is not limited here.

[0054] Optionally, encoding the first voice information to obtain the first coded information includes performing a time-domain-frequency domain conversion on the first voice information to obtain spectrum information of the first voice information, and encoding the spectrum information to obtain the first coded information. Thus, the spectrum information of the first voice information can be encoded to obtain the first coded information.

[0055] For example, STFT (Short-Time Fourier Transform) can be performed on the first voice information according to a 32-millisecond window length and a 10-millisecond frame shift to obtain a time-frequency spectrum of the first voice information. Feature extraction is performed on the time-frequency spectrum of the first voice information to obtain a 64-dimensional logarithmic Mel spectrum as the frequency spectrum information of the first voice information.

[0056] The 4 frames of logarithmic mel spectrum are encoded to obtain a 768-dimensional embedding vector 1 as the first encoding information. It should be noted that in this embodiment, a 768-dimensional embedding vector can be generated from 40 milliseconds of first speech information.

[0057] Optionally, encoding the first speech information to obtain the first encoded information includes encoding the first speech information using a pre-trained audio encoder to obtain the first encoded information. Thus, the first encoded information can be generated using the pre-trained audio encoder, eliminating the need to retrain a new audio encoder, thereby saving model training time. Furthermore, the pre-trained audio encoder is trained on a large-scale dataset and can learn rich feature representations, thus having good latent space feature encoding capabilities, thereby helping to improve the quality of the first encoded information.

[0058] Optionally, the audio encoder includes a CNN (Convolutional Neural Network) and multiple visual encoders. The visual encoder can be any visual encoder in the related art, without further limitation, for example, it can include a Vision Transformer (ViT).

[0059] For example, the audio encoder consists of one CNN and 12 ViTs. The CNN has one input channel and 768 output channels, with convolution kernels and movement strides of 4 and 64 in the time and frequency dimensions, respectively. Each ViT contains 12 attention heads, with both input and output dimensions of 768.

[0060] S103: Perform coding mapping associated with target speech processing on the first coding information to obtain second coding information.

[0061] It should be noted that the feature dimensions of the first coding information and the second coding information may be inconsistent, and the coding mapping strategies associated with different target speech processing may be inconsistent. The coding mapping strategy can adopt any coding mapping strategy in the relevant technology, and no excessive restrictions are made here.

[0062] For example, the 768-dimensional embedding vector 1 may be subjected to coding mapping associated with the target speech processing to obtain a 768-dimensional embedding vector 2 as the second coding information.

[0063] Optionally, performing coding mapping associated with the target speech processing on the first coding information to obtain the second coding information includes mapping the first coding information to a feature space corresponding to the target speech processing to obtain the second coding information.

[0064] Optionally, the first coding information is subjected to coding mapping associated with the target speech processing to obtain the second coding information, including obtaining a target coding mapping strategy corresponding to the target speech processing, and coding mapping the first coding information according to the target coding mapping strategy to obtain the second coding information.

[0065] It can be understood that there is a corresponding relationship between the candidate speech processing and the candidate coding mapping strategy.

[0066] For example, there is a corresponding relationship between noise reduction processing and code mapping strategy 1, there is a corresponding relationship between dereverberation processing and code mapping strategy 2, and there is a corresponding relationship between bandwidth extension processing and code mapping strategy 3.

[0067] Taking the target speech processing including noise reduction processing as an example, the code mapping strategy 1 corresponding to the noise reduction processing can be obtained, and the first code information is code-mapped according to the code mapping strategy 1 to obtain the second code information. It should be noted that the target code mapping strategy includes the code mapping strategy 1 in this embodiment.

[0068] Taking the target speech processing including dereverberation processing as an example, the code mapping strategy 2 corresponding to the dereverberation processing can be obtained, and the first code information is code-mapped according to the code mapping strategy 2 to obtain the second code information. It should be noted that the target code mapping strategy in this embodiment includes the code mapping strategy 2.

[0069] Taking the target speech processing including bandwidth extension processing as an example, a code mapping strategy 3 corresponding to the bandwidth extension processing can be obtained, and the first code information is code-mapped according to the code mapping strategy 3 to obtain the second code information. It should be noted that in this embodiment, the target code mapping strategy includes the code mapping strategy 3.

[0070] Optionally, performing coding mapping associated with target speech processing on the first coding information to obtain second coding information includes performing coding mapping associated with target speech processing on the first coding information through a target encoder to obtain the second coding information.

[0071] Optionally, the target encoder includes multiple visual encoders.

[0072] For example, the target encoder includes 3 ViTs, each of which contains a 12-head attention structure, with both input and output dimensions of 768.

[0073] S104: Decode the second coded information to obtain second voice information as voice information for completing target voice processing.

[0074] Currently, voice processing technology is widely used in scenarios such as phone calls, assisted driving, smart homes, smart offices, and business processing. For example, in phone calls, the original call voice can be processed to generate the target call voice. In smart home scenarios, for example, the original user voice can be processed to generate the target user voice, and control instructions for smart home devices can be obtained based on the target user voice. However, the voice processing methods in related technologies suffer from poor voice quality.

[0075] In the present disclosure, the first coded information can be subjected to coding mapping associated with the target speech processing to obtain the second coded information, and the second coded information can be decoded to obtain the second speech information, so that the target speech processing can be performed on the first speech information in the latent space to obtain the second speech information, thereby improving the accuracy of the speech processing and further improving the quality of the speech information that completes the target speech processing, and is suitable for scenarios such as noise reduction processing, dereverberation processing, and bandwidth extension processing.

[0076] In addition, it is also suitable for scenarios such as calls, assisted driving, smart home, smart office, business processing, etc., and has good applicability and scalability.

[0077] It should be noted that the second voice information refers to the voice information obtained by performing target voice processing on the first voice information.

[0078] Optionally, decoding the second coded information to obtain the second voice information includes decoding the second coded information through a vocoder to obtain the second voice information.

[0079] For example, Figure 5 As shown, the speech processing model includes a pre-trained audio encoder, a target encoder, and a vocoder. The audio encoder encodes first speech information to obtain first coded information. The target encoder performs coding mapping associated with target speech processing on the first coded information to obtain second coded information. The vocoder decodes the second coded information to obtain second speech information.

[0080] Optionally, after obtaining the second voice information, the method further includes performing at least one of the following operations:

[0081] Operation 1: When the first voice information includes call voice information collected by the first terminal device and not sent to the second terminal device, the second voice information is sent to the second terminal device, and the second voice information is used for voice playback on the second terminal device.

[0082] Therefore, in a call scenario, if the first voice information includes call voice information collected by the first terminal device and not sent to the second terminal device, after obtaining the second voice information, the second voice information can be sent to the second terminal device, and the second voice information is used for voice playback on the second terminal device to realize the call between the first terminal device and the second terminal device.

[0083] For example, taking mobile phone A as the execution subject, mobile phone A can respond to the call instruction, collect call voice information 1, encode call voice information 1 to obtain first encoded information, perform encoding mapping associated with the target voice processing on the first encoded information to obtain second encoded information, decode the second encoded information to obtain second voice information, and send the second voice information to mobile phone B. The second voice information is used for voice playback by mobile phone B. It should be noted that in this embodiment, mobile phone A is the first terminal device and mobile phone B is the second terminal device. Call voice information 1 refers to the call voice information collected by mobile phone A and not sent to mobile phone B. The first voice information includes call voice information 1.

[0084] For example, using a server as the execution subject, the server may receive call voice information 1 sent by mobile phone A, encode call voice information 1 to obtain first encoded information, perform encoding mapping associated with the target voice processing on the first encoded information to obtain second encoded information, decode the second encoded information to obtain second voice information, and send the second voice information to mobile phone B. The second voice information is used for voice playback by mobile phone B. It should be noted that in this embodiment, mobile phone A is the first terminal device, mobile phone B is the second terminal device, call voice information 1 refers to the call voice information collected by mobile phone A and not sent to mobile phone B, and the first voice information includes call voice information 1.

[0085] Operation 2: When the first voice information includes call voice information sent by the first terminal device to the second terminal device, play the second voice information, or send the second voice information to the second terminal device, and the second voice information is used for voice playback on the second terminal device.

[0086] Therefore, in a call scenario, if the first voice information includes call voice information sent by the first terminal device to the second terminal device, after obtaining the second voice information, if the execution subject is the second terminal device, the second voice information can be played, or, if the execution subject is not the second terminal device, the second voice information can be sent to the second terminal device, and the second voice information is used for voice playback on the second terminal device to realize a call between the first terminal device and the second terminal device.

[0087] For example, using mobile phone B as the execution subject, mobile phone B can receive call voice information 2, encode call voice information 2 to obtain first encoded information, perform code mapping associated with the target voice processing on the first encoded information to obtain second encoded information, decode the second encoded information to obtain second voice information, and play the second voice information. It should be noted that in this embodiment, mobile phone A is the first terminal device, mobile phone B is the second terminal device, call voice information 2 refers to the call voice information sent from mobile phone A to mobile phone B, and the first voice information includes call voice information 2.

[0088] For example, taking the execution subject as a server, the server may receive call voice information 2 sent by mobile phone B, encode call voice information 2 to obtain first encoded information, perform encoding mapping associated with the target voice processing on the first encoded information to obtain second encoded information, decode the second encoded information to obtain second voice information, and send the second voice information to mobile phone B. The second voice information is used for voice playback by mobile phone B. It should be noted that in this embodiment, mobile phone A is the first terminal device and mobile phone B is the second terminal device. Call voice information 2 refers to the call voice information sent by mobile phone A to mobile phone B, and the first voice information includes call voice information 2.

[0089] Operation 3: When the first voice information includes interactive voice information between the user and the electronic device, perform voice recognition on the second voice information to obtain a voice interaction instruction, execute the voice interaction instruction, or send the voice interaction instruction to the electronic device.

[0090] Therefore, in the voice interaction scenario, after obtaining the second voice information, voice recognition can be performed on the second voice information to obtain a voice interaction instruction. When the execution subject is an electronic device that performs voice interaction with the user, the voice interaction instruction is executed, or a voice interaction instruction is sent to the electronic device. The voice interaction instruction is used for execution by the electronic device to realize voice interaction between the user and the electronic device.

[0091] For example, using smart glasses as an example, the smart glasses can collect interactive voice information 3 between the user and the smart glasses, encode the interactive voice information 3 to obtain first encoded information, perform encoding mapping associated with the target voice processing on the first encoded information to obtain second encoded information, decode the second encoded information to obtain second voice information, perform voice recognition on the second voice information, obtain a voice interaction instruction, and execute the voice interaction instruction. It should be noted that in this embodiment, the smart glasses are electronic devices that perform voice interaction with the user, and the first voice information includes the interactive voice information 3.

[0092] For example, taking the execution subject as a server, the server may receive interactive voice information 3 sent by the smart glasses, encode the interactive voice information 3 to obtain first encoded information, perform encoding mapping associated with the target voice processing on the first encoded information to obtain second encoded information, decode the second encoded information to obtain second voice information, perform voice recognition on the second voice information to obtain a voice interaction instruction, and send the voice interaction instruction to the smart glasses. It should be noted that in this embodiment, the smart glasses are electronic devices that perform voice interaction with the user, and the first voice information includes the interactive voice information 3.

[0093] It should be noted that the present disclosure does not limit the execution sequence of steps S101 to S104. Figure 1 The steps S101 to S104 are merely executed in sequence for example.

[0094] The speech processing method provided by the embodiments of the present disclosure obtains first speech information to be subjected to target speech processing, encodes the first speech information to obtain first coded information, performs a coding mapping associated with the target speech processing on the first coded information to obtain second coded information, and decodes the second coded information to obtain second speech information as the speech information that has completed the target speech processing. Thus, the first coded information can be subjected to a coding mapping associated with the target speech processing to obtain second coded information, and the second coded information can be decoded to obtain second speech information. This allows the target speech processing to be performed on the first speech information in a latent space to obtain the second speech information, thereby improving the accuracy of the speech processing and, in turn, the quality of the speech information that has completed the target speech processing. The method is suitable for scenarios such as noise reduction, dereverberation, and bandwidth extension.

[0095] Figure 2 is a flow chart of a speech processing method according to another exemplary embodiment. Figure 2 As shown, the speech processing method of the embodiment of the present disclosure includes the following steps.

[0096] S201: Acquire first voice information to be processed by target voice.

[0097] S202: Encode the first voice information to obtain first encoded information.

[0098] The relevant contents of steps S201-S202 can be found in the above embodiment and will not be repeated here.

[0099] S203, obtaining a target encoder corresponding to the target speech processing, where the target encoder includes multiple visual encoders.

[0100] S204: Perform coding mapping associated with target speech processing on the first coding information through a target encoder to obtain second coding information.

[0101] In this embodiment, there is a correspondence between the target speech processing and the target encoder. Multiple candidate encoders can be set based on the speech processing type to cover multiple speech processing types, and encoding mapping is performed through the target encoder corresponding to the target speech processing. That is, different target encoders can be used for encoding mapping under different speech processing types, which improves the diversity of encoding mapping logic and is suitable for scenarios such as noise reduction processing, dereverberation processing, and bandwidth extension processing, and helps to improve the quality of speech information that completes the target speech processing.

[0102] In addition, when the type of speech processing changes, only the target encoder needs to be replaced, which is simple and easy.

[0103] Optionally, the correspondence between the target speech processing and the target encoder includes at least one of the following correspondences:

[0104] The target speech processing includes noise reduction processing, and the corresponding target encoder includes a noise reduction encoder;

[0105] The target speech processing includes dereverberation processing, and the corresponding target encoder includes a dereverberation encoder;

[0106] The target speech processing includes bandwidth extension processing, and the corresponding target encoder includes a bandwidth extension encoder;

[0107] For example, if the target speech processing includes noise reduction processing, the noise reduction encoder may perform coding mapping associated with the noise reduction processing on the first coding information to obtain second coding information.

[0108] For example, if the target speech processing includes dereverberation processing, the dereverberation encoder may perform coding mapping associated with the dereverberation processing on the first coded information to obtain second coded information.

[0109] For example, if the target speech processing includes bandwidth extension processing, the bandwidth extension encoder may perform coding mapping associated with the bandwidth extension processing on the first coding information to obtain second coding information.

[0110] S205: Decode the second coded information to obtain second voice information as voice information for completing target voice processing.

[0111] The relevant contents of step S205 can be found in the above embodiment and will not be repeated here.

[0112] It should be noted that the present disclosure does not limit the execution sequence of steps S201 to S205. Figure 2 The steps S201 to S205 are merely executed in sequence for example.

[0113] The speech processing method provided by the embodiments of the present disclosure obtains a target encoder corresponding to the target speech processing, the target encoder including multiple visual encoders. The target encoder performs encoding mapping associated with the target speech processing on the first encoding information to obtain the second encoding information. Thus, multiple candidate encoders can be set based on the speech processing type to cover multiple speech processing types, and encoding mapping can be performed using the target encoder corresponding to the target speech processing. That is, different target encoders can be used for encoding mapping under different speech processing types, thereby increasing the diversity of encoding mapping logic. This is suitable for scenarios such as noise reduction processing, dereverberation processing, and bandwidth expansion processing, and helps to improve the quality of speech information that completes the target speech processing.

[0114] Figure 3 FIG. 1 is a flow chart showing a method for training a speech processing model according to an exemplary embodiment. Figure 3 As shown, the training method of the speech processing model of the embodiment of the present disclosure includes the following steps.

[0115] S301 , encoding sample first speech information through an audio encoder to obtain sample first encoded information.

[0116] S302, based on the sample first coding information, the target encoder and the vocoder are trained, the target encoder is used to perform coding mapping associated with the target speech processing on the sample first coding information, and the vocoder is used to decode the predicted coding information obtained by the coding mapping of the target encoder.

[0117] In the present disclosure, a speech processing model includes a pre-trained audio encoder, a target encoder, and a vocoder. During the training of the target encoder and vocoder, the parameters of the audio encoder are fixed. The audio encoder is used to encode sample first speech information to obtain sample first coded information. The target encoder is used to perform code mapping associated with target speech processing on the sample first coded information to obtain predicted coded information. The vocoder is used to decode the predicted coded information obtained by the target encoder code mapping to obtain predicted speech information.

[0118] During the training process, the target encoder can learn the coding mapping capability associated with the target speech processing, and the vocoder can learn the ability to reconstruct speech information based on the coding information. Thus, the trained target encoder can perform coding mapping associated with the target speech processing on the first coding information to obtain second coding information, and the trained vocoder can decode the second coding information to obtain second speech information.

[0119] In addition, the trained speech processing model can perform target speech processing on the first speech information in the latent space to obtain the second speech information, thereby improving the accuracy of speech processing and thereby improving the quality of the speech information that completes the target speech processing. It is suitable for scenarios such as noise reduction processing, dereverberation processing, and bandwidth expansion processing.

[0120] It should be noted that the target encoder and vocoder can be trained independently or jointly, and there are no excessive restrictions here.

[0121] Optionally, based on the sample first coding information, the target encoder and vocoder are trained, including taking the sample first voice information as the voice information to be subjected to target voice processing, obtaining the sample second voice information that has completed the target voice processing, performing coding mapping associated with the target voice processing on the sample first coding information through the target encoder to obtain predicted coding information, decoding the predicted coding information through the vocoder to obtain predicted voice information, and training the target encoder and vocoder based on the sample second voice information and the predicted voice information.

[0122] For example, based on the sample second speech information and the predicted speech information, the target encoder and vocoder are trained, including obtaining the loss function of the speech processing model based on the sample second speech information and the predicted speech information, and training the target encoder and vocoder based on the loss function.

[0123] It should be noted that the present disclosure does not limit the execution sequence of steps S301 to S302. Figure 3 The following is an example of the sequential execution of steps S301 to S302. The relevant contents of steps S301 to S302 can be referred to the above embodiment and will not be repeated here.

[0124] The speech processing model training method provided by the embodiments of the present disclosure encodes sample first speech information through an audio encoder to obtain the sample first encoded information, and trains a target encoder and a vocoder based on the sample first encoded information. Thus, during the training process, the target encoder can learn the code mapping capabilities associated with the target speech processing, and the vocoder can learn the ability to reconstruct speech information based on the encoded information. The trained target encoder can then perform code mapping associated with the target speech processing on the first encoded information to obtain second encoded information, and the trained vocoder can decode the second encoded information to obtain second speech information.

[0125] Figure 4 FIG. 1 is a flow chart showing a method for training a speech processing model according to another exemplary embodiment. Figure 4 As shown, the training method of the speech processing model of the embodiment of the present disclosure includes the following steps.

[0126] S401: Encode sample first speech information through an audio encoder to obtain sample first encoded information.

[0127] The relevant contents of step S401 can be found in the above embodiment and will not be repeated here.

[0128] S402: Based on the sample first voice information, obtain the sample second voice information to be subjected to target voice processing, where the sample first voice information is the voice information on which the target voice processing is completed.

[0129] In this embodiment, the first sample voice information refers to voice information obtained by performing target voice processing on the sample second voice information.

[0130] Optionally, obtaining the sample second voice information to be subjected to target voice processing based on the sample first voice information includes at least one of the following operations:

[0131] Operation 1: When the target speech processing includes noise reduction processing and the sample first speech information does not contain noise information, speech synthesis is performed on the sample first speech information and the sample noise information to obtain sample second speech information to be subjected to noise reduction processing.

[0132] In this embodiment, the sample first voice information does not contain noise information, and the sample first voice information is also called denoised voice information or clean voice information, and the sample second voice information is noisy voice information.

[0133] Operation 2: When the target speech processing includes dereverberation processing and the sample first speech information does not contain reverberation information, speech synthesis is performed on the sample first speech information and the sample reverberation information to obtain sample second speech information to be subjected to dereverberation processing.

[0134] In this embodiment, the sample first voice information does not contain reverberation information, and the sample first voice information is also called dereverberation voice information, and the sample second voice information is reverberation voice information.

[0135] Operation 3: When the target speech processing includes bandwidth extension and the sample first speech information is wideband speech information, downsample the sample first speech information to obtain narrowband speech information as the sample second speech information to be subjected to bandwidth extension.

[0136] In this embodiment, the sample first voice information is narrowband voice information, and the sample second voice information is wideband voice information.

[0137] S403: Encode the sample second speech information through an audio encoder to obtain sample second encoded information.

[0138] S404: Train the target encoder based on the sample first encoding information and the sample second encoding information.

[0139] Optionally, the target encoder is trained based on the sample first coding information and the sample second coding information, including performing coding mapping associated with the target speech processing on the sample second coding information through the target encoder to obtain predicted coding information, and training the target encoder based on the sample first coding information and the predicted coding information.

[0140] For example, based on the sample first coding information and the predicted coding information, the target encoder is trained, including obtaining the loss function of the target encoder based on the sample first coding information and the predicted coding information, and training the target encoder based on the loss function of the target encoder.

[0141] S405: Decode the first coded information of the sample through a vocoder to obtain predicted speech information.

[0142] S406: Training the vocoder based on the sample first voice information and the predicted voice information.

[0143] Optionally, the vocoder is trained based on the sample first speech information and the predicted speech information, including obtaining a loss function for the vocoder based on the sample first speech information and the predicted speech information, and training the vocoder based on the vocoder loss function. For example, in the loss function, an adversarial training method is employed, a multi-cycle and multi-resolution discriminator is used to calculate the adversarial loss, and a mean square error of the mel spectrum is added as the loss function.

[0144] It should be noted that the present disclosure does not limit the execution sequence of steps S401 to S406. Figure 4 The following example is only performed by sequentially executing steps S401 to S406. Steps S401 to S404 can be implemented as independent embodiments, and steps S401 to S402 and steps S405 to S406 can be implemented as independent embodiments.

[0145] The speech processing model training method provided by the embodiments of the present disclosure obtains sample second speech information to be subjected to target speech processing based on sample first speech information, wherein the sample first speech information is speech information that has completed the target speech processing. The sample second speech information is encoded by an audio encoder to obtain sample second encoded information. The target encoder is trained based on the sample first encoded information and the sample second encoded information. Thus, the audio encoder can be used to generate the sample first encoded information and the sample second encoded information, respectively, to train the target encoder.

[0146] In addition, the sample first coded information is decoded by a vocoder to obtain predicted speech information, and the vocoder is trained based on the sample first speech information and the predicted speech information.

[0147] Based on any of the above embodiments, when the target speech processing includes noise reduction processing, the speech processing model includes a pre-trained audio encoder, a noise reduction encoder, and a vocoder.

[0148] like Figure 6 As shown, the sample first voice information does not contain noise information, and speech synthesis is performed on the sample first voice information and the sample noise information to obtain the sample second voice information to be subjected to noise reduction processing.

[0149] The sample first speech information is encoded by an audio encoder to obtain sample first encoded information.

[0150] The sample second speech information is encoded by an audio encoder to obtain sample second encoded information.

[0151] The denoising encoder performs coding mapping associated with denoising processing on the second coded information of the sample to obtain predicted coded information, obtains a loss function of the denoising encoder based on the first coded information of the sample and the predicted coded information, and trains the denoising encoder based on the loss function of the denoising encoder.

[0152] The first encoded information of the sample is decoded by a vocoder to obtain predicted speech information, a loss function of the vocoder is obtained based on the first speech information of the sample and the predicted speech information, and the vocoder is trained based on the loss function of the vocoder.

[0153] Figure 7 The figure is a structural diagram of a speech processing device according to an exemplary embodiment.

[0154] Reference Figure 7 The speech processing device 700 of the embodiment of the present disclosure includes: an acquisition module 701, a first encoding module 702, a second encoding module 703 and a decoding module 704.

[0155] An acquisition module 701 is configured to acquire first speech information to be subjected to target speech processing;

[0156] A first encoding module 702 is configured to encode the first voice information to obtain first encoded information;

[0157] A second encoding module 703 is configured to perform encoding mapping associated with the target speech processing on the first encoding information to obtain second encoding information;

[0158] The decoding module 704 is configured to decode the second coded information to obtain second voice information as the voice information for completing the target voice processing.

[0159] In some possible implementations, the acquisition module 701 is further configured to: collect the first voice information; or receive the first voice information.

[0160] In some possible implementations, before collecting the first voice information, the acquisition module 701 is further configured to: acquire at least one of a call instruction and an instruction for starting a voice interaction function.

[0161] In some possible implementations, the acquisition module 701 is further configured to: receive the first voice information sent by a terminal device.

[0162] In some possible implementations, the first voice information includes at least one of the following voice information:

[0163] Call voice information between the first terminal device and the second terminal device;

[0164] Interactive voice information between users and electronic devices.

[0165] In some possible implementations, after obtaining the second voice information, the decoding module 704 is further configured to perform at least one of the following operations:

[0166] In a case where the first voice information includes call voice information sent by the first terminal device to the second terminal device, the second voice information is sent to the second terminal device, and the second voice information is used for voice playback by the second terminal device;

[0167] In the case where the first voice information includes call voice information sent by the first terminal device to the second terminal device, playing the second voice information, or sending the second voice information to the second terminal device, the second voice information is used for voice playback on the second terminal device;

[0168] In the case where the first voice information includes interactive voice information between a user and an electronic device, voice recognition is performed on the second voice information to obtain a voice interaction instruction, and the voice interaction instruction is executed, or the voice interaction instruction is sent to the electronic device.

[0169] In some possible implementations, the decoding module 704 is further configured to: perform time domain-frequency domain conversion on the first voice information to obtain spectrum information of the first voice information; and encode the spectrum information to obtain the first encoded information.

[0170] In some possible implementations, the first encoding module 702 is further configured to: encode the first speech information through a pre-trained audio encoder to obtain the first encoded information, where the audio encoder includes a convolutional neural network and multiple visual encoders.

[0171] In some possible implementations, the second encoding module 703 is further configured to: obtain a target encoder corresponding to the target speech processing, the target encoder including multiple visual encoders; and perform encoding mapping associated with the target speech processing on the first encoding information through the target encoder to obtain the second encoding information.

[0172] In some possible implementations, the correspondence between the target speech processing and the target encoder includes at least one of the following correspondences:

[0173] The target speech processing includes noise reduction processing, and the corresponding target encoder includes a noise reduction encoder;

[0174] The target speech processing includes dereverberation processing, and the corresponding target encoder includes a dereverberation encoder;

[0175] The target speech processing includes bandwidth extension processing, and the corresponding target encoder includes a bandwidth extension encoder.

[0176] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0177] The speech processing device provided by the embodiments of the present disclosure obtains first speech information to be subjected to target speech processing, encodes the first speech information to obtain first coded information, performs a coding mapping associated with the target speech processing on the first coded information to obtain second coded information, and decodes the second coded information to obtain second speech information as the speech information that has completed the target speech processing. Thus, the first coded information can be subjected to a coding mapping associated with the target speech processing to obtain second coded information, and the second coded information can be decoded to obtain second speech information. This allows the target speech processing to be performed on the first speech information in a latent space to obtain the second speech information, thereby improving the accuracy of the speech processing and, in turn, the quality of the speech information that has completed the target speech processing. The device is suitable for scenarios such as noise reduction, dereverberation, and bandwidth extension.

[0178] Figure 8 The present invention is a structural diagram of a training device for a speech processing model according to an exemplary embodiment. The speech processing model includes a pre-trained audio encoder, a target encoder, and a vocoder.

[0179] Reference Figure 8 The speech processing model training device 800 of the embodiment of the present disclosure includes: a processing module 801 and a training module 802.

[0180] The processing module 801 is configured to encode the sample first speech information through the audio encoder to obtain the sample first encoded information;

[0181] A training module 802 is configured to train the target encoder and the vocoder based on the sample first coding information, wherein the target encoder is configured to perform code mapping associated with target speech processing on the sample first coding information, and the vocoder is configured to decode the predicted coding information obtained by the code mapping of the target encoder;

[0182] Wherein, during the process of training the target encoder and the vocoder, the parameters of the audio encoder are fixed.

[0183] In some possible implementations, the training module 802 is further configured to: obtain sample second voice information to be subjected to target voice processing based on the sample first voice information, where the sample first voice information is the voice information for completing the target voice processing; encode the sample second voice information through the audio encoder to obtain sample second encoding information; and train the target encoder based on the sample first encoding information and the sample second encoding information.

[0184] In some possible implementations, the training module 802 is further configured to: perform coding mapping associated with the target speech processing on the sample second coding information through the target encoder to obtain predicted coding information; and train the target encoder based on the sample first coding information and the predicted coding information.

[0185] In some possible implementations, the training module 802 is further configured to perform at least one of the following operations:

[0186] When the target speech processing includes noise reduction processing and the sample first speech information does not contain noise information, performing speech synthesis on the sample first speech information and the sample noise information to obtain sample second speech information to be subjected to noise reduction processing;

[0187] When the target speech processing includes dereverberation processing and the sample first speech information does not contain reverberation information, performing speech synthesis on the sample first speech information and the sample reverberation information to obtain sample second speech information to be subjected to dereverberation processing;

[0188] When the target speech processing includes bandwidth extension and the sample first speech information is wideband speech information, the sample first speech information is downsampled to obtain narrowband speech information as the sample second speech information to be subjected to bandwidth extension.

[0189] In some possible implementations, the training module 802 is further configured to: decode the sample first encoded information through the vocoder to obtain predicted speech information; and train the vocoder based on the sample first speech information and the predicted speech information.

[0190] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0191] The speech processing model training device provided by the embodiments of the present disclosure encodes sample first speech information using an audio encoder to obtain the sample first encoded information, and trains a target encoder and a vocoder based on the sample first encoded information. Thus, during the training process, the target encoder can learn the code mapping capabilities associated with the target speech processing, and the vocoder can learn the ability to reconstruct speech information based on the encoded information. As a result, the trained target encoder can perform code mapping associated with the target speech processing on the first encoded information to obtain second encoded information, and the trained vocoder can decode the second encoded information to obtain second speech information.

[0192] In order to implement the above embodiments, the present disclosure also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, it implements the steps of the speech processing method provided by the present disclosure, and / or implements the steps of the training method of the speech processing model provided by the present disclosure.

[0193] Figure 9 9 is a schematic diagram illustrating the structure of an electronic device according to an exemplary embodiment. For example, the electronic device 900 may be a vehicle, a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0194] Reference Figure 9 The electronic device 900 may include one or more of the following components: a processing component 902 , a memory 904 , a power component 906 , a multimedia component 908 , an audio component 910 , an input / output (I / O) interface 912 , a sensor component 914 , and a communication component 916 .

[0195] The processing component 902 generally controls the overall operation of the electronic device 900, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 902 may include one or more processors 920 to execute instructions to complete all or part of the steps of the above-mentioned speech processing method, and / or to complete all or part of the steps of the above-mentioned speech processing model training method. In addition, the processing component 902 may include one or more modules to facilitate interaction between the processing component 902 and other components. For example, the processing component 902 may include a multimedia module to facilitate interaction between the multimedia component 908 and the processing component 902.

[0196] The memory 904 is configured to store various types of data to support operations on the electronic device 900. Examples of such data include instructions for any application or method operating on the electronic device 900, contact data, phone book data, messages, pictures, videos, etc. The memory 904 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0197] The power component 906 provides power to the various components of the electronic device 900. The power component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 900.

[0198] The multimedia component 908 includes a screen that provides an output interface between the electronic device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor can not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 908 includes a front camera and / or a rear camera. When the electronic device 900 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

[0199] The audio component 910 is configured to output and / or input audio signals. For example, the audio component 910 includes a microphone (MIC), and when the electronic device 900 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 904 or transmitted via the communication component 916. In some embodiments, the audio component 910 also includes a speaker for outputting audio signals.

[0200] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0201] The sensor assembly 914 includes one or more sensors for providing various aspects of the status assessment of the electronic device 900. For example, the sensor assembly 914 can detect the open / closed state of the electronic device 900, the relative positioning of components, such as the display and keypad of the electronic device 900. The sensor assembly 914 can also detect changes in the position of the electronic device 900 or a component of the electronic device 900, the presence or absence of user contact with the electronic device 900, the orientation or acceleration / deceleration of the electronic device 900, and changes in the temperature of the electronic device 900. The sensor assembly 914 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 914 can also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 914 can also include an accelerometer, a gyroscope, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0202] The communication component 916 is configured to facilitate wired or wireless communication between the electronic device 900 and other devices. The electronic device 900 can access a wireless network based on a communication standard, such as WiFi, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 916 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 916 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0203] In an exemplary embodiment, the electronic device 900 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the steps of the above-mentioned speech processing method and / or the steps of the above-mentioned speech processing model training method.

[0204] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is further provided, such as a memory 904 including instructions, wherein the instructions can be executed by the processor 920 of the electronic device 900 to perform the above-mentioned speech processing method and / or to complete the above-mentioned speech processing model training method. For example, the non-transitory computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0205] In order to implement the above embodiments, the present disclosure also proposes a computer-readable storage medium on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the speech processing method provided by the present disclosure are implemented, and / or the steps of the training method of the speech processing model provided by the present disclosure are implemented.

[0206] Alternatively, the computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0207] In order to implement the above embodiments, the present disclosure also proposes a chip, which includes an interface circuit and a processing circuit coupled to each other, the interface circuit is used to input or output signals, and the processing circuit is configured to implement the steps of the speech processing method provided by the present disclosure, and / or, implement the steps of the training method of the speech processing model provided by the present disclosure.

[0208] Figure 10FIG. 1 is a schematic diagram showing the structure of a chip according to an exemplary embodiment. Figure 10 The structure of the chip 1000 is shown, but is not limited thereto.

[0209] Chip 1000 includes a processing circuit 1001, which is configured to execute the steps of any of the above speech processing methods and / or execute the steps of any of the above speech processing model training methods.

[0210] In some embodiments, chip 1000 further includes one or more interface circuits 1002. Optionally, interface circuit 1002 is connected to memory 1003. Interface circuit 1002 can be used to receive signals from memory 1003 or other devices, and interface circuit 1002 can be used to send signals to memory 1003 or other devices. For example, interface circuit 1002 can read instructions stored in memory 1003 and send the instructions to processing circuit 1001.

[0211] In some embodiments, the interface circuit 1002 performs at least one of the communication steps such as sending and / or receiving in the above method, and the processing circuit 1001 performs the other steps.

[0212] In some embodiments, terms such as interface circuit, interface, transceiver pin, and transceiver may be used interchangeably.

[0213] In some embodiments, the chip 1000 further includes one or more memories 1003 for storing instructions. Alternatively, all or part of the memory 1003 may be located outside the chip 1000 .

[0214] In order to implement the above embodiments, the present disclosure also proposes a computer program product, including a computer program. When the computer program is executed by a processor, it implements the steps of the speech processing method provided by the present disclosure, and / or implements the steps of the training method of the speech processing model provided by the present disclosure.

[0215] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0216] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A speech processing method, characterized in that: The method comprises: Acquire first voice information to be subjected to target voice processing; Encoding the first voice information to obtain first coded information; Performing code mapping associated with the target speech processing on the first code information to obtain second code information; The second coded information is decoded to obtain second voice information as the voice information for completing the target voice processing.

2. The method according to claim 1, characterized in that The step of obtaining first voice information to be subjected to target voice processing includes: collecting the first voice information; or, Receive the first voice information.

3. The method according to claim 2, characterized in that Before collecting the first voice information, the method further includes: Obtain at least one of a call instruction and an instruction for starting a voice interaction function.

4. The method according to claim 2, characterized in that The receiving the first voice information includes: Receive the first voice information sent by the terminal device.

5. The method according to claim 1, wherein The first voice information includes at least one of the following voice information: Call voice information between the first terminal device and the second terminal device; Interactive voice information between users and electronic devices.

6. The method according to claim 1, wherein After obtaining the second voice information, the method further includes performing at least one of the following operations: In a case where the first voice information includes call voice information sent by the first terminal device to the second terminal device, the second voice information is sent to the second terminal device, and the second voice information is used for voice playback by the second terminal device; In the case where the first voice information includes call voice information sent by the first terminal device to the second terminal device, playing the second voice information, or sending the second voice information to the second terminal device, the second voice information is used for voice playback on the second terminal device; In the case where the first voice information includes interactive voice information between a user and an electronic device, voice recognition is performed on the second voice information to obtain a voice interaction instruction, and the voice interaction instruction is executed, or the voice interaction instruction is sent to the electronic device.

7. The method according to claim 1, characterized in that The encoding of the first voice information to obtain first coded information includes: Performing a time domain-frequency domain conversion on the first voice information to obtain frequency spectrum information of the first voice information; The spectrum information is encoded to obtain the first encoded information.

8. The method according to any one of claims 1 to 7, characterized in that The encoding of the first voice information to obtain first coded information includes: The first speech information is encoded by a pre-trained audio encoder to obtain the first encoded information, where the audio encoder includes a convolutional neural network and multiple visual encoders.

9. The method according to any one of claims 1 to 7, characterized in that The performing coding mapping associated with the target speech processing on the first coding information to obtain second coding information includes: Obtaining a target encoder corresponding to the target speech processing, wherein the target encoder includes a plurality of visual encoders; The target encoder performs coding mapping associated with the target speech processing on the first coding information to obtain the second coding information.

10. The method according to claim 9, characterized in that The correspondence between the target speech processing and the target encoder includes at least one of the following correspondences: The target speech processing includes noise reduction processing, and the corresponding target encoder includes a noise reduction encoder; The target speech processing includes dereverberation processing, and the corresponding target encoder includes a dereverberation encoder; The target speech processing includes bandwidth extension processing, and the corresponding target encoder includes a bandwidth extension encoder.

11. A method for training a speech processing model, characterized in that: The speech processing model includes a pre-trained audio encoder, a target encoder, and a vocoder, and the method includes: Encoding the sample first speech information by the audio encoder to obtain sample first encoded information; Based on the sample first coding information, the target encoder and the vocoder are trained, the target encoder is used to perform coding mapping associated with target speech processing on the sample first coding information, and the vocoder is used to decode the predicted coding information obtained by the coding mapping of the target encoder; Wherein, during the process of training the target encoder and the vocoder, the parameters of the audio encoder are fixed.

12. The method according to claim 11, characterized in that The training of the target encoder and the vocoder based on the sample first encoding information includes: Based on the sample first voice information, obtaining sample second voice information to be subjected to target voice processing, wherein the sample first voice information is voice information that has completed the target voice processing; Encoding the sample second voice information by the audio encoder to obtain sample second encoded information; The target encoder is trained based on the sample first encoding information and the sample second encoding information.

13. The method according to claim 12, characterized in that The training of the target encoder based on the first encoding information of the sample and the second encoding information of the sample includes: Performing coding mapping associated with the target speech processing on the second coding information of the sample by the target encoder to obtain predictive coding information; The target encoder is trained based on the sample first encoding information and the predicted encoding information.

14. The method according to claim 12, characterized in that The step of obtaining the sample second voice information to be subjected to target voice processing based on the sample first voice information includes at least one of the following operations: When the target speech processing includes noise reduction processing and the sample first speech information does not contain noise information, performing speech synthesis on the sample first speech information and the sample noise information to obtain sample second speech information to be subjected to noise reduction processing; When the target speech processing includes dereverberation processing and the sample first speech information does not contain reverberation information, performing speech synthesis on the sample first speech information and the sample reverberation information to obtain sample second speech information to be subjected to dereverberation processing; When the target speech processing includes bandwidth extension and the sample first speech information is wideband speech information, the sample first speech information is downsampled to obtain narrowband speech information as the sample second speech information to be subjected to bandwidth extension.

15. The method according to claim 11, characterized in that The training of the target encoder and the vocoder based on the sample first encoding information includes: Decoding the first encoded information of the sample by the vocoder to obtain predicted speech information; The vocoder is trained based on the sample first speech information and the predicted speech information.

16. A speech processing device, characterized in that: include: An acquisition module configured to acquire first voice information to be subjected to target voice processing; A first encoding module is configured to encode the first voice information to obtain first encoded information; a second encoding module configured to perform encoding mapping associated with the target speech processing on the first encoding information to obtain second encoding information; The decoding module is configured to decode the second coded information to obtain second voice information as the voice information for completing the target voice processing.

17. The device according to claim 16, characterized in that The acquisition module is further configured to: collecting the first voice information; or, Receive the first voice information.

18. The device according to claim 17, characterized in that Before collecting the first voice information, the acquisition module is further configured to: Obtain at least one of a call instruction and an instruction for starting a voice interaction function.

19. A training device for a speech processing model, characterized in that: The speech processing model includes a pre-trained audio encoder, a target encoder and a vocoder, and the apparatus includes: a processing module configured to encode the sample first speech information through the audio encoder to obtain sample first encoded information; a training module configured to train the target encoder and the vocoder based on the sample first coding information, the target encoder being configured to perform coding mapping associated with target speech processing on the sample first coding information, and the vocoder being configured to decode predicted coding information obtained by the coding mapping of the target encoder; Wherein, during the process of training the target encoder and the vocoder, the parameters of the audio encoder are fixed.

20. The device according to claim 19, characterized in that The training module is further configured to: Based on the sample first voice information, obtaining sample second voice information to be subjected to target voice processing, wherein the sample first voice information is voice information that has completed the target voice processing; Encoding the sample second voice information by the audio encoder to obtain sample second encoded information; The target encoder is trained based on the sample first encoding information and the sample second encoding information.

21. The device according to claim 20, characterized in that The training module is further configured to: Performing coding mapping associated with the target speech processing on the second coding information of the sample by the target encoder to obtain predictive coding information; The target encoder is trained based on the sample first encoding information and the predicted encoding information.

22. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method according to any one of claims 1 to 15 are implemented.

23. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

24. A chip, characterized in that: The chip includes an interface circuit and a processing circuit coupled to each other, the interface circuit is used to input or output signals, and the processing circuit is configured to implement the steps of the method according to any one of claims 1 to 15.

25. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 15.

Images