Voice processing method and device, storage medium and computer equipment

Abstract

The embodiment of the invention provides a voice processing method and device, a storage medium and computer equipment. The voice processing method comprises the steps that fused voice data obtained after at least two sound sources are mixed is acquired, and fused voice features are extracted; then, the fused speech features are input into a trained feature perceptron to determine a speech feature mask, and speech feature characterization is determined based on the speech feature mask. And determining a voice feature mask through a feature sensor, and further determining voice features. And finally, inputting the voice features into a trained decoder, and decoding voice data corresponding to each sound source. Blind source separation is carried out by sensing the voice characteristics through the characteristic sensor, the difference between the separated voice and the original voice is reduced, the difference between the separated voice and the original voice is reduced, the consistency is improved, and the voice processing accuracy is further improved.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech processing method, apparatus, storage medium and computer equipment. Background Art

[0002] In modern life, people are often surrounded by multiple sounds, such as mixed voices of multiple speakers, background noise and music. The selective attention mechanism enables people to focus on the voice of the target speaker in a complex noise environment. With the rapid progress and widespread application of artificial intelligence (AI) technology, the blind source separation (BSS) algorithm has been proposed, which aims to separate different sound sources into independent sound sources from the mixed voices of multiple speakers. High-quality speech separation systems can play an important role in teleconferencing systems, voice recording systems, hearing aids and hearing enhancement devices, smart homes and security monitoring.

[0003] In related technologies, although AI technology has significantly improved the performance of BSS tasks, in real environments, the lack of labeled data will lead to the inability to perform deep learning training, which in turn leads to low consistency between the separated speech and the original speech when the speech separation system performs blind source separation in real scenarios. Summary of the invention

[0004] The main purpose of this application is to provide a speech processing method, apparatus, storage medium and computer equipment, which can reduce the difference between the separated speech and the original speech, improve the consistency, and thus improve the accuracy of speech processing.

[0005] In a first aspect, an embodiment of the present application provides a speech processing method, comprising:

[0006] Acquire fused voice data, perform feature extraction on the fused voice data, and obtain fused voice features, wherein the fused voice data includes voice data generated by mixing at least two sound sources;

[0007] Inputting the fused speech features into the trained feature sensor to determine the speech feature mask of the speech data corresponding to each sound source;

[0008] Determining the speech characteristic representation of the speech data corresponding to each sound source from the fused speech features based on each of the speech characteristic masks;

[0009] Inputting each of the speech feature representations into a trained feature sensor to determine a speech feature mask of the speech data corresponding to each sound source;

[0010] Determining the speech features of the speech data corresponding to each sound source from the fused speech features based on each of the speech feature masks;

[0011] Each of the speech features is input into a trained decoder to decode the speech data corresponding to each of the sound sources.

[0012] In a second aspect, an embodiment of the present application provides a speech processing device, including:

[0013] A feature extraction unit is used to obtain fused speech data, perform feature extraction on the fused speech data, and obtain fused speech features, wherein the fused speech data includes speech data generated by mixing at least two sound sources;

[0014] A feature perception unit, used for inputting the fused speech feature into a trained feature perception unit to determine a speech feature mask of speech data corresponding to each sound source;

[0015] A first determining unit, configured to determine a speech feature representation of speech data corresponding to each sound source from the fused speech features based on each of the speech feature masks;

[0016] A feature perception unit, used for inputting each of the speech feature representations into a trained feature perception unit to determine a speech feature mask of the speech data corresponding to each sound source;

[0017] A second determining unit, configured to determine the speech features of the speech data corresponding to each sound source from the fused speech features based on each of the speech feature masks;

[0018] The decoding unit is used to input each of the speech features into a trained decoder to decode the speech data corresponding to each of the sound sources.

[0019] In some embodiments, the apparatus further comprises:

[0020] Acquire a first sample fused speech feature of the first sample fused speech data, and first sample real speech data of each first sound source included in the first sample fused speech data;

[0021] Inputting the first sample fusion speech feature into the feature sensor to be trained, and determining the first sample speech feature mask of the speech data corresponding to each first sound source;

[0022] Determining a first predicted feature of the speech data corresponding to each first sound source by using each of the first sample speech feature masks and the predictor to be trained;

[0023] Inputting each of the first sample real speech data into the trained feature extraction model to extract the first real feature of the speech data corresponding to each first sound source;

[0024] Determining a first loss value according to a first predicted characteristic of each first sound source and a corresponding first real characteristic;

[0025] When the first loss value is greater than the first preset loss value, the network parameters of the feature sensor to be trained and the predictor to be trained are adjusted according to the first loss value, and the step of inputting the first sample fusion speech feature into the feature sensor to be trained and determining the first sample speech feature mask of the speech data corresponding to each first sound source is returned until the first loss value is less than or equal to the first preset loss value, thereby obtaining the initially trained feature sensor and the trained predictor.

[0026] In some embodiments, an apparatus comprises:

[0027] Determining a first sample speech characteristic representation of speech data corresponding to each sound source from the first sample fused speech features based on each of the first sample speech characteristic masks;

[0028] Each of the first sample speech feature representations is input into the predictor to be trained to obtain a first predicted feature of the speech data corresponding to each first sound source.

[0029] In some embodiments, an apparatus comprises:

[0030] Determining a similarity between a first predicted feature of each first sound source and a corresponding first real feature;

[0031] Selecting a target first sound source with the greatest similarity from each first sound source;

[0032] Obtaining a similarity between the first predicted feature of the target first sound source and the first predicted feature of each other first sound source to obtain a plurality of first predicted similarities, wherein the other first sound sources are first sound sources other than the target first sound source in each first sound source;

[0033] A first loss value is determined according to a target similarity between a first predicted feature of the target first sound source and a corresponding first real feature, and a plurality of the first predicted similarities.

[0034] In some embodiments, the apparatus further comprises:

[0035] Acquire second sample fused speech features of the second sample fused speech data, and second sample real speech data of each second sound source included in the second sample fused speech data;

[0036] Inputting the second sample fusion speech feature into the preliminarily trained feature sensor to determine the second sample speech feature mask of the speech data corresponding to each second sound source;

[0037] Determining a second sample speech characteristic representation of the speech data corresponding to each second sound source from the second sample fused speech features based on each second sample speech characteristic mask;

[0038] Inputting each of the second sample speech feature representations into a feature sensor to be trained, and determining a sample speech feature mask of speech data corresponding to each second sound source;

[0039] Determining, from the second sample fused speech features based on each of the sample speech feature masks, sample speech features of the speech data corresponding to each second sound source;

[0040] Inputting each of the sample speech features into a decoder to be trained, and decoding the second sample predicted speech data corresponding to each second sound source;

[0041] Determine a second loss value based on the second sample predicted speech data of each second sound source and the corresponding second sample real speech data;

[0042] When the second loss value is greater than the second preset loss value, the network parameters of the feature sensor to be trained are adjusted according to the second loss value, and the step of inputting the second sample fusion speech feature into the feature sensor after preliminary training and determining the second sample speech feature mask of the speech data corresponding to each second sound source is returned until the second loss value is less than or equal to the second preset loss value, thereby obtaining the trained feature sensor, the trained feature sensor and the trained decoder.

[0043] In some embodiments, an apparatus comprises:

[0044] Determine the speech distortion corresponding to each second sound source according to the second sample predicted speech data of each second sound source and the corresponding second sample real speech data;

[0045] A target speech distortion degree with the minimum speech distortion degree is selected from each of the speech distortion degrees, and the target speech distortion degree is determined as the second loss value.

[0046] In some embodiments, an apparatus comprises:

[0047] For each second sound source, calculating a similarity energy value of a similar part between the corresponding second sample predicted speech data and the corresponding second sample real speech data;

[0048] Calculate the difference energy value of the difference part between the corresponding second sample predicted speech data and the corresponding second sample real speech data;

[0049] The speech distortion degree corresponding to each second sound source is determined according to the energy ratio of the similarity energy value to the difference energy value.

[0050] In a third aspect, an embodiment of the present application provides a storage medium, wherein the computer-readable storage medium stores a plurality of instructions, which are suitable for loading by a processor to execute any of the above speech processing methods.

[0051] In a fourth aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above speech processing methods when executing the computer program.

[0052] In an embodiment of the present application, by acquiring fused speech data, feature extraction is performed on the fused speech data to obtain fused speech features, wherein the fused speech data includes speech data generated by mixing at least two sound sources; the fused speech features are input into a trained feature sensor to determine the speech feature mask of the speech data corresponding to each sound source; based on each of the speech feature masks, the speech feature representation of the speech data corresponding to each sound source is determined from the fused speech features; each of the speech feature representations is input into a trained feature sensor to determine the speech feature mask of the speech data corresponding to each sound source; based on each of the speech feature masks, the speech features of the speech data corresponding to each sound source are determined from the fused speech features; each of the speech features is input into a trained decoder to decode the speech data corresponding to each of the sound sources. Compared with the related art, in which blind source separation cannot be performed or the consistency of the separated speech is low due to the lack of labeled data during the training process, the embodiment of the present application uses a feature sensor to perceive the speech features of the speech data corresponding to different sound sources, thereby performing blind source separation according to the speech features, reducing the difference between the separated speech and the original speech, improving the consistency, and thus improving the accuracy of speech processing.

[0053] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or understood by practicing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1A scenario diagram of a speech processing system provided in an embodiment of the present application.

[0056] Figure 2 A flowchart of a speech processing method provided in an embodiment of the present application.

[0057] Figure 3 A schematic diagram of the architecture of the speech processing method provided in an embodiment of the present application.

[0058] Figure 4 This is a schematic diagram of the training architecture of the feature sensor in this application.

[0059] Figure 5 This is a schematic diagram of the training architecture of the feature sensor in this application.

[0060] Figure 6 A schematic diagram of the structure of a speech processing device provided in an embodiment of the present application.

[0061] Figure 7 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0063] It should be noted that in some processes described in the specification, claims and the above-mentioned drawings, multiple steps appearing in a specific order are included, but it should be clearly understood that these steps may not be executed in the order in which they appear in this document or may be executed in parallel. The step numbers are only used to distinguish different steps, and the numbers themselves do not represent any execution order. In addition, descriptions such as "first", "second" or "target" in this document are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0064] Before further describing the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are described. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations:

[0065] Artificial Intelligence (AI) is an important part of the intelligent discipline, attempting to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial Intelligence is a very broad science, including robotics, speech recognition, image recognition, natural language processing, expert systems, machine learning, computer vision, etc.

[0066] An encoder is a component or algorithm that converts input data into a specific encoding or feature representation. In computer science and machine learning, the role of an encoder is to compress input information, extract key features, or perform some form of transformation for subsequent processing, analysis, or transmission. The design and function of an encoder depends on the specific application scenario and task requirements. Its purpose is to make data easier to process, classify, generate, or compare with other data in a new representation.

[0067] For example, in deep learning, an encoder is usually part of a neural network that maps input images, text, or other data to a low-dimensional feature space to achieve an effective representation of the data. This encoded feature usually contains the key information of the input data while removing some redundant or unimportant parts.

[0068] Decoder is a neural network module corresponding to the encoder, which is mainly used to convert the encoded low-dimensional representation (usually generated by the encoder) back to the original data space or generate new data samples. The main functions and features include: 1. Reconstructing data: One of the main functions of the decoder is to reconstruct the original data from the encoded low-dimensional representation. For example, in an autoencoder, the encoder compresses the input data into a low-dimensional encoding, and the decoder tries to recover the output from this encoding as close to the original input as possible. By training the autoencoder, the decoder can learn how to map the encoded representation back to the original data space, thereby realizing data reconstruction. 2. Generate new data: The decoder can also be used to generate new data samples. In models such as generative adversarial networks (GANs) and variational autoencoders (VAEs), decoders are often combined with random noise or latent variables to generate new data samples that are similar to the training data. For example, in VAEs, the encoder maps the input data to a latent space, and then the decoder samples from this latent space and generates new data samples. By adjusting the value of the latent variable, the characteristics of the generated data can be controlled. 3. Working with the encoder: The decoder usually works closely with the encoder. The encoder converts the input data into a low-dimensional representation, while the decoder converts this low-dimensional representation back to the original data space or generates new data. This encoder-decoder structure is very effective in many tasks, such as image generation, machine translation, speech synthesis, etc. During training, the encoder and decoder are usually optimized together to minimize the reconstruction error or the difference between the generated data and the real data.

[0069] At present, in real environments, people are often surrounded by multiple sounds, such as mixed voices of multiple speakers, background noise and music. The selective attention mechanism enables people to focus on the voice of the target speaker in a complex noise environment. In some scenarios, such as remote conference scenarios, in order to enable participants to hear the voice of the target participant clearly, the blind source separation (BSS) algorithm is used to separate the voice of the target participant in a noisy environment from the noisy fusion voice, so that participants can focus on the voice of the target participant.

[0070] However, due to the lack of labeled data during the training process, the methods in the related art may result in low consistency between the separated speech and the original speech when the speech separation system performs blind source separation in real scenarios.

[0071] In order to solve the above problems, the embodiment of the present application obtains fused speech data, extracts features from the fused speech data, and obtains fused speech features, wherein the fused speech data includes speech data generated by mixing at least two sound sources; the fused speech features are input into the trained feature sensor to determine the speech feature mask of the speech data corresponding to each sound source; based on each of the speech feature masks, the speech feature representation of the speech data corresponding to each sound source is determined from the fused speech features; each of the speech feature representations is input into the trained feature sensor to determine the speech feature mask of the speech data corresponding to each sound source; based on each of the speech feature masks, the speech features of the speech data corresponding to each sound source are determined from the fused speech features; each of the speech features is input into the trained decoder to decode the speech data corresponding to each of the sound sources. Compared with the related art, the blind source separation cannot be performed or the consistency of the separated speech is low due to the lack of labeled data in the training process. The embodiment of the present application uses a feature sensor to perceive the speech features of the speech data corresponding to different sound sources, thereby performing blind source separation according to the speech features, reducing the difference between the separated speech and the original speech, and improving the consistency. Please continue to refer to the following specific embodiments for details.

[0072] See also Figure 1 , Figure 1 A schematic diagram of a scenario of a speech processing system provided in an embodiment of the present application, which includes a terminal 140, the Internet 130, a gateway 120, a computer device 110, etc.

[0073] The terminal 140 includes but is not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. In addition, it can be a single device or a collection of multiple devices. The terminal 140 can communicate with the Internet 130 in a wired or wireless manner to exchange data.

[0074] Computer equipment refers to a computer system that can provide certain services to the terminal 140. Compared with ordinary terminals 140, computer equipment 110 has higher requirements in terms of stability, security, performance, etc. Computer equipment 110 can be a high-performance computer in a network platform, a cluster of multiple high-performance computers, a part of a high-performance computer (such as a virtual machine), a combination of parts of multiple high-performance computers (such as virtual machines), etc.

[0075] The gateway 120 is also called an internetwork connector or a protocol converter. The gateway realizes network interconnection at the transport layer and is a computer system or device that acts as a converter. The gateway is a translator between two systems that use different communication protocols, data formats or languages, or even completely different architectures. At the same time, the gateway can also provide filtering and security functions. The message sent by the terminal 140 to the computer device 110 must be sent to the corresponding computer device 110 through the gateway 120. The message sent by the computer device 110 to the terminal 140 must also be sent to the corresponding terminal 140 through the gateway 120.

[0076] The data transmission method according to the embodiment of the present disclosure may be implemented on the computer device 110 .

[0077] It should be noted that Figure 1 The scenario diagram of the speech processing system shown is only an example. The speech processing system and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of image processing technology and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0078] In this embodiment, the description will be made from the perspective of a speech processing device, which may be specifically integrated into a computer device having a storage unit and a microprocessor installed therein and having computing capabilities.

[0079] See also Figure 2 , Figure 2 A flow chart of a speech processing method provided in an embodiment of the present application. The speech processing method comprises:

[0080] In step 201, fused speech data is acquired, and features are extracted from the fused speech data to obtain fused speech features, wherein the fused speech data includes speech data generated by mixing at least two sound sources.

[0081] The fused voice data is the voice data generated by mixing at least two sound sources. These sound sources can be the voices of different people or other audio recorded by at least two characters. For example, in a multi-person conversation scene, the fused voice data is an audio signal in which the voices of multiple speakers are mixed together. The fused voice data can be obtained from the actual environment through an audio acquisition device such as a microphone, or it can be read from an existing audio file.

[0082] Specifically, the process of extracting features from the fused speech data is performed by a trained encoder, and the encoder is mainly used to convert the original fused speech data into more abstract and representative fused speech features in this process.

[0083] Specifically, the feature extraction process of the encoder can be to slide the convolution kernel on the input data, perform weighted summation on the local area, and obtain a new feature map. Different convolution kernels can extract different features, such as components of different frequencies, different speech patterns, etc. As the convolution layer deepens, more advanced and abstract features can be extracted. The feature map output by the convolution layer is downsampled to reduce the feature dimension while retaining the main features.

[0084] In step 202, the fused speech features are input into the trained feature sensor to determine the speech feature mask of the speech data corresponding to each sound source.

[0085] Among them, the feature sensor is a trained neural network model, which is used to extract the speech feature mask of each sound source corresponding to the speech data from the fused speech features. The features are the syllables, rhythm, vocabulary and syntactic information of the speech in the audio data. Different people have different speaking characteristics, so the feature sensor is used to perceive the speech features of each sound source to form a speech feature mask.

[0086] Specifically, the input of the feature sensor is the fused speech feature, and the output is the speech feature mask with the same number as the sound sources. Each speech feature mask is a vector or matrix with the same dimension as the fused speech feature, which represents the part of the fused speech feature related to the speech feature of a certain sound source.

[0087] The process of determining the speech feature mask is as follows: when the fused speech features are input to the feature sensor, the feature sensor will perform a series of nonlinear transformations and calculations on the fused speech features. These transformations and calculations usually include convolution operations, fully connected layers, activation functions, etc. The feature sensor gradually adjusts its parameters by learning the speech feature patterns in the training data, so that the output speech feature mask can accurately represent the speech features of each sound source.

[0088] For example, if the speech feature of source A is that the rhythm is repeated sounds, and the speech feature of source B is that the rhythm is repeated rhymes, then for source A, the corresponding speech feature mask output by the feature sensor will have a larger value in the repeated sounds part, and a smaller value in the rhythm parts other than repeated sounds; for source B, the corresponding speech feature mask output by the feature sensor will have a larger value in the repeated rhymes part, and a smaller value in the rhythm parts other than repeated rhymes. In this way, the trained feature sensor can quickly determine the speech feature mask of the speech data corresponding to each source based on the input fusion speech features.

[0089] In step 203, based on each of the speech feature masks, a speech feature representation of the speech data corresponding to each sound source is determined from the fused speech features.

[0090] After the speech feature mask of each sound source is determined in step S202, the speech feature representation of the speech data corresponding to each sound source can be determined from the fused speech features. The speech feature representation is an abstract representation of the speech data of each sound source in terms of specific speech features. It contains important feature information of the speech data of the sound source, such as syllables, rhythm, vocabulary and syntactic information. This information is very important for subsequent speech separation and recognition.

[0091] Specifically, the specific method for determining the speech feature representation of the speech data corresponding to each sound source is as follows: since the speech feature mask is a vector or matrix with the same dimension as the fused speech feature, each element in the speech feature mask is the weight of the corresponding element in the fused speech feature, so the tensor product of the speech feature mask of each sound source and the fused speech feature can be calculated to obtain the result tensor, which is the speech feature representation of the speech data corresponding to each sound source. It highlights the part of the fused speech feature that is related to the speech feature of the sound source, and suppresses other irrelevant parts.

[0092] In step 204, each of the speech feature representations is input into a trained feature sensor to determine a speech feature mask of the speech data corresponding to each sound source.

[0093] Among them, the feature sensor is also a trained neural network model. Its function is to further extract the speech feature mask of the speech data corresponding to each sound source from the speech feature representation to ensure that the speech features determined according to the speech feature mask will not be distorted after being decoded into speech data.

[0094] The input of the feature sensor is the speech feature representation of each sound source, and the output is the speech feature mask with the same number of sound sources. Each speech feature mask is a vector or matrix with the same dimension as the speech feature representation, which represents the part of the speech feature representation related to the specific speech features of a certain sound source. Specifically, similar to the feature sensor, the feature sensor will also perform a series of nonlinear transformations and calculations on the input speech feature representation. The feature sensor will enable the output speech feature mask to more accurately represent the speech features of each sound source.

[0095] In step 205, the speech features of the speech data corresponding to each sound source are determined from the fused speech features based on each of the speech feature masks.

[0096] After the speech feature mask of each sound source is determined in step S204, the speech feature of the speech data corresponding to each sound source is determined again from the fused speech feature. The method is similar to determining the speech feature representation from the fused speech feature, and the tensor product calculation is performed on each speech feature mask and the fused speech feature to realize the element-level multiplication operation of each speech feature mask and the fused speech feature. The result is the speech feature of the speech data corresponding to each sound source, which more accurately represents the information of the speech data of each sound source in terms of specific speech features.

[0097] Specifically, speech features are a more specific and detailed representation of the speech data of each sound source. It contains various characteristic information of the speech data of the sound source, such as frequency, amplitude, phase, etc. This information is very critical for the final speech separation and decoding.

[0098] In step 206, each of the speech features is input into a trained decoder to decode the speech data corresponding to each of the sound sources.

[0099] The decoder is a trained neural network model that decodes the speech features of each sound source into corresponding speech data. The input of the decoder is the speech features of each sound source, and the output is the separated speech data corresponding to each sound source.

[0100] Specifically, the decoding process of the decoder includes: the decoder will perform a series of inverse transformations and calculations on the input speech features to restore the original speech signal. These inverse transformations and calculations usually include inverse convolution operations, upsampling, activation functions, etc. The decoder gradually adjusts its parameters by learning the relationship between the speech features in the training data and the original speech data, so that the output speech data can be as close to the original source speech data as possible. After being processed by the decoder, the separated speech data corresponding to each sound source is finally obtained, achieving the goal of speech separation.

[0101] In this way, by acquiring fused speech data, feature extraction is performed on the fused speech data to obtain fused speech features, wherein the fused speech data includes speech data generated by mixing at least two sound sources; the fused speech features are input into the trained feature sensor to determine the speech feature mask of the speech data corresponding to each sound source; based on each of the speech feature masks, the speech feature representation of the speech data corresponding to each sound source is determined from the fused speech features; each of the speech feature representations is input into the trained feature sensor to determine the speech feature mask of the speech data corresponding to each sound source; based on each of the speech feature masks, the speech features of the speech data corresponding to each sound source are determined from the fused speech features; each of the speech features is input into the trained decoder to decode the speech data corresponding to each of the sound sources. The speech features of the speech data corresponding to different sound sources are perceived by the feature sensor, so as to perform blind source separation according to the speech features, reduce the difference between the separated speech and the original speech, improve consistency, and thus improve the accuracy of speech processing.

[0102] For details, please refer to Figure 3 , Figure 3 A schematic diagram of the architecture of the speech processing method provided for the embodiment of the present application. When the fused speech data is obtained, the fused speech data is feature extracted by the trained encoder to obtain the fused speech features; the fused speech features are analyzed by the trained feature sensor to determine the speech feature mask of the speech data corresponding to each sound source. Based on these speech feature masks, the speech feature representation of the speech data corresponding to each sound source can be determined from the fused speech features. Then, each speech feature representation is input into the trained feature sensor, and the feature sensor determines the speech feature mask of the speech data corresponding to each sound source. Based on these speech feature masks, the speech features of the speech data corresponding to each sound source are determined from the fused speech features. Finally, the speech features of each sound source are input into the trained decoder, and the decoder decodes these speech features to separate the speech data corresponding to each sound source. By gradually extracting the independent speech information of each sound source from the fused speech, effective separation of speech data is achieved.

[0103] In some implementations, before inputting the fused speech features into the trained feature sensor to determine the speech feature mask of the speech data corresponding to each sound source, the method further includes:

[0104] (1) obtaining first sample fusion speech features of first sample fusion speech data and first sample real speech data of each first sound source included in the first sample fusion speech data;

[0105] (2) inputting the first sample fusion speech feature into the feature sensor to be trained to determine the first sample speech feature mask of the speech data corresponding to each first sound source;

[0106] (3) determining a first predicted feature of the speech data corresponding to each first sound source by using each of the first sample speech feature masks and the predictor to be trained;

[0107] (4) inputting each of the first sample real speech data into the trained feature extraction model to extract the first real feature of the speech data corresponding to each first sound source;

[0108] (5) determining a first loss value according to a first predicted characteristic of each first sound source and a corresponding first real characteristic;

[0109] (6) When the first loss value is greater than the first preset loss value, the network parameters of the feature sensor to be trained and the predictor to be trained are adjusted according to the first loss value, and the step of inputting the first sample fusion speech feature into the feature sensor to be trained and determining the first sample speech feature mask of the speech data corresponding to each first sound source is returned until the first loss value is less than or equal to the first preset loss value, thereby obtaining the initially trained feature sensor and the trained predictor.

[0110] Among them, see Figure 4 , Figure 4 This is a schematic diagram of the training architecture of the feature sensor for this application. The first sample fused speech data is speech data mixed from multiple first sound sources. The first sample fused speech feature is obtained through the encoder to be trained. This feature is an abstract representation of the fused speech data. At the same time, it is also necessary to obtain the first sample real speech data of each first sound source as the real value for subsequent comparison and calculation of loss.

[0111] The feature perceptron to be trained is a neural network model, which aims to identify the speech features related to each first sound source from the fused speech features. After inputting the first sample fused speech features, the feature perceptron outputs the first sample speech feature mask corresponding to each first sound source after a series of calculations and learning. This mask can filter out the part of the fused speech features related to a specific first sound source.

[0112] The first predicted feature of the speech data corresponding to each first sound source is determined by using each first sample speech feature mask and the predictor to be trained. The first sample speech feature mask obtained is further processed in combination with the predictor to be trained. The predictor predicts the first predicted feature of the speech data corresponding to each first sound source based on the mask and other information. The predicted feature is an estimate of the real feature of the first sound source.

[0113] Each first sample real speech data is input into the trained feature extraction model to extract the first real feature of the speech data corresponding to each first sound source.

[0114] For each first sample real speech data of the first sound source, the trained feature extraction model is used to extract its real features. This real feature is used as the standard for subsequent loss calculation to measure the accuracy of the predicted features.

[0115] A first loss value is determined based on the first predicted feature of each first sound source and the corresponding first real feature. By comparing the first predicted feature and the first real feature of each first sound source, a certain loss function (such as mean square error, etc.) can be used to calculate the first loss value. The loss value reflects the degree of difference between the predicted feature and the real feature.

[0116] When the first loss value is greater than the first preset loss value, the network parameters of the feature perceptron to be trained and the predictor to be trained are adjusted according to the first loss value, and the step of inputting the first sample fusion speech feature into the feature perceptron to be trained and determining the first sample speech feature mask of the speech data corresponding to each first sound source is returned until the first loss value is less than or equal to the first preset loss value, thereby obtaining the initially trained feature perceptron and the trained predictor.

[0117] If the first loss value is greater than the preset loss value, it means that the performance of the current feature sensor and predictor is not good enough. At this time, the network parameters of the encoder, feature sensor, and predictor are adjusted according to the loss value through back propagation and other algorithms, so that they can learn and predict better. Then iterate and optimize continuously until the first loss value is less than or equal to the preset loss value. At this point, the encoder, feature sensor, and predictor are preliminarily trained, and they can more accurately extract the characteristics of each first sound source from the fused speech data.

[0118] In this way, by inputting the first sample real speech data into the trained feature extraction model, the first real feature is extracted as a standard for measuring the prediction accuracy. By comparing the first predicted feature and the first real feature, a first loss value is calculated using a loss function, and the loss value reflects the degree of difference between the prediction and the reality. When the first loss value is greater than the first preset loss value, the network parameters of the feature sensor and the predictor are adjusted by the back propagation algorithm, and the optimization is continuously iterated until the loss value is less than or equal to the preset value. The final preliminarily trained feature sensor and trained predictor can more accurately extract the characteristics of each first sound source from the fused speech data, which helps to improve the feature perception ability of the feature sensor.

[0119] In some implementations, determining the first predicted feature of the speech data corresponding to each first sound source by using each of the first sample speech feature masks and the predictor to be trained includes:

[0120] (3.1) determining a first sample speech feature representation of speech data corresponding to each sound source from the first sample fused speech features based on each of the first sample speech feature masks;

[0121] (3.2) Inputting each of the first sample speech feature representations into the predictor to be trained to obtain a first predicted feature of the speech data corresponding to each first sound source.

[0122] Among them, the first predicted feature of the speech data corresponding to each sound source is to first calculate the tensor product of each first sample speech feature mask and the first sample fusion speech feature to obtain the first sample speech feature representation of the speech data corresponding to each sound source; thereby, the first sample speech feature representation is input into the predictor to be trained to obtain the first predicted feature of the speech data corresponding to each first sound source predicted by the predictor.

[0123] In some implementations, determining the first loss value according to the first predicted feature and the corresponding first real feature of each first sound source includes:

[0124] (5.1) determining the similarity between the first predicted feature of each first sound source and the corresponding first real feature;

[0125] (5.2) selecting a target first sound source with the greatest similarity from each first sound source;

[0126] (5.3) obtaining a similarity between the first predicted feature of the target first sound source and the first predicted feature of each other first sound source, to obtain a plurality of first predicted similarities, wherein the other first sound sources are first sound sources other than the target first sound source in each first sound source;

[0127] (5.4) Determine a first loss value based on a target similarity between the first predicted feature of the target first sound source and the corresponding first real feature, and a plurality of the first predicted similarities.

[0128] Among them, the similarity between the first predicted feature of each first sound source obtained by the predictor to be trained and the first real feature extracted from the actual real speech data by the trained feature extraction model is measured. Various similarity measurement methods can be used, such as cosine similarity, Pearson correlation coefficient, etc. This similarity value reflects the closeness between the prediction result and the actual situation. The higher the similarity, the more accurate the prediction.

[0129] After calculating the similarity between the first predicted feature of each first sound source and the corresponding first real feature, the sound source with the highest similarity is found from all the first sound sources as the target first sound source. The purpose of this step is to determine which sound source's prediction result is closest to the real situation in the current state, so as to further analyze the information related to the sound source later.

[0130] After determining the target first sound source, the similarity between the first predicted feature of the target first sound source and the first predicted features of all other non-target first sound sources is calculated. In this way, multiple first predicted similarities can be obtained. These similarity values ​​can reflect the degree of difference between the target first sound source and other sound sources in the predicted features, which is helpful to evaluate the uniqueness of the target first sound source and the degree of distinction from other sound sources.

[0131] Finally, the first loss value is determined by comprehensively considering the similarity between the predicted characteristics and the real characteristics of the target first sound source (target similarity) and the predicted similarity between the target first sound source and other sound sources (multiple first predicted similarities). One possible approach is to design a loss function that considers the relationship between the target similarity and multiple first predicted similarities, so that the target similarity is as high as possible, while the predicted similarity between the target first sound source and other sound sources is as low as possible, so as to ensure the prediction accuracy and uniqueness of the target first sound source. By adjusting the parameters of the feature sensor and predictor to be trained, this loss value is minimized, thereby improving the performance of the model.

[0132] Specifically, the calculation method of the first loss value can refer to the following formula:

[0133]

[0134] The noise contrast estimation (NCE) loss function is used. Its purpose is to learn effective feature representation by maximizing the similarity between positive sample pairs and minimizing the similarity between negative sample pairs. i The first real characteristic of the target first sound source, is the first prediction characteristic of the target first sound source, ω is the temperature coefficient, is any similarity function, is the target similarity between the first predicted feature of the target first sound source and the corresponding first real feature, Q U For each first sound source, the first predicted feature, is the similarity between the first predicted feature of the target first sound source and the first predicted feature of each first sound source. The value of the similarity function is mapped to the positive real number domain through the exponential function exp(), so that the sample pairs with higher similarity have larger exponential values. It is the result of the exponential transformation of the sum of the similarities between the first predicted feature of the target first sound source and the first predicted feature of each other first sound source. The value of the entire loss function. By taking the logarithm, the division operation can be converted into a subtraction operation, which is convenient for calculation and optimization. The goal of the loss function is to make its value as small as possible, which means that the similarity of the positive sample pair should be as high as possible, and the similarity with the negative sample pair should be as low as possible.

[0135] In some implementations, before inputting each of the speech feature representations into the trained feature sensor to determine the speech feature mask of the speech data corresponding to each sound source, the method further includes:

[0136] (7) obtaining second sample fused speech features of the second sample fused speech data, and second sample real speech data of each second sound source included in the second sample fused speech data;

[0137] (8) inputting the second sample fusion speech feature into the preliminarily trained feature sensor to determine a second sample speech feature mask of the speech data corresponding to each second sound source;

[0138] (9) determining a second sample speech feature representation of the speech data corresponding to each second sound source from the second sample fused speech features based on each of the second sample speech feature masks;

[0139] (10) inputting each of the second sample speech feature representations into a feature sensor to be trained to determine a sample speech feature mask of speech data corresponding to each second sound source;

[0140] (11) determining, based on each of the sample speech feature masks, from the second sample fused speech features, sample speech features of the speech data corresponding to each second sound source;

[0141] (12) inputting each of the sample speech features into a decoder to be trained, and decoding the second sample predicted speech data corresponding to each second sound source;

[0142] (13) determining a second loss value based on the second sample predicted speech data of each second sound source and the corresponding second sample real speech data;

[0143] (14) When the second loss value is greater than the second preset loss value, the network parameters of the feature sensor to be trained are adjusted according to the second loss value, and the step of inputting the second sample fusion speech feature into the feature sensor after preliminary training and determining the second sample speech feature mask of the speech data corresponding to each second sound source is returned until the second loss value is less than or equal to the second preset loss value, thereby obtaining the trained feature sensor, the trained feature sensor and the trained decoder.

[0144] Among them, see Figure 5 , Figure 5 This is a schematic diagram of the training architecture for the feature sensor of this application. For the training process of the feature sensor, a new set of sample data will be prepared. The second sample fused speech data is the speech data of multiple second sound sources mixed together. Its fused speech features are extracted by a specific method, and the real speech data of each individual second sound source is obtained at the same time, providing a basis for subsequent training and evaluation. The new fused speech features are processed using the feature sensor obtained through preliminary training in the previous steps. The feature sensor generates a corresponding speech feature mask for each second sound source through learning and calculation, which is used to filter out the specific part of the fused speech feature related to the sound source.

[0145] Using the obtained speech feature mask, the speech feature representation of each second sound source is extracted from the fused speech features. This representation highlights the features related to the sound source and provides more specific information for further analysis and processing.

[0146] The feature perceptron to be trained takes the speech feature representation as input, and determines the sample speech feature mask of each second sound source through learning and calculation. This mask is more accurate than the speech feature mask for the specific speech features of each sound source.

[0147] The sample speech feature mask is used to extract the sample speech features of each second sound source from the fused speech features, and these features more accurately reflect the unique characteristics of each sound source.

[0148] The decoder to be trained takes the sample speech features as input, and attempts to recover the predicted speech data of each second sound source through inverse transformation and calculation, thereby realizing the conversion from features to speech.

[0149] By comparing the predicted speech data and the real speech data of each second sound source, a second loss value is calculated using a suitable loss function. This loss value reflects the performance of the decoder and the feature perceptron, that is, the degree of difference between the predicted speech and the real speech.

[0150] If the second loss value is greater than the preset value, it means that the current feature sensor and decoder performance is not good enough. According to the loss value, the network parameters of the feature sensor to be trained are adjusted through algorithms such as back propagation, and then the subsequent steps are re-executed, and the optimization is continuously iterated until the loss value is less than or equal to the preset value. At this point, the trained encoder, feature sensor, feature sensor and decoder are obtained, which can more accurately separate the speech data of each second sound source from the fused speech data, so that the separated speech data will not be distorted or the possibility of distortion is reduced.

[0151] In some implementations, determining the second loss value based on the predicted speech sample data of each second sound source and the corresponding second sample real speech data includes:

[0152] (13.1) determining the speech distortion corresponding to each second sound source according to the second sample predicted speech data of each second sound source and the corresponding second sample real speech data;

[0153] (13.2) A target speech distortion degree with the minimum speech distortion degree is selected from each of the speech distortion degrees, and the target speech distortion degree is determined as the second loss value.

[0154] Among them, the speech distortion is used as the loss function for training the feature perceptron. Therefore, the speech distortion corresponding to each second sound source is determined according to the second sample predicted speech data of each second sound source and the corresponding second sample real speech data; after obtaining the speech distortion corresponding to each second sound source, the minimum speech distortion is selected as the target speech distortion. This is because during the training process, it is hoped to find the parameter setting that makes the overall prediction result closest to the actual situation. This minimum speech distortion is determined as the second loss value, which is used to judge the performance of the model and adjust the model parameters in the subsequent process. If the second loss value is greater than the second preset loss value, it means that the prediction result of the model is quite different from the real data, and the model parameters need to be further adjusted to reduce the distortion; if the second loss value is less than or equal to the second preset loss value, it means that the performance of the model is good and the training can be stopped.

[0155] In some implementations, determining the speech distortion corresponding to each second sound source according to the second sample predicted speech data of each second sound source and the corresponding second sample real speech data includes:

[0156] (13.1.1) For each second sound source, calculate a similarity energy value between similar parts of the corresponding second sample predicted speech data and the corresponding second sample real speech data;

[0157] (13.1.2) calculating a difference energy value of a difference part between the corresponding second sample predicted speech data and the corresponding second sample real speech data;

[0158] (13.1.3) Determine the speech distortion corresponding to each second sound source based on the energy ratio between the similarity energy value and the difference energy value.

[0159] The speech distortion degree corresponding to each second sound source can be realized by referring to the following formula:

[0160]

[0161] Among them, L SI-SDRIt is the scale-invariant signal-to-distortion ratio loss function, which is used to measure the second sample prediction speech data The difference between the real speech data s and the second sample, where the numerator is is the energy of the normalized projection of the second sample predicted speech data in the direction of the second sample real speech data, that is, the similarity energy value of the similar part, and the denominator is the energy difference between the normalized projection and the predicted signal, that is, the difference energy value of the difference part. According to this formula, the speech distortion corresponding to each second sound source can be calculated.

[0162] The specific implementation of the above steps can be found in the previous embodiments, which will not be described in detail here.

[0163] In order to better implement the speech processing method provided in the embodiment of the present application, the embodiment of the present application also provides a device based on the above speech processing method. The meanings of the nouns are the same as those in the above speech processing method, and the specific implementation details can refer to the description in the method embodiment.

[0164] See also Figure 6 , Figure 6 The structure diagram of the speech processing device provided in the embodiment of the present application is applied to a computer device. The speech processing device may include a feature extraction unit 601, a feature perception unit 602, a first determination unit 603, a feature perception unit 604, a second determination unit 605, and a decoding unit 606.

[0165] The feature extraction unit 601 is used to obtain fused speech data, perform feature extraction on the fused speech data, and obtain fused speech features, wherein the fused speech data includes speech data generated by mixing at least two sound sources;

[0166] A feature perception unit 602, used for inputting the fused speech feature into the trained feature perception unit to determine the speech feature mask of the speech data corresponding to each sound source;

[0167] A first determining unit 603 is used to determine the speech characteristic representation of the speech data corresponding to each sound source from the fused speech features based on each of the speech characteristic masks;

[0168] A feature perception unit 604 is used to input each of the speech feature representations into a trained feature perception unit to determine a speech feature mask of the speech data corresponding to each sound source;

[0169] A second determining unit 605 is used to determine the speech features of the speech data corresponding to each sound source from the fused speech features based on each of the speech feature masks;

[0170] The decoding unit 606 is used to input each of the speech features into a trained decoder to decode the speech data corresponding to each of the sound sources.

[0171] In some embodiments, the apparatus further comprises:

[0172] Acquire a first sample fused speech feature of the first sample fused speech data, and first sample real speech data of each first sound source included in the first sample fused speech data;

[0173] Inputting the first sample fusion speech feature into the feature sensor to be trained, and determining the first sample speech feature mask of the speech data corresponding to each first sound source;

[0174] Determining a first predicted feature of the speech data corresponding to each first sound source by using each of the first sample speech feature masks and the predictor to be trained;

[0175] Inputting each of the first sample real speech data into the trained feature extraction model to extract the first real feature of the speech data corresponding to each first sound source;

[0176] Determining a first loss value according to a first predicted characteristic of each first sound source and a corresponding first real characteristic;

[0177] When the first loss value is greater than the first preset loss value, the network parameters of the feature sensor to be trained and the predictor to be trained are adjusted according to the first loss value, and the step of inputting the first sample fusion speech feature into the feature sensor to be trained and determining the first sample speech feature mask of the speech data corresponding to each first sound source is returned until the first loss value is less than or equal to the first preset loss value, thereby obtaining the initially trained feature sensor and the trained predictor.

[0178] In some embodiments, an apparatus comprises:

[0179] Determining a first sample speech characteristic representation of speech data corresponding to each sound source from the first sample fused speech features based on each of the first sample speech characteristic masks;

[0180] Each of the first sample speech feature representations is input into the predictor to be trained to obtain a first predicted feature of the speech data corresponding to each first sound source.

[0181] In some embodiments, an apparatus comprises:

[0182] Determining a similarity between a first predicted feature of each first sound source and a corresponding first real feature;

[0183] Selecting a target first sound source with the greatest similarity from each first sound source;

[0184] Obtaining a similarity between the first predicted feature of the target first sound source and the first predicted feature of each other first sound source to obtain a plurality of first predicted similarities, wherein the other first sound sources are first sound sources other than the target first sound source in each first sound source;

[0185] A first loss value is determined according to a target similarity between a first predicted feature of the target first sound source and a corresponding first real feature, and a plurality of the first predicted similarities.

[0186] In some embodiments, the apparatus further comprises:

[0187] Acquire second sample fused speech features of the second sample fused speech data, and second sample real speech data of each second sound source included in the second sample fused speech data;

[0188] Inputting the second sample fusion speech feature into the preliminarily trained feature sensor to determine the second sample speech feature mask of the speech data corresponding to each second sound source;

[0189] Determining a second sample speech characteristic representation of the speech data corresponding to each second sound source from the second sample fused speech features based on each second sample speech characteristic mask;

[0190] Inputting each of the second sample speech feature representations into a feature sensor to be trained, and determining a sample speech feature mask of speech data corresponding to each second sound source;

[0191] Determining, from the second sample fused speech features based on each of the sample speech feature masks, sample speech features of the speech data corresponding to each second sound source;

[0192] Inputting each of the sample speech features into a decoder to be trained, and decoding the second sample predicted speech data corresponding to each second sound source;

[0193] Determine a second loss value based on the second sample predicted speech data of each second sound source and the corresponding second sample real speech data;

[0194] When the second loss value is greater than the second preset loss value, the network parameters of the feature sensor to be trained are adjusted according to the second loss value, and the step of inputting the second sample fusion speech feature into the feature sensor after preliminary training and determining the second sample speech feature mask of the speech data corresponding to each second sound source is returned until the second loss value is less than or equal to the second preset loss value, thereby obtaining the trained feature sensor, the trained feature sensor and the trained decoder.

[0195] In some embodiments, an apparatus comprises:

[0196] Determine the speech distortion corresponding to each second sound source according to the second sample predicted speech data of each second sound source and the corresponding second sample real speech data;

[0197] A target speech distortion degree with the minimum speech distortion degree is selected from each of the speech distortion degrees, and the target speech distortion degree is determined as the second loss value.

[0198] In some embodiments, an apparatus comprises:

[0199] For each second sound source, calculating a similarity energy value of a similar part between the corresponding second sample predicted speech data and the corresponding second sample real speech data;

[0200] Calculate the difference energy value of the difference part between the corresponding second sample predicted speech data and the corresponding second sample real speech data;

[0201] The speech distortion degree corresponding to each second sound source is determined according to the energy ratio of the similarity energy value to the difference energy value.

[0202] The specific implementation of each of the above units can be found in the previous embodiments, which will not be described in detail here.

[0203] As can be seen from the above, the embodiment of the present application obtains fused speech data through the feature extraction unit 601, performs feature extraction on the fused speech data, and obtains fused speech features, wherein the fused speech data includes speech data generated by mixing at least two sound sources; the feature perception unit 602 inputs the fused speech features into the trained feature sensor, and determines the speech feature mask of the speech data corresponding to each sound source; the first determination unit 603 determines the speech feature representation of the speech data corresponding to each sound source from the fused speech features based on each of the speech feature masks; the feature perception unit 604 inputs each of the speech feature representations into the trained feature sensor, and determines the speech feature mask of the speech data corresponding to each sound source; the second determination unit 605 determines the speech features of the speech data corresponding to each sound source from the fused speech features based on each of the speech feature masks; the decoding unit 606 inputs each of the speech features into the trained decoder, and decodes the speech data corresponding to each of the sound sources.

[0204] In this way, by acquiring fused speech data, feature extraction is performed on the fused speech data to obtain fused speech features, wherein the fused speech data includes speech data generated by mixing at least two sound sources; the fused speech features are input into the trained feature sensor to determine the speech feature mask of the speech data corresponding to each sound source; based on each of the speech feature masks, the speech feature representation of the speech data corresponding to each sound source is determined from the fused speech features; each of the speech feature representations is input into the trained feature sensor to determine the speech feature mask of the speech data corresponding to each sound source; based on each of the speech feature masks, the speech features of the speech data corresponding to each sound source are determined from the fused speech features; each of the speech features is input into the trained decoder to decode the speech data corresponding to each of the sound sources. The speech features of the speech data corresponding to different sound sources are perceived by the feature sensor, so as to perform blind source separation according to the speech features, reduce the difference between the separated speech and the original speech, improve consistency, and thus improve the accuracy of speech processing.

[0205] The specific implementation of each of the above units can be found in the previous embodiments, which will not be described in detail here.

[0206] Reference Figure 7 , Figure 7The block diagram of the structure of part of the computer device 110 for implementing the embodiment of the present disclosure. The computer device 110 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 622 (for example, one or more processors) and memory 632, and one or more storage media 630 (for example, one or more mass storage devices) storing application programs 642 or data 644. Among them, the memory 632 and the storage medium 630 can be temporary storage or permanent storage. The program stored in the storage medium 630 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server 600. Furthermore, the central processing unit 622 can be configured to communicate with the storage medium 630 and execute a series of instruction operations in the storage medium 630 on the server 600.

[0207] The computer device 110 may also include one or more power supplies 626, one or more wired or wireless network interfaces 650, one or more input and output interfaces 658, and / or one or more operating systems 641, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0208] The central processor 622 in the computer device 110 may be used to execute the speech processing method of the embodiment of the present disclosure, for example:

[0209] Acquire fused voice data, perform feature extraction on the fused voice data, and obtain fused voice features, wherein the fused voice data includes voice data generated by mixing at least two sound sources;

[0210] Inputting the fused speech features into the trained feature sensor to determine the speech feature mask of the speech data corresponding to each sound source;

[0211] Determining the speech characteristic representation of the speech data corresponding to each sound source from the fused speech features based on each of the speech characteristic masks;

[0212] Inputting each of the speech feature representations into a trained feature sensor to determine a speech feature mask of the speech data corresponding to each sound source;

[0213] Determining the speech features of the speech data corresponding to each sound source from the fused speech features based on each of the speech feature masks;

[0214] Each of the speech features is input into a trained decoder to decode the speech data corresponding to each of the sound sources.

[0215] The embodiments of the present disclosure further provide a computer-readable storage medium, which is used to store program codes, and the program codes are used to execute the speech processing methods of the aforementioned embodiments.

[0216] The present disclosure also provides a computer program product, which includes a computer program. The processor of a computer device reads and executes the computer program, so that the computer device executes the above-mentioned speech processing method. For example:

[0217] Acquire fused voice data, perform feature extraction on the fused voice data, and obtain fused voice features, wherein the fused voice data includes voice data generated by mixing at least two sound sources;

[0218] Inputting the fused speech features into the trained feature sensor to determine the speech feature mask of the speech data corresponding to each sound source;

[0219] Determining the speech characteristic representation of the speech data corresponding to each sound source from the fused speech features based on each of the speech characteristic masks;

[0220] Inputting each of the speech feature representations into a trained feature sensor to determine a speech feature mask of the speech data corresponding to each sound source;

[0221] Determining the speech features of the speech data corresponding to each sound source from the fused speech features based on each of the speech feature masks;

[0222] Each of the speech features is input into a trained decoder to decode the speech data corresponding to each of the sound sources.

[0223] In addition, the terms "comprises" and "includes" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to such process, method, product or apparatus.

[0224] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0225] It should be understood that in the description of the embodiments of the present application, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to not include the number, and above, below, within, etc. are understood to include the number.

[0226] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0227] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0228] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0229] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0230] It should also be understood that the various implementations provided in the embodiments of the present application can be combined arbitrarily to achieve different technical effects.

[0231] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0232] The above is a specific description of the implementation method of the present application, but the present application is not limited to the above-mentioned implementation method. Technical personnel familiar with the field can also make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A speech processing method, characterized in that: include: Acquire fused voice data, perform feature extraction on the fused voice data, and obtain fused voice features, wherein the fused voice data includes voice data generated by mixing at least two sound sources; Inputting the fused speech features into the trained feature sensor to determine the speech feature mask of the speech data corresponding to each sound source; Determining the speech characteristic representation of the speech data corresponding to each sound source from the fused speech features based on each of the speech characteristic masks; Inputting each of the speech feature representations into a trained feature sensor to determine a speech feature mask of the speech data corresponding to each sound source; Determining the speech features of the speech data corresponding to each sound source from the fused speech features based on each of the speech feature masks; Each of the speech features is input into a trained decoder to decode the speech data corresponding to each of the sound sources.

2. The speech processing method according to claim 1, characterized in that: Before inputting the fused speech features into the trained feature sensor to determine the speech feature mask of the speech data corresponding to each sound source, the method further includes: Acquire a first sample fused speech feature of the first sample fused speech data, and first sample real speech data of each first sound source included in the first sample fused speech data; Inputting the first sample fusion speech feature into the feature sensor to be trained, and determining the first sample speech feature mask of the speech data corresponding to each first sound source; Determining a first predicted feature of the speech data corresponding to each first sound source by using each of the first sample speech feature masks and the predictor to be trained; Inputting each of the first sample real speech data into the trained feature extraction model to extract the first real feature of the speech data corresponding to each first sound source; Determining a first loss value according to a first predicted characteristic of each first sound source and a corresponding first real characteristic; When the first loss value is greater than the first preset loss value, the network parameters of the feature sensor to be trained and the predictor to be trained are adjusted according to the first loss value, and the step of inputting the first sample fusion speech feature into the feature sensor to be trained and determining the first sample speech feature mask of the speech data corresponding to each first sound source is returned until the first loss value is less than or equal to the first preset loss value, thereby obtaining the initially trained feature sensor and the trained predictor.

3. The speech processing method according to claim 2, characterized in that: The step of determining the first predicted feature of the speech data corresponding to each first sound source by using each of the first sample speech feature masks and the predictor to be trained comprises: Determining a first sample speech characteristic representation of speech data corresponding to each sound source from the first sample fused speech features based on each of the first sample speech characteristic masks; Each of the first sample speech feature representations is input into the predictor to be trained to obtain a first predicted feature of the speech data corresponding to each first sound source.

4. The speech processing method according to claim 2, characterized in that: The determining of the first loss value according to the first predicted characteristic and the corresponding first real characteristic of each first sound source includes: Determining a similarity between a first predicted feature of each first sound source and a corresponding first real feature; Selecting a target first sound source with the greatest similarity from each first sound source; Obtaining a similarity between the first predicted feature of the target first sound source and the first predicted feature of each other first sound source to obtain a plurality of first predicted similarities, wherein the other first sound sources are first sound sources other than the target first sound source in each first sound source; A first loss value is determined according to a target similarity between a first predicted feature of the target first sound source and a corresponding first real feature, and a plurality of the first predicted similarities.

5. The speech processing method according to claim 2, characterized in that: Before inputting each of the speech feature representations into the trained feature sensor to determine the speech feature mask of the speech data corresponding to each sound source, the method further includes: Acquire second sample fused speech features of the second sample fused speech data, and second sample real speech data of each second sound source included in the second sample fused speech data; Inputting the second sample fusion speech feature into the preliminarily trained feature sensor to determine the second sample speech feature mask of the speech data corresponding to each second sound source; Determining a second sample speech characteristic representation of the speech data corresponding to each second sound source from the second sample fused speech features based on each second sample speech characteristic mask; Inputting each of the second sample speech feature representations into a feature sensor to be trained, and determining a sample speech feature mask of speech data corresponding to each second sound source; Determining, from the second sample fused speech features based on each of the sample speech feature masks, sample speech features of the speech data corresponding to each second sound source; Inputting each of the sample speech features into a decoder to be trained, and decoding the second sample predicted speech data corresponding to each second sound source; Determine a second loss value based on the second sample predicted speech data of each second sound source and the corresponding second sample real speech data; When the second loss value is greater than the second preset loss value, the network parameters of the feature sensor to be trained are adjusted according to the second loss value, and the step of inputting the second sample fusion speech feature into the feature sensor after preliminary training and determining the second sample speech feature mask of the speech data corresponding to each second sound source is returned until the second loss value is less than or equal to the second preset loss value, thereby obtaining the trained feature sensor, the trained feature sensor and the trained decoder.

6. The speech processing method according to claim 5, characterized in that: The determining of the second loss value based on the predicted speech sample data of each second sound source and the corresponding second sample real speech data includes: Determine the speech distortion corresponding to each second sound source according to the second sample predicted speech data of each second sound source and the corresponding second sample real speech data; A target speech distortion degree with the minimum speech distortion degree is selected from each of the speech distortion degrees, and the target speech distortion degree is determined as the second loss value.

7. The speech processing method according to claim 6, characterized in that: The step of determining the speech distortion corresponding to each second sound source according to the second sample predicted speech data of each second sound source and the corresponding second sample real speech data comprises: For each second sound source, calculating a similarity energy value of a similar part between the corresponding second sample predicted speech data and the corresponding second sample real speech data; Calculate the difference energy value of the difference part between the corresponding second sample predicted speech data and the corresponding second sample real speech data; The speech distortion degree corresponding to each second sound source is determined according to the energy ratio of the similarity energy value to the difference energy value.

8. A speech processing device, characterized in that: include: A feature extraction unit is used to obtain fused speech data, perform feature extraction on the fused speech data, and obtain fused speech features, wherein the fused speech data includes speech data generated by mixing at least two sound sources; A feature perception unit, used for inputting the fused speech feature into a trained feature perception unit to determine a speech feature mask of speech data corresponding to each sound source; A first determining unit, configured to determine a speech feature representation of speech data corresponding to each sound source from the fused speech features based on each of the speech feature masks; A feature perception unit, used for inputting each of the speech feature representations into a trained feature perception unit to determine a speech feature mask of the speech data corresponding to each sound source; A second determining unit, configured to determine the speech features of the speech data corresponding to each sound source from the fused speech features based on each of the speech feature masks; The decoding unit is used to input each of the speech features into a trained decoder to decode the speech data corresponding to each of the sound sources.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the speech processing method according to any one of claims 1 to 7.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the speech processing method according to any one of claims 1 to 7 is implemented.

Images