Voice separation method and device, electronic equipment, storage medium and product

Abstract

The invention provides a voice separation method which can be applied to the technical field of audio and video. The method comprises the following steps: acquiring mixed voice data of a plurality of speaking objects, and determining the actual pitch of each speaking object; performing separation processing on the mixed voice data to obtain an initial voice signal of each speaking object; performing feature processing on each initial voice signal to obtain pitch probability distribution of each speaking object; obtaining an estimated pitch of each speaking object through the pitch probability distribution of each speaking object; if the error between the estimated pitch and the actual pitch does not meet the preset error range, adjusting the pitch probability distribution for multiple times until the error between the new estimated pitch and the actual pitch meets the preset error range; and updating the initial voice signal of each speaking object through the latest adjusted pitch probability distribution to obtain a target voice signal of each speaking object. The invention further provides a voice separation device, electronic equipment, a storage medium and a product.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio and video, and more particularly, to a speech separation method and device, electronic equipment, storage medium and product. BACKGROUND

[0002] Speech separation aims to separate the speech of different speakers in a speech signal containing multiple speakers, so as to process each speech signal independently. In the past few decades, speech separation has been widely studied in the field of signal processing.

[0003] According to the number of microphones, speech separation methods can be divided into single-channel and multi-channel. Single-channel methods process a single microphone and distinguish different speakers according to their inherent characteristics. Multi-channel methods use additional spatial information of speech, which has the advantage of suppressing interference in different directions. However, under certain conditions, the role of spatial information is not obvious, while single-channel technology has a wider range of applications, as it requires lower hardware costs and resources, and shows good prospects for development.

[0004] In addition, single-channel speech separation can be divided into unsupervised and supervised methods according to whether it is supervised. Traditional unsupervised algorithms are limited by performance, while supervised algorithms can effectively handle various complex scenarios by learning complex deep features. In recent years, they have been favored by researchers.

[0005] Currently, the processing methods for speech separation mainly include non-deep learning and deep learning. Non-deep learning is based on the characteristics of the target speaker, such as pitch and signal continuity, and formulates the separation problem as a matrix decomposition problem, which decomposes the time-frequency spectrum representation by learning a non-negative reconstruction basis and weight. The speech separation model based on deep learning relies on a supervised training framework and extracts features through convolution and pooling to obtain the spectral features of the speaker. However, mixed speech containing speakers of the same gender has similar spectral and speech features, and the above two methods are difficult to make accurate separation. SUMMARY

[0006] In view of the above problems, the present application provides a speech separation method, device, electronic equipment, storage medium and product.

[0007] The application provides a speech separation method, including: obtaining mixed speech data containing multiple speaking objects, determining the actual pitch of each speaking object; separating the mixed speech data to obtain the initial speech signal of each speaking object; performing feature processing on the initial speech signal of each speaking object to obtain the pitch probability distribution of each speaking object; obtaining the estimated pitch of each speaking object through the pitch probability distribution of each speaking object; if the error between the estimated pitch and the actual pitch does not meet the preset error range, adjusting the pitch probability distribution corresponding to the estimated pitch that does not meet the preset error range multiple times, obtaining a new estimated pitch using the adjusted pitch probability distribution until the error between the new estimated pitch and the actual pitch meets the preset error range; updating the initial speech signal of each speaking object through the latest adjusted pitch probability distribution to obtain the target speech signal of each speaking object.

[0008] According to the embodiments of the application, the mixed speech data is divided into multiple overlapping speech segments according to a preset length; the convolutional neural network is used to perform feature transformation on the multiple overlapping speech segments to obtain the time domain features of the multiple overlapping speech segments; and the time domain features of the multiple overlapping speech segments are separated and reconstructed to obtain the initial speech signal of each speaking object.

[0009] According to the embodiments of the application, multiple time-frequency masks are estimated through the multiple overlapping speech segments; each time-frequency mask is used to perform separation processing on the time domain features of each corresponding overlapping speech segment to obtain multiple speech time-frequency spectra; and the decoder is used to reconstruct the multiple speech time-frequency spectra to obtain the initial speech signal of each speaking object.

[0010] According to the embodiments of the application, the initial speech signal of each speaking object is transformed; a residual module is constructed, and the transformed initial speech signal is processed through the residual module to obtain the pitch probability distribution of each speaking object. According to the embodiments of the application, the initial speech signal of each speaking object is frame segmented to obtain multiple discrete speech segments; and each discrete speech segment is linearly transformed to obtain the frequency spectrum feature corresponding to each discrete speech segment.

[0011] According to the embodiments of the application, the residual module is used to process each frequency spectrum feature to obtain multiple frequency spectrum fusion features; and the multiple frequency spectrum fusion features are mapped to obtain the pitch probability distribution of each speaking object.

[0012] According to an embodiment of the present application, a difference between each pitch probability distribution corresponding to an estimated pitch that does not satisfy a preset error range is calculated to obtain a cross-entropy loss of each pitch probability distribution; a weight coefficient of the cross-entropy loss of each pitch probability distribution is adjusted multiple times to optimize the cross-entropy loss of each pitch probability distribution to obtain an adjusted pitch probability distribution; a new estimated pitch is obtained by using the adjusted pitch probability distribution until an error between the new estimated pitch and an actual pitch satisfies the preset error range, and a latest adjusted pitch probability distribution is determined.

[0013] Another aspect of the present application provides a speech separation device, comprising: a speech acquisition module configured to acquire mixed speech data containing a plurality of speaking objects and determine an actual pitch of each speaking object; a speech separation module configured to separate the mixed speech data to obtain an initial speech signal of each speaking object; a pitch estimation module configured to perform feature processing on the initial speech signal of each speaking object to obtain a pitch probability distribution of each speaking object, process the pitch probability distribution of each speaking object to obtain an estimated pitch of each speaking object, and if an error between the estimated pitch and the actual pitch does not satisfy a preset error range, adjust the pitch probability distribution corresponding to the estimated pitch that does not satisfy the preset error range multiple times, obtain a new estimated pitch by using the adjusted pitch probability distribution until an error between the new estimated pitch and the actual pitch satisfies the preset error range; and a speech update module configured to update the initial speech signal of each speaking object by using the latest adjusted pitch probability distribution to obtain a target speech signal of each speaking object.

[0014] Another aspect of the present application provides an electronic device, comprising: one or more processors; and a memory configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method as above.

[0015] Another aspect of the present application provides a computer-readable storage medium storing a computer program or instructions, which, when executed, are used to implement the method as above.

[0016] Another aspect of the present application provides a computer program product comprising a computer program or instructions, which, when executed by a processor, are used to implement the method as above.

[0017] The speech separation method, device, electronic device, storage medium and product provided by the embodiments of the present application can achieve the following beneficial effects:

[0018] (1) Through the serial network structure, the speech separation module can effectively process mixed speech data, accurately separate the speech signals of the same gender and different speakers, adopt permutation processing, ensure that the separated speech is consistent with the original input source, effectively solve the permutation problem in speech separation, and improve the consistency of the separation result.

[0019] (2) The pitch estimation network is introduced, the pitch of the speaker's voice is estimated, the speech information is more comprehensively obtained, and the comprehensive performance of the audio processing is improved.

[0020] (3) By jointly training the speech separation module and the pitch estimation module, using the joint optimization method of scale invariant signal-to-noise ratio loss and cross-entropy loss of the pitch estimation module, the two modules learn and supplement each other, and the accuracy of speech separation and pitch estimation is further improved.

[0021] (4) The introduction of the TCN (Temporal Convolutional Network) structure allows the model to fully utilize the long-time dependence of the speech signal, effectively expands the time context window of the model through the incremental expansion factor, improves the capture ability of the speech signal, and the residual connection and batch normalization layer in the model structure help to prevent the problem of gradient disappearance or gradient explosion, improve the robustness of the network, and speed up the training speed. BRIEF DESCRIPTION OF DRAWINGS

[0022] The above and other objects, features and advantages of the present application will become more apparent from the following description of embodiments of the present application, taken in conjunction with the accompanying drawings, in which:

[0023] Figure 1 The application scenario diagram of speech separation according to the embodiments of the present application is schematically shown;

[0024] Figure 2A The flowchart of the speech separation method according to the embodiments of the present application is schematically shown;

[0025] Figure 2B The network structure diagram of the speech separation method according to the embodiments of the present application is schematically shown;

[0026] Figure 3 The flowchart of obtaining the initial speech signal of each speaker according to the embodiments of the present application is schematically shown;

[0027] Figure 4 The flowchart of processing the time domain features of the overlapping speech segments in the mixed speech data according to the embodiments of the present application is schematically shown;

[0028] Figure 5 The flowchart of obtaining the pitch probability distribution of each speaker according to the embodiments of the present application is schematically shown;

[0029] Figure 6 A schematic diagram of the residual module according to an embodiment of this application is shown.

[0030] Figure 7 A flowchart illustrating the determination of the latest adjusted pitch probability distribution according to an embodiment of this application is shown schematically.

[0031] Figure 8 A schematic diagram illustrating the structure of a speech separation apparatus according to an embodiment of this application is shown.

[0032] Figure 9 A block diagram schematically illustrates an electronic device suitable for implementing a speech separation method according to an embodiment of this application. Detailed Implementation

[0033] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0034] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0035] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0036] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, application, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0037] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this application all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0038] Figure 1 The illustration shows an application scenario diagram of speech separation according to an embodiment of this application.

[0039] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0040] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0041] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0042] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0043] It should be noted that the speech separation method provided in this application embodiment can generally be executed by server 105. Correspondingly, the speech separation device provided in this application embodiment can generally be located in server 105. The speech separation method provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the speech separation device provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0044] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0045] The following will be based on Figure 1 The scenario described herein will be described in detail with reference to the voice method of the embodiments of this application.

[0046] Figure 2A A flowchart illustrating a speech separation method according to an embodiment of this application is shown schematically; Figure 2B A network structure diagram of a speech separation method according to an embodiment of this application is illustrated.

[0047] like Figure 2A and Figure 2B As shown, the speech separation method according to the embodiments of this application includes steps S1 to S6.

[0048] In step S1, mixed speech data containing multiple speakers is acquired, and the actual pitch of each speaker is determined.

[0049] Record audio of multiple subjects speaking simultaneously using a multi-microphone array or a single microphone, ensuring coverage of speakers of different genders, ages, and pitches. Record audio of each subject speaking individually, and determine the actual pitch of each subject by analyzing the audio of each subject speaking.

[0050] For example, a multi-microphone array can be one of the following: a linear array (microphones are evenly arranged along a straight line), a planar array (microphones are distributed in a two-dimensional plane and arranged in a rectangular, circular or spiral pattern), a circular array (microphones are evenly distributed on the circumference), a spherical array (microphones are distributed on the surface of a sphere), or a random sparse array (microphones are not evenly distributed and the spacing may be random or optimized).

[0051] In step S2, the mixed speech data is separated to obtain the initial speech signal of each speaker.

[0052] The mixed speech data is framed and windowed by the encoder. Based on a one-dimensional convolutional neural network, the mixed speech data is converted into time-domain features after one-dimensional convolution, nonlinear activation and pooling. The speech of different speakers is highly overlapping in the time-frequency domain (such as the spectrogram). By calculating the spectrum of the mixed speech data, the energy ratio of each speaker at the time-frequency point is estimated, which is the time-frequency mask. The decoder performs phase reconstruction to smooth the time-frequency mask and obtain the separated initial speech signal.

[0053] For example, the algorithms used in time-frequency masks include ideal binary masks, ideal scaling masks, Wiener filters, etc.

[0054] In step S3, feature processing is performed on the initial speech signal of each speaker to obtain the pitch probability distribution of each speaker.

[0055] The initial speech signal of each speaker is segmented and windowed to reduce spectral leakage. Pitch analysis is performed only on valid speech segments to avoid interference from silent segments. The fundamental frequency of each speech segment is calculated based on a deep learning algorithm. The pitch values ​​of all valid frames are statistically analyzed, and the pitch probability distribution of each speaker is fitted. The pitch probability distribution is generated independently for each speaker. It supports the differential modeling of fundamental frequencies of different sound sources (such as male and female voices) in overlapping speech, and solves the problem of speech cross-interference in traditional methods.

[0056] For example, in addition to using deep learning algorithms to calculate the fundamental frequency of each frame of speech, the autocorrelation method (suitable for clean speech) or the fundamental frequency period can be found through the difference function.

[0057] For example, fitting the pitch probability distribution of each speaker can be done using a Gaussian mixture model (suitable for multimodal distributions) or kernel density estimation (suitable for non-parametric estimation of smooth distributions).

[0058] In step S4, the estimated pitch of each speaker is obtained through the pitch probability distribution of each speaker.

[0059] For example, pitch estimation can be performed using the mean method (suitable for stationary speech with a single-peak distribution), the median method (suitable for resisting outliers, such as burst noise), the main modality method (suitable for multi-peak distribution, such as singing vibrato), or the weighted mean method (suitable for mixed pitch feature analysis).

[0060] In step S5, if the error between the estimated pitch and the actual pitch does not meet the preset error range, the pitch probability distribution corresponding to the estimated pitch that does not meet the preset error range is adjusted multiple times, and a new estimated pitch is obtained using the adjusted pitch probability distribution, until the error between the new estimated pitch and the actual pitch meets the preset error range.

[0061] When the error between the estimated pitch and the actual pitch exceeds the preset range, the pitch probability distribution model parameters are automatically adjusted and iterated until the accuracy requirements are met.

[0062] In step S6, the initial speech signal of each speaker is updated using the latest adjusted pitch probability distribution to obtain the target speech signal of each speaker.

[0063] By utilizing the iteratively optimized pitch probability distribution, time-frequency domain reconstruction is performed on the separated initial speech signal to generate a high-quality target speech signal that conforms to the target pitch characteristics. This method is suitable for speech enhancement involving multiple speakers. Since the separated initial speech signal exhibits signal offset, correction according to the target distribution can improve the accuracy of speaker recognition.

[0064] For example, LPC (Linear Predictive Coding) analysis can be used to extract the formant frequencies of the actual pitch, keeping the relative positions of the formants between the actual pitch and the estimated pitch unchanged during spectrum shifting.

[0065] For example, in offline banking scenarios, when multiple users simultaneously consult bank tellers about related services, the overlapping and interference of voice signals from multiple users makes it difficult to identify each user's specific needs and instructions. This forces tellers to repeatedly confirm information, which not only prolongs the processing time for individual users but also easily leads to dissatisfaction among other users.

[0066] The method described in this application can reduce noise and separate superimposed voice signals in real time, accurately extract each user's business needs, and automatically synchronize them to the teller's interface. This not only improves the bank's service efficiency and quality but also reduces user waiting time and communication costs, thus optimizing the user experience.

[0067] For example, in online banking consultation scenarios, such as when a bank's telephone customer service receives multiple inquiries from users simultaneously, the voice signals of multiple users overlap in the call line and there is background noise, making it difficult to identify the specific questions and information needs of each user. This causes customer service to have to ask users to repeat themselves, which not only prolongs the call time but may also lead to incorrect answers due to information confusion, thus reducing customers' trust in telephone banking services.

[0068] The method described in this application can accurately separate the clear voice of each customer, allowing customer service to quickly understand the needs of each user. This not only shortens call processing time but also improves the accuracy of answers, effectively enhancing the service efficiency and user satisfaction of telephone banking.

[0069] Figure 3 A flowchart illustrating the process of obtaining the initial speech signal for each speaker according to an embodiment of this application is shown.

[0070] like Figure 3 As shown, obtaining the initial voice signal of each speaker according to the embodiments of this application includes steps S21 to S23.

[0071] Step S21: Divide the mixed speech data into multiple overlapping speech segments according to a preset length.

[0072] The mixed speech data is input into the encoder, which is a feature extraction model based on a one-dimensional convolutional neural network (1D CNN). The encoder divides the mixed speech data into overlapping speech segments of length L, where L is the size of the encoder's convolution kernel.

[0073] Step S22: Perform feature transformation on multiple overlapping speech segments using a convolutional neural network to obtain the temporal features of the multiple overlapping speech segments.

[0074] A one-dimensional convolutional neural network is used to perform feature transformation on speech segments. A normalization layer transforms the speech segments into a one-dimensional sequence. One-dimensional convolution (sliding window filtering) and pooling are then applied to this one-dimensional sequence to obtain a dimensionality-reduced feature sequence or global feature vector, i.e., the temporal features of multiple overlapping speech segments. .

[0075]

[0076] In the formula, This represents a one-dimensional convolution operation. Indicates a speech segment, Indicates the sequence index of a speech segment. It represents the total number of segments.

[0077] Step S23: Separate and reconstruct the temporal features of multiple overlapping speech segments to obtain the initial speech signal of each speaker.

[0078] The TNT structure is used to separate the temporal features of multiple overlapping speech segments. The TNT structure can capture the dependencies in long time series to extract the temporal features of the speech signal. The TNT structure includes multiple residual blocks, each of which consists of a one-dimensional convolutional layer and residual connections, which can realize cross-layer information transfer. The output of the convolution operation is added to the input through the residual block connection to prevent gradient vanishing or gradient exploding problems. The decoder reconstructs the separated temporal features to obtain the initial speech signal of each speaker.

[0079] Figure 4A flowchart illustrating the processing of temporal features of overlapping speech segments in mixed speech data according to an embodiment of this application is shown.

[0080] like Figure 4 As shown, the temporal feature processing of overlapping speech segments in mixed speech data according to the embodiments of this application includes steps S231 to S233.

[0081] Step S231: Estimate the corresponding time-frequency mask using multiple overlapping speech segments.

[0082] By analyzing the time-frequency characteristics of multiple overlapping speech segments, the time-frequency mask of each sound source is accurately estimated, effectively distinguishing the energy distribution of different speakers in the mixed signal. The time-frequency mask is used to perform directional filtering of the time-domain characteristics of the original signal, preserving the spectral components of the target sound source while suppressing interfering sound sources.

[0083] For example, estimating time-frequency masks The following formula is used specifically:

[0084]

[0085] In the formula, Indicates a separate network. It represents the time-domain characteristics.

[0086] Step S232: Separate the temporal features of each overlapping speech segment using each time-frequency mask to obtain multiple speech time-spectrums.

[0087] For example, the separation process specifically uses the following formula:

[0088]

[0089] In the formula, Indicates a separate network. Represents an S-shaped curve function. This represents a one-dimensional convolution operation. This represents the activation function. This represents the original signal input to the separation network.

[0090] Step S233: The time spectrum of multiple speech signals is reconstructed by the decoder to obtain the initial speech signal of each speaker.

[0091] The decoder converts the separated time spectrum into a time domain signal through inverse mapping, restoring the clarity and naturalness of the speech to the greatest extent, ensuring that the pronunciation is comprehensible and preserving the speaker's personality characteristics (such as timbre and intonation).

[0092] For example, the initial estimated speech signal is obtained through element-wise multiplication of the decoder, and the reconstruction loss of the spectrum is calculated when reconstructing the speech. Optimize the initial estimated speech signal Obtain the initial speech signal The specific formula is as follows:

[0093]

[0094]

[0095]

[0096] In the formula, Indicates time-frequency mask, Indicates a speech segment, Represents the reconstruction loss function. Indicates the weighting coefficient. This represents the auxiliary loss function.

[0097] For example, the decoder optimizes the initial estimated speech signal using a scale-invariant signal-to-noise ratio loss function.

[0098] Figure 5 A flowchart illustrating the acquisition of the pitch probability distribution of each speaker according to an embodiment of this application is shown. Figure 6 A schematic diagram of the residual module according to an embodiment of this application is shown.

[0099] like Figure 5 and Figure 6 As shown, the temporal feature processing of overlapping speech segments in mixed speech data according to the embodiments of this application includes steps S31 to S34.

[0100] Step S31: Perform frame segmentation on the initial speech signal of each speaker to obtain multiple discrete speech segments.

[0101] The initial speech signal output by the decoder is segmented into frames (discrete speech segments) within multiple time windows. The purpose is to decompose the continuous initial speech signal into discrete time segments for subsequent processing.

[0102] Step S32: Perform a linear transformation on each discrete speech segment to obtain the spectral features corresponding to each discrete speech segment.

[0103] Performing a linear transformation on each frame (discrete speech segment) can suppress high-frequency noise and avoid signal ambiguity. It converts the initial speech signal in the time domain into a frequency domain representation or other forms of feature representation to extract spectral features within each time window, providing input for the subsequent residual module.

[0104] For example, linear transformations can be achieved using Fourier transform or wavelet transform.

[0105] For example, after linear transformation The initial speech signal after Spectral features obtained Specifically, it can be expressed by the following formula:

[0106]

[0107] In the formula, It also represents the original input of the residual module.

[0108] Step S33: Construct a residual module and use the residual module to process each spectral feature to obtain multiple spectral fusion features.

[0109] The residual module comprises multiple residual blocks and models the temporal structure of the fundamental frequency profile through multi-layer convolutional operations. It employs multiple causal convolutional layers, with the span between each layer being a power of 2, to expand the receptive field of the convolutional layers while maintaining causality. Through multi-layer convolution, the network learns a high-level representation of the speech signal and predicts the fundamental frequency based on contextual information.

[0110] For example, the inflation factor for an 8-layer residual module is The filter length is 5, which can generate a receptive field of 71 frames.

[0111] For example, spectral fusion features Specifically, it can be expressed by the following formula:

[0112]

[0113]

[0114] In the formula, This represents the convolution operation. Represents the residual module. This represents the original input / spectral characteristics of the residual module. This represents the hyperbolic tangent activation function.

[0115] Step S34: Map the multiple spectral fusion features to obtain the pitch probability distribution of each speaking object.

[0116] The pitch probability distribution reflects the probability of possible pitch states within each time window.

[0117] For example, pitch probability distribution Specifically, it can be expressed by the following formula:

[0118]

[0119] In the formula, This represents the output layer of the residual module. This represents the spectral fusion characteristics of each speaker.

[0120] Figure 7 A flowchart illustrating the determination of the latest adjusted pitch probability distribution according to an embodiment of this application is shown schematically.

[0121] like Figure 7 As shown, determining the latest adjusted pitch probability distribution according to the embodiments of this application includes steps S51 to S53.

[0122] Step S51: Calculate the difference between the pitch probability distributions corresponding to each estimated pitch that does not meet the preset error range, and obtain the cross-entropy loss of each pitch probability distribution.

[0123] Cross-entropy loss is often used to measure the difference between two probability distributions; here, it is used to measure the difference between the estimated pitch and the actual pitch.

[0124] For example, cross-entropy loss Specifically, it can be expressed by the following formula:

[0125]

[0126] In the formula, Represents the probability distribution of pitch. Represents the cross-entropy loss function. This represents the reference true value corresponding to the actual pitch.

[0127] Step S52: Adjust the weight coefficient of the cross-entropy loss of each pitch probability distribution multiple times to optimize the cross-entropy loss of each pitch probability distribution and obtain the adjusted pitch probability distribution.

[0128] For example, the total loss of the pitch estimation module Other loss items Combined, they constitute the overall target loss. Specifically, it can be expressed by the following formula:

[0129]

[0130] In the formula, It is the weighting coefficient.

[0131] By adjusting the weighting coefficients This optimizes the total loss.

[0132] For example, the function used for optimizing the cross-entropy loss of the pitch probability distribution and the function used by the decoder to optimize the initial estimated speech signal are the same, both using the scale-invariant signal-to-noise ratio loss function.

[0133] Step S53: Obtain a new estimated pitch using the adjusted pitch probability distribution until the error between the new estimated pitch and the actual pitch meets the preset error range, and determine the latest adjusted pitch probability distribution.

[0134] In summary, the speech separation method provided in this application can achieve the following beneficial effects:

[0135] (1) Accurately separate mixed speech. Through the serial network structure, the speech separation module can effectively process mixed speech waveforms and accurately separate the speech signals of different speakers of the same gender.

[0136] (2) To solve the permutation problem, permutation processing is adopted to ensure that the separated speech is consistent with the original input source, which effectively solves the permutation problem in speech separation and improves the consistency of the separation results.

[0137] (3) Pitch information prediction: A pitch estimation network is introduced to estimate the pitch of the speaker's speech, thereby obtaining speech information more comprehensively and improving the overall performance of audio processing.

[0138] (4) Joint optimization improves accuracy. By jointly training the speech separation module and the pitch estimation module, a joint optimization method of scale-invariant signal-to-noise ratio loss and cross-entropy loss of the pitch estimation module is adopted, so that the two modules learn from each other and complement each other, which further improves the accuracy of speech separation and pitch estimation.

[0139] (5) Long-term dependency processing: The introduction of the TCN structure allows the model to make full use of the long-term dependency of the speech signal. By increasing the inflation factor, the model’s time context window is effectively expanded, which improves the ability to capture speech signals.

[0140] (6) Improved robustness and training speed: The residual connections and batch normalization layers in the model structure help prevent gradient vanishing or gradient explosion problems, which improves the robustness of the network and accelerates the training speed.

[0141] Based on the above speech separation method, this application also provides a speech separation device. The following will be combined with... Figure 8 The device is described in detail.

[0142] Figure 8 A schematic block diagram of a speech separation device according to an embodiment of this application is shown.

[0143] like Figure 8 As shown, the speech separation device 800 of this embodiment includes a speech acquisition module 810, a speech separation module 820, a pitch estimation module 830, and a speech update module 840.

[0144] The voice acquisition module 810 is used to acquire mixed voice data containing multiple speakers and determine the actual pitch of each speaker. In one embodiment, the voice acquisition module 810 can be used to perform step S1 described above, which will not be repeated here.

[0145] The speech separation module 820 is used to separate the mixed speech data to obtain the initial speech signal of each speaker. In one embodiment, the speech separation module 820 can be used to perform step S2 described above, which will not be repeated here.

[0146] The pitch estimation module 830 performs feature processing on the initial speech signal of each speaker to obtain a pitch probability distribution for each speaker. It then processes this pitch probability distribution to obtain an estimated pitch for each speaker. If the error between the estimated pitch and the actual pitch does not meet a preset error range, the pitch probability distribution corresponding to the estimated pitch that does not meet the preset error range is adjusted multiple times. A new estimated pitch is obtained using the adjusted pitch probability distribution, until the error between the new estimated pitch and the actual pitch meets the preset error range. In one embodiment, the pitch estimation module 830 can be used to execute steps S3 to S5 described above, which will not be repeated here.

[0147] The voice update module 840 is used to update the initial voice signal of each speaker using the latest adjusted pitch probability distribution to obtain the target voice signal for each speaker. In one embodiment, the voice update module 840 can be used to perform step S6 described above, which will not be repeated here.

[0148] According to an embodiment of this application, the speech separation module 820 includes a speech segmentation unit, a feature transformation unit, and a separation and reconstruction unit.

[0149] The speech segmentation unit is used to divide mixed speech data into multiple overlapping speech segments according to a preset length.

[0150] The feature transformation unit is used to perform feature transformation on multiple overlapping speech segments through a convolutional neural network to obtain the temporal features of the multiple overlapping speech segments.

[0151] The separation and reconstruction unit is used to separate and reconstruct the temporal features of multiple overlapping speech segments to obtain the initial speech signal of each speaker.

[0152] According to an embodiment of this application, the separation and reconstruction unit includes a time-frequency mask subunit, a separation processing subunit, and a spectrum reconstruction subunit.

[0153] The time-frequency mask subunit is used to estimate multiple time-frequency masks corresponding to multiple overlapping speech segments.

[0154] The separation processing subunit is used to separate the temporal features of each corresponding overlapping speech segment through each time-frequency mask to obtain multiple speech time-spectrums.

[0155] The spectrum reconstruction subunit is used to reconstruct the time spectrum of multiple speech signals using a decoder to obtain the initial speech signal for each speaker.

[0156] According to an embodiment of this application, the pitch estimation module 830 includes a speech conversion unit and a speech processing unit.

[0157] The speech transformation unit is used to transform the initial speech signal of each speaker.

[0158] The speech processing unit is used to construct a residual module, which processes the transformed initial speech signal to obtain the pitch probability distribution of each speaker.

[0159] According to embodiments of this application, the speech transformation unit includes a speech segmentation subunit and a linear transformation subunit.

[0160] The speech segmentation subunit is used to segment the initial speech signal of each speaker into frames, resulting in multiple discrete speech segments.

[0161] The linear transformation subunit is used to perform a linear transformation on each discrete speech segment to obtain the spectral features corresponding to each discrete speech segment.

[0162] According to an embodiment of this application, the speech processing unit includes a spectrum fusion subunit and a fusion mapping subunit.

[0163] The spectrum fusion subunit is used to process each spectrum feature using the residual module to obtain multiple spectrum fusion features.

[0164] The fusion mapping subunit is used to map multiple spectral fusion features to obtain the pitch probability distribution of each speaking object.

[0165] According to an embodiment of this application, the pitch estimation module 830 further includes a loss calculation unit, a loss optimization unit, and a probability determination unit.

[0166] The loss calculation unit calculates the difference between the pitch probability distributions corresponding to each estimated pitch that does not meet the preset error range, and obtains the cross-entropy loss of each pitch probability distribution.

[0167] The loss optimization unit repeatedly adjusts the weight coefficients of the cross-entropy loss for each pitch probability distribution to optimize the cross-entropy loss for each pitch probability distribution, thus obtaining the adjusted pitch probability distribution.

[0168] The probability determination unit uses the adjusted pitch probability distribution to obtain a new estimated pitch until the error between the new estimated pitch and the actual pitch meets the preset error range, and then determines the latest adjusted pitch probability distribution.

[0169] According to embodiments of this application, any multiple modules among the voice acquisition module 810, voice separation module 820, pitch estimation module 830, and voice update module 840 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the voice acquisition module 810, voice separation module 820, pitch estimation module 830, and voice update module 840 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the speech acquisition module 810, speech separation module 820, pitch estimation module 830, and speech update module 840 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0170] Figure 9 A block diagram schematically illustrates an electronic device suitable for implementing a speech separation method according to an embodiment of this application.

[0171] like Figure 9 As shown, an electronic device 900 according to an embodiment of this application includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0172] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 902 and / or RAM 903. It should be noted that the programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0173] According to embodiments of this application, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the input / output (I / O) interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0174] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0175] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0176] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the speech separation method provided in the embodiments of this application.

[0177] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0178] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0179] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0180] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0181] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0182] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A speech separation method, characterized in that, include: Acquire mixed speech data containing multiple speakers, and determine the actual pitch of each speaker; The mixed speech data is separated to obtain the initial speech signal for each speaker; The initial speech signal of each speaker is subjected to feature processing to obtain the pitch probability distribution of each speaker. The estimated pitch of each speaker is obtained by using the pitch probability distribution of each speaker. If the error between the estimated pitch and the actual pitch does not meet the preset error range, the pitch probability distribution corresponding to the estimated pitch that does not meet the preset error range is adjusted multiple times, and a new estimated pitch is obtained by using the adjusted pitch probability distribution until the error between the new estimated pitch and the actual pitch meets the preset error range. The initial speech signal of each speaker is updated by the latest adjusted pitch probability distribution to obtain the target speech signal of each speaker.

2. The method according to claim 1, characterized in that, The step of separating the mixed speech data to obtain the initial speech signal for each speaker includes: The mixed speech data is divided into multiple overlapping speech segments according to a preset length; The multiple overlapping speech segments are transformed using a convolutional neural network to obtain their temporal features. The temporal features of the multiple overlapping speech segments are separated and reconstructed to obtain the initial speech signal of each speaker.

3. The method according to claim 2, characterized in that, The step of separating and reconstructing the temporal features of the multiple overlapping speech segments to obtain the initial speech signal for each speaker includes: Multiple time-frequency masks are estimated using the multiple overlapping speech segments; The temporal features of each overlapping speech segment are separated by each time-frequency mask to obtain multiple speech time-spectrums; The initial speech signal of each speaker is obtained by reconstructing the time spectrum of the multiple speech signals using a decoder.

4. The method according to claim 1, characterized in that, The step of performing feature processing on the initial speech signal of each speaker to obtain the pitch probability distribution of each speaker includes: Transform the initial speech signal for each of the speaking subjects; A residual module is constructed, and the transformed initial speech signal is processed through the residual module to obtain the pitch probability distribution of each speaker.

5. The method according to claim 4, characterized in that, The transformation of the initial speech signal for each of the speaking subjects includes: The initial speech signal of each speaker is segmented into frames to obtain multiple discrete speech segments; Each discrete speech segment is linearly transformed to obtain the spectral features corresponding to each discrete speech segment.

6. The method according to claim 5, characterized in that, The step of processing the transformed initial speech signal through the residual module to obtain the pitch probability distribution of each speaker includes: The residual module is used to process each spectral feature to obtain multiple spectral fusion features; The multiple spectral fusion features are mapped to obtain the pitch probability distribution of each speaking object.

7. The method according to claim 1, characterized in that, The step of repeatedly adjusting the pitch probability distribution corresponding to the estimated pitch that does not meet the preset error range, and using the adjusted pitch probability distribution to obtain a new estimated pitch, until the error between the new estimated pitch and the actual pitch meets the preset error range includes: Calculate the difference between the pitch probability distributions corresponding to each estimated pitch that does not meet the preset error range, and obtain the cross-entropy loss of each pitch probability distribution; The weighting coefficients of the cross-entropy loss for each pitch probability distribution are adjusted multiple times to optimize the cross-entropy loss for each pitch probability distribution, resulting in an adjusted pitch probability distribution. A new estimated pitch is obtained using the adjusted pitch probability distribution until the error between the new estimated pitch and the actual pitch meets a preset error range, and the latest adjusted pitch probability distribution is determined.

8. A speech separation device, characterized in that, include: The speech acquisition module is used to acquire mixed speech data containing multiple speakers and determine the actual pitch of each speaker. The speech separation module is used to separate the mixed speech data to obtain the initial speech signal of each speaker; The pitch estimation module is used to perform feature processing on the initial speech signal of each speaker to obtain the pitch probability distribution of each speaker, and to obtain the estimated pitch of each speaker through the pitch probability distribution of each speaker. If the error between the estimated pitch and the actual pitch does not meet the preset error range, the pitch probability distribution corresponding to the estimated pitch that does not meet the preset error range is adjusted multiple times, and a new estimated pitch is obtained using the adjusted pitch probability distribution, until the error between the new estimated pitch and the actual pitch meets the preset error range. The voice update module is used to update the initial voice signal of each of the speaking objects using the latest adjusted pitch probability distribution, so as to obtain the target voice signal of each of the speaking objects.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

Images