Intelligent speech separation system based on deep learning

Abstract

The present invention discloses an intelligent speech separation system based on deep learning. The present invention preprocesses a high-dimensional feature vector sequence, calculates the pitch frequency and MFCC coefficient of the audio frame, performs quantization, normalization and fusion operations, and then obtains a feature sequence of separated speech signal components through a series of calculations. The influence of timbre and pitch characteristics on speech source correlation is fully considered, and the accuracy of speech separation is improved. After the separation is completed, the separation quality index of each separated speech signal component is calculated, and the optimized signal component is marked based on the result of the separation quality index comparison. The optimized signal component is optimized by repeatedly inputting the model, and the separation accuracy is further improved.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to an intelligent speech separation system based on deep learning. Background Art

[0002] In today's digital information age, speech signal processing has a wide range of application scenarios, such as voice communication, speech recognition, multimedia processing, etc. However, in actual environments, speech signals are often affected by multiple interference sources, such as background noise, overlapping speech generated by multiple people speaking at the same time, etc., which makes it a very challenging task to accurately separate the target speech from the mixed speech signal.

[0003] However, the intelligent speech separation system in the prior art still has the following deficiencies:

[0004] When processing speech signals, the impact of spectrum leakage and frequency resolution on speech feature extraction is not fully considered, resulting in limited speech separation accuracy;

[0005] In addition, the impact of timbre and pitch characteristics on speech source correlation was not fully considered during the speech separation process. After separation, the separated speech signal components were not comprehensively evaluated to determine whether the deep neural model needed to be adjusted and then input into the model again for separation, resulting in poor separation accuracy and effect.

[0006] To this end, an intelligent speech separation system based on deep learning was launched. Summary of the invention

[0007] In view of this, the present invention provides an intelligent speech separation system based on deep learning to solve the problems raised by the above background technology.

[0008] The purpose of the present invention can be achieved by the following technical solution: An intelligent speech separation system based on deep learning, comprising:

[0009] Speech feature module: obtains audio data containing mixed speech and preprocesses the received audio data. After the preprocessing is completed, the audio data is framed according to the preset time length to obtain an audio frame sequence. After performing corresponding operations on the obtained audio frame sequence, a high-dimensional feature vector sequence is obtained and sent to the speech separation module; each high-dimensional feature vector represents the deep feature representation of each audio frame;

[0010] Speech separation module: receives a high-dimensional feature vector sequence and inputs it into a pre-built deep neural model. After the input is completed, the deep neural model executes the corresponding steps to obtain the final feature sequence composed of the separated speech signal components;

[0011] Speech reconstruction module: Perform inverse feature transformation on the final feature sequence composed of each speech signal component to obtain the preliminary reconstruction result of the time domain waveform of each speech signal component, use the preliminary reconstruction result of the time domain waveform as the time domain speech signal of each speech signal component, and post-process the reconstructed time domain speech signal of each speech signal component. After the post-processing is completed, the output mode is determined according to the user's settings;

[0012] User feedback module: collects feedback parameters of speech separation effect within a set time period, conducts comprehensive analysis, and determines whether the deep neural model needs to be adjusted based on the analysis results; feedback parameters include the number of repeated inputs, the number of optimization marks, and user feedback scores.

[0013] In some embodiments, corresponding operations are performed on the obtained audio frame sequence, specifically:

[0014] M1: Set different Hamming window coefficient combinations as variables, and use each Hamming window coefficient combination to perform windowing operations on the audio frames in the audio frame sequence in sequence;

[0015] M2: Perform fast Fourier transform on the windowed audio frame to obtain a spectrum diagram, analyze the spectrum leakage ratio SLR and frequency resolution FR under each Hamming window coefficient combination, and obtain the spectrum leakage ratio SLR and frequency resolution FR of each group corresponding to the audio frame sequence under different Hamming window coefficient combinations. Take the average of each group of spectrum leakage ratio SLR and frequency resolution FR under the same Hamming window coefficient combination to obtain the spectrum leakage average ratio and frequency resolution average ratio under the same Hamming window coefficient combination.

[0016] In some embodiments, performing corresponding operations on the obtained audio frame sequence further includes:

[0017] M3: Mark the spectrum leakage average ratio and frequency resolution average ratio as sq and se respectively, and preset the passing spectrum leakage ratio and passing frequency resolution corresponding to the spectrum leakage average ratio sq and the frequency resolution average ratio se respectively, according to the formula The spectrum leakage average ratio sq and the frequency resolution average ratio se under different Hamming window coefficient combinations are weightedly calculated to obtain the comprehensive performance value su of different Hamming window coefficient combinations for the current audio frame sequence; and They represent the passing spectrum leakage ratio and passing frequency resolution respectively, a1 and a2 are the weight factors of the spectrum leakage average ratio sq and the frequency resolution average ratio se respectively; the Hamming window coefficient combination with the largest comprehensive value su is selected as the optimal Hamming window coefficient combination for the current audio frame sequence;

[0018] M4: Use a convolutional neural network to extract features from the audio frame spectrogram to obtain a feature vector sequence output by the convolutional neural network, and input the feature vector sequence into a long short-term memory network. The long short-term memory network processes the input feature vector sequence to learn the temporal characteristics of the audio signal, and finally obtains a high-dimensional feature vector sequence corresponding to the audio frame sequence.

[0019] In some embodiments, after the input is completed, the deep neural model performs corresponding steps, specifically:

[0020] P1: The deep neural model preprocesses the received high-dimensional feature vector sequence;

[0021] P2: Extract the audio frame sequence, calculate the autocorrelation function for each frame of audio, identify the delay time corresponding to the periodic peak in the autocorrelation function, and obtain the pitch frequency of each frame of audio. Quantize and normalize the obtained pitch frequency value, and after processing, mark the pitch feature vector corresponding to each time step as d1;

[0022] Perform short-time Fourier transform on each frame of audio, then process it through a Mel filter to obtain a Mel spectrum, then take the logarithm of the Mel spectrum and perform discrete cosine transform, finally obtain the MFCC coefficient corresponding to each frame of audio, perform dimensionality reduction on the obtained MFCC coefficients, select the first X coefficients as the key timbre features; X>5; and quantize and normalize the key timbre features; after processing, the key timbre feature vector corresponding to each time step is d2;

[0023] For the hidden representation of the speech source at all time steps, the corresponding pitch feature vector d1 and the key timbre feature vector d2 are fused, and the fused vector becomes D+d1+d2, where D represents the hidden representation of the speech source, and the fused hidden representation vector sequence is obtained;

[0024] The fused hidden representation vector sequence is reshaped into H×T×(D+d1+d2), where H is the number of speech sources and T is the time step; the dot product operation is used to calculate the relevance score of each time step, and a relevance score matrix consisting of each group of relevance scores is constructed from this;

[0025] For each time step, the softmax function is used to calculate and the calculation results of all time steps are integrated to obtain the attention weight distribution matrix;

[0026] The hidden representations of the speech sources are weighted and summed according to the attention weight distribution matrix to obtain a weighted representation focused on the corresponding speech source at each time step. The weighted representations of each speech source are integrated as a feature sequence composed of the separated speech signal components.

[0027] In some embodiments, after the input is completed, the deep neural model performs corresponding steps, further comprising:

[0028] P3: For each speech signal component separated in step P2, calculate the signal-to-noise ratio and the speech distortion; respectively preset the value intervals corresponding to the signal-to-noise ratio and the speech distortion, respectively, set each value interval corresponding to the signal-to-noise ratio to correspond to a signal-to-noise ratio score, and each value interval corresponding to the speech distortion to correspond to a distortion score;

[0029] The weight coefficients corresponding to the preset signal-to-noise ratio score and the distortion score are respectively multiplied by the corresponding preset weight coefficients, and then the sum is calculated to obtain the separation quality index of each speech signal component.

[0030] In some embodiments, the final feature sequence composed of the separated speech signal components is obtained, specifically:

[0031] P4: The preset minimum allowable index corresponding to the separation quality index. If the separation quality index of a speech signal component is lower than the preset minimum allowable index, the speech signal component is marked as an optimized signal component and input into the deep neural model again to repeat steps P2 and P3;

[0032] P5: Compare the separation quality index obtained after re-input in step P4 with the preset minimum allowable index. If it is still lower than the preset minimum allowable index, calculate the difference between the current separation quality index and the previous set of separation quality indexes, and mark the difference as the improvement value; then re-input into the deep neural model to repeat steps P2 and P3, compare the obtained separation quality index with the preset minimum allowable index, if it is still lower than the preset minimum allowable index, calculate the improvement value again, if the improvement value obtained by two consecutive rounds of repeated input is lower than the preset reference value of the improvement, stop input, take the result obtained by the last round of separation of the optimized signal component as the final result, and replace it;

[0033] P6: After the replacement is completed, the final feature sequence composed of the separated speech signal components is obtained.

[0034] In some embodiments, feedback parameters of speech separation effects within a set time period are collected and comprehensively analyzed, and it is determined whether the deep neural model needs to be adjusted based on the analysis results, specifically:

[0035] Count the number of repeated inputs received by the deep neural model within a set time period as the number of repeated inputs within the set time period; count the number of marked optimization signal components within the set time period as the number of optimized marks within the set time period; preset weight coefficients corresponding to the number of repeated inputs and the number of optimized marks, respectively, multiply the number of repeated inputs and the number of optimized marks by the corresponding preset weight coefficients, and then sum them to obtain the performance evaluation value f1 of the deep neural model within the set time period;

[0036] After the user sets the output mode for output, the user's scores on the speech separation effect are collected as the user feedback score of the current speech separation effect; the average of the user feedback scores of each group within the set time period is calculated as the feedback average score f2;

[0037] Using the formula The performance evaluation value f1 and the feedback average f2 within the set time period are weighted to obtain the speech separation evaluation index qw within the set time period; wherein and They represent the maximum value allowed for performance evaluation and the minimum value allowed for feedback score respectively; s1 and s2 are the weight factors affecting the performance evaluation value f1 and the feedback average score f2 respectively;

[0038] The preset minimum allowable value corresponding to the speech separation evaluation index qw is compared with the preset minimum allowable value within the set time period. If the speech separation evaluation index qw is lower than the minimum allowable value, an adjustment signal is triggered and sent to the technician to remind the technician to adjust the deep neural model.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The present invention calculates the pitch frequency and MFCC coefficient of the audio frame after preprocessing the high-dimensional feature vector sequence, performs quantization, normalization and fusion operations, and then obtains the feature sequence of the separated speech signal component through a series of calculations, fully considering the influence of timbre and pitch characteristics on the correlation of speech sources, and improving the accuracy of speech separation. At the same time, after the separation is completed, the separation quality index of each separated speech signal component is calculated, and the optimized signal component is marked based on the result of the separation quality index comparison, and the optimized signal component is optimized by repeatedly inputting the model, thereby further improving the separation accuracy;

[0041] The present invention sets different combinations of Hamming window coefficients, performs windowing and fast Fourier transform on audio frames, analyzes spectrum leakage ratio and frequency resolution, obtains performance comprehensive value by weighted calculation, selects the best combination, effectively reduces spectrum leakage, ensures a certain frequency resolution, and provides a better spectrum data foundation for subsequent speech feature extraction and separation;

[0042] The present invention collects feedback parameters such as the number of repeated inputs within a set time period, the number of optimization marks and user feedback scores, and comprehensively calculates the speech separation evaluation index, so as to effectively determine whether the deep neural model needs to be adjusted, thereby making up for the shortcomings of the existing technology in the evaluation mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Further details, features and advantages of the present application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0044] Figure 1 It is a principle block diagram of the present invention. DETAILED DESCRIPTION

[0045] Several embodiments of the present application will be described in more detail below with reference to the accompanying drawings so that those skilled in the art can implement the present application. The present application can be embodied in many different forms and purposes and should not be limited to the embodiments described herein. These embodiments are provided to make the present application comprehensive and complete, and to fully convey the scope of the present application to those skilled in the art. The embodiments do not limit the present application.

[0046] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this application belongs. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the relevant art and / or the context of this specification, and will not be interpreted in an idealized or overly formal sense unless explicitly defined as such herein.

[0047] See also Figure 1 As shown, the intelligent speech separation system based on deep learning includes a speech feature module, a speech separation module and a speech reconstruction module;

[0048] The speech feature module is used to obtain audio data containing mixed speech and preprocess the received audio data; the preprocessing includes denoising, enhancement, format unification and removal of DC components; after the preprocessing is completed, the audio data is framed according to the preset time length to obtain an audio frame sequence; the length of each frame is set to 20-40 milliseconds, the frame shift is 10-20 milliseconds, and an appropriate overlapping part is set to ensure the continuity of the speech signal; after performing corresponding operations on the obtained audio frame sequence, a high-dimensional feature vector sequence is obtained and sent to the speech separation module; each high-dimensional feature vector represents the deep feature representation of each audio frame;

[0049] The corresponding operations are performed on the obtained audio frame sequence, specifically:

[0050] M1: Set different combinations of Hamming window coefficients as variables; for example, in addition to the traditional combination of (0.54, 0.46), other combinations of coefficients can be selected within a preset range with a set step size; use each combination of Hamming window coefficients to perform windowing operations on the audio frames in the audio frame sequence in sequence;

[0051] M2: Perform fast Fourier transform on the audio frame obtained by windowing to obtain a spectrum diagram, analyze the spectrum leakage ratio SLR and frequency resolution FR under each Hamming window coefficient combination, and obtain the spectrum leakage ratio SLR and frequency resolution FR of each group corresponding to the audio frame sequence under different Hamming window coefficient combinations, and take the average of each group of spectrum leakage ratio SLR and frequency resolution FR under the same Hamming window coefficient combination to obtain the spectrum leakage average ratio and frequency resolution average ratio under the same Hamming window coefficient combination; the SLR is obtained by calculating the ratio of the sum of the main lobe energy and the side lobe energy in the spectrum diagram, and the frequency resolution is determined by analyzing the minimum frequency interval that can be clearly distinguished in the spectrum diagram. The larger the SLR value, the smaller the spectrum leakage, and the smaller the FR, the higher the spectrum resolution;

[0052] M3: Mark the spectrum leakage average ratio and frequency resolution average ratio as sq and se respectively, and preset the passing spectrum leakage ratio and passing frequency resolution corresponding to the spectrum leakage average ratio sq and the frequency resolution average ratio se respectively, according to the formula The spectrum leakage average ratio sq and the frequency resolution average ratio se under different Hamming window coefficient combinations are weightedly calculated to obtain the comprehensive performance value su of different Hamming window coefficient combinations for the current audio frame sequence; and They represent the passing spectrum leakage ratio and passing frequency resolution respectively, a1 and a2 are the weight factors of the spectrum leakage average ratio sq and the frequency resolution average ratio se respectively; the Hamming window coefficient combination with the largest comprehensive value su is selected as the optimal Hamming window coefficient combination for the current audio frame sequence;

[0053] It should be noted that this optimal combination achieves a good balance in reducing spectrum leakage and ensuring a certain frequency resolution, and can provide a better spectrum data foundation for subsequent tasks such as speech feature extraction and speech separation;

[0054] M4: Use a convolutional neural network to extract features from the audio frame spectrogram to obtain a feature vector sequence output by the convolutional neural network, and input the feature vector sequence into a long short-term memory network. The long short-term memory network processes the input feature vector sequence to learn the temporal characteristics of the audio signal, and finally obtains a high-dimensional feature vector sequence corresponding to the audio frame sequence;

[0055] M4-401: The first convolutional layer of the convolutional neural network uses a smaller convolution kernel (3×3) and an appropriate number of convolution kernels (16);

[0056] The second convolutional layer of the convolutional neural network uses different sizes of convolution kernels (5×5) and different numbers of convolution kernels (32);

[0057] For the audio frame spectrogram, the convolution operation after multiple convolution layers continuously extracts more complex spectral features. At the same time, the pooling operation is used between the convolution layers to reduce the data dimension, reduce the amount of calculation and enhance the translation invariance of the features.

[0058] After multiple convolutional layers and pooling layers, a fully connected layer is added to further integrate and map the features to obtain a feature vector sequence;

[0059] M4-402: The LSTM network is configured with multiple hidden layers (2-3 layers), each layer having a certain number of LSTM network units (64-128);

[0060] The long short-term memory network unit processes the input feature data in time series, learns the dynamic change rules of the audio signal in the time dimension, and captures the temporal characteristics of the speech signal;

[0061] After being processed by the long short-term memory network, a high-dimensional feature vector sequence with rich semantics and feature expression capabilities is obtained, and each feature vector corresponds to the deep feature representation of an audio frame.

[0062] It should be noted that the high-dimensional feature vector sequence contains rich speech feature information, which can provide a strong basis for the subsequent speech separation module so that it can perform accurate speech separation operations based on these features.

[0063] The speech separation module is used to receive a high-dimensional feature vector sequence and input it into a pre-built deep neural model. After the input is completed, the deep neural model executes the corresponding steps to obtain the final feature sequence composed of the separated speech signal components;

[0064] Specifically:

[0065] P1: The deep neural model preprocesses the received high-dimensional feature vector sequence; the preprocessing includes format checking and adjustment as well as data normalization;

[0066] P2: Extract the audio frame sequence, calculate the autocorrelation function for each frame of audio, identify the delay time corresponding to the periodic peak in the autocorrelation function, and obtain the pitch frequency of each frame of audio. The obtained pitch frequency value is quantized and normalized; that is, the pitch frequency value is divided according to the set interval, each interval corresponds to a quantized value, and then normalized. Usually, the quantized interval number is normalized to the [0,1] interval. Each quantized interval number is converted into a value in the [0,1] interval. These normalized quantized values ​​are formed into a vector form; after processing, the pitch feature vector corresponding to each time step is marked as d1;

[0067] Perform short-time Fourier transform on each frame of audio, then process it through a Mel filter to obtain a Mel spectrum, then take the logarithm of the Mel spectrum and perform discrete cosine transform, finally obtain the MFCC coefficient corresponding to each frame of audio, perform dimension reduction operation on the obtained MFCC coefficient, and select the first X coefficients as the key timbre features; X>5, set by technical personnel; and quantize and normalize the key timbre features; the quantization and normalization methods are similar to the processing of pitch frequency values, but the specific interval division and quantization method are set separately according to the characteristics of the timbre features; after processing, the key timbre feature vector corresponding to each time step is d2;

[0068] For the hidden representation of the speech source at all time steps, the corresponding pitch feature vector d1 and the key timbre feature vector d2 are fused, and the fused vector becomes D+d1+d2, where D represents the hidden representation of the speech source, and the fused hidden representation vector sequence is obtained;

[0069] The fused hidden representation vector sequence is reshaped into H×T×(D+d1+d2), where H is the number of speech sources and T is the time step; the dot product operation is used to calculate the relevance score of each time step, and a relevance score matrix consisting of each group of relevance scores is constructed from this;

[0070] The matrix comprehensively considers the impact of timbre and pitch features on speech source relevance;

[0071] For each time step, the softmax function is used to calculate and the calculation results of all time steps are integrated to obtain the attention weight distribution matrix;

[0072] The hidden representations of the speech sources are weighted and summed according to the attention weight distribution matrix to obtain the weighted representation focused on the corresponding speech source at each time step. The weighted representations of each speech source are integrated as the feature sequence composed of each speech signal component after separation.

[0073] P3: For each speech signal component separated in step P2, calculate the signal-to-noise ratio and speech distortion;

[0074] A higher signal-to-noise ratio means that the noise content in the separated speech signal is lower and the separation effect is better; the smaller the speech distortion, the higher the separation quality;

[0075] Preset the value intervals corresponding to the signal-to-noise ratio and the speech distortion respectively, set the value intervals corresponding to the signal-to-noise ratio to correspond to a signal-to-noise ratio score, and set the value intervals corresponding to the speech distortion to correspond to a distortion score; the signal-to-noise ratio score and the distortion score range are both set to 1-5, the higher the signal-to-noise ratio, the higher the corresponding signal-to-noise ratio score, and the lower the speech distortion, the higher the corresponding distortion score;

[0076] Preset the weight coefficients corresponding to the signal-to-noise ratio score and the distortion score, respectively, multiply the signal-to-noise ratio score and the distortion score of each speech signal component by the corresponding preset weight coefficients, and then sum them to obtain the separation quality index of each speech signal component;

[0077] P4: The preset minimum allowable index corresponding to the separation quality index. If the separation quality index of a speech signal component is lower than the preset minimum allowable index, the speech signal component is marked as an optimized signal component and input into the deep neural model again to repeat steps P2 and P3;

[0078] It should be noted that when the data is input into the deep neural model again, the technicians will adjust the relevant calculation mechanism based on the experience of the first round of separation.

[0079] P5: Compare the separation quality index obtained after re-input in step P4 with the preset minimum allowable index. If it is still lower than the preset minimum allowable index, calculate the difference between the current separation quality index and the previous set of separation quality indexes, and mark the difference as the improvement value; then re-input into the deep neural model to repeat steps P2 and P3, compare the obtained separation quality index with the preset minimum allowable index, if it is still lower than the preset minimum allowable index, calculate the improvement value again, if the improvement value obtained by two consecutive rounds of repeated input is lower than the preset reference value of the improvement, stop input, take the result obtained by the last round of separation of the optimized signal component as the final result, and replace it;

[0080] P6: After the replacement is completed, the final feature sequence composed of the separated speech signal components is obtained;

[0081] The speech reconstruction module is used to perform inverse feature transformation on the final feature sequence composed of each speech signal component, obtain the preliminary reconstruction result of the time domain waveform of each speech signal component, and use the preliminary reconstruction result of the time domain waveform as the time domain speech signal of each speech signal component; convert the frequency domain feature sequence back to the time domain signal representation form through inverse fast Fourier transform to obtain the preliminary reconstruction result of the time domain waveform of each speech signal component; and post-process the reconstructed time domain speech signal of each speech signal component, and after the post-processing is completed, determine the output mode according to the user's settings; the post-processing includes smoothing processing and volume normalization;

[0082] It should be noted that if the audio playback device output is selected, the reconstructed separated voice signal is sent to the specified audio playback device (such as speakers, headphones, etc.) through the audio driver, and appropriate playback parameters such as volume and channel balance are set to achieve real-time playback of the separated voice;

[0083] If you choose to store the audio file, the separated voice signal will be encoded and compressed according to the specified audio format (such as WAV, MP3, etc.) to generate an audio file, and then stored in the specified storage medium (such as hard disk, mobile storage device, network storage, etc.). When storing, you can set the file naming rules, storage path and other information to facilitate the user's subsequent search and use;

[0084] The user feedback module is used to collect feedback parameters of speech separation effects within a set time period, conduct a comprehensive analysis, and determine whether the deep neural model needs to be adjusted based on the analysis results; the feedback parameters include the number of repeated inputs, the number of optimization marks, and the user feedback score;

[0085] Specifically:

[0086] Counting the number of repeated inputs received by the deep neural model within a set time period as the number of repeated inputs within the set time period;

[0087] Counting the number of signal components marked for optimization within a set time period as the number of optimization marks within the set time period;

[0088] Preset the weight coefficients corresponding to the number of repeated inputs and the number of optimization marks, respectively, multiply the number of repeated inputs and the number of optimization marks by the corresponding preset weight coefficients, and then sum them up to obtain the performance evaluation value f1 of the deep neural model within the set time period;

[0089] It should be noted that the higher the number of repeated inputs and the number of optimization tags, the worse the deep neural model performs in the first round of speech separation processing.

[0090] After the user sets the output mode and performs the output, the user's rating on the speech separation effect is collected as the user feedback rating of the current speech separation effect; the rating range is set to 1-10 and is a positive integer. The more satisfied the user is, the higher the corresponding feedback rating is;

[0091] The average of the feedback scores of each group of users within the set time period is calculated as the average feedback score f2;

[0092] It should be noted that the higher the feedback average score f2 is, the more satisfied the user is with the effect of speech separation.

[0093] Using the formula The performance evaluation value f1 and the feedback average f2 within the set time period are weighted to obtain the speech separation evaluation index qw within the set time period; wherein and They represent the maximum value allowed for performance evaluation and the minimum value allowed for feedback score respectively; s1 and s2 are the weight factors affecting the performance evaluation value f1 and the feedback average score f2 respectively;

[0094] The minimum allowable value corresponding to the preset speech separation evaluation index qw is compared with the preset minimum allowable value within the set time period. If the speech separation evaluation index qw is lower than the minimum allowable value, an adjustment signal is triggered and sent to the technician to remind the technician to adjust the deep neural model;

[0095] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. Intelligent speech separation system based on deep learning, characterized by: include: Speech feature module: obtains audio data containing mixed speech and preprocesses the received audio data. After the preprocessing is completed, the audio data is framed according to the preset time length to obtain an audio frame sequence. After performing corresponding operations on the obtained audio frame sequence, a high-dimensional feature vector sequence is obtained and sent to the speech separation module; each high-dimensional feature vector represents the deep feature representation of each audio frame; The corresponding operations are performed on the obtained audio frame sequence, specifically: M1: Set different Hamming window coefficient combinations as variables, and use each Hamming window coefficient combination to perform windowing operations on the audio frames in the audio frame sequence in sequence; M2: Perform fast Fourier transform on the audio frame obtained by windowing to obtain a spectrum diagram, analyze the spectrum leakage ratio SLR and frequency resolution FR under each Hamming window coefficient combination, and obtain the spectrum leakage ratio SLR and frequency resolution FR of each group corresponding to the audio frame sequence under different Hamming window coefficient combinations. The average of each group of spectrum leakage ratio SLR and frequency resolution FR under the same Hamming window coefficient combination is calculated to obtain the spectrum leakage average ratio and frequency resolution average ratio under the same Hamming window coefficient combination; M3: Mark the spectrum leakage average ratio and frequency resolution average ratio as sq and se respectively, and preset the passing spectrum leakage ratio and passing frequency resolution corresponding to the spectrum leakage average ratio sq and the frequency resolution average ratio se respectively, according to the formula The spectrum leakage average ratio sq and the frequency resolution average ratio se under different Hamming window coefficient combinations are weightedly calculated to obtain the comprehensive performance value su of different Hamming window coefficient combinations for the current audio frame sequence; and They represent the passing spectrum leakage ratio and passing frequency resolution respectively, a1 and a2 are the influencing weight factors of the spectrum leakage average ratio sq and the frequency resolution average ratio se respectively; Select the Hamming window coefficient combination with the largest comprehensive value su as the optimal Hamming window coefficient combination for the current audio frame sequence; M4: Use a convolutional neural network to extract features from the audio frame spectrogram to obtain a feature vector sequence output by the convolutional neural network, and input the feature vector sequence into a long short-term memory network. The long short-term memory network processes the input feature vector sequence to learn the temporal characteristics of the audio signal, and finally obtains a high-dimensional feature vector sequence corresponding to the audio frame sequence; Speech separation module: receives a high-dimensional feature vector sequence and inputs it into a pre-built deep neural model. After the input is completed, the deep neural model executes the corresponding steps to obtain the final feature sequence composed of the separated speech signal components; Speech reconstruction module: Perform inverse feature transformation on the final feature sequence composed of each speech signal component to obtain the preliminary reconstruction result of the time domain waveform of each speech signal component, use the preliminary reconstruction result of the time domain waveform as the time domain speech signal of each speech signal component, and post-process the reconstructed time domain speech signal of each speech signal component. After the post-processing is completed, the output mode is determined according to the user's settings; User feedback module: collects feedback parameters of speech separation effect within a set time period, conducts comprehensive analysis, and determines whether the deep neural model needs to be adjusted based on the analysis results; feedback parameters include the number of repeated inputs, the number of optimization marks, and user feedback scores.

2. The deep learning-based intelligent speech separation system according to claim 1, characterized in that: After the input is completed, the deep neural model performs the corresponding steps, specifically: P1: The deep neural model preprocesses the received high-dimensional feature vector sequence; P2: Extract the audio frame sequence, calculate the autocorrelation function for each frame of audio, identify the delay time corresponding to the periodic peak in the autocorrelation function, and obtain the pitch frequency of each frame of audio. Quantize and normalize the obtained pitch frequency value, and after processing, mark the pitch feature vector corresponding to each time step as d1; Perform short-time Fourier transform on each frame of audio, then process it through a Mel filter to obtain a Mel spectrum, then take the logarithm of the Mel spectrum and perform discrete cosine transform, finally obtain the MFCC coefficient corresponding to each frame of audio, perform dimensionality reduction on the obtained MFCC coefficients, select the first X coefficients as the key timbre features; X>5; and quantize and normalize the key timbre features; after processing, the key timbre feature vector corresponding to each time step is d2; For the hidden representation of the speech source at all time steps, the corresponding pitch feature vector d1 and the key timbre feature vector d2 are fused, and the fused vector becomes D+d1+d2, where D represents the hidden representation of the speech source, and the fused hidden representation vector sequence is obtained; The fused hidden representation vector sequence is reshaped into H×T×(D+d1+d2), where H is the number of speech sources and T is the time step; the dot product operation is used to calculate the relevance score of each time step, and a relevance score matrix consisting of each group of relevance scores is constructed from this; For each time step, the softmax function is used to calculate and the calculation results of all time steps are integrated to obtain the attention weight distribution matrix; The hidden representations of the speech sources are weighted and summed according to the attention weight distribution matrix to obtain a weighted representation focused on the corresponding speech source at each time step. The weighted representations of each speech source are integrated as a feature sequence composed of the separated speech signal components.

3. The deep learning-based intelligent speech separation system according to claim 2, characterized in that: After the input is completed, the deep neural model executes the corresponding steps, including: P3: For each speech signal component separated in step P2, calculate the signal-to-noise ratio and the speech distortion; respectively preset the value intervals corresponding to the signal-to-noise ratio and the speech distortion, respectively, set each value interval corresponding to the signal-to-noise ratio to correspond to a signal-to-noise ratio score, and each value interval corresponding to the speech distortion to correspond to a distortion score; The weight coefficients corresponding to the preset signal-to-noise ratio score and the distortion score are respectively multiplied by the corresponding preset weight coefficients, and then the sum is calculated to obtain the separation quality index of each speech signal component.

4. The deep learning-based intelligent speech separation system according to claim 3, characterized in that: The final feature sequence composed of the separated speech signal components is obtained, specifically: P4: The preset minimum allowable index corresponding to the separation quality index. If the separation quality index of a speech signal component is lower than the preset minimum allowable index, the speech signal component is marked as an optimized signal component and input into the deep neural model again to repeat steps P2 and P3; P5: Compare the separation quality index obtained after re-input in step P4 with the preset minimum allowable index. If it is still lower than the preset minimum allowable index, calculate the difference between the current separation quality index and the previous set of separation quality indexes, and mark the difference as the improvement value; then re-input into the deep neural model to repeat steps P2 and P3, compare the obtained separation quality index with the preset minimum allowable index, if it is still lower than the preset minimum allowable index, calculate the improvement value again, if the improvement value obtained by two consecutive rounds of repeated input is lower than the preset reference value of the improvement, stop input, take the result obtained by the last round of separation of the optimized signal component as the final result, and replace it; P6: After the replacement is completed, the final feature sequence composed of the separated speech signal components is obtained.

5. The deep learning-based intelligent speech separation system according to claim 4, characterized in that: Collect feedback parameters of speech separation effect within a set time period and conduct comprehensive analysis. According to the analysis results, determine whether the deep neural model needs to be adjusted. Specifically: Count the number of repeated inputs received by the deep neural model within a set time period as the number of repeated inputs within the set time period; count the number of marked optimization signal components within the set time period as the number of optimized marks within the set time period; preset weight coefficients corresponding to the number of repeated inputs and the number of optimized marks, respectively, multiply the number of repeated inputs and the number of optimized marks by the corresponding preset weight coefficients, and then sum them to obtain the performance evaluation value f1 of the deep neural model within the set time period; After the user sets the output mode for output, the user's scores on the speech separation effect are collected as the user feedback score of the current speech separation effect; the average of the user feedback scores of each group within the set time period is calculated as the feedback average score f2; Using the formula The performance evaluation value f1 and the feedback average f2 within the set time period are weighted to obtain the speech separation evaluation index qw within the set time period; wherein and They represent the maximum value allowed for performance evaluation and the minimum value allowed for feedback score respectively; s1 and s2 are the weight factors affecting the performance evaluation value f1 and the feedback average score f2 respectively; The preset minimum allowable value corresponding to the speech separation evaluation index qw is compared with the preset minimum allowable value within the set time period. If the speech separation evaluation index qw is lower than the minimum allowable value, an adjustment signal is triggered and sent to the technician to remind the technician to adjust the deep neural model.

Images