Voice processing method and device
By employing a speech processing method that combines multi-scale short-time Fourier transform and multi-dimensional feature fusion, the problem of poor noise processing in embedded devices is solved, achieving rapid and accurate processing and quality improvement of speech signals.
Patent Information
- Application Number
- CN202511679035.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-14
- Publication Date
- 2026-02-13
AI Technical Summary
Existing speech enhancement technologies have high computational complexity in embedded devices and poor noise processing performance. In particular, their separation performance deteriorates as noise intensity increases. Furthermore, the traditional DUET algorithm lacks robustness and cannot effectively remove noise, resulting in poor speech quality.
Multi-scale short-time Fourier transform is used to obtain frequency domain signals at different scales. A mask matrix is generated by multi-dimensional feature extraction and fusion processing is performed. Power spectrum difference distribution is judged by time delay features, energy features, coherence coefficient features and JS divergence, and a target mask matrix is generated for speech signal processing.
It achieves rapid and accurate noise processing, improves voice quality, solves the problem of voice separation under noise, and enhances the voice signal processing performance of embedded devices.
Smart Images
Figure CN121528233A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of speech processing technology, and more specifically, to a speech processing method and apparatus. Background Technology
[0002] In the field of speech signal processing, especially in speech enhancement tasks, mainstream methods are divided into traditional signal processing-based algorithms and deep learning-based algorithms. While deep learning-based methods have demonstrated superior performance in speech enhancement and separation tasks, they suffer from inherently high computational complexity. These methods employ end-to-end optimization strategies, relying on large-scale neural network models (such as the Transformer architecture) to fit massive amounts of training data, and their computational complexity increases linearly with the duration of the speech. Although some lightweight models compress the number of parameters, they still fundamentally contradict the requirements for low power consumption and low latency in some speech enhancement scenarios. This efficiency bottleneck is particularly prominent in resource-constrained scenarios such as embedded devices. To address these issues, the DUET algorithm can be used. As a classic blind source separation method, the performance of the DUET algorithm is limited by its strong dependence on phase and energy differences. In the processing of noisy frames, because the noise signal disrupts the phase consistency and energy distribution pattern, the single-source point screening based on cosine angles produces bias points and interference points, ultimately leading to a significant increase in noise residue in the separated speech. Furthermore, the traditional DUET algorithm lacks robustness due to its neglect of feature types and limited scale, causing its performance to deteriorate sharply with increasing noise. This algorithm employs a local time-frequency analysis strategy, relying solely on the signal characteristics of the current frequency point for separation decisions, lacking utilization of global information. When noise intensity increases, frequency-level features are prone to falling into a "misjudgment trap"—they cannot distinguish between energy fluctuations caused by noise and the characteristics of the true sound source during single-source screening, leading to a precipitous drop in separation performance. In addition, when the microphone spacing is large, spatial aliasing occurs, with a phase difference greater than 2π, resulting in distortion of phase difference features. Summary of the Invention
[0003] This application provides a speech processing method and apparatus to at least solve the technical problem in the related art where a large amount of noise in the received speech leads to poor speech quality.
[0004] According to one aspect of the embodiments of this application, a speech processing method is provided, comprising: receiving raw speech data to be processed; performing short-time Fourier transforms on the raw speech data at various scales to obtain frequency domain signals at different scales; processing the frequency domain signals at different scales to obtain mask matrices at different scales, and fusing the mask matrices at different scales to obtain a target mask matrix; processing the frequency domain signals corresponding to the raw speech data using the target mask matrix to obtain processed frequency domain signals; and determining a processed speech signal based on the processed frequency domain signals.
[0005] Optionally, the frequency domain signals at different scales are processed to obtain mask matrices at different scales, including: extracting features from the frequency domain signals at different scales to obtain multi-dimensional features at different scales, wherein the multi-dimensional features include: time delay features, energy features, coherence coefficient features and target features for representing power spectrum difference distribution; and generating mask matrices at different scales based on the multi-dimensional features at different scales.
[0006] Optionally, the target feature is determined by the following methods: acquiring the energy of a first speech signal and the energy of a second speech signal in the original speech data; converting the energy of the first speech signal and the energy of the second speech signal into a first probability distribution and a second probability distribution, respectively; acquiring the JS divergence of the first probability distribution and the second probability distribution; and normalizing the JS divergence to obtain the target feature.
[0007] Optionally, fusing the mask matrices of different scales to obtain a target mask matrix includes: unifying the first-scale mask matrix and the second-scale mask matrix among the mask matrices of different scales according to multiple dimensions to obtain a first mask matrix; unifying the third-scale mask matrix and the second-scale mask matrix among the different mask matrices according to multiple dimensions to obtain a second mask matrix, wherein the scales of the first-scale mask matrix, the second-scale mask matrix, and the third-scale mask matrix increase sequentially; and fusing the first mask matrix, the second mask matrix, and the second-scale mask matrix to obtain the target mask matrix.
[0008] Optionally, unifying the first-scale mask matrix and the second-scale mask matrix in the mask matrices of different scales according to multiple dimensions to obtain the first mask matrix includes: performing frame alignment on the first-scale mask matrix and the second-scale mask matrix using an averaging operation to obtain a first-scale mask matrix with unified completion time; and adjusting the frequency of the first-scale mask matrix with unified completion time to be the same as the frequency corresponding to the second-scale mask matrix by linear interpolation to obtain the first mask matrix.
[0009] Optionally, the third-scale mask matrix and the second-scale mask matrix in the different mask matrices are unified according to multiple dimensions to obtain a second mask matrix, including: when the frame index of the second-scale mask matrix is an even index, the third-scale mask matrix is determined as the third-scale mask matrix with unified completion time; when the frame index of the second-scale mask matrix is an odd index, an averaging operation is used to perform frame alignment between the third-scale mask matrix and the second-scale mask matrix to obtain the third-scale mask matrix with unified completion time.
[0010] Optionally, the method further includes: obtaining the frequency interval of each frequency point in the third-scale mask matrix with uniform completion time, and selecting a target frequency point from the frequency points of the third-scale mask matrix according to twice the frequency interval; mapping the target frequency point in the third-scale mask matrix with the frequency points in the second-scale mask matrix to obtain the second mask matrix.
[0011] Optionally, fusing the first mask matrix, the second mask matrix, and the second-scale mask matrix to obtain the target mask matrix includes: determining a mask statistical function by summing the first mask matrix, the second mask matrix, and the second-scale mask matrix; and determining the target mask matrix based on a preset threshold and the mask statistical function.
[0012] According to another aspect of the embodiments of this application, a speech processing apparatus is also provided, comprising: a receiving module for receiving raw speech data to be processed; a transform module for performing short-time Fourier transforms on the raw speech data at various scales to obtain frequency domain signals at different scales; a first processing module for processing the frequency domain signals at different scales to obtain mask matrices at different scales, and fusing the mask matrices at different scales to obtain a target mask matrix; a second processing module for processing the frequency domain signal corresponding to the raw speech data using the target mask matrix to obtain a processed frequency domain signal; and a determining module for determining the processed speech signal based on the processed frequency domain signal.
[0013] According to another aspect of the embodiments of this application, a computer device is also provided, including: a memory and a processor, wherein the memory is used to store program instructions; and the processor, connected to the memory, is used to execute the above-described voice processing method.
[0014] According to another aspect of the embodiments of this application, a computer program product is also provided, including computer instructions that, when executed by a processor, implement the above-described voice processing method.
[0015] In this embodiment, the method involves receiving raw speech data to be processed; performing short-time Fourier transforms on the raw speech data at various scales to obtain frequency domain signals at different scales; processing the frequency domain signals at different scales to obtain mask matrices at different scales; fusing the mask matrices at different scales to obtain a target mask matrix; processing the frequency domain signals corresponding to the raw speech data using the target mask matrix to obtain processed frequency domain signals; determining the processed speech signal based on the processed frequency domain signals. By converting the raw speech data into frequency domain signals at different scales and processing the frequency domain signals at different scales to obtain mask matrices at different scales, and then processing the frequency domain signals corresponding to the raw speech data using the mask matrices to obtain the processed speech signal, the method achieves the goal of quickly and accurately processing noise in the raw speech data, thereby improving speech quality and solving the technical problem in related technologies where a large amount of noise in the received speech leads to poor speech quality. Attached Figure Description
[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0017] Figure 1 This is a hardware structure block diagram of a computer target terminal for implementing a voice processing method according to an embodiment of this application;
[0018] Figure 2 This is a flowchart of a speech processing method according to an embodiment of this application;
[0019] Figure 3 This is a schematic diagram of an application scenario according to an embodiment of this application;
[0020] Figure 4 This is a flowchart of another speech processing method according to an embodiment of this application;
[0021] Figure 5 This is a flowchart of a target feature determination method according to an embodiment of this application;
[0022] Figure 6 This is a flowchart of a mask matrix fusion method according to an embodiment of this application;
[0023] Figure 7 This is a structural diagram of a voice processing device according to an embodiment of this application. Detailed Implementation
[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.
[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0026] The information collected in this application embodiment is information and data authorized by the user or fully authorized by all parties. The collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant regions, and necessary confidentiality measures have been taken. It does not violate public order and good morals, and provides corresponding operation entry points for users to choose to authorize or reject the automated decision results. If the user chooses to reject, the process will proceed to the expert decision-making process.
[0027] The technical terms used in this application are explained as follows:
[0028] DUET (Degenerate Unmixing Estimation Technique) is a blind source separation algorithm based on a dual-microphone array that uses the time delay difference (TDOA) and amplitude ratio information between signals to achieve sound source separation.
[0029] STFT (Short-time Fourier Transform): A method for converting speech from the time domain to the frequency domain.
[0030] JS divergence (Jensen-Shannon Divergence) is a metric that quantifies the difference between two probability distributions. Essentially, it is a symmetric and normalized improvement of KL divergence (Kullback-Leibler Divergence), which solves the problems of asymmetry and unlimited value of KL divergence, making it more suitable for measuring the "similarity" between distributions.
[0031] JS-PLD (Power Level Difference based Jensen-Shannon Divergence): A feature designed in this application embodiment to determine the distribution characteristics of PLD using JS divergence.
[0032] To address the problems existing in related technologies, this application provides a speech processing method that can run on... Figure 1 The computer target terminal shown is explained below.
[0033] The speech processing method embodiments provided in this application can be executed in a mobile target terminal, a computer target terminal, or a similar computing device. Figure 1 A hardware block diagram of a computer target terminal for implementing a speech processing method is shown. Figure 1 As shown, the computer target terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions connected via wired and / or wireless networks. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, the computer target terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.
[0034] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be implemented wholly or partially as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be wholly or partially integrated into any other element in the computer target terminal 10. As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor target terminal path connected to an interface).
[0035] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the voice processing method in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the aforementioned voice processing method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer target terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0036] The transmission module 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer target terminal 10. In one example, the transmission module 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission module 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.
[0037] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer target terminal 10.
[0038] It should be noted here that, in some optional embodiments, the above... Figure 1 The computer target terminal shown may include hardware components (including circuitry), software components (including computer code stored on a computer-readable medium), or a combination of both hardware and software components. It should be noted that... Figure 1 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computer target terminal.
[0039] In the above operating environment, this application provides an embodiment of a voice processing method. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than that shown here.
[0040] Figure 2 This is a flowchart of a speech processing method according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:
[0041] Step S202: Receive the raw voice data to be processed;
[0042] like Figure 3 As shown, raw voice data can be acquired through a dual-microphone headset (MIC1 and MIC2).
[0043] Step S204: Perform short-time Fourier transforms on the original speech data at various scales to obtain frequency domain signals at different scales.
[0044] Specifically, multi-scale STFT is performed on the acquired noisy speech, mainly divided into fine-scale (first scale), standard-scale (second scale), and coarse-scale (third scale). The fine-scale scale obtains more refined temporal features, while the coarse-scale scale obtains more refined frequency features. Different scales correspond to different frame lengths. For example: Fine-scale STFT parameters: frame length = 16ms, frame shift = 4ms, to capture more refined temporal features. Standard-scale STFT parameters: frame length = 32ms, frame shift = 8ms, balancing time and frequency resolution. Coarse-scale STFT parameters: frame length = 64ms, frame shift = 16ms, to obtain more refined frequency features.
[0045] Step S206: Process the frequency domain signals of different scales respectively to obtain mask matrices of different scales, and fuse the mask matrices of different scales to obtain the target mask matrix;
[0046] Step S208: The target mask matrix is used to process the frequency domain signal corresponding to the original speech data to obtain the processed frequency domain signal;
[0047] Step S210: Determine the processed speech signal based on the processed frequency domain signal.
[0048] Through steps S202 to S210 above, the process involves receiving raw speech data to be processed; performing short-time Fourier transforms on the raw speech data at various scales to obtain frequency domain signals at different scales; processing the frequency domain signals at different scales to obtain mask matrices at different scales; fusing the mask matrices at different scales to obtain a target mask matrix; using the target mask matrix to process the frequency domain signal corresponding to the raw speech data to obtain a processed frequency domain signal; and determining the processed speech signal based on the processed frequency domain signal. By converting the raw speech data into frequency domain signals at different scales and processing these signals to obtain mask matrices at different scales, and then using these mask matrices to process the frequency domain signal corresponding to the raw speech data to obtain the processed speech signal, the process achieves the goal of rapidly and accurately processing noise in the raw speech data, thereby improving speech quality and solving the technical problem in related technologies where a large amount of noise in the received speech leads to poor speech quality. The following is a detailed explanation.
[0049] To better illustrate the speech processing method proposed in the embodiments of this application, another speech processing method is also proposed in the embodiments of this application, such as... Figure 4 As shown, the process includes: performing multi-scale STFT on the original speech data (noisy speech) to obtain a multi-scale frequency domain signal; estimating frequency point and frame-level features on the multi-scale frequency domain signal to obtain multi-dimensional features; performing mask estimation on the multi-dimensional features; processing the frequency domain signal of the original speech data using the mask; and finally performing ISTFT (Inverse Short-Time Fourier Transform) on the processed frequency domain signal to obtain the processed speech data (clean speech).
[0050] In some embodiments of this application, the specific steps for processing frequency domain signals at different scales to obtain mask matrices at different scales are as follows: feature extraction is performed on frequency domain signals at different scales to obtain multi-dimensional features at different scales, wherein the multi-dimensional features include: time delay features, energy features, coherence coefficient features, and target features for representing power spectrum difference distribution; mask matrices at different scales are generated based on the multi-dimensional features at different scales.
[0051] Specifically, time delay features are extracted based on frequency domain signals obtained at three scales. In this embodiment, time delay can be measured using an angular vector. To unify the measurement range, normalization is performed after extracting the time delay features. The formula is as follows:
[0052] In the formula, This represents the time delay feature matrix, used to measure the signal at the nth frequency point and the kth time frame. With signal The time delay difference between them, the signal With signal These are used to represent the frequency domain representations of the speech signals (first and second channel speech signals) received by different microphones, where n represents the frequency index and k represents the time frame index. This represents the argument function, used for signal extraction. With signal The phase difference between them Indicates frequency, This indicates the maximum delay.
[0053] In addition, the energy features extracted from the frequency domain signals obtained at the three scales are as follows: energy is measured using the power spectral density ratio, and the calculation method is as follows:
[0054]
[0055] In the formula, This represents the energy characteristic matrix, used to measure the signal at the nth frequency point and the kth time frame. With signal The power ratio between them Represents the logarithmic function with base 10. Represents the maximum value of the energy characteristic. It represents the absolute value.
[0056] It should also be noted that coherence coefficient features are extracted from frequency domain signals obtained at three scales. Compared with time delay and energy features, coherence coefficients have the advantages of better robustness and higher comprehensiveness. Therefore, coherence coefficients are used as a supplement to the original features.
[0057]
[0058] In the formula, This represents the coherence coefficient matrix, used to represent the signal at the nth frequency point and the kth time frame. With signal The coherence between them This represents the transpose of a matrix.
[0059] In some embodiments of this application, the target feature is determined by: acquiring the energy of a first speech signal and the energy of a second speech signal from the original speech data; converting the energy of the first speech signal and the energy of the second speech signal into a first probability distribution and a second probability distribution, respectively; acquiring the JS divergence of the first probability distribution and the second probability distribution; and normalizing the JS divergence to obtain the target feature.
[0060] Specifically, the process of calculating the energy of the two speech signals is shown in the following formula:
[0061]
[0062]
[0063] In the formula, This represents the energy of the first voice signal. This represents the energy of the second voice signal (the energy value of the signal at the k-th time frame and the n-th frequency point).
[0064] Since JS divergence requires the input to be a probability distribution (non-negative, normalized), the energy matrix needs to be... and Transform into probability distribution and First, nonnegation correction is needed to ensure that all elements are nonnegative;
[0065] Normalization: Normalizes the sum of all elements of a matrix to 1.
[0066]
[0067]
[0068] In the formula, Let represent the first probability matrix, which represents the signal at each time frame k and frequency point n. The probability value at that time frame and frequency point. The second probability matrix represents the signal at each time frame k and frequency point n. The probability value at that time frame and frequency point. This represents the cumulative sum of the energy values of the first-channel speech signal across all time frames and frequency points. This represents the sum of the energy values of the second voice signal across all time frames and frequency points.
[0069] like Figure 5 As shown, calculate the probability distribution. and The process of calculating the JS divergence is shown in the following formula:
[0070]
[0071] In the formula, express and JS divergence between them express and KL divergence between them express and The KL divergence between them.
[0072] in, yes and The average distribution of is calculated using the following formula:
[0073]
[0074] KL represents the KL divergence, and the formula for its calculation is:
[0075]
[0076]
[0077] It can be obtained This is a value that takes values in the range (0, ...). To ensure a consistent measurement range, functions of this type need to be normalized.
[0078]
[0079] In the formula, Indicates the target features.
[0080] At this point, the frame-level mask feature JS-PLD (target feature) has been obtained. This method can be used to obtain JS-PLDs at three scales, namely... (First Scale) (Second scale) and (Third scale).
[0081] In some embodiments of this application, the specific steps for fusing the mask matrices of different scales to obtain the target mask matrix are as follows: unifying the first-scale mask matrix and the second-scale mask matrix among the mask matrices of different scales according to multiple dimensions to obtain a first mask matrix; unifying the third-scale mask matrix and the second-scale mask matrix among the different mask matrices according to multiple dimensions to obtain a second mask matrix, wherein the scales of the first-scale mask matrix, the second-scale mask matrix, and the third-scale mask matrix increase sequentially; fusing the first mask matrix, the second mask matrix, and the second-scale mask matrix to obtain the target mask matrix.
[0082] The specific steps for unifying the first-scale mask matrix and the second-scale mask matrix in the mask matrices of different scales according to multiple dimensions to obtain the first mask matrix are as follows: The first-scale mask matrix and the second-scale mask matrix are frame-aligned using an averaging operation to obtain a first-scale mask matrix with unified completion time; the frequency of the first-scale mask matrix with unified completion time is adjusted to be the same as the frequency corresponding to the second-scale mask matrix using linear interpolation to obtain the first mask matrix.
[0083] It should be noted that the specific steps for unifying the third-scale mask matrix and the second-scale mask matrix in the different mask matrices according to multiple dimensions to obtain the second mask matrix are as follows: when the frame index of the second-scale mask matrix is an even index, the third-scale mask matrix is determined as the third-scale mask matrix with unified completion time; when the frame index of the second-scale mask matrix is an odd index, the third-scale mask matrix and the second-scale mask matrix are frame aligned using an averaging operation to obtain the third-scale mask matrix with unified completion time.
[0084] The second mask matrix is determined as follows: obtain the frequency interval of each frequency point in the third-scale mask matrix with uniform completion time, and select the target frequency point from the frequency points of the third-scale mask matrix according to twice the frequency interval; map the target frequency point in the third-scale mask matrix with the frequency points in the second-scale mask matrix one by one to obtain the second mask matrix.
[0085] It is understandable that when determining the second mask matrix, if the frame index of the second-scale mask matrix is an even index, the value corresponding to the third-scale mask matrix is determined as the value corresponding to the second mask matrix. If the frame index of the second-scale mask matrix is an odd index, an averaging operation is used to align the value corresponding to the third-scale mask matrix and the value corresponding to the second-scale mask matrix to obtain the second mask matrix.
[0086] The mask matrix is determined as shown in the following formula:
[0087]
[0088] In the formula, This represents the minimum value of the time delay matrix. The minimum value of the energy matrix. This represents the minimum value of the target feature. This indicates that multiple conditions are met simultaneously. This indicates that multiple conditions are combined.
[0089] In one embodiment and Take 0.2, Take 0.3, Let's take 0.4. Based on this method, we can obtain the first-scale mask matrix at three different scales. Third-scale mask matrix Second-scale mask matrix .
[0090] Through three scales of STFT and a series of processing steps, the first-scale mask matrix at the three scales has been obtained. Second-scale mask matrix and third-scale mask matrix However, due to the different scales, the dimensions of the three mask matrices are not the same, so it is necessary to unify the scale. In this application, the standard scale is used as the core, and the mask matrices of the fine scale and coarse scale are standardized to unify the dimensions with the standard scale mask matrix.
[0091] like Figure 6 As shown, the specific process for unifying the dimensions of the fine-scale mask matrix with the standard-scale mask matrix is as follows:
[0092] The transition from a fine scale to a standard scale consists of two parts: unification of the time dimension and unification of the frequency dimension. The method for unifying the time dimension is as follows:
[0093]
[0094] In the formula, This represents the first-scale mask matrix with uniform completion time. , These represent two adjacent time frames within the same time frame k in the first-scale mask, corresponding to the second-scale mask. , The mask matrix value.
[0095] The unified calculation method for the frequency dimension is as follows:
[0096] Let the frequency index of the fine scale (first scale) be... (0, 128), corresponding to the frequency Let the standard scale (second scale) frequency index be... (0, 256), corresponding to the frequency For each Find its two adjacent frequencies. and ( < < The standardized mask is obtained through linear interpolation:
[0097]
[0098] In the formula, Denotes the first mask matrix. and These represent the frequencies in the first-scale mask matrix corresponding to the frequencies in the second-scale mask, respectively, within the first-scale mask matrix with uniform completion time. two adjacent frequency points and The mask matrix value.
[0099] The specific process for unifying the dimensions of the coarse-scale mask matrix with those of the standard-scale mask matrix is as follows:
[0100] The transition from a coarse scale to a standard scale is also divided into two parts: time dimension unification and frequency dimension unification. The method for time dimension unification is as follows:
[0101] When frame index When it is even:
[0102]
[0103] In the formula, This represents a third-scale mask matrix with uniform completion time. This represents the third-scale mask matrix. This represents the frequency index of the third-scale mask matrix. The frame index represents the third-scale mask matrix. This represents the frame index of the second-scale mask matrix.
[0104] When frame index When it is an odd number:
[0105]
[0106] In the formula, and These represent the adjacent time frames in the third-scale mask matrix that correspond to the time frames in the second-scale mask matrix. and The mask value.
[0107] The unified calculation method for the frequency dimension is as follows:
[0108] Let the fine-scale frequency index be... (0, 512), corresponding to the frequency Let the standard scale frequency index be... (0, 256), corresponding to the frequency The mask after dimensional unification is as follows:
[0109]
[0110] In the formula, Denotes the second mask matrix, representing The value of the nth frequency point is taken from the value of the 2nth frequency point in the third-scale mask matrix with unified completion time. .
[0111] In some embodiments of this application, the specific steps for fusing the first mask matrix, the second mask matrix, and the third-scale mask matrix to obtain the target mask matrix are as follows: determining the mask statistical function by summing the first mask matrix, the second mask matrix, and the third-scale mask matrix; and determining the target mask matrix based on a preset threshold and the mask statistical function.
[0112] Specifically, the mask statistics function is as follows:
[0113]
[0114] In the formula, This represents the mask statistical function. This represents the second-scale mask matrix. Denotes the first mask matrix. This represents the second mask matrix.
[0115] It is understandable that the first mask matrix is obtained by unifying the array dimensions of the first-scale mask matrix and the second-scale mask matrix, and the second mask matrix is obtained by unifying the array dimensions of the third-scale mask matrix and the second-scale mask matrix. It should be noted that the array dimensions include the time dimension and the frequency dimension.
[0116] Figure 7 A speech processing device is shown, the device comprising:
[0117] Receiver module 70 is used to receive raw voice data to be processed;
[0118] The transformation module 72 is used to perform short-time Fourier transforms on the original speech data at various different scales to obtain frequency domain signals at different scales.
[0119] The first processing module 74 is used to process the frequency domain signals of different scales respectively to obtain mask matrices of different scales, and to fuse the mask matrices of different scales to obtain the target mask matrix.
[0120] The second processing module 76 is used to process the frequency domain signal corresponding to the original speech data using the target mask matrix to obtain the processed frequency domain signal.
[0121] The determination module 78 is used to determine the processed speech signal based on the processed frequency domain signal.
[0122] The aforementioned speech processing device receives raw speech data to be processed; performs short-time Fourier transforms on the raw speech data at various scales to obtain frequency domain signals at different scales; processes the frequency domain signals at different scales to obtain mask matrices at different scales, and fuses the mask matrices at different scales to obtain a target mask matrix; processes the frequency domain signals corresponding to the raw speech data using the target mask matrix to obtain processed frequency domain signals; and determines the processed speech signal based on the processed frequency domain signals. By converting the raw speech data into frequency domain signals at different scales and processing the frequency domain signals at different scales to obtain mask matrices at different scales, and then processing the frequency domain signals corresponding to the raw speech data using the mask matrices to obtain the processed speech signal, the device achieves the goal of rapidly and accurately processing noise in the raw speech data, thereby improving speech quality and solving the technical problem in related technologies where a large amount of noise in the received speech leads to poor speech quality.
[0123] The first processing module 74 includes: a first processing submodule, used to process frequency domain signals of different scales to obtain mask matrices of different scales, including: extracting features from frequency domain signals of different scales to obtain multi-dimensional features at different scales, wherein the multi-dimensional features include: time delay features, energy features, coherence coefficient features and target features for representing power spectrum difference distribution; and generating mask matrices of different scales based on the multi-dimensional features at different scales.
[0124] The first processing submodule includes a first processing unit, used to determine the target feature by: acquiring the energy of a first speech signal and the energy of a second speech signal in the original speech data; converting the energy of the first speech signal and the energy of the second speech signal into a first probability distribution and a second probability distribution, respectively; acquiring the JS divergence of the first probability distribution and the second probability distribution; and normalizing the JS divergence to obtain the target feature.
[0125] The first processing submodule further includes a fusion unit, used to fuse the mask matrices of different scales to obtain a target mask matrix, including: unifying the first-scale mask matrix and the second-scale mask matrix among the mask matrices of different scales according to multiple dimensions to obtain a first mask matrix; unifying the third-scale mask matrix and the second-scale mask matrix among the different mask matrices according to multiple dimensions to obtain a second mask matrix, wherein the scales of the first-scale mask matrix, the second-scale mask matrix, and the third-scale mask matrix increase sequentially; and fusing the first mask matrix, the second mask matrix, and the second-scale mask matrix to obtain the target mask matrix.
[0126] The fusion unit includes a first subunit, used to unify the first-scale mask matrix and the second-scale mask matrix in the mask matrices of different scales according to multiple dimensions to obtain a first mask matrix. This includes: performing frame alignment on the first-scale mask matrix and the second-scale mask matrix using an averaging operation to obtain a first-scale mask matrix with unified completion time; and adjusting the frequency of the first-scale mask matrix with unified completion time to be the same as the frequency corresponding to the second-scale mask matrix using linear interpolation to obtain the first mask matrix.
[0127] The fusion unit further includes a second subunit, used to unify the third-scale mask matrix and the second-scale mask matrix in the different mask matrices according to multiple dimensions to obtain a second mask matrix, including: when the frame index of the second-scale mask matrix is an even index, determining the third-scale mask matrix as a third-scale mask matrix with unified completion time; when the frame index of the second-scale mask matrix is an odd index, using an averaging operation to perform frame alignment between the third-scale mask matrix and the second-scale mask matrix to obtain the third-scale mask matrix with unified completion time.
[0128] The second subunit further includes a mapping subunit, used to obtain the frequency interval of each frequency point in the third-scale mask matrix with unified completion time, and select a target frequency point from the frequency points of the third-scale mask matrix with unified completion time according to twice the frequency interval; and map the target frequency point in the third-scale mask matrix to the frequency points in the second-scale mask matrix one by one to obtain the second mask matrix.
[0129] The fusion unit further includes a third subunit, used to fuse the first mask matrix, the second mask matrix, and the second-scale mask matrix to obtain the target mask matrix, including: determining a mask statistical function by summing the first mask matrix, the second mask matrix, and the second-scale mask matrix; and determining the target mask matrix according to a preset threshold and the mask statistical function.
[0130] It should be noted that, Figure 7 The voice processing device shown is used to perform Figure 2 The speech processing method shown above also applies to this speech processing device, and will not be repeated here.
[0131] This application also provides a computer device, including: a memory and a processor, wherein the memory is used to store program instructions; and the processor, connected to the memory, is used to execute the above-described voice processing method.
[0132] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the speech processing method in this application.
[0133] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0134] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0135] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.
[0136] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0137] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0138] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.
[0139] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A speech processing method, characterized in that, include: Receive raw voice data to be processed; The original speech data is subjected to short-time Fourier transforms at various scales to obtain frequency domain signals at different scales. The frequency domain signals at different scales are processed separately to obtain mask matrices of different scales, and the mask matrices of different scales are fused to obtain the target mask matrix; The target mask matrix is used to process the frequency domain signal corresponding to the original speech data to obtain the processed frequency domain signal; The processed speech signal is determined based on the processed frequency domain signal.
2. The method according to claim 1, characterized in that, Frequency domain signals of different scales are processed to obtain mask matrices of different scales, including: Feature extraction is performed on frequency domain signals at different scales to obtain multi-dimensional features at different scales. The multi-dimensional features include: time delay features, energy features, coherence coefficient features, and target features for representing power spectrum difference distribution. Mask matrices of different scales are generated based on multi-dimensional features at different scales.
3. The method according to claim 2, characterized in that, The target features are determined in the following ways, including: The energy of the first speech signal and the energy of the second speech signal in the original speech data are obtained respectively. The energy of the first speech signal and the energy of the second speech signal are respectively converted into a first probability distribution and a second probability distribution; Obtain the JS divergence of the first probability distribution and the second probability distribution; The target feature is obtained by normalizing the JS divergence.
4. The method according to claim 2, characterized in that, The mask matrices of different scales are fused to obtain the target mask matrix, including: The first-scale mask matrix and the second-scale mask matrix in the mask matrices of different scales are unified according to multiple dimensions to obtain the first mask matrix; The third-scale mask matrix and the second-scale mask matrix in the mask matrices of different scales are unified according to multiple dimensions to obtain the second mask matrix, wherein the scales of the first-scale mask matrix, the second-scale mask matrix and the third-scale mask matrix increase sequentially. The first mask matrix, the second mask matrix, and the second scale mask matrix are fused to obtain the target mask matrix.
5. The method according to claim 4, characterized in that, The first-scale mask matrix and the second-scale mask matrix in the mask matrices of different scales are unified according to multiple dimensions to obtain the first mask matrix, which includes: An averaging operation is used to perform frame alignment between the first-scale mask matrix and the second-scale mask matrix to obtain a first-scale mask matrix with uniform completion time. The frequency of the first-scale mask matrix with unified completion time is adjusted to be the same as the frequency corresponding to the second-scale mask matrix by linear interpolation, thus obtaining the first mask matrix.
6. The method according to claim 4, characterized in that, The third-scale mask matrix and the second-scale mask matrix from the different scale mask matrices are unified according to multiple dimensions to obtain the second mask matrix, including: If the frame index of the second-scale mask matrix is an even index, the third-scale mask matrix is determined to be a third-scale mask matrix with uniform completion time. When the frame index of the second-scale mask matrix is an odd number, an averaging operation is used to align the third-scale mask matrix and the second-scale mask matrix to obtain the third-scale mask matrix with uniform completion time.
7. The method according to claim 6, characterized in that, The method further includes: Obtain the frequency interval of each frequency point in the third-scale mask matrix with uniform completion time, and select the target frequency point from the frequency points of the third-scale mask matrix according to twice the frequency interval; The target frequency points in the third-scale mask matrix with unified completion time are mapped one-to-one with the frequency points in the second-scale mask matrix to obtain the second mask matrix.
8. The method according to claim 4, characterized in that, The target mask matrix is obtained by fusing the first mask matrix, the second mask matrix, and the second-scale mask matrix, including: The sum of the first mask matrix, the second mask matrix, and the second-scale mask matrix determines the mask statistical function; The target mask matrix is determined based on a preset threshold and the mask statistical function.
9. A voice processing device, characterized in that, include: The receiving module is used to receive the raw voice data to be processed; The transformation module is used to perform short-time Fourier transforms on the original speech data at various scales to obtain frequency domain signals at different scales. The first processing module is used to process the frequency domain signals of different scales respectively to obtain mask matrices of different scales, and to fuse the mask matrices of different scales to obtain the target mask matrix. The second processing module is used to process the frequency domain signal corresponding to the original speech data using the target mask matrix to obtain the processed frequency domain signal. The determination module is used to determine the processed speech signal based on the processed frequency domain signal.
10. A computer device, characterized in that, include: A memory and a processor, wherein the memory is used to store program instructions; The processor, connected to the memory, is used to execute the speech processing method according to any one of claims 1 to 8.
11. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the speech processing method according to any one of claims 1 to 8.