Audio processing method and device, medium and electronic equipment

Abstract

The invention belongs to the technical field of artificial intelligence, and particularly relates to an audio processing method, an audio processing device, a computer readable medium, electronic equipment and a computer program product. The audio processing method comprises the following steps: performing feature extraction on an audio signal to obtain frequency domain features of a plurality of audio frames; the audio signal carries a voice signal and a noise signal; serialized modeling is carried out on the frequency domain features of the audio frames, intra-frame sequence features are obtained, and the intra-frame sequence features are used for representing the frequency sequence relation among a plurality of frequency points in the single audio frame; serialization modeling is carried out on the intra-frame sequence features of the multiple audio frames to obtain inter-frame sequence features, and the inter-frame sequence features are used for representing the time sequence relation among the multiple audio frames; and performing signal modulation on the audio signal according to the inter-frame sequence features to obtain a voice signal after noise signal removal. According to the embodiment of the invention, the speech enhancement noise reduction effect can be improved.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to an audio processing method, an audio processing device, a computer-readable medium, an electronic device, and a computer program product. Background Art

[0002] The rapid advancement of hardware computing power, particularly the superior parallel computing capabilities of graphics processing units (GPUs), has led to a surge in the development of deep learning technology. Currently, deep learning-based speech enhancement and noise reduction methods are widely used in video conferencing, phone calls, intelligent voice assistants, speech recognition, and other scenarios, bringing significant benefits to people's daily lives.

[0003] However, due to the complexity of the noise environment in real application scenarios, deep learning-based speech enhancement and noise reduction methods usually have obvious noise residue or speech damage problems, and therefore have certain application limitations. Summary of the Invention

[0004] The present application provides an audio processing method, an audio processing device, a computer-readable medium, an electronic device, and a computer program product, the purpose of which is to improve the effect of speech enhancement and noise reduction.

[0005] According to one aspect of an embodiment of the present application, there is provided an audio processing method, the method comprising:

[0006] Extracting features from an audio signal to obtain frequency domain features of multiple audio frames; the audio signal carries a speech signal and a noise signal;

[0007] Performing serial modeling on the frequency domain features of the audio frame to obtain intra-frame sequence features, where the intra-frame sequence features are used to represent a frequency sequence relationship between multiple frequency points in a single audio frame;

[0008] Performing serial modeling on intra-frame sequence features of the plurality of audio frames to obtain inter-frame sequence features, where the inter-frame sequence features are used to represent a temporal sequence relationship between the plurality of audio frames;

[0009] The audio signal is modulated according to the inter-frame sequence feature to obtain a speech signal after the noise signal is removed.

[0010] According to one aspect of an embodiment of the present application, there is provided an audio processing device, the device comprising:

[0011] an extraction module configured to perform feature extraction on an audio signal to obtain frequency domain features of a plurality of audio frames; the audio signal carries a speech signal and a noise signal;

[0012] A first modeling module is configured to perform serial modeling on the frequency domain features of the audio frame to obtain an intra-frame sequence feature, where the intra-frame sequence feature is used to represent a frequency sequence relationship between multiple frequency points in a single audio frame;

[0013] a second modeling module configured to perform serial modeling on the intra-frame sequence features of the plurality of audio frames to obtain inter-frame sequence features, wherein the inter-frame sequence features are used to represent a temporal sequence relationship between the plurality of audio frames;

[0014] The modulation module is configured to perform signal modulation on the audio signal according to the inter-frame sequence feature to obtain a speech signal after removing the noise signal.

[0015] In some embodiments of the present application, based on the above technical solution, the modulation module includes:

[0016] a feature restoration module configured to decode and restore the inter-frame sequence features to obtain mask information for removing noise signals;

[0017] The audio modulation module is configured to perform signal modulation on the audio signal according to the mask information to obtain a speech signal after the noise signal is removed.

[0018] In some embodiments of the present application, based on the above technical solution, the audio modulation module includes:

[0019] a frequency domain spectrum modulation module, configured to perform signal modulation on the frequency domain spectrum of the audio signal according to the mask information to obtain a frequency domain spectrum of the speech signal after noise removal;

[0020] The frequency domain spectrum transformation module is configured to perform an inverse discrete cosine transform on the frequency domain spectrum of the speech signal to obtain a time domain spectrum of the speech signal.

[0021] In some embodiments of the present application, based on the above technical solution, the feature restoration module is further configured to: decode and restore the inter-frame sequence features in sequence through multiple serially connected decoders to obtain mask information for removing noise signals; wherein the input data of the decoder includes the output data of the previous decoder and the output data of an encoder with the same depth as the decoder, and the encoder is used to encode and compress the audio signal.

[0022] In some embodiments of the present application, based on the above technical solution, the decoder includes a transposed convolution layer, a batch normalization layer and an activation layer connected in sequence, and the encoder includes a convolution layer, a batch normalization layer and an activation layer connected in sequence; the transposed convolution layer is used to perform transposed convolution processing on the input data to obtain data with increased dimension, the convolution layer is used to perform convolution processing on the input data to obtain data with reduced dimension, the batch normalization layer is used to normalize the input data, and the activation layer is used to introduce nonlinear features to the input data.

[0023] In some embodiments of the present application, based on the above technical solution, the convolution step size of the convolution layer in the frequency domain space is greater than one, the convolution step size of the convolution layer in the time domain space is equal to one, and the transposed convolution layer has the same convolution step size as the convolution layer.

[0024] In some embodiments of the present application, based on the above technical solution, the extraction module is further configured to: perform frame and window processing on the audio signal to obtain time domain spectra of multiple audio frames; perform discrete cosine transform on the time domain spectrum of the audio frame to obtain frequency domain spectrum of the audio frame; and encode and compress the frequency domain spectrum of the audio frame in sequence through multiple serially connected encoders to obtain frequency domain features of the audio frame.

[0025] In some embodiments of the present application, based on the above technical solution, the first modeling module includes:

[0026] a sorting module, configured to sort the frequency domain features of the audio frames according to a specified frequency order to form a frequency domain feature sequence;

[0027] a forward mapping module configured to map the frequency domain features of the audio frame in the frequency domain feature sequence in a forward-to-backward order to obtain forward sequence features of each frequency point; the forward sequence features of the current frequency point have a mapping relationship with the forward sequence features of the previous frequency point;

[0028] a reverse mapping module configured to map the frequency domain features of the audio frame in the frequency domain feature sequence from back to front to obtain reverse sequence features of each frequency point; the reverse sequence feature of the current frequency point has a mapping relationship with the reverse sequence feature of the next frequency point;

[0029] The feature determination module is configured to determine the intra-frame sequence feature of the current frequency point according to the forward sequence feature and the reverse sequence feature of the current frequency point.

[0030] In some embodiments of the present application, based on the above technical solution, the forward mapping module or the reverse mapping module includes:

[0031] A first splicing module is configured to splice the sequence characteristics of the reference frequency point with the frequency domain characteristics of the current frequency point to obtain first input data, wherein the reference frequency point is the reference frequency point or the subsequent frequency point of the current frequency point;

[0032] a first mapping module configured to perform mapping processing on the first input data to obtain a reset weight and an update weight corresponding to the current frequency point, the reset weight being used to reset the influence of the sequence characteristics of the reference frequency point on the input data, and the update weight being used to update the sequence characteristics of the current frequency point;

[0033] A second splicing module is configured to perform a weighted operation on the sequence features of the reference frequency points according to the reset weights, and then splice the results with the frequency domain features of the current frequency points to obtain second input data;

[0034] A second mapping module is configured to perform mapping processing on the second input data to obtain the initial sequence feature of the current frequency point;

[0035] The feature weighting module is configured to perform a weighted operation on the sequence feature of the reference frequency point and the initial sequence feature of the current frequency point according to the updated weight to obtain the updated sequence feature of the current frequency point.

[0036] In some embodiments of the present application, based on the above technical solution, the second modeling module includes:

[0037] a time-frequency transformation module configured to perform time-frequency transformation on the intra-frame sequence features of the plurality of audio frames to obtain time domain distribution information of the intra-frame sequence features at a plurality of frequency points;

[0038] The time domain modeling module is configured to perform serial modeling on the time domain distribution information at each of the frequency points to obtain inter-frame sequence features.

[0039] In some embodiments of the present application, based on the above technical solution, the time domain modeling module is further configured to: map the time domain distribution information of each moment on the frequency point in sequence according to a specified time sequence to obtain inter-frame sequence features; the inter-frame sequence features of the current moment have a mapping relationship with the inter-frame sequence features of the reference moment, and the reference moment is the moment before or after the current moment.

[0040] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the audio processing method in the above technical solution is implemented.

[0041] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to implement the audio processing method in the above technical solution.

[0042] According to one aspect of an embodiment of the present application, a computer program product is provided, including a computer program, which implements the audio processing method in the above technical solution when executed by a processor.

[0043] In the technical solution provided in the embodiment of the present application, frequency domain features of multiple audio frames are obtained by extracting features from the audio signal, and then the frequency domain features of the audio frames are serialized and modeled to obtain intra-frame sequence features. The intra-frame sequence features of multiple audio frames are serialized and modeled to obtain inter-frame sequence features, and then the audio signal is modulated according to the inter-frame sequence features to obtain a speech signal after the noise signal is removed. The embodiment of the present application serializes and models the audio frame features based on the two dimensions of intra-frame and inter-frame, which can take into account both short-term and long-term temporal information, and comprehensively analyze the full-band information in the frequency domain space, thereby improving the speech enhancement and noise reduction effects in the audio signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0045] Figure 1 A system architecture block diagram using the technical solution of this application is shown.

[0046] Figure 2 A flowchart of an audio processing method in one embodiment of the present application is shown.

[0047] Figure 3 A flowchart of serialized modeling of frequency domain features of audio frames in one embodiment of the present application is shown.

[0048] Figure 4 A flowchart of mapping frequency domain features of an audio frame in one embodiment of the present application is shown.

[0049] Figure 5 A flowchart of an embodiment of the present application for sequentially extracting intra-frame sequence features and inter-frame sequence features from frequency domain features of audio data in an application scenario is shown.

[0050] Figure 6 The encoder structure and the corresponding decoder structure used in an application scenario of an embodiment of the present application are shown.

[0051] Figure 7 The figure shows the model structure of the audio processing model used in an application scenario of an embodiment of the present application and a schematic diagram of the process of audio processing based on the audio processing model.

[0052] Figure 8 FIG. 4 is a schematic diagram showing a comparison of scores of a speech perceptual quality parameter PESQ.

[0053] Figure 9 A schematic diagram of score comparison of the scale invariance signal-to-noise ratio parameter SISNR is shown.

[0054] Figure 10 FIG. 4 is a schematic diagram showing a score comparison of the simulated subjective audio quality perception parameter MOSOVL.

[0055] Figure 11 The structural block diagram of the audio processing device provided in an embodiment of the present application is schematically shown.

[0056] Figure 12 The following schematically shows a block diagram of a computer system structure of an electronic device suitable for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0057] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.

[0058] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner.In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps etc. can be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0059] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0060] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0061] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0062] In the specific implementation of this application, related data such as user voice is involved. When the various embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0063] Figure 1 A system architecture block diagram using the technical solution of this application is shown.

[0064] like Figure 1 As shown, the system architecture applying the technical solution of the present application may include a terminal device 110 and a server 130. The terminal device 110 may include various electronic devices such as smart phones, tablet computers, laptops, desktop computers, smart speakers, smart wearable devices, smart car devices, smart payment terminals, etc. The server 130 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Between the terminal device 110 and the server 130, various connection types of communication media for providing communication links may be included, such as a wired communication link or a wireless communication link.

[0065] Audio processing model 120 is a machine learning model used to perform audio noise reduction tasks. For example, when performing voice recognition on a user's voice commands, it can perform noise reduction processing on the collected audio data to obtain noise-free voice data. For another example, in video calls or video conferencing scenarios, audio processing model 120 can be used to perform noise reduction on the audio portion of communication data, improving the audio communication quality between the communicating parties.

[0066] In one application scenario of an embodiment of the present application, the audio processing model 120 can be pre-deployed on the server 130, and the server 130 can train the audio processing model 120. During the model training process, the loss error can be determined based on the processing results of the audio processing model 120 on the training samples, and then the model parameters of the audio processing model 120 can be iteratively updated based on the loss error. Through continuous training, the loss error of the model can be gradually reduced, thereby improving the recognition accuracy of the model.

[0067] After the audio processing model 120 is trained, it can provide audio noise reduction services to the terminal device 110. For example, the terminal device 110 can upload audio data to the server 130. The audio processing model 120 deployed on the server 130 performs noise reduction on the audio data and outputs noise-removed speech data. The server 130 then returns the speech data to the terminal device 110, which then presents the noise-reduced speech data to the user or implements other scenario requirements based on the audio noise reduction results.

[0068] In other application scenarios, the trained image recognition model 120 can also be deployed directly on the terminal device 110, allowing the terminal device 110 to run the image recognition model locally. When image recognition is required, the terminal device 110 can input the image to be recognized into the trained image recognition model 120, and the image recognition model 120 will recognize the image to be recognized and output the recognition result.

[0069] The audio processing model provided in the embodiments of this application can be applied to a variety of different online business scenarios, including cloud technology, artificial intelligence, smart transportation, assisted driving, and other scenarios. For example, the voice call function involved in social software or instant messaging software, the recognition of user voice commands in autonomous driving or assisted driving services, etc.

[0070] In the related technologies of this application, pure speech can be separated from audio signals through deep neural networks, thereby achieving the effect of speech enhancement and noise reduction. Speech enhancement and noise reduction methods based on deep learning can be mainly divided into four categories: time-frequency masking (MASK)-based methods, feature mapping (MAP)-based methods, signal approximation (SA)-based methods, and end-to-end (End2End)-based methods.

[0071] Due to the limitations of convolution kernel size, neural network models based on the aforementioned deep learning methods are typically only able to analyze and extract local features in the frequency domain, lacking global information. Low-frequency harmonics in the frequency domain are more energy-intensive, hiding noise between them and making their features less distinct. High-frequency consonants, on the other hand, have weaker energy distribution, resulting in relatively high noise intensity and a distinct noise signature. Neural network models fail to comprehensively analyze information across the entire frequency band, thus failing to comprehensively and meticulously analyze both speech and noise signal characteristics. Inadequate analysis of high-frequency speech features can easily lead to over-suppression of noise and speech damage, resulting in muffled or even stuttering speech. Inadequate analysis of low-frequency noise features can easily lead to residual noise, significantly compromising noise reduction effectiveness.

[0072] In response to the problems existing in the above-mentioned related technologies, the embodiments of the present application provide a speech enhancement and noise reduction solution that can simultaneously perform intra-frame and inter-frame sequence modeling on speech signals, perform feature extraction and analysis in the single-frame frequency dimension and single-frequency time dimension of the time-spectrum of noisy speech, and comprehensively and meticulously utilize the time-spectrum sequence features to realize the extraction of speech signals, and finally reconstruct the speech signal of pure human voice from the mixed audio signal interfered by noise.

[0073] Compared with conventional technical solutions, the audio processing method provided by the embodiment of the present application has better noise suppression and voice protection capabilities, and can be better applied to various scenarios. There is less residual noise between the harmonics in the mid- and low-frequency bands, and the high-frequency human voice components are better preserved, which makes the voice sound more natural, full, and smooth. The embodiment of the present application can be widely used in scenarios such as call noise reduction, video conferencing, smart hardware devices, voice recognition front-ends, live broadcast on-demand applications, etc. By processing noisy speech, suppressing the energy of noise components, improving the intelligibility and naturalness of speech, and ultimately improving the user's comprehensive audio listening experience.

[0074] The following describes in detail the technical solutions provided by the present application, including the audio processing method, audio processing device, computer-readable medium, electronic device, and computer program product, in conjunction with specific implementation methods.

[0075] Figure 2 A flowchart of an audio processing method in an embodiment of the present application is shown. The audio processing method can be performed by Figure 1The terminal device or server shown in the figure can also execute the method together. The embodiment of the present application takes the audio processing method executed by the terminal device as an example to illustrate. Figure 1 As shown, the audio processing method may include the following steps S210 to S240.

[0076] S210: Extract features from the audio signal to obtain frequency domain features of multiple audio frames; the audio signal carries a speech signal and a noise signal.

[0077] S220: Serializing and modeling the frequency domain features of the audio frame to obtain intra-frame sequence features, where the intra-frame sequence features are used to represent a frequency sequence relationship between multiple frequency points in a single audio frame.

[0078] S230: performing serial modeling on intra-frame sequence features of the plurality of audio frames to obtain inter-frame sequence features, where the inter-frame sequence features are used to represent a temporal sequence relationship between the plurality of audio frames.

[0079] S240: performing signal modulation on the audio signal according to the inter-frame sequence feature to obtain a speech signal after removing the noise signal.

[0080] The embodiment of the present application serializes and models the audio frame features based on the intra-frame and inter-frame dimensions, can take into account both short-term timing information and long-term timing information, and perform a comprehensive analysis of the full-band information in the frequency domain space, thereby improving the speech enhancement and noise reduction effects in the audio signal.

[0081] The following describes in detail how to implement each step of the audio processing method in the embodiment of the present application in a specific application scenario.

[0082] In step S210 , feature extraction is performed on the audio signal to obtain frequency domain features of multiple audio frames.

[0083] In one embodiment of the present application, a method for extracting features from an audio signal may include: performing frame and window processing on the audio signal to obtain time domain spectra of multiple audio frames; performing discrete cosine transform on the time domain spectra of the audio frames to obtain frequency domain spectra of the audio frames; and encoding and compressing the frequency domain spectra of the audio frames in sequence through multiple serially connected encoders to obtain frequency domain features of the audio frames.

[0084] First, the audio signal is framed, which can be divided into short audio segments of fixed length, i.e., audio frames. For example, in this embodiment, the original audio signal can be divided into multiple short frames of fixed length using a single frame length of 1024 and a frame shift of 512. The overlap length of two adjacent frames is 512, and each frame signal is modulated using a Hamming window to prevent spectrum leakage.

[0085] Even for periodic signals, if the truncation time is not an integer multiple of the period (period truncation), the truncated signal will exhibit leakage. To minimize this leakage error, a weighting function, also known as a window function, is used. Windowing is primarily used to ensure that the time-domain signal better meets the periodic requirements of time-frequency transformation processing and reduce leakage. Since directly truncating the signal with a rectangular window will produce frequency leakage, a non-rectangular window, typically a Hamming window, is used to mitigate this. The amplitude-frequency characteristic of a Hamming window results in significant sidelobe attenuation, with the mainlobe peak and the first sidelobe peak attenuated by up to 40dB.

[0086] After the frame and window operation is completed, the modulated signal is subjected to discrete cosine transform (DCT) operation to extract the frequency domain features and obtain the noisy speech signal x. n Frequency domain representation of X k The combination of audio signal framing and windowing with cosine transform operation is also called short-time discrete cosine transform (SDCT).

[0087] Discrete cosine transform is used to convert signals or data from the time domain to the frequency domain. Through a series of mathematical operations, it converts discrete signal data into the coefficients of the cosine function to better analyze the frequency characteristics of the signal.

[0088] In some optional implementations, the frequency domain features of the audio frame may be features obtained through short-time Fourier transform, or may be various acoustic features such as amplitude spectrum, power spectrum, and Mel spectrum.

[0089] In step S220 , the frequency domain features of the audio frame are serialized and modeled to obtain intra-frame sequence features. The intra-frame sequence features are used to represent the frequency sequence relationship between multiple frequency points in a single audio frame.

[0090] Figure 3 FIG. 1 shows a flow chart of serializing and modeling the frequency domain features of an audio frame in one embodiment of the present application. Figure 3 As shown, based on the above embodiment, step S220 may further include the following steps S310 to S340.

[0091] S310: Sort the frequency domain features of the audio frames according to a specified frequency order to form a frequency domain feature sequence.

[0092] The specified frequency order can be the order from low to high frequency, or the order from high to low frequency. For example, an audio frame includes several frequency points, and after sorting them according to the specified frequency order, the frequency domain feature sequence can be {x1, x2...x n-1 、xn 、x n+1}. Among them, x n Indicates the frequency domain characteristics corresponding to the nth frequency point.

[0093] S320: Mapping the frequency domain features of the audio frame in the frequency domain feature sequence from front to back to obtain forward sequence features of each frequency point; the forward sequence features of the current frequency point have a mapping relationship with the forward sequence features of the previous frequency point.

[0094] The frequency domain features of each frequency point in the frequency domain feature sequence and the sequence features of the previous frequency point are used as input data. After forward mapping, the forward sequence features of the current frequency point can be obtained. In other words, the forward sequence features of the current frequency point have a mapping relationship with the forward sequence features of the previous frequency point.

[0095] For example, after forward mapping the frequency domain features of each frequency point in the frequency domain feature sequence, the forward sequence features obtained can be expressed as {y1, y2...y n-1 、y n 、y n+1}. Among them, y n =F1(x n ,y n-1 ), represents the forward sequence feature of the nth frequency point. F1() is a preset forward mapping function, the parameters of which can be obtained through neural network model fitting training.

[0096] S330: Mapping the frequency domain features of the audio frame in the frequency domain feature sequence from back to front to obtain reverse sequence features of each frequency point; the reverse sequence feature of the current frequency point has a mapping relationship with the reverse sequence feature of the next frequency point.

[0097] The frequency domain features of each frequency point in the frequency domain feature sequence and the sequence features of the next frequency point are used as input data. After forward mapping, the reverse sequence features of the current frequency point can be obtained. In other words, the reverse sequence features of the current frequency point have a mapping relationship with the reverse sequence features of the next frequency point.

[0098] For example, after the frequency domain features of each frequency point in the frequency domain feature sequence are reverse mapped, the obtained reverse sequence features can be expressed as {z1, z2...z n-1 、z n 、z n+1}. Among them, z n =F2(x n , z n+1 ), represents the reverse sequence feature of the nth frequency point. F2() is a preset reverse mapping function, the parameters of which can be obtained through neural network model fitting training.

[0099] S340: Determine the intra-frame sequence feature of the current frequency point according to the forward sequence feature and the reverse sequence feature of the current frequency point.

[0100] In one embodiment of the present application, the forward sequence feature and the reverse sequence feature of the current frequency point can be added together to obtain the intra-frame sequence feature of the current frequency point. For example, the forward sequence feature of the nth frequency point is y n , the reverse sequence characteristic of the nth frequency point is z n , then the corresponding intra-frame sequence feature of the nth frequency point can be expressed as y n +z n .

[0101] In some other optional implementations, the forward sequence features and the reverse sequence features of the current frequency point may be weighted according to a specified weighting coefficient to obtain the intra-frame sequence features of the current frequency point. The specified weighting coefficient may be obtained through neural network model fitting training.

[0102] The embodiment of the present application adopts a method that combines forward mapping and reverse mapping to map the frequency domain features of each frequency point in turn from two different directions, and can find the intrinsic connection between frequency points from low frequency to high frequency and from high frequency to low frequency respectively. And because the current frequency point has a mapping relationship with the adjacent previous frequency point or the next frequency point, it can be ensured that each frequency point has a greater mutual influence with the surrounding closer frequency points, and has less influence on the farther frequency points. In this way, full-band information sharing can be achieved within a single frame signal, high-frequency voice information can be better retained according to the low-frequency harmonic characteristics, and low-frequency noise information can be more completely characterized and suppressed according to the high-frequency noise characteristics, thereby comprehensively improving the listening quality of the enhanced voice.

[0103] Figure 4 FIG. 1 shows a flow chart of mapping the frequency domain features of an audio frame in one embodiment of the present application. Figure 4 As shown, based on the above embodiment, the method of performing mapping processing on the frequency domain features of the audio frame in step S320 or step S330 may include the following steps S410 to S450.

[0104] S410: Concatenate the sequence feature of the reference frequency point with the frequency domain feature of the current frequency point to obtain first input data, where the reference frequency point is a frequency point before or after the current frequency point.

[0105] When forward mapping is performed in the frequency domain feature sequence from front to back, the reference frequency for the current frequency point n can be the previous frequency point n-1. When reverse mapping is performed in the frequency domain feature sequence from back to front, the reference frequency for the current frequency point n can be the next frequency point n+1.

[0106] Taking forward mapping as an example, the previous frequency point is used as the reference frequency point, and the sequence feature obtained after the previous frequency point is mapped (i.e., the forward sequence feature) can be expressed as h n-1 . The sequence feature h of the previous frequency point n-1 and the frequency domain characteristics x of the current frequency point n After the splicing process, the first input data is obtained as (h n-1 , x n ).

[0107] Taking reverse mapping as an example, the next frequency point is used as the reference frequency point, and the sequence feature (i.e., the backward sequence feature) obtained after the mapping process of the next frequency point can be expressed as h n+1 . The sequence feature h of the next frequency point n+1 and the frequency domain characteristics x of the current frequency point n After the splicing process, the first input data is obtained as (h n+1 , x n ).

[0108] For the sake of convenience, the following description uses the previous frequency point as the reference frequency point for forward mapping as an example.

[0109] S420: Mapping the first input data to obtain reset weights and update weights corresponding to the current frequency point. The reset weights are used to reset the influence of the sequence characteristics of the reference frequency point on the input data, and the update weights are used to update the sequence characteristics of the current frequency point.

[0110] In one embodiment of the present application, a pre-trained sigmoid function may be used to map the first input data to obtain reset weights and update weights corresponding to the current frequency point, respectively. The sigmoid function may transform the data into a value within a range of 0-1.

[0111] For example, the reset weight r of the current frequency point can be expressed as r=σ1(h n-1 , x n ), the updated weight u of the current frequency point can be expressed as u=σ2(h n-1 , x n ). Among them, the sigmoid function used for mapping to obtain the reset weight and the sigmoid function used for mapping to obtain the update weight can both be obtained through neural network model fitting training.

[0112] S430: Performing a weighted operation on the sequence features of the reference frequency points according to the reset weights and then performing a splicing process on the sequence features of the reference frequency points and the frequency domain features of the current frequency points to obtain second input data.

[0113] According to the reset weight, the input data can be reset by weighted operation, that is, the influence of the sequence characteristics of the reference frequency point on the input data can be reset in the input data. For example, if the reset weight is r, the second input data can be expressed as (r·h n-1 , x n ).

[0114] S440: Perform mapping processing on the second input data to obtain an initial sequence feature of the current frequency point.

[0115] In one embodiment of the present application, a pre-trained tanh activation function can be used to map the second data to obtain the initial sequence feature of the current frequency point. For example, the initial sequence feature of the current frequency point can be expressed as h'=tanh(r·h n-1 , x n ). Among them, the tanh activation function can be obtained through neural network model fitting training.

[0116] S450: Performing a weighted operation on the sequence feature of the reference frequency point and the initial sequence feature of the current frequency point according to the updated weight to obtain an updated sequence feature of the current frequency point.

[0117] According to the updated weight, the sequence characteristics of the current frequency point can be updated through weighted operation, that is, the influence of the sequence characteristics of the reference frequency point on the sequence characteristics of the current frequency point can be adjusted by updating the weight. For example, if the updated weight is u, the updated sequence characteristics of the current frequency point can be expressed as h n =(1-u)·h n-1 +u·h'.

[0118] By calculating reset weights and update weights, the embodiments of the present application can adjust the influence of different parameters during the feature mapping process. The reset weights are used to adjust the influence of the sequence characteristics of the reference frequency point on the input data, while the update weights are used to adjust the influence of the sequence characteristics of the reference frequency point on the output data (i.e., the sequence characteristics of the current frequency point). Using reset weights and update weights can more comprehensively explore the inherent connection between the current frequency point and the reference frequency point (i.e., the previous or next frequency point), improving the speech enhancement and denoising effects of audio processing.

[0119] In step S230 , intra-frame sequence features of the multiple audio frames are serialized and modeled to obtain inter-frame sequence features, where the inter-frame sequence features are used to represent a temporal sequence relationship between the multiple audio frames.

[0120] In one embodiment of the present application, a method for serializing and modeling intra-frame sequence features of multiple audio frames to obtain inter-frame sequence features may include: performing time-frequency transformation on the intra-frame sequence features of multiple audio frames to obtain time domain distribution information of the intra-frame sequence features at multiple frequency points; and serializing and modeling the time domain distribution information at each frequency point to obtain inter-frame sequence features.

[0121] In the frequency domain space of the intra-frame sequence feature distribution obtained after serializing and modeling the frequency domain features of the audio frame in step S220, in order to further explore the inter-frame sequence features, the embodiment of the present application can first perform time-frequency transformation on the intra-frame sequence features to obtain the time domain distribution information of the intra-frame sequence features at multiple frequency points, and then serialize and model the time domain distribution information at each frequency point.

[0122] In one embodiment of the present application, a method for serializing and modeling the time domain distribution information at each frequency point to obtain inter-frame sequence features may include: mapping the time domain distribution information at each moment on the frequency point in sequence according to a specified time sequence to obtain inter-frame sequence features; the inter-frame sequence features at the current moment have a mapping relationship with the inter-frame sequence features at the reference moment, and the reference moment is the moment before or after the current moment.

[0123] For example, when the time domain distribution information is forward mapped in a time sequence from front to back, the reference moment can be the moment before the current moment, that is, the inter-frame sequence characteristics of the previous moment and the time domain distribution information of the current moment are mapped to obtain the inter-frame sequence characteristics of the current moment.

[0124] For another example, when the time domain distribution information is reversely mapped in chronological order from back to front, the reference moment can be the moment after the current moment, that is, the inter-frame sequence characteristics of the latter moment and the time domain distribution information of the current moment are mapped to obtain the inter-frame sequence characteristics of the current moment.

[0125] In some optional implementations, the forward sequence features and reverse sequence features at the current moment may be obtained by forward mapping and reverse mapping respectively, and then the forward sequence features and reverse sequence features may be summed or weighted to obtain the inter-frame sequence features at the current moment.

[0126] Figure 5 The flowchart of an embodiment of the present application shows how to extract intra-frame sequence features and inter-frame sequence features from the frequency domain features of audio data in an application scenario. Intra-frame sequence feature extraction corresponds to the frequency sequence modeling process, and inter-frame sequence feature extraction corresponds to the time sequence modeling process.

[0127] like Figure 5 As shown, after performing frame processing and time-frequency conversion on the audio data, frequency domain features 501 of multiple audio frames can be obtained. Among them, a data channel Channel represents an audio frame corresponding to a moment.

[0128] The frequency domain features of each frequency point in an audio frame are subjected to bidirectional frequency serialization modeling in a forward mapping and reverse mapping manner, and then superimposed to obtain the intra-frame sequence features 502 of each frequency point.

[0129] A time-frequency conversion is performed on the intra-frame sequence features 502 corresponding to each audio frame, that is, the data is converted from the frequency domain space to the time domain space, and the time domain distribution information 503 on each frequency point (FrequencyBin) is obtained.

[0130] After performing one-way time series modeling on the time domain distribution information of each moment within a frequency point, the inter-frame sequence features 504 of each moment are obtained.

[0131] In one embodiment of the present application, frequency series modeling can be performed by a bidirectional gated recurrent unit (BiGRU). BiGRU is an improved recurrent neural network (RNN) structure that consists of two independent gated recurrent unit (GRU) layers, one processing the sequence in the forward direction and the other processing the sequence in the reverse direction. This bidirectional structure enables BiGRU to capture long-range dependencies in the sequence, thereby improving the performance of the model. Time series modeling can be performed by a unidirectional gated recurrent unit (GRU).

[0132] In step S240, the audio signal is modulated according to the inter-frame sequence feature to obtain a speech signal after the noise signal is removed.

[0133] In one embodiment of the present application, performing signal modulation on an audio signal according to inter-frame sequence features to obtain a speech signal after removing the noise signal may further include: decoding and restoring the inter-frame sequence features to obtain mask information for removing the noise signal; performing signal modulation on the audio signal according to the mask information to obtain a speech signal after removing the noise signal.

[0134] By decoding and restoring the inter-frame sequence features to obtain mask information, the audio signal can be modulated based on the mask information to remove the noise contained in it, resulting in a noise-removed speech signal. For example, the mask information can be used to preserve the signal strength of the speech signal while reducing the signal strength of the noise signal, thereby achieving the goal of noise reduction and speech enhancement.

[0135] In one embodiment of the present application, the method of performing signal modulation on an audio signal according to mask information may include: performing signal modulation on the frequency domain spectrum of the audio signal according to the mask information to obtain the frequency domain spectrum of the speech signal after noise is removed; and performing an inverse discrete cosine transform on the frequency domain spectrum of the speech signal to obtain the time domain spectrum of the speech signal.

[0136] In order to perform serial modeling on the audio signal to extract intra-frame sequence features and inter-frame sequence features respectively, the audio signal can first be converted into a time-frequency domain, for example, the audio signal can be converted from the time domain space to the frequency domain space by discrete cosine transform to obtain the frequency domain spectrum of the corresponding audio signal. On this basis, after obtaining the mask information, the embodiment of the present application performs signal modulation on the frequency domain spectrum of the audio signal by the mask information to obtain the frequency domain spectrum of the speech signal after removing the noise; and then performs a time-frequency conversion on the frequency domain spectrum of the speech signal, for example, the speech signal can be converted from the frequency domain space to the time domain space by inverse discrete cosine transform to obtain the time domain spectrum of the speech signal.

[0137] In one embodiment of the present application, the frequency domain features of audio frames are extracted using a pre-trained encoder model using encoding compression. Based on this, after obtaining inter-frame sequence features, the embodiment of the present application can obtain mask information using a decoder model trained together with the encoder model using decoding and restoration. The encoder and decoder models can have symmetrical network model structures.

[0138] In one embodiment of the present application, decoding and restoring inter-frame sequence features to obtain mask information for removing noise signals may further include: sequentially decoding and restoring inter-frame sequence features through multiple serially connected decoders to obtain mask information for removing noise signals; wherein the input data of the decoder includes the output data of the previous decoder and the output data of an encoder having the same depth as the decoder, and the encoder is used to encode and compress the audio signal.

[0139] For example, the encoder model includes multiple encoders connected in series, and the decoder model includes the same number of decoders connected in series. The output data of the first encoder is not only passed to the second encoder as input data for the second encoder, but is also passed to the first decoder as input data for the first decoder. Similarly, the output data of the mth encoder is not only passed to the m+1th encoder as input data for the m+1th encoder, but is also passed to the mth decoder as input data for the mth decoder.

[0140] The embodiment of the present application can train the corresponding decoder model to learn the residual data of input data and output data by transferring data between the encoder and decoder of the same depth, thereby reducing the amount of data calculation and improving the training efficiency of the model.

[0141] Figure 6 The encoder structure and the corresponding decoder structure used in an application scenario of an embodiment of the present application are shown.

[0142] like Figure 6 As shown in the figure, the decoder includes a transposed convolutional layer, a batch normalization layer, and an activation layer connected in sequence, while the encoder includes a convolutional layer, a batch normalization layer, and an activation layer connected in sequence. The transposed convolutional layer is used to perform transposed convolution on the input data to obtain data with increased dimensionality, the convolutional layer is used to perform convolution on the input data to obtain data with reduced dimensionality, the batch normalization layer is used to normalize the input data, and the activation layer is used to introduce nonlinear features to the input data.

[0143] By configuring batch normalization layers and activation layers in each encoder and each decoder, the embodiment of the present application can perform feature normalization and introduce nonlinear features in the feature learning process at different depths, thereby improving the end-to-end model's ability to learn audio features and enhancing the accuracy and reliability of audio signal processing.

[0144] In one embodiment of the present application, the convolution step size of the convolution layer in the frequency domain space is greater than one, the convolution step size of the convolution layer in the time domain space is equal to one, and the transposed convolution layer and the convolution layer have the same convolution step size.

[0145] By configuring the convolutional layer in the frequency domain with a convolution step size greater than one, the present embodiment can fully utilize the data compression effect between high-dimensional and low-dimensional data in the frequency domain during feature encoding and compression and decoding, thereby reducing the amount of data computation required for feature processing. At the same time, the number of audio frames in the time domain can be maintained unchanged, fully exploiting global time domain information and improving the extraction of temporal information from both intra-frame and inter-frame sequence features.

[0146] Figure 7 The figure shows the model structure of the audio processing model used in an application scenario of an embodiment of the present application and a schematic diagram of the process of audio processing based on the audio processing model.

[0147] like Figure 7 As shown, the audio processing model in the embodiment of the present application is mainly divided into three functional units, namely, an audio signal pre-processing and feature extraction unit, a neural network model inference unit and a post-processing speech generation unit.

[0148] The pre-processing and feature extraction unit first processes the noisy speech signal x nResampling is performed to resample all audio data of different sampling rates to 48kHz. After the resampling operation is completed, the long audio signal is subjected to time domain frame division and windowing. According to the single frame length of 1024 and the frame shift of 512 (overlap 512), the original audio signal is divided into multiple frames of short signals with fixed lengths, and each frame signal is modulated using a Hamming window to prevent spectrum leakage. After the frame division and windowing operation is completed, the modulated signal is subjected to a discrete cosine transform (DCT) operation to extract the frequency domain features and obtain the noisy speech signal x. n Frequency domain representation of X k The combination of audio signal framing and windowing with cosine transform operation is also called short-time discrete cosine transform (SDCT).

[0149] In the deep learning network inference unit, the embodiment of the present application adopts the Encoder-Decoder framework. The Encoder part is mainly composed of the EncConv2d structure with two-dimensional convolution (Conv2d) as the kernel. The kernel size of the convolution kernel of each layer of EncConv2d is (5, 2), which means that the frequency domain field of view is 5 and the time domain field of view is 2. The analysis and processing of each frame signal feature will refer to the previous frame signal. The convolution stride is (2, 1), which can reduce the number of signal frequency domain features by half layer by layer, while the number of time domain frames remains unchanged, which plays a role in dimensionality reduction and reducing the amount of calculation. The Decoder part is mainly composed of DecTConv2d with transposed two-dimensional convolution (ConvTranspose2d) as the kernel. The parameters of DecTConv2d of each layer are the same as those of the corresponding EncConv2d, which realizes the restoration of signal dimension.

[0150] Between the Encoder and the Decoder, the embodiment of the present application designs a dual-path recurrent neural network module IIGRUs composed of a stack of intra-frame inter-frame sequence structures (Inter-frame Intra-frame Gated Recurrent Units, IIGRU) modules. The network structure can be referred to Figure 5 The IIGRU can be considered to consist of two cascaded recurrent neural network modules. The first RNN is used to serialize and model frequency domain features, using a bidirectional GRU network to extract features between frequency points and analyze their correlation. The second RNN is used to analyze the timing information of the audio signal, used to analyze the time domain characteristics of speech and noise.

[0151] The workflow of the deep learning network inference unit is as follows: the Encoder receives the short-time cosine transform representation of the noisy speech obtained after signal preprocessing, and then extracts high-dimensional features layer by layer through EncConv2d. The corresponding output is given to DecTConv2d through a skip connection. IIGRUs receives the output features from its last layer of EncConv2d, extracts and analyzes the frequency domain and time domain sequence information, and gives the input to the Decoder. The Decoder receives the output from IIGRUs and the Encoder, and after layer-by-layer dimensionality increase processing, finally obtains the cosine transform mask.

[0152] After obtaining the short-time cosine transform mask of the speech signal, the post-processing speech generation unit modulates the short-time cosine spectrum of the original noisy speech to obtain the short-time cosine spectrum estimate of the speech signal, which is expressed as follows:

[0153]

[0154] After obtaining the short-time cosine spectrum of the speech signal, the inverse short-time discrete transform (iSDCT) corresponding to the SDCT is performed to obtain the time domain estimate of the enhanced speech signal.

[0155] The embodiment of the present application can train the above-mentioned audio processing model based on a deep learning speech enhancement and noise reduction scheme, generate a noisy speech signal by mixing a clean speech data set and a noise data set, and simulate the noisy speech signal-to-noise ratio in different noise environments by controlling the noise mixing ratio.

[0156] Assume that the pure speech signal is s n , the noise signal is d n , the corresponding short-time cosine transforms are S k and D k , then the noisy speech signal can be expressed as:

[0157] x n =s n +d n

[0158] The corresponding short-time cosine feature can be expressed as:

[0159] X k =S k +D k

[0160] The ideal mask output by the network model can be expressed as:

[0161]

[0162] The loss function can be expressed as: The loss function may be, for example, MSE, MAE, SI-SNR or other functions.

[0163] In order to verify the speech noise separation effect of the audio processing model provided in the embodiment of the present application, a batch of test data was generated according to the signal-to-noise ratio range of [-10,30]dB, with a step of 2dB, totaling 1000 sets of test data. Multiple evaluation indicators were used to compare the audio processing model provided in the embodiment of the present application with other models in the related technology.

[0164] Figure 8 The figure shows a comparison diagram of the score of the perceptual evaluation of speech quality (PESQ). Figure 9 A schematic diagram of the score comparison of the scale-invariant signal-to-noise ratio (SISNR) parameter is shown. Figure 10 The figure shows a comparison diagram of the scores of the simulated subjective audio quality perception parameter (Mean Opinion Score-Overall, MOSOVL).

[0165] The horizontal axis in each score comparison diagram represents the signal-to-noise ratio, and the vertical axis represents the indicator score corresponding to each signal-to-noise ratio. Three sets of data are provided for each signal-to-noise ratio: from left to right, the noisy speech data, the data after noise reduction processing using the DCTCRN model in the related art, and the data after noise reduction processing using the audio processing model provided in the embodiments of this application.

[0166] Based on the comparison of the above indicator scores, it can be seen that the solution provided in the embodiment of the present application can effectively suppress noise and improve voice quality.

[0167] The embodiment of the present application provides a technical solution design for speech enhancement and noise reduction based on intra-frame and inter-frame sequence modeling, which improves the frequency domain feature extraction and analysis capabilities of the algorithm model, thereby enabling the algorithm to have better low- and medium-frequency noise suppression capabilities and high-frequency speech protection capabilities. Compared with other conventional technical solutions, the embodiment of the present application can better capture the frequency domain characteristics of speech, especially in the frequency bands where the harmonic energy of low- and medium-frequency speech is strong and noise residues are easily retained. The embodiment of the present application can better suppress noise. In the high-frequency part, the embodiment of the present application can also better protect speech due to the problem of excessive suppression and speech damage caused by weak speech characteristics.

[0168] It should be noted that although the steps of the method of the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all steps must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0169] The following introduces an embodiment of the device of the present application, which can be used to execute the audio processing method in the above embodiment of the present application. Figure 11 The structural block diagram of the audio processing device provided by the embodiment of the present application is schematically shown. Figure 11 As shown, the audio processing device 1100 includes:

[0170] The extraction module 1110 is configured to extract features from the audio signal to obtain frequency domain features of multiple audio frames; the audio signal carries a speech signal and a noise signal;

[0171] A first modeling module 1120 is configured to perform serial modeling on the frequency domain features of the audio frame to obtain an intra-frame sequence feature, where the intra-frame sequence feature is used to represent a frequency sequence relationship between multiple frequency points in a single audio frame;

[0172] The second modeling module 1130 is configured to perform serial modeling on the intra-frame sequence features of the plurality of audio frames to obtain inter-frame sequence features, where the inter-frame sequence features are used to represent a temporal sequence relationship between the plurality of audio frames;

[0173] The modulation module 1140 is configured to perform signal modulation on the audio signal according to the inter-frame sequence feature to obtain a speech signal after the noise signal is removed.

[0174] In some embodiments of the present application, based on the above technical solution, the modulation module 1140 includes:

[0175] a feature restoration module configured to decode and restore the inter-frame sequence features to obtain mask information for removing noise signals;

[0176] The audio modulation module is configured to perform signal modulation on the audio signal according to the mask information to obtain a speech signal after the noise signal is removed.

[0177] In some embodiments of the present application, based on the above technical solution, the audio modulation module includes:

[0178] a frequency domain spectrum modulation module, configured to perform signal modulation on the frequency domain spectrum of the audio signal according to the mask information to obtain a frequency domain spectrum of the speech signal after noise removal;

[0179] The frequency domain spectrum transformation module is configured to perform an inverse discrete cosine transform on the frequency domain spectrum of the speech signal to obtain a time domain spectrum of the speech signal.

[0180] In some embodiments of the present application, based on the above technical solution, the feature restoration module is further configured to: decode and restore the inter-frame sequence features in sequence through multiple serially connected decoders to obtain mask information for removing noise signals; wherein the input data of the decoder includes the output data of the previous decoder and the output data of an encoder with the same depth as the decoder, and the encoder is used to encode and compress the audio signal.

[0181] In some embodiments of the present application, based on the above technical solution, the decoder includes a transposed convolution layer, a batch normalization layer and an activation layer connected in sequence, and the encoder includes a convolution layer, a batch normalization layer and an activation layer connected in sequence; the transposed convolution layer is used to perform transposed convolution processing on the input data to obtain data with increased dimension, the convolution layer is used to perform convolution processing on the input data to obtain data with reduced dimension, the batch normalization layer is used to normalize the input data, and the activation layer is used to introduce nonlinear features to the input data.

[0182] In some embodiments of the present application, based on the above technical solution, the convolution step size of the convolution layer in the frequency domain space is greater than one, the convolution step size of the convolution layer in the time domain space is equal to one, and the transposed convolution layer has the same convolution step size as the convolution layer.

[0183] In some embodiments of the present application, based on the above technical solution, the extraction module 1110 is further configured to: perform frame and window processing on the audio signal to obtain time domain spectra of multiple audio frames; perform discrete cosine transform on the time domain spectrum of the audio frame to obtain frequency domain spectrum of the audio frame; and encode and compress the frequency domain spectrum of the audio frame in sequence through multiple serially connected encoders to obtain frequency domain features of the audio frame.

[0184] In some embodiments of the present application, based on the above technical solution, the first modeling module 1120 includes:

[0185] a sorting module, configured to sort the frequency domain features of the audio frames according to a specified frequency order to form a frequency domain feature sequence;

[0186] a forward mapping module configured to map the frequency domain features of the audio frame in the frequency domain feature sequence in a forward-to-backward order to obtain forward sequence features of each frequency point; the forward sequence features of the current frequency point have a mapping relationship with the forward sequence features of the previous frequency point;

[0187] a reverse mapping module configured to map the frequency domain features of the audio frame in the frequency domain feature sequence from back to front to obtain reverse sequence features of each frequency point; the reverse sequence feature of the current frequency point has a mapping relationship with the reverse sequence feature of the next frequency point;

[0188] The feature determination module is configured to determine the intra-frame sequence feature of the current frequency point according to the forward sequence feature and the reverse sequence feature of the current frequency point.

[0189] In some embodiments of the present application, based on the above technical solution, the forward mapping module or the reverse mapping module includes:

[0190] A first splicing module is configured to splice the sequence characteristics of the reference frequency point with the frequency domain characteristics of the current frequency point to obtain first input data, wherein the reference frequency point is the reference frequency point or the subsequent frequency point of the current frequency point;

[0191] a first mapping module configured to perform mapping processing on the first input data to obtain a reset weight and an update weight corresponding to the current frequency point, the reset weight being used to reset the influence of the sequence characteristics of the reference frequency point on the input data, and the update weight being used to update the sequence characteristics of the current frequency point;

[0192] A second splicing module is configured to perform a weighted operation on the sequence features of the reference frequency points according to the reset weights, and then splice the results with the frequency domain features of the current frequency points to obtain second input data;

[0193] A second mapping module is configured to perform mapping processing on the second input data to obtain the initial sequence feature of the current frequency point;

[0194] The feature weighting module is configured to perform a weighted operation on the sequence feature of the reference frequency point and the initial sequence feature of the current frequency point according to the updated weight to obtain the updated sequence feature of the current frequency point.

[0195] In some embodiments of the present application, based on the above technical solution, the second modeling module 1130 includes:

[0196] a time-frequency transformation module configured to perform time-frequency transformation on the intra-frame sequence features of the plurality of audio frames to obtain time domain distribution information of the intra-frame sequence features at a plurality of frequency points;

[0197] The time domain modeling module is configured to perform serial modeling on the time domain distribution information at each of the frequency points to obtain inter-frame sequence features.

[0198] In some embodiments of the present application, based on the above technical solution, the time domain modeling module is further configured to: map the time domain distribution information of each moment on the frequency point in sequence according to a specified time sequence to obtain inter-frame sequence features; the inter-frame sequence features of the current moment have a mapping relationship with the inter-frame sequence features of the reference moment, and the reference moment is the moment before or after the current moment.

[0199] The specific details of the audio processing device provided in each embodiment of the present application have been described in detail in the corresponding method embodiments and will not be repeated here.

[0200] Figure 12 The block diagram schematically shows a computer system structure of an electronic device used to implement an embodiment of the present application.

[0201] It should be noted that Figure 12 The computer system 1200 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0202] like Figure 12 As shown, the computer system 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1202 or the program loaded from the storage part 1208 into the random access memory (RAM) 1203. Various programs and data required for system operation are also stored in the random access memory 1203. The CPU 1201, the read-only memory 1202, and the random access memory 1203 are connected to each other via a bus 1204. An input / output interface 1205 (i.e., an I / O interface) is also connected to the bus 1204.

[0203] The following components are connected to the input / output interface 1205: an input section 1206 including a keyboard, a mouse, and the like; an output section 1207 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1208 including a hard disk; and a communication section 1209 including a network interface card such as a local area network card or a modem. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the input / output interface 1205 as needed. Removable media 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1210 as needed, so that computer programs read therefrom can be installed into the storage section 1208 as needed.

[0204] In particular, according to an embodiment of the present application, the processes described in the various method flow charts can be implemented as computer software programs. For example, an embodiment of the present application includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods shown in the flow charts. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1209 and / or installed from a removable medium 1211. When the computer program is executed by the central processing unit 1201, the various functions defined in the system of the present application are performed.

[0205] It should be noted that the computer-readable medium shown in the embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0206] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0207] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0208] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0209] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed herein.

[0210] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An audio processing method, characterized in that: include: Extract features from the audio signal to obtain frequency domain features of multiple audio frames; The audio signal carries a speech signal and a noise signal; Performing serial modeling on the frequency domain features of the audio frame to obtain intra-frame sequence features, where the intra-frame sequence features are used to represent a frequency sequence relationship between multiple frequency points in a single audio frame; Performing serial modeling on intra-frame sequence features of the plurality of audio frames to obtain inter-frame sequence features, where the inter-frame sequence features are used to represent a temporal sequence relationship between the plurality of audio frames; The audio signal is modulated according to the inter-frame sequence feature to obtain a speech signal after the noise signal is removed.

2. The audio processing method according to claim 1, wherein: Performing signal modulation on the audio signal according to the inter-frame sequence feature to obtain a speech signal after removing the noise signal includes: Decoding and restoring the inter-frame sequence features to obtain mask information for removing noise signals; The audio signal is modulated according to the mask information to obtain a speech signal after the noise signal is removed.

3. The audio processing method according to claim 2, characterized in that Performing signal modulation on the audio signal according to the mask information to obtain a speech signal after removing the noise signal includes: Performing signal modulation on the frequency domain spectrum of the audio signal according to the mask information to obtain the frequency domain spectrum of the speech signal after noise removal; Performing an inverse discrete cosine transform on the frequency domain spectrum of the speech signal to obtain a time domain spectrum of the speech signal.

4. The audio processing method according to claim 2, wherein: Decoding and restoring the inter-frame sequence features to obtain mask information for removing noise signals includes: The inter-frame sequence features are decoded and restored in sequence by multiple serially connected decoders to obtain mask information for removing noise signals; wherein the input data of the decoder includes the output data of the previous decoder and the output data of an encoder with the same depth as the decoder, and the encoder is used to encode and compress the audio signal.

5. The audio processing method according to claim 4, characterized in that: The decoder includes a transposed convolution layer, a batch normalization layer, and an activation layer connected in sequence, and the encoder includes a convolution layer, a batch normalization layer, and an activation layer connected in sequence; the transposed convolution layer is used to perform transposed convolution processing on the input data to obtain data with increased dimension, the convolution layer is used to perform convolution processing on the input data to obtain data with reduced dimension, the batch normalization layer is used to normalize the input data, and the activation layer is used to introduce nonlinear features into the input data.

6. The audio processing method according to claim 5, characterized in that: The convolution step size of the convolution layer in the frequency domain space is greater than one, the convolution step size of the convolution layer in the time domain space is equal to one, and the transposed convolution layer and the convolution layer have the same convolution step size.

7. The audio processing method according to claim 1, characterized in that: Perform feature extraction on the audio signal to obtain frequency domain features of multiple audio frames, including: Performing frame and window processing on the audio signal to obtain time domain spectra of multiple audio frames; Performing discrete cosine transform on the time domain spectrum of the audio frame to obtain a frequency domain spectrum of the audio frame; The frequency domain spectrum of the audio frame is encoded and compressed in sequence by multiple serially connected encoders to obtain the frequency domain features of the audio frame.

8. The audio processing method according to any one of claims 1 to 7, characterized in that: The frequency domain features of the audio frame are serialized and modeled to obtain intra-frame sequence features, including: Sorting the frequency domain features of the audio frames according to a specified frequency order to form a frequency domain feature sequence; Mapping the frequency domain features of the audio frame in the frequency domain feature sequence from front to back to obtain forward sequence features of each frequency point; the forward sequence features of the current frequency point have a mapping relationship with the forward sequence features of the previous frequency point; Mapping the frequency domain features of the audio frame in the frequency domain feature sequence from back to front to obtain reverse sequence features of each frequency point; the reverse sequence feature of the current frequency point has a mapping relationship with the reverse sequence feature of the next frequency point; The intra-frame sequence feature of the current frequency point is determined according to the forward sequence feature and the reverse sequence feature of the current frequency point.

9. The audio processing method according to claim 8, characterized in that: Mapping the frequency domain features of the audio frame includes: Concatenating the sequence features of a reference frequency point with the frequency domain features of a current frequency point to obtain first input data, wherein the reference frequency point is a reference frequency point or a subsequent frequency point of the current frequency point; Mapping the first input data to obtain a reset weight and an update weight corresponding to the current frequency point, the reset weight being used to reset the degree of influence of the sequence characteristics of the reference frequency point on the input data, and the update weight being used to update the sequence characteristics of the current frequency point; Performing a weighted operation on the sequence features of the reference frequency point according to the reset weight and then concatenating the sequence features with the frequency domain features of the current frequency point to obtain second input data; Performing mapping processing on the second input data to obtain an initial sequence feature of the current frequency point; A weighted operation is performed on the sequence feature of the reference frequency point and the initial sequence feature of the current frequency point according to the updated weight to obtain the updated sequence feature of the current frequency point.

10. The audio processing method according to any one of claims 1 to 7, characterized in that: Performing serial modeling on the intra-frame sequence features of the plurality of audio frames to obtain inter-frame sequence features, including: Performing time-frequency transformation on the intra-frame sequence features of the plurality of audio frames to obtain time domain distribution information of the intra-frame sequence features at a plurality of frequency points; The time domain distribution information at each of the frequency points is serialized and modeled to obtain inter-frame sequence features.

11. The audio processing method according to claim 10, characterized in that: The time domain distribution information at each of the frequency points is serialized and modeled to obtain inter-frame sequence features, including: The time domain distribution information of each moment on the frequency point is mapped in sequence according to the specified time sequence to obtain the inter-frame sequence characteristics; the inter-frame sequence characteristics of the current moment have a mapping relationship with the inter-frame sequence characteristics of the reference moment, and the reference moment is the previous moment or the next moment of the current moment.

12. An audio processing device, characterized in that: include: An extraction module is configured to perform feature extraction on the audio signal to obtain frequency domain features of multiple audio frames; The audio signal carries a speech signal and a noise signal; A first modeling module is configured to perform serial modeling on the frequency domain features of the audio frame to obtain an intra-frame sequence feature, where the intra-frame sequence feature is used to represent a frequency sequence relationship between multiple frequency points in a single audio frame; a second modeling module configured to perform serial modeling on the intra-frame sequence features of the plurality of audio frames to obtain inter-frame sequence features, wherein the inter-frame sequence features are used to represent a temporal sequence relationship between the plurality of audio frames; The modulation module is configured to perform signal modulation on the audio signal according to the inter-frame sequence feature to obtain a speech signal after removing the noise signal.

13. A computer-readable medium, characterized in that The computer-readable medium stores a computer program, and when the computer program is executed by a processor, the audio processing method according to any one of claims 1 to 11 is implemented.

14. An electronic device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to execute the executable instructions to implement the audio processing method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the audio processing method according to any one of claims 1 to 11 is implemented.

Images