An audio data processing method, device and readable storage medium

Abstract

The application discloses an audio data processing method, device and readable storage medium. The method comprises the following steps: obtaining target audio data frames and K historical audio data frames associated with original audio data; obtaining M first-order time derivatives and M second-order time derivatives based on N target cepstrum coefficients of the obtained target audio data frames; obtaining N historical cepstrum coefficients corresponding to each historical audio data frame respectively, and determining a spectral dynamic feature based on the obtained K*N historical cepstrum coefficients; inputting the N target cepstrum coefficients, the M first-order time derivatives, the M second-order time derivatives and the spectral dynamic feature into a target mask estimation model to obtain a target mask corresponding to the target audio data frame; and the target mask is used to suppress noise data in the original audio data to obtain enhanced audio data corresponding to the original audio data. By using the application, noise data in the audio data can be effectively suppressed, and the speech fidelity is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an audio data processing method, apparatus, and readable storage medium. Background Technology

[0002] Currently, in some audio and video acquisition scenarios (such as audio and video conferencing scenarios), it is necessary to acquire audio data. However, in these acquired audio data, there is a high risk of non-stationary noise that interferes with the target speech in the current audio data, thereby reducing the acquisition quality of the target speech in the currently acquired audio data.

[0003] However, among these non-stationary noises, there may also be a type of Babble Noise composed of conversations between multiple speakers. Babble Noise is a type of non-stationary background noise that easily interferes with the target speech in the current audio data. Because the components of this non-stationary background noise are similar to those of the target speech, when performing speech enhancement processing on target speech containing this non-stationary background noise, speech data with similar speech components to the non-stationary background noise in the target speech is easily mistakenly cancelled, thus reducing the speech fidelity after noise suppression of the audio data. Summary of the Invention

[0004] This application provides an audio data processing method, apparatus, and readable storage medium that can effectively suppress noise data in audio data and improve voice fidelity.

[0005] One embodiment of this application provides an audio data processing method, including:

[0006] Obtain the target audio data frame and K historical audio data frames associated with the original audio data; both the target audio data frame and the K historical audio data frames are spectrum frames, and each of the K historical audio data frames is a spectrum frame preceding the target audio data frame, where K is a positive integer;

[0007] When N target cepstral coefficients of the target audio data frame are obtained, M first-order time derivatives and M second-order time derivatives associated with the target audio data frame are obtained based on the N target cepstral coefficients; N is a positive integer greater than 1, and M is a positive integer less than N;

[0008] Obtain N historical cepstral coefficients corresponding to each historical audio data frame, and determine the spectral dynamic characteristics associated with the target audio data frame based on the obtained K*N historical cepstral coefficients;

[0009] The target cepstral coefficients, M first-order time derivatives, M second-order time derivatives, and spectral dynamic features are input into the target mask estimation model, which outputs the target mask corresponding to the target audio data frame. The target mask is used to suppress noise data in the original audio data to obtain the enhanced audio data corresponding to the original audio data.

[0010] One embodiment of this application provides an audio data processing method, including:

[0011] Obtain the target sample audio data frame and K historical sample audio data associated with the sample audio data, and obtain the sample mask corresponding to the target sample audio data frame; the target sample audio data frame and the K historical sample audio data frames are all spectrum frames, and each of the K historical sample audio data frames is a spectrum frame preceding the target sample audio data frame, where K is a positive integer;

[0012] When N cepstral coefficients of the target sample audio data frame are obtained, M first-order time derivatives and M second-order time derivatives associated with the target sample audio data frame are obtained based on the N cepstral coefficients; N is a positive integer greater than 1, and M is a positive integer less than N.

[0013] Obtain N historical sample cepstral coefficients corresponding to each historical sample audio data frame, and determine the dynamic characteristics of the sample spectrum associated with the target sample audio data frame based on the obtained K*N historical sample cepstral coefficients;

[0014] The cepstral coefficients of N target samples, the first-order time derivatives of M samples, the second-order time derivatives of M samples, and the dynamic features of the sample spectrum are input into the initial mask estimation model, and the initial mask estimation model outputs the predicted mask corresponding to the audio data frame of the target sample.

[0015] The initial mask estimation model is iteratively trained based on the predicted mask and the sample mask to obtain the target mask estimation model used to output the target audio data frame associated with the original audio data; the target mask is used to suppress noise data in the original audio data to obtain the enhanced audio data corresponding to the original audio data.

[0016] One embodiment of this application provides an audio data processing apparatus, including:

[0017] The first acquisition module is used to acquire the target audio data frame and K historical audio data frames associated with the original audio data; the target audio data frame and the K historical audio data frames are all spectrum frames, and each of the K historical audio data frames is a spectrum frame preceding the target audio data frame, where K is a positive integer.

[0018] The second acquisition module is used to acquire, when N target cepstral coefficients of the target audio data frame are acquired, M first-order time derivatives and M second-order time derivatives associated with the target audio data frame based on the N target cepstral coefficients; N is a positive integer greater than 1 and M is a positive integer less than N;

[0019] The third acquisition module is used to acquire N historical cepstral coefficients corresponding to each historical audio data frame, and to determine the spectral dynamic features associated with the target audio data frame based on the acquired K*N historical cepstral coefficients.

[0020] The mask estimation module is used to input N target cepstral coefficients, M first-order time derivatives, M second-order time derivatives, and spectral dynamic features into the target mask estimation model, and output the target mask corresponding to the target audio data frame from the target mask estimation model. The target mask is used to suppress noise data in the original audio data to obtain the enhanced audio data corresponding to the original audio data.

[0021] The first acquisition module mentioned above includes:

[0022] The audio preprocessing unit is used to perform frame-by-frame windowing preprocessing on the raw audio data to obtain H audio data segments; H is a positive integer greater than 1.

[0023] The time-frequency conversion unit is used to perform time-frequency conversion on each audio data segment to obtain the audio data frame corresponding to each audio data segment.

[0024] The data frame determination unit is used to determine the target audio data frame and the K historical audio data frames preceding the target audio data frame from H audio data frames; K is less than H.

[0025] Among them, H audio data segments include audio data segment i, where i is a positive integer less than or equal to H;

[0026] The aforementioned time-frequency transformation unit is specifically used to perform a Fourier transform on the audio data segment i to obtain the DC component frequency point and 2S frequency points of the audio data segment i in the frequency domain; the 2S frequency points include S frequency points related to the first frequency point type and S frequency points related to the second frequency point type; S is a positive integer; the S frequency points related to the first frequency point type are obtained from the 2S frequency points, and the audio data frame corresponding to the audio data segment i is determined based on the S frequency points related to the first frequency point type and the DC component frequency points.

[0027] The target audio data frame contains S1 frequency points, including a DC component frequency point and S2 frequency points related to the frequency point type, where S1 and S2 are both positive integers.

[0028] The above-mentioned device also includes:

[0029] The frequency band mapping module is used to map S1 frequency points to N acoustic frequency bands; S1 is greater than or equal to N;

[0030] The cepstral processing module is used to perform cepstral processing on each acoustic frequency band to obtain the target cepstral coefficients corresponding to each acoustic frequency band.

[0031] Among them, N acoustic frequency bands include acoustic frequency band j, where j is a positive integer less than or equal to N;

[0032] The above cepstral processing module includes:

[0033] The energy acquisition unit is used to acquire the band energy of acoustic band j, and to perform a logarithmic transformation on the band energy of acoustic band j to obtain the logarithmic band energy of acoustic band j.

[0034] The cosine transform unit is used to perform discrete cosine transform on the logarithmic band energy of acoustic band j to obtain the target cepstral coefficients corresponding to acoustic band j.

[0035] The second acquisition module mentioned above includes:

[0036] The first differential unit is used to perform differential operations on N target cepstral coefficients to obtain (N-1) differential operation values. Each differential operation value in the (N-1) differential operation values ​​is used as the first time derivative of the corresponding target cepstral coefficient. M first time derivatives associated with the target audio data frame are obtained from the (N-1) first time derivatives.

[0037] The second difference unit is used to perform a second difference operation on (N-1) first-order time derivatives to obtain (N-2) difference operation values. Each difference operation value among the (N-2) difference operation values ​​is used as the second-order time derivative of the corresponding target cepstral coefficient. M second-order time derivatives associated with the target audio data frame are obtained from the (N-2) second-order time derivatives.

[0038] The third acquisition module mentioned above includes:

[0039] The data frame acquisition unit is used to acquire any two adjacent historical audio data frames from K historical audio data frames as the first historical audio data frame and the second historical audio data frame; the second historical audio data frame is a spectrum frame obtained after the first historical audio data frame.

[0040] The coefficient acquisition unit is used to acquire N historical cepstral coefficients corresponding to the first historical audio data frame from the cache associated with the target audio data frame, and to acquire N historical cepstral coefficients corresponding to the second historical audio data frame; the acquired N historical cepstral coefficients corresponding to the first historical audio data frame are used as the first historical cepstral coefficients, and the acquired N historical cepstral coefficients corresponding to the second historical audio data frame are used as the second historical cepstral coefficients.

[0041] The difference determination unit is used to take the frequency band difference value between the first historical cepstral coefficient and the second historical cepstral coefficient as the inter-frame difference value between the first historical audio data frame and the second historical audio data frame.

[0042] The feature determination unit is used to obtain the inter-frame difference value between the K historical audio data frames when each adjacent historical audio data frame in the K historical audio data frames is used as the first historical audio data frame and the second historical audio data frame, and to determine the spectral dynamic features associated with the target audio data frame based on the inter-frame difference value between the K historical audio data frames.

[0043] The aforementioned difference determination unit includes:

[0044] The coefficient difference acquisition subunit is used to acquire the historical cepstral coefficient Lp from the N historical cepstral coefficients contained in the first historical cepstral coefficient, and to acquire the historical cepstral coefficient Lq from the N historical cepstral coefficients contained in the second historical cepstral coefficient; p and q are both positive integers less than or equal to N, and p = q; and to acquire the coefficient difference value between the historical cepstral coefficient Lp and the historical cepstral coefficient Lq.

[0045] The difference value determination subunit is used to determine the frequency band difference value between the first historical cepstral coefficient and the second historical cepstral coefficient based on the coefficient difference value, and to use the frequency band difference value as the inter-frame difference value between the first historical audio data frame and the second historical audio data frame.

[0046] The target mask estimation model includes a mask estimation network layer and a mask output layer.

[0047] The mask estimation module mentioned above includes:

[0048] The mask estimation unit is used to take N target cepstral coefficients, M first-order time derivatives, M second-order time derivatives and spectral dynamic features as target audio features of the target audio data frame, input the target audio features into the mask estimation network layer, and perform mask estimation on the target audio features through the mask estimation network layer to obtain the hidden features corresponding to the target audio features.

[0049] The mask output unit is used to input hidden features into the mask output layer, and to merge the hidden features through the mask output layer to obtain the target mask corresponding to the target audio data frame.

[0050] The mask estimation network layer includes a first mask estimation network layer, a second mask estimation network layer, and a third mask estimation network layer with skip connections.

[0051] The aforementioned mask estimation unit includes:

[0052] The first estimation subunit is used to input the target audio features into the first mask estimation network layer and output the first intermediate features through the first mask estimation network layer;

[0053] The second estimation subunit is used to perform feature concatenation on the first intermediate feature and the target audio feature according to the skip connection between the first mask estimation network layer and the second mask estimation network layer to obtain the second intermediate feature, input the second intermediate feature into the second mask estimation network layer, and output the third intermediate feature through the second mask estimation network layer.

[0054] The third estimation subunit is used to perform feature concatenation on the third intermediate feature, the target audio feature, and the first intermediate feature based on the skip connections between the first mask estimation network layer and the third mask estimation network layer, as well as the skip connections between the second mask estimation network layer and the third mask estimation network layer, to obtain the fourth intermediate feature. The fourth intermediate feature is then input into the third mask estimation network layer, and the hidden feature corresponding to the target audio feature is output through the third mask estimation network layer.

[0055] The aforementioned device also includes:

[0056] The noise suppression module is used to interpolate the target mask to obtain an interpolated mask. The length of the interpolated mask is the same as the length of the target audio data frame. The interpolated mask is multiplied by the target audio data frame, and the result of the multiplication is subjected to an inverse Fourier transform to obtain the target audio data after noise suppression. After noise suppression is performed on each audio data frame associated with the original audio data, the enhanced audio data corresponding to the original audio data is obtained.

[0057] One embodiment of this application provides an audio data processing apparatus, including:

[0058] The first acquisition module is used to acquire a target sample audio data frame and K historical sample audio data associated with the sample audio data, and to acquire the sample mask corresponding to the target sample audio data frame; the target sample audio data frame and the K historical sample audio data frames are all spectrum frames, and each of the K historical sample audio data frames is a spectrum frame preceding the target sample audio data frame, where K is a positive integer;

[0059] The second acquisition module is used to acquire, based on the N target sample cepstral coefficients of the target sample audio data frame, the first-order time derivatives of M samples and the second-order time derivatives of M samples associated with the target sample audio data frame; N is a positive integer greater than 1 and M is a positive integer less than N.

[0060] The third acquisition module is used to acquire N historical sample cepstral coefficients corresponding to each historical sample audio data frame, and to determine the dynamic characteristics of the sample spectrum associated with the target sample audio data frame based on the acquired K*N historical sample cepstral coefficients.

[0061] The mask prediction module is used to input the cepstral coefficients of N target samples, the first-order time derivatives of M samples, the second-order time derivatives of M samples, and the dynamic features of the sample spectrum into the initial mask estimation model, and the initial mask estimation model outputs the predicted mask corresponding to the audio data frame of the target sample.

[0062] The model training module is used to iteratively train the initial mask estimation model based on the prediction mask and the sample mask to obtain the target mask estimation model used to output the target audio data frame associated with the original audio data; the target mask is used to suppress noise data in the original audio data to obtain the enhanced audio data corresponding to the original audio data.

[0063] One embodiment of this application provides a computer device, including: a processor and a memory;

[0064] The processor is connected to a memory, which stores a computer program. When the computer program is executed by the processor, it causes the computer device to perform the method provided in the embodiments of this application.

[0065] One aspect of this application provides a computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor, so that a computer device having the processor performs the method provided in this application.

[0066] One embodiment of this application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method provided in this application embodiment.

[0067] In this embodiment, a target audio data frame and K historical audio data frames associated with the original audio data can be obtained. Both the target audio data frame and the K historical audio data frames are spectral frames, and each of the K historical audio data frames is a spectral frame preceding the target audio data frame. Further, based on the N target cepstral coefficients of the obtained target audio data frame, M first-order time derivatives and M second-order time derivatives associated with the target audio data frame can be obtained. Additionally, based on the N historical cepstral coefficients corresponding to each historical audio data frame, spectral dynamic features associated with the target audio data frame can be determined. Subsequently, the obtained N target cepstral coefficients, M first-order time derivatives, M second-order time derivatives, and spectral dynamic features can be input into a target mask estimation model associated with the original audio data. This target mask estimation model can then output a target mask corresponding to the target audio data frame. This target mask can be used to suppress noise data (such as Babble Noise) in the original audio data to obtain enhanced audio data corresponding to the original audio data. Therefore, this application embodiment, when performing speech enhancement on the original audio data, can comprehensively consider multiple audio features, including target cepstral coefficients, first-order time derivatives, second-order time derivatives, and spectral dynamic features. This allows for a more accurate description of the time-frequency relationship between clean speech data and background noise data, resulting in a more accurate target mask. Thus, applying each output mask to the corresponding audio data frame effectively suppresses noise data in the audio data and improves speech fidelity. Furthermore, unlike directly estimating the mask from noisy frequency data using a neural network model, this application embodiment first uses digital signal processing technology to extract corresponding audio features from the noisy frequency data, and then inputs the extracted audio features into a trained neural network model (i.e., the target mask estimation model) for mask estimation. Therefore, this application embodiment requires lower network complexity, thereby reducing computational complexity and CPU (Central Processing Unit) consumption, and ultimately improving audio data processing efficiency. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 This is a schematic diagram of a system architecture provided in an embodiment of this application;

[0070] Figure 2 This is a schematic diagram of an audio data processing scenario provided in an embodiment of this application;

[0071] Figure 3 This is a flowchart illustrating an audio data processing method provided in an embodiment of this application;

[0072] Figure 4a This is a schematic diagram of an audio preprocessing scenario provided in an embodiment of this application;

[0073] Figure 4b This is a schematic diagram of an audio preprocessing scenario provided in an embodiment of this application;

[0074] Figure 5 This is a schematic diagram of a cepstral coefficient difference operation provided in an embodiment of this application;

[0075] Figure 6 This is a schematic diagram illustrating a scenario for obtaining inter-frame difference values ​​provided in an embodiment of this application;

[0076] Figure 7 This is a schematic diagram of the network structure of a mask estimation model provided in an embodiment of this application;

[0077] Figure 8 This is a flowchart illustrating an audio data processing method provided in an embodiment of this application;

[0078] Figure 9 This is a schematic diagram of a model training process provided in an embodiment of this application;

[0079] Figure 10 This is a schematic diagram illustrating a noise reduction effect provided in an embodiment of this application;

[0080] Figure 11 This is a schematic diagram of the structure of an audio data processing device provided in an embodiment of this application;

[0081] Figure 12 This is a schematic diagram of the structure of an audio data processing device provided in an embodiment of this application;

[0082] Figure 13 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application;

[0083] Figure 14 This is a schematic diagram of the structure of an audio data processing system provided in an embodiment of this application. Detailed Implementation

[0084] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0085] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0086] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0087] Speech enhancement (SE) technology refers to the technique of extracting useful speech signals from a noisy background and suppressing or reducing noise interference when speech signals are interfered with or even submerged by various noises. Speech enhancement technology can separate speech noise from non-speech noise to ensure speech intelligibility; that is, it extracts the purest possible original speech from noisy speech. Speech enhancement has a wide range of applications, including voice calls, teleconferencing, real-time audio and video conferencing, scene recording, hearing aids, and speech recognition devices, and has become a preprocessing module in many speech coding and recognition systems. There are many existing speech enhancement methods, which can be divided into two categories according to the technology they employ: digital signal processing (DSP) speech enhancement technology and machine learning-based speech enhancement technology. DSP speech enhancement technology can handle stationary noise well in real-time online speech enhancement, but its ability to suppress non-stationary noise is poor. Among traditional DSP speech enhancement technologies, based on the number of channels, it can be further divided into single-channel speech enhancement technology and microphone array speech enhancement technology. The unique characteristic of machine learning / deep learning-based speech enhancement technology in noise suppression is its ability to reduce highly non-stationary noise and background sounds, and it is being commercially applied in real-time communication. It should be understood that non-stationary noise here refers to noise whose statistical characteristics change over time, such as dog barks, banging sounds from kitchen utensils, baby cries, construction noise, or traffic noise captured along with the target speech during audio and video acquisition.

[0088] The solutions provided in this application relate to digital signal processing technology. Digital signal processing (DSP) is a technology that converts analog information (such as audio, video, and images) into digital information. It utilizes computers or dedicated processing equipment to perform digital processing on signals, including acquisition, transformation, filtering, estimation, enhancement, compression, and recognition, to obtain a signal form that meets human needs. Digital signal processing involves multiple disciplines such as mathematics, network theory, signals and systems, cybernetics, communication theory, artificial intelligence, pattern recognition, and neural networks, and has been widely applied in fields such as communications. In this application embodiment, digital signal processing technology can be used to extract target audio features from target audio data frames, including target cepstral coefficients, first-order time derivatives, second-order time derivatives, and spectral dynamic characteristics.

[0089] The solutions provided in this application relate to machine learning technology within the field of artificial intelligence. It is understood that machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning typically includes techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and formulaic learning. In this application embodiment, the target mask estimation model is an AI model based on machine learning technology, which can be used to estimate the corresponding mask from the input audio features.

[0090] Please see Figure 1 , Figure 1 This is a schematic diagram of a system architecture provided in an embodiment of this application. For example... Figure 1 As shown, the system architecture may include a business server 100 and a user terminal cluster. The user terminal cluster may include one or more user terminals; the number of user terminals in the user terminal cluster is not limited here. Figure 1 As shown, the user terminal cluster may specifically include user terminal 200a, user terminal 200b, user terminal 200c, ..., user terminal 200n. Communication connections may exist between user terminals in the cluster; for example, user terminal 200a and user terminal 200b may be connected, as may user terminal 200a and user terminal 200c. Simultaneously, any user terminal in the user terminal cluster may have a communication connection with the service server 100, enabling each user terminal in the cluster to interact with the service server 100 through this communication connection. For example, user terminal 200a may have a communication connection with the service server 100. The communication connection method is not limited; it can be established directly or indirectly through wired communication, wireless communication, or other methods. This application does not impose any restrictions on this method.

[0091] It should be understood that, such as Figure 1 Each user terminal in the user terminal cluster shown can have an application client installed. When the application client runs on each user terminal, it can interact with the aforementioned... Figure 1The business servers 100 shown interact with each other. The application client can be a social client, instant messaging client (e.g., a conferencing client), entertainment client (e.g., a game client, a live streaming client), multimedia client (e.g., a video client), information client (e.g., a news client), shopping client, in-vehicle client, smart home client, or any other client capable of displaying text, images, audio, and video data. For example, in some embodiments, the application client can be a client with audio and video communication capabilities. It should be noted that this audio and video communication capability can be a simple audio communication capability or a video communication capability. This capability can be widely used in various business scenarios involving audio and video acquisition, such as audio and video conferencing, audio and video calls, and audio and video live streaming, in different fields such as enterprise office, instant messaging, online education, telemedicine, and digital finance. The application client can be a standalone client or an embedded sub-client integrated into a client (e.g., a social client, a video client, etc.), and there is no limitation on this. Taking an instant messaging client as an example, the business server 100 can be a collection of multiple servers, including the backend server corresponding to the instant messaging client and the data processing server. Therefore, each user terminal can transmit data with the business server 100 through the instant messaging client. For example, each user terminal can collect relevant audio and video data in real time and send the collected audio and video data to other user terminals through the business server 100 to realize audio and video communication (e.g., conducting remote real-time audio and video conferences).

[0092] It should be understood that in practical application scenarios (such as real-time audio and video communication scenarios), the audio data acquired during the audio and video acquisition process is inevitably interfered with by external noise, especially the noise composed of background human voices (i.e., Babble Noise), which is even more difficult to eliminate from the target speech in the current audio data. In order to improve the acquisition quality of the target speech, it is necessary to suppress such noise. Based on this, this application provides a method for real-time noise suppression of audio data. This method combines traditional digital signal processing with neural networks with excellent nonlinear fitting capabilities to suppress noise data (such as Babble Noise) in the audio and video communication process while maintaining extremely high speech fidelity. For ease of subsequent understanding and explanation, this application can refer to the source object of the target speech as the business object (e.g., the user speaking during audio and video communication, or the speaker), and the audio data to be processed associated with the business object as the raw audio data. It can be understood that the raw audio data here can be obtained by collecting the sound in the real environment where the business object is located through audio devices, and may simultaneously contain the speech data generated by the business object (i.e., the speech data of the target speech) and the noise data in the environment. The noise data in this application embodiment refers to non-stationary background noise data, which may include real conversations around the business object (i.e., Babble Noise), singing or talking sounds carried by a multimedia file being played, and other similar non-stationary background noise. The multimedia file can be a video file carrying both image and audio data, such as short videos, TV series, movies, music videos (MVs), animations, etc.; or it can be an audio file mainly composed of audio data, such as songs, audiobooks, radio dramas, radio programs, etc. This application embodiment does not limit the type, content, source, or format of the multimedia file. Furthermore, this application embodiment may refer to the neural network model used for mask estimation of audio features extracted from the original audio data as a target mask estimation model.

[0093] Optionally, the audio device can be a hardware component located in the user terminal, such as a microphone of the user terminal; or, optionally, the audio device can also be a hardware device connected to the user terminal, such as a microphone connected to the user terminal, used to provide the user terminal with raw audio data acquisition services, and the audio device can include an audio sensor, a microphone, etc.

[0094] It is understood that the methods provided in the embodiments of this application can be executed by a computer device, which includes, but is not limited to, a user terminal (e.g., Figure 1 (any user terminal in the user terminal cluster shown) or service server (e.g., Figure 1 The business server 100 shown is a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud databases, cloud services, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The user terminal can be a smartphone, tablet, laptop, desktop computer, PDA, wearable device (e.g., smartwatch, smart bracelet, smart hearing aid), smart computer, smart vehicle, or other smart terminal capable of running the aforementioned application clients. The user terminal and the business server can be connected directly or indirectly via wired or wireless means; this embodiment does not impose any limitations on this.

[0095] For ease of understanding, user terminal 200a and user terminal 200b will be used as examples. Assume that business object 1 communicates with business object 2 corresponding to user terminal 200b via an application client on user terminal 200a (e.g., business object 1 and business object 2 are having a voice-only conference). When business object 1 speaks, user terminal 200a can obtain the raw audio data associated with business object 1 through relevant audio devices (e.g., the microphone on user terminal 200a). It should be noted that in this embodiment, the original audio data can be a mixed audio signal in the time domain. Since it is difficult to directly extract the pure speech signal from the mixed audio signal in the time domain, this application will solve the speech separation problem from the frequency domain. Specifically, the original audio data can be preprocessed to obtain multiple spectrum frames in the frequency domain (which can also be called audio data frames in this embodiment). Each spectrum frame contains a portion of the spectrum of the original audio data in the frequency domain. For easy distinction, this embodiment can refer to any one of the spectrum frames to be processed as the target audio data frame. Correspondingly, the spectrum frame located before the target audio data frame in the frequency domain can be called the historical audio data frame. That is to say, the historical audio data frame is the spectrum frame obtained before the target audio data frame.

[0096] Based on this, the user terminal 200a can further acquire a target audio data frame associated with the original audio data and K historical audio data frames, where K is a positive integer. This embodiment does not limit the specific number of historical audio data frames. Both the target audio data frame and the K historical audio data frames are spectral frames, and each of the K historical audio data frames is a spectral frame preceding the target audio data frame. To improve the accuracy and intelligibility of speech separation, this application utilizes various audio features for mask estimation. For ease of understanding, the target audio data frame is used as an example; the processing of other spectral frames is consistent with the processing of the target audio data frame. Specifically, user terminal 200a can acquire N target cepstral coefficients of the target audio data frame, and then, based on these N target cepstral coefficients, acquire M first-order time derivatives and M second-order time derivatives associated with the target audio data frame, where N is a positive integer greater than 1 and M is a positive integer less than N. This embodiment does not limit the specific number of target cepstral coefficients, the specific number of first-order time derivatives, or the specific number of second-order time derivatives. Furthermore, user terminal 200a can also acquire N historical cepstral coefficients corresponding to each historical audio data frame, and can determine the spectral dynamic characteristics associated with the target audio data frame based on the acquired K*N historical cepstral coefficients.

[0097] In this embodiment, the cepstral coefficients, first-order time derivatives, second-order time derivatives, and spectral dynamic features associated with each spectral frame can be collectively referred to as audio features. For ease of distinction, the audio features corresponding to the target audio data frame can be called target audio features. It can be understood that the target cepstral coefficients can be used to characterize the acoustic features of the target audio data frame, while the associated first-order time derivatives, second-order time derivatives, and spectral dynamic features can characterize the temporal correlation characteristics (or the stability characteristics of the audio signals) between audio signals. Therefore, the user terminal 200a can input the aforementioned N target cepstral coefficients, M first-order time derivatives, M second-order time derivatives, and spectral dynamic features into a trained target mask estimation model. The target mask estimation model outputs the target mask corresponding to the target audio data frame. This target mask can be used to suppress noise data in the original audio data to obtain enhanced audio data corresponding to the original audio data. In other words, by using the mask (i.e., Mask, also called a masking device, etc.) corresponding to each spectral frame, speech data and noise data in the original audio data can be effectively separated, achieving speech enhancement in the audio-visual communication process. It is understandable that the mask obtained by modeling multiple audio features is more accurate, and therefore, the speech fidelity of the enhanced audio data obtained using this mask is also very high.

[0098] In the embodiments of this application, the target mask may include, but is not limited to, an ideal ratio mask (IRM), an ideal binary mask (IBM), an optimal ratio mask (ORM), etc. The type of target mask will not be limited here.

[0099] It is understandable that user terminal 200a can subsequently send the obtained enhanced audio data to service server 100, and service server 100 can then distribute the enhanced audio data to terminal device 200b. Similarly, it is understandable that when service object 2 speaks, terminal device 200b can also perform a similar voice enhancement process to send the obtained enhanced audio data related to service object 2 to terminal device 200a. In this way, service object 1 and service object 2 can always hear the high-quality voice sent by the other party during audio and video communication, thereby achieving high-quality audio and video communication and improving user experience.

[0100] Optionally, in some embodiments, the application client can also be a client with audio and video editing capabilities. This capability allows for voice enhancement processing of the raw audio data to be processed. This capability can be applied to business scenarios involving audio and video acquisition, such as audio and video production and recording. The raw audio data can be obtained by recording the sounds of the real-world environment of the business object (here referring to the target user for whom voice enhancement needs to be performed) in real time using an audio device. Alternatively, the raw audio data can be obtained from the multimedia file to be processed (which may include video and audio files). This application embodiment does not impose any limitations on this. Similarly, the raw audio data can also be an audio signal that mixes the voice data of the business object with noise data from the environment in which the business object is located. The process of performing voice enhancement processing on this raw audio data is similar to the voice enhancement processing process described above for audio and video communication scenarios. The resulting clearer enhanced audio data can be directly stored or sent, or it can be used to replace the raw audio data in the multimedia file to be processed. Compared to real-time audio and video communication scenarios, the real-time requirements for voice enhancement are lower in non-real-time audio and video editing scenarios, but the user's need for high-quality voice can still be met.

[0101] Optionally, and understandably, the service server can also obtain the raw audio data sent by the user terminal, and by loading a trained target mask estimation model, obtain the target mask corresponding to the target audio data frame associated with the raw audio data, thereby achieving speech enhancement. Figure 1The system architecture shown can have one or more service servers. A user terminal can be connected to a service server. Each service server can obtain the raw audio data uploaded by the user terminal connected to it and perform voice enhancement on it.

[0102] It is understood that the above system architecture is applicable to a variety of business scenarios involving audio and video acquisition, including: audio and video conferencing scenarios, audio and video call scenarios, audio and video live streaming scenarios, audio and video interview scenarios, remote visitation scenarios, real-time noise reduction scenarios such as hearing aid voice enhancement and speech recognition, as well as non-real-time noise reduction scenarios such as audio and video recording and post-production, or other business scenarios that require voice enhancement processing of the acquired audio data, especially business scenarios that require real-time suppression of Babble Noise. Specific business scenarios will not be listed one by one here.

[0103] For ease of understanding, please refer to the following: Figure 2 , Figure 2 This is a schematic diagram of an audio data processing scenario provided in an embodiment of this application. Wherein, as... Figure 2 The computer device 20 shown can be the above-mentioned Figure 1 The corresponding embodiment refers to either the service server 100 or any user terminal in the user terminal cluster (e.g., user terminal 200a), which is not limited here.

[0104] like Figure 2 As shown, the original audio data 201 can be a mixed audio signal containing voice data of the business object and noise data in the environment. The original audio data 201 can be audio data collected in real time by the computer device 20 through relevant audio devices, audio data obtained by the computer device 20 from the multimedia file to be processed, or audio data sent by other computer devices to the computer device 20 for audio processing. This application embodiment does not limit this.

[0105] It is understood that after acquiring the original audio data 201, the computer device 20 can suppress the noise data to obtain audio data with better speech quality. To achieve this goal, the computer device 20 can first use traditional digital signal processing techniques to extract the audio features of the original audio data 201. Before this, the computer device 20 can perform audio preprocessing on the original audio data 201, specifically including frame-by-frame windowing preprocessing, time-frequency transformation, and other operations, thereby obtaining an audio data frame set 202 associated with the original audio data 201. This audio data frame set 202 can include multiple audio data frames located in the frequency domain (i.e., spectrum frames). Here, the number of audio data frames included in the audio data frame set 202 is not limited. Subsequently, the computer device 20 can perform audio feature extraction, mask estimation, noise suppression, and other processing operations on each audio data frame in the audio data frame set 202. In this embodiment, the processing order of each audio data frame is not limited. For example, multiple audio data frames can be processed in parallel, or each audio data frame can be processed serially according to the order of acquisition time. For ease of understanding and explanation, in this embodiment, any audio data frame to be processed in the audio data frame set 202 can be used as the target audio data frame. For example, audio data frame 203 in the audio data frame set 202 can be used as the target audio data frame. When other audio data frames are used as target audio data frames, the corresponding processing procedure is the same as the processing procedure for audio data frame 203. In addition, the computer device 20 can also obtain a historical audio data frame set 204 related to audio data frame 203 from the audio data frame set 202. The historical audio data frame set 204 can include K historical audio data frames preceding the target audio data frame. For example, these K historical audio data frames can be audio data frames A1, ..., audio data frame A1 in sequence. K Where K is a positive integer, and the specific value of K is not limited here. It can be understood that audio data frame A1 to audio data frame A... K All of these are spectrum frames prior to audio data frame 203.

[0106] Furthermore, the computer device 20 can extract audio features from the target audio data frame. Taking audio data frame 203 as an example, the computer device 20 can obtain the cepstral coefficient set 205 corresponding to the audio data frame 203. This cepstral coefficient set 205 can be used to characterize the acoustic features of the audio data frame 203. The cepstral coefficient set 205 can include N target cepstral coefficients of the audio data frame 203. For example, the N target cepstral coefficients can specifically include cepstral coefficient B1, cepstral coefficient B2, ..., cepstral coefficient B... NN is a positive integer greater than 1, and the specific value of N is not limited here. Subsequently, the computer device 20 can obtain M first-order time derivatives and M second-order time derivatives associated with the audio data frame 203 based on the N target cepstral coefficients in the cepstral coefficient set 205, where M is a positive integer less than N, and the specific value of M is not limited here. Among them, the first-order time derivatives can be obtained by applying the above cepstral coefficients B1, B2, ..., B... N The second-order time derivative is obtained by performing a difference operation on the first-order time derivative. The specific calculation process can be found later. Figure 3 The relevant description of step S102 in the corresponding embodiment. For example... Figure 2 As shown, after the corresponding calculations, the computer device 20 can obtain a set of first-order time derivatives 206 and a set of second-order time derivatives 207. The set of first-order time derivatives 206 may include M first-order time derivatives associated with the audio data frame 203. For example, the M first-order time derivatives may specifically include first-order time derivatives C1, ..., C2. M Similarly, the set of second-order time derivatives 207 may include M second-order time derivatives associated with the audio data frame 203. For example, the M second-order time derivatives may specifically include second-order time derivatives D1, ..., D2. M .

[0107] Furthermore, in order to more accurately characterize the stability of the original audio data, the computer device 20 can also acquire the spectral dynamics associated with the target audio data frame. Taking audio data frame 203 as an example, after acquiring the aforementioned historical audio data frame set 204, the computer device 20 can acquire N historical cepstral coefficients corresponding to each historical audio data frame in the historical audio data frame set 204. For example, it can acquire the N historical cepstral coefficients corresponding to audio data frame A1, including cepstral coefficient A... 11 cepstral coefficients A 12 ..., cepstral coefficients A 1N ...; Obtain audio data frame A K The corresponding N historical cepstral coefficients, including cepstral coefficient A K1 cepstral coefficients A K2 ..., cepstral coefficients A KNIn this embodiment, the obtained K*N historical cepstral coefficients can be used as a cepstral coefficient set 208. It is understood that the process of obtaining the N historical cepstral coefficients corresponding to each historical audio data frame is similar to the process of obtaining the N target cepstral coefficients corresponding to the audio data frame 203, and will not be repeated here. Furthermore, the computer device 20 can determine the spectral dynamic features 209 associated with the audio data frame 203 based on the K*N historical cepstral coefficients in the cepstral coefficient set 208; the specific process can be found in the following sections. Figure 3 The relevant description of step S103 in the corresponding embodiment.

[0108] Furthermore, after acquiring the audio features of audio data frame 203, computer device 20 can load a pre-trained target mask estimation model (e.g., mask estimation model 210). The aforementioned cepstral coefficient set 205, first-order time derivative set 206, second-order time derivative set 207, and spectral dynamic features 209 can then be input into the mask estimation model 210. The mask estimation model 210 performs mask estimation on the input audio features to obtain the target mask (e.g., mask 211) corresponding to audio data frame 203. Subsequently, computer device 20 can apply the obtained mask 211 to audio data frame 203 to suppress noise data. It can be understood that the mask's function is equivalent to preserving as much of the original audio data as possible of the speech data of the business object, while eliminating noise data that causes interference (e.g., conversations of other people near the business object). In addition, computer device 20 can also perform mask estimation on other audio data frames (e.g., audio data frames A1, ..., audio data frames A...). K The processing of (etc.) is similar to that of audio data frame 203, and will not be described in detail here.

[0109] Finally, after the computer device 20 performs noise suppression on each audio data frame in the audio data frame set 202, it can obtain the enhanced audio data 212 corresponding to the original audio data 201. At this time, the noise content in the enhanced audio data 212 is extremely low, and the voice data of the business object is effectively preserved, resulting in extremely high voice fidelity.

[0110] It is understandable that computer device 20 can utilize an audio database containing massive amounts of audio data to train a neural network to obtain the aforementioned mask estimation model 210. For details of the training process, please refer to the subsequent sections. Figure 8 The corresponding implementation examples.

[0111] It is understandable that the source of the original audio data may differ in different business scenarios, and correspondingly, the purpose of the resulting enhanced audio data may also differ. For example, in real-time audio and video communication scenarios such as audio and video conferencing, audio and video calls, audio and video live streaming, audio and video interviews, and remote visits, computer device 20 can send the enhanced audio data F1 obtained by real-time voice enhancement processing of the original audio data E1 to the user terminals of other users communicating with business object 1 via audio and video. As another example, in a hearing aid voice enhancement scenario, computer device 20 can perform voice enhancement processing on the original audio data E2 associated with business object 2 obtained by the hearing aid, thereby returning the enhanced audio data F2 containing clear voice data of business object 2 to the hearing aid for playback. Yet another example, in a speech recognition scenario, after obtaining the original audio data E3 input by business object 3, computer device 20 can first perform voice enhancement processing on it to obtain enhanced audio data F3, and then perform speech recognition on the high-quality voice data contained in the enhanced audio data F3, thereby improving the accuracy of speech recognition. For example, in an audio / video recording scenario, computer device 20 can perform voice enhancement processing on the original audio data E4 entered by business object 4, and can store the obtained enhanced audio data F4 (e.g., store it in the local cache of computer device 20 or upload it to cloud storage) or send it (e.g., send it as an audio / video session message in an instant messaging process to other user terminals for playback); for example, in an audio / video production scenario, computer device 20 can obtain the original audio data E5 from the multimedia file to be processed and perform voice enhancement processing on it, and then replace the original audio data E5 in the multimedia file to be processed with the obtained enhanced audio data F5, thereby improving the audio quality in the multimedia file.

[0112] In this process, computer device 20 obtains a target mask estimation model by training an initial mask estimation model, acquires the target audio features of the target audio data frame associated with the original audio data, and performs mask estimation on the target audio features using the target mask estimation model. The specific implementation method for noise suppression using the target mask can be found below. Figures 3-10 The description in the corresponding embodiments.

[0113] Please see Figure 3 , Figure 3This is a flowchart illustrating an audio data processing method provided in an embodiment of this application. It is understood that the method provided in this embodiment can be executed by a computer device, which includes, but is not limited to, a user terminal or a service server running a target mask estimation model. For ease of understanding, this embodiment uses a user terminal as an example to illustrate the specific process of performing audio processing (such as speech enhancement) on raw audio data in that user terminal. Figure 3 As shown, the method may include at least the following steps S101-S104:

[0114] Step S101: Obtain the target audio data frame and K historical audio data frames associated with the original audio data;

[0115] Specifically, the user terminal can acquire raw audio data containing voice data of the business object and noise data in the environment. This raw audio data can be audio data acquired in real time by the user terminal through an audio device, audio data obtained from a multimedia file to be processed, or audio data sent from other related user terminals. There are no restrictions on this.

[0116] It's understandable that, from a statistical perspective, speech data exhibits certain stationary properties. For example, within a single articulation unit lasting from tens to hundreds of milliseconds, speech data can demonstrate significant stability and regularity. Based on this, speech enhancement processing can be performed on a segment of audio data using smaller articulation units (such as phonemes, words, or bytes). Therefore, before extracting audio features from the original audio data, the user terminal can perform audio preprocessing to obtain multiple spectral frames located in the frequency domain.

[0117] In one implementation, a sliding window can be used to extract short segments from the original audio data. Specifically, the user terminal can perform frame-segmentation and windowing preprocessing on the original audio data to obtain H audio data segments, where H is a positive integer greater than 1. This application embodiment does not limit the specific number of audio data segments. It should be noted that frame-segmentation and windowing preprocessing can include frame-segmentation and windowing operations. First, the user terminal can perform frame-segmentation on the original audio data to obtain H audio signal frames in the time domain. It is understood that since the beginning and end of each audio signal frame will be discontinuous after framing, the more audio signal frames are segmented, the greater the error compared to the original audio data. Therefore, this application embodiment can solve this problem by using windowing operations to make the framed signal continuous, and each frame signal can exhibit the characteristics of a periodic function. That is, the user terminal can perform windowing operations on each of the H audio signal frames obtained above to obtain H audio data segments with continuous signals. In this application embodiment, when performing windowing operations, each audio signal frame is multiplied sequentially by a window function to obtain the corresponding audio data segment. The window functions include, but are not limited to, Vorbis window, Hamming window, rectangular window, Hanning window, etc. In practical applications, appropriate window functions can be selected as needed, and this application embodiment does not limit them.

[0118] It should be noted that the user terminal can determine the number of audio signal frames that can be divided based on the length of the original audio data, the frame length used in the framing operation, and the frame shift. The frame length refers to the length of an audio signal frame, which can be expressed in various ways, such as time or the number of sampling points. Optionally, if expressed as time, the length of an audio signal frame can typically be between 15ms and 30ms. In practical applications, a suitable frame length can be selected as needed, and this application does not limit this. For example, in some embodiments, the frame length can be set to 20ms, and an audio signal frame with a frame length of 20ms refers to an audio signal with a duration of 20ms. Optionally, it can also be expressed as the number of sampling points. For example, in some embodiments, assuming the sampling rate of the original audio data is 16kHz and the frame length is 20ms, then an audio signal frame can consist of 16kHz * 20ms = 320 sampling points. The frame shift refers to the distance moved during each framing operation. Starting from the beginning of the first audio signal frame, a frame shift is moved to begin the next frame. This can also be expressed in two ways. For example, in some embodiments, it can be expressed in terms of time, with the frame shift set to 12ms; or in some embodiments, it can be expressed in terms of the number of sampling points, with the frame shift set to 192 sampling points for raw audio data with a sampling rate of 16kHz.

[0119] For easier understanding, please refer to Figures 4a-4b , Figures 4a-4b This is a schematic diagram of an audio preprocessing scenario provided in an embodiment of this application. For example... Figure 4a As shown, when performing frame segmentation on raw audio data of length T, the frame length can be set to T1 (e.g., 20ms), and the frame shift to T2 (e.g., 12ms). Starting from the beginning of the raw audio data, an audio signal of frame length T1 is taken, resulting in the first audio signal frame, i.e., audio signal frame 1. Subsequently, a frame shift of length T2 is performed, and another audio signal of frame length T1 is taken from the current position, resulting in the second audio signal frame, i.e., audio signal frame 2. This process continues until H audio signal frames are obtained, where H = (T - T1) ÷ T2 + 1. It is understandable that, in particular, during the frame segmentation operation, there may be a situation where the remaining signal length is less than one frame. In this case, zero-padding can be performed on this remaining signal segment to make it reach the length of one frame (i.e., T1), or it can be discarded directly, because the last frame is at the very end of the raw audio data and is mostly a silent segment. Further details can be found in [link to relevant documentation]. Figure 4b ,like Figure 4b As shown, after obtaining H audio signal frames through the above framing operation, the user terminal can apply a window function to each audio signal frame sequentially to obtain the corresponding audio data segment. For example, multiplying audio signal frame 1 by the window function yields audio data segment 1; multiplying audio signal frame 2 by the window function yields audio data segment 2; ...; multiplying audio signal frame H by the window function yields audio data segment H. It can be understood that audio data segments 1 to H are arranged in chronological order.

[0120] In some embodiments, both the analysis window and the synthesis window can be Vorbis windows, which satisfy the Princen-Bradley criterion. The analysis window refers to the window function used in frame-by-frame windowing preprocessing. To achieve perfect reconstruction of the speech signal and reduce its distortion, a synthesis window can be added during the subsequent process of restoring the frequency-domain speech spectrum to the time-domain speech signal. The specific implementation process is not elaborated in this embodiment. The definition of the Vorbis window can be found in the following formula (1):

[0121]

[0122] Where n refers to the index of the sampling point currently applied by the Vorbis window, N is the window length, and 0≤n≤N-1.

[0123] After acquiring H audio data segments, the user terminal can further perform time-frequency transformation on each audio data segment to obtain the corresponding audio data frame for each segment. In other words, the audio data segments in the time domain can be transformed into audio data frames in the frequency domain to obtain spectral frames that are easier to suppress noise. For ease of understanding and explanation, this application embodiment will use any one of the H audio data segments as an example to illustrate the specific process of time-frequency transformation. Assuming that H audio data segments include audio data segment i, where i is a positive integer less than or equal to H, the user terminal can first perform a Fourier transform (e.g., Fast Fourier Transform, FFT) on audio data segment i to obtain the DC component frequency point and 2S frequency points of audio data segment i in the frequency domain. That is, a total of (1+2S) frequency points can be obtained after the Fourier transform, where S is a positive integer. This application does not limit the number of frequency points. It can be understood that the number of sampling points of the audio signal frame corresponding to each audio data segment can be the same as or different from the number of frequency points corresponding to that audio data segment. In practical applications, the number of frequency points obtained after the Fourier transform can be set as needed. For example, in some embodiments, the number of sampling points corresponding to each audio signal frame is 320, and when performing time-frequency transformation, the number of frequency points corresponding to each audio data segment can be set to 512.

[0124] As can be seen from the properties of the Fourier transform, the (1+2S) frequency points obtained after the Fourier transform are all complex numbers, each corresponding to a frequency. The magnitude of this complex number can represent the amplitude characteristic of that frequency, and this amplitude characteristic has a specific proportional relationship with the amplitude of the corresponding audio signal. It should be noted that, except for the first complex number (i.e., the DC component frequency point), the remaining 2S complex numbers are conjugate symmetric about their center. Since the magnitudes (or amplitudes) of two conjugate symmetric complex numbers are the same, it is only necessary to select the spectrum of half of these 2S frequency points. For ease of distinction, in the embodiments of this application, the first S frequency points of these 2S frequency points can be determined as frequency points related to the first frequency point type, and correspondingly, the last S frequency points of these 2S frequency points can be determined as frequency points related to the second frequency point type. That is, these 2S frequency points can include S frequency points related to the first frequency point type and S frequency points related to the second frequency point type. It is understood that the S frequency points associated with the first frequency point type and the S frequency points associated with the second frequency point type are conjugate symmetric about their center. Subsequently, the user terminal can obtain the S frequency points associated with the first frequency point type from the aforementioned 2S frequency points, and can determine the audio data frame corresponding to audio data segment i based on the S frequency points associated with the first frequency point type and the DC component frequency point. Alternatively, optionally, due to the conjugate symmetry characteristic, the audio data frame corresponding to audio data segment i can also be determined based on the S frequency points associated with the second frequency point type and the DC component frequency point; this application embodiment does not limit this. It is understood that the audio data frame corresponding to audio data segment i is a spectral frame in the frequency domain. For example, in some embodiments, after time-frequency transformation, each audio data segment corresponds to 513 frequency points, including 1 DC component frequency point and 512 frequency points with conjugate symmetry. Then, the first half of the 512 frequency points (i.e., the frequency points associated with the first frequency point type) and the DC component frequency point can be taken to form the corresponding audio data frame.

[0125] For example, suppose that after performing a Fourier transform on audio data segment i, five frequency points are obtained (i.e., S = 2), including one DC component frequency point (a + bi), and frequency points (c + di), (e + fi), (c - di), and (e - fi). Among them, frequency points (c + di) and (c - di) are a pair of conjugate complex numbers, and frequency points (e + fi) and (e - fi) are also a pair of conjugate complex numbers. Therefore, frequency points (c + di) and (e + fi) can be used as frequency points related to the first frequency point type, and frequency points (c - di) and (e - fi) can be used as frequency points related to the second frequency point type. Furthermore, the audio data frame corresponding to audio data segment i can be determined based on the DC component frequency points (a + bi), (c + di), and (e + fi), or the audio data frame corresponding to audio data segment i can be determined based on the DC component frequency points (a + bi), (c - di), and (e - fi).

[0126] For better understanding, please refer again. Figure 4b ,like Figure 4b As shown, after performing a time-frequency transformation on audio data segment 1, audio data frame 1 can be obtained; after performing a time-frequency transformation on audio data segment 2, audio data frame 2 can be obtained; ...; after performing a time-frequency transformation on audio data segment H, audio data frame H can be obtained. It can be understood that the order of these H audio data frames in the frequency domain is consistent with the order of the H audio data segments in the time domain.

[0127] It is understood that after obtaining H audio data frames, the user terminal can determine the target audio data frame and the K historical audio data frames preceding the target audio data frame from these H audio data frames. Both the target audio data frame and the K historical audio data frames are spectrum frames. The target audio data frame can be any one of the H audio data frames to be processed, and each of the K historical audio data frames is a spectrum frame preceding the target audio data frame. K is a positive integer less than H, and the value of K is not limited in this embodiment. Please refer again. Figure 4b Assuming that audio data frame 4 is taken as the target audio data frame, the spectrum frames before audio data frame 4 are audio data frame 1, audio data frame 2 and audio data frame 3. For example, when K=2, audio data frame 2 and audio data frame 3, which are closest to audio data frame 4, can be taken as the required historical audio data frames.

[0128] It should be noted that when processing each audio data frame, it is necessary to obtain the K previous historical audio data frames. Specifically, if the number of previous historical audio data frames is less than the specified K, zero-padding can be used to bring the number of previous historical audio data frames to K. For example, combining the above... Figure 4b Assuming that audio data frame 1 is taken as the target audio data frame and K=2, it can be seen that there are no spectrum frames before audio data frame 1 after time-frequency transformation. Therefore, two spectrum frames with all zeros can be added in front of audio data frame 1 as the historical audio data frames before audio data frame 1.

[0129] Step S102: When N target cepstral coefficients of the target audio data frame are obtained, based on the N target cepstral coefficients, M first-order time derivatives and M second-order time derivatives associated with the target audio data frame are obtained.

[0130] It is understandable that after obtaining the target audio data frame through audio preprocessing, the user terminal can extract audio features from the target audio data frame. Specifically, it can obtain N cepstral coefficients that characterize the acoustic features of the target audio data frame. For ease of distinction, these N cepstral coefficients can be collectively referred to as target cepstral coefficients. It can also obtain M first-order time derivatives, M second-order time derivatives, and spectral dynamic features that characterize the time correlation between different speech signals, where N is a positive integer greater than 1, and M is a positive integer less than N. The following will elaborate on the process of obtaining the target cepstral coefficients, first-order time derivatives, and second-order time derivatives associated with the target audio data frame.

[0131] The specific process for obtaining the N target cepstral coefficients of the target audio data frame can be as follows: Assume the target audio data frame contains a total of S1 frequency points, including one DC component frequency point and S2 frequency points related to the frequency point type. S1 and S2 are both positive integers, and S1 = 1 + S2. Based on the description of the time-frequency transformation process in step S101 above, the S2 frequency points related to the frequency point type can be either S frequency points related to the first frequency point type or S frequency points related to the second frequency point type, where S2 = S. Based on this, the user terminal can map these S1 (e.g., 256+1) frequency points to N (e.g., 56) acoustic frequency bands, where S1 is greater than or equal to N. That is, the frequencies of the S1 frequency points can be divided into a coarser frequency scale (i.e., acoustic frequency bands in this embodiment), thereby reducing the complexity of subsequent calculations. Furthermore, cepstral processing can be performed on each acoustic frequency band separately to obtain the target cepstral coefficients corresponding to each acoustic frequency band. For ease of understanding, let's assume here that there are N acoustic frequency bands, including acoustic frequency band j, where j is a positive integer less than or equal to N. The specific process of cepstral processing for acoustic frequency band j can be as follows: First, the band energy of acoustic frequency band j can be obtained. In some embodiments, triangular filtering can be performed on the frequency point data to obtain the band energy of each corresponding acoustic frequency band. For example, a triangular filter (e.g., triangular filter j) associated with acoustic frequency band j can be obtained from a triangular filter bank containing N triangular filters. Then, each filter point in the triangular filter j can be applied to the corresponding frequency point in acoustic frequency band j to obtain the band energy of acoustic frequency band j. In some embodiments, when the number of acoustic frequency bands is 56, 56 triangular filters are also required. Furthermore, a logarithmic transformation can be performed on the band energy of acoustic band j to obtain its logarithmic band energy. Then, a Discrete Cosine Transform (DCT) can be performed on the logarithmic band energy of acoustic band j to obtain the target cepstral coefficients corresponding to acoustic band j. It can be understood that after cepstral processing of each acoustic band, N target cepstral coefficients of the target audio data frame can be obtained. The process of obtaining the cepstral coefficients of other audio data frames is the same as that of obtaining the target cepstral coefficients, and will not be elaborated here.

[0132] It should be noted that existing technologies that directly estimate the size of the frequency bin (which can be understood as the interval between samples in the frequency domain) through neural networks lead to extremely high computational complexity. To address this issue, this application does not directly process samples or the spectrum. Assuming that the spectral envelope of speech and noise is sufficiently flat, a coarser resolution than the frequency bin can be used, that is, the frequency of each frequency point is divided into a coarser frequency scale to reduce computational complexity. In this application, such a coarser frequency scale can be referred to as an acoustic frequency band. Different acoustic frequency bands can be used to characterize the nonlinear characteristics of human ear perception of sound. The acoustic frequency band in this application can be the Bark frequency scale, the Mel frequency scale, or other frequency scales, without limitation. For example, the Bark frequency scale is a frequency scale that matches human ear perception of sound. The Bark frequency scale is in Hz and maps frequencies to 24 psychoacoustic critical bands. The 25th critical band occupies a frequency range of approximately 16kHz to 20kHz, and the width of one critical band is equal to one Bark. Simply put, the Bark frequency scale converts physical frequencies to psychoacoustic frequencies. Combining the above steps, if the frequency bin is used directly, it is necessary to consider the (complex) spectral values ​​of S1 (e.g., 257) frequency points. Then the amount of data subsequently fed into the target mask estimation model will be very large. Therefore, this embodiment of the application utilizes the characteristics of the frequency band envelope to re-divide the S1 frequency points into N acoustic frequency bands, thereby achieving the purpose of reducing the amount of computation.

[0133] For example, in some embodiments, the Bark domain can be approximated using various approximation functions. Assuming a sampling rate of 16kHz, a window length of 512, a frame length of 20ms, and a frame shift of 12ms, an audio data frame after Fourier transform contains 257 frequency points. These 257 frequency points can then be divided into 56 acoustic frequency bands based on a set frequency band approximation function. The code for this division is as follows:

[0134] static const opus_int16 eband5ms[]={ / / eband20ms---56ok

[0135] 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,34,36,38,40,42,44,46,48,56,64,72,80,92,104,116,128,144,160,176,192,208,232,256

[0136] };

[0137] In other words, frequency point 0 (the first frequency point, representing the DC component) can be assigned to the first acoustic band, frequency point 1 (the second frequency point) to the second acoustic band, and so on, with frequencies 232 to 255 assigned to the 55th acoustic band and frequency point 256 assigned to the 56th acoustic band. Subsequently, cepstral transformation (i.e., logarithmic transformation of the band energy of each acoustic band followed by DCT) can be performed on each of these 56 acoustic bands, ultimately yielding 56 Bark-frequency cepstral coefficients (BFCCs).

[0138] It is understandable that, in addition to the aforementioned N target cepstral coefficients, the first and second time derivatives of these N target cepstral coefficients are also considered. The specific process by which a user terminal obtains M first-order time derivatives and M second-order time derivatives associated with a target audio data frame based on N target cepstral coefficients can be as follows: First, a difference operation is performed on the N target cepstral coefficients to obtain (N-1) difference operation values. Then, each of the (N-1) difference operation values ​​can be used as the first-order time derivative of the corresponding target cepstral coefficient. Subsequently, the M first-order time derivatives associated with the target audio data frame can be obtained from these (N-1) first-order time derivatives. Similarly, a second difference operation can be performed on the obtained (N-1) first-order time derivatives to obtain (N-2) difference operation values. Then, each of the (N-2) difference operation values ​​can be used as the second-order time derivative of the corresponding target cepstral coefficient. Subsequently, the M second-order time derivatives associated with the target audio data frame can be obtained from the (N-2) second-order time derivatives. In this embodiment, the value of M is not limited. For example, in some embodiments, M can be set to 6.

[0139] For ease of understanding, please refer to the following: Figure 5 , Figure 5 This is a schematic diagram illustrating a scenario of cepstral coefficient difference operation provided in an embodiment of this application. For example... Figure 5As shown, assume an audio data frame corresponds to 56 cepstral coefficients (e.g., BFCC), namely cepstral coefficient 1, cepstral coefficient 2, cepstral coefficient 3, ..., cepstral coefficient 54, cepstral coefficient 55, and cepstral coefficient 56. The first-order time derivative 1 is obtained by performing a difference operation on cepstral coefficient 1 and cepstral coefficient 2 (e.g., cepstral coefficient 2 - cepstral coefficient 1); the first-order time derivative 2 is obtained by performing a difference operation on cepstral coefficient 2 and cepstral coefficient 3; ...; the first-order time derivative 54 is obtained by performing a difference operation on cepstral coefficient 54 and cepstral coefficient 55; and the first-order time derivative 55 is obtained by performing a difference operation on cepstral coefficient 55 and cepstral coefficient 56. Then, a second difference operation can be performed on the obtained first-order time derivatives 1 to 55. For example, performing a second difference operation on first-order time derivatives 1 and 2 (e.g., first-order time derivative 2 minus first-order time derivative 1) yields second-order time derivative 1; ...; performing a second difference operation on first-order time derivatives 54 and 55 yields second-order time derivative 54. In some embodiments, M can be set to 6, taking the first 6 first-order time derivatives and the first 6 second-order time derivatives, i.e., taking first-order time derivatives 1 to 6 and second-order time derivatives 1 to 6.

[0140] Step S103: Obtain N historical cepstral coefficients corresponding to each historical audio data frame, and determine the spectral dynamic features associated with the target audio data frame based on the obtained K*N historical cepstral coefficients;

[0141] In addition to the target cepstral coefficients, first-order time derivatives, and second-order time derivatives mentioned in step S102 above, the stationarity measure of past audio data frames relative to the current audio data frame, i.e., the spectral dynamic characteristics, can also be considered. These characteristics can be obtained based on the frequency band difference values ​​corresponding to the past K historical audio data frames. In some embodiments, since the acquisition of the N historical cepstral coefficients corresponding to each historical audio data frame can be obtained when processing the historical audio data frame (i.e., when the audio data frame is used as the target audio data frame, the specific process can be referred to in step S102 above for the acquisition of the N target cepstral coefficients), the user terminal can use its cache (e.g., a circular buffer structure) to store the N historical cepstral coefficients corresponding to the latest K historical audio data frames before the current target audio data frame. When each target audio data frame is updated to its next audio data frame, the historical cepstral coefficients in the cache should also be updated accordingly.

[0142] Specifically, among the K historical audio data frames preceding the target audio data frame, any two adjacent historical audio data frames can be obtained as the first historical audio data frame and the second historical audio data frame, where the second historical audio data frame is the spectrum frame obtained after the first historical audio data frame. Then, N historical cepstral coefficients corresponding to the first historical audio data frame and N historical cepstral coefficients corresponding to the second historical audio data frame can be obtained from the cache associated with the target audio data frame (e.g., the local cache of the user terminal). In this embodiment, for ease of distinction, the N historical cepstral coefficients corresponding to the first historical audio data frame can be used as the first historical cepstral coefficients, and the N historical cepstral coefficients corresponding to the second historical audio data frame can be used as the second historical cepstral coefficients. Further, the frequency band difference value between the first and second historical cepstral coefficients can be used as the inter-frame difference value between the first and second historical audio data frames. Specifically, this can be achieved by obtaining historical cepstral coefficient L from the N historical cepstral coefficients included in the first historical cepstral coefficients. p Simultaneously, the historical cepstral coefficient L can be obtained from the N historical cepstral coefficients contained in the second historical cepstral coefficient. q Where p and q are both positive integers less than or equal to N, and p = q. Furthermore, the historical cepstral coefficients L can be obtained. p Compared with historical cepstral coefficients L q The coefficient differences between them (e.g., historical cepstral coefficients L) p -Historical cepstral coefficients L q Then, the frequency band difference between the first and second historical cepstral coefficients can be determined based on this coefficient difference value. It can be understood that when each historical cepstral coefficient in the first historical cepstral coefficient is used as a historical cepstral coefficient L... p And each historical cepstral coefficient in the second historical cepstral coefficients is used as a historical cepstral coefficient L. q At this point, the frequency band difference value between the first historical cepstral coefficient and the second historical cepstral coefficient can be obtained, where the frequency band difference value includes N coefficient difference values. Subsequently, this frequency band difference value can be used as the inter-frame difference value between the first historical audio data frame and the second historical audio data frame.

[0143] It is understood that when each adjacent historical audio data frame in the aforementioned K historical audio data frames is used as the first historical audio data frame and the second historical audio data frame, the inter-frame difference values ​​among the K historical audio data frames can be obtained. Subsequently, the spectral dynamic characteristics associated with the target audio data frame can be determined based on the inter-frame difference values ​​among the K historical audio data frames. Optionally, in some embodiments, the sum of the difference values ​​of the N coefficient difference values ​​contained in all inter-frame difference values ​​can be obtained, and then the average of the sum of difference values ​​(e.g., the sum of difference values / K) can be taken to obtain the corresponding spectral dynamic characteristics.

[0144] For ease of understanding, please refer to the following: Figure 6 , Figure 6 This is a schematic diagram illustrating a scenario for obtaining inter-frame difference values ​​provided in an embodiment of this application. Figure 6As shown, assume there are currently 8 historical audio data frames (i.e., K=8), namely historical audio data frame 1, historical audio data frame 2, ..., historical audio data frame 7, and historical audio data frame 8. Each historical audio data frame corresponds to 56 historical cepstral coefficients (i.e., N=56). For example, historical audio data frame 1 corresponds to cepstral coefficients A1 to A56, historical audio data frame 2 corresponds to cepstral coefficients B1 to B56, ..., historical audio data frame 7 corresponds to cepstral coefficients C1 to C56, and historical audio data frame 8 corresponds to cepstral coefficients D1 to D56. When historical audio data frame 1 is used as the first historical audio data frame and historical audio data frame 2 is used as the second historical audio data frame, the first historical cepstral coefficients include cepstral coefficients A1 to A56, and the second historical cepstral coefficients include cepstral coefficients B1 to B56. At this time, the coefficient difference value AB1 between cepstral coefficients A1 and B1 (e.g., cepstral coefficient A1 - cepstral coefficient B1), the coefficient difference value AB2 between cepstral coefficients A2 and B2, ..., the coefficient difference value AB55 between cepstral coefficients A55 and B55, and the coefficient difference value AB56 between cepstral coefficients A56 and B56 can be obtained. Then, the frequency band difference value containing 56 coefficient difference values ​​(i.e., coefficient difference value AB1 to coefficient difference value AB56) can be used as the inter-frame difference value 1 between historical audio data frame 1 and historical audio data frame 2. Similarly, when historical audio data frame 7 is used as the first historical audio data frame and historical audio data frame 8 is used as the second historical audio data frame, the frequency band difference value containing 56 coefficient difference values ​​(i.e., coefficient difference values ​​CD1 to CD56) can be obtained as the inter-frame difference value 7 between historical audio data frame 7 and historical audio data frame 8. Subsequently, the spectral dynamic characteristics associated with the current audio data frame can be determined based on the inter-frame difference values ​​(i.e., inter-frame difference values ​​1 to 7) between the above 8 historical audio data frames. For example, all 56 coefficient difference values ​​contained in each inter-frame difference value can be added together to obtain the corresponding sum of difference values, and then the average of the sum of difference values ​​can be taken to obtain the corresponding spectral dynamic characteristics, i.e., spectral dynamic characteristics = (coefficient difference value AB1 + ... + coefficient difference value AB56 + ... + coefficient difference value CD1 + ... + coefficient difference value CD56) ÷ 8.

[0145] It should be noted that if the number of historical audio data frames preceding the current audio data frame does not meet the specified K (e.g., 8) requirements, the embodiments of this application can obtain K historical audio data frames through zero-padding. The historical audio data frames obtained through zero-padding are all-zero spectrum frames, and correspondingly, the N cepstral coefficients corresponding to such all-zero spectrum frames can also be set to zero values.

[0146] Step S104: Input N target cepstral coefficients, M first-order time derivatives, M second-order time derivatives, and spectral dynamic features into the target mask estimation model, and output the target mask corresponding to the target audio data frame from the target mask estimation model.

[0147] It is understandable that intelligent speech enhancement technology based on machine learning / deep learning is inspired by the time-frequency (TF) masking concept in Computational Auditory Scene Analysis (CASA). For supervised speech enhancement, the choice of training objective is crucial for both learning and generalization. Typically, the training objective is defined on the TF representation of the speech signal, such as the spectrum calculated from the short-time Fourier transform. These training objectives are mainly divided into two categories: one is masking-based objectives, such as IRM, which describes the time-frequency relationship between clean speech and background noise; the other is mapping-based objectives, such as logarithmic power spectrum, which corresponds to the spectral representation of clean speech. This application's embodiments adopt the former approach, utilizing the nonlinear fitting capability of neural networks to estimate the mask from the input features. Then, the mask is multiplied by the spectrum of the noisy speech signal (i.e., the original audio data in this application's embodiments) to reconstruct the time-domain waveform for enhancement.

[0148] Specifically, for ease of differentiation, the user terminal can use the N target cepstral coefficients, M first-order time derivatives, M second-order time derivatives, and spectral dynamic features obtained in the above steps—a total of (N+2M+1) features—as the target audio features of the target audio data frame. These target audio features can then be input into the target mask estimation model for mask estimation. For example, in some embodiments, the number of target cepstral coefficients is 56, and the number of first-order and second-order time derivatives is 6 each. Therefore, the size of the target audio features input into the target mask estimation model is 56 + 6*2 + 1 = 69.

[0149] It is understandable that target mask estimation models based on neural networks have extremely strong nonlinear fitting capabilities. Therefore, an initial mask estimation model can be trained to learn how to calculate the mask from noisy frequency features. The specific process of model training will be discussed later. Figure 8 The corresponding implementation examples.

[0150] In one implementation, the target mask estimation model may include a mask estimation network layer and a mask output layer. First, the obtained target audio features can be input into the mask estimation network layer. This layer performs mask estimation on the input target audio features to obtain the hidden features corresponding to the target audio features. Further, these hidden features can be input into the mask output layer, where they are combined to obtain the target mask corresponding to the target audio data frame. The length of the target mask is N (the same as the number of acoustic frequency bands defined above). In this embodiment, the target mask can be used to suppress noise data in the original audio data to obtain enhanced audio data corresponding to the original audio data.

[0151] In this embodiment, the mask can also be called gain or band gain, and may include, but is not limited to, Ideal Ratio Mask (IRM), Ideal Binary Mask (IBM), Optimal Ratio Mask (ORM), etc. The type of target mask will not be limited here. IRM, based on the assumption that the target speech and background noise are orthogonal (i.e., uncorrelated), directly characterizes the ratio of clean speech energy to noisy speech energy within a time-frequency unit, and is a widely used masking method. The IRM value can be between 0 and 1; a larger value indicates a higher proportion of target speech within the time-frequency unit. Since the target speech is sparsely distributed in the time-frequency domain, the energy difference between the target speech and background noise is usually quite large for a specific time-frequency unit. Therefore, the signal-to-noise ratio (SNR) in most time-frequency units is either extremely high or extremely low. IBM is a simplified description of this reality, discretizing the SNR of continuous time-frequency units into two states: 1 and 0. Within a time-frequency unit: if the target speech is dominant (high SNR), it is marked as 1; conversely, if background noise is dominant (low SNR), it is marked as 0. Finally, multiplying IBM by the noisy speech effectively sets the time-frequency units with low SNR to zero, thereby eliminating background noise. Therefore, IBM can be considered a binary version of IRM. Meanwhile, ORM is defined by minimizing the mean square error between clean speech and the estimated target speech, which is very similar to IRM.

[0152] Optionally, in some embodiments, the aforementioned mask estimation network layer may include a first mask estimation network layer, a second mask estimation network layer, and a third mask estimation network layer with skip connections. It is understood that the skip connections between the first, second, and third mask estimation network layers can prevent network overfitting. Therefore, the specific process of mask estimation of the input target audio features through this mask estimation network layer can be as follows: the target audio features are input to the first mask estimation network layer, and the first mask estimation network layer outputs a first intermediate feature; then, based on the skip connections between the first and second mask estimation network layers, the first intermediate feature is estimated. The first intermediate feature is obtained by concatenating the intermediate features and the target audio features. This second intermediate feature is then input into the second mask estimation network layer, which outputs the third intermediate feature. Further, based on the skip connections between the first and third mask estimation network layers, and between the second and third mask estimation network layers, the third intermediate feature, the target audio features, and the first intermediate feature are concatenated to obtain the fourth intermediate feature. This fourth intermediate feature is then input into the third mask estimation network layer, which outputs the hidden features corresponding to the target audio features. Optionally, the target mask estimation model can employ more or fewer mask estimation network layers; the specific number of mask estimation network layers is not limited in this embodiment.

[0153] The first, second, and third mask estimation network layers in the mask estimation network layer can adopt network structures such as Gated Recurrent Units (GRU) or Long Short-Term Memory (LSTM), while the mask output layer can adopt a fully connected layer or other network structures. The specific structures of the mask estimation network layer and the mask output layer are not limited in this application embodiment.

[0154] GRU is a gating mechanism in recurrent neural networks. It is similar to LSTM with a forget gate. GRU includes update gate and reset gate, but it has fewer output gates than LSTM. It has fewer parameters than LSTM. Therefore, if GRU is used to design the mask estimation network layer, a lightweight mask estimation model can be obtained.

[0155] For ease of understanding, please refer to the following: Figure 7 , Figure 7 This is a schematic diagram of the network structure of a mask estimation model provided in an embodiment of this application. Figure 7As shown, after obtaining the corresponding audio features (e.g., audio features of size 69), they can be input into the mask estimation model 70 (i.e., the target mask estimation model). This model can include a gated recurrent network layer 1 (i.e., the first mask estimation network layer), a gated recurrent network layer 2 (i.e., the second mask estimation network layer), a gated recurrent network layer 3 (i.e., the third mask estimation network layer), and a fully connected layer (i.e., the mask output layer). It uses three simple GRU neural networks to model the audio features, and the last fully connected layer is used for output gain (i.e., the mask). Optionally, in this embodiment, the number of features corresponding to each audio data frame of the input model can be 69, and the number of nodes (also called neurons or perceptrons) in the three gated recurrent network layers can be 64, 96, and 96 respectively. Correspondingly, the feature dimension of the first intermediate feature output by gated recurrent network layer 1 is 64, the feature dimension of the third intermediate feature output by gated recurrent network layer 2 is 96, and the feature dimension of the hidden feature output by gated recurrent network layer 3 is 96. In addition, the number of nodes in the fully connected layer can be 56, so the dimension of the final output mask is 56 (i.e., 56 mask values ​​are output). Each network layer can use a suitable activation function. For example, gated recurrent network layer 1 can use the ReLU function (Rectified Linear Unit), gated recurrent network layer 2 can use the ReLU function, and gated recurrent network layer 3 can use the tanh function (hyperbolic tangent function). If the mask uses an IRM, the fully connected layer can use the sigmoid function to ensure that the output mask ranges from (0,1). Each of the above network layers can also use other functions as activation functions, and this application does not limit this.

[0156] It should be noted that the target mask estimation model used in the embodiments of this application is a lightweight neural network. The three mask estimation network layers can achieve good mask estimation results, and the number of network parameters is small and the network complexity is low, thereby reducing computation time and CPU consumption.

[0157] It's understandable that, assuming noise and speech are uncorrelated, the energy of noisy speech will necessarily be greater than that of clean speech. Dividing the frequency domain into N acoustic bands and calculating the energy, for each acoustic band, the less noise it contains, the cleaner the speech, and the greater the band gain. Based on this, for noisy speech, multiplying each acoustic band by a gain means that when the noise in that band is high, a smaller gain can be applied, and vice versa. This can enhance the speech and suppress noise.

[0158] In this embodiment, after obtaining the target mask (i.e., frequency band gain) through the above steps, the user terminal can use the target mask for noise suppression. Specifically, when the length of the target mask (i.e., N) is less than the length of the target audio data frame (i.e., S1), the target mask needs to be interpolated to obtain a corresponding interpolated mask. The length of the interpolated mask obtained at this time is the same as the length of the target audio data frame (e.g., 257 frequency points). Further, the interpolated mask can be multiplied by the target audio data frame, that is, each mask value in the interpolated mask is applied to each frequency point obtained by Fourier transform in the target audio data frame. Then, the multiplication result can be subjected to inverse Fourier transform to obtain the target audio data after noise suppression of the target audio data frame, i.e., restored to the enhanced time-domain speech signal. It can be understood that the process of noise suppression for other audio data is similar to the process of noise suppression for the target audio data frame, and will not be elaborated here. Finally, after noise suppression is performed on each audio data frame associated with the original audio data, the enhanced audio data corresponding to the original audio data can be obtained. The noise content in the enhanced audio data is very low, and there is no false cancellation of the voice data of the business object. Therefore, the enhanced audio data has extremely high voice fidelity.

[0159] As described above, when performing speech enhancement on the original audio data, this embodiment of the application can comprehensively consider multiple audio features, including target cepstral coefficients, first-order time derivatives, second-order time derivatives, and spectral dynamic features. This allows for a more accurate description of the time-frequency relationship between the speech data of the business object and the background noise data, resulting in a more accurate target mask without suppressing the desired speech along with the background noise. Therefore, applying each set of output masks to the corresponding audio data frames can effectively suppress noise data in the audio data and improve speech fidelity. Especially in real-time audio and video communication scenarios (e.g., real-time audio and video conferencing scenarios), it can provide users with high-quality and high-definition speech, enhancing the user experience. Furthermore, this embodiment of the application first uses digital signal processing technology to extract corresponding audio features from the noisy frequency data, and then inputs the extracted audio features into a lightweight neural network model (i.e., the target mask estimation model) for rapid mask estimation. Therefore, this embodiment of the application requires lower network complexity, thereby reducing computational complexity and CPU (Central Processing Unit) consumption, and thus improving audio data processing efficiency.

[0160] Please see Figure 8 , Figure 8This is a flowchart illustrating an audio data processing method provided in an embodiment of this application. It is understood that the method provided in this embodiment can be executed by a computer device, which includes, but is not limited to, a user terminal or a service server. For ease of understanding, this embodiment uses a user terminal as an example to illustrate the specific process of training an initial mask estimation model on that user terminal. Figure 8 As shown, the method may include at least the following steps S201-S205:

[0161] Step S201: Obtain the target sample audio data frame and K historical sample audio data associated with the sample audio data, and obtain the sample mask corresponding to the target sample audio data frame.

[0162] It is understandable that the user terminal can obtain sample audio data from an audio database containing massive amounts of audio data. This sample audio data can be noisy speech signals (e.g., audio data carrying Babble Noise and sample object speech data). Subsequently, by performing frame-segmentation, windowing preprocessing, time-frequency transformation, and other operations on the sample audio data, a target sample audio data frame and K historical sample audio data associated with that sample audio data can be obtained. Both the target sample audio data frame and the K historical sample audio data frames are spectral frames, and each of the K historical sample audio data frames is a spectral frame preceding the target sample audio data frame. K is a positive integer. The specific process can be found above. Figure 3 Step S101 in the corresponding embodiment will not be described again here. Furthermore, in order to calculate the loss function subsequently, the user terminal can also obtain the sample mask corresponding to the target sample audio data frame.

[0163] Step S202: When the N target sample cepstral coefficients of the target sample audio data frame are obtained, the first-order time derivatives and the second-order time derivatives of the M samples associated with the target sample audio data frame are obtained based on the N target sample cepstral coefficients.

[0164] It can be understood that the user terminal can map multiple frequency points contained in the target sample audio data frame to N divided sample acoustic frequency bands, and obtain the target sample cepstral coefficients corresponding to each sample acoustic frequency band by performing cepstral processing on each sample acoustic frequency band. Then, based on the obtained N target sample cepstral coefficients, the M sample first-order time derivatives and M sample second-order time derivatives associated with the target sample audio data frame can be obtained, where N is a positive integer greater than 1 and M is a positive integer less than N. The specific implementation of this step can be found above. Figure 3 Step S102 in the corresponding embodiment will not be described again here.

[0165] Step S203: Obtain N historical sample cepstral coefficients corresponding to each historical sample audio data frame, and determine the dynamic characteristics of the sample spectrum associated with the target sample audio data frame based on the obtained K*N historical sample cepstral coefficients.

[0166] It can be understood that the user terminal can obtain N historical sample cepstral coefficients corresponding to any two adjacent historical sample audio data frames from K historical sample audio data frames. Then, based on the obtained two sets of historical sample cepstral coefficients, the inter-frame difference value between these two adjacent historical sample audio data frames can be determined. Finally, based on the inter-frame difference value between every two adjacent historical sample audio data frames from the K historical sample audio data frames, the dynamic characteristics of the sample spectrum associated with the target sample audio data frame can be determined. The specific implementation method of this step can be found above. Figure 3 Step S103 in the corresponding embodiment will not be described again here.

[0167] Step S204: Input the cepstral coefficients of N target samples, the first-order time derivatives of M samples, the second-order time derivatives of M samples, and the dynamic features of the sample spectrum into the initial mask estimation model, and output the predicted mask corresponding to the audio data frame of the target sample from the initial mask estimation model.

[0168] It can be understood that the user terminal can use the obtained N target sample cepstral coefficients, M sample first-order time derivatives, M sample second-order time derivatives, and sample spectral dynamic features as sample audio features of the target sample audio data frame. Subsequently, these sample audio features can be input into the initial mask estimation model, which outputs the predicted mask corresponding to the target sample audio data frame. An exemplary network structure of the initial mask estimation model can be found above. Figure 7 The corresponding implementation method. For a detailed explanation of this step, please refer to the above. Figure 3 Step S104 in the corresponding embodiment will not be described again here.

[0169] Step S205: Iteratively train the initial mask estimation model based on the predicted mask and the sample mask to obtain the target mask estimation model used to output the target mask corresponding to the target audio data frame associated with the original audio data.

[0170] It is understandable that the user terminal can generate a loss function based on the predicted mask and the sample mask, and then use this loss function to correct the model parameters in the initial mask estimation model. Through multiple iterations of training, a target mask estimation model can finally be obtained, which is used to output the target audio data frame associated with the original audio data. The target mask can be used to suppress noise data in the original audio data to obtain enhanced audio data corresponding to the original audio data.

[0171] In one alternative implementation, the loss function used for model training can be Huber loss (a parameterized loss function used for regression problems), whose formula is described as follows:

[0172]

[0173] Among them, g true This refers to the sample mask, g pred This refers to the prediction mask, and the hyperparameter d in the loss function can be set to 0.1.

[0174] In addition, other forms of loss functions may be used, and this application does not limit the specific loss function used.

[0175] For ease of understanding, please refer to the following: Figure 9 , Figure 9 This is a schematic diagram of a model training process provided in an embodiment of this application. For example... Figure 9As shown, a neural network model can be designed to model the extracted audio features to obtain the corresponding mask, which is used to suppress background noise in the audio data. Specifically, after obtaining the training speech (i.e., sample audio data), speech preprocessing (i.e., audio preprocessing) can be performed on it to obtain multiple related sample audio data frames. Then, audio features can be extracted from each sample audio data frame in sequence to obtain the sample audio features corresponding to each sample audio data frame. Subsequently, these sample audio features can be input into the initial mask estimation model for model training to obtain the intermediate mask estimation model. Understandably, at this point, it's necessary to further verify the generalization performance of the obtained intermediate mask estimation model. Similarly, the acquired test speech (also known as test audio data, which can be obtained together with the sample audio data) can be preprocessed to obtain multiple test audio data frames. Then, audio features can be extracted from each test audio data frame sequentially to obtain the corresponding test audio features. These test audio features can then be input into the intermediate mask estimation model, which outputs a corresponding mask. This mask can be applied to the corresponding test audio data frame to obtain the spectrum after suppressing background noise. Subsequently, an inverse Fourier transform can be performed on the obtained spectrum to reconstruct the time-domain speech signal, thereby achieving speech enhancement. Understandably, when the obtained test results meet expectations, this intermediate mask estimation model can be used as a target mask estimation model that can be directly used subsequently.

[0176] Furthermore, please also refer to Figure 10 , Figure 10 This is a schematic diagram illustrating a noise reduction effect provided in an embodiment of this application. For example... Figure 10 As shown, the spectrum corresponding to a speech segment containing Babble Noise is as follows: Figure 10 The spectrum of the noisy speech in the image is 10A. After noise suppression of the noisy speech using the method and target mask estimation model provided in this application, the spectrum of the enhanced speech is obtained as follows: Figure 10 The spectrum 10B is shown in the image. A comparison of the two spectra reveals that the method provided in this application can effectively suppress background noise such as Babble Noise while preserving relatively complete speech.

[0177] As described above, this embodiment of the application, by training an initial mask estimation model, can obtain a target mask estimation model for outputting the mask corresponding to the audio data frame. Since this model is a lightweight neural network model, it can reduce the computational complexity in the speech enhancement process, ensure the size of the installation package in the deployment scenario, and reduce CPU consumption. In addition, the trained target mask estimation model can automatically and quickly output the estimated mask, thereby improving the efficiency of speech enhancement processing of audio data.

[0178] Please see Figure 11 This is a schematic diagram of the structure of an audio data processing device provided in an embodiment of this application. The audio data processing device 1 can be a computer program (including program code) running on a computer device; for example, the audio data processing device 1 is an application software. This device can be used to execute corresponding steps in the audio data processing method provided in the embodiments of this application. Figure 11 As shown, the audio data processing device 1 may include: a first acquisition module 11, a second acquisition module 12, a third acquisition module 13, a mask estimation module 14, a frequency band mapping module 15, a cepstral processing module 16, and a noise suppression module 17.

[0179] The first acquisition module 11 is used to acquire a target audio data frame and K historical audio data frames associated with the original audio data; the target audio data frame and the K historical audio data frames are all spectrum frames, and each of the K historical audio data frames is a spectrum frame preceding the target audio data frame, where K is a positive integer.

[0180] The first acquisition module 11 may include: an audio preprocessing unit 111, a time-frequency conversion unit 112, and a data frame determination unit 113;

[0181] The audio preprocessing unit 111 is used to perform frame-by-frame windowing preprocessing on the original audio data to obtain H audio data segments; H is a positive integer greater than 1.

[0182] The time-frequency conversion unit 112 is used to perform time-frequency conversion on each audio data segment to obtain the audio data frame corresponding to each audio data segment.

[0183] In one implementation, the H audio data segments include audio data segment i, where i is a positive integer less than or equal to H;

[0184] The time-frequency transformation unit 112 is specifically used to perform a Fourier transform on the audio data segment i to obtain the DC component frequency point and 2S frequency points of the audio data segment i in the frequency domain; the 2S frequency points include S frequency points related to the first frequency point type and S frequency points related to the second frequency point type; S is a positive integer; the S frequency points related to the first frequency point type are obtained from the 2S frequency points, and the audio data frame corresponding to the audio data segment i is determined based on the S frequency points related to the first frequency point type and the DC component frequency points;

[0185] The data frame determination unit 113 is used to determine the target audio data frame and the K historical audio data frames preceding the target audio data frame from H audio data frames; K is less than H.

[0186] The specific functional implementation methods of the audio preprocessing unit 111, the time-frequency conversion unit 112, and the data frame determination unit 113 can be found in the above description. Figure 3 Step S101 in the corresponding embodiment will not be described again here.

[0187] The second acquisition module 12 is used to acquire, based on the N target cepstral coefficients of the target audio data frame, M first-order time derivatives and M second-order time derivatives associated with the target audio data frame; N is a positive integer greater than 1 and M is a positive integer less than N.

[0188] The second acquisition module 12 may include: a first differential unit 121 and a second differential unit 122;

[0189] The first differential unit 121 is used to perform differential operations on N target cepstral coefficients to obtain (N-1) differential operation values. Each differential operation value in the (N-1) differential operation values ​​is used as the first time derivative of the corresponding target cepstral coefficient. M first time derivatives associated with the target audio data frame are obtained from the (N-1) first time derivatives.

[0190] The second difference unit 122 is used to perform a second difference operation on (N-1) first-order time derivatives to obtain (N-2) difference operation values. Each difference operation value in the (N-2) difference operation values ​​is used as the second-order time derivative of the corresponding target cepstral coefficient. M second-order time derivatives associated with the target audio data frame are obtained from the (N-2) second-order time derivatives.

[0191] The specific functional implementation methods of the first differential unit 121 and the second differential unit 122 can be found in the above description. Figure 3 Step S102 in the corresponding embodiment will not be described again here.

[0192] The third acquisition module 13 is used to acquire N historical cepstral coefficients corresponding to each historical audio data frame, and to determine the spectral dynamic features associated with the target audio data frame based on the acquired K*N historical cepstral coefficients.

[0193] The third acquisition module 13 may include: a data frame acquisition unit 131, a coefficient acquisition unit 132, a difference determination unit 133, and a feature determination unit 134.

[0194] The data frame acquisition unit 131 is used to acquire any two adjacent historical audio data frames from K historical audio data frames as the first historical audio data frame and the second historical audio data frame; the second historical audio data frame is a spectrum frame obtained after the first historical audio data frame.

[0195] The coefficient acquisition unit 132 is used to acquire N historical cepstral coefficients corresponding to the first historical audio data frame from the cache associated with the target audio data frame, and acquire N historical cepstral coefficients corresponding to the second historical audio data frame; the acquired N historical cepstral coefficients corresponding to the first historical audio data frame are used as the first historical cepstral coefficients, and the acquired N historical cepstral coefficients corresponding to the second historical audio data frame are used as the second historical cepstral coefficients.

[0196] The difference determination unit 133 is used to take the frequency band difference value between the first historical cepstral coefficient and the second historical cepstral coefficient as the inter-frame difference value between the first historical audio data frame and the second historical audio data frame.

[0197] The difference determination unit 133 may include: a coefficient difference acquisition subunit 1331 and a difference value determination subunit 1332;

[0198] The coefficient difference acquisition subunit 1331 is used to acquire the historical cepstral coefficient Lp from the N historical cepstral coefficients contained in the first historical cepstral coefficient, and to acquire the historical cepstral coefficient Lq from the N historical cepstral coefficients contained in the second historical cepstral coefficient; p and q are both positive integers less than or equal to N, and p = q; and to acquire the coefficient difference value between the historical cepstral coefficient Lp and the historical cepstral coefficient Lq.

[0199] The difference value determination subunit 1332 is used to determine the frequency band difference value between the first historical cepstral coefficient and the second historical cepstral coefficient based on the coefficient difference value, and to use the frequency band difference value as the inter-frame difference value between the first historical audio data frame and the second historical audio data frame.

[0200] The specific functional implementation methods of the coefficient difference acquisition subunit 1331 and the difference value determination subunit 1332 can be found above. Figure 3 Step S103 in the corresponding embodiment will not be described again here.

[0201] The feature determination unit 134 is used to obtain the inter-frame difference value between the K historical audio data frames when each adjacent historical audio data frame in the K historical audio data frames is used as the first historical audio data frame and the second historical audio data frame, and to determine the spectral dynamic features associated with the target audio data frame based on the inter-frame difference value between the K historical audio data frames.

[0202] The specific functional implementation methods of the data frame acquisition unit 131, coefficient acquisition unit 132, difference determination unit 133, and feature determination unit 134 can be found above. Figure 3 Step S103 in the corresponding embodiment will not be described again here.

[0203] The mask estimation module 14 is used to input N target cepstral coefficients, M first-order time derivatives, M second-order time derivatives, and spectral dynamic features into the target mask estimation model, and output the target mask corresponding to the target audio data frame from the target mask estimation model; the target mask is used to suppress noise data in the original audio data to obtain the enhanced audio data corresponding to the original audio data;

[0204] In one embodiment, the target mask estimation model includes a mask estimation network layer and a mask output layer; the mask estimation module 14 may include: a mask estimation unit 141 and a mask output unit 142;

[0205] The mask estimation unit 141 is used to take N target cepstral coefficients, M first-order time derivatives, M second-order time derivatives and spectral dynamic features as target audio features of the target audio data frame, input the target audio features into the mask estimation network layer, and perform mask estimation on the target audio features through the mask estimation network layer to obtain the hidden features corresponding to the target audio features.

[0206] In one embodiment, the mask estimation network layer includes a first mask estimation network layer, a second mask estimation network layer, and a third mask estimation network layer with skip connections; the mask estimation unit 141 may include: a first estimation subunit 1411, a second estimation subunit 1412, and a third estimation subunit 1413.

[0207] The first estimation subunit 1411 is used to input the target audio features into the first mask estimation network layer and output the first intermediate features through the first mask estimation network layer;

[0208] The second estimation subunit 1412 is used to perform feature concatenation on the first intermediate feature and the target audio feature according to the skip connection between the first mask estimation network layer and the second mask estimation network layer to obtain the second intermediate feature, input the second intermediate feature into the second mask estimation network layer, and output the third intermediate feature through the second mask estimation network layer.

[0209] The third estimation subunit 1413 is used to perform feature concatenation on the third intermediate feature, the target audio feature, and the first intermediate feature based on the skip connections between the first mask estimation network layer and the third mask estimation network layer and the skip connections between the second mask estimation network layer and the third mask estimation network layer, to obtain the fourth intermediate feature, input the fourth intermediate feature into the third mask estimation network layer, and output the hidden feature corresponding to the target audio feature through the third mask estimation network layer.

[0210] The specific functional implementation of the first estimation subunit 1411, the second estimation subunit 1412, and the third estimation subunit 1413 can be found above. Figure 3Step S104 in the corresponding embodiment will not be described again here.

[0211] The mask output unit 142 is used to input the hidden features into the mask output layer, and to merge the hidden features through the mask output layer to obtain the target mask corresponding to the target audio data frame.

[0212] The specific functional implementation of the mask estimation unit 141 and the mask output unit 142 can be found in the above description. Figure 3 Step S104 in the corresponding embodiment will not be described again here.

[0213] In one implementation, the target audio data frame contains S1 frequency points, including a DC component frequency point and S2 frequency points related to the frequency point type, where S1 and S2 are both positive integers.

[0214] The frequency band mapping module 15 is used to map S1 frequency points to N acoustic frequency bands; S1 is greater than or equal to N.

[0215] The cepstral processing module 16 is used to perform cepstral processing on each acoustic frequency band to obtain the target cepstral coefficients corresponding to each acoustic frequency band.

[0216] In one implementation, the N acoustic frequency bands include acoustic frequency band j, where j is a positive integer less than or equal to N;

[0217] The cepstral processing module 16 may include: an energy acquisition unit 161 and a cosine transform unit 162;

[0218] The energy acquisition unit 161 is used to acquire the frequency band energy of the acoustic frequency band j, and to perform a logarithmic transformation on the frequency band energy of the acoustic frequency band j to obtain the logarithmic frequency band energy of the acoustic frequency band j.

[0219] The cosine transform unit 162 is used to perform discrete cosine transform on the logarithmic frequency band energy of acoustic frequency band j to obtain the target cepstral coefficients corresponding to acoustic frequency band j.

[0220] The specific functional implementation methods of the energy harvesting unit 161 and the cosine transform unit 162 can be found in the above description. Figure 3 Step S102 in the corresponding embodiment will not be described again here.

[0221] The noise suppression module 17 is used to interpolate the target mask to obtain an interpolated mask; the length of the interpolated mask is the same as the length of the target audio data frame; the interpolated mask is multiplied by the target audio data frame, and the multiplication result is subjected to an inverse Fourier transform to obtain the target audio data after noise suppression of the target audio data frame; after noise suppression is performed on each audio data frame associated with the original audio data, the enhanced audio data corresponding to the original audio data is obtained.

[0222] The specific functional implementation methods of the first acquisition module 11, the second acquisition module 12, the third acquisition module 13, the mask estimation module 14, the frequency band mapping module 15, the cepstral processing module 16, and the noise suppression module 17 can be found above. Figure 3 Steps S101-S104 in the corresponding embodiments will not be described again here. Furthermore, the beneficial effects of using the same method will also not be described again.

[0223] Please see Figure 12 This is a schematic diagram of the structure of an audio data processing device provided in an embodiment of this application. The audio data processing device 2 can be a computer program (including program code) running on a computer device; for example, the audio data processing device 2 is an application software. This device can be used to execute corresponding steps in the audio data processing method provided in the embodiments of this application. Figure 12 As shown, the audio data processing device 2 may include: a first acquisition module 21, a second acquisition module 22, a third acquisition module 23, a mask prediction module 24, and a model training module 25;

[0224] The first acquisition module 21 is used to acquire a target sample audio data frame and K historical sample audio data associated with the sample audio data, and to acquire the sample mask corresponding to the target sample audio data frame; the target sample audio data frame and the K historical sample audio data frames are all spectrum frames, and each of the K historical sample audio data frames is a spectrum frame preceding the target sample audio data frame, where K is a positive integer.

[0225] The second acquisition module 22 is used to acquire, based on the N target sample cepstral coefficients of the target sample audio data frame, the first-order time derivatives of M samples and the second-order time derivatives of M samples associated with the target sample audio data frame; N is a positive integer greater than 1 and M is a positive integer less than N.

[0226] The third acquisition module 23 is used to acquire N historical sample cepstral coefficients corresponding to each historical sample audio data frame, and to determine the dynamic characteristics of the sample spectrum associated with the target sample audio data frame based on the acquired K*N historical sample cepstral coefficients.

[0227] The mask prediction module 24 is used to input the cepstral coefficients of N target samples, the first-order time derivatives of M samples, the second-order time derivatives of M samples, and the dynamic features of the sample spectrum into the initial mask estimation model, and output the predicted mask corresponding to the audio data frame of the target sample from the initial mask estimation model.

[0228] The model training module 25 is used to iteratively train the initial mask estimation model based on the prediction mask and the sample mask to obtain the target mask estimation model used to output the target audio data frame associated with the original audio data; the target mask is used to suppress noise data in the original audio data to obtain the enhanced audio data corresponding to the original audio data.

[0229] The specific functional implementation methods of the first acquisition module 21, the second acquisition module 22, the third acquisition module 23, the mask prediction module 24, and the model training module 25 can be found above. Figure 8 Steps S201-S205 in the corresponding embodiments will not be described again here. Furthermore, the beneficial effects of using the same method will also not be described again.

[0230] Please see Figure 13 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 13 As shown, the computer device 1000 may include a processor 1001, a network interface 1004, and a memory 1005. Furthermore, the computer device 1000 may also include a user interface 1003 and at least one communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as at least one disk storage device. Optionally, the memory 1005 may also be at least one storage device located remotely from the processor 1001. Figure 13 As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

[0231] In such Figure 13In the computer device 1000 shown, the network interface 1004 provides network communication functionality; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to execute the aforementioned... Figure 3 , Figure 8 The description of the audio data processing method in any corresponding embodiment will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0232] Furthermore, it should be noted that this application embodiment also provides a computer-readable storage medium, which stores a computer program executed by the aforementioned audio data processing device 1 and audio data processing device 2. The computer program includes program instructions, and when the processor executes the program instructions, it can execute the aforementioned... Figure 3 , Figure 8 The description of the audio data processing method in any corresponding embodiment is already provided, and therefore will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer-readable storage medium embodiments related to this application, please refer to the description of the method embodiments of this application.

[0233] The aforementioned computer-readable storage medium can be an internal storage unit of the audio data processing apparatus provided in any of the foregoing embodiments or the computer device, such as a hard disk or memory of the computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device. Furthermore, the computer-readable storage medium may include both internal storage units and external storage devices of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0234] Furthermore, it should be noted that this application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned... Figure 3 , Figure 8The method is provided in any of the corresponding embodiments. Furthermore, the beneficial effects of using the same method will not be repeated here. For technical details not disclosed in the computer program products or computer program embodiments involved in this application, please refer to the description of the method embodiments of this application.

[0235] For further details, please see Figure 14 , Figure 14 This is a schematic diagram of the structure of an audio data processing system provided in an embodiment of this application. The audio data processing system 3 may include an audio data processing device 1a and an audio data processing device 2a. The audio data processing device 1a may be the aforementioned... Figure 11 The audio data processing device 1 in the corresponding embodiment can be understood to be integrated into the above-mentioned audio data processing device 1a. Figure 2 The computer device 20 in the corresponding embodiment will not be described in detail here. The audio data processing device 2a can be the one described above. Figure 12 The audio data processing device 2 in the corresponding embodiment can be understood to be integrated into the above-mentioned audio data processing device 2a. Figure 2 The computer device 20 in the corresponding embodiment will not be described again here. Furthermore, the beneficial effects of using the same method will also not be described again. For technical details not disclosed in the embodiments of the audio data processing system involved in this application, please refer to the description of the method embodiments of this application.

[0236] The terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0237] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0238] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. An audio data processing method, characterized in that, include: Obtain the target audio data frame and K historical audio data frames associated with the original audio data; the target audio data frame and the K historical audio data frames are both spectrum frames, and each of the K historical audio data frames is a spectrum frame preceding the target audio data frame, where K is a positive integer; When N target cepstral coefficients of the target audio data frame are obtained, M first-order time derivatives and M second-order time derivatives associated with the target audio data frame are obtained based on the N target cepstral coefficients. N is a positive integer greater than 1, and M is a positive integer less than N; Obtain N historical cepstral coefficients corresponding to each historical audio data frame, and determine the spectral dynamic features associated with the target audio data frame based on the obtained K*N historical cepstral coefficients; The N target cepstral coefficients, the M first-order time derivatives, the M second-order time derivatives, and the spectral dynamic features are input into the target mask estimation model, and the target mask estimation model outputs the target mask corresponding to the target audio data frame; the target mask is used to suppress noise data in the original audio data to obtain enhanced audio data corresponding to the original audio data.

2. The method according to claim 1, characterized in that, The acquisition of the target audio data frame and K historical audio data frames associated with the original audio data includes: The original audio data is preprocessed by frame segmentation and windowing to obtain H audio data segments; H is a positive integer greater than 1. Perform time-frequency transformation on each audio data segment to obtain the audio data frame corresponding to each audio data segment; Determine the target audio data frame and the K historical audio data frames preceding the target audio data frame from H audio data frames; K is less than H.

3. The method according to claim 2, characterized in that, The H audio data segments include audio data segment i, where i is a positive integer less than or equal to H; The step of performing time-frequency transformation on each audio data segment to obtain the audio data frame corresponding to each audio data segment includes: Perform a Fourier transform on the audio data segment i to obtain the DC component frequency point and 2S frequency points of the audio data segment i in the frequency domain; the 2S frequency points include S frequency points related to the first frequency point type and S frequency points related to the second frequency point type; S is a positive integer; S frequency points related to the first frequency point type are obtained from the 2S frequency points, and the audio data frame corresponding to the audio data segment i is determined based on the S frequency points related to the first frequency point type and the DC component frequency point.

4. The method according to claim 1, characterized in that, The target audio data frame contains S1 frequency points, including a DC component frequency point and S2 frequency points related to the frequency point type, where S1 and S2 are both positive integers. The method further includes: Map the S1 frequency points to N acoustic frequency bands; S1 is greater than or equal to N; Cepstral processing is performed on each acoustic frequency band to obtain the target cepstral coefficients corresponding to each acoustic frequency band.

5. The method according to claim 4, characterized in that, The N acoustic frequency bands include acoustic frequency band j, where j is a positive integer less than or equal to N; The step of performing cepstral processing on each acoustic frequency band to obtain the target cepstral coefficients corresponding to each acoustic frequency band includes: Obtain the band energy of the acoustic band j, and perform a logarithmic transformation on the band energy of the acoustic band j to obtain the logarithmic band energy of the acoustic band j; The target cepstral coefficients corresponding to the acoustic frequency band j are obtained by performing a discrete cosine transform on the logarithmic frequency band energy of the acoustic frequency band j.

6. The method according to claim 1, characterized in that, The step of obtaining M first-order time derivatives and M second-order time derivatives associated with the target audio data frame based on the N target cepstral coefficients includes: Perform differential operations on the N target cepstral coefficients to obtain (N-1) differential operation values. Take each of the (N-1) differential operation values ​​as the first time derivative of the corresponding target cepstral coefficient. Obtain M first time derivatives associated with the target audio data frame from the (N-1) first time derivatives. Perform a second difference operation on the (N-1) first-order time derivatives to obtain (N-2) difference operation values. Use each of the (N-2) difference operation values ​​as the second-order time derivative of the corresponding target cepstral coefficient. Obtain M second-order time derivatives associated with the target audio data frame from the (N-2) second-order time derivatives.

7. The method according to claim 1, characterized in that, The step of obtaining N historical cepstral coefficients corresponding to each historical audio data frame, and determining the spectral dynamic features associated with the target audio data frame based on the obtained K*N historical cepstral coefficients, includes: In the K historical audio data frames, any two adjacent historical audio data frames are selected as the first historical audio data frame and the second historical audio data frame; the second historical audio data frame is a spectrum frame obtained after the first historical audio data frame. Obtain N historical cepstral coefficients corresponding to the first historical audio data frame from the cache associated with the target audio data frame, and obtain N historical cepstral coefficients corresponding to the second historical audio data frame; The N historical cepstral coefficients corresponding to the first historical audio data frame are used as the first historical cepstral coefficients, and the N historical cepstral coefficients corresponding to the second historical audio data frame are used as the second historical cepstral coefficients. The frequency band difference between the first historical cepstral coefficient and the second historical cepstral coefficient is used as the inter-frame difference between the first historical audio data frame and the second historical audio data frame. When each adjacent historical audio data frame in the K historical audio data frames is used as the first historical audio data frame and the second historical audio data frame, the inter-frame difference value between the K historical audio data frames is obtained, and the spectral dynamic characteristics associated with the target audio data frame are determined based on the inter-frame difference value between the K historical audio data frames.

8. The method according to claim 7, characterized in that, The step of using the frequency band difference value between the first historical cepstral coefficient and the second historical cepstral coefficient as the inter-frame difference value between the first historical audio data frame and the second historical audio data frame includes: From the N historical cepstral coefficients included in the first historical cepstral coefficient, obtain the historical cepstral coefficient L. p And from the N historical cepstral coefficients contained in the second historical cepstral coefficient, obtain the historical cepstral coefficient L. q p and q are both positive integers less than or equal to N, and p = q; Obtain the historical cepstral coefficients L p With the historical cepstral coefficient L q The coefficient difference between them; Based on the coefficient difference value, the frequency band difference value between the first historical cepstral coefficient and the second historical cepstral coefficient is determined, and the frequency band difference value is used as the inter-frame difference value between the first historical audio data frame and the second historical audio data frame.

9. The method according to claim 1, characterized in that, The target mask estimation model includes a mask estimation network layer and a mask output layer; The step of inputting the N target cepstral coefficients, the M first-order time derivatives, the M second-order time derivatives, and the spectral dynamic features into the target mask estimation model, and outputting the target mask corresponding to the target audio data frame by the target mask estimation model, includes: The N target cepstral coefficients, the M first-order time derivatives, the M second-order time derivatives, and the spectral dynamic features are used as the target audio features of the target audio data frame. The target audio features are input into the mask estimation network layer, and the mask estimation network layer performs mask estimation on the target audio features to obtain the hidden features corresponding to the target audio features. The hidden features are input into the mask output layer, and the hidden features are merged by the mask output layer to obtain the target mask corresponding to the target audio data frame.

10. The method according to claim 9, characterized in that, The mask estimation network layer includes a first mask estimation network layer, a second mask estimation network layer, and a third mask estimation network layer with skip connections. The step of inputting the target audio features into the mask estimation network layer, and performing mask estimation on the target audio features through the mask estimation network layer to obtain the hidden features corresponding to the target audio features includes: The target audio features are input into the first mask estimation network layer, and the first intermediate features are output through the first mask estimation network layer. Based on the skip connection between the first mask estimation network layer and the second mask estimation network layer, the first intermediate feature and the target audio feature are concatenated to obtain the second intermediate feature. The second intermediate feature is then input into the second mask estimation network layer, and the third intermediate feature is output through the second mask estimation network layer. Based on the skip connections between the first mask estimation network layer and the third mask estimation network layer, as well as the skip connections between the second mask estimation network layer and the third mask estimation network layer, the third intermediate feature, the target audio feature, and the first intermediate feature are concatenated to obtain a fourth intermediate feature. The fourth intermediate feature is then input into the third mask estimation network layer, and the hidden feature corresponding to the target audio feature is output through the third mask estimation network layer.

11. The method according to claim 1, characterized in that, Also includes: The target mask is interpolated to obtain an interpolated mask; the length of the interpolated mask is the same as the length of the target audio data frame. The interpolation mask is multiplied by the target audio data frame, and the multiplication result is subjected to inverse Fourier transform to obtain the target audio data after noise suppression of the target audio data frame. After noise suppression is performed on each audio data frame associated with the original audio data, enhanced audio data corresponding to the original audio data is obtained.

12. An audio data processing method, characterized in that, include: Obtain the target sample audio data frame and K historical sample audio data frames associated with the sample audio data, and obtain the sample mask corresponding to the target sample audio data frame; The target sample audio data frame and the K historical sample audio data frames are both spectrum frames, and each of the K historical sample audio data frames is a spectrum frame preceding the target sample audio data frame, where K is a positive integer. When N target sample cepstral coefficients of the target sample audio data frame are obtained, M sample first-order time derivatives and M sample second-order time derivatives associated with the target sample audio data frame are obtained based on the N target sample cepstral coefficients. N is a positive integer greater than 1, and M is a positive integer less than N; Obtain N historical sample cepstral coefficients corresponding to each historical sample audio data frame, and determine the sample spectral dynamic features associated with the target sample audio data frame based on the obtained K*N historical sample cepstral coefficients; The cepstral coefficients of the N target samples, the first-order time derivatives of the M samples, the second-order time derivatives of the M samples, and the dynamic spectral features of the samples are input into the initial mask estimation model, and the initial mask estimation model outputs the predicted mask corresponding to the audio data frame of the target sample. The initial mask estimation model is iteratively trained based on the predicted mask and the sample mask to obtain a target mask estimation model for outputting the target mask corresponding to the target audio data frame associated with the original audio data; the target mask is used to suppress noise data in the original audio data to obtain enhanced audio data corresponding to the original audio data.

13. An audio data processing apparatus, characterized in that, include: The first acquisition module is used to acquire a target audio data frame and K historical audio data frames associated with the original audio data; the target audio data frame and the K historical audio data frames are both spectrum frames, and each of the K historical audio data frames is a spectrum frame preceding the target audio data frame, where K is a positive integer. The second acquisition module is used to acquire, based on the N target cepstral coefficients of the target audio data frame, M first-order time derivatives and M second-order time derivatives associated with the target audio data frame. N is a positive integer greater than 1, and M is a positive integer less than N; The third acquisition module is used to acquire N historical cepstral coefficients corresponding to each historical audio data frame, and to determine the spectral dynamic features associated with the target audio data frame based on the acquired K*N historical cepstral coefficients. The mask estimation module is used to input the N target cepstral coefficients, the M first-order time derivatives, the M second-order time derivatives, and the spectral dynamic features into the target mask estimation model, and the target mask estimation model outputs the target mask corresponding to the target audio data frame; the target mask is used to suppress noise data in the original audio data to obtain enhanced audio data corresponding to the original audio data.

14. An audio data processing apparatus, characterized in that, include: The first acquisition module is used to acquire a target sample audio data frame and K historical sample audio data frames associated with the sample audio data, and to acquire the sample mask corresponding to the target sample audio data frame; the target sample audio data frame and the K historical sample audio data frames are all spectrum frames, and each of the K historical sample audio data frames is a spectrum frame preceding the target sample audio data frame, where K is a positive integer; The second acquisition module is used to acquire, based on the N target sample cepstral coefficients of the target sample audio data frame, M sample first-order time derivatives and M sample second-order time derivatives associated with the target sample audio data frame. N is a positive integer greater than 1, and M is a positive integer less than N; The third acquisition module is used to acquire N historical sample cepstral coefficients corresponding to each historical sample audio data frame, and to determine the sample spectrum dynamic features associated with the target sample audio data frame based on the acquired K*N historical sample cepstral coefficients. The mask prediction module is used to input the cepstral coefficients of the N target samples, the first-order time derivatives of the M samples, the second-order time derivatives of the M samples, and the dynamic spectral features of the samples into the initial mask estimation model, and the initial mask estimation model outputs the predicted mask corresponding to the audio data frame of the target sample. The model training module is used to iteratively train the initial mask estimation model based on the predicted mask and the sample mask to obtain a target mask estimation model for outputting the target mask corresponding to the target audio data frame associated with the original audio data; the target mask is used to suppress noise data in the original audio data to obtain enhanced audio data corresponding to the original audio data.

15. A computer device, characterized in that, include: Processor and memory; The processor is connected to the memory, wherein the memory is used to store a computer program, and the processor is used to invoke the computer program to cause the computer device to perform the method according to any one of claims 1-12.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded and executed by a processor to cause a computer device having the processor to perform the method of any one of claims 1-12.

17. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium, the computer instructions being adapted to be read and executed by a processor to cause a computer device having the processor to perform the method of any one of claims 1-12.

Images