Audio data processing method and apparatus, device, and medium

Abstract

Embodiments of the present application provide an audio data processing method, device and equipment and a medium. The method comprises: obtaining a recording audio; the recording audio comprises a background reference audio component, a speech audio component and an environmental noise component; obtaining a to-be-matched audio fingerprint of the recording audio, and obtaining a prototype audio matched with the recording audio in an audio database according to the to-be-matched audio fingerprint; obtaining a candidate speech audio from the recording audio according to the prototype audio; the candidate speech audio comprises the speech audio component and the environmental noise component; determining a difference between the recording audio and the candidate speech audio as the background reference audio component contained in the recording audio; performing environmental noise noise reduction processing on the candidate speech audio to obtain a noise-reduced speech audio corresponding to the candidate speech audio, and merging the noise-reduced speech audio and the background reference audio component to obtain a noise-reduced recording audio. The present application can improve the noise reduction effect of the recording audio.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to an audio data processing method, apparatus, device, and medium. Background Technology

[0002] With the rapid popularization of audio and video applications, users are increasingly sharing their everyday music recordings through audio applications. For example, when a user sings along to backing vocals and records using a device with recording capabilities (such as a mobile phone or a sound card with a microphone), the user may be in a noisy environment or using a rudimentary device. This results in the recorded music signal including not only the user's vocals and backing vocals, but also noise from the environment and the device itself. If the unprocessed music recording is directly shared to an audio application, other users will have difficulty hearing the user's voice clearly when playing the recording. Therefore, noise reduction processing is necessary for the recorded music signal.

[0003] Current noise reduction algorithms require clear identification of noise and signal types. For example, they rely on the characteristic distance between human voices and noise based on signal correlation and spectral distribution features, employing statistical or deep learning-based noise reduction methods for suppression. However, music signals come in many types (e.g., classical, folk, rock), some of which resemble certain types of environmental noise, or their spectral characteristics are similar to those of noise. When using the aforementioned noise reduction algorithms to denoise music recordings, the algorithms may misclassify the music signal as noise and suppress it, or vice versa, resulting in less than ideal noise reduction performance. Summary of the Invention

[0004] This application provides an audio data processing method, apparatus, device, and medium that can improve the noise reduction effect of recorded audio.

[0005] One embodiment of this application provides an audio data processing method, including:

[0006] Acquire the recorded audio; the recorded audio includes background reference audio components, speech audio components, and environmental noise components;

[0007] Obtain the audio fingerprint corresponding to the recorded audio, and retrieve the prototype audio that matches the recorded audio from the audio database based on the audio fingerprint.

[0008] Candidate speech audio is obtained from the recorded audio based on the prototype audio; the candidate speech audio includes speech audio components and environmental noise components;

[0009] The difference between the recorded audio and the candidate speech audio is determined as the background reference audio component contained in the recorded audio.

[0010] The candidate speech audio is subjected to environmental noise reduction processing to obtain the denoised speech audio corresponding to the candidate speech audio. The denoised speech audio is then merged with the background reference audio component to obtain the denoised recording audio.

[0011] One embodiment of this application provides an audio data processing method, including:

[0012] Acquire speech sample audio, noise sample audio, and standard sample audio; generate sample recording audio based on the speech sample audio, noise sample audio, and standard sample audio; the speech sample audio and noise sample audio are obtained through recording, and the standard sample audio is clean audio stored in the audio database.

[0013] The first initial network model is used to obtain the sample predicted speech audio from the sample recorded audio. The first initial network model is used to filter the standard sample audio contained in the sample recorded audio. The expected predicted speech audio of the first initial network model is determined by the speech sample audio and the noise sample audio.

[0014] The sample predicted noise reduction frequency corresponding to the sample predicted speech audio is obtained according to the second initial network model; the second initial network model is used to suppress the noise sample audio contained in the sample predicted speech audio, and the expected predicted noise reduction frequency of the second initial network model is determined by the speech sample audio.

[0015] Based on the sample predicted speech audio and the expected predicted speech audio, the network parameters of the first initial network model are adjusted to obtain the first deep network model; the first deep network model is used to filter the recorded audio to obtain candidate speech audio, the recorded audio includes background reference audio components, speech audio components and environmental noise components, and the candidate speech audio includes speech audio components and environmental noise components.

[0016] Based on the sample predicted noise reduction frequency and the expected predicted noise reduction frequency, the network parameters of the second initial network model are adjusted to obtain the second deep network model; the second deep network model is used to perform noise reduction processing on the candidate speech audio to obtain the noise-reduced speech audio.

[0017] One embodiment of this application provides an audio data processing apparatus, including:

[0018] The audio acquisition module is used to acquire recorded audio; the recorded audio includes background reference audio components, speech audio components, and environmental noise components.

[0019] The fingerprint retrieval module is used to obtain the audio fingerprint to be matched corresponding to the recorded audio, and to retrieve the prototype audio that matches the recorded audio from the audio database based on the audio fingerprint to be matched.

[0020] The audio filtering module is used to obtain candidate speech audio from the recorded audio based on the prototype audio; the candidate speech audio includes speech audio components and environmental noise components.

[0021] The audio determination module is used to determine the difference between the recorded audio and the candidate speech audio as the background reference audio component contained in the recorded audio.

[0022] The noise reduction module is used to perform environmental noise reduction on the candidate speech audio to obtain the noise-reduced speech audio corresponding to the candidate speech audio. The noise-reduced speech audio is then merged with the background reference audio component to obtain the noise-reduced recording audio.

[0023] The fingerprint retrieval module includes:

[0024] The frequency domain transformation unit is used to divide the recorded audio into M recording data frames, and to perform frequency domain transformation on the i-th recording data frame among the M recording data frames to obtain the power spectrum data corresponding to the i-th recording data frame; i and M are both positive integers, and i is less than or equal to M;

[0025] The spectrum band division unit is used to divide the power spectrum data corresponding to the i-th recording data frame into N spectrum bands, and construct the sub-fingerprint information corresponding to the i-th recording data frame based on the peak signals in the N spectrum bands; N is a positive integer;

[0026] The audio fingerprint combination unit is used to combine the sub-fingerprint information corresponding to the M recording data frames according to the time order of the M recording data frames in the recorded audio, so as to obtain the audio fingerprint to be matched corresponding to the recorded audio.

[0027] The prototype audio matching unit is used to obtain the audio fingerprint database corresponding to the audio database, perform fingerprint retrieval in the audio fingerprint database based on the audio fingerprint to be matched, and determine the prototype audio that matches the recorded audio in the audio database based on the fingerprint retrieval result.

[0028] Specifically, the prototype audio matching unit is used for:

[0029] The M sub-fingerprint information contained in the audio fingerprint to be matched is mapped to M hash values ​​to be matched, and the recording time corresponding to each of the M hash values ​​to be matched is obtained; the recording time corresponding to a hash value to be matched is used to characterize the time when the sub-fingerprint information corresponding to the hash value to be matched appears in the recorded audio.

[0030] If the p-th hash value among the M hash values ​​to be matched matches the first hash value contained in the audio fingerprint database, then obtain the first time difference between the recording time corresponding to the p-th hash value and the time information corresponding to the first hash value; p is a positive integer less than or equal to M;

[0031] If the q-th hash value among the M hash values ​​to be matched matches the second hash value contained in the audio fingerprint database, then obtain the second time difference between the recording time corresponding to the q-th hash value and the time information corresponding to the second hash value; q is a positive integer less than or equal to M;

[0032] When the first time difference and the second time difference meet the numerical threshold, and the first hash value and the second hash value belong to the same audio fingerprint, the audio fingerprint to which the first hash value belongs is determined as the fingerprint retrieval result, and the audio data corresponding to the fingerprint retrieval result is determined as the prototype audio corresponding to the recorded audio.

[0033] The audio filtering module includes:

[0034] The normalization processing unit is used to acquire the recording power spectrum data corresponding to the recorded audio, and to perform normalization processing on the recording power spectrum data to obtain the first spectral feature.

[0035] The normalization processing unit described above is also used to obtain the prototype power spectrum data corresponding to the prototype audio, perform normalization processing on the prototype power spectrum data to obtain the second spectral feature, and combine the first spectral feature and the second spectral feature into the input feature.

[0036] The first frequency gain output unit is used to input the input features into the first deep network model and output the first frequency gain for the recorded audio through the first deep network model.

[0037] The speech audio acquisition unit is used to acquire candidate speech audio contained in the recorded audio based on the first frequency point gain and recording power spectrum data.

[0038] The first frequency gain output unit includes:

[0039] The feature extraction subunit is used to input the input features into the first deep network model and obtain the temporal distribution features corresponding to the input features based on the feature extraction network layer in the first deep network model.

[0040] The activation subunit is used to obtain the target feature vector corresponding to the temporal distribution characteristics based on the fully connected network layer in the first deep network model, and output the first frequency point gain through the activation layer in the first deep network model.

[0041] Wherein, the first frequency point gain includes the speech gains corresponding to T frequency points respectively, the recording power spectrum data includes the energy values ​​corresponding to T frequency points respectively, and the T speech gains correspond one-to-one with the T energy values; T is a positive integer greater than 1;

[0042] The voice audio acquisition unit includes:

[0043] The frequency weighting subunit is used to weight the energy values ​​belonging to the same frequency point in the recording power spectrum data according to the speech gain corresponding to T frequency points in the first frequency point gain, so as to obtain the weighted energy values ​​corresponding to T frequency points respectively.

[0044] The weighted energy value combination subunit is used to determine the weighted audio domain signal corresponding to the recorded audio based on the weighted energy values ​​corresponding to T frequency points.

[0045] The time-domain transformation subunit is used to perform time-domain transformation on the weighted audio domain signal to obtain the candidate speech audio contained in the recorded audio.

[0046] The noise reduction processing module includes:

[0047] The second frequency gain output unit is used to acquire the speech power spectrum data corresponding to the candidate speech audio, input the speech power spectrum data into the second deep network model, and output the second frequency gain for the candidate speech audio through the second deep network model.

[0048] The signal weighting unit is used to obtain the weighted audio domain signal corresponding to the candidate audio based on the second frequency point gain and the speech power spectrum data;

[0049] The time-domain transformation unit is used to perform time-domain transformation on the weighted speech audio domain signal to obtain the denoised speech audio corresponding to the candidate speech audio.

[0050] The device also includes:

[0051] The audio sharing module is used to share the noise-reduced audio recordings to social media platforms, so that terminal devices on the social media platforms can play the noise-reduced audio recordings when accessing the social media platforms.

[0052] One embodiment of this application provides an audio data processing apparatus, including:

[0053] The sample acquisition module is used to acquire speech sample audio, noise sample audio, and standard sample audio, and generate sample recording audio based on the speech sample audio, noise sample audio, and standard sample audio. The speech sample audio and noise sample audio are obtained through recording, while the standard sample audio is clean audio stored in the audio database.

[0054] The first prediction module is used to obtain the predicted speech audio from the sample audio recordings based on the first initial network model. The first initial network model is used to filter the standard sample audio contained in the sample audio recordings. The expected predicted speech audio of the first initial network model is determined by the speech sample audio and the noise sample audio.

[0055] The second prediction module is used to obtain the sample prediction noise reduction frequency corresponding to the sample prediction speech audio according to the second initial network model; the second initial network model is used to suppress the noise sample audio contained in the sample prediction speech audio, and the expected prediction noise reduction frequency of the second initial network model is determined by the speech sample audio.

[0056] The first adjustment module is used to adjust the network parameters of the first initial network model based on the sample predicted speech audio and the expected predicted speech audio to obtain the first deep network model; the first deep network model is used to filter the recorded audio to obtain candidate speech audio, the recorded audio includes background reference audio components, speech audio components and environmental noise components, and the candidate speech audio includes speech audio components and environmental noise components.

[0057] The second adjustment module is used to adjust the network parameters of the second initial network model based on the sample predicted noise reduction frequency and the expected predicted noise reduction frequency to obtain the second deep network model; the second deep network model is used to perform noise reduction processing on the candidate speech audio to obtain the noise-reduced speech audio.

[0058] The number of sample audio recordings is K, where K is a positive integer;

[0059] The sample acquisition module includes:

[0060] The array construction unit is used to obtain the set of weighted coefficients for the first initial network model and construct K arrays based on the set of weighted coefficients; each array includes coefficients corresponding to the speech sample audio, the noise sample audio, and the standard sample audio respectively;

[0061] The sample recording construction unit is used to weight the speech sample audio, noise sample audio, and standard sample audio according to the coefficients contained in the j-th array of K arrays, to obtain the sample recording audio corresponding to the j-th array; j is a positive integer less than or equal to K.

[0062] One aspect of this application provides a computer device, including a memory and a processor. The memory is connected to the processor, the memory is used to store computer programs, and the processor is used to call the computer programs so that the computer device executes the method provided in one aspect of this application.

[0063] One aspect of this application provides a computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor, so that a computer device having a processor performs the method provided in one aspect of this application.

[0064] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method provided in the above aspect.

[0065] This application embodiment can obtain a recorded audio containing a background reference audio component, a speech audio component, and an environmental noise component. It then obtains a matching audio fingerprint corresponding to the recorded audio, retrieves a matching prototype audio from an audio database based on the fingerprint, and subsequently obtains candidate speech audio from the recorded audio based on the prototype audio. This candidate speech audio includes a speech audio component and an environmental noise component. The difference between the recorded audio and the candidate speech audio is determined as the background reference audio component. Environmental noise reduction processing is then performed on the candidate speech audio to obtain denoised speech audio. This denoised speech audio is then merged with the background reference audio component to obtain the denoised recorded audio. It is evident that the recorded audio can be a mixed audio containing multiple audio components. During the denoising process, the prototype audio corresponding to the recorded audio can be found through audio fingerprint retrieval. Based on this prototype audio, candidate speech audio can be selected from the recorded audio. Subtracting the candidate speech audio from the recorded audio yields the background reference audio component. Noise reduction processing is then performed on the candidate speech audio to obtain the denoised speech audio. Finally, the denoised speech audio is superimposed with the background reference audio component to obtain the denoised recorded audio. In other words, by transforming the noise reduction problem of the recorded audio into the noise reduction problem of the candidate speech audio, we can avoid confusing the background reference audio component and the environmental noise component in the recorded audio, thereby improving the noise reduction effect of the recorded audio. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1This is a schematic diagram of a network architecture provided in an embodiment of this application;

[0068] Figure 2 This is a schematic diagram of a noise reduction scenario for music recording audio provided in an embodiment of this application;

[0069] Figure 3 This is a flowchart illustrating an audio data processing method provided in an embodiment of this application;

[0070] Figure 4 This is a schematic diagram of a music recording scenario provided in an embodiment of this application;

[0071] Figure 5 This is a flowchart illustrating an audio data processing method provided in an embodiment of this application;

[0072] Figure 6 This is a schematic diagram of the structure of a first deep network model provided in an embodiment of this application;

[0073] Figure 7 This is a schematic diagram of the structure of a second deep network model provided in an embodiment of this application;

[0074] Figure 8 This is a schematic diagram of a recording audio noise reduction process provided in an embodiment of this application;

[0075] Figure 9 This is a flowchart illustrating an audio data processing method provided in an embodiment of this application;

[0076] Figure 10 This is a schematic diagram illustrating the training of a deep network model provided in an embodiment of this application;

[0077] Figure 11 This is a schematic diagram of the structure of an audio data processing device provided in an embodiment of this application;

[0078] Figure 12 This is a schematic diagram of the structure of an audio data processing device provided in an embodiment of this application;

[0079] Figure 13 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0080] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0081] Artificial intelligence cloud services are generally also known as "AI as a Service" (AIaaS). This is currently a mainstream service model for artificial intelligence platforms. Specifically, AIaaS platforms break down several common AI services and provide them as independent or packaged services in the cloud. This service model is similar to opening an AI-themed marketplace: all developers can access and use one or more AI services provided by the platform through APIs (Application Programming Interfaces). Some experienced developers can also use the AI ​​frameworks and infrastructure provided by the platform to deploy and maintain their own dedicated cloud AI services.

[0082] The solution provided in this application relates to AI noise reduction service in artificial intelligence cloud services. In this application embodiment, the AI ​​noise reduction service can be accessed through API interface. The AI ​​noise reduction service is used to perform noise reduction processing on the recorded audio shared to social platforms (e.g., music recording sharing applications) to improve the noise reduction effect of the recorded audio.

[0083] Please see Figure 1 , Figure 1 This is a schematic diagram of a network architecture provided in an embodiment of this application. Figure 1 As shown, this network architecture may include a server 10d and a user terminal cluster. The user terminal cluster may include one or more user terminals; the number of user terminals is not limited here. Figure 1 As shown, the user terminal cluster can specifically include user terminal 10a, user terminal 10b, and user terminal 10c. Server 10d can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. User terminals 10a, 10b, and 10c can include, but are not limited to: smartphones, tablets, laptops, PDAs, mobile internet devices (MIDs), wearable devices (such as smartwatches and smart bracelets), and smart TVs with recording functions, or sound card devices with microphones. Figure 1 As shown, user terminals 10a, 10b, and 10c can each connect to server 10d via a network, so that each user terminal can interact with server 10d through the network connection.

[0084] by Figure 1 Taking the user terminal 10a shown as an example, the user terminal 10a can integrate a recording function. When a user wants to record their own or others' audio data, they can use an audio playback device to play background reference audio (the background reference audio here can be background music, or background audio and subtitle dubbing audio in a video, etc.), and start the recording function in the user terminal 10a to begin recording a mixed audio including the background reference audio played by the aforementioned audio playback device. This application refers to this mixed audio as recorded audio, and the background reference audio can be used as the background reference audio component in the recorded audio. Wherein, when the user terminal 10a has an audio playback function, the aforementioned audio playback device can be the user terminal 10a itself; or, the audio playback device can also be other devices with audio playback functions besides the user terminal 10a; the aforementioned recorded audio can include the background reference audio played by the audio playback device, the ambient noise in the environment where the audio playback device / user is located, and the softened audio of the user's voice. The recorded background reference audio can be used as the background reference audio component in the recorded audio, the recorded ambient noise can be used as the ambient noise component in the recorded audio, and the recorded user voice can be used as the voice audio component in the recorded audio. User terminal 10a can upload recorded audio to a social media platform; for example, if user terminal 10a has a social media platform client installed, it can upload the recorded audio to the social media platform client, which can then transmit the recorded audio to the social media platform's backend server (e.g., the aforementioned...). Figure 1 Server 10d shown.

[0085] Furthermore, since the recorded audio contains environmental noise components, the social media platform's backend server needs to perform noise reduction processing on the recorded audio. The noise reduction process can be as follows: obtain the audio fingerprint to be matched corresponding to the recorded audio; based on the audio fingerprint, retrieve the prototype audio that matches the recorded audio from the audio database (the prototype audio can be understood as the official, genuine audio corresponding to the background reference audio component in the recorded audio); based on the prototype audio, candidate speech audio (including the aforementioned environmental noise and user speech) can be obtained from the recorded audio, and the difference between the recorded audio and the candidate speech audio can be determined as the background reference audio component; perform noise reduction processing on the candidate speech audio to obtain the corresponding denoised speech audio; superimpose the denoised speech audio with the background reference audio component to obtain the denoised recorded audio, which can then be shared on the social media platform. By transforming the noise reduction problem of the recorded audio into the noise reduction problem of the candidate speech audio, the noise reduction efficiency of the recorded audio can be improved.

[0086] Please see Figure 2 , Figure 2 This is a schematic diagram illustrating a noise reduction scenario for music recording audio provided in an embodiment of this application. For example... Figure 2 The user terminal 20a shown can be a terminal device held by user A (for example, the one mentioned above). Figure 1 (Any user terminal in the user terminal cluster shown) This user terminal 20a integrates recording and audio playback functions, so it can function as both a recording device and an audio playback device. When user A wants to record their own singing, they can activate the recording function on user terminal 20a, start singing while the accompaniment music is playing on user terminal 20a, and begin recording. After recording, the recorded audio 20b can be obtained, which may contain user A's singing voice and the accompaniment music played on user terminal 20a. User terminal 20a can upload the recorded audio 20b to the client corresponding to the music application. After receiving the audio 20b, the client transmits it to the backend server corresponding to the music application (e.g., the one mentioned above). Figure 1 The server 10d shown enables the backend server to store and share the music recording audio 20b.

[0087] In a real-world music recording scenario, user A might be in a noisy environment. Therefore, the music recording audio 20b recorded by user terminal 20a will contain not only user A's singing and the backing vocals played by user terminal 20a, but also environmental noise. That is, music recording audio 20b can include three audio components: noise, backing vocals, and user's singing. For example, if user A is on the street, the noise in the music recording audio 20b recorded by user terminal 20a could be vehicle horns, street vendor shouts, or pedestrian conversations; of course, the noise in music recording audio 20b can also include electronic noise. If the backend server directly shares the music recording audio 20b uploaded by user terminal 20a, other terminal devices will be unable to clearly hear the music recorded by user A when accessing music applications and playing music recording audio 20a. Therefore, before sharing the music recording audio 20b in a music application, it is necessary to perform noise reduction processing on the music recording audio 20b. Then, the noise-reduced music recording audio is shared so that other terminal devices can play the noise-reduced music recording audio when accessing the music application, thus understanding user A's true singing ability. In other words, user terminal 20a is only responsible for the acquisition and uploading of the music recording audio 20b; the noise reduction process of the music recording audio 20b can be performed by the background server corresponding to the music application. Optionally, after acquiring the music recording audio 20b, user terminal 20a can perform noise reduction processing on the music recording audio 20b and upload the noise-reduced music recording audio to the music application. After receiving the noise-reduced music recording audio, the background server corresponding to the music application can directly share the noise-reduced music recording audio, meaning that the noise reduction processing of the music recording audio 20b can be performed by user terminal 20a.

[0088] The following description uses a music application's backend server (e.g., server 10d mentioned above) as an example to illustrate the noise reduction process for recorded music audio 20b. The essence of this noise reduction process is to suppress noise in the recorded music audio 20b while preserving the backing vocals and user A's singing voice. In other words, noise reduction of recorded music audio 20b means eliminating noise as much as possible while preserving the backing vocals and user A's singing voice as much as possible.

[0089] like Figure 2As shown, after the backend server of a music application (e.g., server 10d mentioned above) obtains the music recording audio 20b, it can perform a frequency domain transformation on the music recording audio 20b, that is, transform the music recording audio 20b from the time domain to the frequency domain, to obtain the frequency domain power spectrum corresponding to the music recording audio 20b; the frequency domain power spectrum can include the energy values ​​corresponding to each frequency point, and the frequency domain power spectrum can be as follows: Figure 2 As shown in the frequency domain power spectrum 20i, one energy value in the frequency domain power spectrum 20i corresponds to one frequency point, and one frequency point is one frequency sampling point.

[0090] Furthermore, based on the frequency domain power spectrum corresponding to the music recording audio 20b, the audio fingerprint 20c corresponding to the music recording audio 20b can be extracted; whereby the audio fingerprint can refer to a unique digital feature in the form of an identifier representing a segment of audio. The backend server can obtain the music library 20d in the music application, and the audio fingerprint library 20e corresponding to the music library 20d. The music library 20d can include all music audios stored in the music application, and the audio fingerprint library 20e can include the audio fingerprint corresponding to each music audio in the music library 20d. Then, based on the audio fingerprint 20c corresponding to the music recording audio 20b, an audio fingerprint search can be performed in the audio fingerprint library 20e to obtain the fingerprint search result corresponding to the audio fingerprint 20c (i.e., the audio fingerprint in the audio fingerprint library 20e that matches the audio fingerprint 20b). Based on the fingerprint search result, the music prototype audio 20f that matches the music recording audio 20b (such as the music prototype corresponding to the accompaniment in the music recording audio 20b) can be determined from the music library 20d. Similarly, the music prototype audio 20f can be transformed in the frequency domain, that is, the music prototype audio 20 is transformed from the time domain to the frequency domain to obtain the frequency domain power spectrum corresponding to the music prototype audio 20.

[0091] The frequency domain power spectrum corresponding to the music recording audio 20b is combined with the frequency domain power spectrum corresponding to the original music, and the combined frequency domain power spectrum is input into the first-order deep network model 20g. The first-order deep network model 20g outputs the frequency point gain. The first-order deep network model 20g can be a pre-trained network model capable of removing music from the music recording audio. The training process of the first-order deep network model 20g can be described in step S304 below. By multiplying the frequency point gain output by the first-order deep network model 20g with the frequency domain power spectrum corresponding to the music recording audio 20b, a weighted recording audio domain signal is obtained. The weighted recording audio domain signal is then transformed in the time domain, i.e., transformed from the frequency domain to the time domain, to obtain the music-removed audio 20k. Here, the music-removed audio 20k refers to the audio signal from which the backing vocals are filtered out of the music recording audio 20b.

[0092] like Figure 2 As shown, assuming the frequency gain output by the first-order deep network model 20g is a frequency gain sequence 20h, this frequency gain sequence 20h includes the speech gains corresponding to 5 frequencies: speech gain 5 for frequency 1, speech gain 7 for frequency 2, speech gain 8 for frequency 3, speech gain 10 for frequency 4, and speech gain 3 for frequency 5. Assuming the frequency domain power spectrum corresponding to the music recording audio 20b is a frequency domain power spectrum 20i, this frequency domain power spectrum 20i also includes the energy values ​​corresponding to the above 5 frequencies, specifically energy value 1 for frequency 1, energy value 2 for frequency 2, energy value 3 for frequency 3, energy value 2 for frequency 4, and energy value 1 for frequency 5. The weighted audio domain signal 20j is obtained by multiplying the speech gain at each frequency point in the frequency gain sequence 20h with the energy value corresponding to the same frequency point in the frequency power spectrum 20i. Specifically, the calculation process is as follows: The speech gain 5 corresponding to frequency point 1 in the frequency gain sequence 20h is multiplied with the energy value 1 corresponding to frequency point 1 in the frequency power spectrum 20i to obtain the weighted energy value 5. This weighted energy value 5 is the energy value 5 for frequency point 1 in the weighted audio domain signal 20j. The speech gain 7 corresponding to frequency point 2 in the frequency gain sequence 20h is multiplied with the energy value 2 corresponding to frequency point 2 in the frequency power spectrum 20i to obtain the energy value 5 for frequency point 1 in the weighted audio domain signal 20j. The energy value of 2 is 14; the product of the speech gain 8 corresponding to frequency point 3 in the frequency gain sequence 20h and the energy value 3 corresponding to frequency point 3 in the frequency power spectrum 20i is calculated to obtain the energy value 24 for frequency point 3 in the weighted audio domain signal 20j; the product of the speech gain 10 corresponding to frequency point 4 in the frequency gain sequence 20h and the energy value 2 corresponding to frequency point 4 in the frequency power spectrum 20i is calculated to obtain the energy value 20 for frequency point 4 in the weighted audio domain signal 20j; the product of the speech gain 3 corresponding to frequency point 5 in the frequency gain sequence 20h and the energy value 1 corresponding to frequency point 4 in the frequency power spectrum 20i is calculated to obtain the energy value 3 for frequency point 5 in the weighted audio domain signal 20j. By performing a time-domain transformation on the weighted audio domain signal 20j, the music-de-music audio 20k can be obtained, which can contain two components: noise and user singing.

[0093] After obtaining the de-music audio 20k, the backend server can determine the difference between the music recording audio 20b and the de-music audio 20k as the pure music audio 20p contained in the music recording audio 20b. Here, the pure music audio 20p can be the backing track played by the music playback device. Simultaneously, a frequency domain transformation can be performed on the de-music audio 20k to obtain its corresponding frequency domain power spectrum. This power spectrum is then input into the second-order deep network model 20m, which outputs the frequency gain corresponding to the de-music audio 20k. The second-order deep network model 20m can be a pre-trained network model capable of denoising noisy speech audio. The training process of the second-order deep network model 20m can be described in step S305 below. By multiplying the frequency gain output by the second-order deep network model 20m with the frequency domain power spectrum corresponding to the de-music audio 20k, a weighted speech domain signal is obtained. This weighted speech domain signal is then transformed in the time domain to obtain the denoised human voice frequency 20n. Here, the denoised human voice frequency 20n can refer to the audio signal obtained after noise suppression of the de-music audio 20k, such as the singing voice of user A in the music recording audio 20b. The first-order deep network model 20g and the second-order deep network model 20m can be deep networks with different network structures. The calculation process of the denoised human voice frequency 20n is similar to that of the de-music audio 20k and will not be elaborated further here.

[0094] The backend server can superimpose the pure music audio 20p with the denoised vocal audio 20n to obtain the denoised music recording audio 20q. By separating the pure music audio 20q from the music recording audio 20b, the denoising process of the music recording audio 20b is converted into denoising process of the denoised vocal audio 20k (which can be understood as vocal audio). This allows the denoised music recording audio 20q to retain user A's singing and backing vocals while suppressing noise in the music recording audio 20b to the greatest extent, thus improving the denoising effect of the music recording audio 20b.

[0095] Please see Figure 3 , Figure 3 This is a flowchart illustrating an audio data processing method provided in an embodiment of this application. It is understood that this audio data processing method can be executed by a computer device, which can be a user terminal, a server, or a computer program application (including program code) within the computer device; no specific limitations are made here. Figure 3 As shown, the audio data processing method may include the following steps S101-S105:

[0096] Step S101: Obtain the recorded audio; the recorded audio includes background reference audio components, speech audio components, and environmental noise components.

[0097] Specifically, the computer device can acquire recorded audio that includes background reference audio components, speech audio components, and environmental noise components. This recorded audio can be a mixed audio obtained by jointly recording and capturing audio from a target object in the target environment and an audio playback device. The recording device can be a device with recording capabilities, such as a sound card with a microphone or a mobile phone; the audio playback device can be a device with audio playback capabilities, such as a mobile phone, a music player, or a speaker; and the target object can be the user whose voice needs to be recorded, as mentioned above. Figure 2 In the corresponding embodiment, User A; the target environment can be the recording environment where the target object and the audio playback device are located, such as the indoor space or outdoor space (e.g., street, park) where the target object and the audio playback device are located. When a device has both recording and audio playback functions, the device can be used as both a recording device and an audio playback device. That is, the audio playback device and the recording device in this application can be the same device, as described above. Figure 2 The corresponding embodiment is user terminal 20a. It should be noted that the audio recording obtained by the computer device can be recording data transmitted from the recording device to the computer device, or it can be recording data collected by the computer device itself. If the computer device has both recording and audio playback functions, it can also be used as both a recording device and an audio playback device. The computer device can have audio applications installed, and the recording function in the audio application can be used to realize the above-mentioned audio recording process.

[0098] Optionally, assuming the target person wants to record their own singing, they can activate the recording function on the recording device, play backing vocals on an audio playback device, sing the song against the background music, and begin recording. After recording, the recorded music can be used as the aforementioned audio recording. This audio recording can include the backing vocals played on the audio playback device and the target person's singing. If the target environment is noisy, the audio recording can also include ambient noise from that environment. The recorded backing vocals can be used as the background audio component in the audio recording, as described above. Figure 2 The corresponding embodiment shows the music accompaniment played by the user terminal 20a; the recorded singing voice of the target object can be used as the speech audio component in the recorded audio, as described above. Figure 2 The corresponding embodiment is user A's singing voice; the recorded ambient noise can be used as the ambient noise component in the recorded audio, as described above. Figure 2The recorded audio can be used to record noise in the environment where the user terminal 20a is located, as described above. Figure 2 The corresponding embodiment includes the music recording audio 20b.

[0099] Optionally, assuming the target user wants to record their own voice-over audio, the target can activate the recording function on the recording device and play the background audio of the segment to be voiced using an audio playback device. The target then performs the voice-over based on the background audio and begins recording. After recording, the recorded voice-over audio can be used as the aforementioned audio recording. This audio recording can include the background audio played by the audio playback device and the target's voice-over. If the target environment is noisy, the audio recording can also include ambient noise from that environment. The recorded background audio can be used as the background reference audio component in the audio recording; the recorded voice-over can be used as the speech audio component; and the recorded ambient noise can be used as the ambient noise component.

[0100] In other words, the audio recordings acquired by the computer device can include audio played by the audio playback device (e.g., the background audio in the aforementioned music accompaniment or dubbing segment), the speech output by the target object (e.g., the user's dubbing or singing), and environmental noise in the target environment. It is understood that the music recording and dubbing recording scenarios described above are merely examples in this application. This application can also be applied to other audio recording scenarios, such as human-computer question-and-answer interaction scenarios between the target object and the audio playback device, and language performance scenarios (such as crosstalk performances) between the target object and the audio playback device. This application does not limit these applications.

[0101] Step S102: Obtain the audio fingerprint to be matched corresponding to the recorded audio, and obtain the prototype audio that matches the recorded audio from the audio database based on the audio fingerprint to be matched.

[0102] Specifically, the recorded audio acquired by computer equipment may contain not only the audio output by the target object and the audio played by the audio playback device, but also environmental noise from the target environment. For example, if the target environment is a shopping mall, the environmental noise in the recorded audio could be the mall's promotional announcements, the shouts of shop assistants, and the electronic noise of the recording equipment; if the target environment is an office, the environmental noise could be the sound of the air conditioner or fan, and the electronic noise of the recording equipment. Therefore, computer equipment needs to perform noise reduction processing on the acquired recorded audio. The goal of this noise reduction processing is to suppress the environmental noise in the recorded audio as much as possible, while keeping the audio output by the target object and the audio played by the audio playback device unchanged.

[0103] During the noise reduction process of recorded audio, the computer device can compress the audio data, mapping the audio to digital digest information. This digital digest information can be referred to as the audio fingerprint to be matched corresponding to the recorded audio. The data size of the audio fingerprint to be matched is much smaller than the data size of the recorded audio. The computer device can also obtain an audio database and the corresponding audio fingerprint database. It then matches the audio fingerprint to be matched with the audio fingerprints contained in the audio fingerprint database, finding the matching audio fingerprint in the database. The audio data corresponding to the matching audio fingerprint is then identified as the prototype audio corresponding to the recorded audio (e.g., the aforementioned...). Figure 2 The corresponding embodiment includes the music prototype audio 20f); in other words, the computer device can retrieve the prototype audio matching the recorded audio from the audio database based on audio fingerprint retrieval technology. The audio database can include all audio data contained in audio applications, and the audio fingerprint database can include the audio fingerprint corresponding to each audio data in the audio database. Both the audio database and the audio fingerprint database can be pre-configured; for example, when the recorded audio is music recording, the audio database can be a database containing all music sequences; when the recorded audio is dubbing recording, the audio database can be a database containing audio from all video data; and so on. When performing audio fingerprint retrieval on the recorded audio, the computer device can directly access the audio database and the audio fingerprint database to retrieve the prototype audio matching the recorded audio. The prototype audio can refer to the original audio corresponding to the audio played by the voice playback device in the recorded audio; for example, when the recorded audio is music recording, the prototype audio can be the music prototype corresponding to the accompaniment included in the music recording; when the recorded audio is dubbing recording, the prototype audio can be the prototype dubbing corresponding to the video background audio included in the dubbing recording, etc.

[0104] The audio fingerprint retrieval technology used by the computer device may include, but is not limited to: Philips audio retrieval technology (a retrieval technology that may include a highly robust fingerprint extraction method and an efficient fingerprint search strategy) and Shazam audio retrieval technology (an audio retrieval technology that may include audio fingerprint extraction and audio fingerprint matching). This application can select a suitable audio retrieval technology to retrieve the aforementioned prototype audio based on actual needs. For example, based on improvements to the above two audio fingerprint retrieval technologies, this application does not limit the audio retrieval technology used. In the audio fingerprint retrieval technology, the audio fingerprint to be matched extracted by the computer device can be represented by common audio features of the recorded audio, which may include, but are not limited to: Fourier coefficients, Mel-Frequency Cepstral Coefficients (MFCC), spectral flatness, sharpness, and LPC (Linear Prediction Coefficient) coefficients. The audio fingerprint matching algorithm used by the computer device may include, but is not limited to: distance-based matching algorithm (when the computer device finds the shortest distance between audio fingerprint A and the audio fingerprint to be matched in the audio fingerprint database, it indicates that the audio data corresponding to the audio fingerprint A is the prototype audio corresponding to the recorded audio), index-based matching method, and threshold-based matching method; this application may select appropriate audio fingerprint extraction algorithm and audio fingerprint matching algorithm according to actual needs, and this application does not limit them.

[0105] Step S103: Obtain candidate speech audio from the recorded audio based on the prototype audio; the candidate speech audio includes speech audio components and environmental noise components.

[0106] Specifically, after the computer device retrieves the prototype audio that matches the recorded audio from the audio database, it can filter the recorded audio based on the prototype audio to obtain candidate speech audio (also known as human voice signals carrying noise, as mentioned above) contained in the recorded audio. Figure 2 In the corresponding embodiment, the candidate speech audio (20k) can include the speech audio component and the environmental noise component in the recorded audio. In other words, the candidate speech audio can be understood as the recorded audio after filtering the audio output by the audio playback device. That is, the above-mentioned candidate speech audio can be obtained by removing the audio output by the audio playback device from the recorded audio.

[0107] Furthermore, the computer device can perform frequency domain transformation on the recorded audio to obtain the first spectral feature corresponding to the recorded audio; and perform frequency domain transformation on the prototype audio to obtain the second spectral feature corresponding to the prototype audio. The frequency domain transformation method in this application may include, but is not limited to: Fourier Transform (FT), Laplace Transform, Z-transform, and variations or improvements of the above three frequency domain transformation methods, such as Fast Fourier Transform (FFT), Discrete Fourier Transform (DFT), etc.; this application does not limit the frequency domain transformation method used. The first spectral feature mentioned above can be the power spectrum data obtained after frequency domain transformation of the recorded audio, or it can be the result obtained after normalizing the power spectrum data. The acquisition process of the second spectral feature mentioned above is the same as that of the first spectral feature. For example, if the first spectral feature is the power spectrum data corresponding to the recorded audio, the second spectral feature is the power spectrum data corresponding to the original audio; if the first spectral feature is the normalized power spectrum data, the second spectral feature is the normalized power spectrum data. The normalization processing method used for the first and second spectral features is the same. The normalization processing method mentioned above can include, but is not limited to, iLN (instant layer normalization), LN (Layer Normalization), IN (Instance Normalization), GN (Group Normalization), SN (Switchable Normalization), etc. This application does not limit the normalization processing method used.

[0108] The computer device can concatenate the first and second spectral features, and input the concatenated spectral features into a first deep network model (e.g., the first deep network model 20g in the embodiment corresponding to 2 above). The first deep network model can output a first frequency gain (e.g., the aforementioned...). Figure 2 In the corresponding embodiment, the frequency gain sequence 20h is multiplied by the power spectrum data corresponding to the recorded audio, and then subjected to a time-domain transformation to obtain the aforementioned candidate speech audio. The time-domain transformation here is the inverse of the frequency-domain transformation mentioned above. For example, if the frequency transformation uses a Fourier transform, the time-domain transformation used here is an inverse Fourier transform. The calculation process for the candidate speech audio can be found above. Figure 2The calculation process for removing 20kbps of music audio in the corresponding embodiment will not be elaborated here. The first deep network model described above can be used to filter the audio output by the audio playback device in the recorded audio. The first deep neural network can include, but is not limited to, Gate Recurrent Unit (GRU), Long Short Term Memory (LSTM), Deep Neural Networks (DNN), Convolutional Neural Networks (CNN), and any variation of the above network models, or a combination of two network models, etc. This application does not limit the network structure of the first deep network model used. It should be noted that the second deep network model mentioned below can also include, but is not limited to, the above network models. The second deep network model is used to perform noise reduction processing on candidate speech audio. The second deep network model can have the same network structure as the first deep network model, but have different model parameters (the two network models have different functions); or, the second deep network model and the first deep network model can have different network structures and different model parameters. The type of the second deep network model will not be elaborated hereafter.

[0109] Step S104: The difference between the recorded audio and the candidate speech audio is determined as the background reference audio component contained in the recorded audio.

[0110] Specifically, after obtaining candidate speech audio based on the first deep network model, the computer device can subtract the candidate speech audio from the recorded audio to obtain the audio output by the audio playback device; in this application, the audio output by the audio device can be referred to as the background reference audio component in the recorded audio (e.g., the aforementioned...). Figure 2 The corresponding embodiment includes pure music audio (20p). The candidate speech audio includes environmental noise components and speech audio components in the recorded audio. The result obtained by subtracting the recorded audio from the candidate speech audio is the background reference audio component contained in the recorded audio.

[0111] The difference between the recorded audio and the candidate audio can be a waveform difference in the time domain or a spectral difference in the frequency domain. When both the recorded audio and the candidate audio are time-domain waveform signals, the first signal waveform corresponding to the recorded audio and the second signal waveform corresponding to the candidate audio can be obtained. Both the first and second signal waveforms can be represented in a two-dimensional coordinate system (the horizontal axis can represent time, and the vertical axis can represent signal strength, also known as signal amplitude). The first and second signal waveforms can then be subtracted to obtain the waveform difference between the recorded audio and the candidate audio in the time domain. When subtracting the recorded audio from the candidate audio in the time domain, the horizontal axes of the first and second signal waveforms remain unchanged; only the vertical axis values ​​corresponding to the horizontal axis values ​​are subtracted to obtain a new waveform signal. This new waveform signal can be considered as the time-domain waveform signal corresponding to the background reference audio component.

[0112] Optionally, when both the recorded audio and the candidate speech audio are frequency domain signals, the recording power spectrum data corresponding to the recorded audio can be subtracted from the speech power spectrum data corresponding to the candidate speech audio to obtain the spectral difference between the two. This spectral difference can be considered as the frequency domain signal corresponding to the background reference audio component. For example, assuming the recording power spectrum data corresponding to the recorded audio is (5, 8, 10, 9, 7) and the speech power spectrum data corresponding to the candidate speech audio is (2, 4, 1, 5, 6), the spectral difference obtained after subtracting the two can be (3, 4, 9, 4, 1). This spectral difference (3, 4, 9, 4, 1) can be called the frequency domain signal corresponding to the background reference audio component.

[0113] Step S105: Perform environmental noise reduction processing on the candidate speech audio to obtain the denoised speech audio corresponding to the candidate speech audio. Merge the denoised speech audio with the background reference audio component to obtain the denoised recording audio.

[0114] Specifically, computer equipment can perform noise reduction processing on candidate speech audio, that is, suppress environmental noise in the candidate speech audio to obtain the noise-reduced speech audio corresponding to the candidate speech audio (for example, the above). Figure 2 The corresponding embodiment uses a human voice noise reduction frequency of 20 Hz.

[0115] The noise reduction processing of the candidate speech audio can be achieved through the aforementioned second deep network model. The computer device can perform frequency domain transformation on the candidate speech audio to obtain the power spectrum data (which can be called speech power spectrum data) corresponding to the candidate speech audio. This speech power spectrum data is then input into the second deep network model, which outputs a second frequency gain. Multiplying this second frequency gain by the speech power spectrum data corresponding to the candidate speech audio and then performing a time domain transformation yields the aforementioned noise-reduced speech audio. Furthermore, the noise-reduced speech audio can be superimposed with the aforementioned background reference audio components to obtain the noise-reduced recording audio (e.g., the aforementioned...). Figure 2 The noise-reduced music recording audio 20q in the corresponding embodiment.

[0116] Optionally, the computer device can share the denoised audio recording to a social media platform, so that terminal devices on the social media platform can play the denoised audio recording when accessing it. Here, the aforementioned social media platform refers to applications, web pages, etc., that can be used to share and disseminate audio and video data. For example, the social media platform can be an audio application, a video application, or a content sharing platform, etc.

[0117] For example, in a music recording scenario, the denoised audio recording can be used as a music recording. A computer device can share this denoised audio recording to a content sharing platform (in this case, the social media platform is the default content sharing platform). When a terminal device accesses the denoised audio recording shared on the content sharing platform, it can play the denoised audio recording. Please see [link to relevant documentation]. Figure 4 , Figure 4 This is a schematic diagram of a music recording scenario provided in an embodiment of this application. Figure 4The server 30a shown can be the backend server of the content sharing platform, and the user terminal 30b can be the terminal device used by user A, who shares the noise-reduced music recording audio 30e on the content sharing platform. User terminal 30c can be the terminal device used by user B, and user terminal 30d can be the terminal device used by user C. When server 30a obtains the noise-reduced music recording audio 30e, it can share it to the content sharing platform. At this time, the content sharing platform on user terminal 30b can display the noise-reduced music recording audio 30e, as well as information such as the sharing time of the noise-reduced music recording audio 30e. When user terminal 30c used by user B accesses the content sharing platform, it can display content shared by different users, which may include the noise-reduced music recording audio 30e shared by user A. Clicking on the noise-reduced music recording audio 30e will play it on user terminal 30c. Similarly, when user C accesses the content sharing platform using user terminal 30d, the noise-reduced music recording audio 30e shared by user A can be displayed on the content sharing platform of user terminal 30d. After clicking on the noise-reduced music recording audio 30e, the noise-reduced music recording audio 30e can be played on user terminal 30d.

[0118] In this embodiment, the recorded audio can be a mixture of speech audio components, background reference audio components, and environmental noise components. During the noise reduction process, the prototype audio corresponding to the recorded audio can be found through audio fingerprint retrieval. Based on this prototype audio, candidate speech audio can be selected from the recorded audio. Subtracting the candidate speech audio from the recorded audio yields the background reference audio component. Noise reduction processing can then be applied to the candidate speech audio to obtain denoised speech audio. Superimposing the denoised speech audio with the background reference audio component yields the denoised recorded audio. In other words, by transforming the noise reduction problem of the recorded audio into a noise reduction problem of the candidate speech audio, the misunderstanding of the background reference audio component in the recorded audio being confused with environmental noise can be avoided, thereby improving the noise reduction effect of the recorded audio.

[0119] Please see Figure 5 , Figure 5 This is a flowchart illustrating an audio data processing method provided in an embodiment of this application. It is understood that this audio data processing method can be executed by a computer device, which can be a user terminal, a server, or a computer program application (including program code) within the computer device; no specific limitations are made here. Figure 3As shown, the audio data processing method may include the following steps S101-S105:

[0120] Step S201: Obtain the recorded audio; the recorded audio includes background reference audio components, speech audio components, and environmental noise components.

[0121] The specific implementation of step S201 can be found above. Figure 3 Step S101 in the corresponding embodiment will not be described again here.

[0122] Step S202: Divide the recorded audio into M recording data frames, and perform frequency domain transformation on the i-th recording data frame among the M recording data frames to obtain the power spectrum data corresponding to the i-th recording data frame; i and M are both positive integers, and i is less than or equal to M.

[0123] Specifically, the computer device can perform frame segmentation processing on the recorded audio, dividing the recorded audio into M recording data frames. A frequency domain transformation is then performed on the i-th recording data frame, such as a Fourier transform, to obtain the power spectrum data corresponding to the i-th recording data frame. Here, M can be a positive integer greater than 1, such as M taking values ​​of 2, 3, ..., and i can be a positive integer less than or equal to M. The computer device can implement frame segmentation processing of the recorded audio using a sliding window, thus obtaining M recording data frames. To maintain the continuity between adjacent recording data frames, an overlapping segmentation method is typically used for frame segmentation. The size of the recording data frame can be related to the size of the sliding window.

[0124] For each of the M recorded data frames, a frequency domain transformation (such as Fourier transform) can be performed independently to obtain the power spectrum data corresponding to each recorded data frame. The power spectrum data can include the energy value corresponding to each frequency point (the energy value here can also be called the amplitude of the power spectrum data). One energy value in the power spectrum data corresponds to one frequency point, and one frequency point can be understood as one frequency sampling point during frequency domain transformation.

[0125] Step S203: Divide the power spectrum data corresponding to the i-th recording data frame into N spectrum bands, and construct the sub-fingerprint information corresponding to the i-th recording data frame based on the peak signals in the N spectrum bands; N is a positive integer.

[0126] Specifically, the computer device can construct sub-fingerprint information for each recorded data frame based on the power spectrum data corresponding to each recorded data frame. The key to constructing sub-fingerprint information lies in selecting the energy value with the highest distinguishability from the power spectrum data corresponding to each recorded data frame. The following description uses the i-th recorded data frame as an example to illustrate the sub-fingerprint information construction process. The computer device can divide the power spectrum data corresponding to the i-th recorded data frame into N spectrum bands, selecting the peak signal (i.e., the maximum value in each spectrum band, which can also be understood as the maximum energy value in each spectrum band) as the signature of that spectrum band, thereby constructing the sub-fingerprint information corresponding to the i-th recorded data frame. N can be a positive integer, such as 1, 2, ... In other words, the sub-fingerprint information corresponding to the i-th recorded data frame can include the peak signals corresponding to each of the N spectrum bands.

[0127] Step S204: According to the time order of the M recording data frames in the recorded audio, combine the sub-fingerprint information corresponding to the M recording data frames respectively to obtain the audio fingerprint to be matched corresponding to the recorded audio.

[0128] Specifically, the computer device can obtain the sub-fingerprint information corresponding to each of the M recording data frames as described in step S203 above. Then, according to the chronological order of the M recording data frames in the recorded audio, the sub-fingerprint information corresponding to each of the M recording data frames can be combined sequentially to obtain the audio fingerprint to be matched corresponding to the recorded audio. By selecting the peak signal to construct the audio fingerprint to be matched, it is possible to ensure that the audio fingerprint to be matched remains unchanged under various noise and distortion environments as much as possible.

[0129] Step S205: Obtain the audio fingerprint database corresponding to the audio database, perform fingerprint retrieval in the audio fingerprint database according to the audio fingerprint to be matched, and determine the prototype audio that matches the recorded audio in the audio database according to the fingerprint retrieval result.

[0130] Specifically, the computer device can acquire an audio database and a corresponding audio fingerprint database. Each audio data point in the audio database can be assigned an audio fingerprint according to steps S201-S204 described above. The audio fingerprints corresponding to each audio data point constitute the audio fingerprint database corresponding to the audio database. The audio fingerprint database is pre-built. After acquiring the audio fingerprint to be matched corresponding to the recorded audio, the computer device can directly acquire the audio fingerprint database and perform a fingerprint search within it. This retrieves the audio fingerprint that matches the audio fingerprint to be matched. The matching audio fingerprint can serve as the fingerprint search result corresponding to the audio fingerprint to be matched, and the audio data corresponding to the fingerprint search result can then be identified as the prototype audio that matches the recorded audio.

[0131] Optionally, the computer device can store audio fingerprints as keys in an audio retrieval hash table. Each audio data frame can correspond to a sub-fingerprint, and each sub-fingerprint can correspond to a key in the audio retrieval hash table. The sub-fingerprints corresponding to all audio data frames in each audio data set can form the audio fingerprint corresponding to that audio data. For ease of searching, each sub-fingerprint can be used as a key in the hash table. Each key can point to the time when the sub-fingerprint appears in its corresponding audio data, and can also point to the identifier of the audio data to which the sub-fingerprint belongs. For example, after a sub-fingerprint is converted into a hash value, this hash value can be stored as a key in the audio retrieval hash table. This key points to the time when the sub-fingerprint appears in its corresponding audio data as 02:30, and the identifier of the audio data it points to is: Audio Data 1. Understandably, the aforementioned audio fingerprint database can include one or more hash values ​​corresponding to each audio data set in the audio database.

[0132] When a recorded audio is divided into M audio data frames, the audio fingerprint to be matched corresponding to the recorded audio can include M sub-fingerprint information, with one sub-fingerprint information corresponding to one audio data frame. The computer device can map the M sub-fingerprint information contained in the audio fingerprint to be matched into M hash values ​​to be matched, and obtain the recording time corresponding to each of the M hash values ​​to be matched. The recording time corresponding to a hash value to be matched is used to characterize the time when the sub-fingerprint information corresponding to the hash value to be matched appears in the recorded audio. If the p-th hash value to be matched among the M hash values ​​to be matched matches the first hash value contained in the audio fingerprint database, then the first time difference between the recording time corresponding to the p-th hash value to be matched and the time information corresponding to the first hash value is obtained, where p is a positive integer less than or equal to M. If the q-th hash value to be matched among the M hash values ​​to be matched matches the second hash value contained in the audio fingerprint database, then the second time difference between the recording time corresponding to the q-th hash value to be matched and the time information corresponding to the second hash value is obtained, where q is a positive integer less than or equal to M. When the first time difference and the second time difference meet the numerical threshold, and the first hash value and the second hash value belong to the same audio fingerprint, the audio fingerprint to which the first hash value belongs can be determined as the fingerprint retrieval result, and the audio data corresponding to the fingerprint retrieval result can be determined as the prototype audio corresponding to the recorded audio. Furthermore, the computer device can match the above M hash values ​​to be matched with the hash values ​​in the audio fingerprint database. For each successfully matched hash value, a time difference can be calculated. After all M hash values ​​to be matched have been matched, the maximum value of the same time difference can be counted. At this time, the maximum value can be set as the above numerical threshold, and the audio data corresponding to the maximum value is determined as the prototype audio corresponding to the recorded audio.

[0133] For example, there are M hash values ​​to be matched, including hash value 1, hash value 2, hash value 3, hash value 4, hash value 5, and hash value 6. Hash value A in the audio fingerprint database matches hash value 1, and hash value A points to audio data 1. The time difference between hash value A and hash value 1 is t1. Hash value B in the audio fingerprint database matches hash value 2, and hash value B points to audio data 1. The time difference between hash value B and hash value 2 is t2. Hash value C in the audio fingerprint database matches hash value 3, and hash value C points to audio data 1. Data 1 has a time difference of t3 between hash value C and hash value 3; hash value D in the audio fingerprint database matches hash value 4, and hash value D points to audio data 1, with a time difference of t4 between hash value D and hash value 4; hash value E in the audio fingerprint database matches hash value 5, and hash value E points to audio data 2, with a time difference of t5 between hash value E and hash value 5; hash value F in the audio fingerprint database matches hash value 6, and hash value 6 points to audio data 2, with a time difference of t6 between hash value F and hash value 6. If the time differences t1, t2, t3, and t4 are the same, and the time differences t5 and t6 are the same, then audio data 1 can be used as the prototype audio corresponding to the recorded audio.

[0134] Step S206: Obtain the recording power spectrum data corresponding to the recorded audio, normalize the recording power spectrum data to obtain the first spectral feature; obtain the prototype power spectrum data corresponding to the prototype audio, normalize the prototype power spectrum data to obtain the second spectral feature, and combine the first spectral feature and the second spectral feature into the input feature.

[0135] Specifically, the computer device can acquire the recording power spectrum data corresponding to the recorded audio. This recording power spectrum data can be composed of the power spectrum data corresponding to the M audio data frames mentioned above. The recording power spectrum data can include the energy values ​​corresponding to each frequency point in the recorded audio. The recording power spectrum data is normalized to obtain a first spectral feature. If the normalization process here is iLN, the energy values ​​corresponding to each frequency point in the recording power spectrum data can be independently normalized. Of course, other normalization processes, such as BN, can also be used in this application. Optionally, embodiments of this application can also directly use the recording power spectrum data as the first spectral feature without performing normalization processing on the recording power spectrum data. Similarly, for the prototype audio, the same frequency domain transformation (to obtain prototype power spectrum data) and normalization processing operations as described above for the recorded audio can be performed to obtain the second spectral feature corresponding to the prototype audio. Then, the first spectral feature and the second spectral feature can be combined into an input feature by concat.

[0136] Step S207: Input the input features into the first deep network model, and output the first frequency gain for the recorded audio through the first deep network model.

[0137] Specifically, the computer device can input the input features into the first deep network model, and the first deep network model can output the first frequency gain for the recorded audio. The first frequency gain can include the speech gain corresponding to each frequency point in the recorded audio.

[0138] When the first deep network model includes a GRU (which can be used as a feature extraction network layer), a fully connected network (which can be used as a fully connected network layer), and a Sigmoid function (which can be called an activation layer and can be used as an output layer in this application), the input features are first input to the feature extraction network layer in the first deep network model. Based on the feature extraction network layer, the temporal distribution features corresponding to the input features can be obtained. These temporal distribution features can be used to represent the contextual semantics in the recorded audio. Based on the fully connected network layer in the first deep network model, the target feature vector corresponding to the temporal distribution features is obtained. Through the activation layer in the first deep network model, the first frequency point gain is output. For example, the speech gain corresponding to each frequency point contained in the recorded audio can be output by the Sigmoid function.

[0139] Step S208: Based on the first frequency gain and recording power spectrum data, obtain the candidate speech audio contained in the recorded audio; determine the difference between the recorded audio and the candidate speech audio as the background reference audio component contained in the recorded audio; the candidate speech audio includes speech audio components and environmental noise components.

[0140] Specifically, assuming the recorded audio includes T frequency points (T is a positive integer greater than 1), the first frequency point gain can include the speech gains corresponding to each of the T frequency points, and the recording power spectrum data includes the energy values ​​corresponding to each of the T frequency points. There is a one-to-one correspondence between the T speech gains and the T energy values. The computer device can weight the energy values ​​belonging to the same frequency point in the recording power spectrum data based on the speech gains corresponding to the T frequency points in the first frequency point gain, obtaining the weighted energy values ​​corresponding to the T frequency points. Then, based on the weighted energy values ​​corresponding to the T frequency points, the weighted recording audio domain signal corresponding to the recorded audio can be determined. By performing a time-domain transformation on the weighted recording audio domain signal (which is the inverse of the aforementioned frequency-domain transformation), the candidate speech audio contained in the recorded audio can be obtained. For example, when the first frequency gain output by the first deep network model is (2, 3) and the recording power spectrum data is (1, 2), it indicates that the recorded audio may include two frequency points (here, T is 2). The first frequency point has a speech gain of 2 in the first frequency gain and an energy value of 1 in the recording power spectrum data. The second frequency point has a speech gain of 3 in the first frequency gain and an energy value of 2 in the recording power spectrum data. The weighted audio domain signal can be calculated as (2, 6). By performing a time-domain transformation on the weighted audio domain signal, the candidate speech audio contained in the recorded audio can be obtained. Furthermore, the difference between the recorded audio and the candidate speech audio can be determined as the background reference audio component, i.e., the audio output by the audio playback device.

[0141] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a first deep network model provided in an embodiment of this application; taking a music recording scenario as an example, the network structure of the first deep network model is explained. Figure 6As shown, after the computer device retrieves the music prototype audio 40b (i.e., prototype audio) corresponding to the music recording audio 40a (i.e., the recorded audio) from the audio database, it can perform Fast Fourier Transform (FFT) on the music recording audio 40a and the music prototype audio 40b respectively to obtain the power spectrum data 40c (i.e., the recorded power spectrum data) corresponding to the music recording audio 40a and the power spectrum data 40d (i.e., the prototype power spectrum data) corresponding to the music prototype audio 40b. The above-mentioned Fast Fourier Transform is only an example in this embodiment. This application can also use other frequency domain transformation methods, such as Discrete Fourier Transform. After iLN normalization of the power spectra of each frame in power spectrum data 40c and power spectrum data 40d, features are combined using concat. The combined input features are used as input data for the first deep network model 40e. This first deep network model 40e can be composed of gated recurrent unit 1, gated recurrent unit 2, and fully connected network 1. Finally, the first frequency gain is output through the Sigmoid function. The speech gain of each frequency point included in the first frequency gain is multiplied by the energy value (also called the frequency power spectrum) of the corresponding frequency point in power spectrum data 40c, and then subjected to inverse Fourier transform (iFFT) to obtain the music audio 40f (i.e., the aforementioned candidate speech audio). The inverse Fourier transform can be a time-domain transform method, that is, transforming from the frequency domain to the time domain. It can be understood that, as Figure 6 The network structure of the first deep network model 40e shown is only an example. The first deep network model used in this application embodiment can also be based on the first deep network model 40e by adding gated recurrent units or fully connected network structures. This application does not limit this.

[0142] Step S209: Obtain the speech power spectrum data corresponding to the candidate speech audio, input the speech power spectrum data into the second deep network model, and output the second frequency point gain for the candidate speech audio through the second deep network model.

[0143] Specifically, after acquiring candidate speech audio, the computer device can perform frequency domain transformation on the candidate speech audio to obtain the speech power spectrum data corresponding to the candidate speech audio. The speech power spectrum data is then input into a second deep network model. Through the feature extraction network layer (which can be GRU), the fully connected network layer (which can be a fully connected network), and the activation layer (Sigmoid function) in the second deep network model, a second frequency point gain for the candidate speech audio can be output. The second frequency point gain can include the noise reduction gain corresponding to each frequency point in the candidate speech audio, and can be the output value of the Sigmoid function.

[0144] Step S210: Based on the second frequency gain and speech power spectrum data, obtain the weighted speech domain signal corresponding to the candidate speech audio; perform time-domain transformation on the weighted speech domain signal to obtain the denoised speech audio corresponding to the candidate speech audio; merge the denoised speech audio with the background reference audio component to obtain the denoised recording audio.

[0145] Specifically, assuming the candidate speech audio includes D frequency points (D is a positive integer greater than 1, where D can be equal to or different from the aforementioned T, and the values ​​of D and T are not limited in this application), then the second frequency point gain can include the noise reduction gain corresponding to each of the D frequency points, and the speech power spectrum data includes the energy values ​​corresponding to each of the D frequency points. The D noise reduction gains correspond one-to-one with the D energy values. The computer device can weight the energy values ​​belonging to the same frequency point in the speech power spectrum data based on the noise reduction gains corresponding to the D frequency points in the second frequency point gain, obtaining the weighted energy values ​​corresponding to the D frequency points. Then, based on the weighted energy values ​​corresponding to the D frequency points, the weighted speech audio domain signal corresponding to the candidate speech audio can be determined. By performing a time-domain transformation on the weighted speech audio domain signal (which is the inverse of the aforementioned frequency-domain transformation), the noise-reduced speech audio corresponding to the candidate speech audio can be obtained. For example, when the second frequency gain output by the second deep network model is (0.1, 0.5) and the speech power spectrum data is (5, 8), it indicates that the candidate speech audio can include two frequency points (here, D is 2). The first frequency point has a noise reduction gain of 0.1 in the second frequency gain and an energy value of 5 in the speech power spectrum data. The second frequency point has a noise reduction gain of 0.5 in the second frequency gain and an energy value of 8 in the speech power spectrum data. The weighted speech domain signal can be calculated as (0.5, 4). By performing a time-domain transformation on the weighted speech domain signal, the noise-reduced speech audio corresponding to the candidate speech audio can be obtained. Furthermore, the noise-reduced speech audio can be superimposed with the background reference audio component to obtain the noise-reduced recording audio.

[0146] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a second deep network model provided in an embodiment of this application. For example... Figure 7 As shown above, Figure 6In the corresponding embodiment, after the computer device obtains the music-de-sound audio 40f through the first deep network model 40e, it can perform a Fast Fourier Transform (FFT) on the music-de-sound audio 40f to obtain the power spectrum data 40g corresponding to the music-de-sound audio 40f (i.e., the aforementioned speech power spectrum data). The power spectrum data 40g is used as input data for the second deep network model 40h, which can be composed of a fully connected network 2, a gated loop unit 3, a gated loop unit 4, and a fully connected network 3. Finally, the second frequency gain can be output through the Sigmoid function. The noise reduction gain of each frequency point included in the second frequency gain is multiplied by the energy value of the corresponding frequency point in the power spectrum data 40g, and then subjected to an inverse Fourier Transform (iFFT) to obtain the human voice noise-de-sound frequency 40i (i.e., the aforementioned noise-de-sound audio). It is understood that, as Figure 7 The network structure of the second deep network model 40h shown is only an example. The second deep network model used in this application embodiment can also be based on the second deep network model 40h by adding gated recurrent units or fully connected network structures. This application does not limit this.

[0147] Please see Figure 8 , Figure 8 This is a schematic diagram of a recording audio noise reduction process provided in an embodiment of this application. Figure 8 As shown, this embodiment takes a music recording scenario as an example. After the computer device obtains the music recording audio 50a, it can obtain the audio fingerprint 50b corresponding to the music recording audio 50a. Based on the audio fingerprint 50b, it performs audio fingerprint retrieval in the audio fingerprint database 50d corresponding to the music library 50c (i.e., the aforementioned audio database). When the audio fingerprint corresponding to a certain audio data in the music library 50c matches the audio fingerprint 50b, the audio data in block 50c can be identified as the music prototype audio 50e corresponding to the music recording audio 50a. The extraction process of the audio fingerprint 50b and the audio fingerprint retrieval process of the audio fingerprint 50b can be referred to the description in the aforementioned steps S202-S205, and will not be repeated here.

[0148] Furthermore, spectral features can be extracted from the music recording audio 50a and the original music audio 50e respectively. The obtained spectral features are then combined and input into the first-order deep network 50h (i.e., the aforementioned first-order deep network model). The music audio 50i can be obtained through the first-order deep network 50h (the process of obtaining the music audio 50i can be found above). Figure 6The corresponding embodiments will not be described in detail here; the spectral feature extraction process may include frequency domain transformation such as Fourier transform and normalization processing such as iLN. Then, by subtracting the music recording audio 50a from the music-free audio 50i, the pure music audio 50j (i.e., the aforementioned background reference audio component) can be obtained.

[0149] After performing a Fast Fourier Transform on the de-audio audio 50i, its corresponding power spectrum data can be obtained. This power spectrum data is used as the input of the second-order deep network 50k (i.e., the second-order deep network model mentioned above). The second-order deep network 50k can then be used to obtain the de-noising human voice frequency 50m (the process of obtaining the de-noising human voice frequency 50m can be found in the above). Figure 7 The corresponding embodiments will not be described in detail here; then the pure music audio 50j and the human voice noise-reduced audio 50m can be superimposed to obtain the final noise-reduced music recording audio 50n (i.e. the noise-reduced recording audio).

[0150] In this embodiment, the recorded audio can be a mixed audio component containing a speech audio component, a background reference audio component, and an environmental noise component. During the noise reduction process, the prototype audio corresponding to the recorded audio can be found through audio fingerprint retrieval. Based on this prototype audio, candidate speech audio can be selected from the recorded audio. Subtracting the candidate speech audio from the recorded audio yields the background reference audio component. Noise reduction processing can then be performed on the candidate speech audio to obtain the denoised speech audio. Superimposing the denoised speech audio with the background reference audio component yields the denoised recorded audio. In other words, by transforming the noise reduction problem of the recorded audio into a noise reduction problem of the candidate speech audio, the misunderstanding of the background reference audio component in the recorded audio being confused with environmental noise can be avoided, thereby improving the noise reduction effect of the recorded audio. Using audio fingerprint retrieval technology to obtain the prototype audio improves retrieval accuracy.

[0151] Before using the aforementioned first and second deep network models in a recording scenario, they need to be trained. The following will demonstrate this through appendices. Figure 9 and attached Figure 10 The training process of the first deep network model and the second deep network model is described.

[0152] Please see Figure 9 , Figure 9 This is a flowchart illustrating an audio data processing method provided in an embodiment of this application. It is understood that this audio data processing method can be executed by a computer device, which can be a user terminal, a server, or a computer program application (including program code) within the computer device; no specific limitations are made here. Figure 9As shown, the audio data processing method may include the following steps S301-S305:

[0153] Step S301: Obtain speech sample audio, noise sample audio, and standard sample audio; generate sample recording audio based on speech sample audio, noise sample audio, and standard sample audio.

[0154] Specifically, computer equipment can pre-acquire a large number of speech sample audios, a large number of noise sample audios, and a large number of standard sample audios. The speech sample audios can be audio sequences containing only human voices; for example, these could be pre-recorded sequences of various users singing or dubbing for various users. The noise sample audios can be audio sequences containing only noise, which could be pre-recorded noises from different scenarios; for example, noise sample audios could be various types of noise such as vehicle horns, keyboard tapping, or the sound of striking various metal objects. The standard sample audios can be clean audio stored in an audio database; for example, these standard sample audios could be music sequences or video dubbing sequences. In other words, the speech sample audios and noise sample audios can be acquired through recording, and the standard sample audios can be clean audio stored on various platforms. Computer equipment needs authorization from a platform to acquire standard sample audios from that platform. For example, in a music recording scenario, the speech sample audios could be sequences of human voices, the noise sample audios could be sequences of noise from different scenarios, and the standard sample audios could be music sequences.

[0155] Computer equipment can superimpose speech sample audio, noise sample audio, and standard sample audio to obtain sample recording audio. To construct more sample recording audio, not only can different speech sample audio, noise sample audio, and standard sample audio be randomly combined, but different coefficients can also be used to weight the same set of speech sample audio, noise sample audio, and standard sample audio, resulting in different sample recording audio. Specifically, the computer equipment can obtain a set of weighted coefficients for a first initial network model. This set of weighted coefficients can be a set of randomly generated floating-point numbers. K arrays can be constructed based on this set of weighted coefficients. Each array can contain three values ​​in a specific order. Three values ​​with different orders can form different arrays. The three values ​​in an array represent the coefficients of the speech sample audio, noise sample audio, and standard sample audio, respectively. Based on the coefficients contained in the j-th array of the K arrays, the speech sample audio, noise sample audio, and standard sample audio are weighted respectively to obtain the sample recording audio corresponding to the j-th array. In other words, for any speech sample audio, a noise sample audio, and a standard sample audio, K different sample recording audio can be constructed.

[0156] For example, suppose the K arrays include the following four arrays (where K is 4): [0.1, 0.5, 0.3], [0.5, 0.6, 0.8], [0.6, 0.1, 0.4], and [1, 0.7, 0.3]. For the speech sample audio a, the noise sample audio b, and the standard sample audio c, the following sample audio recordings can be constructed: Sample audio recording y1 = 0.1a + 0.5b + 0.3c, Sample audio recording y2 = 0.5a + 0.6b + 0.8c, Sample audio recording y3 = 0.6a + 0.1b + 0.4c, and Sample audio recording y4 = a + 0.7b + 0.3c.

[0157] Step S302: Obtain the sample predicted speech audio from the sample recorded audio according to the first initial network model; the first initial network model is used to filter the standard sample audio contained in the sample recorded audio, and the expected predicted speech audio of the first initial network model is determined by the speech sample audio and the noise sample audio.

[0158] Specifically, for all sample audio recordings used to train the two initial network models (including the first and second initial network models), the processing of each sample audio recording is the same in both initial network models. During the training phase, the sample audio recordings can be input into the first initial network model in batches, that is, all sample audio recordings are trained in batches. For ease of description, the training process of the two initial network models is described below using any one of the sample audio recordings as an example.

[0159] Please see Figure 10 , Figure 10 This is a schematic diagram illustrating the training of a deep network model provided in an embodiment of this application. For example... Figure 10As shown, the sample audio recording y can be determined by the speech sample audio x1, the noise sample sequence x2, and the standard sample audio in the sample database 60a, such as sample audio recording y = r1×x1 + r2×x2 + r3×x3. The computer device can perform a frequency domain transformation on the sample audio recording y to obtain the sample power spectrum data corresponding to the sample audio recording y, and then normalize the sample power spectrum data (e.g., iLN normalization) to obtain the sample spectral features corresponding to the sample audio recording y. These sample spectral features are then input into the first initial network model 60b, which outputs the first sample frequency gain corresponding to the sample spectral features. This first sample frequency gain can include the speech gain of each frequency point corresponding to the sample audio recording. The first sample frequency gain here is the actual output result of the first initial network model 60b for the aforementioned sample audio recording y. The first initial network model 60b can refer to a first deep network model in the training phase, trained to filter the standard sample audio contained in the sample audio recording.

[0160] The computer device can obtain the sample predicted speech audio 60c based on the first sample frequency point gain and sample power spectrum data. The calculation process of the sample predicted speech audio 60c is similar to the calculation process of the aforementioned candidate speech audio, and will not be repeated here. Among them, the expected predicted speech audio corresponding to the first initial network model 60b can be determined by the speech sample audio x1 and the noise sample audio x2. The expected predicted speech audio can be the signal (r1×x1+r2×x2) in the aforementioned sample recorded audio y; that is, the expected output result of the first initial network model 60b can be the square root result of the energy value (or power spectrum value) of each frequency point in the power spectrum data of the signal (r1×x1+r2×x2) divided by the corresponding frequency point energy value in the sample power spectrum data.

[0161] Step S303: Obtain the sample prediction noise reduction frequency corresponding to the sample prediction speech audio according to the second initial network model; the second initial network model is used to suppress the noise sample audio contained in the sample prediction speech audio, and the expected prediction noise reduction frequency of the second initial network model is determined by the speech sample audio.

[0162] Specifically, such as Figure 10As shown, the computer device can input the power spectrum data corresponding to the sample predicted speech audio 60c into the second initial network model 60f. The second initial network model 60f can output the second sample frequency point gain corresponding to the sample predicted speech audio 60c. This second sample frequency point gain can include the noise reduction gain of each frequency point corresponding to the sample predicted speech audio 60c. The second sample frequency point gain here is the actual output result of the second initial network model 60f for the aforementioned sample predicted speech audio 60c. Here, the second initial network model 60f can refer to a second deep network model in the training phase. Training the second initial network model 60f is to suppress the noise contained in the sample predicted speech audio. It should be noted that the training samples of the second initial network model 60f need to be aligned with some samples of the first initial network model 60b. For example, the training samples of the second initial network model 60f can be the sample predicted speech audio 60c determined based on the first initial network model 60b.

[0163] The computer device can obtain the sample predicted noise reduction frequency 60g based on the second sample frequency gain and the power spectrum data of the sample predicted speech audio 60c. The calculation process of the sample predicted noise reduction frequency 60g is similar to the calculation process of the aforementioned noise reduction speech audio, and will not be repeated here. Among them, the expected predicted noise reduction frequency corresponding to the second initial network model 60f can be determined by the speech sample audio x1. The expected predicted noise reduction frequency can be the signal (r1×x1) in the aforementioned sample recorded audio y; that is, the expected output result of the second initial network model 60f can be the square root of the energy value (or power spectrum value) of each frequency point in the power spectrum data of the signal (r1×x1) divided by the corresponding frequency point energy value in the power spectrum data of the sample predicted speech audio 60c.

[0164] Step S304: Based on the sample predicted speech audio and the expected predicted speech audio, the network parameters of the first initial network model are adjusted to obtain the first deep network model; the first deep network model is used to filter the recorded audio to obtain candidate speech audio, the recorded audio includes background reference audio components, speech audio components and environmental noise components, and the candidate speech audio includes speech audio components and environmental noise components.

[0165] Specifically, such as Figure 10As shown, based on the difference between the sample predicted speech audio 60c and the expected predicted speech audio (r1×x1+r2×x2) corresponding to the first initial network model 60b, a first loss function 60d is determined for the first initial network model 60b. By optimizing the first loss function 60d to its minimum value, i.e., minimizing the training loss, the network parameters of the first initial network model 60b are adjusted until the number of training iterations reaches the preset maximum number of iterations (or the training of the first initial network model 60b reaches convergence). At this point, the first initial network model 60b can be used as the first deep network model 60e. The trained first deep network model 60e can be used to filter the recorded audio to obtain candidate speech audio. The usage process of the first deep network model 60e can be referred to the description in step S207 above. Optionally, the first loss function 60d can also be the square term between the expected output result of the first initial network model 60b and the first frequency point gain (actual output result).

[0166] Step S305: Based on the sample predicted noise reduction frequency and the expected predicted noise reduction frequency, adjust the network parameters of the second initial network model to obtain the second deep network model; the second deep network model is used to perform noise reduction processing on the candidate speech audio to obtain the noise-reduced speech audio.

[0167] Specifically, such as Figure 10 As shown, based on the difference between the sample predicted noise reduction frequency 60g and the expected predicted speech audio (r1×x1) corresponding to the second initial network model 60f, a second loss function 60h is determined for the second initial network model 60f. By optimizing the second loss function 60h to its minimum value, i.e., minimizing the training loss, the network parameters of the second initial network model 60f are adjusted until the number of training iterations reaches the preset maximum number of iterations (or the training of the second initial network model 60f reaches convergence). At this point, the second initial network model can be used as the second deep network model 60i. The trained second deep network model 60i can be used to denoise the candidate speech audio to obtain the denoised speech audio. The usage process of the second deep network model 60i can be referred to the description in step S209 above. Optionally, the second loss function 60h can also be the square term between the expected output result of the second initial network model 60f and the second frequency point gain (actual output result).

[0168] In this embodiment, by weighting the speech sample audio, noise sample audio, and standard sample audio with different coefficients, the number of sample audio recordings can be expanded. Training the first and second initial network models with these sample audio recordings can improve the generalization ability of the network models. Aligning the training samples of the second initial network model with a portion of the training samples of the first initial network model (partial signals contained in the sample audio recordings) can enhance the overall correlation between the first and second initial network models. When using the trained first and second deep network models for noise reduction, the noise reduction effect of the audio recordings can be improved.

[0169] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of an audio data processing device provided in an embodiment of this application. Figure 11 As shown, the audio data processing device 1 may include: an audio acquisition module 11, a fingerprint retrieval module 12, an audio filtering module 13, an audio determination module 14, and a noise reduction processing module 15.

[0170] Audio acquisition module 11 is used to acquire recorded audio; the recorded audio includes background reference audio components, speech audio components and environmental noise components;

[0171] The fingerprint retrieval module 12 is used to obtain the audio fingerprint to be matched corresponding to the recorded audio, and to obtain the prototype audio that matches the recorded audio from the audio database based on the audio fingerprint to be matched.

[0172] Audio filtering module 13 is used to obtain candidate speech audio from the recorded audio based on the prototype audio; the candidate speech audio includes speech audio components and environmental noise components;

[0173] The audio determination module 14 is used to determine the difference between the recorded audio and the candidate speech audio as the background reference audio component contained in the recorded audio.

[0174] The noise reduction processing module 15 is used to perform environmental noise reduction processing on the candidate speech audio to obtain the noise-reduced speech audio corresponding to the candidate speech audio, and to merge the noise-reduced speech audio with the background reference audio component to obtain the noise-reduced recording audio.

[0175] The specific functional implementation methods of the audio acquisition module 11, fingerprint retrieval module 12, audio filtering module 13, audio determination module 14, and noise reduction processing module 15 can be found above. Figure 3 Steps S101-S105 in the corresponding embodiments will not be described again here.

[0176] In one or more embodiments, the fingerprint retrieval module 12 may include: a frequency domain transformation unit 121, a spectrum band division unit 122, an audio fingerprint combination unit 123, and a prototype audio matching unit 124.

[0177] The frequency domain transformation unit 121 is used to divide the recorded audio into M recording data frames, and to perform frequency domain transformation on the i-th recording data frame among the M recording data frames to obtain the power spectrum data corresponding to the i-th recording data frame; i and M are both positive integers, and i is less than or equal to M.

[0178] The spectrum band division unit 122 is used to divide the power spectrum data corresponding to the i-th recording data frame into N spectrum bands, and construct the sub-fingerprint information corresponding to the i-th recording data frame based on the peak signals in the N spectrum bands; N is a positive integer.

[0179] The audio fingerprint combination unit 123 is used to combine the sub-fingerprint information corresponding to the M recording data frames according to the time order of the M recording data frames in the recorded audio, so as to obtain the audio fingerprint to be matched corresponding to the recorded audio.

[0180] The prototype audio matching unit 124 is used to obtain the audio fingerprint library corresponding to the audio database, perform fingerprint retrieval in the audio fingerprint library according to the audio fingerprint to be matched, and determine the prototype audio that matches the recorded audio in the audio database according to the fingerprint retrieval result.

[0181] Specifically, the prototype audio matching unit 124 is used for:

[0182] The M sub-fingerprint information contained in the audio fingerprint to be matched is mapped to M hash values ​​to be matched, and the recording time corresponding to each of the M hash values ​​to be matched is obtained; the recording time corresponding to a hash value to be matched is used to characterize the time when the sub-fingerprint information corresponding to the hash value to be matched appears in the recorded audio.

[0183] If the p-th hash value among the M hash values ​​to be matched matches the first hash value contained in the audio fingerprint database, then obtain the first time difference between the recording time corresponding to the p-th hash value and the time information corresponding to the first hash value; p is a positive integer less than or equal to M;

[0184] If the q-th hash value among the M hash values ​​to be matched matches the second hash value contained in the audio fingerprint database, then obtain the second time difference between the recording time corresponding to the q-th hash value and the time information corresponding to the second hash value; q is a positive integer less than or equal to M;

[0185] When the first time difference and the second time difference meet the numerical threshold, and the first hash value and the second hash value belong to the same audio fingerprint, the audio fingerprint to which the first hash value belongs is determined as the fingerprint retrieval result, and the audio data corresponding to the fingerprint retrieval result is determined as the prototype audio corresponding to the recorded audio.

[0186] The specific functional implementations of the frequency domain transformation unit 121, the spectrum band division unit 122, the audio fingerprint combination unit 123, and the prototype audio matching unit 124 can be found above. Figure 5 Steps S202 and S205 in the corresponding embodiments will not be described again here.

[0187] In one or more embodiments, the audio filtering module 13 may include: a normalization processing unit 131, a first frequency point gain output unit 132, and a speech audio acquisition unit 133.

[0188] The normalization processing unit 131 is used to acquire the recording power spectrum data corresponding to the recorded audio, and to perform normalization processing on the recording power spectrum data to obtain the first spectral feature.

[0189] The normalization processing unit 131 described above is also used to obtain the prototype power spectrum data corresponding to the prototype audio, perform normalization processing on the prototype power spectrum data to obtain the second spectral feature, and combine the first spectral feature and the second spectral feature into the input feature.

[0190] The first frequency gain output unit 132 is used to input the input features into the first deep network model and output the first frequency gain for the recorded audio through the first deep network model.

[0191] The speech audio acquisition unit 133 is used to acquire candidate speech audio contained in the recorded audio based on the first frequency point gain and recording power spectrum data.

[0192] Optionally, the first frequency gain output unit 132 may include: a feature extraction subunit 1321 and an activation subunit 1322;

[0193] The feature extraction subunit 1321 is used to input the input features into the first deep network model and obtain the temporal distribution features corresponding to the input features based on the feature extraction network layer in the first deep network model.

[0194] The activation subunit 1322 is used to obtain the target feature vector corresponding to the temporal distribution features based on the fully connected network layer in the first deep network model, and output the first frequency point gain through the activation layer in the first deep network model.

[0195] Optionally, the first frequency gain includes the speech gains corresponding to T frequency points respectively, and the recording power spectrum data includes the energy values ​​corresponding to T frequency points respectively. The T speech gains correspond one-to-one with the T energy values; T is a positive integer greater than 1.

[0196] The speech audio acquisition unit 133 may include: a frequency point weighting subunit 1331, a weighted energy value combination subunit 1332, and a time domain transformation subunit 1333;

[0197] The frequency weighting subunit 1331 is used to weight the energy values ​​belonging to the same frequency point in the recording power spectrum data according to the speech gain corresponding to the T frequency points in the first frequency point gain, so as to obtain the weighted energy values ​​corresponding to the T frequency points respectively.

[0198] The weighted energy value combination subunit 1332 is used to determine the weighted audio domain signal corresponding to the recorded audio based on the weighted energy values ​​corresponding to T frequency points.

[0199] The time-domain transformation subunit 1333 is used to perform time-domain transformation on the weighted audio domain signal to obtain the candidate speech audio contained in the recorded audio.

[0200] The specific functional implementation methods of the normalization processing unit 131, the first frequency point gain output unit 132, the speech audio acquisition unit 133, the feature extraction subunit 1321, the activation subunit 1322, the frequency point weighting subunit 1331, the weighted energy value combination subunit 1332, and the time domain transformation subunit 1333 can be found above. Figure 5 Steps S206 and S208 in the corresponding embodiments will not be described again here.

[0201] In one or more embodiments, the noise reduction processing module 15 may include: a second frequency point gain output unit 151, a signal weighting unit 152, and a time domain transformation unit 153.

[0202] The second frequency gain output unit 151 is used to acquire the speech power spectrum data corresponding to the candidate speech audio, input the speech power spectrum data into the second deep network model, and output the second frequency gain for the candidate speech audio through the second deep network model.

[0203] The signal weighting unit 152 is used to obtain the weighted audio domain signal corresponding to the candidate audio based on the second frequency point gain and the speech power spectrum data.

[0204] The time-domain transformation unit 153 is used to perform time-domain transformation on the weighted speech audio domain signal to obtain the denoised speech audio corresponding to the candidate speech audio.

[0205] The specific functional implementation of the second frequency gain output unit 151, the signal weighting unit 152, and the time-domain transformation unit 153 can be found above. Figure 5 Steps S209 and S210 in the corresponding embodiments will not be described again here.

[0206] In one or more embodiments, the audio data processing device 1 may further include: an audio sharing module 16;

[0207] The audio sharing module 16 is used to share the noise-reduced audio recording to a social platform, so that terminal devices on the social platform can play the noise-reduced audio recording when accessing the social platform.

[0208] The specific implementation of the audio sharing module 16 can be found in the above description. Figure 3 Step S105 in the corresponding embodiment will not be described again here.

[0209] In this application, the aforementioned modules, units, and sub-units can implement the aforementioned... Figure 3 , Figure 5 The descriptions in any method embodiment, including the beneficial effects of using the same method, will not be repeated here.

[0210] Please see Figure 12 , Figure 12 This is a schematic diagram of the structure of an audio data processing device provided in an embodiment of this application. Figure 12 As shown, the audio data processing device 2 may include: a sample acquisition module 21, a first prediction module 22, a second prediction module 23, a first adjustment module 24, and a second adjustment module 25;

[0211] The sample acquisition module 21 is used to acquire speech sample audio, noise sample audio, and standard sample audio, and generate sample recording audio based on the speech sample audio, noise sample audio, and standard sample audio; the speech sample audio and noise sample audio are obtained through recording, and the standard sample audio is clean audio stored in the audio database.

[0212] The first prediction module 22 is used to obtain the sample predicted speech audio in the sample recorded audio according to the first initial network model; the first initial network model is used to filter the standard sample audio contained in the sample recorded audio, and the expected predicted speech audio of the first initial network model is determined by the speech sample audio and the noise sample audio.

[0213] The second prediction module 23 is used to obtain the sample prediction noise reduction frequency corresponding to the sample prediction speech audio according to the second initial network model; the second initial network model is used to suppress the noise sample audio contained in the sample prediction speech audio, and the expected prediction noise reduction frequency of the second initial network model is determined by the speech sample audio.

[0214] The first adjustment module 24 is used to adjust the network parameters of the first initial network model based on the sample predicted speech audio and the expected predicted speech audio to obtain the first deep network model; the first deep network model is used to filter the recorded audio to obtain candidate speech audio, the recorded audio includes background reference audio components, speech audio components and environmental noise components, and the candidate speech audio includes speech audio components and environmental noise components.

[0215] The second adjustment module 25 is used to adjust the network parameters of the second initial network model based on the sample predicted noise reduction frequency and the expected predicted noise reduction frequency to obtain the second deep network model; the second deep network model is used to perform noise reduction processing on the candidate speech audio to obtain the noise-reduced speech audio.

[0216] The specific functional implementation methods of the sample acquisition module 21, the first prediction module 22, the second prediction module 23, the first adjustment module 24, and the second adjustment module 25 can be found above. Figure 9 Steps S301-S305 in the corresponding embodiments will not be described again here.

[0217] In one or more embodiments, the number of sample audio recordings is K, where K is a positive integer;

[0218] The sample acquisition module 21 may include: an array construction unit 211 and a sample recording construction unit 212;

[0219] The array construction unit 211 is used to obtain the set of weighted coefficients for the first initial network model and construct K arrays based on the set of weighted coefficients; each array includes coefficients corresponding to the speech sample audio, the noise sample audio, and the standard sample audio respectively;

[0220] The sample recording construction unit 212 is used to weight the speech sample audio, noise sample audio and standard sample audio respectively according to the coefficients contained in the j-th array of K arrays to obtain the sample recording audio corresponding to the j-th array; j is a positive integer less than or equal to K.

[0221] The specific functional implementation methods of the array construction unit 211 and the sample recording construction unit 212 can be found in the above description. Figure 9 Step S301 in the corresponding embodiment will not be described again here.

[0222] In this application, the aforementioned modules, units, and sub-units can implement the aforementioned... Figure 9 The descriptions in the corresponding method embodiments, as well as the beneficial effects of using the same method, will not be repeated here.

[0223] Further, please see Figure 13 , Figure 13 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 13 As shown, the computer device 1000 can be a user terminal, for example, the one described above. Figure 1 The user terminal 10a in the corresponding embodiment can also be a server, for example, as described above. Figure 1 The server 10d in the corresponding embodiment will not be limited here. For ease of understanding, this application takes a computer device as a user terminal as an example. The computer device 1000 may include: a processor 1001, a network interface 1004, and a memory 1005. In addition, the computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a Wi-Fi interface). The memory 1004 may be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage device. The memory 1005 may optionally be at least one storage device located remotely from the aforementioned processor 1001. Figure 13 As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

[0224] The network interface 1004 in the computer device 1000 can also provide network communication functions, and the optional user interface 1003 can also include a display screen and a keyboard. Figure 13 In the computer device 1000 shown, the network interface 1004 provides network communication functionality; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0225] Acquire the recorded audio; the recorded audio includes background reference audio components, speech audio components, and environmental noise components;

[0226] Obtain the audio fingerprint corresponding to the recorded audio, and retrieve the prototype audio that matches the recorded audio from the audio database based on the audio fingerprint.

[0227] Candidate speech audio is obtained from the recorded audio based on the prototype audio; the candidate speech audio includes speech audio components and environmental noise components;

[0228] The difference between the recorded audio and the candidate speech audio is determined as the background reference audio component contained in the recorded audio.

[0229] The candidate speech audio is subjected to environmental noise reduction processing to obtain the denoised speech audio corresponding to the candidate speech audio. The denoised speech audio is then merged with the background reference audio component to obtain the denoised recording audio.

[0230] Alternatively, processor 1001 can also achieve:

[0231] Acquire speech sample audio, noise sample audio, and standard sample audio; generate sample recording audio based on the speech sample audio, noise sample audio, and standard sample audio; the speech sample audio and noise sample audio are obtained through recording, and the standard sample audio is clean audio stored in the audio database.

[0232] The first initial network model is used to obtain the sample predicted speech audio from the sample recorded audio. The first initial network model is used to filter the standard sample audio contained in the sample recorded audio. The expected predicted speech audio of the first initial network model is determined by the speech sample audio and the noise sample audio.

[0233] The sample predicted noise reduction frequency corresponding to the sample predicted speech audio is obtained according to the second initial network model; the second initial network model is used to suppress the noise sample audio contained in the sample predicted speech audio, and the expected predicted noise reduction frequency of the second initial network model is determined by the speech sample audio.

[0234] Based on the sample predicted speech audio and the expected predicted speech audio, the network parameters of the first initial network model are adjusted to obtain the first deep network model; the first deep network model is used to filter the recorded audio to obtain candidate speech audio, the recorded audio includes background reference audio components, speech audio components and environmental noise components, and the candidate speech audio includes speech audio components and environmental noise components.

[0235] Based on the sample predicted noise reduction frequency and the expected predicted noise reduction frequency, the network parameters of the second initial network model are adjusted to obtain the second deep network model; the second deep network model is used to perform noise reduction processing on the candidate speech audio to obtain the noise-reduced speech audio.

[0236] It should be understood that the computer device 1000 described in the embodiments of this application can execute the foregoing text. Figure 3 , Figure 5 as well as Figure 9 The description of the audio data processing method in any of the corresponding embodiments can also be performed as described above. Figure 11 The description of the audio data processing device 1 in the corresponding embodiment, or Figure 12 The description of the audio data processing device 2 in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0237] Furthermore, it should be noted that this application embodiment also provides a computer-readable storage medium, which stores a computer program executed by the aforementioned audio data processing device 1 and audio data processing device 2. The computer program includes program instructions, and when the processor executes the program instructions, it can execute the aforementioned... Figure 3 , Figure 5 as well as Figure 9 The description of the audio data processing method in any corresponding embodiment is already provided and will not be repeated here. Similarly, the beneficial effects of using the same method will not be repeated here. For technical details not disclosed in the computer-readable storage medium embodiments related to this application, please refer to the description of the method embodiments of this application. As an example, program instructions may be deployed and executed on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network. These multiple computing devices distributed across multiple locations and interconnected via a communication network can constitute a blockchain system.

[0238] Furthermore, it should be noted that this application also provides a computer program product or computer program, which may include computer instructions, which may be stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor may execute the computer instructions, causing the computer device to perform the aforementioned actions. Figure 3 , Figure 5 as well as Figure 9 The description of the audio data processing method in any corresponding embodiment is already provided, and therefore will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer program products or computer program embodiments related to this application, please refer to the description of the method embodiments of this application.

[0239] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0240] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0241] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0242] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0243] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. An audio data processing method, characterized in that, include: Get the recorded audio; The recorded audio includes a background reference audio component, a speech audio component, and an environmental noise component. Obtain the audio fingerprint to be matched corresponding to the recorded audio, and obtain the prototype audio that matches the recorded audio from the audio database based on the audio fingerprint to be matched; Candidate speech audio is obtained from the recorded audio based on the prototype audio; the candidate speech audio includes the speech audio component and the environmental noise component; The difference between the recorded audio and the candidate speech audio is determined as the background reference audio component contained in the recorded audio. Obtain the speech power spectrum data corresponding to the candidate speech audio, input the speech power spectrum data into the second deep network model, and output the second frequency point gain for the candidate speech audio through the second deep network model; Based on the second frequency gain and the speech power spectrum data, the weighted speech domain signal corresponding to the candidate speech audio is obtained; The weighted audio domain signal is transformed in the time domain to obtain the denoised audio corresponding to the candidate audio. The denoised audio is then merged with the background reference audio component to obtain the denoised recording audio.

2. The method according to claim 1, characterized in that, The step of obtaining the audio fingerprint to be matched corresponding to the recorded audio, and obtaining the prototype audio that matches the recorded audio from the audio database based on the audio fingerprint to be matched, includes: The recorded audio is divided into M recording data frames. The i-th recording data frame is subjected to frequency domain transformation to obtain the power spectrum data corresponding to the i-th recording data frame. i and M are both positive integers, and i is less than or equal to M. The power spectrum data corresponding to the i-th recording data frame is divided into N spectrum bands, and the sub-fingerprint information corresponding to the i-th recording data frame is constructed based on the peak signals in the N spectrum bands; N is a positive integer. According to the time order of the M recording data frames in the recorded audio, the sub-fingerprint information corresponding to the M recording data frames is combined to obtain the audio fingerprint to be matched corresponding to the recorded audio. Obtain the audio fingerprint database corresponding to the audio database, perform fingerprint retrieval in the audio fingerprint database according to the audio fingerprint to be matched, and determine the prototype audio that matches the recorded audio in the audio database according to the fingerprint retrieval result.

3. The method according to claim 2, characterized in that, The step of performing a fingerprint search in the audio fingerprint database based on the audio fingerprint to be matched, and determining the prototype audio that matches the recorded audio in the audio database based on the fingerprint search results, includes: The M sub-fingerprint information contained in the audio fingerprint to be matched is mapped to M hash values ​​to be matched, and the recording time corresponding to the M hash values ​​to be matched is obtained respectively; the recording time corresponding to a hash value to be matched is used to characterize the time when the sub-fingerprint information corresponding to the hash value to be matched appears in the audio recording. If the p-th hash value among the M hash values ​​to be matched matches the first hash value contained in the audio fingerprint database, then the first time difference between the recording time corresponding to the p-th hash value and the time information corresponding to the first hash value is obtained; p is a positive integer less than or equal to M; If the q-th hash value among the M hash values ​​to be matched matches the second hash value contained in the audio fingerprint database, then the second time difference between the recording time corresponding to the q-th hash value and the time information corresponding to the second hash value is obtained; q is a positive integer less than or equal to M; When the first time difference and the second time difference meet the numerical threshold, and the first hash value and the second hash value belong to the same audio fingerprint, the audio fingerprint to which the first hash value belongs is determined as the fingerprint retrieval result, and the audio data corresponding to the fingerprint retrieval result is determined as the prototype audio corresponding to the recorded audio.

4. The method according to claim 1, characterized in that, The step of obtaining candidate speech audio from the recorded audio based on the prototype audio includes: Obtain the recording power spectrum data corresponding to the recorded audio, and normalize the recording power spectrum data to obtain the first spectral feature; Obtain the prototype power spectrum data corresponding to the prototype audio, normalize the prototype power spectrum data to obtain the second spectral feature, and combine the first spectral feature and the second spectral feature into the input feature; The input features are input into a first deep network model, and the first deep network model outputs a first frequency gain for the recorded audio. Based on the first frequency gain and the recording power spectrum data, candidate speech audio contained in the recorded audio is obtained.

5. The method according to claim 4, characterized in that, The step of inputting the input features into a first deep network model and outputting a first frequency gain through the first deep network model includes: The input features are input into a first deep network model, and the temporal distribution features corresponding to the input features are obtained based on the feature extraction network layer in the first deep network model. Based on the fully connected network layer in the first deep network model, the target feature vector corresponding to the temporal distribution features is obtained, and the first frequency point gain is output through the activation layer in the first deep network model.

6. The method according to claim 4, characterized in that, The first frequency gain includes the speech gains corresponding to T frequency points respectively, and the recording power spectrum data includes the energy values ​​corresponding to the T frequency points respectively. The T speech gains correspond one-to-one with the T energy values; T is a positive integer greater than 1. The step of obtaining candidate speech audio contained in the recorded audio based on the first frequency point gain and the recording power spectrum data includes: Based on the speech gains corresponding to the T frequency points in the first frequency point gain, the energy values ​​belonging to the same frequency point in the recording power spectrum data are weighted to obtain the weighted energy values ​​corresponding to the T frequency points respectively. Based on the weighted energy values ​​corresponding to the T frequency points, the weighted audio domain signal corresponding to the recorded audio is determined; The weighted audio domain signal is transformed in the time domain to obtain the candidate speech audio contained in the recorded audio.

7. The method according to claim 1, characterized in that, Also includes: The noise-reduced audio recording is shared to a social media platform so that terminal devices on the social media platform can play the noise-reduced audio recording when accessing the social media platform.

8. An audio data processing method, characterized in that, include: Acquire speech sample audio, noise sample audio, and standard sample audio; generate sample recording audio based on the speech sample audio, noise sample audio, and standard sample audio; the speech sample audio and the noise sample audio are obtained through recording, and the standard sample audio is clean audio stored in an audio database; Sample predicted speech audio is obtained from the sample recorded audio according to the first initial network model; the first initial network model is used to filter the standard sample audio contained in the sample recorded audio, and the expected predicted speech audio of the first initial network model is determined by the speech sample audio and the noise sample audio; The sample predicted noise reduction frequency corresponding to the sample predicted speech audio is obtained according to the second initial network model; the second initial network model is used to suppress the noise sample audio contained in the sample predicted speech audio, and the expected predicted noise reduction frequency of the second initial network model is determined by the speech sample audio; Based on the sample predicted speech audio and the expected predicted speech audio, the network parameters of the first initial network model are adjusted to obtain a first deep network model; the first deep network model is used to filter the recorded audio to obtain candidate speech audio, the recorded audio includes background reference audio components, speech audio components and environmental noise components, and the candidate speech audio includes the speech audio components and environmental noise components. Based on the sample predicted noise reduction frequency and the expected predicted noise reduction frequency, the network parameters of the second initial network model are adjusted to obtain a second deep network model; the second deep network model is used to perform noise reduction processing on the candidate speech audio to obtain noise-reduced speech audio.

9. The method according to claim 8, characterized in that, The number of sample audio recordings is K, where K is a positive integer; The step of generating sample recording audio based on the speech sample audio, the noise sample audio, and the standard sample audio includes: Obtain a set of weighted coefficients for the first initial network model, and construct K arrays based on the set of weighted coefficients; each array includes coefficients corresponding to the speech sample audio, the noise sample audio, and the standard sample audio, respectively; Based on the coefficients contained in the j-th array of the K arrays, the speech sample audio, the noise sample audio, and the standard sample audio are weighted respectively to obtain the sample recording audio corresponding to the j-th array; j is a positive integer less than or equal to K.

10. An audio data processing apparatus, characterized in that, include: The audio acquisition module is used to acquire recorded audio. The recorded audio includes a background reference audio component, a speech audio component, and an environmental noise component. The fingerprint retrieval module is used to obtain the audio fingerprint to be matched corresponding to the recorded audio, and to obtain the prototype audio that matches the recorded audio from the audio database based on the audio fingerprint to be matched. An audio filtering module is used to obtain candidate speech audio from the recorded audio based on the prototype audio; the candidate speech audio includes the speech audio component and the environmental noise component. An audio determination module is used to determine the difference between the recorded audio and the candidate speech audio as the background reference audio component contained in the recorded audio. The noise reduction processing module is used to acquire the speech power spectrum data corresponding to the candidate speech audio, input the speech power spectrum data into the second deep network model, and output the second frequency point gain for the candidate speech audio through the second deep network model. The noise reduction processing module is further configured to obtain the weighted audio domain signal corresponding to the candidate audio based on the second frequency point gain and the speech power spectrum data; The noise reduction processing module is further configured to perform time-domain transformation on the weighted audio domain signal to obtain the noise-reduced audio corresponding to the candidate audio, and merge the noise-reduced audio with the background reference audio component to obtain the noise-reduced recording audio.

11. An audio data processing apparatus, characterized in that, include: The sample acquisition module is used to acquire speech sample audio, noise sample audio, and standard sample audio, and generate sample recording audio based on the speech sample audio, the noise sample audio, and the standard sample audio; the speech sample audio and the noise sample audio are obtained through recording, and the standard sample audio is clean audio stored in an audio database; A first prediction module is used to obtain sample predicted speech audio from the sample recorded audio according to a first initial network model; the first initial network model is used to filter the standard sample audio contained in the sample recorded audio, and the expected predicted speech audio of the first initial network model is determined by the speech sample audio and the noise sample audio. The second prediction module is used to obtain the sample predicted noise reduction frequency corresponding to the sample predicted speech audio according to the second initial network model; the second initial network model is used to suppress the noise sample audio contained in the sample predicted speech audio, and the expected predicted noise reduction frequency of the second initial network model is determined by the speech sample audio. The first adjustment module is used to adjust the network parameters of the first initial network model based on the sample predicted speech audio and the expected predicted speech audio to obtain a first deep network model; the first deep network model is used to filter the recorded audio to obtain candidate speech audio, the recorded audio includes a background reference audio component, a speech audio component and an environmental noise component, and the candidate speech audio includes the speech audio component and the environmental noise component. The second adjustment module is used to adjust the network parameters of the second initial network model based on the sample predicted noise reduction frequency and the expected predicted noise reduction frequency to obtain a second deep network model; the second deep network model is used to perform noise reduction processing on the candidate speech audio to obtain noise-reduced speech audio.

12. A computer device, characterized in that, Including memory and processor; The memory is connected to the processor, the memory is used to store computer programs, and the processor is used to invoke the computer programs so that the computer device performs the method according to any one of claims 1-9.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded and executed by a processor to cause a computer device having the processor to perform the method of any one of claims 1-9.

14. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, implements the method described in any one of claims 1-9.

Images