Audio processing method, audio processing model training method, and device, storage medium and program product

Abstract

An audio processing method, an audio processing model training method, and a device, a storage medium and a program product. The audio processing method comprises: determining audio to be processed, and inputting said audio into an audio processing model (202); converting said audio into a speech spectrogram by means of a feature extraction layer in the audio processing model, and performing feature extraction on the speech spectrogram to obtain a plurality of spectral features, wherein there is an overlapping feature between two adjacent spectral features among the plurality of spectral features (204); performing information extraction on the plurality of spectral features by means of an information sampling layer in the audio processing model, so as to obtain hidden-layer features corresponding to the plurality of spectral features (206); and by means of an attention pooling layer in the audio processing model, and on the basis of the hidden-layer features corresponding to the plurality of spectral features, calculating noise information corresponding to said audio and outputting the noise information (208). By performing feature extraction and information sampling on a speech spectrogram, key features are learned and extracted from speech signals, and an attention mechanism is used to make a model focus on analyzing noise components, so as to improve the generalization capability of the model in terms of evaluating noise, thereby accurately calculating noise information of audio.

Description

Audio processing and model training method, device, storage medium and program product

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on April 3, 2024, with application number 202410405445.1 and application name “Audio Processing Method and Device”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments of this specification relate to the field of artificial intelligence technology, and in particular to an audio processing and model training method, a computing device, a computer storage medium, and a computer program product. Background Art

[0003] With the widespread use of mobile communications and smart devices, clear, high-quality voice communication is becoming an increasingly basic user requirement. However, in everyday life, voice signals contain various noise interferences, which not only affect the user's communication experience but also hinder the accuracy of voice recognition systems. Currently, various methods exist to assess and eliminate noise, such as training noise recognition models through manual annotation and comparing original sounds with reference sounds. However, these methods require significant human and material resources to implement and, in practice, struggle to ensure real-time performance and accuracy. Therefore, how to quickly and accurately analyze noise signals in audio signals is an urgent problem that needs to be solved. Summary of the Invention

[0004] In view of this, embodiments of this specification provide an audio processing method. One or more embodiments of this specification also relate to an audio processing apparatus, a computing device, a computer-readable storage medium, and a computer program product to address technical deficiencies in the prior art.

[0005] According to a first aspect of the embodiments of this specification, there is provided an audio processing method, including:

[0006] Determining audio to be processed, and inputting the audio to be processed into an audio processing model;

[0007] The audio to be processed is converted into a speech spectrogram by a feature extraction layer in the audio processing model, and feature extraction is performed on the speech spectrogram to obtain a plurality of spectral features, wherein two adjacent spectral features in the plurality of spectral features have overlapping features;

[0008] Extracting information from the multiple spectral features through the information sampling layer in the audio processing model to obtain hidden layer features corresponding to the multiple spectral features;

[0009] Through the attention pooling layer in the audio processing model, the noise information corresponding to the audio to be processed is calculated and output according to the hidden layer features corresponding to the multiple spectral features.

[0010] According to a second aspect of the embodiments of this specification, a method for training an audio processing model is provided, comprising:

[0011] Determining sample audio and sample noise information corresponding to the sample audio, and inputting the sample audio into an audio initial processing model;

[0012] The sample audio is converted into a speech spectrogram by a feature extraction layer in the audio initial processing model, and feature extraction is performed on the speech spectrogram to obtain a plurality of spectral features, wherein two adjacent spectral features in the plurality of spectral features have overlapping features;

[0013] Extracting information from the multiple spectral features through the information sampling layer in the initial audio processing model to obtain hidden layer features corresponding to the multiple spectral features;

[0014] Calculating predicted noise information corresponding to the sample audio according to hidden layer features corresponding to the multiple spectral features through the attention pooling layer in the initial audio processing model;

[0015] A model loss value is calculated according to the sample noise information and the predicted noise information, and model parameters of the audio initial processing model are adjusted based on the model loss value, and the audio initial processing model is continuously trained until an audio processing model that meets the training conditions is obtained.

[0016] According to a third aspect of the embodiments of this specification, there is provided an audio processing method, including:

[0017] Receive and parse the audio file input by the user, determine the audio to be processed, and input the audio to be processed into the audio processing model;

[0018] The audio to be processed is converted into a speech spectrogram by a feature extraction layer in the audio processing model, and feature extraction is performed on the speech spectrogram to obtain a plurality of spectral features, wherein two adjacent spectral features in the plurality of spectral features have overlapping features;

[0019] Extracting information from the multiple spectral features through the information sampling layer in the audio processing model to obtain hidden layer features corresponding to the multiple spectral features;

[0020] Through the attention pooling layer in the audio processing model, noise information corresponding to the audio to be processed is calculated according to the hidden layer features corresponding to the multiple spectral features and fed back to the user.

[0021] According to a fourth aspect of the embodiments of this specification, there is provided an audio processing device, including:

[0022] An input module, configured to determine audio to be processed and input the audio to be processed into an audio processing model;

[0023] a conversion module configured to convert the to-be-processed audio into a speech spectrogram through a feature extraction layer in the audio processing model, perform feature extraction on the speech spectrogram, and obtain a plurality of spectral features, wherein two adjacent spectral features in the plurality of spectral features have overlapping features;

[0024] an information extraction module configured to extract information from the plurality of spectral features through an information sampling layer in the audio processing model to obtain hidden layer features corresponding to the plurality of spectral features;

[0025] The calculation and output module is configured to calculate and output the noise information corresponding to the audio to be processed based on the hidden layer features corresponding to the multiple spectral features through the attention pooling layer in the audio processing model.

[0026] According to a fifth aspect of the embodiments of this specification, there is provided a training device for an audio processing model, comprising:

[0027] An input module, configured to determine a sample audio and sample noise information corresponding to the sample audio, and input the sample audio into an audio initial processing model;

[0028] a conversion module configured to convert the sample audio into a speech spectrogram through a feature extraction layer in the audio initial processing model, perform feature extraction on the speech spectrogram, and obtain a plurality of spectral features, wherein two adjacent spectral features in the plurality of spectral features have overlapping features;

[0029] An information extraction module is configured to extract information from the multiple spectral features through the information sampling layer in the audio initial processing model to obtain hidden layer features corresponding to the multiple spectral features;

[0030] a calculation module configured to calculate predicted noise information corresponding to the sample audio according to hidden layer features corresponding to the multiple spectral features through an attention pooling layer in the initial audio processing model;

[0031] The training module is configured to calculate a model loss value based on the sample noise information and the predicted noise information, adjust the model parameters of the audio initial processing model based on the model loss value, and continue to train the audio initial processing model until an audio processing model that meets the training conditions is obtained.

[0032] According to a sixth aspect of the embodiments of this specification, there is provided an audio processing device, including:

[0033] An input module is configured to receive and parse an audio file input by a user, determine audio to be processed, and input the audio to be processed into an audio processing model;

[0034] a conversion module configured to convert the to-be-processed audio into a speech spectrogram through a feature extraction layer in the audio processing model, perform feature extraction on the speech spectrogram, and obtain a plurality of spectral features, wherein two adjacent spectral features in the plurality of spectral features have overlapping features;

[0035] an information extraction module configured to extract information from the plurality of spectral features through an information sampling layer in the audio processing model to obtain hidden layer features corresponding to the plurality of spectral features;

[0036] The calculation and feedback module is configured to calculate the noise information corresponding to the audio to be processed and feed it back to the user based on the hidden layer features corresponding to the multiple spectral features through the attention pooling layer in the audio processing model.

[0037] According to a seventh aspect of the embodiments of this specification, a computing device is provided, including:

[0038] memory and processor;

[0039] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the above method are implemented.

[0040] According to an eighth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, and the computer program / instruction implements the steps of the above method when executed by a processor.

[0041] According to a ninth aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0042] One embodiment of this specification implements a feature extraction layer within an audio processing model that converts the processed audio into a speech spectrogram and performs feature extraction on the speech spectrogram to obtain multiple spectral features. This allows for better learning and extraction of key features from the speech signal, with overlapping features between adjacent spectral features, further improving feature extraction accuracy. Subsequently, an information sampling layer extracts information from the multiple spectral features to obtain the corresponding hidden features. Then, an attention pooling layer uses an attention mechanism to enable the model to focus on analyzing the noise components in the speech, improving the model's generalization ability in noise assessment and thereby more accurately calculating the noise information corresponding to the processed audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] FIG1 is a schematic diagram of a scenario of an audio processing method provided by an embodiment of this specification;

[0044] FIG2 is a flow chart of an audio processing method provided by one embodiment of this specification;

[0045] FIG3 is a schematic diagram of a speech spectrum diagram in an audio processing method provided by one embodiment of this specification;

[0046] FIG4 is a flowchart of a processing process of an audio processing method provided by one embodiment of this specification;

[0047] FIG5 is a flowchart of a method for training an audio processing model provided by one embodiment of this specification;

[0048] FIG6 is a flowchart of another audio processing method provided by an embodiment of this specification;

[0049] FIG7 is a schematic structural diagram of an audio processing device provided by one embodiment of this specification;

[0050] FIG8 is a schematic diagram of the structure of a training device for an audio processing model provided by one embodiment of this specification;

[0051] FIG9 is a schematic structural diagram of another audio processing device provided by an embodiment of this specification;

[0052] FIG10 is a structural block diagram of a computing device provided by one embodiment of this specification;

[0053] FIG. 11 is a schematic diagram of an exemplary system to which the embodiments of the present disclosure are applicable. DETAILED DESCRIPTION

[0054] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0055] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0056] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0057] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0058] First, the terms involved in one or more embodiments of this specification are explained.

[0059] Speech noisiness assessment: Speech noisiness assessment (SQA) is to evaluate the impact of noise in speech on the subjective hearing perception of the human ear.

[0060] Noisiness score: Noisiness score represents a person's subjective perception of noise in speech.

[0061] With the widespread adoption of mobile communications and smart devices, clear, high-quality voice communication is becoming an increasingly basic user requirement. However, in real-world environments, voice signals are often interfered with by various noises, which not only affects the user's communication experience but also poses a challenge to the accuracy of voice recognition systems. Therefore, accurately assessing the level of noise in voice signals is crucial for improving voice processing algorithms, enhancing communication quality, and optimizing the user experience. In the current communications field, despite various approaches to assessing and eliminating noise, many challenges remain. Existing noise assessment methods typically rely on active signals and require a clean reference signal for comparison, which is often impractical in practical applications. Furthermore, these methods are based on traditional signal processing and have high latency when processing real-time signals, making them difficult to meet the needs of real-time communication.

[0062] Based on this, this specification provides an audio processing method, which also relates to an audio processing device, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0063] Refer to Figure 1, which is a scene diagram of an audio processing method provided according to an embodiment of this specification, wherein the audio to be processed can be understood as the voice audio input by the user. For example, when the user inputs the user's speech through the voice software, the voice software will receive the user's voice and convert it into the audio to be processed. When the audio to be processed is subjected to noise evaluation, the audio to be processed will be input into the audio processing model. The audio processing model is used to evaluate the noise level of the audio. After receiving the audio to be processed, the feature extraction layer in the audio processing model will convert it into a speech spectrogram, and then extract multiple spectral features from the speech spectrogram, and input the multiple spectral features into the information extraction layer. The information extraction layer extracts multiple hidden features from the multiple spectral features, and inputs the multiple hidden features into the attention pooling layer. The attention pooling layer identifies the noise components in the hidden features through the attention mechanism, thereby calculating the noise information. The audio processing model can accurately identify the noise level in the audio, which is beneficial for subsequent users to judge whether to re-input the audio or use the audio directly based on the noise information, thereby providing users with better voice services.

[0064] Referring to FIG. 2 , FIG. 2 shows a flow chart of an audio processing method provided according to an embodiment of the present specification, which specifically includes the following steps.

[0065] Step 202: Determine the audio to be processed, and input the audio to be processed into the audio processing model.

[0066] In practical applications, the audio processing method provided in this specification can be applied to scenarios of voice recognition, such as intelligent voice assistants, which provide users with intelligent command services by recognizing the user's language. In the application of in-vehicle voice recognition, intelligent in-vehicle services can be provided to users. For example, if the user inputs the voice "open the sun visor for me", the intelligent command to open the sun visor can be activated after recognizing the user's voice, providing users with more convenient and intelligent voice services. However, during the recognition process, the audio to be processed may contain noise due to the environment or other human voices, making the final recognition result inaccurate and affecting the user experience. Therefore, the noise level of the audio to be processed can be evaluated. If the noise level is greater than a certain threshold, inaccurate recognition may occur. At this time, the user can be asked to re-enter the input, thereby providing users with accurate voice recognition capabilities.

[0067] The audio to be processed can be understood as audio input by the user, such as a piece of audio recorded by the user for speech recognition; it can also be a piece of audio downloaded by the user for specific noise level assessment. After receiving the audio to be processed, it can be input into the audio processing model, which is used to assess the noise level of the audio. After the audio processing model outputs noise information, the next step of the processing strategy for the audio to be processed can be carried out based on the noise information, such as guiding the optimization of noise suppression and speech enhancement algorithms, performing audio analysis, or asking the user to re-enter.

[0068] Furthermore, in order to better enable the audio processing model to identify the audio to be processed, the audio input by the user can be preprocessed so that the subsequent audio processing model can process the audio according to preset conditions. The specific determination of the audio to be processed includes: determining the original audio and the audio adjustment strategy corresponding to the original audio; intercepting and processing the original audio according to the audio adjustment strategy to obtain the audio to be processed.

[0069] In actual applications, because users may input audio that is too long to be properly input into the audio processing model, the original audio input by the user can be intercepted and processed. The original audio is the audio input received by the user, and the audio adjustment policy corresponding to the original audio can be understood as the adjustment policy pre-set by the audio processing model for the audio, such as limiting the audio length to 20 seconds. The original audio is intercepted according to the audio adjustment policy, so that the subsequent audio processing model can properly analyze the processed audio.

[0070] In a specific embodiment of this specification, user audio input is received. The original audio is a 60-second segment of user speech. The original audio needs to be trimmed and processed. A trimming strategy is determined to truncate the audio to 20 seconds. The original audio is trimmed according to the trimming strategy to obtain a 20-second segment of audio as the audio to be processed, which is then output to the audio processing model for processing and analysis.

[0071] Based on this, by adjusting the audio according to the audio processing strategy, the processed audio can be processed and analyzed by the audio processing model.

[0072] Step 204: The audio to be processed is converted into a speech spectrogram through a feature extraction layer in the audio processing model, and feature extraction is performed on the speech spectrogram to obtain a plurality of spectral features, wherein two adjacent spectral features in the plurality of spectral features have overlapping features.

[0073] In actual applications, after the audio to be processed is input into the audio processing model, the feature extraction layer in the audio processing model will perform feature extraction on the audio to be processed. Since the audio to be processed is a time domain speech signal, in order to be able to analyze and process the audio, it needs to be converted into frequency domain data first. The feature extraction layer will first convert the audio to be processed into a speech spectrogram. The speech spectrogram is a way of representing audio features. In the embodiment of this specification, the speech spectrogram is a Mel-Spectrogram. The speech spectrogram converts the original audio signal into a two-dimensional image form through a series of preprocessing steps (such as framing, windowing, Fourier transform, Mel filter bank application, logarithmic compression, etc.). The two dimensions of the Mel-Spectrogram usually correspond to time and frequency: the horizontal axis represents time (for example, in frames), and the vertical axis represents frequency (in Mel scales). Each pixel value represents the acoustic energy intensity at the corresponding time and frequency point. After the audio to be processed is converted into a speech spectrogram, feature extraction can be performed on it to obtain multiple spectral features.

[0074] Among them, the spectral features can be understood as audio features obtained through feature extraction from the audio to be processed. See Figure 3, which is a schematic diagram of a speech spectrogram in an audio processing method provided in this specification, wherein (A) in Figure 3 is a speech spectrogram, and (B) in Figure 3 is a plurality of spectral features extracted from the speech spectrogram. It can be found that the plurality of spectral features are generated by segmenting the speech spectrogram. In order to avoid the loss of relevant information in the audio due to segmentation or the occurrence of recognition errors due to segmentation, it is necessary to generate two adjacent spectral features by overlapping so that there are overlapping features between the two adjacent spectral features.

[0075] Furthermore, in order to convert the audio to be processed into a speech spectrogram, it is necessary to perform spectrogram conversion on the audio to be processed. Specifically, converting the audio to be processed into a speech spectrogram includes: performing short-time Fourier transform on the audio to be processed to obtain the initial speech spectrogram corresponding to the audio to be processed; filtering the initial speech spectrogram through a preset filter to obtain a speech spectrogram.

[0076] The Short-Time Fourier Transform (STFT) is a transformation method that decomposes a time-domain signal into a frequency-domain representation. By performing an STFT on the audio being processed, we obtain a STFT spectrum, or the initial speech spectrogram. This initial speech spectrogram is then filtered through a preset filter to obtain a speech spectrogram. In practical applications, the STFT spectrum is converted to a logarithmic scale, such as the Mel scale, to obtain the speech spectrogram corresponding to the audio being processed.

[0077] In a specific embodiment of the present specification, a short-time Fourier transform spectrum is obtained by performing a short-time Fourier transform on the audio to be processed, and the short-time Fourier transform spectrum is subjected to Mel-scale conversion and amplitude squared through a preset filter to obtain a speech spectrum corresponding to the audio to be processed.

[0078] Based on this, by converting the audio to be processed into a speech spectrogram, it is convenient to perform subsequent audio analysis and processing based on the speech spectrogram.

[0079] Furthermore, after obtaining the speech spectrogram, in order to better analyze and process the speech spectrogram, the speech spectrum can be segmented, and specifically, feature extraction is performed on the speech spectrogram to obtain multiple spectral features, including: determining the audio extraction rules corresponding to the audio processing model, and determining the extraction parameters corresponding to the speech spectrogram according to the audio extraction rules; performing feature extraction on the speech spectrogram according to the extraction parameters to obtain multiple spectral features.

[0080] In practical applications, after obtaining the speech spectrogram, feature extraction can be performed on the speech spectrogram, so the extraction parameters need to be determined. The audio processing model presets extraction rules for audio. After determining the audio extraction rules, feature extraction can be performed on the speech spectrogram according to the extraction parameters in the audio extraction rules, thereby obtaining multiple spectral features.

[0081] The audio extraction rule can be a pre-defined extraction rule in the audio processing model. The extraction rule can include corresponding extraction parameters, such as extraction width and extraction quantity. The extraction parameter can be understood as the extraction size parameter of the spectral feature. Feature extraction is performed on the speech spectrogram based on the extraction parameter to obtain multiple spectrograms that meet the extraction size parameters. In specific implementation, after determining the extraction parameters, the speech spectrogram can be intercepted according to the extraction parameters. The extraction parameters include feature width T, feature height F, feature quantity L, and feature total length B.

[0082] In a specific embodiment of the present specification, the audio extraction rules of the audio processing model are determined, and the extraction parameters are determined according to the audio extraction rules. The extraction parameters include the width (time), height (Mel scale), number and total length of each spectral feature. According to the extraction parameters, multiple spectral features of size [B, L, T, F] can be obtained.

[0083] Furthermore, the actual processing process of feature extraction of the speech spectrogram is to segment the speech spectrogram, specifically to extract features of the speech spectrogram according to the extraction parameters to obtain multiple spectral features, including: segmenting the speech spectrogram according to the extraction parameters to obtain multiple initial spectral features; determining the overlapping spectral features corresponding to the multiple initial spectral features, and generating multiple spectral features based on the overlapping spectral features and the multiple initial spectral features.

[0084] Among them, after the speech spectrum graph is segmented, multiple initial spectrum features after segmentation can be obtained. After combining the multiple initial spectrum features, the entire speech spectrum graph is obtained. In order to avoid information loss or subsequent analysis errors caused by segmentation, it is necessary to overlap the initial spectrum features, that is, overlap adjacent spectrum features. Therefore, after obtaining multiple initial spectrum features, overlapping spectrum features can be determined based on the adjacent spectrum features of each initial spectrum feature. The overlapping spectrum feature is the spectrum feature that needs to be superimposed on each initial spectrum feature. After determining each initial spectrum feature and its corresponding overlapping spectrum feature that needs to be overlapped, multiple spectrum features can be generated. In actual applications, the overlapping spectrum feature corresponding to each initial spectrum feature includes two overlapping spectrum features, namely, a preceding overlapping spectrum feature and a following overlapping spectrum feature. The preceding overlapping spectrum feature is determined based on the previous initial spectrum feature of the initial spectrum feature, and the following overlapping spectrum feature is determined based on the next initial spectrum feature of the initial spectrum feature.

[0085] In a specific embodiment of the present specification, after the speech spectrogram is segmented according to the extraction parameters, multiple initial spectral features after segmentation can be obtained, and the overlapping spectral features corresponding to each initial spectral feature are determined. The initial spectral features are superimposed according to the overlapping spectral features corresponding to each initial spectral feature, thereby generating multiple spectral features.

[0086] Based on this, by overlapping the spectral features, it is ensured that there is no information loss between each spectral feature, and the subsequent audio processing and analysis can be accurately performed.

[0087] Step 206: extract information from the multiple spectral features through the information sampling layer in the audio processing model to obtain hidden layer features corresponding to the multiple spectral features.

[0088] In practical applications, after extracting multiple spectral features from the audio through the feature extraction layer in the audio processing model, the spectral features can be input into the information sampling layer, and then the hidden features can be extracted through the information sampling layer. The hidden features are high-dimensional hidden layer feature values. The information sampling layer is the downsampling layer. The information sampling layer can adopt a CNN convolutional neural network and use 2D convolution to extract information from the spectral features, thereby obtaining high-dimensional hidden layer feature values. In specific implementation, the information sampling layer can include three sampling modules, each of which includes a convolutional layer, a pooling layer, and a normalization layer. The three sampling modules form a multi-level downsampling structure. Each level of downsampling will further reduce the spatial resolution of the feature while capturing higher-level abstract features.

[0089] Based on this, the information sampling layer reduces the spatial dimensionality of the input features while retaining key feature information. This helps reduce the number of model parameters and computational complexity, improves model efficiency, and helps extract higher-level, more abstract features, which is more beneficial for subsequent analysis. Downsampling also helps increase the model's robustness to small translations, rotations, or scaling of the input data, thereby improving the model's translation invariance.

[0090] Furthermore, the information extraction process of any one of the multiple spectral features includes: performing a convolution operation on a target spectral feature among the multiple spectral features through the convolution layer in the information sampling layer to obtain a convolution feature map corresponding to the target spectral feature; performing a pooling operation on the convolution feature map through the pooling layer in the information sampling layer to obtain a pooling feature map corresponding to the convolution feature map; and performing a normalization operation on the pooling feature map through the normalization layer in the information sampling layer to obtain a target hidden layer feature corresponding to the target spectral feature.

[0091] In practical applications, since the information sampling layer includes multiple sampling modules, each sampling module includes a convolution layer, a pooling layer and a normalization layer. Therefore, after the sampling module receives input data such as spectral features or the output data of the previous level sampling module, it will first perform a convolution operation on the input data through the convolution layer to obtain a convolution feature map, and then use the pooling layer to perform a pooling operation on the convolution feature map to obtain a pooled feature map, and then perform a normalization operation through the normalization layer to obtain the hidden layer features.

[0092] In specific implementations, multiple spectral features can be sampled separately, so that the target spectral feature that needs to be sampled can be determined from the multiple spectral features. This is then input into the sampling module. Through the convolutional layer, pooling layer, and normalization layer in the sampling module, the target hidden layer feature of the target spectral feature is finally output. The target hidden layer feature can then be input into the next level sampling module until the final hidden layer feature is obtained. The hidden layer feature is a three-dimensional tensor [B, L, H], where B is the batch size, L is the number of features, and H is the hidden feature length.

[0093] Based on this, the sampling module reduces the spatial dimension of the input feature map through the above processing while retaining key feature information.

[0094] Step 208: Calculate and output the noise information corresponding to the audio to be processed based on the hidden layer features corresponding to the multiple spectral features through the attention pooling layer in the audio processing model.

[0095] In practical applications, after obtaining high-dimensional hidden layer feature values, the hidden layer features can be input into the attention pooling layer in the audio processing model. The attention pooling layer calculates the noise information in each hidden layer feature value and summarizes the noise information corresponding to the final audio to be processed.

[0096] The attention pooling layer is a pooling operation layer that integrates the attention mechanism. This allows the pooling operation to more intelligently weight and aggregate input features, improving the model's ability to capture key features and interpret input data. The attention pooling layer allows the model to focus more on learning the noise-related components of speech and, based on these components, calculate a Noisiness score that represents the "noise level" of the speech.

[0097] Furthermore, since, when calculating the noise information of the audio to be processed, the sub-noise information corresponding to each spectral feature is first calculated, and then the noise information of the audio to be processed is summarized and generated based on the sub-noise information corresponding to each spectral feature, the noise information corresponding to the audio to be processed is calculated specifically based on the hidden layer features corresponding to the multiple spectral features, including: calculating the sub-noise information corresponding to each hidden layer feature based on the hidden layer features corresponding to the multiple spectral features; and generating the noise information corresponding to the audio to be processed based on each sub-noise information.

[0098] The sub-noise information can be understood as the noise score calculated from the noise components included in each hidden layer feature, and then the noise information of the entire audio to be processed is generated according to the sub-noise information corresponding to each hidden layer feature.

[0099] In practical applications, the corresponding sub-noise information can be calculated based on the hidden features corresponding to each spectral feature. This sub-noise information is then aggregated using attention weights to generate the noise information for the processed audio. In summary, attention-weighted pooling applies an attention mechanism to the input speech feature sequence, dynamically assigning each position a weight reflecting the significance of its noise component. The value vectors are then weighted and summed according to these weights to extract the comprehensive feature that best represents the noise information. Finally, a fully connected layer maps this comprehensive feature into a single Noisiness score, which serves as a key metric for measuring the noise level of speech.

[0100] Furthermore, the process of calculating the sub-noise information corresponding to any one of the sub-noise information corresponding to each hidden layer feature includes: determining the hidden layer feature to be processed, and calculating the key vector, value vector and query vector corresponding to the hidden layer feature to be processed; according to the preset attention mechanism, using the query vector and the key vector to calculate the attention score; performing weighted summation on the value vector according to the attention score to obtain the feature vector corresponding to the hidden layer feature to be processed, and mapping the feature vector to sub-noise information through a fully connected layer.

[0101] The latent features to be processed can be understood as the latent features selected from multiple latent features for which sub-noise information needs to be calculated. In practical applications, the input latent feature sequence can be mapped to a key vector, a value vector, and a query vector. The key and value vectors together describe the feature information at each position, while the query vector represents the context or goal for identifying noise components. An attention mechanism (such as dot-product attention, multi-head attention, or self-attention) is used to calculate the similarity or correlation between the query vector and the key vector at each position. These similarity scores are normalized (e.g., using the softmax function) and converted into an attention weight distribution, where the weight at each position reflects the importance of the feature at that position for identifying noise components. Using the attention weight distribution obtained in the previous step, a weighted sum (or weighted average) is performed on the value vectors of the input sequence to focus on important features. The comprehensive feature vector obtained by attention weighted pooling is further processed through an additional fully connected layer to compress it into a scalar value, the Noisiness score. This score quantifies the overall strength or proportion of noise components in the input speech and can be used for subsequent analysis, decision-making, or optimization tasks.

[0102] In a specific embodiment of the present specification, the key vector, value vector and query vector corresponding to each hidden layer feature are calculated, and the attention score is calculated using the query vector and the key vector according to the preset attention mechanism. The value vector is weighted and summed according to the attention score to obtain the feature vector corresponding to the target hidden layer feature, and the feature vector is mapped to sub-noise information through the fully connected layer.

[0103] This specification provides an audio processing method, including determining an audio to be processed and inputting the audio to be processed into an audio processing model; converting the audio to be processed into a speech spectrogram through a feature extraction layer in the audio processing model, performing feature extraction on the speech spectrogram, and obtaining a plurality of spectral features, wherein two adjacent spectral features in the plurality of spectral features have overlapping features; performing information extraction on the plurality of spectral features through an information sampling layer in the audio processing model, and obtaining hidden layer features corresponding to the plurality of spectral features; and calculating and outputting noise information corresponding to the audio to be processed based on the hidden layer features corresponding to the plurality of spectral features through an attention pooling layer in the audio processing model. This method realizes that, according to the feature extraction layer in the audio processing model, the audio to be processed can be converted into a speech spectrogram, and feature extraction is performed on the speech spectrogram to obtain a plurality of spectral features, so that key features can be better learned and extracted from the speech signal, and there are overlapping features between two adjacent spectral features, further improving the accuracy of feature extraction. Subsequently, the information sampling layer extracts information from multiple spectral features to obtain the hidden features corresponding to the multiple spectral features. Then, the attention pooling layer uses the attention mechanism to enable the model to focus on analyzing the noise components in the speech, thereby improving the model's generalization ability to evaluate noise, and thus more accurately calculating the noise information corresponding to the audio to be processed.

[0104] The following further describes the audio processing method provided in this specification using the application of the audio processing method in speech recognition as an example, in conjunction with FIG4 . FIG4 shows a flowchart of the processing process of an audio processing method provided in one embodiment of this specification, which specifically includes the following steps.

[0105] Step 402: Determine the original audio and the audio adjustment strategy corresponding to the original audio.

[0106] In one feasible manner, the original audio is an audio segment input by a user through a chat software, and an audio adjustment strategy for the original audio is determined, where the audio adjustment strategy is used to adjust the duration of the audio.

[0107] Step 404: intercept and process the original audio according to the audio adjustment strategy to obtain the audio to be processed.

[0108] In one achievable manner, the audio duration of the original audio is cut to 20 seconds according to the audio adjustment strategy, thereby obtaining the audio to be processed.

[0109] Step 406: Input the audio to be processed into the audio processing model.

[0110] In one feasible method, the audio to be processed is input into the audio processing model carried by the chat software. The audio processing model is used to evaluate the noise level of the audio to be processed, so that the noise level of the current input audio can be reflected to the user based on the noise level, so that the user can determine whether the audio needs to be sent based on the noise level.

[0111] Step 408: Perform short-time Fourier transform on the audio to be processed through the feature extraction layer in the audio processing model to obtain an initial speech spectrogram corresponding to the audio to be processed.

[0112] In one feasible method, the audio to be processed is subjected to a short-time Fourier transform through a feature extraction layer in an audio processing model to obtain a short-time Fourier transform spectrum, which is used as an initial speech spectrum corresponding to the audio to be processed.

[0113] Step 410: Filter the initial speech spectrogram using a preset filter to obtain a speech spectrogram.

[0114] In one feasible manner, the short-time Fourier transform spectrum is filtered by a preset filter to obtain a speech spectrogram.

[0115] Step 412: Determine an audio extraction rule corresponding to the audio processing model, and determine extraction parameters corresponding to the speech spectrogram according to the audio extraction rule.

[0116] In one achievable manner, an audio extraction rule is determined, where the audio extraction rule includes extraction parameters, such as feature width, feature quantity, and other parameters.

[0117] Step 414: extracting features from the speech spectrogram according to the extraction parameters to obtain a plurality of spectrum features.

[0118] In one feasible manner, feature extraction is performed on the speech spectrogram according to the extraction parameters, and a plurality of spectrum features with a size of [B, L, T, F] are output.

[0119] Step 416: extract information from the multiple spectral features through the information sampling layer in the audio processing model to obtain hidden layer features corresponding to the multiple spectral features.

[0120] In one feasible method, a convolution operation is performed on multiple spectral features through a convolution layer in the information sampling layer to obtain a corresponding convolution feature map, a pooling operation is performed on the convolution feature map through a pooling layer in the information sampling layer to obtain a pooling feature map corresponding to the convolution feature map, and a normalization operation is performed on the pooling feature map through a normalization layer in the information sampling layer to obtain the corresponding hidden layer features.

[0121] Step 418: Calculate the sub-noise information corresponding to each hidden layer feature based on the hidden layer features corresponding to the multiple spectral features through the attention pooling layer in the audio processing model.

[0122] Step 420: Generate noise information corresponding to the audio to be processed according to each sub-noise information.

[0123] This specification provides an audio processing method that, based on the feature extraction layer in the audio processing model, can convert the processed audio into a speech spectrogram and perform feature extraction on the speech spectrogram to obtain multiple spectral features. This method can better learn and extract key features from the speech signal, and the overlapping features between two adjacent spectral features further improve the accuracy of feature extraction. Subsequently, the information sampling layer extracts information from the multiple spectral features to obtain the hidden layer features corresponding to the multiple spectral features. Then, through the attention pooling layer, the attention mechanism is used to enable the model to focus on analyzing the noise components in the speech, improving the model's generalization ability in evaluating noise, thereby more accurately calculating the noise information corresponding to the processed audio.

[0124] Referring to FIG5 , FIG5 shows a flowchart of a method for training an audio processing model according to an embodiment of the present specification, which specifically includes the following steps.

[0125] Step 502: Determine sample audio and sample noise information corresponding to the sample audio, and input the sample audio into an initial audio processing model.

[0126] The sample audio can be understood as a sample for the model, and the sample noise information corresponding to the sample audio can be understood as a sample label for the sample audio. In one embodiment, the sample label is the noise score of the sample audio. The initial audio processing model can be understood as a pre-trained model. For example, the initial audio processing model of this embodiment can be a model with the structure shown in Figure 1.

[0127] Step 504: The sample audio is converted into a speech spectrogram through the feature extraction layer in the audio initial processing model, and feature extraction is performed on the speech spectrogram to obtain a plurality of spectral features, wherein two adjacent spectral features in the plurality of spectral features have overlapping features.

[0128] The feature extraction layer extracts features from the sample audio. Since the sample audio is a time-domain speech signal, it must first be converted into frequency-domain data for analysis and processing. The feature extraction layer first converts the sample audio into a speech spectrogram, then extracts features from it to obtain multiple spectral features. Because multiple spectral features are generated by segmenting the speech spectrogram, to avoid losing relevant information in the audio due to segmentation or causing recognition errors due to segmentation, it is necessary to overlap two adjacent spectral features to generate overlapping features.

[0129] In an optional method, after obtaining the speech spectrogram, in order to better analyze and process the speech spectrogram, the speech spectrum can be segmented, and specifically, feature extraction is performed on the speech spectrogram to obtain multiple spectral features, including: determining the audio extraction rules corresponding to the audio initial processing model, and determining the extraction parameters corresponding to the speech spectrogram according to the audio extraction rules; performing feature extraction on the speech spectrogram according to the extraction parameters to obtain multiple spectral features.

[0130] The audio extraction rules may be pre-set extraction rules for the initial audio processing model, and the extraction rules may include corresponding extraction parameters, such as extraction width and extraction quantity. The extraction parameters can be understood as extraction size parameters for spectral features. Feature extraction is performed on the speech spectrogram based on the extraction parameters, thereby obtaining multiple spectrograms that meet the extraction size parameters. In specific implementations, after determining the extraction parameters, the speech spectrogram can be intercepted based on the extraction parameters. The extraction parameters include feature width T, feature height F, feature quantity L, and total feature length B.

[0131] In a specific embodiment of the present specification, the audio extraction rules of the audio initial processing model are determined, and the extraction parameters are determined according to the audio extraction rules. The extraction parameters include the width (time), height (Mel scale), number and total length of each spectral feature. According to the extraction parameters, multiple spectral features of size [B, L, T, F] can be obtained.

[0132] Furthermore, the actual processing process of feature extraction of the speech spectrogram is to segment the speech spectrogram. Specifically, feature extraction of the speech spectrogram is performed according to the extraction parameters, and obtaining multiple spectral features may include: segmenting the speech spectrogram according to the extraction parameters to obtain multiple initial spectral features; determining the overlapping spectral features corresponding to the multiple initial spectral features, and generating multiple spectral features based on the overlapping spectral features and the multiple initial spectral features.

[0133] Among them, after the speech spectrum graph is segmented, multiple initial spectrum features after segmentation can be obtained. After combining the multiple initial spectrum features, the entire speech spectrum graph is obtained. In order to avoid information loss or subsequent analysis errors caused by segmentation, it is necessary to overlap the initial spectrum features, that is, overlap adjacent spectrum features. Therefore, after obtaining multiple initial spectrum features, overlapping spectrum features can be determined based on the adjacent spectrum features of each initial spectrum feature. The overlapping spectrum feature is the spectrum feature that needs to be superimposed on each initial spectrum feature. After determining each initial spectrum feature and its corresponding overlapping spectrum feature that needs to be overlapped, multiple spectrum features can be generated. In actual applications, the overlapping spectrum feature corresponding to each initial spectrum feature includes two overlapping spectrum features, namely, a preceding overlapping spectrum feature and a following overlapping spectrum feature. The preceding overlapping spectrum feature is determined based on the previous initial spectrum feature of the initial spectrum feature, and the following overlapping spectrum feature is determined based on the next initial spectrum feature of the initial spectrum feature.

[0134] In a specific embodiment of the present specification, after the speech spectrogram is segmented according to the extraction parameters, multiple initial spectral features after segmentation can be obtained, and the overlapping spectral features corresponding to each initial spectral feature are determined. The initial spectral features are superimposed according to the overlapping spectral features corresponding to each initial spectral feature, thereby generating multiple spectral features.

[0135] Based on this, by overlapping the spectral features, it is ensured that there is no information loss between each spectral feature, and the subsequent audio processing and analysis can be accurately performed.

[0136] Step 506: extract information from the multiple spectral features through the information sampling layer in the initial audio processing model to obtain hidden layer features corresponding to the multiple spectral features.

[0137] After extracting multiple spectral features from the audio through the feature extraction layer in the audio initial processing model, the spectral features can be input into the information sampling layer, and then the hidden features can be extracted through the information sampling layer. The hidden features are high-dimensional hidden layer feature values. The information sampling layer is the downsampling layer. The information sampling layer can adopt a CNN convolutional neural network and use 2D convolution to extract information from the spectral features, thereby obtaining high-dimensional hidden layer feature values. In specific implementation, the information sampling layer can include three sampling modules, each of which includes a convolution layer, a pooling layer, and a normalization layer. The three sampling modules form a multi-level downsampling structure. Each level of downsampling will further reduce the spatial resolution of the feature while capturing higher-level abstract features.

[0138] Based on this, the information sampling layer reduces the spatial dimensionality of the input features while retaining key feature information. This helps reduce the number of model parameters and computational complexity, improves model efficiency, and helps extract higher-level, more abstract features, which is more beneficial for subsequent analysis. Downsampling also helps increase the model's robustness to small translations, rotations, or scaling of the input data, thereby improving the model's translation invariance.

[0139] Step 508: Calculate the predicted noise information corresponding to the sample audio according to the hidden layer features corresponding to the multiple spectral features through the attention pooling layer in the initial audio processing model.

[0140] The attention pooling layer is a pooling operation layer that integrates the attention mechanism. This allows the pooling operation to more intelligently perform weighted aggregation on input features, improving the model's ability to capture key features and interpret input data. The attention pooling layer allows the model to focus more on learning the noise-related components of speech and, based on these components, calculate a Noisiness score that represents the "noise level" of the speech.

[0141] Furthermore, when calculating the noise information of the sample audio, the sub-noise information corresponding to each spectral feature is first calculated, and then the noise information of the sample audio is summarized and generated based on the sub-noise information corresponding to each spectral feature. Specifically, the noise information corresponding to the sample audio is calculated based on the hidden layer features corresponding to the multiple spectral features, including: calculating the sub-noise information corresponding to each hidden layer feature based on the hidden layer features corresponding to the multiple spectral features; and generating predicted noise information corresponding to the sample audio based on each sub-noise information.

[0142] The sub-noise information can be understood as the noise score calculated from the noise components included in each hidden layer feature. Then, the predicted noise information of the entire sample audio is generated based on the sub-noise information corresponding to each hidden layer feature.

[0143] In practical applications, the corresponding sub-noise information can be calculated based on the hidden features corresponding to each spectral feature. This sub-noise information is then aggregated using attention weights to generate the noise information for the sample audio. In summary, attention-weighted pooling applies an attention mechanism to the input speech feature sequence, dynamically assigning each position a weight reflecting the significance of its noise component. The value vectors are then weighted and summed according to these weights to extract the comprehensive feature that best represents the noise information. Finally, a fully connected layer maps this comprehensive feature into a single Noisiness score, which serves as a key metric for measuring the noise level of speech.

[0144] Furthermore, the process of calculating the sub-noise information corresponding to any hidden layer feature in the sub-noise information corresponding to each hidden layer feature includes: determining the hidden layer feature to be processed, and calculating the key vector, value vector and query vector corresponding to the hidden layer feature to be processed; according to the preset attention mechanism, using the query vector and the key vector to calculate the attention score; performing weighted summation on the value vector according to the attention score to obtain the feature vector corresponding to the hidden layer feature to be processed, and mapping the feature vector to sub-noise information through the fully connected layer.

[0145] Based on the above processing results of the attention pooling layer, the predicted noise information can be obtained. The predicted noise information can be understood as the noise score estimated by the initial audio processing model for the sample model.

[0146] Step 510: Calculate a model loss value based on the sample noise information and the predicted noise information, adjust the model parameters of the audio initial processing model based on the model loss value, and continue training the audio initial processing model until an audio processing model that meets the training conditions is obtained.

[0147] In actual applications, after the sample audio is predicted by the audio initial processing model, the predicted noise information is obtained, and then the model loss value can be calculated based on the predicted noise information and the sample noise information. The model loss value is then used to adjust the parameters of the audio initial processing model, and training is continued until an audio processing model that meets the model training conditions is trained. In one example, the loss value of the model can be calculated based on a preset loss function. In this embodiment, the loss function can be implemented by those skilled in the art according to actual needs, selecting any appropriate supervised loss function. This embodiment does not limit the specific implementation of the loss function. Among them, the training conditions may include meeting the training rounds or the model parameters reaching a threshold.

[0148] Through the above training process of this embodiment, an audio processing model that can implement the above audio processing method can be obtained.

[0149] This manual provides a training method for an audio processing model. By inputting a large amount of training data, as well as using a neural network and self-attention mechanism, the model can adapt to various noise conditions and improve the generalization ability and accuracy of evaluation.

[0150] Referring to FIG. 6 , FIG. 6 shows a flowchart of another audio processing method provided according to an embodiment of this specification, which specifically includes the following steps.

[0151] Step 602: Receive and parse the audio file input by the user, determine the audio to be processed, and input the audio to be processed into the audio processing model.

[0152] The audio file input by the user can be understood as an audio file generated by converting the user's voice, or it can be an audio file directly input by the user. By parsing the audio file, the audio to be processed can be determined, and the processed audio can be input into the audio processing model, so that the noise score of the processed audio can be evaluated by the audio processing model.

[0153] Step 604: The audio to be processed is converted into a speech spectrogram through a feature extraction layer in the audio processing model, and feature extraction is performed on the speech spectrogram to obtain a plurality of spectral features, wherein two adjacent spectral features in the plurality of spectral features have overlapping features.

[0154] Step 606: extract information from the multiple spectral features through the information sampling layer in the audio processing model to obtain hidden layer features corresponding to the multiple spectral features.

[0155] Step 608: Calculate the noise information corresponding to the audio to be processed based on the hidden layer features corresponding to the multiple spectral features through the attention pooling layer in the audio processing model and feed it back to the user.

[0156] In practical applications, after the noise information of the audio to be processed is calculated by the audio processing model, the noise information can be fed back to the user so that the user can determine whether to continue using the audio based on the noise information.

[0157] Furthermore, after calculating the noise information corresponding to the audio to be processed and feeding it back to the user, it also includes: receiving an adjustment instruction from the user for the audio file based on the noise information; adjusting the audio information of the audio file according to the adjustment instruction, and feeding back the adjusted audio file to the user.

[0158] In practical applications, adjustment instructions can be understood as user-made adjustments to audio files based on noise information, such as suppressing audio noise. For example, in a conference recording scenario, when the audio file is a recording of a conference, the recording software uses an embedded audio processing model to provide the user with feedback on the noise information, i.e., the noise score, of the conference recording. This allows the user to understand the quality of the conference recording. Upon learning the noise score, the user can issue an adjustment instruction to the audio file. The system can then automatically perform noise reduction on the conference recording and return the noise-reduced audio file to the user. In some cases, the adjustment instruction can also be an instruction to reselect the audio file, i.e., the user can re-enter the audio file or reselect the audio file. For example, in a chat application scenario, after a user inputs a voice audio segment using the chat application, the chat application uses an embedded audio processing model to provide the user with feedback on the noise score of the voice audio segment, so that the user is aware of the clarity of the voice to be sent, allowing the user to choose whether to send it or re-enter it. After the adjustment, the audio that meets the user's hearing perception can be fed back to the user, or the adjusted audio file can be evaluated and the audio file that meets the evaluation requirements can be fed back to the user. In specific implementation, after obtaining the noise information of the audio file, the audio file can be directly adjusted without receiving the user's adjustment instructions, such as noise suppression, automatic noise reduction, etc., so as to directly feedback clear audio to the user and provide the user with a more convenient and better quality voice audio service.

[0159] This specification provides an audio processing method that, based on the feature extraction layer in the audio processing model, can convert the audio to be processed into a speech spectrogram, and perform feature extraction on the speech spectrogram to obtain multiple spectral features, which can better learn and extract key features from the speech signal, and there are overlapping features between two adjacent spectral features, further improving the accuracy of feature extraction. Subsequently, the information sampling layer is used to extract information from the multiple spectral features to obtain the hidden layer features corresponding to the multiple spectral features. Then, through the attention pooling layer, the attention mechanism is used to enable the model to focus on analyzing the noise components in the speech, thereby improving the generalization ability of the model to evaluate noise, and thus more accurately calculating the noise information corresponding to the audio to be processed. This facilitates users to make decisions on the audio to be processed based on the noise information, providing users with better speech recognition services.

[0160] Corresponding to the above method embodiment, this specification also provides an audio processing device embodiment. FIG7 shows a schematic diagram of the structure of an audio processing device provided by one embodiment of this specification. As shown in FIG7, the device includes:

[0161] An input module 702 is configured to determine audio to be processed and input the audio to be processed into an audio processing model;

[0162] The conversion module 704 is configured to convert the to-be-processed audio into a speech spectrogram through a feature extraction layer in the audio processing model, perform feature extraction on the speech spectrogram, and obtain a plurality of spectral features, wherein two adjacent spectral features in the plurality of spectral features have overlapping features;

[0163] An information extraction module 706 is configured to extract information from the plurality of spectral features through an information sampling layer in the audio processing model to obtain hidden layer features corresponding to the plurality of spectral features;

[0164] The calculation and output module 708 is configured to calculate and output the noise information corresponding to the audio to be processed based on the hidden layer features corresponding to the multiple spectral features through the attention pooling layer in the audio processing model.

[0165] Optionally, the input module 702 is further configured to determine original audio and an audio adjustment strategy corresponding to the original audio; and intercept and process the original audio according to the audio adjustment strategy to obtain audio to be processed.

[0166] Optionally, the conversion module 704 is further configured to perform short-time Fourier transform on the audio to be processed to obtain an initial speech spectrogram corresponding to the audio to be processed; and filter the initial speech spectrogram through a preset filter to obtain a speech spectrogram.

[0167] Optionally, the conversion module 704 is further configured to determine the audio extraction rules corresponding to the audio processing model, and determine the extraction parameters corresponding to the speech spectrogram according to the audio extraction rules; perform feature extraction on the speech spectrogram according to the extraction parameters to obtain multiple spectrum features.

[0168] Optionally, the conversion module 704 is further configured to segment the speech spectrogram according to the extraction parameters to obtain multiple initial spectral features; determine overlapping spectral features corresponding to the multiple initial spectral features, and generate multiple spectral features based on the overlapping spectral features and the multiple initial spectral features.

[0169] Optionally, the information extraction module 706 is further configured to perform a convolution operation on a target spectral feature among the multiple spectral features through the convolution layer in the information sampling layer to obtain a convolution feature map corresponding to the target spectral feature; perform a pooling operation on the convolution feature map through the pooling layer in the information sampling layer to obtain a pooling feature map corresponding to the convolution feature map; and perform a normalization operation on the pooling feature map through the normalization layer in the information sampling layer to obtain a target hidden layer feature corresponding to the target spectral feature.

[0170] Optionally, the calculation and output module 708 is further configured to calculate sub-noise information corresponding to each hidden layer feature according to the hidden layer features corresponding to the multiple spectral features; and generate noise information corresponding to the audio to be processed according to each sub-noise information.

[0171] Optionally, the calculation and output module 708 is further configured to determine the hidden layer features to be processed, and calculate the key vector, value vector and query vector corresponding to the hidden layer features to be processed; calculate the attention score using the query vector and the key vector according to the preset attention mechanism; perform weighted summation on the value vector according to the attention score to obtain the feature vector corresponding to the hidden layer features to be processed, and map the feature vector to sub-noise information through a fully connected layer.

[0172] This specification provides an audio processing device that, based on a feature extraction layer in an audio processing model, can convert the audio to be processed into a speech spectrogram and perform feature extraction on the speech spectrogram to obtain multiple spectral features. This device can better learn and extract key features from the speech signal, and the overlapping features between adjacent spectral features further improve the accuracy of feature extraction. Subsequently, the information sampling layer extracts information from the multiple spectral features to obtain the hidden layer features corresponding to the multiple spectral features. Then, through the attention pooling layer, the attention mechanism is used to enable the model to focus on analyzing the noise components in the speech, improving the model's generalization ability in evaluating noise, thereby more accurately calculating the noise information corresponding to the audio to be processed.

[0173] The above is a schematic scheme of an audio processing device of this embodiment. It should be noted that the technical scheme of the audio processing device and the technical scheme of the above-mentioned audio processing method are based on the same concept. For details not described in detail in the technical scheme of the audio processing device, please refer to the description of the technical scheme of the above-mentioned audio processing method.

[0174] Corresponding to the above method embodiments, this specification also provides an embodiment of a training device for an audio processing model. FIG8 shows a schematic diagram of the structure of a training device for an audio processing model provided in one embodiment of this specification. As shown in FIG8 , the device includes:

[0175] An input module 802 is configured to determine a sample audio and sample noise information corresponding to the sample audio, and input the sample audio into an initial audio processing model;

[0176] The conversion module 804 is configured to convert the sample audio into a speech spectrogram through a feature extraction layer in the audio initial processing model, perform feature extraction on the speech spectrogram, and obtain a plurality of spectral features, wherein two adjacent spectral features in the plurality of spectral features have overlapping features;

[0177] The information extraction module 806 is configured to extract information from the multiple spectral features through the information sampling layer in the audio initial processing model to obtain hidden layer features corresponding to the multiple spectral features;

[0178] A calculation module 808 is configured to calculate predicted noise information corresponding to the sample audio according to hidden layer features corresponding to the multiple spectral features through an attention pooling layer in the initial audio processing model;

[0179] The training module 810 is configured to calculate the model loss value based on the sample noise information and the predicted noise information, adjust the model parameters of the audio initial processing model based on the model loss value, and continue to train the audio initial processing model until an audio processing model that meets the training conditions is obtained.

[0180] The above is a schematic diagram of a training device for an audio processing model according to this embodiment. It should be noted that the technical solution of the training device for the audio processing model is based on the same concept as the technical solution of the audio processing method described above. For details not described in detail in the technical solution of the training device for the audio processing model, please refer to the description of the technical solution of the training method for the audio processing model described above.

[0181] Corresponding to the above method embodiment, this specification also provides an audio processing device embodiment. FIG9 shows a schematic diagram of the structure of another audio processing device provided in one embodiment of this specification. As shown in FIG9, the device includes:

[0182] An input module 902 is configured to receive and parse an audio file input by a user, determine audio to be processed, and input the audio to be processed into an audio processing model;

[0183] The conversion module 904 is configured to convert the to-be-processed audio into a speech spectrogram through a feature extraction layer in the audio processing model, perform feature extraction on the speech spectrogram, and obtain a plurality of spectral features, wherein two adjacent spectral features in the plurality of spectral features have overlapping features;

[0184] An information extraction module 906 is configured to extract information from the multiple spectral features through the information sampling layer in the audio processing model to obtain hidden layer features corresponding to the multiple spectral features;

[0185] The calculation and feedback module 908 is configured to calculate the noise information corresponding to the audio to be processed and feed it back to the user based on the hidden layer features corresponding to the multiple spectral features through the attention pooling layer in the audio processing model.

[0186] The above is a schematic scheme of an audio processing device of this embodiment. It should be noted that the technical scheme of the audio processing device and the technical scheme of the above-mentioned audio processing method are based on the same concept. For details not described in detail in the technical scheme of the audio processing device, please refer to the description of the technical scheme of the above-mentioned audio processing method.

[0187] Figure 10 shows a block diagram of a computing device 1000 according to one embodiment of this specification. Components of the computing device 1000 include, but are not limited to, a memory 1010 and a processor 1020. The processor 1020 is connected to the memory 1010 via a bus 1030, and a database 1050 is used to store data.

[0188] The computing device 1000 also includes an access device 1040 that enables the computing device 1000 to communicate via one or more networks 1060. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1040 may include one or more of any type of network interface (e.g., a network interface controller (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, or a near field communication (NFC) interface.

[0189] In one embodiment of the present specification, the aforementioned components of the computing device 1000 and other components not shown in FIG10 may also be connected to each other, for example, via a bus. It should be understood that the computing device structure block diagram shown in FIG10 is for illustrative purposes only and does not limit the scope of this specification. Those skilled in the art may add or replace other components as needed.

[0190] Computing device 1000 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 1000 may also be a mobile or stationary server.

[0191] Among them, the processor 1020 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-mentioned audio processing method and audio processing model training method.

[0192] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the computing device embodiment is generally similar to the audio processing method and audio processing model training method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the audio processing method and audio processing model training method embodiments.

[0193] An embodiment of the present specification also provides a computer-readable storage medium storing a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned audio processing method and audio processing model training method.

[0194] Each embodiment in this specification is described in a progressive manner. Similar or identical parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the computer-readable storage medium embodiment is generally similar to the audio processing method and audio processing model training method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the audio processing method and audio processing model training method embodiments.

[0195] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned audio processing method and audio processing model training method.

[0196] The above is a schematic diagram of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product is based on the same concept as the technical solutions of the aforementioned audio processing method and audio processing model training method. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solutions of the aforementioned audio processing method and audio processing model training method.

[0197] FIG11 shows an exemplary system for an audio processing method applicable to an embodiment of this specification. As shown in FIG11 , the system 1100 may include a cloud server 1102, a network 1060, and / or one or more user devices 1106. FIG11 shows an example of multiple user devices. It should be noted that the audio processing method provided in the embodiment of this specification can be deployed in the cloud server 1102 and implemented by the cloud server 1102. However, the method is not limited thereto. In another feasible embodiment, the solution of the embodiment of this specification can also be deployed in a user device 1106 having higher hardware and software performance and implemented by the user device 1106.

[0198] The cloud server 1102 can be any appropriate device for storing information, data, programs and / or any other suitable type of content, including but not limited to distributed storage system devices, server clusters, computing cloud server clusters, etc. In some embodiments, the cloud server 1102 can perform any appropriate function. For example, in some embodiments, when the solution of the embodiments of this specification is deployed on the cloud server, the cloud server 1102 can be used to calculate the noise information corresponding to the audio to be processed. As an optional example, in some embodiments, the cloud server 1102 can convert the audio to be processed into a speech spectrogram, perform feature extraction on the speech spectrogram, and obtain multiple spectral features, wherein there are overlapping features between two adjacent spectral features in the multiple spectral features; then perform information extraction on the multiple spectral features to obtain hidden layer features corresponding to the multiple spectral features; and then calculate the noise information corresponding to the audio to be processed based on the hidden layer features corresponding to the multiple spectral features. In some embodiments, the cloud server 1102 can be used to send the noise information to the user device 1106.

[0199] In some embodiments, the user device 1106 can be connected to the network 1060 via one or more communication links (e.g., communication link 1112), and the network 1060 can be linked to the cloud service 1102 via one or more communication links (e.g., communication link 1114). The communication link can be any communication link suitable for transmitting data between the user device 1106 and the cloud service 1102, such as a network link, a dial-up link, a wireless link, a hardwired link, any other suitable communication link, or any suitable combination of such links.

[0200] User device 1106 may include any one or more user devices equipped with an audio capture device, such as a microphone, an audio output device, such as a speaker, and suitable for user interaction. However, when implementing the embodiments of this specification, user device 1106 may calculate and output noise information corresponding to the processed audio based on the processed audio. As an alternative example, in some embodiments, user device 1106 may convert the processed audio into a speech spectrogram using a feature extraction layer in an audio processing model, perform feature extraction on the speech spectrogram, and obtain multiple spectral features, wherein adjacent spectral features in the multiple spectral features have overlapping features. Furthermore, information extraction may be performed on the multiple spectral features using an information sampling layer in the audio processing model to obtain hidden features corresponding to the multiple spectral features. Finally, an attention pooling layer in the audio processing model may be used to calculate and output noise information corresponding to the processed audio based on the hidden features corresponding to the multiple spectral features. In some embodiments, user device 1106 may include any suitable type of device. For example, in some embodiments, user device 1106 may include a mobile device, a tablet computer, a laptop computer, a desktop computer, a wearable computer, and / or any other suitable type of user device.

[0201] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0202] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0203] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0204] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0205] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An audio processing method, comprising: Determining audio to be processed, and inputting the audio to be processed into an audio processing model; The audio to be processed is converted into a speech spectrogram by a feature extraction layer in the audio processing model, and feature extraction is performed on the speech spectrogram to obtain a plurality of spectral features, wherein two adjacent spectral features in the plurality of spectral features have overlapping features; Extracting information from the multiple spectral features through the information sampling layer in the audio processing model to obtain hidden layer features corresponding to the multiple spectral features; Through the attention pooling layer in the audio processing model, the noise information corresponding to the audio to be processed is calculated and output according to the hidden layer features corresponding to the multiple spectral features.

2. The method according to claim 1, wherein Identify the audio to be processed, including: Determining original audio and an audio adjustment strategy corresponding to the original audio; The original audio is intercepted and processed according to the audio adjustment strategy to obtain audio to be processed.

3. The method according to claim 1 or 2, wherein: Converting the audio to be processed into a speech spectrogram, comprising: Performing a short-time Fourier transform on the audio to be processed to obtain an initial speech spectrogram corresponding to the audio to be processed; The initial speech spectrogram is filtered using a preset filter to obtain a speech spectrogram.

4. The method according to any one of claims 1 to 3, wherein Feature extraction is performed on the speech spectrogram to obtain multiple spectrum features, including: Determining an audio extraction rule corresponding to the audio processing model, and determining an extraction parameter corresponding to the speech spectrogram according to the audio extraction rule; Feature extraction is performed on the speech spectrogram according to the extraction parameters to obtain multiple spectrum features.

5. The method according to claim 4, wherein Performing feature extraction on the speech spectrogram according to the extraction parameters to obtain a plurality of spectrum features, including: Segmenting the speech spectrogram according to the extraction parameters to obtain a plurality of initial spectrum features; Overlapping spectrum features corresponding to the multiple initial spectrum features are determined, and multiple spectrum features are generated according to the overlapping spectrum features and the multiple initial spectrum features.

6. The method according to any one of claims 1 to 5, wherein Extracting information from any one of the plurality of spectrum features includes: Performing a convolution operation on a target spectrum feature among the multiple spectrum features through a convolution layer in the information sampling layer to obtain a convolution feature map corresponding to the target spectrum feature; Performing a pooling operation on the convolution feature map through the pooling layer in the information sampling layer to obtain a pooled feature map corresponding to the convolution feature map; The pooling feature map is normalized by the normalization layer in the information sampling layer to obtain the target hidden layer features corresponding to the target spectrum features.

7. The method according to any one of claims 1 to 6, wherein Calculating noise information corresponding to the audio to be processed according to hidden layer features corresponding to the multiple spectral features includes: Calculating sub-noise information corresponding to each hidden layer feature according to the hidden layer features corresponding to the multiple spectral features; Noise information corresponding to the audio to be processed is generated according to each sub-noise information.

8. The method according to claim 7, wherein: The process of calculating the sub-noise information corresponding to any hidden layer feature in the sub-noise information corresponding to each hidden layer feature includes: Determine the hidden layer features to be processed, and calculate the key vector, value vector and query vector corresponding to the hidden layer features to be processed; Calculating an attention score using the query vector and the key vector according to a preset attention mechanism; The value vector is weightedly summed according to the attention score to obtain a feature vector corresponding to the hidden layer feature to be processed, and the feature vector is mapped into sub-noise information through a fully connected layer.

9. A method for training an audio processing model, comprising: Determining sample audio and sample noise information corresponding to the sample audio, and inputting the sample audio into an audio initial processing model; The sample audio is converted into a speech spectrogram by a feature extraction layer in the audio initial processing model, and feature extraction is performed on the speech spectrogram to obtain a plurality of spectral features, wherein two adjacent spectral features in the plurality of spectral features have overlapping features; Extracting information from the multiple spectral features through the information sampling layer in the initial audio processing model to obtain hidden layer features corresponding to the multiple spectral features; Calculating predicted noise information corresponding to the sample audio according to hidden layer features corresponding to the multiple spectral features through the attention pooling layer in the initial audio processing model; A model loss value is calculated according to the sample noise information and the predicted noise information, and model parameters of the audio initial processing model are adjusted based on the model loss value, and the audio initial processing model is continuously trained until an audio processing model that meets the training conditions is obtained.

10. The method according to claim 9, wherein: Feature extraction is performed on the speech spectrogram to obtain multiple spectrum features, including: Determining an audio extraction rule corresponding to the audio initial processing model, and determining an extraction parameter corresponding to the speech spectrogram according to the audio extraction rule; Feature extraction is performed on the speech spectrogram according to the extraction parameters to obtain multiple spectrum features.

11. The method according to claim 10, wherein: Performing feature extraction on the speech spectrogram according to the extraction parameters to obtain a plurality of spectrum features, including: Segmenting the speech spectrogram according to the extraction parameters to obtain a plurality of initial spectrum features; Overlapping spectrum features corresponding to the multiple initial spectrum features are determined, and multiple spectrum features are generated according to the overlapping spectrum features and the multiple initial spectrum features.

12. The method according to any one of claims 9 to 11, wherein: Calculating predicted noise information corresponding to the sample audio according to hidden layer features corresponding to the multiple spectral features, including: Calculating sub-noise information corresponding to each hidden layer feature according to the hidden layer features corresponding to the multiple spectral features; The predicted noise information corresponding to the sample audio is generated according to each sub-noise information.

13. The method according to claim 12, wherein: The process of calculating the sub-noise information corresponding to any hidden layer feature in the sub-noise information corresponding to each hidden layer feature includes: Determine the hidden layer features to be processed, and calculate the key vector, value vector and query vector corresponding to the hidden layer features to be processed; Calculating an attention score using the query vector and the key vector according to a preset attention mechanism; The value vector is weightedly summed according to the attention score to obtain a feature vector corresponding to the hidden layer feature to be processed, and the feature vector is mapped into sub-noise information through a fully connected layer.

14. An audio processing method, comprising: Receive and parse the audio file input by the user, determine the audio to be processed, and input the audio to be processed into the audio processing model; The audio to be processed is converted into a speech spectrogram by a feature extraction layer in the audio processing model, and feature extraction is performed on the speech spectrogram to obtain a plurality of spectral features, wherein two adjacent spectral features in the plurality of spectral features have overlapping features; Extracting information from the multiple spectral features through the information sampling layer in the audio processing model to obtain hidden layer features corresponding to the multiple spectral features; Through the attention pooling layer in the audio processing model, noise information corresponding to the audio to be processed is calculated according to the hidden layer features corresponding to the multiple spectral features and fed back to the user.

15. The method according to claim 14, wherein After calculating the noise information corresponding to the audio to be processed and feeding it back to the user, the method further includes: receiving an instruction from the user to adjust the audio file based on noise information; The audio information of the audio file is adjusted according to the adjustment instruction, and the adjusted audio file is fed back to the user.

16. A computing device comprising: memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method as described in any one of claims 1 to 8 are implemented; or, the steps of the method as described in any one of claims 9 to 13 are implemented; or, the steps of the method as described in any one of claims 14 to 15 are implemented.

17. A computer-readable storage medium storing a computer program / instruction, which, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 8; or, implements the steps of the method as described in any one of claims 9 to 13; or, implements the steps of the method as described in any one of claims 14 to 15.

18. A computer program product comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8; or, implements the steps of the method according to any one of claims 9 to 13; or, implements the steps of the method according to any one of claims 14 to 15.

Images