Audio detection method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses an audio detection method and device, equipment and a storage medium. The method comprises the following steps: converting an original mixed audio signal into an audio spectrum in a frequency domain; performing feature enhancement processing on the spectrum features of the audio spectrum through a frequency sensing joint module constructed in the target audio detection model to obtain frequency sensing spectrum features corresponding to the audio spectrum; performing feature analysis on the frequency sensing spectrum features through a feature analysis joint module constructed in a target audio detection model to obtain a feature analysis result corresponding to each audio frame in the original mixed audio signal; and for each audio frame in the original mixed audio signal, determining an audio type detection result of the audio frame according to the feature analysis result corresponding to the audio frame. The improved audio detection model is adopted to participate in audio detection, so that the parameter quantity and the calculation quantity of audio detection are reduced on the basis of ensuring the audio processing accuracy, and the calculation performance and the processing time of audio detection are effectively improved.

Description

Technical Field

[0001] This disclosure relates to the field of audio processing technology, and in particular to an audio detection method, apparatus, device, and storage medium. Background Technology

[0002] In daily life, the mixed audio that is actually recorded or played back is generally composed of three categories: speech, music, and noise. Detecting these three types of audio in the mixed audio can effectively assist subsequent audio effects processing or encoding / decoding processing. For example, by detecting the audio type, corresponding loudness equalization can be performed to improve sound quality, or noise reduction strategies can be adjusted to better protect the music and remove noise. Furthermore, the bitrate of encoding / decoding can be dynamically adjusted based on the detection results to ensure sound quality and save costs.

[0003] Current audio detection methods are mainly based on network models. However, when using existing network models for audio detection, problems arise such as the network model structure being too complex to be integrated into lightweight devices, or the need to significantly increase the number of network model parameters to ensure the accuracy of audio detection, resulting in a greater computational burden. Summary of the Invention

[0004] This disclosure provides an audio detection method, apparatus, device, and storage medium that, while ensuring the accuracy of audio detection, effectively reduces the amount of network parameters used and the amount of computing resources consumed in the audio detection process.

[0005] In a first aspect, embodiments of this disclosure provide an audio detection method, the method comprising:

[0006] Convert the original mixed audio signal into an audio spectrum in the frequency domain;

[0007] The frequency-aware joint module built in the target audio detection model performs feature enhancement processing on the spectral features of the audio spectrum to obtain the frequency-aware spectral features corresponding to the audio spectrum. The frequency-aware joint module includes multiple frequency-aware network layers.

[0008] The feature analysis joint module built in the target audio detection model performs feature analysis on the frequency-aware spectral features to obtain the feature analysis results corresponding to each audio frame in the original mixed audio signal. The feature analysis results include the probability values ​​of the audio frames being audio signals of different audio types.

[0009] For each audio frame in the original mixed audio signal, the audio type detection result of the audio frame is determined based on the feature analysis results corresponding to the audio frame.

[0010] Secondly, embodiments of this disclosure also provide an audio detection device, the device comprising:

[0011] The audio conversion module is used to convert the original mixed audio signal into an audio spectrum in the frequency domain;

[0012] The enhancement processing module is used to perform feature enhancement processing on the spectral features of the audio spectrum through the frequency sensing joint module built in the target audio detection model to obtain the frequency sensing spectral features corresponding to the audio spectrum. The frequency sensing joint module includes multiple frequency sensing network layers.

[0013] The audio analysis module is used to perform feature analysis on the frequency-aware spectral features through the feature analysis joint module built in the target audio detection model, and obtain the feature analysis results corresponding to each audio frame in the original mixed audio signal. The feature analysis results include the probability values ​​of the audio frames being audio signals of different audio types.

[0014] The result detection module is used to determine the audio type detection result of each audio frame in the original mixed audio signal based on the feature analysis results corresponding to the audio frame.

[0015] Thirdly, embodiments of this disclosure also provide a computer device, the computer device comprising:

[0016] One or more processors;

[0017] Storage device for storing one or more programs.

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the audio detection method provided in any embodiment of this disclosure.

[0019] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the audio detection method provided in any embodiment of this disclosure.

[0020] Fifthly, embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements the audio detection method provided in any embodiment of this disclosure.

[0021] The technical solution of this disclosure specifically discloses an audio detection method, apparatus, device, and storage medium. The method first converts the original mixed audio signal into an audio spectrum in the frequency domain. Then, through a frequency-aware joint module built in the target audio detection model, feature enhancement processing is performed on the spectral features of the audio spectrum to obtain the frequency-aware spectral features corresponding to the audio spectrum. The frequency-aware joint module includes multiple frequency-aware network layers. Next, through a feature analysis joint module built in the target audio detection model, feature analysis is performed on the frequency-aware spectral features to obtain the feature analysis results corresponding to each audio frame in the original mixed audio signal. The feature analysis results include probability values ​​for audio frames representing different audio types. Finally, for each audio frame in the original mixed audio signal, the audio type detection result of the audio frame is determined based on the feature analysis results corresponding to the audio frame. In existing audio detection implementations, audio detection based on network models often suffers from problems such as overly complex network structures unsuitable for integration into lightweight devices, or the need to significantly increase the number of network model parameters to ensure audio detection accuracy, resulting in increased computational demands. This technical solution addresses this by using an audio detection model that includes a frequency-aware joint module and a feature analysis joint module. The frequency-aware joint module comprises multiple frequency-aware network layers. By improving this module, the number of parameters in the audio detection model is reduced, thus lowering the computational load while maintaining audio processing accuracy. This lightweight approach effectively improves the computational performance and processing time of audio detection. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the exemplary embodiments of this disclosure, the accompanying drawings used in describing the embodiments are briefly introduced below. Obviously, the accompanying drawings described are only a portion of the embodiments to be described in this disclosure, and not all of them. For those skilled in the art, other drawings can be obtained from these drawings without any creative effort.

[0023] Figure 1a A schematic flowchart of an audio detection method provided in an embodiment of this disclosure;

[0024] Figure 1b A structural example diagram of the target audio detection model involved in the audio detection method provided in this embodiment is given;

[0025] Figure 1c A structural example diagram of the frequency-aware convolutional layer of the target audio detection model involved in the audio detection method provided in this embodiment is given;

[0026] Figure 1d A structural example diagram of the frequency-aware filter layer of the target audio detection model involved in the audio detection method provided in this embodiment is given;

[0027] Figure 2 This is a schematic diagram of the structure of an audio detection device provided in an embodiment of the present disclosure;

[0028] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure. Detailed Implementation

[0029] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0030] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0031] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0032] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules, or units, and are not used to limit the order of functions performed by these devices, modules, or units or their interdependencies. It should also be noted that the modifications of "a" and "a plurality of" mentioned in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0033] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0034] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0035] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0036] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0037] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0038] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0039] Figure 1a This is a flowchart illustrating an audio detection method provided in an embodiment of the present disclosure. This embodiment is applicable to audio detection. The method can be executed by an audio detection device, which can be implemented by software and / or hardware and can be configured in a terminal and / or server to implement the audio detection method in this embodiment of the present disclosure.

[0040] It should be noted that one application scenario of this embodiment can be described as: audio detection of mixed audio signals containing multiple audio frame types. Applications requiring this can include audio effects processing, audio denoising, and speech extraction. Audio detection technology aims to detect the audio type of each audio frame in a mixed audio segment. In daily life, the mixed audio actually recorded or played is generally composed of three categories: speech, music, and noise. Detecting these three types of audio in mixed audio can effectively assist subsequent audio effects processing or encoding / decoding processing. For example, by detecting the audio type, corresponding loudness equalization can be performed to improve sound quality, or noise reduction strategies can be adjusted to better protect the music and remove noise, and the bitrate of encoding / decoding can be dynamically adjusted based on the detection results to ensure sound quality and save costs.

[0041] With the development of deep learning, numerous detection algorithms have emerged in the field of audio classification, most of which are derived from image classification and automatic speech recognition. Current audio event detection algorithms can be broadly categorized into two types: one is based on transfer learning from pre-trained models with large parameters. These pre-trained models typically use image-based pre-trained models to initialize parameters, then optimize them using audio event detection data. Examples include large-scale pre-trained audio neural networks (PANNs) for audio pattern recognition and audio spectrogram transformers (ASTs). These models usually use a large amount of weakly labeled data for training, first classifying events and then interpolating to obtain audio detection results. The other type is based on self-supervised learning (SSL) pre-trained models, such as the BEATs pre-trained model. BEATs iteratively optimize an acoustic tagger and a multi-task self-supervised learning model to obtain an optimal audio classification and training model. However, these large-parameter pre-trained models incur significant computational costs, making them unsuitable for integration into lightweight devices.

[0042] Another type is the sequence classification model based on Convolutional Recurrent Neural Networks (CRNNs), which can directly achieve audio classification and frame-level audio detection. Convolutional Neural Networks (CNNs) were initially used in the field of two-dimensional image recognition, possessing translation invariance in both dimensions of images. However, for the two-dimensional spectrum of audio data, translation invariance is not present in the frequency dimension. Furthermore, researchers have found that audio event detection is highly correlated with frequency. Based on this, Frequency Dynamic Convolution was proposed, imposing frequency component dependence on the convolution kernel of two-dimensional convolutions to enhance its recognition advantage in frequency-related patterns, thereby improving the accuracy of audio detection. To further enhance the expressive power of Frequency Dynamic Convolution, Multi-dimensional Dynamic Convolution (MDConv) was proposed, endowing the convolution kernel with adaptive dynamic characteristics along multiple dimensions of frequency, further enhancing the feature extraction capability of the convolution kernel. However, both of these methods lead to a significant increase in the number of convolution parameters and computational cost.

[0043] As described above, when performing audio detection based on existing network models, there are issues such as the network model structure being too complex to be integrated into lightweight devices, or the need to significantly increase the number of network model parameters to ensure the accuracy of audio detection, resulting in a greater computational burden.

[0044] Based on this, this embodiment provides an audio detection method that can effectively reduce the number of parameters and computational load in the audio detection process, thereby improving the processing performance of audio detection. Specifically, as shown below... Figure 1a As shown, the audio detection method provided in this embodiment may include:

[0045] S101. Convert the original mixed audio signal into an audio spectrum in the frequency domain.

[0046] In this embodiment, the original mixed audio signal can be considered as the mixed audio signal to be detected, which may include multiple source audio signals, such as speech signals, background music audio signals, and noise signals. The original mixed audio signal can be an audio signal captured over a period of time, or a recorded audio signal of a certain duration.

[0047] It is known that audio detection can be implemented by processing the original mixed audio signal in the time domain or in the frequency domain. This embodiment preferably processes the original mixed audio signal in the frequency domain, thereby converting the original mixed audio signal to the frequency domain and obtaining the corresponding audio spectrum.

[0048] In this embodiment, the original mixed audio signal can be transformed using a feature transformation to obtain a log-Mel spectrum, which is then converted into the audio spectrum in the frequency domain. For example, assuming the original mixed audio signal is represented as x(t), one conversion method is to perform a short-time Fourier transform on x(t) with N frequency points per frame, followed by a Mel transform with F frequency points to obtain the Mel spectrum. Finally, the logarithm of the Mel spectrum is taken, thus obtaining a T-frame, F-dimensional log-Mel spectrum relative to x(t). This log-Mel spectrum is equivalent to the audio spectrum of the original mixed audio signal, where T can be obtained from the duration t, and F can be obtained from the number of frequency points N. For example, the specific value of F can be half of N plus 1. This log-Mel spectrum can be represented as a T-row, F-column spectrum matrix, as shown below.

[0049] S102. The frequency-aware joint module built in the target audio detection model performs feature enhancement processing on the spectral features of the audio spectrum to obtain the frequency-aware spectral features corresponding to the audio spectrum. The frequency-aware joint module includes multiple frequency-aware network layers.

[0050] Specifically, the target audio detection model can be understood as a model used to determine the probability values ​​corresponding to each audio frame in a mixed audio signal being identified as an audio signal of a different audio type. The target audio detection model can be obtained by training an initial audio detection model based on a sample training set. The sample training set can be configured to include at least one sample tuple, which contains sample mixed audio and audio frame detection labels. The audio frame detection labels record the true audio type possessed by each audio frame in the sample mixed audio.

[0051] In this embodiment, the determined audio spectrum can be used as input information for the target audio detection model. In this step, the audio spectrum can be processed by the target audio detection model. The target audio detection model includes a frequency-aware joint module and a feature analysis joint module. The processing performed by the frequency-aware joint module may include: performing feature enhancement processing on the audio spectrum to obtain frequency-aware spectral features. The processing performed by the feature analysis joint module includes: performing feature analysis on the frequency-aware spectral features output by the frequency-aware joint module to obtain the feature analysis results for each audio frame. This step describes the processing steps of the frequency-aware joint module.

[0052] It should be noted that the network model used in this embodiment is configured to include a frequency-aware joint module and a feature analysis joint module. By improving the frequency-aware joint module, when using a target audio detection model with such a structure to participate in audio detection, compared with existing detection models, the number of parameters and computational load can be reduced while ensuring the same detection effect. Preferably, the convolutional layers involved in the original network structure have been adjusted to adaptive two-dimensional convolutions plus frequency-aware filters. By performing this structural processing on the original network model, the initial audio detection model is used as the initial audio detection model. Then, the initial audio detection model is trained based on the sample training set to obtain the target audio detection model.

[0053] In this embodiment, to detect the original mixed audio signal, it is necessary to know the characteristics of different types of audio, i.e., to perform feature analysis on the audio spectrum. Considering that directly analyzing the spectral features of the audio spectrum may lead to low accuracy, this embodiment first performs feature enhancement processing on the spectral features of the audio spectrum based on the frequency-aware joint module before performing feature analysis. Based on the above processing, enhanced features are obtained, which are considered to be the frequency-aware spectral features corresponding to the audio spectrum. After obtaining the frequency-aware spectral features, they are then analyzed.

[0054] In this embodiment, the frequency-aware joint module includes multiple frequency-aware network layers, each of which contains a frequency-aware convolutional layer and a frequency-aware filter layer. Each frequency-aware network layer adopts a streaming input-output method. Each frequency-aware network layer performs convolution processing on the input features based on the frequency-aware convolutional layer to perform basic feature extraction, and then performs filtering processing on the initially extracted features based on the frequency-aware filter layer to further refine the feature extraction.

[0055] Following the above description, the processing of each frequency-aware network layer can be described as follows: The original mixed audio signal is converted into the spectral characteristics of the audio spectrum in the frequency domain, which are then used as the initial input features. This input is fed into the first frequency-aware layer in the frequency-aware joint module. After a series of frequency-aware feature enhancement processes, the output of the first frequency-aware network layer is obtained. This process is repeated sequentially, using the output of the previous frequency-aware network layer as the input feature of the current frequency-aware network layer, and performing frequency-aware feature enhancement processes until the feature enhancement process of the last frequency-aware network layer is completed. The output of the last frequency-aware network layer is then used as the frequency-aware spectral feature after feature enhancement, for subsequent feature analysis.

[0056] S103. Through the feature analysis joint module constructed in the target audio detection model, feature analysis is performed on the frequency-aware spectral features to obtain the feature analysis results corresponding to each audio frame in the original mixed audio signal. The feature analysis results include the probability values ​​of the audio frames being audio signals of different audio types.

[0057] In this embodiment, after enhancing the spectral features using the frequency-aware joint module to obtain frequency-aware spectral features, this step performs feature analysis on the frequency-aware spectral features using the feature analysis joint module. This further determines the probability value of each audio frame being an audio signal of a different audio type, and uses this probability value as the feature analysis result for that audio frame. After performing feature analysis on the frequency-aware spectral features, it is necessary to determine the feature analysis result corresponding to each audio frame contained in the original mixed audio signal. That is, the final output of the target audio detection model is the feature analysis result output in frames. The feature analysis result characterizes the probability value of each audio frame being an audio signal of a different audio type.

[0058] For example, assuming the original mixed audio signal includes speech, background music, and noise, the target audio detection model will ultimately output frame-level probability values ​​for the speech, background music, and noise channels. The output, in frames, is denoted as the feature analysis result. This detects whether a segment of the original mixed audio signal contains speech, music, and noise; speech, background music, and noise represent three different audio types. By performing feature analysis on the frequency-aware spectral features, the probability value of each audio frame relative to these three audio types can be determined, and this result is used as the feature analysis result for each audio frame.

[0059] In this embodiment, the feature analysis joint module includes a gated loop layer, a fully connected layer, and an activation processing layer. Based on these structures, feature analysis is performed on the frequency-sensing spectral features, and the analysis results obtained based on the above analysis are recorded as feature analysis results.

[0060] For example, Figure 1b A structural example diagram of the target audio detection model involved in the audio detection method provided in this embodiment is given, such as... Figure 1b As shown, the target audio detection model includes a frequency-aware joint module and a feature analysis joint module. The frequency-aware joint module is a combination of multiple frequency-aware network layers, each of which includes a frequency-aware convolutional layer and a frequency-aware filter layer. The target audio detection model employs a series of frequency-adaptive convolutions and Fourier filters in the frequency-aware joint module to replace standard two-dimensional convolutions for feature extraction. The feature analysis joint module includes a bidirectional gated recurrent layer, a fully connected layer, and an activation processing layer. The input to the target audio detection module is the audio spectrum of the original mixed audio signal in the frequency domain, and the output is the probability value corresponding to each audio frame being identified as an audio signal of different audio types.

[0061] S104. For each audio frame in the original mixed audio signal, determine the audio type detection result of the audio frame based on the feature analysis results corresponding to the audio frame.

[0062] In this embodiment, after obtaining the feature analysis results, it is equivalent to knowing the probability value of each audio frame in the original mixed audio signal as an audio signal of different audio types. For example, by obtaining the probability value of each audio frame relative to these three audio types, it is possible to determine whether the current audio frame corresponds to speech, music, or noise. The audio type with the highest probability value can be taken as the audio type of the audio frame. Alternatively, it is preferable to smooth and binarize the probability values ​​output by the model, and finally output an audio type discrimination result for each frame in the priority order of music, speech, and noise, as the audio type detection result for that audio frame.

[0063] In this step, the feature analysis results obtained after feature transformation of the original mixed audio signal are used for inference analysis. Finally, based on the probability values ​​of different audio types, the specific audio type corresponding to this audio frame can be determined. For example, first, a threshold is set to binarize the output probability values. Then, median filtering is used to smooth the binarized classification results of each channel. Finally, the detection result of one channel is output according to the priority of whether music, speech, or noise is present. That is, each audio frame has only one classification result.

[0064] This embodiment provides an audio detection method that addresses the challenges of existing audio detection implementations. Existing audio detection methods based on network models often suffer from overly complex structures unsuitable for integration into lightweight devices, or require significantly increased network model parameters to maintain accuracy, leading to higher computational demands. This solution addresses this by employing an audio detection model comprising a frequency sensing joint module and a feature analysis joint module. The frequency sensing joint module includes multiple frequency sensing network layers. By improving this module and using an audio detection model with this structure, the number of parameters in the audio detection model is reduced. This reduces the computational load while maintaining audio processing accuracy, effectively improving computational performance and processing time while maintaining a lightweight model.

[0065] As a first optional embodiment of this example, based on the above optimization, the frequency-aware joint module constructed in the target audio detection model can be used to perform feature enhancement processing on the spectral features of the audio spectrum to obtain the frequency-aware spectral features corresponding to the audio spectrum. Specifically, the optimization is as follows:

[0066] a1) The spectral features of the audio spectrum are used as input features and input to the first frequency-aware network layer in the frequency-aware joint module for frequency-aware feature enhancement processing.

[0067] In this embodiment, the frequency-aware joint module consists of multiple frequency-aware network layers, each employing a streaming input-output approach. The spectral characteristics of the original mixed audio signal converted into an audio spectrum in the frequency domain are used as the initial input features and input to the first frequency-aware layer in the frequency-aware joint module. After a series of frequency-aware feature enhancement processes, the output of the first frequency-aware network layer is obtained.

[0068] b1) Starting from the second frequency-aware network layer in the frequency-aware joint module, the output of the previous frequency-aware network layer is used as the input feature of the current frequency-aware network layer, and frequency-aware feature enhancement processing is performed sequentially.

[0069] Specifically, the output of the first frequency-aware network layer is used as the input feature of the second frequency-aware network layer. After a series of frequency-aware enhancement processes in the second frequency-aware network layer, the output of the second frequency-aware network layer is obtained. This process is repeated, with the frequency-aware network layer that performs the enhancement process designated as the current frequency-aware network layer. The output of the previous frequency-aware network layer is used as the input feature of the current frequency-aware network layer, and frequency-aware feature enhancement processes are performed sequentially until the feature enhancement process of the last frequency-aware network layer is completed.

[0070] c1) The output of the last frequency sensing network layer is determined as the frequency sensing spectral feature after feature enhancement of the spectral feature.

[0071] In this embodiment, the output of the last frequency sensing network is used as the frequency sensing spectrum feature after feature enhancement, which is then used for subsequent feature analysis.

[0072] The first optional embodiment of the above technical solution provides that the frequency sensing joint module includes multiple frequency sensing network layers. Each frequency sensing network layer performs feature enhancement processing on the spectral features once. By performing multiple streaming feature enhancement processing on the spectral features through multiple frequency sensing network layers, the accuracy of audio detection can be guaranteed.

[0073] As a specific implementation method, frequency-aware feature enhancement processing through a frequency-aware network layer includes:

[0074] a11) Obtain the input features of the frequency-aware network layer, and perform a first frequency-aware enhancement processing on the input features through the frequency-aware convolutional layer in the frequency-aware network layer to obtain the first frequency-aware features.

[0075] It is important to understand that in this embodiment, the frequency-aware network layer includes convolutional layers and filter layers. The convolutional layers in the frequency-aware network layer are referred to as frequency-aware convolutional layers, and the filter layers are referred to as frequency-aware filter layers. In this embodiment, the input features are processed by convolution and other basic operations using the frequency-aware convolutional layers, and then the processed results are filtered using the frequency-aware filter layers to further refine and extract features, resulting in enhanced features.

[0076] For any frequency-aware network layer, the processing can be described as follows: The input features of the frequency-aware network layer are acquired; the input features are then convolved using a frequency-aware convolutional layer (this convolution is denoted as the first frequency-aware enhancement process); and the enhanced features are obtained, denoted as the first frequency-aware features. For example, performing the first frequency-aware enhancement process on the input features using a frequency-aware convolutional layer can be described as follows: The input features are convolved based on certain weights, and then the convolution result is processed according to certain weights to obtain the first frequency-aware features.

[0077] Furthermore, the step of performing a first frequency-aware enhancement processing on the input features through the frequency-aware convolutional layer in the frequency-aware network layer to obtain the first frequency-aware features can be optimized, including:

[0078] a111) The frequency-aware convolutional layer, combined with the determined input frequency attention weights, performs convolution processing on the input features to obtain the convolution processing result.

[0079] In this embodiment, basic feature extraction is first performed on the input features using a frequency-aware convolutional layer. The first frequency-aware enhancement processing of the input features by the frequency-aware convolutional layer can be understood as performing frequency-adaptive convolution on the input features. Specifically, the input features are convolved based on the determined input frequency attention weights, and the processed result is denoted as the convolution processing result.

[0080] a112) The convolution processing result is processed using the determined output frequency attention weights, and the processing result is determined as the first frequency-aware feature;

[0081] The input frequency attention weight and output frequency attention weight are obtained by performing feature compression, feature space transformation and activation processing on the input features.

[0082] In this embodiment, after obtaining the convolution processing result, the convolution processing result will be further processed based on the output frequency attention weight, and the processed result will be recorded as the first frequency-aware feature.

[0083] It's important to understand that both the input frequency attention weights and the output frequency attention weights are pre-obtained through feature compression, feature space transformation, and activation processing of the input features. For example, Figure 1c The following is a structural example diagram of the frequency-aware convolutional layer of the target audio detection model involved in the audio detection method provided in this embodiment, as shown in the figure. Figure 1c As shown, the steps for determining the input frequency attention weights can be described as follows: assuming the input features are represented as... Among them, C inThe input features are represented by the number of channels, T represents the number of audio frames, and F represents the number of frequency points. The input features M are first compressed along the time dimension to obtain the features. Next, a one-dimensional convolution is used to transform the feature m into a latent space variable. Where H represents the number of hidden layer channels; finally, two one-dimensional convolutions are used to activate the latent space variable h, and each one-dimensional convolution receives the weights, which are divided into input and output weights. and Where C out This indicates the number of output channels for the convolution.

[0084] For example, continue to refer to Figure 1c The computation process of frequency-adaptive convolution of input features through frequency-aware convolutional layers can be described as: N = Conv2d(W in M)·W out ,in, The first frequency-aware feature is represented by Conv2d(), which represents a standard two-dimensional convolution.

[0085] The above technical solution proposes to process input features by combining input frequency attention weights, output attention weights, and convolution. In order to further improve the modeling ability of frequency-related features, the input and output of the two-dimensional convolution are restricted based on frequency adaptive convolution, thereby realizing feature enhancement and providing basic data for obtaining accurate audio types in the future.

[0086] a12) The first frequency sensing feature is subjected to second frequency sensing enhancement processing through the frequency sensing filter layer in the frequency sensing network layer to obtain the second frequency sensing feature as the output result.

[0087] In this embodiment, after each first frequency-aware enhancement process (i.e., after each frequency-adaptive convolution), the result of the frequency-aware convolution is further refined and extracted using a frequency-aware filter layer. This process is referred to as the second frequency-aware enhancement process. For any frequency-aware network layer, the process can be described as follows: First frequency-aware features obtained after the first frequency-aware enhancement process are acquired in the frequency-aware network layer; the first frequency-aware features are then filtered using a frequency-aware filter layer for further feature refinement and extraction. This process is referred to as the second frequency-aware enhancement process, resulting in enhanced features, denoted as the second frequency-aware features. For example, performing the second frequency-aware enhancement process on the first frequency-aware features using a frequency-aware filter layer can be described as follows: The existing frequency-aware filter is updated based on certain weights to obtain an updated frequency-aware filter; then, the frequency domain spatial features of the first frequency-aware features are determined, and the updated frequency-aware filter is used to perform a dot product on the frequency domain spatial features, followed by an inverse transform on the dot product result to obtain the filtered features of the first frequency-aware features; finally, the filtered features and the first frequency-aware features are fused to obtain the output result.

[0088] Furthermore, the step of performing second frequency sensing enhancement processing on the first frequency sensing feature through the frequency sensing filter layer in the frequency sensing network layer to obtain the second frequency sensing feature as the output result can be optimized, including:

[0089] (a121) The frequency sensing filter layer, combined with the determined frequency filter update weights, updates the existing frequency sensing filter to obtain the updated frequency sensing filter. The frequency filter update weights are obtained by performing feature compression, feature space transformation, and excitation processing on the first frequency sensing features.

[0090] In this embodiment, the existing frequency-aware filter can be understood as a learnable filter obtained through continuous learning during the model training phase. During the actual audio detection phase, the existing frequency-aware filter in the frequency-aware filter layer is updated based on a weight, resulting in an updated frequency-aware filter. The first frequency-aware feature is then processed based on this updated frequency-aware filter. The method of updating the existing frequency-aware filter based on the updated frequency filter weights can be described as follows: the updated frequency filter weights are multiplied by the existing frequency-aware filter, and the result of this multiplication is used as the updated frequency-aware filter.

[0091] The calculation method for the frequency filter update weights is similar to that in the frequency-aware convolutional layer described above. The frequency filter update weights are pre-obtained by performing feature compression, feature space transformation, and activation processing on the first frequency-aware features. For example, Figure 1d A structural example diagram of the frequency-aware filter layer of the target audio detection model involved in the audio detection method provided in this embodiment is given, as follows: Figure 1d As shown, assuming the first frequency-sensing feature is represented as The compression-excitation mode is used to first compress N in the time and channel dimensions to obtain features. Next, a linear layer is used to transform n into a latent space variable. Where H' represents the number of hidden layer channels; finally, the hidden space variable is excited by a linear layer to obtain the frequency filter update weights.

[0092] Following the above description, the frequency filter update weights are multiplied by the learnable frequency-aware filter to obtain the updated frequency-aware filter. Let the learnable frequency-aware filter be represented as... The updated frequency sensing filter is then expressed as: F l =W f ⊙F,

[0093] a122) Through the frequency sensing filter layer, the frequency domain spatial features of the first frequency sensing feature are determined, and the frequency domain spatial features are multiplied by the updated frequency sensing filter, and the multiplication result is inversely transformed to obtain the filtered features of the first frequency sensing feature.

[0094] Continue to refer to Figure 1d In this embodiment, the first frequency-sensing feature is first processed by a normalization module through a frequency-sensing filter layer, and then subjected to a two-dimensional Fourier transform to obtain the frequency domain spatial features. For example, the calculation process can be represented as: N fd =fft2d(LN(N)), Where, N fd For frequency domain spatial features, it is a complex tensor, fft2d() represents two-dimensional fast Fourier transform, and LN() represents normalization module.

[0095] Then, the updated frequency-aware filter is applied to the frequency domain spatial features. Specifically, the updated frequency-aware filter is used to perform a dot product on the frequency domain spatial features to extract frequency-related features. The result of the dot product is then restored using a two-dimensional inverse Fourier transform. The resulting features are denoted as the filtered features of the first frequency-aware feature. That is, the updated frequency-aware filter is used to perform a dot product on the frequency domain spatial features to extract frequency-related features: N' fd =N fd ⊙F l , Then, by performing an inverse Fourier transform, we obtain the real tensor of the original domain: N' = ifft2d(N' fd ),

[0096] (a123) The filtering features and the first frequency sensing features are fused together to obtain the second frequency sensing features as the output.

[0097] Considering that retaining only the filtered features might result in the loss of some features, it is necessary to fuse the filtered features with the original first-frequency sensing features. For details, please refer to [link / reference needed]. Figure 1d The filtered features are passed through a normalization layer and a multilayer perceptron, and then added to the original first frequency perceptron features to obtain the processed features, which are used as the second frequency perceptron features of the output result.

[0098] For example, the second frequency-sensing feature can be represented as: Y = N + MLP(LN(N')), Where MLP() represents a multilayer perceptron, N represents the first frequency sensing feature, N' represents the filtering feature, and LN() represents the normalization layer.

[0099] It is important to understand that each frequency-aware network layer is processed by frequency-aware convolutional layers and frequency-aware filter layers. The final output features after processing by each frequency-aware network layer in the frequency-aware module are considered frequency-aware spectral features. In this embodiment, a frequency-adaptive convolution is used to impose restrictions on the input and output of the two-dimensional convolution, achieving audio detection results consistent with multi-dimensional dynamic frequency convolution with less computation and fewer parameters.

[0100] The above technical solution specifies the steps for the frequency-aware filter layer in the frequency-aware network layer to perform second-frequency-aware enhancement processing on the first frequency-aware feature to obtain the second frequency-aware feature. Based on the frequency-aware filter structure in this technical solution, it achieves the same effect as existing audio detection techniques with a smaller number of parameters and computational load. It can better model and extract frequency-related features with smaller parameters, thereby improving the accuracy of audio detection.

[0101] As a second optional embodiment of this example, based on the above optimization, the feature analysis joint module constructed in the target audio detection model can be used to perform feature analysis on the frequency-aware spectral features to obtain the feature analysis results corresponding to each audio frame in the original mixed audio signal. Specifically, the optimization is as follows:

[0102] a2) The frequency-aware spectral features are modeled using the gated loop layer in the feature analysis joint module.

[0103] In this embodiment, after feature enhancement processing of the spectral features by the frequency-aware joint module, the frequency-aware spectral features corresponding to the audio spectrum are obtained. After obtaining the frequency-aware spectral features, further analysis is required using the feature analysis joint module in the target audio detection model. The feature analysis joint module includes a gated loop layer, which is used to perform temporal context modeling processing on the frequency-aware spectral features to obtain temporal dimension features.

[0104] b2) Through the fully connected layer and activation processing layer in the feature analysis joint module, linear and activation processing are performed on the time dimension features after modeling to obtain the feature analysis results corresponding to each audio frame in the original mixed audio signal.

[0105] The feature analysis results for an audio frame include the probability values ​​of the audio frames being audio signals of different audio types.

[0106] In this embodiment, the feature analysis joint module further includes a fully connected layer and an activation processing layer to further perform linear processing and activation processing on the temporal dimension features, thereby obtaining the feature analysis results corresponding to each audio frame in the original mixed audio signal. Each audio frame corresponds to one feature analysis result, which specifically includes the probability values ​​of the audio frame representing different audio types. For example, assuming an original mixed audio signal contains music, sound, and noise, with probability values ​​represented as values ​​between 0 and 1, and assuming the probability value corresponding to the speech type is 0.1, the probability value corresponding to the music type is 0.2, and the probability value corresponding to the noise type is 0.9, then the probability value of the audio frame being identified as the speech type, and the probability values ​​corresponding to the music type and the noise type are used as the feature analysis results corresponding to that audio frame.

[0107] The second optional embodiment of this example specifies the steps of performing feature analysis on the frequency-aware spectral features through a feature analysis joint module built in the target audio detection model to obtain the feature analysis results corresponding to each audio frame in the original mixed audio signal. First, the frequency-aware spectral features are processed by temporal context modeling through a gated recurrent layer in the feature analysis joint module to obtain temporal dimension features. Then, the temporal dimension features are processed by linearization and activation processing through a fully connected layer and an activation processing layer to obtain the feature analysis results corresponding to each audio frame in the original mixed audio signal. This achieves feature analysis and provides a basis for subsequently obtaining the audio type detection results of the corresponding audio frames in the original mixed audio signal.

[0108] As a third optional embodiment of this example, based on the above optimizations, the audio type detection result of each audio frame in the original mixed audio signal can be determined according to the feature analysis results corresponding to the audio frame. Specifically, the optimization is as follows:

[0109] a3) For each audio frame in the original mixed audio signal, analyze the feature analysis results corresponding to the audio frame to determine the audio type detected relative to the audio frame.

[0110] In this embodiment, for each audio frame in the original mixed audio signal, the feature analysis results corresponding to the audio frame are analyzed. First, a threshold is set to binarize the output probability. Then, median filtering is used to smooth the binarized classification results of each channel to obtain the label values ​​corresponding to different audio types. Based on the label values ​​corresponding to different audio types, the audio type detected relative to the audio frame is determined.

[0111] As a specific implementation method, the step of optimizing the parsing of the feature analysis results corresponding to the audio frame and determining the audio type detected relative to the audio frame can include:

[0112] a31) Analyze the feature analysis results corresponding to the audio frame to obtain the probability values ​​corresponding to different audio signal types when the audio frame is identified in the feature analysis results.

[0113] For example, assuming the original mixed audio signal contains three audio types: speech, background music, and noise, the feature analysis results corresponding to the audio frames in the original mixed audio signal are analyzed to obtain the probability value corresponding to each audio frame being identified as speech, background music, or noise.

[0114] a32) Binarize the probability values ​​corresponding to the different audio types to obtain the label values ​​corresponding to the different audio types.

[0115] In this embodiment, a pre-set threshold is used to binarize the probability values ​​corresponding to different audio types. Then, median filtering is used to smooth the binarized classification results of each channel, obtaining label values ​​corresponding to different audio types. Continuing the example above, assuming the probability value is between 0 and 1, and the threshold is set to 0.5, audio types with probability values ​​less than 0.5 are binarized to 0, and audio types with probability values ​​greater than or equal to 0.5 are binarized to 1. Assuming the probability of an audio frame being identified as speech is 0.7, as music is 0.8, and as noise is 0.1, after binarizing the probability values ​​corresponding to different audio types, the binarized value for speech is 1, for music is 1, and for noise is 0. After binarizing the probability values ​​corresponding to the audio types, median filtering can be used to smooth them. The smoothed value is used as the label value corresponding to the audio type, which is equivalent to smoothing the audio frame by referring to the label values ​​of the preceding and following frames.

[0116] a33) The audio type with a tag value set is determined as the audio type detected relative to the audio frame.

[0117] In this embodiment, the set value can be set to 1, which determines the audio type with a tag value of 1 as the audio type detected relative to the audio frame. It is understood that for each audio frame, its relative audio type will be determined. The audio type detected relative to the audio frame may contain only one audio type, or it may contain two or more audio types.

[0118] b3) If an audio type exists relative to the audio frame, then the audio type is used as the audio type detection result of the audio frame.

[0119] Specifically, if an audio type exists for a relative audio frame, that audio type is used as the audio type detection result for the audio frame. For example, assuming the audio type of the relative audio frame is noise, the noise type is used as the audio detection result for that audio frame.

[0120] c3) If there are at least two audio types relative to the audio frame, the audio type with the highest priority shall be used as the audio type detection result of the audio frame.

[0121] Specifically, if a relative audio frame contains two or more audio types, the audio type with the highest priority is output as the audio type detection result for that audio frame. For example, assuming a relative audio frame contains background music, speech, and noise, with speech having a higher priority than background music, which in turn has a higher priority than noise, then the detection result for one channel is output based on the presence or absence of background music, speech, and noise. That is, the speech type is used as the audio type detection result for each audio frame. It is important to understand that each audio frame has only one classification result, which serves as the audio type detection result for that audio frame.

[0122] This third optional embodiment specifies that the above technical solution determines the audio type detection result of the audio frame based on the feature analysis results corresponding to the audio frame. For each audio frame in the original mixed audio signal, the audio type detected relative to the audio frame is determined by parsing the feature analysis results, and then the detection result of the audio frame is determined based on the audio type present in the audio frame. This achieves accurate detection of audio frames.

[0123] It is understandable that determining the audio type detection result for each audio frame in the original mixed audio signal can effectively assist subsequent business processing. As a fourth optional embodiment of this example, based on the above optimizations, the method can be further optimized to include:

[0124] a4) For audio frames whose audio type detection result is speech type, perform the first service processing corresponding to the speech type to obtain the first service audio signal corresponding to the original mixed audio signal.

[0125] In this embodiment, the detection result of the audio frame may be speech, noise, or music. For each audio type, a corresponding service processing is set. The service processing corresponding to the speech type is designated as the first service processing, the service processing corresponding to the noise type as the second service processing, and the service processing corresponding to the music type as the third service processing. There are no specific restrictions on which service processing is included for each type; they can be preset according to actual needs. For example, the service processing may include speech enhancement processing, adjusting the codec bitrate, response equalization processing to improve sound quality, and adjusting noise strategies to better protect the music while removing noise.

[0126] In this embodiment, if an audio frame is detected as speech, a first service processing is performed on the audio frame to obtain a service audio signal corresponding to the original mixed audio signal, denoted as the first service audio signal. For example, the first service processing may include speech enhancement processing, adjusting the codec bitrate, etc.

[0127] b4) For audio frames whose audio type detection result is noise type, perform the second service processing corresponding to the noise type to obtain the second service audio signal corresponding to the original mixed audio signal.

[0128] In this embodiment, if an audio frame is detected as noise, a second service processing is performed on the audio frame to obtain the service audio signal corresponding to the original mixed audio signal, which is denoted as the second service audio signal. For example, the second service processing may be noise removal processing, etc.

[0129] c4) For audio frames whose audio type detection result is music type, perform the third service processing corresponding to the music type to obtain the third service audio signal corresponding to the original mixed audio signal.

[0130] In this embodiment, if an audio frame is detected as music, a third service processing is performed on the corresponding audio frame to obtain the service audio signal corresponding to the original mixed audio signal, denoted as the third service audio signal. For example, the third service processing can improve sound quality in response to equalization processing.

[0131] The above technical solution adds audio type detection results based on audio frames, which can effectively assist subsequent business processing.

[0132] As a fifth optional embodiment of this example, based on the above optimizations, the training steps of the target audio detection model can be specifically optimized as follows:

[0133] a4) Obtain the sample training set and the two identical initial audio detection models that have been constructed, and denote the two initial audio detection models as the student model and the teacher model, respectively. The sample training set includes at least one sample tuple, which contains sample mixed audio and audio frame detection labels. The audio frame detection labels record the real audio type of each audio frame in the sample mixed audio.

[0134] In this embodiment, to obtain the target audio detection model, an initial model needs to be pre-constructed, denoted as the initial audio detection model, and a sample training set needs to be obtained. The initial audio detection model is trained based on the sample training set, and the trained model is used as the target audio detection model for audio detection in the actual detection stage. It should be noted that the processing of mixed audio samples in the sample training set based on the initial audio detection model during the training stage is the same as the processing of the original mixed audio signal based on the target audio detection model during the actual detection stage, and will not be repeated here. First, the construction of the sample training set is described. For example, clear speech, noise, and music are mixed to construct the training data required for audio classification, and binary labels for the three types of audio are generated in advance as training targets. An initial audio detection model based on deep learning is constructed. The mixed temporal audio undergoes feature transformation to obtain a log-Mel spectrum, which is then used as the input to the initial audio classification model. The output of the model is the frame-level probability of the presence or absence of the three types of audio.

[0135] Specifically, let the clear speech signal be s(t), the noise signal be n(t), and the music signal be m(t), then the constructed mixed audio signal can be expressed as: x(t) = g s *s(t)+g n *n(t)+g m *m(t), where g s g n g m Let x(t) represent the gains of the speech signal, noise signal, and music signal, respectively, with values ​​ranging from [0,1]. A short-time Fourier transform is performed on x(t), followed by an F-point Mel transform. Finally, the logarithm of the Mel spectrum is taken to obtain the F-dimensional logarithmic Mel spectrum of T frames. These are the features of the input model. Based on the start and end points of each signal class when generating the mixed signal, corresponding frame-level binary labels are generated. As the training target of the model, in this embodiment, a sample training set is constructed based on the generated mixed audio and its corresponding labels. The generated mixed audio is used as the sample mixed audio, and the real audio type of each audio frame in the mixed audio is used as the audio frame detection label. Furthermore, the sample mixed audio and the corresponding audio frame detection label are used as a sample tuple. Multiple sample tuples are used to construct a sample training set for training the initial audio detection model.

[0136] In this embodiment, a deep learning-based audio classification model is constructed as the initial audio detection model. This embodiment uses a classic convolutional recurrent network as a framework and the complex spectrum of mixed audio as the basis. As input, the initial audio detection network, exemplarily, primarily comprises a series of joint modules of frequency-adaptive convolutions and Fourier filters, used to replace standard two-dimensional convolutions for feature extraction; secondly, it includes two layers of bidirectional gated recurrent modules to model the contextual information of the sequence; and finally, it includes a classification module that uses a linear layer to map high-dimensional features to low-dimensional classification logic, and then uses an activation function to convert the logical values ​​into probability values. In this embodiment, a semi-supervised learning strategy (mean-teacher) is adopted in the training of the initial audio detection model, specifically, two identical models are used: a student model and a teacher model.

[0137] b4) Input the sample mixed audio from the sample binary into the student model to obtain the feature analysis prediction results corresponding to each audio frame in the sample mixed audio.

[0138] Specifically, the sample mixed audio from the sample binary is input into the student model. The model outputs the feature analysis results for each audio frame in the sample mixed audio, that is, the predicted probability value corresponding to each audio frame being determined as an audio signal of different audio types. The predicted probability values ​​output above are recorded as the feature analysis prediction results. That is, this step will obtain the feature analysis prediction results for each audio frame in the sample mixed audio.

[0139] c4) The loss function value is determined by using a loss function based on the cross-entropy of the two academies, combined with the audio frame detection labels and the feature analysis prediction results.

[0140] In this embodiment, when calculating the loss function, it is necessary to calculate the binary cross-entropy loss between the student model's output and the audio frame detection labels, as well as the consistency loss between the student model and the teacher model. The overall loss function for calculating the loss function value can be expressed as:

[0141]

[0142] L = L BSE +w con *L CON ,

[0143] Among them, L BSE L represents the binary cross-entropy loss between the student model's output and the audio frame detection labels. CON w represents the consistency loss between the student model's output and the teacher model's output. con The weight representing the control over consistency loss, in this embodiment w con The value is controlled by a time scheduler. This represents the predicted output of the student model. This represents the predicted output of the teacher model.

[0144] d4) Update the network parameters of the student model according to the loss function value, and update the teacher model with the updated network parameters. Return to re-determine the feature analysis and prediction results until the training end condition is met. After training, determine the corresponding teacher model as the target audio detection model.

[0145] It's important to understand that during training, only the student model performs gradient backpropagation; the teacher model's weights are updated using an Exponential Moving Average (EMA) strategy, which can be expressed as follows: in, and α and β represent the model weights of the teacher model and student model at step t, respectively, and α represents the network parameters during EMA update.

[0146] In this embodiment, the network parameters of the student model are updated based on the loss function value, and the teacher model is updated using the updated network parameters. The process then returns to update the student model again, and the feature analysis prediction results are determined again. This process is repeated multiple times until the training termination condition is met. This embodiment does not impose specific restrictions on the training termination condition; for example, it can be set as the loss function value of the model's output feature analysis prediction result and the audio frame detection label being within a set loss threshold range.

[0147] The fifth optional embodiment of the present technical solution specifies the training steps of the target audio detection model. Based on the improved target audio detection model, the number of model parameters and the amount of audio detection computation are reduced while ensuring the accuracy of audio processing, effectively improving the computational performance and processing time of audio detection.

[0148] To more clearly illustrate the effects achieved by this technical solution, relevant verification was conducted using test data. The test data consisted of manually labeled online mixed data, including 7.3 hours of music, 2.7 hours of speech, and 4.1 hours of noise. The accuracy of each audio frame in the feature analysis prediction results was used as the evaluation metric. All mixed audio was resampled to 16kHz, and a short-time Fourier transform was performed on the mixed audio with a frame length of 2048 points and a frame shift of 256 sampling points. After performing a 2048-point FFT on the framed signal, a Mel transform and logarithmic transformation were performed to obtain the log-Mel spectrum of 128 frequency bands. The number of channels in the convolution was [32, 64, 128, 256, 256, 256, 256], and the number of hidden units in the bidirectional gated recurrent unit was 256.

[0149] Table 1

[0150]

[0151] In the testing experiments, the existing technical solution used the original multidimensional frequency dynamic convolution MDDF-CRNN. ​​Table 1 shows a parameter example table comparing the performance of the existing and proposed technical solutions. As shown in Table 1, the existing technical solution achieved a speech detection accuracy of 0.905 on the test data, while the proposed technical solution achieved a speech detection accuracy of 0.912. It is evident that the proposed technical solution has a higher speech detection accuracy than the existing technical solution. Similarly, for music and noise detection, the proposed technical solution also demonstrates higher speech detection accuracy than the existing technical solution. Furthermore, the statistics on the number of parameters and computational load also show that the proposed technical solution has significantly fewer parameters and less computational load than the existing technical solution.

[0152] Tests show that, compared to the original multidimensional frequency dynamic convolution MDDF-CRNN, the frequency adaptive convolution and Fourier filter structure proposed in this technical solution achieves the same effect with fewer parameters and less computation. This indicates that the proposed technical solution can better model and extract frequency-related features with fewer parameters, thereby increasing the accuracy of audio detection.

[0153] Figure 2 This is a schematic diagram of an audio detection device provided in an embodiment of the present disclosure. This embodiment is applicable to audio detection. The device can be implemented by software and / or hardware and can be configured in a terminal and / or server to implement the audio detection method in this embodiment of the present disclosure. Specifically, the device may include: an audio conversion module 21, an enhancement processing module 22, an audio analysis module 23, and a result detection module 24.

[0154] The audio conversion module 21 is used to convert the original mixed audio signal into an audio spectrum in the frequency domain.

[0155] The enhancement processing module 22 is used to perform feature enhancement processing on the spectral features of the audio spectrum through the frequency sensing joint module built in the target audio detection model to obtain the frequency sensing spectral features corresponding to the audio spectrum. The frequency sensing joint module includes multiple frequency sensing network layers.

[0156] The audio analysis module 23 is used to perform feature analysis on the frequency-aware spectral features through the feature analysis joint module built in the target audio detection model, and obtain the feature analysis results corresponding to each audio frame in the original mixed audio signal. The feature analysis results include the probability values ​​of the audio frames being audio signals of different audio types.

[0157] The result detection module 24 is used to determine the audio type detection result of each audio frame in the original mixed audio signal based on the feature analysis results corresponding to the audio frame.

[0158] This embodiment provides an audio detection device. In existing audio detection implementations, when audio detection is based on a network model, problems arise such as the network model structure being too complex to be integrated into lightweight devices, or the need to significantly increase the number of network model parameters to ensure audio detection accuracy, resulting in increased computational load. The present technical solution uses an audio detection model that includes a frequency sensing joint module and a feature analysis joint module. The frequency sensing joint module includes multiple frequency sensing network layers. By improving the frequency sensing joint module, the number of parameters in the audio detection model is reduced when using such a structure. This reduces the computational load of audio detection while maintaining audio processing accuracy, effectively improving the computational performance and processing time of audio detection while maintaining a lightweight model.

[0159] Furthermore, the enhanced processing module 22 may include:

[0160] The first enhancement processing unit is used to take the spectral features of the audio spectrum as input features and input them into the first frequency-aware network layer in the frequency-aware joint module to perform frequency-aware feature enhancement processing.

[0161] The second enhancement processing unit is used to start from the second frequency sensing network layer in the frequency sensing joint module, take the output result of the previous frequency sensing network layer as the input feature of the current frequency sensing network layer, and perform frequency sensing-based feature enhancement processing in sequence.

[0162] An effective feature determination unit is used to determine the output of the last frequency sensing network layer as the frequency sensing spectral feature after feature enhancement of the spectral feature.

[0163] Based on the above optimizations, the first enhancement processing unit or the second enhancement processing unit is used to perform the step of frequency-aware feature enhancement processing through the frequency-aware network layer, including:

[0164] The input features of the frequency-aware network layer are obtained, and the input features are subjected to a first frequency-aware enhancement processing through the frequency-aware convolutional layer in the frequency-aware network layer to obtain the first frequency-aware features.

[0165] The first frequency-sensing feature is subjected to second frequency-sensing enhancement processing through the frequency-sensing filter layer in the frequency-sensing network layer to obtain the second frequency-sensing feature as the output result.

[0166] Further, the step of performing a first frequency-aware enhancement processing on the input features through a frequency-aware convolutional layer in the frequency-aware network layer to obtain first frequency-aware features includes:

[0167] The frequency-aware convolutional layer, combined with the determined input frequency attention weights, performs convolution processing on the input features to obtain the convolution processing result.

[0168] The convolution processing result is processed using the determined output frequency attention weights, and the processing result is determined as the first frequency-aware feature;

[0169] The input frequency attention weight and output frequency attention weight are obtained by performing feature compression, feature space transformation and activation processing on the input features.

[0170] Further, the step of performing second frequency-sensing enhancement processing on the first frequency-sensing feature through the frequency-sensing filter layer in the frequency-sensing network layer to obtain the second frequency-sensing feature as the output includes:

[0171] By using the frequency sensing filter layer and combining it with the determined frequency filter update weights, the existing frequency sensing filter is updated to obtain the updated frequency sensing filter.

[0172] The frequency domain spatial features of the first frequency-sensing feature are determined through the frequency-sensing filter layer, and the frequency domain spatial features are multiplied by the updated frequency-sensing filter. The result of the multiplication is then inversely transformed to obtain the filtered features of the first frequency-sensing feature.

[0173] By fusing the filtering features and the first frequency sensing features, a second frequency sensing feature is obtained as the output result.

[0174] The frequency filter update weights are obtained by performing feature compression, feature space transformation, and excitation processing on the first frequency sensing features.

[0175] Furthermore, the audio analysis module 23 is specifically used for:

[0176] The frequency-aware spectral features are modeled using the gated loop layer in the feature analysis joint module.

[0177] The fully connected layer and activation processing layer in the feature analysis joint module are used to perform linear and activation processing on the time dimension features after modeling, so as to obtain the feature analysis results corresponding to each audio frame in the original mixed audio signal.

[0178] Furthermore, the result detection module 24 may include:

[0179] The result parsing unit is used to analyze the feature analysis results corresponding to each audio frame in the original mixed audio signal and determine the audio type detected relative to the audio frame.

[0180] The first result determination unit is used to determine the audio type as the audio type detection result of the audio frame if an audio type exists relative to the audio frame.

[0181] The second result determination unit is used to determine the audio type detection result of the audio frame if there are at least two audio types relative to the audio frame, by taking the audio type with the highest priority.

[0182] Based on the above optimizations, the result parsing unit is specifically used for:

[0183] Analyze the feature analysis results corresponding to the audio frame to obtain the probability values ​​corresponding to different audio types of audio signals when the audio frame is identified in the feature analysis results;

[0184] Binarize the probability values ​​corresponding to the different audio types to obtain the label values ​​corresponding to the different audio types;

[0185] The audio type with a set tag value is determined as the audio type detected relative to the audio frame.

[0186] Furthermore, the device also includes a service processing module for:

[0187] For audio frames whose audio type detection result is speech, perform the first service processing corresponding to the speech type to obtain the first service audio signal corresponding to the original mixed audio signal;

[0188] For audio frames whose audio type detection result is noise type, perform the second service processing corresponding to the noise type to obtain the second service audio signal corresponding to the original mixed audio signal;

[0189] For audio frames whose audio type detection result is music, perform the third service processing corresponding to the music type to obtain the third service audio signal corresponding to the original mixed audio signal.

[0190] Furthermore, the device also includes a model training module, specifically used for:

[0191] Obtain a sample training set and two identical initial audio detection models that have been constructed, and denote the two initial audio detection models as the student model and the teacher model, respectively. The sample training set includes at least one sample tuple, which contains sample mixed audio and audio frame detection labels. The audio frame detection labels record the real audio type of each audio frame in the sample mixed audio.

[0192] The sample mixed audio in the sample binary is input into the student model to obtain the feature analysis and prediction results corresponding to each audio frame in the sample mixed audio;

[0193] The loss function value is determined by using a loss function based on the cross-entropy of the two academies, combined with the audio frame detection labels and the feature analysis prediction results;

[0194] The network parameters of the student model are updated based on the loss function value, and the teacher model is updated using the updated network parameters. The feature analysis and prediction results are then re-determined until the training termination condition is met. The teacher model corresponding to the training termination is then determined as the target audio detection model.

[0195] The above-described apparatus can execute the methods provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the methods.

[0196] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0197] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure. Reference is made below. Figure 3 It illustrates a computer device suitable for implementing embodiments of the present disclosure (e.g., Figure 3 The diagram below shows the structure of the terminal device or server 30. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 3 The computer device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0198] like Figure 3As shown, the computer device 30 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 31, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 32 or a program loaded from a storage device 38 into a random access memory (RAM) 33. The RAM 33 also stores various programs and data required for the operation of the computer device 30. The processing unit 31, the ROM 32, and the RAM 33 are interconnected via a bus 35. An edit / output (I / O) interface 34 is also connected to the bus 35.

[0199] Typically, the following devices can be connected to I / O interface 34: input devices 36 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 37 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 38 including, for example, magnetic tapes, hard disks, etc.; and communication devices 39. Communication device 39 allows computer device 30 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 A computer device 30 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0200] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 39, or installed from a storage device 38, or installed from a ROM 32. When the computer program is executed by the processing device 31, it performs the functions defined in the methods of embodiments of this disclosure.

[0201] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0202] The computer device provided in this embodiment and the audio detection method provided in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0203] This disclosure provides a computer storage medium storing a computer program that, when executed by a processor, implements the audio detection method provided in the above embodiments.

[0204] It should be noted that the computer-readable medium described above in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0205] In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0206] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0207] The aforementioned computer-readable medium may be included in the aforementioned computer device; or it may exist independently and not assembled into the computer device.

[0208] The aforementioned computer-readable medium carries one or more programs that, when executed by the computer device, cause the computer device to:

[0209] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0210] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0211] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0212] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0213] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0214] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0215] Furthermore, although the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while some specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0216] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An audio detection method, characterized in that, include: Convert the original mixed audio signal into an audio spectrum in the frequency domain; The frequency-aware joint module built in the target audio detection model performs feature enhancement processing on the spectral features of the audio spectrum to obtain the frequency-aware spectral features corresponding to the audio spectrum. The frequency-aware joint module includes multiple frequency-aware network layers. The feature analysis joint module built in the target audio detection model performs feature analysis on the frequency-aware spectral features to obtain the feature analysis results corresponding to each audio frame in the original mixed audio signal. The feature analysis results include the probability values ​​of the audio frames being audio signals of different audio types. For each audio frame in the original mixed audio signal, the audio type detection result of the audio frame is determined based on the feature analysis results corresponding to the audio frame.

2. The method according to claim 1, characterized in that, The frequency-aware joint module, constructed within the target audio detection model, performs feature enhancement processing on the spectral features of the audio spectrum to obtain the frequency-aware spectral features corresponding to the audio spectrum, including: The spectral features of the audio spectrum are used as input features and input to the first frequency-aware network layer in the frequency-aware joint module for frequency-aware feature enhancement processing. Starting from the second frequency-aware network layer in the frequency-aware joint module, the output of the previous frequency-aware network layer is used as the input feature of the current frequency-aware network layer, and frequency-aware feature enhancement processing is performed sequentially. The output of the last frequency sensing network layer is determined as the frequency sensing spectral feature after feature enhancement of the spectral feature.

3. The method according to claim 2, characterized in that, Frequency-aware feature enhancement processing via frequency-aware network layers includes: The input features of the frequency-aware network layer are obtained, and the input features are subjected to a first frequency-aware enhancement processing through the frequency-aware convolutional layer in the frequency-aware network layer to obtain the first frequency-aware features. The first frequency-sensing feature is subjected to second frequency-sensing enhancement processing through the frequency-sensing filter layer in the frequency-sensing network layer to obtain the second frequency-sensing feature as the output result.

4. The method according to claim 3, characterized in that, The step of performing a first frequency-aware enhancement processing on the input features through a frequency-aware convolutional layer in the frequency-aware network layer to obtain first frequency-aware features includes: The frequency-aware convolutional layer, combined with the determined input frequency attention weights, performs convolution processing on the input features to obtain the convolution processing result. The convolution processing result is processed using the determined output frequency attention weights, and the processing result is determined as the first frequency-aware feature; The input frequency attention weight and output frequency attention weight are obtained by performing feature compression, feature space transformation and activation processing on the input features.

5. The method according to claim 3, characterized in that, The step of performing second frequency sensing enhancement processing on the first frequency sensing feature through the frequency sensing filter layer in the frequency sensing network layer to obtain the second frequency sensing feature as the output result includes: By using the frequency sensing filter layer and combining it with the determined frequency filter update weights, the existing frequency sensing filter is updated to obtain the updated frequency sensing filter. The frequency domain spatial features of the first frequency-sensing feature are determined through the frequency-sensing filter layer, and the frequency domain spatial features are multiplied by the updated frequency-sensing filter. The result of the multiplication is then inversely transformed to obtain the filtered features of the first frequency-sensing feature. By fusing the filtering features and the first frequency sensing features, a second frequency sensing feature is obtained as the output result. The frequency filter update weights are obtained by performing feature compression, feature space transformation, and excitation processing on the first frequency sensing features.

6. The method according to claim 1, characterized in that, The feature analysis joint module built into the target audio detection model performs feature analysis on the frequency-aware spectral features to obtain the feature analysis results corresponding to each audio frame in the original mixed audio signal, including: The frequency-aware spectral features are modeled using the gated loop layer in the feature analysis joint module. By using the fully connected layer and activation processing layer in the feature analysis joint module, linear and activation processing is performed on the time dimension features after modeling, thereby obtaining the feature analysis results corresponding to each audio frame in the original mixed audio signal.

7. The method according to claim 1, characterized in that, The step of determining the audio type detection result of each audio frame in the original mixed audio signal based on the feature analysis results corresponding to the audio frame includes: For each audio frame in the original mixed audio signal, the feature analysis results corresponding to the audio frame are analyzed to determine the audio type detected relative to the audio frame; If an audio type exists relative to the audio frame, then the audio type is used as the audio type detection result of the audio frame; If there are at least two audio types relative to the audio frame, the audio type with the highest priority is used as the audio type detection result of the audio frame.

8. The method according to claim 7, characterized in that, The step of parsing the feature analysis results corresponding to the audio frame to determine the audio type detected relative to the audio frame includes: Analyze the feature analysis results corresponding to the audio frame to obtain the probability values ​​corresponding to different audio types of audio signals when the audio frame is identified in the feature analysis results; Binarize the probability values ​​corresponding to the different audio types to obtain the label values ​​corresponding to the different audio types; The audio type with a set tag value is determined as the audio type detected relative to the audio frame.

9. The method according to any one of claims 1-8, characterized in that, Also includes: For audio frames whose audio type detection result is speech, perform the first service processing corresponding to the speech type to obtain the first service audio signal corresponding to the original mixed audio signal; For audio frames whose audio type detection result is noise type, perform the second service processing corresponding to the noise type to obtain the second service audio signal corresponding to the original mixed audio signal; For audio frames whose audio type detection result is music, perform the third service processing corresponding to the music type to obtain the third service audio signal corresponding to the original mixed audio signal.

10. The method according to any one of claims 1-8, characterized in that, The training steps of the target audio detection model include: Obtain a sample training set and two identical initial audio detection models that have been constructed, and denote the two initial audio detection models as the student model and the teacher model, respectively. The sample training set includes at least one sample tuple, which contains sample mixed audio and audio frame detection labels. The audio frame detection labels record the real audio type of each audio frame in the sample mixed audio. The sample mixed audio in the sample binary is input into the student model to obtain the feature analysis and prediction results corresponding to each audio frame in the sample mixed audio; The loss function value is determined by using a loss function based on the cross-entropy of the two academies, combined with the audio frame detection labels and the feature analysis prediction results; The network parameters of the student model are updated based on the loss function value, and the teacher model is updated using the updated network parameters. The feature analysis and prediction results are then re-determined until the training termination condition is met. The teacher model corresponding to the training termination is then determined as the target audio detection model.

11. An audio detection device, characterized in that, include: The audio conversion module is used to convert the original mixed audio signal into an audio spectrum in the frequency domain; The enhancement processing module is used to perform feature enhancement processing on the spectral features of the audio spectrum through the frequency sensing joint module built in the target audio detection model to obtain the frequency sensing spectral features corresponding to the audio spectrum. The frequency sensing joint module includes multiple frequency sensing network layers. The audio analysis module is used to perform feature analysis on the frequency-aware spectral features through the feature analysis joint module built in the target audio detection model, and obtain the feature analysis results corresponding to each audio frame in the original mixed audio signal. The feature analysis results include the probability values ​​of the audio frames being audio signals of different audio types. The result detection module is used to determine the audio type detection result of each audio frame in the original mixed audio signal based on the feature analysis results corresponding to the audio frame.

12. A computer device, characterized in that, The computer device includes: One or more processors; a storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the audio detection method as described in any one of claims 1-10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the audio detection method as described in any one of claims 1-10.

14. A computer program product comprising a computer program that, when executed by a processor, implements the audio detection method according to any one of claims 1-10.

Images