Audio signal processing method, electronic device, storage medium and program product

Abstract

The embodiment of the invention provides an audio signal processing method, electronic equipment, a storage medium and a program product, and the method comprises the steps: carrying out the time-frequency conversion of a to-be-processed audio segment, and extracting a key frequency band feature, a time-frequency fusion feature and an audio semantic feature; comprehensive representation of the audio signal in three dimensions of local frequency domain details, time-frequency dynamic change and high-level semantic content is realized; the first fusion feature with complementary information and higher representation capability is generated by fusing the three features, and quality analysis is performed based on the first fusion feature, so that the accuracy and reliability of audio quality evaluation are remarkably improved, and the audio quality evaluation is closer to human subjective auditory perception.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an audio signal processing method, electronic device, storage medium, and program product. Background Technology

[0002] Audio Quality Assessment (AQA) algorithms are divided into subjective assessment and objective assessment. Subjective assessment involves a person subjectively comparing and scoring the original audio signal and the audio signal to be evaluated. Objective assessment aims to predict the auditory quality of an audio signal using objective algorithms, simulating the subjective scoring results of a person. Objective assessment is further divided into referenced assessment and non-referenced assessment.

[0003] No-reference evaluation is an algorithm that can predict the auditory quality of an audio signal to be evaluated without using the original audio signal as a reference. With the development of artificial intelligence technology, no-reference audio quality evaluation models based on deep learning have emerged. These models use convolutional neural networks (CNNs) to extract acoustic features of the audio signal to be evaluated and then judge the quality of the audio signal based on the extracted acoustic features.

[0004] However, the prediction results of the above evaluation model differ significantly from the subjective evaluation results, and the accuracy of the quality prediction results is not high. Summary of the Invention

[0005] This application provides an audio signal processing method, electronic device, storage medium, and program product for accurately evaluating audio quality.

[0006] This application provides an audio signal processing method, including: performing time-frequency conversion on an audio segment to be processed to obtain at least one spectrogram; using an audio processing network to extract features from the at least one spectrogram to obtain key frequency band features, time-frequency fusion features, and audio semantic features corresponding to the audio segment; fusing the key frequency band features, time-frequency fusion features, and audio semantic features corresponding to the audio segment to obtain a first fused feature; and evaluating the quality of the audio segment based on the first fused feature.

[0007] This application also provides an audio signal processing method, including: sampling an audio signal in a live or short video scene to obtain an audio segment to be processed; using an audio processing network to extract features from at least one spectrogram to obtain key frequency band features, time-frequency fusion features, and audio semantic features corresponding to the audio segment; fusing the key frequency band features, time-frequency fusion features, and audio semantic features corresponding to the audio segment to obtain a first fused feature; evaluating the quality of the audio segment based on the first fused feature; and if it is determined based on the quality information that the audio segment does not meet the sound quality requirements of the live or short video scene, enhancing or repairing the sound quality of the audio segment to play the enhanced or repaired audio segment.

[0008] This application also provides an electronic device, which includes: a memory and a processor; the memory for storing a computer program; and the processor coupled to the memory for executing the computer program to perform steps in an audio signal processing method.

[0009] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the audio signal processing method.

[0010] This application also provides a computer program product, including a computer program / instructions, which, when executed by a processor, enable the processor to implement the steps in the audio signal processing method.

[0011] The audio signal processing method provided in this application, by performing time-frequency conversion on the audio segment to be processed and extracting key frequency band features, time-frequency fusion features, and audio semantic features, achieves a comprehensive representation of the audio signal in three dimensions: local frequency domain details, time-frequency dynamic changes, and high-level semantic content. By fusing these three types of features to generate a first fusion feature with complementary information and stronger representational capabilities, and performing quality analysis based on this feature, it can more accurately and robustly identify quality defects such as distortion and noise in the audio, effectively avoiding misjudgments caused by content changes, thereby significantly improving the accuracy and reliability of audio quality assessment and making it closer to human subjective auditory perception. Attached Figure Description

[0012] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0013] Figure 1 A flowchart illustrating an audio signal processing method provided in this application embodiment;

[0014] Figure 2This is a schematic diagram of an exemplary audio processing network.

[0015] Figure 3 This is a schematic diagram of an exemplary frequency domain convolution model;

[0016] Figure 4 This is a schematic diagram of an exemplary time-frequency fusion model;

[0017] Figure 5 This is a schematic diagram of the structure of an exemplary semantic understanding model;

[0018] Figure 6 This is a schematic diagram illustrating the working principle of an exemplary multi-head self-attention mechanism;

[0019] Figure 7 This is a schematic diagram illustrating the working principle of an exemplary single-head self-attention mechanism.

[0020] Figure 8 A flowchart illustrating an audio signal processing method provided in this application embodiment;

[0021] Figure 9 This is a schematic diagram of the structure of an audio signal processing device provided in an embodiment of this application;

[0022] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] In the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the access relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone, where A and B can be singular or plural. In the textual description of this application, the character " / " generally indicates that the preceding and following associated objects have an "or" relationship. Furthermore, in the embodiments of this application, "first," "second," "third," etc., are only used to distinguish the content of different objects and have no other special meaning.

[0025] It should be noted that, in the cases involving user information in the embodiments of this application, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse. In addition, the various models involved in this application (including but not limited to language models or large models) comply with relevant laws and standards.

[0026] The technical solutions of this application and how they solve the aforementioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The technical solutions provided by each embodiment of this application are described in detail below with reference to the accompanying drawings.

[0027] Figure 1 A flowchart illustrating an audio signal processing method provided in an embodiment of this application. See also... Figure 1 The method may include the following steps:

[0028] 101. Perform time-frequency conversion on the audio segment to be processed to obtain at least one spectrogram.

[0029] 102. Use an audio processing network to extract features from at least one spectrogram to obtain key frequency band features, time-frequency fusion features, and audio semantic features corresponding to the audio segment.

[0030] 103. The key frequency band features, time-frequency fusion features and audio semantic features corresponding to the audio segment are fused to obtain the first fused feature.

[0031] 104. Based on the first fusion feature, the quality of the audio segment is evaluated and processed.

[0032] The audio signal processing method provided in this application is a referenceless quality assessment algorithm that can analyze the quality information of the audio segment to be processed without the need for reference audio. It can be applied to audio quality assessment in various scenarios, including but not limited to: live streaming scenarios, short video scenarios, voice call scenarios, etc., and is especially suitable for scenarios where it is not easy to obtain reference audio or where reference audio cannot be obtained.

[0033] For audio segments requiring audio quality analysis, referred to as the audio segment to be processed, the first step is to perform a time-frequency transformation to obtain at least one spectrogram. Time-frequency transformation converts a one-dimensional time-domain signal into a two-dimensional time-frequency domain representation, revealing the signal's characteristics in both time and frequency dimensions. Types of spectrograms include, but are not limited to, Mel spectrograms, linear amplitude spectrograms, and linear-frequency spectrograms. For ease of understanding and differentiation, the Mel spectrogram will be used as an example below. In a Mel spectrogram, the horizontal axis represents time, and the vertical axis represents Mel frequency. The color or brightness of each point in the graph represents the energy level of the signal at that time and frequency. The Mel spectrogram clearly displays the energy distribution of the audio signal at different times and frequencies. It is worth noting that one channel of a Mel spectrogram corresponds to one Mel band. When generating a Mel spectrogram, the linear frequency axis is mapped to a set of non-linear frequency bands using a Mel scale. For example, setting "48 channels" means dividing the entire frequency range into 48 frequency bands.

[0034] In practical applications, the audio segment can undergo at least one time-frequency transformation to obtain at least one spectrogram. The scale information of each spectrogram can be the same or different; under the premise of using an audio processing network for audio quality evaluation, the scale information of each spectrogram is the same, and the scale information of the spectrogram input to all branches of the audio processing network is the same, which can greatly save inference speed. This means that all branches of the audio processing network can use the same spectrogram.

[0035] It is worth noting that multiple time-frequency transformations result in multiple spectrograms of different sizes. For example, different spectrograms may have different time spans, different numbers of channels, or different sampling rates. These multiple spectrograms of different sizes can form a multi-scale input representation of the same audio segment to be processed. This ensures that the audio processing network can capture various details in the audio signal, leading to better audio quality assessment. For example, high-frequency resolution (corresponding to a larger number of channels) helps identify subtle sonic differences, while a longer time span helps understand the overall structure or background of the audio signal. Changing the sampling rate can affect the quality and processing efficiency of the audio signal; a higher sampling rate can capture more high-frequency information, theoretically providing a clearer and more natural sound.

[0036] For example, the audio segment to be processed is 12 seconds long and has a sampling rate of 48 kHz. Optionally, the audio segment to be processed can undergo three time-frequency conversions to obtain three Mel spectrograms. Obtaining three Mel spectrograms through three time-frequency conversions is only an example; three Mel spectrograms can also be obtained through a single time-frequency conversion, which includes three parallel conversion channels, and different channels can produce different Mel spectrograms. The process of obtaining three Mel spectrograms through three time-frequency conversions is explained below:

[0037] The audio segment to be processed undergoes a first time-frequency transformation to obtain a Mel spectrogram with a duration of 12 seconds, a sampling rate of 48 kHz, and 48 channels. For example, the audio segment to be processed can be mapped using a Short-Time Fourier Transform (STFT) and a filter bank consisting of 48 Mel filters to generate a Mel spectrogram with a duration of 12 seconds, a sampling rate of 48 kHz, and 48 Mel frequency bands (channels). The duration of 12 seconds, the sampling rate of 48 kHz, and the number of channels are merely examples of parameters for a Mel spectrogram and are not limited to these.

[0038] A second time-frequency transformation is performed on the audio segment to be processed, resulting in a Mel spectrogram with a duration of 10 seconds, a sampling rate of 48 kHz, and 64 channels. For example, the audio segment to be processed can be trimmed or aligned to extract a 10-second effective segment (still maintaining the 48 kHz sampling rate). This segment is then mapped using a short-time Fourier transform (STFT) and a filter bank consisting of 64 Mel filters to generate a 10-second Mel spectrogram with a sampling rate of 48 kHz and 64 channels, thereby enhancing the model's ability to perceive frequency details. The 10-second duration, 48 kHz sampling rate, and 64 channels are merely examples of parameters for the Mel spectrogram and are not limited to this.

[0039] A third time-frequency transformation is performed on the audio segment to be processed, resulting in a Mel spectrogram with a duration of 30 seconds, a sampling rate of 16 kHz, and 80 channels. For example, the audio segment to be processed can be extended to a length of 30 seconds through interpolation or cyclic expansion, and then downsampled to 16 kHz. This is then mapped using a short-time Fourier transform (STFT) and a filter bank consisting of 80 Mel filters to generate a 30-second Mel spectrogram with a sampling rate of 16 kHz and 80 channels, providing a longer temporal context and higher frequency resolution. The 30-second duration, 16 kHz sampling rate, and 80 channels are merely examples of Mel spectrogram parameters and are not limited to these.

[0040] The above example illustrates how time-frequency conversion of the audio segment to be processed yields three Mel spectrograms. Three Mel spectrograms are an example of at least one spectrogram, but are not limited to this.

[0041] In this embodiment, after performing time-frequency conversion on the audio segment to be processed to obtain at least one spectrogram, the audio processing network is used to extract features from the at least one spectrogram to obtain key frequency band features, time-frequency fusion features, and audio semantic features corresponding to the audio segment; then the key frequency band features, time-frequency fusion features, and audio semantic features corresponding to the audio segment can be fused to obtain a first fused feature; based on the first fused feature, the quality information of the audio segment is analyzed.

[0042] The audio processing network has an audio quality assessment function. The quality information of the audio segments output by the audio processing network is close to the subjective evaluation results of humans and has high accuracy. This quality information can be used to accurately assess the quality level of audio segments.

[0043] Optionally, during the model training phase of the audio processing network, the model is trained with the goal of ensuring consistency between the audio quality evaluation results output by the audio processing network and the manually labeled audio quality evaluation results. In this way, the audio quality evaluation results output by the trained audio processing network are less different from the subjective human scores, and the audio quality evaluation results are better.

[0044] Optionally, the audio processing network may include, but is not limited to, deep learning models, such as, convolutional neural networks (CNN), recurrent neural networks (RNN), long short-term memory networks (LSTM), gated recurrent units (GRU), encoder-decoder architectures (such as Transformer), convolutional recurrent neural networks (CRNN), and variations or combinations thereof.

[0045] Optionally, audio datasets from different scenarios can be used to train the model, resulting in a pre-trained audio processing network with good generalization ability. Then, the pre-trained audio processing network can be fine-tuned using audio datasets from specific scenarios to obtain an audio processing network with good implementation results in specific scenarios (such as live streaming scenarios).

[0046] Firstly, during the training sample preparation phase, multiple audio segments are collected, and a sufficient number of evaluators assign a Mean Opinion Score (MOS) value to each audio segment. Each audio segment and its manually labeled MOS value are used as a training sample. Next, the audio segments are converted into time-frequency spectrograms (such as Mel spectrograms) and used as model input to the audio processing network. The predicted MOS values ​​output by the audio processing network are obtained. The audio processing network is then adjusted based on the loss function between the predicted MOS values ​​and the manually labeled MOS values. The above steps are iteratively executed until the audio processing network meets the model iteration stopping condition, such as reaching the maximum number of training iterations or the model parameters of the audio processing network meeting the requirements.

[0047] MOS (Mean Opinion Score) is a standardized subjective audio quality assessment method. It involves organizing a group of evaluators (typically 20-30 people) to rate audio segments perceptually (e.g., on a 1-5 scale). Statistical methods are then used to remove outliers and clean the data from the raw scores given by the evaluators. Finally, the average score of each audio segment across multiple evaluators is calculated. The MOS value effectively reflects the subjective experience of audio quality by the human auditory system and is an important benchmark for assessing speech clarity, naturalness, and overall listening satisfaction.

[0048] Optionally, during the model training phase, in addition to focusing on the regression error between the predicted MOS values ​​of the audio processing network and the manually labeled MOS values, a learning-to-rank mechanism is further introduced. This mechanism optimizes the ranking consistency between the model output of the audio processing network and the subjective MOS scores of humans through optimization algorithms such as the ranking loss function. The aim is to ensure that the relative judgment of audio quality by the audio processing network is highly consistent with the order of human perception. Since in practical applications, users are more concerned with whether "audio A sounds better than audio B" rather than the precise matching of absolute scores, improving ranking accuracy is particularly crucial.

[0049] Regression errors include, but are not limited to: Mean Squared Error (MSE), Mean Absolute Error (MAE), and Root Mean Squared Error (RMSE).

[0050] Ranking loss functions are a class of loss functions used to learn the relative order of samples and are widely used in ranking tasks. Examples of ranking loss functions include, but are not limited to: Pairwise Ranking Loss, Spearman Rank Correlation Loss, Kullback-Leibler Divergence Loss, and so on.

[0051] Taking fine-tuning training in a live streaming scenario as an example, the first step is to construct an audio dataset for the live streaming scenario. For instance, a data sampling process from the live streaming room to the audio files is established, collecting a large number of audio segments (e.g., more than 10,000). Taking a 12-second duration for each audio segment and a sampling rate of 48kHz as an example, a sufficient number of evaluators assign corresponding MOS scores to each audio segment, using each scored audio segment as a training sample. Next, the audio segments are converted to time-frequency representations such as Mel-spectrum graphs, which are then used as model input to the audio processing network. The predicted MOS values ​​output by the audio processing network are obtained. The model parameters of the audio processing network are adjusted based on the regression loss function and ranking loss function between the predicted MOS values ​​output by the audio processing network and their manually labeled MOS values. Finally, the ranking consistency between the optimized model output and the manually labeled MOS values ​​is optimized through an algorithm. These steps are iteratively executed until the audio processing network meets the model iteration stopping condition, such as reaching the maximum number of training iterations or the model parameters of the audio processing network meeting the requirements. Descriptions of the regression loss function and ranking loss function can be found above and will not be repeated here.

[0052] In practical applications, the trained audio processing network can be deployed and quantized, the entire live broadcast can be sampled to obtain audio segments to be processed, and the audio processing network can be used to predict scores.

[0053] In this embodiment, the audio processing network also has multiple feature extraction functions during the audio quality assessment process. It can extract key frequency band features, time-frequency fusion features and audio semantic features of the audio segment to be processed, fuse the extracted features, and analyze the quality information of the audio segment based on the fused features.

[0054] The audio processing network can extract features from at least one spectrogram to obtain frequency domain features of different frequency bands, and fuse the frequency domain features of different frequency bands to obtain the key frequency band features of the audio segment to be processed.

[0055] The audio processing network can extract features from at least one spectrogram to obtain time-domain features of different time ranges and frequency-domain features of different frequency bands. It then fuses the time-domain features of different time ranges and the frequency-domain features of different frequency bands to obtain the time-frequency fusion features of the audio segment to be processed.

[0056] The audio processing network can perform semantic understanding on at least one spectrogram to obtain the audio semantic features of the audio segment to be processed.

[0057] In this embodiment, the network structure of the audio processing network is not limited. Optionally, for better feature extraction, see [link to relevant documentation]. Figure 2 The audio processing network may include: a frequency domain convolutional model, a time-frequency fusion model, and a semantic understanding model; correspondingly, at least one spectrogram includes: a first spectrogram, a second spectrogram, and a third spectrogram. The implementation method of using the audio processing network to extract features from at least one spectrogram to obtain key frequency band features, time-frequency fusion features, and audio semantic features corresponding to the audio segment is as follows: The frequency domain convolutional model is used to extract and fuse frequency band features from the first spectrogram at the first time block granularity to obtain key frequency band features corresponding to the audio segment; the time-frequency fusion model is used to extract and fuse global time-frequency features from the second spectrogram at the time-frequency block granularity to obtain time-frequency fusion features corresponding to the audio segment; and the semantic understanding model is used to perform speech semantic understanding processing on the third spectrogram to obtain audio semantic features corresponding to the audio segment.

[0058] In this embodiment, a frequency domain convolution model is used to extract and fuse frequency band features from the first spectrogram at the granularity of first time blocks to obtain the key frequency band features corresponding to the audio segment. Specifically, the first spectrogram is divided into blocks along the time dimension to obtain multiple block results, each block being called a first time block. The frequency domain convolution model treats each first time block as a frame, and performs feature extraction on each frame using the frequency domain convolution model to obtain the frequency domain features of the corresponding frequency band for each frame. The frequency domain features of different frequency bands are then fused to obtain the key frequency band features of the audio segment to be processed.

[0059] Optionally, to better extract key frequency band features, a frequency domain convolution model is used to extract and fuse frequency band features from the first spectrogram at the granularity of the first time block. The implementation method for obtaining the key frequency band features corresponding to the audio segment is as follows: the first spectrogram is divided into blocks along the time dimension according to the first duration granularity to obtain multiple first time blocks; frequency domain convolution model is used to extract and fuse frequency band features from each of the multiple first time blocks to obtain the key frequency band features corresponding to the multiple first time blocks. The first duration granularity can be flexibly set as needed, each first time block has the same duration, and the sum of the durations of all first time blocks is the duration corresponding to the first spectrogram.

[0060] See also Figure 2 Taking a Mel spectrogram with a duration of 12 seconds, a sampling rate of 48 kHz, and 48 channels as an example, the size of this Mel spectrogram is 1x1x48x1201. After time-blocking, the Mel spectrogram is divided into 297 sub-graphs, each with a duration of approximately 40.4 milliseconds, and each sub-graph serves as a first time block. After time-blocking, the Mel spectrogram is converted into a new tensor with a size of 297x1x48x15, and each sub-graph has a size of 1x48x15. The sub-graphs obtained after time-blocking are input into a frequency domain convolutional model for feature extraction to obtain the key frequency band features of each sub-graph. The key frequency band features of all sub-graphs are fused into a 1x297x384 feature vector. The four dimensions of the size information in the Mel spectrogram are batch, channel, height, and width. Batch represents the number of input objects; channel represents the dimension of features extracted from the input objects, with more channels indicating richer features extracted by the model; height and width represent the size information of the input image or feature map.

[0061] In this embodiment, the frequency domain convolution model can be any CNN network capable of frequency domain feature extraction. Optionally, the frequency domain convolution model includes: multiple convolutional modules with N-fold downsampling, where N is a positive integer greater than or equal to 2; the implementation method of using the frequency domain convolution model to extract and fuse frequency band features of multiple first time blocks to obtain key frequency band features corresponding to multiple first time blocks is as follows: multiple first time blocks are input into the frequency domain convolution model, and then sequentially passed through multiple convolutional modules with N-fold downsampling for convolution and downsampling operations to obtain key frequency band features corresponding to multiple first time blocks.

[0062] Specifically, by stacking multiple convolutional layers and downsampling operations, the frequency domain convolutional model achieves a progressive abstraction from local acoustic details to global semantic features, enabling the extraction of key frequency band features that determine audio quality. Each convolutional module performs convolution and downsampling operations on the input data, and the output of each convolutional module is downsampled by a factor of N relative to its input data, where N is greater than or equal to 2. For example, Figure 3The frequency domain convolutional model in the example has four convolutional modules with a 2x downsampling. The first convolutional module (Conv2d) has an input data size of 297x1x48x15 and an output feature map size of 297x16x24x7. The second convolutional module has an input data size of 297x16x24x7 and an output feature map size of 297x32x12x5. The third convolutional module has an input data size of 297x32x12x5 and an output feature map size of 297x64x6x3. The fourth convolutional module has an input data size of 297x64x6x3 and an output feature map size of 1x297x384. The size information of the output feature map of the fourth convolutional module has three dimensions: batch, time, and channel. Time represents the features in the time dimension.

[0063] The first convolutional module processes the original spectrogram blocks. The convolutional kernel in this layer is small and the receptive field is limited. It mainly captures local and subtle acoustic patterns, such as single harmonics, brief noise impulses, subtle spectral defects, and basic phoneme features (such as plosive consonants and formant origins of vowels).

[0064] The role of the middle-layer convolutional modules (such as the second and third convolutional modules) is as follows: after one downsampling, the temporal and frequency dimensions of the feature map are halved, and the receptive field of each feature point becomes larger. The convolutional kernel begins to integrate a wider range of information: on the frequency axis, it begins to fuse information from adjacent frequency bands, enabling the identification of wider formant structures, frequency band noise, etc. On the time axis, it can capture longer temporal patterns, such as the fluctuation of syllables and brief periods of silence.

[0065] The role of higher-level convolutional modules (such as the fourth layer and above) is to reduce the spatial size of the feature maps through multiple downsampling steps. At this point, each feature point corresponds to a fairly large region (in both time and frequency) of the original input. These features represent a relatively high-level abstraction. The network automatically learns and focuses on the frequency bands and time segments that contribute most to distinguishing audio quality at this layer. For example, it might learn that for speech quality, clarity in the 300Hz-3400Hz range (the main frequency band for telephone speech) is important. For music quality, the fullness of low frequencies (bass) and the clarity of high frequencies (such as cymbals) may be more important than mid-frequency frequencies.

[0066] In this embodiment, a time-frequency fusion model is used to perform global time-frequency feature extraction and fusion processing on the second spectrogram at the time-frequency block level to obtain the time-frequency fusion features corresponding to the audio segment. Specifically, the second spectrogram is divided into blocks at the time-frequency block level, with each block having a sufficient frequency and time range. Feature extraction is performed on each block to obtain time-domain features of different time ranges and frequency-domain features of different frequency bands. The time-domain features of different time ranges and the frequency-domain features of different frequency bands are then fused to obtain the time-frequency fusion features of the audio segment to be processed.

[0067] See also Figure 2 Taking the second spectrogram as an example—a Mel spectrogram with a duration of 10 seconds, a sampling rate of 48 kHz, and 64 channels—its dimensions are 1x1x64x1001. After feature extraction using a time-frequency fusion model, a 1x768x2x32 time-frequency fusion feature is obtained. This 1x768x2x32 time-frequency fusion feature is then adapted by an adapter to obtain a 1x128x384 time-frequency fusion feature. This adapter adjusts the feature dimensions to meet the needs of subsequent processing. In this process, an adapter is introduced to optimize the time-frequency fusion model to adapt to specific downstream tasks while maintaining the model parameters unchanged. Of course, the adapter can also be omitted. The four dimensions of the size information here are the same as those mentioned earlier.

[0068] Optionally, to maintain consistency between the time and frequency domains, a time-frequency fusion model is used to perform global time-frequency feature extraction and fusion processing on the second spectrogram at the time-frequency block level to obtain the time-frequency fusion features corresponding to the audio segment. The implementation method is as follows: the second spectrogram is input into the time-frequency fusion model, and the following operations are performed within the time-frequency fusion model: the second spectrogram is divided into blocks in the time-frequency dimension to obtain a time-frequency sub-block sequence; the time-frequency sub-block sequence is vectorized to obtain a vectorized sequence; based on the windowed self-attention mechanism, feature extraction and downsampling processing are performed on the vectorized sequence in the time-frequency domain to obtain the time-frequency fusion features corresponding to the audio segment.

[0069] Optionally, the time-frequency fusion model may include a segmentation network that can divide the second spectrogram into blocks in the time-frequency dimension.

[0070] Optionally, the second spectrogram is divided into blocks in the time-frequency dimension to obtain a time-frequency sub-block sequence as follows: the second spectrogram is divided into blocks in the time dimension according to the second duration granularity to obtain multiple second time blocks; for each second time block, time-frequency blocks are divided into multiple frequency bands contained in the second spectrogram using a time-frequency window to obtain time-frequency sub-blocks contained in the second time block; the time-frequency sub-blocks contained in the multiple second time blocks are organized according to the order between the multiple second time blocks to obtain a time-frequency sub-block sequence.

[0071] See Figure 3 First, the second spectrogram is divided into time-dimension blocks according to the second duration granularity, resulting in multiple sub-graphs, each of which serves as a second time block. If the second spectrogram has a duration of 10 seconds and a second duration granularity of 2.5 seconds, it is divided into four second time blocks, each with a duration of 2.5 seconds. The time-frequency window of the first second time block is denoted as w1, the second as w2, and the nth as w... n n is a positive integer greater than 2. For the first second time block, a time-frequency window w1 is used to divide the first second time block into multiple time-frequency windows w1 along the frequency (F) axis from top to bottom and along the time (T) axis from left to right; for the second second time block, a time-frequency window w2 is used to divide the second second time block into multiple time-frequency windows w2 along the frequency (F) axis from top to bottom and along the time (T) axis from left to right; for the nth second time block, a time-frequency window w1 is used to divide the second second time block into multiple time-frequency windows w2. n The second time block is divided into multiple time-frequency windows along the frequency (F) axis from top to bottom and along the time (T) axis from left to right. n Following the block division order, multiple time-frequency windows w1, multiple time-frequency windows w2, and multiple time-frequency windows w... n Organizing these elements, we obtain a time-frequency sub-block sequence, with each sub-block corresponding to a time-frequency window. A time-frequency window can be understood as a two-dimensional grid in a two-dimensional plane formed by the time axis and the frequency axis, with the horizontal axis being the time axis and the vertical axis being the frequency axis.

[0072] Optionally, the time-frequency fusion model may include a vectorization network to vectorize the time-frequency sub-block sequence to obtain a vectorized sequence.

[0073] See Figure 4 Embedding can be performed on each time-frequency sub-block to obtain its embedding information. Embedding can be considered a vectorization process; the embedding information of a time-frequency sub-block can be seen as its vectorized feature representation. Thus, the embedding information of each time-frequency sub-block in the time-frequency sub-block sequence forms a vectorized sequence, which includes the embedding information of each time-frequency sub-block. The output dimension information of the vectorized sequence is... Where T is the total number of time frames of the Mel spectrogram on the time axis, for example, 1001 frames; F is the number of frequency bands of the Mel spectrogram on the frequency axis, for example, 64 or 128; P is the number of time-frequency sub-blocks; and D is the embedding dimension.

[0074] In this embodiment, the second spectrogram is divided into multiple time-frequency sub-blocks and vectorized to obtain a vectorized sequence. Then, the time-frequency fusion model performs feature extraction and downsampling processing on the vectorized sequence in the time-frequency domain based on a windowed self-attention mechanism to obtain the time-frequency fusion features corresponding to the audio segment. In this way, the time-frequency fusion model can not only obtain local features within the time-frequency sub-blocks, but also effectively model various temporal information between the time-frequency sub-blocks through self-attention mechanism and positional encoding. Temporal information includes, but is not limited to, time order, duration, rhythm, periodicity, and long-distance dependence.

[0075] In this embodiment, the time-frequency fusion model uses a windowed self-attention mechanism to extract and downsample features from the vectorized sequence in the time-frequency domain to obtain the time-frequency fusion features corresponding to the audio segment. The time-frequency fusion model can extract and downsample the vectorized feature representation of each time-frequency sub-block in the vectorized sequence in the time-frequency domain using a windowed self-attention mechanism to obtain the time-frequency fusion features corresponding to the audio segment.

[0076] Among them, the windowed self-attention mechanism is a variant of the self-attention mechanism. It constrains the computation range of the attention mechanism by setting a fixed-size window, only calculates the similarity between positions within the window, and normalizes the attention scores to obtain weights. It is mainly used to reduce the high time and space complexity of standard self-attention computation, especially when processing long sequences.

[0077] In some alternative embodiments, see Figure 4 The time-frequency fusion model includes an audio encoder. Based on a windowed self-attention mechanism, it performs feature extraction and downsampling on the vectorized sequence in the time-frequency domain to obtain the time-frequency fusion features corresponding to the audio segment. The implementation method is as follows: the vectorized sequence is input into the audio encoder, and multi-scale feature extraction and downsampling are performed on the vectorized sequence sequentially based on the windowed self-attention mechanism and block-level merging method to obtain the time-frequency fusion features corresponding to the audio segment. Among them, the block-level merging method refers to merging the features corresponding to adjacent time-frequency sub-blocks to achieve downsampling.

[0078] Specifically, the audio encoder can perform multiple feature extractions on the vectorized sequence, and then perform block-level merging downsampling after each feature extraction. First, based on a windowed self-attention mechanism, the vectorized feature representation of each time-frequency sub-block in the vectorized sequence is processed to obtain an intermediate feature representation for each time-frequency sub-block. For two adjacent time-frequency sub-blocks, the intermediate feature representations of the two adjacent time-frequency sub-blocks are merged using a block-level merging method to complete downsampling. Next, the next feature extraction is performed. Based on the windowed self-attention mechanism, the intermediate feature representations of the time-frequency sub-blocks output from the previous feature extraction are processed to obtain a new intermediate feature representation for each time-frequency sub-block. For two adjacent time-frequency sub-blocks, the intermediate feature representations of the two adjacent time-frequency sub-blocks are merged using a block-level merging method to complete downsampling. These steps are repeated until the time-frequency fusion features corresponding to the audio segment are obtained.

[0079] In some optional embodiments, the audio encoder employs a hierarchical design of multiple feature extraction networks. The input features of the first feature extraction network are vectorized sequences, and the input features of subsequent feature extraction networks are the output features of the previous feature extraction network. The input or output features correspond to multiple time-frequency sub-blocks, and the size of the output features of the multiple feature extraction networks decreases sequentially. Optionally, the feature extraction networks here can adopt an encoder-decoder architecture, such as a Transformer network structure, but are not limited to this.

[0080] Optionally, the feature extraction network may include a Swin-Transformer and a Patch-Merge component. Some feature extraction networks may consist of only a Swin-Transformer. For example, see [link to example]. Figure 4 The audio encoder comprises four feature extraction networks. The first three feature extraction networks include a windowing transformer and a block merging component, while the last feature extraction network includes a block merging component. The windowing transformer extracts features from any given time-frequency sub-block, and the block merging component merges the features corresponding to two adjacent time-frequency sub-blocks. Figure 4 It can be seen that the size of the input data for the first feature extraction network is... The size of the input data for the second feature extraction network is The size of the input data for the third feature extraction network is The size of the input data for the fourth feature extraction network is After being extracted by each feature extraction network, the feature map was downsampled.

[0081] Based on the above, the vectorized sequence is input into the audio encoder, and multi-scale feature extraction and downsampling are performed on the vectorized sequence sequentially based on the windowed self-attention mechanism and block-level merging method to obtain the time-frequency fusion features corresponding to the audio segment. The implementation method is as follows: the vectorized sequence is input into the audio encoder and processed sequentially through multiple feature extraction networks until the output feature of the last feature extraction network is obtained as the time-frequency fusion feature corresponding to the audio segment; in any feature extraction network, the input features are extracted in the time-frequency domain based on the windowed self-attention mechanism to obtain the intermediate features of multiple time-frequency sub-blocks corresponding to the input features; the intermediate features of multiple time-frequency sub-blocks are downsampled using the block-level merging method to obtain the output feature of any feature extraction network.

[0082] In some optional embodiments, the time-frequency fusion model further includes a text encoder; a contrastive learning approach is used to train an initial model formed by the audio encoder and the text encoder based on audio samples and text samples to obtain a time-frequency fusion model with cross-modal semantic alignment capabilities. Thus, using a time-frequency fusion model with cross-modal semantic alignment capabilities can better extract time-frequency fusion features with richer semantic information corresponding to audio segments, thereby better capturing the essence of the audio content.

[0083] In practical applications, time-frequency fusion models can be deep learning models that achieve cross-modal alignment of audio and text through contrastive learning. During model training, a large number of audio and text samples are prepared. Semantically matching audio and text samples are used as positive samples, and semantically mismatched audio and text samples are used as negative samples. Audio samples can be understood as audio segments participating in model training, and text samples can be understood as text data participating in model training. The initial model can be various deep learning models to be trained, including an audio encoder and a text encoder. During contrastive learning, the audio encoder is used to extract feature vectors from the audio samples; the text encoder is used to extract feature vectors from the text samples. By optimizing the contrastive loss function, positive samples are made closer in the feature representation space, i.e., the distance between the feature vectors of the audio samples and the text samples in the positive samples is reduced; negative samples are made farther apart in the feature representation space, i.e., the distance between the feature vectors of the audio samples and the text samples in the negative samples is increased. In this way, through contrastive learning, the time-frequency fusion model learns to shorten the distance between the feature vectors of audio samples and text samples in positive sample pairs, while simultaneously widening the distance between the feature vectors of audio samples and text samples in negative sample pairs; the closer the distance, the greater the similarity. Ultimately, the time-frequency fusion model achieves cross-modal representation alignment in a shared semantic space, bringing semantically related audio and text closer to each other in the vector space.

[0084] In this embodiment, a semantic understanding model is used to perform speech semantic understanding processing on the third spectrogram to obtain the audio semantic features corresponding to the audio segment.

[0085] Optionally, the third spectrogram can be preprocessed before being input into the semantic understanding model. Preprocessing can be performed by a neural network layer containing one-dimensional convolution (Conv1d), activation functions (such as GeLU), and transpose operations. Specifically, this neural network layer first performs a one-dimensional convolution operation on the input third spectrogram to extract local features in the temporal direction; then, it introduces nonlinearity through an activation function to enhance the model's expressive power; finally, it adjusts the dimensional order of the tensors through a transpose operation to make the output format more suitable for the input requirements of the subsequent semantic understanding model. This preprocessing helps compress the input data, supporting the effective extraction of audio semantic features.

[0086] See also Figure 2 Taking a Mel spectrogram with a duration of 30 seconds, a sampling rate of 16kHz, and 80 channels as an example, the size of this Mel spectrogram is 1x1x80x3000. After preprocessing, the size of the preprocessed third spectrogram is 1x1500x384. A semantic understanding model is then used to extract semantic features from the preprocessed third spectrogram, yielding the audio semantic features corresponding to the audio segment (size 1x1500x384). The audio semantic features corresponding to the audio segment can also have their feature dimensions adjusted using an adapter to suit subsequent processing needs. Of course, an adapter can also be omitted. The four dimensions of the size information here are the same as the four dimensions mentioned earlier.

[0087] In this embodiment, the semantic understanding model can be any model with powerful semantic understanding capabilities, and there is no limitation thereto. Optionally, the semantic understanding model is obtained by fine-tuning a pre-trained speech processing model using an audio quality evaluation task, and the pre-trained speech processing model is obtained by training using multiple speech processing tasks.

[0088] The pre-trained speech processing model is trained using large-scale, diverse speech datasets through various self-supervised or weakly supervised pre-training tasks. This model is capable of performing multiple speech processing tasks, such as automatic speech recognition (ASR), speech translation, and speech understanding. The training data for the pre-trained model covers a large amount of speech from real-world scenarios, typically including complex factors such as background noise, reverberation, different accents, and recording conditions. This data characteristic allows the model to learn robustness to noise and interference during pre-training, thus exhibiting good generalization ability and noise perception capability in downstream tasks.

[0089] A semantic understanding model can be obtained by fine-tuning a pre-trained speech processing model for an audio quality evaluation task, thus giving the semantic understanding model a strong semantic understanding capability for audio signals.

[0090] In this embodiment, the model structure of the semantic understanding model is not limited. Optionally, in order to enable the semantic understanding model to perform deeper speech semantic understanding, the semantic understanding model may include multiple self-attention modules connected in sequence, with the output of the previous self-attention module serving as the input data for the next self-attention module.

[0091] Optionally, the semantic understanding model includes multiple sequentially connected self-attention modules. The implementation method of using the semantic understanding model to perform speech semantic understanding processing on the third spectrogram to obtain the audio semantic features corresponding to the audio segment is as follows: repeatedly execute the step of inputting the semantic understanding result output by the previous self-attention module into the next self-attention module to obtain the semantic understanding result output by the next self-attention module, until the semantic understanding result output by the last self-attention module is obtained, and the semantic understanding result output by the last self-attention module is used as the audio semantic feature corresponding to the audio segment. When the first self-attention module is used as the next self-attention module, the third spectrogram is used as the semantic understanding result of the previous self-attention module of the first self-attention module.

[0092] In this embodiment, the structure of the self-attention module is not limited. Optionally, in order to enable the semantic understanding model to perform deeper speech semantic understanding, the main core components of each self-attention module include, but are not limited to: layer normalization, multi-head self-attention mechanism, and multi-layer perceptron (MLP).

[0093] Based on the above, the semantic understanding result output by the previous self-attention module is input into the next self-attention module to obtain the semantic understanding result output by the next self-attention module. This includes: performing layer normalization on the semantic understanding result output by the previous self-attention module to obtain the layer normalization result; processing the layer normalization result using a multi-head self-attention mechanism to obtain high-dimensional feature information of the audio segment, which represents the context information of the audio segment; and processing the high-dimensional feature information of the audio segment using a multilayer perceptron to output the semantic understanding result output by the next self-attention module.

[0094] Optionally, to enable the semantic understanding model to perform deeper speech semantic understanding, the implementation method of using a multilayer perceptron to process the high-dimensional feature information of the audio segment and output the semantic understanding result of the next self-attention module includes: performing a residual connection between the layer normalization result and the high-dimensional feature information of the audio segment to obtain the residual connection result; processing the result based on the residual connection using a multilayer perceptron to output the multilayer perceptron processing result; and performing a residual connection between the multilayer perceptron processing result and the first residual connection result to obtain the semantic understanding result output by the next self-attention module.

[0095] For example, Figure 5 The semantic understanding model shown includes four sequentially connected self-attention modules. Within each self-attention module, there is a layer normalization mechanism (LayerNorm), a multi-head attention mechanism, and a multilayer perceptron. Residual connections (Add) follow the multi-head attention mechanism and the multilayer perceptron. The multilayer perceptron can contain two linear layers, possibly with activation functions in between. Figure 5 (Not shown in the image).

[0096] Optional, see Figure 6 The workflow of the multi-head self-attention mechanism is as follows: The input data is sliced ​​into multiple parts for multi-head computation. For example, input data with a size of 1x1500x384 is sliced ​​into 6 heads, each with a feature dimension of 1500x64. Each head undergoes three linear transformations to obtain the corresponding query vector, key vector, and value vector. A single-head self-attention mechanism is then used to process these vectors, yielding the single-head self-attention result for each head. The single-head self-attention results from multiple single-head self-attention mechanisms are concatenated to obtain a concatenated high-dimensional tensor. This concatenated high-dimensional tensor is then subjected to a linear transformation to obtain the output of the multi-head self-attention mechanism. The three dimensions of the size information here are the same as the three dimensions mentioned above, namely batch, time, and channel.

[0097] Optional, see Figure 7The workflow of the single-head self-attention mechanism is as follows: The input data is sliced ​​into three parts: query (Q), key (K), and value (V). Matrix multiplication (Mat Mul) is used to calculate the similarity score between the query (Q) and key (K). The similarity score obtained by matrix multiplication is processed using a normalization function (SoftMax) to obtain normalized attention weights. The normalized attention weights and value (V) are then calculated using matrix multiplication (Mat Mul) to obtain the output of the single-head self-attention mechanism.

[0098] In this embodiment, the key frequency band features, time-frequency fusion features, and audio semantic features corresponding to the audio segment are fused to obtain the first fused feature. The fusion method is not limited and can be flexibly selected according to specific task requirements and model structure.

[0099] Optionally, the key frequency band features, time-frequency fusion features and audio semantic features corresponding to the audio segment are fused in the time domain to obtain the first fused feature.

[0100] Specifically, a temporal alignment and concatenation fusion method is used to fuse the three features to obtain the first fused feature. This fusion method is simple to implement and preserves the original information of each modality. For example, see... Figure 2 The time steps for key frequency band features are 297, the time steps for time-frequency fusion features are 128, the time steps for audio semantic features are 1500, and the time steps for the first fusion feature are 1925.

[0101] Optionally, information fusion is performed on the key frequency band features, time-frequency fusion features and audio semantic features corresponding to the audio segment based on the multi-head attention mechanism to obtain the first fused feature.

[0102] This approach utilizes a multi-head attention mechanism to dynamically calculate the importance weights of different features at different time steps, achieving context-aware feature aggregation. This method automatically learns the correlations and dependencies between features, giving the model stronger representational capabilities and flexibility.

[0103] In this embodiment, the quality of the audio segment is evaluated based on the first fusion feature. For example, global pooling is directly performed on the first fusion feature in the time dimension, and the resulting fixed-dimensional vector is input into one or more fully connected layers to output a quality score. Alternatively, a multilayer perceptron is used to evaluate the quality of the audio segment based on the first fusion feature.

[0104] The quality assessment results for audio segments can be varied. These results may include quality scores, quality grades, or textual descriptions of audio quality, but are not limited to these. Quality scores quantify audio quality numerically, such as a rating from 0 to 100. Audio quality can be categorized into different quality grades, such as "Excellent," "Good," "Average," and "Poor." Textual descriptions can describe audio quality using natural language, such as "high background noise" or "clear vocals with slight distortion."

[0105] Optionally, the method for evaluating the quality of an audio segment based on the first fusion feature is as follows: the first fusion feature is weighted and enhanced with global contextual information using a self-attention mechanism to obtain a second fusion feature; the second fusion feature is aggregated using a pooling layer to obtain the quality score corresponding to the audio segment.

[0106] Specifically, by introducing a self-attention mechanism, the long-range dependencies of audio features over time can be effectively modeled. The second fusion feature contains richer contextual semantic information, making the final quality assessment result more accurate, stable, and closer to human subjective auditory perception. The self-attention mechanism is used to weight and enhance the first fusion feature, highlighting those features more important for quality judgment and suppressing irrelevant or redundant information, thus obtaining a second fusion feature that better reflects the overall audio quality. Then, pooling operations are used to map the second fusion feature to the quality score corresponding to the audio segment.

[0107] The audio signal processing method provided in this application, by performing time-frequency conversion on the audio segment to be processed and extracting key frequency band features, time-frequency fusion features, and audio semantic features, achieves a comprehensive representation of the audio signal in three dimensions: local frequency domain details, time-frequency dynamic changes, and high-level semantic content. By fusing these three types of features to generate a first fusion feature with complementary information and stronger representational capabilities, and performing quality analysis based on this feature, it can more accurately and robustly identify quality defects such as distortion and noise in the audio, effectively avoiding misjudgments caused by content changes, thereby significantly improving the accuracy and reliability of audio quality assessment and making it closer to human subjective auditory perception.

[0108] In the foregoing embodiments of this application, a 48kHz audio input is used, and a large amount of audio data and MOS scores are collected as sample pairs for model training. The trained audio processing network can evaluate broadband distortion. If a hole appears in a certain frequency band, the audio processing network can also output high-accuracy quality information. Furthermore, in the foregoing embodiments of this application, a complex deep learning modeling approach is used, which performs better than a vector machine classification and regression model with very few parameters. Moreover, sample data is obtained from real live streaming scenarios, without synthetic data, resulting in better inference performance on real data and a closer approximation of human subjective rating levels. Furthermore, in the foregoing embodiments of this application, sample data is obtained from real live streaming scenarios. The sample data is rich in content, including information dimensions such as content intelligibility, volume, noticeable stuttering, noise level, whether the voice is smooth and soft or thin and sharp, and the degree of reverberation. The audio processing network trained based on this richer sample data has a high degree of perception of noise, volume, stuttering, and other dimensions. Its overall multi-dimensional perception is closer to human perception, and it has a strong ability to perceive the sound characteristics of live streaming scenarios, enabling accurate quality evaluation of the audio in live streaming scenarios.

[0109] The audio signal processing method provided in the foregoing embodiments of this application can be applied to live streaming or short video scenarios. Taking live streaming as an example, it can score the audio acquisition, audio encoding, audio transmission, audio enhancement and repair processes in the live streaming scenario, provide a voice quality monitoring and feedback system, and guide the quality improvement of the entire live audio link.

[0110] Figure 8 A flowchart illustrating an audio signal processing method provided in an embodiment of this application. See also... Figure 8 The method may include the following steps:

[0111] 801. Sample the audio signal in the live broadcast or short video scene to obtain the audio segment to be processed.

[0112] 802. Use an audio processing network to extract features from at least one spectrogram to obtain key frequency band features, time-frequency fusion features, and audio semantic features corresponding to the audio segment.

[0113] 803. The key frequency band features, time-frequency fusion features and audio semantic features corresponding to the audio segment are fused to obtain the first fused feature.

[0114] 804. Based on the first fusion feature, the quality of the audio segment is evaluated and processed.

[0115] 805. If, based on quality information, it is determined that an audio segment does not meet the audio quality requirements for live streaming or short video scenarios, the audio quality of the audio segment shall be enhanced or repaired in order to play the enhanced or repaired audio segment.

[0116] In this embodiment, if the audio segment is determined based on the quality information obtained from the aforementioned analysis to not meet the preset audio quality requirements for live streaming or short video scenarios (e.g., low signal-to-noise ratio, significant distortion, insufficient speech clarity, excessive background noise, etc.), an audio quality enhancement or repair mechanism will be automatically triggered. Optionally, corresponding audio enhancement modules, such as AI noise reduction, speech enhancement, dereverberation, dynamic range control, and bandwidth expansion modules, can be invoked to process the audio segment to compensate for or eliminate the problems causing substandard quality. After processing, the enhanced or repaired audio segment will be played or output.

[0117] In various audio scenarios such as live streaming and short videos, the quality of the audio signal is a crucial metric. Traditional Audio Quality Assessment (AQA) methods are weak in perceiving dimensions such as noise and stuttering, making it difficult to accurately evaluate audio quality. However, the audio signal processing method provided in this application can accurately evaluate audio quality and enhance or repair the sound quality based on the evaluation results.

[0118] The detailed implementation methods and beneficial effects of each step in this embodiment have been described in detail in the foregoing embodiments, and will not be elaborated here.

[0119] Figure 9 This is a schematic diagram of an audio signal processing device provided in an embodiment of this application. See also... Figure 9 The device may include:

[0120] The time-frequency conversion module 91 is used to perform time-frequency conversion on the audio segment to be processed in order to obtain at least one spectrogram.

[0121] The feature extraction module 92 is used to extract features from at least one spectrogram using an audio processing network to obtain key frequency band features, time-frequency fusion features and audio semantic features corresponding to the audio segment;

[0122] The feature fusion module 93 is used to fuse the key frequency band features, time-frequency fusion features and audio semantic features corresponding to the audio segment to obtain the first fused feature;

[0123] The audio quality evaluation module 94 is used to evaluate the quality of audio segments based on the first fusion feature.

[0124] Optionally, at least one spectrogram includes: a first spectrogram, a second spectrogram, and a third spectrogram; the feature extraction module 92 is specifically used for: using a frequency domain convolution model to perform frequency band feature extraction and fusion processing on the first spectrogram at the granularity of a first time block, to obtain the key frequency band features corresponding to the audio segment; using a time-frequency fusion model to perform global time-frequency feature extraction and fusion processing on the second spectrogram at the granularity of time-frequency blocks, to obtain the time-frequency fusion features corresponding to the audio segment; and using a semantic understanding model to perform speech semantic understanding processing on the third spectrogram, to obtain the audio semantic features corresponding to the audio segment.

[0125] Optionally, when the feature extraction module 92 extracts the key frequency band features corresponding to the audio segment, it is specifically used to: divide the first spectrogram into blocks in the time dimension according to the first duration granularity to obtain multiple first time blocks; and use a frequency domain convolution model to extract and fuse the frequency band features of the multiple first time blocks to obtain the key frequency band features corresponding to the multiple first time blocks.

[0126] Optionally, the frequency domain convolution model includes: multiple convolution modules with N times downsampling, where N is a positive integer greater than or equal to 2; when the feature extraction module 92 extracts the key frequency band features corresponding to the audio segment, it is specifically used to: input multiple first time blocks into the frequency domain convolution model, and sequentially pass them through multiple convolution modules with N times downsampling for convolution and downsampling operations to obtain the key frequency band features corresponding to multiple first time blocks.

[0127] Optionally, when the feature extraction module 92 extracts the time-frequency fusion features corresponding to the audio segment, it specifically performs the following operations: inputting the second spectrogram into the time-frequency fusion model, and performing the following operations within the time-frequency fusion model: dividing the second spectrogram into blocks in the time-frequency dimension to obtain a time-frequency sub-block sequence; vectorizing the time-frequency sub-block sequence to obtain a vectorized sequence; and performing feature extraction and downsampling processing on the vectorized sequence in the time-frequency domain based on a windowed self-attention mechanism to obtain the time-frequency fusion features corresponding to the audio segment.

[0128] Optionally, when the feature extraction module 92 obtains the time-frequency sub-block sequence, it is specifically used to: divide the second spectrogram into blocks in the time dimension according to the second duration granularity to obtain multiple second time blocks; for each second time block, use a time-frequency window to divide it into time-frequency blocks on multiple frequency bands contained in the second spectrogram to obtain time-frequency sub-blocks contained in the second time block; and organize the time-frequency sub-blocks contained in the multiple second time blocks according to the order between the multiple second time blocks to obtain a time-frequency sub-block sequence.

[0129] Optionally, the time-frequency fusion model includes an audio encoder; when the feature extraction module 92 extracts the time-frequency fusion features corresponding to the audio segment, it is specifically used to: input the vectorized sequence into the audio encoder, and perform multi-scale feature extraction and downsampling processing on the vectorized sequence sequentially based on the windowed self-attention mechanism and block-level merging method to obtain the time-frequency fusion features corresponding to the audio segment; wherein, the block-level merging method refers to merging the features corresponding to adjacent time-frequency sub-blocks to achieve downsampling.

[0130] Optionally, the audio encoder employs multiple feature extraction networks with a hierarchical design. The input features of the first feature extraction network are vectorized sequences, and the input features of subsequent feature extraction networks are the output features of the previous feature extraction network. The input features or output features correspond to multiple time-frequency sub-blocks, and the size of the output features of multiple feature extraction networks decreases sequentially. Optionally, the feature extraction network here can adopt an encoder-decoder architecture, such as the Transformer network structure, but is not limited to this.

[0131] When the feature extraction module 92 extracts the time-frequency fusion features corresponding to the audio segment, it specifically performs the following steps: inputs the vectorized sequence into the audio encoder, processes it sequentially through multiple feature extraction networks until the output feature of the last feature extraction network is obtained as the time-frequency fusion feature corresponding to the audio segment; in any feature extraction network, it performs feature extraction on the input features in the time-frequency domain based on a windowed self-attention mechanism to obtain the intermediate features of multiple time-frequency sub-blocks corresponding to the input features; and it downsamples the intermediate features of multiple time-frequency sub-blocks using a block-level merging method to obtain the output feature of any feature extraction network.

[0132] Optionally, the time-frequency fusion model also includes a text encoder. A contrastive learning approach is used to train an initial model formed by the audio encoder and text encoder based on audio and text samples, resulting in a time-frequency fusion model with cross-modal semantic alignment capabilities. In this optional embodiment, the time-frequency fusion model can be a deep learning model that achieves cross-modal alignment of audio and text through contrastive learning. Contrastive learning is a self-supervised learning method whose core idea is to bring similar sample pairs closer together and push dissimilar sample pairs further apart, allowing features with the same semantic meaning to be closer. Since the training process includes text modality input, the audio encoder will simultaneously carry audio and text features. Injecting text modality information into the video fusion model allows audio features to maintain a strong causal order, facilitating the prediction and selection of temporal information. Combining the model structure of the aforementioned time-frequency fusion model, during model training, this contrastive learning training method can be used, employing a windowed self-attention mechanism and a hierarchical multi-scale Transformer network structure. This structure improves inference efficiency and enhances the modeling capabilities of local and global contexts.

[0133] Optionally, the semantic understanding model includes multiple sequentially connected self-attention modules. When the feature extraction module 92 extracts the audio semantic features corresponding to the audio segment, it is specifically used to: repeatedly execute the step of inputting the semantic understanding result output by the previous self-attention module into the next self-attention module to obtain the semantic understanding result output by the next self-attention module, until the semantic understanding result output by the last self-attention module is obtained, and the semantic understanding result output by the last self-attention module is used as the audio semantic feature corresponding to the audio segment. When the first self-attention module is used as the next self-attention module, the third spectrogram is used as the semantic understanding result of the previous self-attention module of the first self-attention module.

[0134] Optionally, the feature fusion module 93 is specifically used to: perform temporal fusion of the key frequency band features, time-frequency fusion features and audio semantic features corresponding to the audio segment to obtain a first fused feature; or, perform information fusion of the key frequency band features, time-frequency fusion features and audio semantic features corresponding to the audio segment based on a multi-head attention mechanism to obtain a first fused feature.

[0135] Optionally, the audio quality evaluation module 94 is specifically used to: perform weighted enhancement processing of the first fusion feature with global context information using a self-attention mechanism to obtain the second fusion feature; and perform aggregation processing of the second fusion feature using a pooling layer to obtain the quality score corresponding to the audio segment.

[0136] The detailed implementation methods and beneficial effects of each module in the device of this embodiment have been described in detail in the foregoing embodiments, and will not be elaborated here.

[0137] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 104 can be device A; or the execution subject of steps 101 and 102 can be device A, and the execution subject of steps 103 to 104 can be device B; and so on.

[0138] Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations appearing in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0139] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 10 As shown, the electronic device includes: a memory 11 and a processor 12;

[0140] Memory 11 is used to store computer programs and can be configured to store various other data to support operation on the computing platform. Examples of this data include instructions for any application or method operating on the computing platform, data structures, contact data, phone book data, messages, pictures, videos, etc.

[0141] The processor 12, coupled to the memory 11, is used to execute the computer program in the memory 11 for: performing steps in the audio signal processing method.

[0142] Optional, such as Figure 10 As shown, the electronic device also includes other components such as a communication component 13, a display 14, a power supply component 15, and an audio component 16. Figure 10 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 10 The components shown. Additionally... Figure 10 The components within the dashed box are optional, not mandatory, and their specific requirements depend on the product form of the electronic device. The electronic device in this embodiment can be a desktop computer, laptop computer, smartphone, or IoT (Internet of Things) device, or a server-side device such as a conventional server, cloud server, or server array. If the electronic device in this embodiment is a desktop computer, laptop computer, or smartphone, it may include... Figure 10 The components within the dashed box; if the electronic device in this embodiment is implemented as a conventional server, cloud server, or server array, etc., it may be omitted. Figure 10 The component within the dashed box.

[0143] The aforementioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0144] The aforementioned communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel.

[0145] The aforementioned display includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen can be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0146] The aforementioned power supply components provide power to various components within the device in which they reside. These power supply components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which they reside.

[0147] The aforementioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0148] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile components, or a combination thereof, and can be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium.

[0149] Accordingly, this application also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, cause the processor to implement the steps in the above method embodiments. It should be understood that each step or combination of steps in the above method flow can be implemented by the computer program or instructions. Furthermore, these computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, enabling the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to function as an apparatus for implementing the corresponding functions in the above method embodiments.

[0150] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0151] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method of audio signal processing, characterized by, The method comprises: performing time-frequency conversion on an audio segment to be processed to obtain at least one spectrogram; extracting features from the at least one spectrogram by using an audio processing network to obtain key frequency band features, time-frequency fusion features and audio semantic features corresponding to the audio segment; fusing the key frequency band features, the time-frequency fusion features and the audio semantic features corresponding to the audio segment to obtain first fusion features; evaluating the quality of the audio segment based on the first fusion features.

2. The method of claim 1, wherein, The at least one spectrogram comprises: a first spectrogram, a second spectrogram and a third spectrogram; extracting features from the at least one spectrogram by using an audio processing network to obtain key frequency band features, time-frequency fusion features and audio semantic features corresponding to the audio segment, comprising: extracting and fusing frequency band features from the first spectrogram by using a frequency domain convolution model with a first time block as a granularity to obtain the key frequency band features corresponding to the audio segment; extracting and fusing global time-frequency features from the second spectrogram by using a time-frequency fusion model with a time-frequency block as a granularity to obtain the time-frequency fusion features corresponding to the audio segment; performing speech semantic understanding processing on the third spectrogram by using a semantic understanding model to obtain the audio semantic features corresponding to the audio segment.

3. The method of claim 2, wherein, extracting and fusing frequency band features from the first spectrogram by using a frequency domain convolution model with a first time block as a granularity to obtain the key frequency band features corresponding to the audio segment, comprising: blocking the first spectrogram in a time dimension according to a first time length granularity to obtain a plurality of first time blocks; extracting and fusing frequency band features from the plurality of first time blocks by using the frequency domain convolution model to obtain key frequency band features corresponding to the plurality of first time blocks.

4. The method of claim 3, wherein, The frequency domain convolution model comprises: a plurality of N times down-sampling convolution modules, N being a positive integer greater than or equal to 2; extracting and fusing frequency band features from the plurality of first time blocks by using the frequency domain convolution model to obtain key frequency band features corresponding to the plurality of first time blocks, comprising: inputting the plurality of first time blocks into the frequency domain convolution model, and sequentially performing convolution operation and down-sampling processing through the plurality of N times down-sampling convolution modules to obtain the key frequency band features corresponding to the plurality of first time blocks.

5. The method according to any one of claims 2 to 4, characterized in that, extracting and fusing global time-frequency features from the second spectrogram by using a time-frequency fusion model with a time-frequency block as a granularity to obtain the time-frequency fusion features corresponding to the audio segment, comprising: inputting the second spectrogram into the time-frequency fusion model, and performing the following operations in the time-frequency fusion model: blocking the second spectrogram in a time-frequency dimension to obtain a time-frequency sub-block sequence; vectorizing the time-frequency sub-block sequence to obtain a vectorized sequence; extracting and down-sampling features from the vectorized sequence in a time-frequency domain based on a windowed self-attention mechanism to obtain the time-frequency fusion features corresponding to the audio segment.

6. The method of claim 5, wherein, blocking the second spectrogram in a time-frequency dimension to obtain a time-frequency sub-block sequence, comprising: performing time dimension blocking on the second spectrogram according to a second time length granularity to obtain a plurality of second time blocks; performing time-frequency blocking on a plurality of frequency bands contained in the second spectrogram by using a time-frequency window for each second time block to obtain time-frequency sub-blocks contained in the second time block; organizing the time-frequency sub-blocks contained in the plurality of second time blocks according to an order between the plurality of second time blocks to obtain the sequence of time-frequency sub-blocks.

7. The method of claim 6, wherein, The time-frequency fusion model comprises an audio encoder; performing feature extraction and down-sampling processing on the vectorized sequence in the time-frequency domain based on a windowed self-attention mechanism to obtain time-frequency fusion features corresponding to the audio segment, comprising: inputting the vectorized sequence into the audio encoder, and sequentially performing multi-scale feature extraction and down-sampling processing on the vectorized sequence based on a windowed self-attention mechanism and a block-level merging manner to obtain time-frequency fusion features corresponding to the audio segment; wherein the block-level merging manner refers to a manner of merging features corresponding to adjacent time-frequency sub-blocks to realize down-sampling.

8. The method of claim 7, wherein, The audio encoder adopts a plurality of feature extraction networks designed in a hierarchical manner, the input feature of a first feature extraction network is the vectorized sequence, the input feature of a non-first feature extraction network is the output feature of a previous feature extraction network, the input feature or the output feature corresponds to a plurality of time-frequency sub-blocks, and the size of the output feature of the plurality of feature extraction networks decreases successively; inputting the vectorized sequence into the audio encoder, and sequentially performing multi-scale feature extraction and down-sampling processing on the vectorized sequence based on a windowed self-attention mechanism and a block-level merging manner to obtain time-frequency fusion features corresponding to the audio segment, comprising: inputting the vectorized sequence into the audio encoder, and sequentially processing through the plurality of feature extraction networks until the output feature of a last feature extraction network is obtained as the time-frequency fusion features corresponding to the audio segment; In any feature extraction network, the input feature is extracted in the time-frequency domain based on a windowed self-attention mechanism to obtain intermediate features of a plurality of time-frequency sub-blocks corresponding to the input feature; and the intermediate features of the plurality of time-frequency sub-blocks are down-sampled by using a block-level merging manner to obtain the output feature of the any feature extraction network.

9. The method according to claim 7 or 8, characterized in that, The time-frequency fusion model further comprises a text encoder; and the method further comprises: performing model training on an initial model formed by the audio encoder and the text encoder based on an audio sample and a text sample by using a contrastive learning manner to obtain a time-frequency fusion model having a cross-modal semantic alignment capability.

10. The method according to any one of claims 2-4 or 6-8, characterized in that, The semantic understanding model comprises a plurality of self-attention modules connected in sequence, and the semantic understanding model is used to perform speech semantic understanding processing on the third spectrogram to obtain audio semantic features corresponding to the audio segment, comprising: The step of inputting the semantic understanding result output by the previous self-attention module into the next self-attention module is repeatedly performed until the semantic understanding result output by the last self-attention module is obtained, and the semantic understanding result output by the last self-attention module is taken as the audio semantic feature corresponding to the audio segment, wherein when the first self-attention module is taken as the next self-attention module, the third spectrum is taken as the semantic understanding result of the previous self-attention module of the first self-attention module.

11. The method according to any one of claims 2-4 or 6-8, characterized in that, The key frequency band feature, the time-frequency fusion feature, and the audio semantic feature corresponding to the audio segment are fused to obtain a first fusion feature, including: The key frequency band feature, the time-frequency fusion feature, and the audio semantic feature corresponding to the audio segment are fused in the time domain to obtain the first fusion feature. Or The key frequency band feature, the time-frequency fusion feature, and the audio semantic feature corresponding to the audio segment are fused based on a multi-head attention mechanism to obtain the first fusion feature.

12. The method of any one of claims 2-4 or 6-8, wherein, Based on the first fusion feature, the quality of the audio segment is evaluated and processed, including: The first fusion feature is weighted and enhanced by using a self-attention mechanism to obtain a second fusion feature; The second fusion feature is aggregated by using a pooling layer to obtain a quality score corresponding to the audio segment.

13. An audio signal processing method, characterized by, Including: An audio signal in a live broadcast or short video scene is sampled to obtain an audio segment to be processed; The steps in the method of any one of claims 1-12 are used to obtain quality information of the audio segment; If it is determined based on the quality information that the audio segment does not meet the audio quality requirement of the live broadcast or short video scene, the audio quality of the audio segment is enhanced or repaired to play the audio segment after the audio quality is enhanced or repaired.

14. An electronic device, comprising: Including: A memory and a processor; The memory is configured to store a computer program; The processor is coupled to the memory and configured to execute the computer program to perform the steps in the method of any one of claims 1-13.

15. A computer readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the processor is enabled to implement the steps in the method of any one of claims 1-13.

16. A computer program product, characterised in that, The computer program / instructions enable the processor to implement the steps in the method of any one of claims 1-13 when the computer program / instructions are executed by the processor.

Images