Audio data evaluation method and device, equipment, storage medium and program product

Abstract

The invention relates to an audio data evaluation method and device, equipment, a storage medium and a program product. The method comprises the following steps: acquiring an original audio, wherein the original audio comprises a reference audio and a to-be-evaluated target audio; human ear perception processing is carried out on the original audio to obtain human ear perception domain audio, and the human ear perception domain audio comprises a reference human ear perception domain audio corresponding to the reference audio and a target human ear perception domain audio corresponding to the target audio; performing evaluation processing on the original audio and the human ear perception domain audio by adopting an evaluation network; and determining difference information between the reference audio and the target audio according to an evaluation processing result, and determining an evaluation result of the target audio according to the difference information. By adopting the method, the accuracy of audio evaluation can be improved, so that the evaluation result better fits the subjective characteristics of human ears.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to an audio data evaluation method, device, equipment, storage medium and program product. Background Art

[0002] In the process of evaluating audio data, the relevant technology considers two quality attributes, namely the objective attribute of measuring the degree of distortion of spatial audio and the spatial quality, and uses multi-task learning and triplet training to enable the neural network to simultaneously learn the basic audio quality and spatial quality attributes of spatial audio, thereby making a more complete evaluation of the overall quality of spatial audio. In this way, only the objective quality of distortion and the spatial quality are considered in the evaluation process, which makes the correlation between the evaluation results and human subjective evaluation low. Summary of the invention

[0003] Based on this, it is necessary to provide an audio data evaluation method, device, electronic device, storage medium and program product that can improve the accuracy of audio data evaluation in response to the above technical problems.

[0004] In a first aspect, the present application provides an audio data evaluation method. The method comprises:

[0005] Acquire original audio, where the original audio includes reference audio and target audio to be evaluated;

[0006] Performing human ear perception processing on the original audio to obtain human ear perception domain audio, wherein the human ear perception domain audio includes reference human ear perception domain audio corresponding to the reference audio and target human ear perception domain audio corresponding to the target audio;

[0007] Using an evaluation network to perform evaluation processing on the original audio and the audio in the human ear perception domain;

[0008] Difference information between the reference audio and the target audio is determined according to the result of the evaluation process, and the evaluation result of the target audio is determined according to the difference information.

[0009] In a second aspect, the present application also provides an audio data evaluation device. The device comprises:

[0010] A first acquisition module, used to acquire original audio, where the original audio includes reference audio and target audio to be evaluated;

[0011] A first processing module is used to perform human ear perception processing on the original audio to obtain human ear perception domain audio, where the human ear perception domain audio includes reference human ear perception domain audio corresponding to the reference audio and target human ear perception domain audio corresponding to the target audio;

[0012] A first evaluation module, configured to use an evaluation network to perform evaluation processing on the original audio and the audio in the human ear perception domain;

[0013] The first determination module is used to determine the difference information between the reference audio and the target audio according to the result of the evaluation process, and determine the evaluation result of the target audio according to the difference information.

[0014] In a third aspect, the present application further provides an electronic device, wherein the electronic device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the first aspect are implemented.

[0015] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the first aspect are implemented.

[0016] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the first aspect are implemented.

[0017] The above-mentioned audio data evaluation method, device, electronic device, storage medium and program product, after obtaining the target audio and reference audio to be evaluated, can obtain the reference human ear perception domain audio and the target human ear perception domain audio by performing human ear perception processing on the reference audio and the target audio respectively. The evaluation network is used to perform evaluation processing on the original audio and the human ear perception domain audio. In this way, since the characteristics of the human ear perception domain are comprehensively considered in the reference human ear perception domain audio and the target human ear perception domain audio, the difference information between the reference audio and the target audio can be determined more accurately according to the result of the evaluation processing, so that the evaluation result of the target audio can be more accurately evaluated according to the difference information, and the correlation between the evaluation result and the subjective evaluation of the human ear is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A diagram showing an application environment of an audio data evaluation method in an embodiment;

[0019] Figure 2 A schematic diagram of a flow chart of an audio data evaluation method in one embodiment;

[0020] Figure 3 Another schematic diagram of a flow chart of an audio data evaluation method in one embodiment;

[0021] Figure 4 1 is another flow chart of an audio data evaluation method in one embodiment;

[0022] Figure 5 1 is another flow chart of an audio data evaluation method in an embodiment;

[0023] Figure 6 Another schematic diagram of a flow chart of an audio data evaluation method in one embodiment;

[0024] Figure 7 A schematic diagram of the implementation flow of a human ear perception processing process in one embodiment;

[0025] Figure 8 1 is a schematic diagram of the implementation flow of the time-frequency conversion process of audio data in one embodiment;

[0026] Fig. 9 A schematic diagram of a real framework of an audio data evaluation method in one embodiment;

[0027] Fig.10 A schematic diagram of the composition structure of an evaluation network in one embodiment;

[0028] Fig.11 A schematic diagram of an implementation architecture of an evaluation network in one embodiment;

[0029] Fig.12 Another schematic diagram of a practical framework of an audio data evaluation method in one embodiment;

[0030] Fig.13 A schematic diagram of a framework for determining an evaluation result of audio data in one embodiment;

[0031] Fig.14 is a structural block diagram of an audio data evaluation device in one embodiment;

[0032] Fig.15 FIG. 4 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0034] In one embodiment, the spatial audio objective evaluation index is determined by the following methods:

[0035] Method 1: Assessment and Prediction of Binaural Aspects of Audio Quality (BAMQ) based on binaural cues. BAMQ takes a pair of matching spatial audio as input, and after front-end processing, it is input into the back-end for objective quality evaluation. The front-end of BAMQ includes two stages: outer ear sound processing and binaural feature extraction. In the outer ear sound processing part, the BAMQ algorithm simulates the sound processing process of the human outer ear, basilar membrane, cochlea, and cochlear hair through four sets of filters, mapping the spatial audio to the human ear perception domain. In the binaural feature extraction stage, BAMQ performs fine structure filtering and modulation filtering on the audio mapped to the human ear perception domain, then calculates the interaural transfer function (ITF), and extracts four binaural features, including interaural phase difference (IPD), interaural time difference (ITD), interaural vector strength (IVS), and interaural level difference (ILD). Finally, the extracted binaural features are input into the back end. At the back end, BAMQ first calculates the corresponding sub-indicators based on the binaural features extracted by the front end, and then inputs the sub-indicators into a series of weights calculated by the Multivariate Adaptive Regression Splines (MARS) method to calculate the final spatial audio objective quality score.

[0036] Method 2: A Deep Perceptual Spatial Audio Localization Metric (DPLM) based on deep feature distance: DPLM first trains a deep neural network that predicts binaural direction of arrival (DOA) at the frame level, and then uses the deep feature distance of the above neural network to evaluate the spatial positioning of spatial audio. Specifically, the input of DPLM is two binaural spatial audios, reference audio x1 and target audio x2. After short-time Fourier transform (STFT), the amplitude spectrum and phase spectrum of the binaural are extracted respectively, and then stacked and input into the neural network. The neural network in DPLM consists of three layers, namely, the feature extraction layer based on the inception network, the time domain aggregation layer based on the long short-term memory network (LSTM), and the fifty-class localization head based on full connection. After the amplitude / phase spectrum of the stacked spatial audio is input into the network, the network calculates the corresponding embedding layer. Finally, the DPLM algorithm embeds the reference audio x1 and the target audio x2 into the deep feature distance between the layers as the localization score of the spatial audio.

[0037] Method 3: Spatial Audio Quality Assessment Metric (SAQAM) based on deep learning: SAQAM is based on a multi-task learning framework. It considers two quality attributes of spatial audio, namely, listening quality (LQ) and spatial quality (SQ). It trains a deep neural network model to perform objective quality evaluation of spatial audio. For spatial quality, SAQAM uses a similar idea to DPLM. By predicting the binaural arrival direction at the frame level, it implicitly learns the spatial quality of spatial audio in the embedding layer. For auditory quality, SAQAM uses contrastive learning and triple training methods. By constructing a triple training sample of <anchor sample, positive sample, negative sample> and ensuring |anchor sample-positive sample|<|anchor sample-negative sample|, the network's embedded implicit learning of the auditory quality of spatial audio is achieved. The network structure of SAQAM consists of a feature extraction layer, a time-domain aggregation layer, and a multi-task head (auditory quality, spatial quality). The feature extraction layer is implemented using an Inception structure similar to DPLM, and the time-domain aggregation block is implemented using a time-domain convolutional network based on causal convolution and dilated convolution. The two multi-task heads are implemented using one-dimensional convolution, weight normalization, and full connection, respectively. For the input reference audio x1 and target audio x2, SAQAM first performs a short-time Fourier transform on the input audio, and stacks the extracted amplitude and phase spectra of the left and right ears, respectively, and inputs them into the network for calculation, and outputs the embedding layer of the feature extraction layer, the time-domain aggregation layer, the auditory quality task head, and the spatial quality task head. Finally, the network calculates the Overall Quality (OVRL) score based on the shared feature extraction layer and the depth feature distance of the time-domain aggregation layer, calculates the auditory quality score based on the feature extraction layer, the time-domain aggregation layer, and the auditory quality task head, and calculates the spatial quality score based on the feature extraction layer, the time-domain aggregation layer, and the spatial quality task head. Among them, the auditory quality of audio data in SAQAM is an indicator that measures the objective attribute of the audio distortion degree, rather than an indicator that measures subjective attributes such as the timbre of the audio. The auditory quality has nothing to do with the subjective perception of the human ear.

[0038] In the first approach, the method takes into account human perception of spatial audio, maps the audio to the human ear perception domain, makes it more consistent with human subjective perception, and predicts the spatial audio quality through binaural cues and multivariate linear regression. However, the BAMQ algorithm has strict requirements on input, requiring the reference audio and the target audio to be time-aligned and of equal length, which makes it difficult to apply to real scenes in some cases.

[0039] In the second method, the concept of deep feature distance is introduced. By predicting the arrival direction of spatial audio and using the implicit information in the network embedding layer, the spatial audio positioning deviation is predicted, which can reflect the spatial quality of spatial audio. However, in addition to spatial quality, the auditory quality of spatial audio (such as changes in fidelity) also affects the overall quality of spatial audio. Therefore, DPLM's objective evaluation of spatial audio is flawed and incomplete.

[0040] In the third method, in order to address the shortcomings of DPLM, two quality attributes of spatial audio, auditory quality and spatial quality, are considered. Through multi-task learning and triple training, the algorithm's neural network can simultaneously learn the auditory quality and spatial quality attributes of spatial audio, and can also make a more objective evaluation of the overall quality of spatial audio. However, SAQAM only trains the neural network through a series of objective quality attributes, ignoring the subjective characteristics of the human ear, and the correlation between the evaluation results and the subjective evaluation of the human ear cannot be guaranteed.

[0041] Of the three implementations above, using only the original spectrogram ignores the complex psychoacoustic phonemes that influence human auditory perception. While capturing the objective distortion level, it does not take into account how the auditory system reacts to different frequencies and masks certain sounds. This can lead to an incomplete understanding of the audio, resulting in poor subjective evaluation results of sound distortion. In contrast, the downsampled spectrogram, which relies solely on the physiological limitations of human hearing, ignores the objective characteristics of the original audio signal. While capturing psychoacoustic properties, it does not cover the entire frequency range and details in the source signal. This can hinder the model's ability to accurately capture various types of audio distortion.

[0042] Based on this, the embodiment of the present application provides an audio data evaluation method. In order to obtain a comprehensive and accurate audio representation, the human ear perception audio data and the original audio data are combined so that the evaluation network can make full use of the original spectrum content, and also take into account the complexity of human ear auditory perception, so as to improve the performance of subjective evaluation. The audio data evaluation method can obtain reference human ear perception domain audio and target human ear perception domain audio by performing human ear perception processing on the obtained reference audio and target audio respectively. The evaluation network is used to perform evaluation processing on the original audio and the human ear perception domain audio. In this way, since the characteristics of the human ear perception domain are comprehensively considered in the reference human ear perception domain audio and the target human ear perception domain audio, the difference information between the reference audio and the target audio can be determined more accurately according to the result of the evaluation processing, so that the evaluation result of the target audio can be more accurately evaluated according to the difference information, and the correlation between the evaluation result and the subjective evaluation of the human ear is high.

[0043] The audio data evaluation method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, after the reference audio 102 and the target audio 103 to be evaluated are obtained in the audio data evaluation terminal 101, the reference human ear perception domain audio and the target human ear perception domain audio can be obtained by performing human ear perception processing on the reference audio and the target audio respectively. The evaluation network is used to evaluate the original audio and the human ear perception domain audio. In this way, since the characteristics of the human ear perception domain are comprehensively considered in the reference human ear perception domain audio and the target human ear perception domain audio, the difference information between the reference audio and the target audio can be determined more accurately according to the result of the evaluation processing, so that the evaluation result of the target audio can be more accurately evaluated according to the difference information, and the correlation between the evaluation result and the subjective evaluation of the human ear is higher. Among them, the audio data evaluation terminal 101 can be, but not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices, and the Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc.

[0044] In one embodiment, Figure 2 As shown, a method for evaluating audio data is provided, and the method is applied to Figure 1 The audio data evaluation terminal in the example is used as an example to illustrate, including the following steps:

[0045] Step 201, obtaining original audio.

[0046] The original audio includes a reference audio and a target audio to be evaluated. The reference audio and the target audio may be time-aligned and equal in length, or may be time-unaligned and unequal in length. The audio content of the reference audio and the target audio may be the same or different.

[0047] Exemplarily, the reference audio may be audio data with good audio quality that can be used as a benchmark, such as undamaged audio data. The target audio may be any type of audio data, such as audio data containing noise, compressed audio data, or damaged audio data. The target audio may be collected by an audio data evaluation terminal, or may be received by receiving target audio sent by other terminals.

[0048] Step 202: Perform human ear perception processing on the original audio to obtain audio in the human ear perception domain.

[0049] The human ear perception domain audio includes reference human ear perception domain audio corresponding to the reference audio and target human ear perception domain audio corresponding to the target audio. The human ear perception processing is to map the reference audio and the target audio from the original original time domain to the human ear perception domain, so as to perform human ear simulation on the reference audio and the target audio, thereby obtaining the reference human ear perception domain audio and the target human ear perception domain audio.

[0050] In some possible implementations, the reference audio and the target audio are filtered respectively by multiple filters to simulate the perception and processing of the human ear to the sound, so as to transfer the original time domain audio data to the human ear perception domain. Taking the human ear perception processing of the reference audio as an example, the reference audio is first filtered by a bandpass filter to simulate the frequency response of the human outer ear and middle ear and the perception and processing of the sound. Then the filtered reference audio is filtered by a high-order filter to simulate the perception and processing of the sound by the human basilar membrane; then, the reference audio after the high-order filtering is instantaneously compressed to simulate the cochlear compression processing of the sound by the human cochlea; finally, the audio data after instantaneous compression is filtered by a low-pass filter to simulate the mechanical and electrical transmission process of the human cochlear hair cells, and the audio data of the reference audio in the human ear perception domain is obtained, that is, the reference human ear perception domain audio. Similarly, the target audio can be converted to the human ear perception domain according to the same steps to obtain the target human ear perception domain audio.

[0051] Step 203: Use an evaluation network to perform evaluation processing on the original audio and the audio in the human ear perception domain.

[0052] The evaluation network is a trained network for evaluating the audio quality of the target audio. The original audio and the audio in the human ear perception domain are evaluated and processed by the evaluation network respectively; that is, the original audio is evaluated and processed by the evaluation network to obtain the evaluation processing result of the original audio, and the audio in the human ear perception domain is evaluated and processed by the evaluation network to obtain the evaluation processing result of the audio in the human ear perception domain.

[0053] Exemplarily, the evaluation network may include an objective distortion quality evaluation subnetwork and a spatial quality evaluation subnetwork, and the original audio is evaluated by the objective distortion quality evaluation subnetwork, and the audio in the human ear perception domain is evaluated by the spatial quality evaluation network, so as to realize the evaluation network to evaluate the original audio and the audio in the human ear perception domain. In the embodiment of the present application, considering the loss caused by the dimensionality reduction of the perceptual audio processing, in order to enhance the correlation between the evaluation result output by the evaluation network and the subjective perception of the human ear, the evaluation network is trained by the human ear perception audio data to assist the original audio, so as to realize the evaluation of the audio data; in this way, it is possible to obtain a more accurate evaluation result and solve the problem of audio loss. Although the auditory quality of audio data in SAQAM is an indicator for measuring the objective distortion degree of audio, the objective distortion degree of audio is not directly related to the auditory perception of the human ear. The way in which the human ear perceives the objective distortion of audio is affected by the complexity of auditory cognition and the physiological limitations of the human ear. Therefore, in the embodiment of the present application, when evaluating the impact of the objective distortion of audio on the auditory perception of the human ear, it is crucial to consider the overall interaction between human psychoacoustic factors and objective distortion.

[0054] Step 204: Determine difference information between the reference audio and the target audio according to the result of the evaluation process, and determine the evaluation result of the target audio according to the difference information.

[0055] The result of the evaluation process may include the result of the evaluation process of the original audio and the result of the evaluation process of the audio in the human ear perception domain. The result of the evaluation process of the original audio can represent the depth characteristics of the original audio in the objective distortion quality evaluation, and the evaluation process of the audio in the human ear perception domain can represent the depth characteristics of the audio in the human ear perception domain and the original audio in terms of spatial quality. The difference information is obtained by determining the distance between the depth characteristics corresponding to different audio data.

[0056] The difference information is used to reflect the difference in space, audio fidelity and overall quality between the target audio and the reference audio with higher audio quality. For example, the difference information represents the spatial feature distance, objective feature distance and overall quality feature distance between the reference audio and the target audio.

[0057] In some possible implementations, the original audio and the audio in the human ear perception domain are subjected to time-frequency conversion to obtain first spectrum information and second spectrum information. The first spectrum information and the second spectrum information are separately input into the evaluation network, and the feature set of the reference audio and the feature set of the target audio are predicted by the evaluation network. For the target audio, since the second spectrum information includes the spectrum information in the original time domain and the spectrum information in the human ear perception domain, the evaluation network first extracts features from the spectrum information in the original time domain and the spectrum information in the human ear perception domain, and performs time domain aggregation and other processing; then, the spatial quality head and the objective distortion quality evaluation head in the evaluation network are used to determine the spatial quality recognition features and the objective distortion quality evaluation recognition features.

[0058] Similarly, the reference audio is processed by the evaluation network to extract features and perform time domain aggregation and other processing; the spatial quality head and objective distortion quality evaluation head in the network are then used to determine the spatial quality recognition features and objective distortion quality evaluation recognition features. Finally, the deep feature distance between the reference audio and the target audio is determined by the features of the reference audio output by the evaluation network and the features of the target audio output by the evaluation network, thus obtaining the difference information.

[0059] Among them, the evaluation results of the target audio include: the spatial quality, objective distortion quality evaluation and overall quality of the target audio. Here, objective distortion quality evaluation is an objective attribute for measuring audio fidelity, and is an objective attribute that is unrelated to the subjective perception of the human ear. Since the expression of audio quality in SAQAM is ambiguous as being understood as measuring subjective attributes such as the timbre of the audio; therefore, in an embodiment of the present application, the objective attribute for measuring audio fidelity is expressed by adopting objective distortion quality evaluation, so that the expression of the evaluation results is more direct and accurate.

[0060] In some possible implementations, the evaluation result of the target audio can be obtained by converting the difference information into a quality score. For example, the objective distortion quality evaluation score of the target audio can be obtained by converting the characteristic distance on the objective distortion quality evaluation in the difference information into a quality score; the spatial quality score of the target audio can be obtained by converting the characteristic distance on the spatial quality in the difference information into a quality score; the overall quality score of the target audio can be obtained by converting the comprehensive characteristic distance in the difference information into a quality score. Afterwards, the objective distortion quality evaluation score, the spatial quality score and the overall quality score are combined to more comprehensively and accurately evaluate the evaluation result of the target audio.

[0061] In an embodiment of the present application, after obtaining the target audio and reference audio to be evaluated, the reference audio and the target audio are respectively subjected to human ear perception processing, so that the reference human ear perception domain audio and the target human ear perception domain audio can be obtained. In this way, since the characteristics of the human ear perception domain are comprehensively considered in the reference human ear perception domain audio and the target human ear perception domain audio, the evaluation network is used to evaluate the original audio and the human ear perception domain audio, so that the result of the evaluation processing can be more in line with the subjectivity of the human ear. Afterwards, according to the result of the evaluation processing, the difference information between the reference audio and the target audio can be determined more accurately, so that the evaluation result of the target audio can be more accurately evaluated according to the difference information, and the correlation between the evaluation result and the subjective evaluation of the human ear is high.

[0062] In one embodiment, the evaluation network includes an objective distortion quality evaluation sub-network and a spatial quality evaluation sub-network. The above step 203 can be performed by Figure 3 The steps shown achieve:

[0063] Step 301: Use the objective distortion quality assessment subnetwork to perform objective assessment processing on at least the original audio to obtain a plurality of reference objective assessment features corresponding to the reference audio and a plurality of target objective assessment features corresponding to the target audio.

[0064] Among them, after the reference audio and the target audio in the original audio are converted into time domain, the amplitude spectrum and phase spectrum obtained after the time domain conversion are input into the objective distortion quality evaluation sub-network to obtain multiple reference objective evaluation features and multiple target objective evaluation features. Alternatively, the amplitude spectrum and phase spectrum corresponding to the original audio and the amplitude spectrum and phase spectrum of the audio in the human ear perception domain are combined and input into the objective distortion quality evaluation sub-network to obtain multiple reference objective evaluation features and multiple target objective evaluation features.

[0065] In some possible implementations, the objective distortion quality assessment subnetwork includes a cascaded first feature extraction layer, a first time domain aggregation layer, and an objective distortion quality assessment head. The above step 301 can be implemented by the following steps 311 to 315 (not shown in the figure):

[0066] Step 311: Use the first feature extraction layer to perform feature extraction processing on the original audio to obtain a first spectrum feature, where the first spectrum feature includes a first reference spectrum feature corresponding to the reference audio and a first target spectrum feature corresponding to the target audio.

[0067] Among them, after the original audio is converted into time-frequency, the spectrum information is obtained and input into the objective distortion quality assessment sub-network. The first feature extraction layer is used to extract features from the spectrum information of the input original audio to obtain the first spectrum feature. Exemplarily, the feature extraction layer includes 6 layers of stacked feature extraction blocks. After the 6th layer of feature extraction blocks, 1*1 convolution is used to compress the channel dimension, and the frequency dimension is compressed on the frequency dimension. The input dimension of the feature extraction block is [B, 4, f, 257], and the dimension of the output first spectrum feature is [B, f, 64]. In this way, by extracting features from the input spectrum information, the spectrum features of the original time domain can be extracted more accurately and fully.

[0068] Step 312: Use the first time domain aggregation layer to perform time domain aggregation processing on the first spectrum feature to obtain a first aggregation feature.

[0069] The first aggregated features include a first reference aggregated feature corresponding to the reference audio and a first target aggregated feature corresponding to the target audio. The first spectral feature output by the first feature extraction layer is input to the first time domain aggregation layer to obtain the aggregated feature output by the time domain aggregation layer.

[0070] In some possible implementations, the first reference spectrum feature and the first target spectrum feature in the original time domain are aggregated through a first time domain aggregation layer to obtain a first aggregate feature. In the first time domain aggregation layer, the first spectrum feature is first convolved with an increasing number of channels, and then convolved with a decreasing number of channels, to obtain the first aggregate feature.

[0071] Step 313: Use the objective distortion quality assessment head to perform objective distortion quality assessment identification processing on the first aggregated feature to obtain an objective distortion quality assessment identification feature, wherein the objective distortion quality assessment identification feature includes a reference objective distortion quality assessment identification feature corresponding to the reference audio and a target objective distortion quality assessment identification feature corresponding to the target audio.

[0072] The first aggregate feature is input into the objective distortion quality assessment head to predict the objective distortion quality assessment of the target audio from the original first aggregate feature, that is, to obtain the objective distortion quality assessment identification feature.

[0073] The objective distortion quality assessment head may include two layers of stacked 1*3 convolution kernels and weight normalization, and two fully connected layers. After the first aggregated feature is input into the objective distortion quality assessment head, it is processed in sequence through two layers of stacked 1*3 convolution kernels and weight normalization, and two fully connected layers to obtain an objective distortion quality assessment identification feature; in this way, the objective distortion quality assessment identification feature is independent of human ear perception, so the objective distortion quality assessment of the target audio in the original time domain can be purely represented by the objective distortion quality assessment identification feature.

[0074] Step 314: Use the first reference spectrum feature, the first reference aggregation feature, and the reference objective distortion quality assessment identification feature as the multiple reference objective assessment features.

[0075] Step 315: Use the first target spectrum feature, the first target aggregation feature, and the target objective distortion quality assessment identification feature as the multiple target objective assessment features.

[0076] In an embodiment of the present application, the original audio is evaluated and processed by the first feature extraction layer, the first time domain aggregation layer and the objective distortion quality assessment head cascaded in the objective distortion quality assessment subnetwork. Since the input features in the entire processing process are the features of the reference audio and the target audio in the original time domain, which are irrelevant to human ear perception, the output of the objective distortion quality assessment subnetwork can intuitively represent the objective distortion quality assessment of the target audio in the original time domain.

[0077] Step 302: Use the spatial quality evaluation subnetwork to perform spatial evaluation processing on the audio in the human ear perception domain, and fuse the evaluation features obtained by the objective evaluation processing during the spatial evaluation processing to obtain multiple reference spatial evaluation features corresponding to the reference audio and multiple target spatial evaluation features corresponding to the target audio.

[0078] Among them, after performing time domain conversion on the reference audio and target audio in the human ear perception domain audio, the amplitude spectrum and phase spectrum obtained after the time domain conversion are input into the objective distortion quality evaluation sub-network to obtain multiple reference objective evaluation features and multiple target objective evaluation features.

[0079] In some possible implementations, the spatial quality assessment subnetwork includes a cascaded second feature extraction layer, a second time domain aggregation layer, and a spatial quality head. The above step 302 may be implemented by the following steps 321 to 326 (not shown):

[0080] Step 321: Use the second feature extraction layer to perform feature extraction processing on the audio in the human ear perception domain to obtain a second spectrum feature, where the second spectrum feature includes a second reference spectrum feature corresponding to the reference audio and a second target spectrum feature corresponding to the target audio.

[0081] The second feature extraction layer has the same structure as the first feature extraction layer and shares parameters. After performing time-frequency conversion on the audio in the human ear perception domain, spectrum information is obtained and input into the spatial quality assessment subnetwork. The second feature extraction layer is used to extract features from the spectrum information of the input audio in the human ear perception domain to obtain a second spectrum feature.

[0082] Step 322: Use the second time domain aggregation layer to perform time domain aggregation processing on the second spectrum feature to obtain a second aggregation feature.

[0083] The second aggregated features include a second reference aggregated feature corresponding to the reference audio and a second target aggregated feature corresponding to the target audio. The second spectral feature output by the second feature extraction layer is input to the second time domain aggregation layer to obtain the second aggregated feature output by the time domain aggregation layer.

[0084] In some possible implementations, the second aggregation layer has the same structure as the first aggregation layer and shares parameters. The second reference spectrum feature and the second target spectrum feature are aggregated through the second time domain aggregation layer to obtain a second aggregate feature. Exemplarily, in the second time domain aggregation layer, the second spectrum feature is first convolved with an increasing number of channels and then convolved with a decreasing number of channels to obtain the second aggregate feature, which can be achieved by the following process:

[0085] In the time domain aggregation layer of the evaluation network, the original time domain features are first convolved with an increasing number of channels, and then convolved with a decreasing number of channels to obtain aggregated features. That is, the above step 322 can be implemented by the following process:

[0086] First, in the second time domain aggregation layer, at least one level of convolution processing is performed on the second spectrum feature according to a plurality of preset channel numbers that increase step by step to obtain a first convolution feature.

[0087] Among them, the second time domain aggregation layer includes two weight normalizations, two 1*3 convolution kernels and a residual module. After the spectral features of the original time domain and the spectral features of the human ear perception domain are input into the time domain aggregation layer, in the four-layer time domain aggregation layer, the number of channels gradually increases to 64, 128, and 256. In this way, in the second time domain aggregation layer, the second spectral features are convolved according to the process of increasing the number of channels from 64 to 128 and 256 to obtain the first convolution feature.

[0088] Secondly, according to the plurality of preset channel numbers that decrease step by step, the first convolution feature is subjected to the at least one level of convolution processing to obtain a second convolution feature;

[0089] Among them, the number of channels gradually decreases to 256, 128, and 64; the first convolution feature is convolved according to the number of channels decreasing from 256 to 128 and 64, and the second convolution feature is output; in this way, the receptive field of the second convolution feature can be expanded by dilated convolution.

[0090] Finally, residual processing is performed on the second convolutional features to obtain the second aggregated features.

[0091] Among them, the second convolution feature is subjected to residual processing through the reference module in the time domain aggregation layer, for example, the second convolution feature is processed by a residual function or a residual network to obtain an output aggregation feature. In this way, by first convolving the spectral feature with a plurality of preset channel numbers that increase step by step, and then convolving with a plurality of preset channel numbers that decrease step by step, the receptive field of the second convolution feature can be improved through the expansion convolution, thereby improving the receptive field of the aggregation feature, so that the aggregation feature can be richer.

[0092] Step 323: Perform weighted fusion processing on the second aggregated feature and the first aggregated feature to obtain a fused feature.

[0093] The fusion feature includes a reference fusion feature corresponding to the reference audio and a target fusion feature corresponding to the target audio. The second aggregate feature and the first aggregate feature are weighted and summed using a preset weight to obtain the fusion feature.

[0094] The preset weight includes two different weight values, and the sum of the two different weight values ​​is 1.

[0095] Step 324: Use the spatial quality head to perform spatial quality identification processing on the fusion feature to obtain a spatial quality identification feature, wherein the spatial quality identification feature includes a reference spatial quality identification feature corresponding to the reference audio and a target spatial quality identification feature corresponding to the target audio.

[0096] The spatial quality identification feature is a multidimensional matrix. The fusion feature is input into the spatial quality head, so that the spatial quality head performs operations such as convolution and weight normalization on the input fusion feature, and outputs the spatial quality identification feature.

[0097] In some possible implementations, in the spatial quality head, the input fusion features are classified by azimuth angle to obtain spatial quality identification features for evaluating the spatial quality of the target audio. That is, the above step 324 can be implemented by the following process:

[0098] First, the fused features are preprocessed to obtain preprocessed features.

[0099] The spatial quality head includes: two layers of stacked 1*3 convolution and weight normalization, two layers of fully connected layers and soft maximum activation function. The input features are convolved and weight normalized by two layers of stacked 1*3 convolution and weight normalization to realize the preprocessing process of fusion features and obtain preprocessed features. In this way, convolution and weight normalization of input features can improve the training speed of the network and thus improve the accuracy of prediction.

[0100] Then, the azimuth angles corresponding to the preprocessing features are classified to obtain the probability distribution of the azimuth angles in a set of preset angle intervals.

[0101] In the spatial quality head, the interval [0°, 360°] is first divided into multiple equidistant sectors, that is, a set of preset angle intervals is obtained. The azimuth corresponding to the preprocessing feature is the azimuth of the audio frame to which the preprocessing feature belongs, including: the horizontal angle and elevation angle of the audio frame in the polar coordinate system. The polar coordinate system takes the human ear as the origin and faces the sound source. The polar coordinates of the vector represented by the preprocessing feature in the polar coordinates, that is, the horizontal angle and the elevation angle, are the azimuth corresponding to the preprocessing feature.

[0102] Exemplarily, the azimuth angles corresponding to the preprocessing features are classified by two fully connected layers and a soft maximum activation function in the spatial quality head to obtain a probability distribution of the azimuth angles in a set of preset angle intervals.

[0103] Finally, the probability distributions corresponding to the preset angle interval set are fused to obtain the spatial quality identification feature.

[0104] After obtaining the probability distribution of the azimuth angle of the preprocessing feature in each preset angle interval, these probability distributions are spliced ​​to obtain a spatial quality identification feature including multiple probability distributions. In this way, by performing angle prediction on the azimuth angle of the audio frame where the preprocessing feature is located, the azimuth angle is classified to obtain the probability that the azimuth angle is in each preset angle interval; thus, by combining the probability distribution corresponding to each preset angle interval, a spatial quality identification feature is formed, so that the value of each latitude in the spatial quality identification feature can reflect the probability distribution of the azimuth angle of the target audio in the preset angle interval.

[0105] Step 325: Use the second reference spectrum feature, the second reference aggregation feature, and the reference space quality identification feature as the multiple reference space evaluation features.

[0106] Step 326: Use the second target spectrum feature, the second target aggregation feature and the target space quality identification feature as the multiple target space evaluation features.

[0107] In an embodiment of the present application, a spatial quality head is used to learn the weighted combined fusion features to achieve the goal of evaluating the spatial quality of the target audio. In this way, the human ear perception domain features and the original time domain features are incorporated into the process of determining the spatial quality identification features, so that the spatial quality identification features can better match the characteristics of the human ear perception domain.

[0108] In one embodiment, at least one of the overall feature distance, the spatial feature distance and the objective feature distance of the difference information can be obtained by step 204. Figure 4 The steps shown achieve:

[0109] Step 401: Determine the overall feature distance between the reference audio and the target audio according to the first spectrum feature, the second spectrum feature, the first aggregate feature, and the second aggregate feature.

[0110] Among them, by determining the distance between the first reference spectral feature of the reference audio and the first target spectral feature of the target audio, the distance between the second reference spectral feature of the reference human ear perception domain audio and the second target spectral feature of the target human ear perception domain audio, determining the distance between the first reference aggregate feature of the reference audio and the first target aggregate feature of the target audio, and the distance between the second reference aggregate feature of the reference human ear perception domain audio and the second target aggregate feature of the target human ear perception domain audio, these distances are summed up and the summed results are averaged to obtain the overall feature distance.

[0111] Step 402: Determine the objective feature distance between the reference audio and the target audio according to the first spectrum feature, the first aggregate feature, and the objective distortion quality assessment identification feature.

[0112] In the same implementation as step 401, the distances between the first spectrum feature, the first aggregation feature and the objective distortion quality assessment identification feature are summed and then averaged to obtain the spatial feature distance.

[0113] Step 403: Determine the spatial feature distance between the reference audio and the target audio according to the second frequency spectrum feature, the second aggregation feature, and the spatial quality identification feature.

[0114] Among them, in the same implementation as step 401, the distances between the second spectrum feature, the second aggregate feature and the spatial quality identification feature are summed and then averaged to obtain the objective feature distance. In this way, by determining the overall feature distance between the reference audio and the target audio, the objective feature distance in the objective distortion quality evaluation and the spatial feature distance in the spatial quality, these feature distances are combined as the difference information between the reference audio and the target audio, and the spatial feature distance takes into account the characteristics of the human ear perception domain, so that the difference information can not only more fully and accurately reflect the gap between the reference audio and the target audio, but also enable the final evaluation result to be related to the human ear perception characteristics.

[0115] After determining the difference information, the audio quality of the target audio relative to the reference audio is obtained through multiple feature distances in the difference information, thereby obtaining an evaluation result of the target audio, which can be achieved through the following process:

[0116] First, based on the overall feature distance in the difference information, the overall quality of the target audio relative to the reference audio is determined.

[0117] In some possible implementations, a mapping relationship table between feature distances and quality scores is first obtained, and then the quality score corresponding to the comprehensive feature distance is determined according to the mapping relationship table, so as to obtain the overall quality.

[0118] Secondly, based on the spatial feature distance, the spatial quality of the target audio relative to the reference audio is determined.

[0119] In some possible implementations, the corresponding quality score is determined in the mapping relationship table according to the spatial feature distance, that is, the spatial quality is obtained. Since the audio features of the human ear perception domain are added when determining the spatial feature distance, the evaluation of the spatial quality of the target audio can be more consistent with the subjective perception characteristics of the human ear.

[0120] Thirdly, based on the objective feature distance, an objective distortion quality evaluation of the target audio relative to the reference audio is determined.

[0121] In some possible implementations, the corresponding quality score is determined in the mapping relationship table according to the objective feature distance, so as to obtain the objective distortion quality evaluation.

[0122] Finally, the overall quality, the spatial quality and the objective distortion quality evaluation are integrated to obtain the evaluation result.

[0123] In some possible implementations, the overall quality, the spatial quality, and the objective distortion quality evaluation are spliced ​​together to form an array, that is, the evaluation result of the target audio is obtained; in this way, the evaluation result can reflect the quality of the target audio from multiple aspects of the overall quality, spatial quality, and objective distortion quality evaluation. In this way, by combining the overall quality, spatial quality, and objective distortion quality evaluation of the target audio relative to the reference audio, the evaluation result can not only accurately reflect the quality of the target audio but also improve the degree of relevance to the human ear perception domain.

[0124] In one embodiment, the reference audio and the reference human ear perception domain audio are subjected to time-frequency transformation to obtain spectrum information that can be input into the evaluation network. The process is as follows: Figure 5 As shown, this can be achieved by following the steps below:

[0125] Step 501: Perform time-frequency transformation on the reference audio and the reference human ear perception domain audio to obtain first spectrum information.

[0126] The reference audio and the reference human ear perception domain audio are converted from the time domain to the frequency domain to obtain first spectrum information, which includes the amplitude spectrum and phase spectrum of the reference audio, and the amplitude spectrum and phase spectrum of the reference human ear perception domain audio.

[0127] In some possible implementations, a short-time Fourier transform is performed on the reference audio and the reference human ear perception domain audio to obtain a time-frequency spectrum, and then an amplitude spectrum and a phase spectrum are extracted from the obtained time-frequency spectrum to obtain first spectrum information. In this way, in the process of time-frequency transformation, the original time-domain audio data and the audio data in the human ear perception domain are combined together, so that the audio data in the human ear perception domain is fully considered in the obtained spectrum information, which facilitates the subsequent combination of the audio data in the human ear perception domain and the original time-domain audio data to jointly evaluate the evaluation results of the target audio.

[0128] Exemplarily, first, the reference audio and the reference human ear perception domain audio are converted to the complex domain respectively to obtain the time spectrum of the reference audio in the complex domain and the time spectrum of the reference human ear perception domain audio in the complex domain; then, the amplitude spectrum and phase spectrum of the time spectrum are extracted; finally, the amplitude spectrum and phase spectrum corresponding to the reference audio, and the amplitude spectrum and phase spectrum corresponding to the reference human ear perception domain audio are stacked to obtain the first spectrum information. In this way, the phase spectrum and amplitude spectrum of the human ear perception domain are fully introduced into the first spectrum information, which can be beneficial to the subsequent audio evaluation process.

[0129] Step 502: Perform time-frequency transformation on the target audio and the target human ear perception domain audio respectively to obtain second spectrum information.

[0130] Among them, the target audio and the target human ear perception domain audio are converted from the time domain to the frequency domain, thereby obtaining the second spectrum information, the amplitude spectrum and phase spectrum of the target audio, and the amplitude spectrum and phase spectrum of the target audio in the human ear perception domain.

[0131] In some possible implementations, a short-time Fourier transform is performed on the target audio and the target human ear perception domain audio to obtain a time-frequency spectrum, and then an amplitude spectrum and a phase spectrum are extracted from the obtained time-frequency spectrum to obtain second frequency spectrum information.

[0132] By performing complex domain conversion on the target audio and the target human ear perception domain audio, and stacking the extracted amplitude spectrum and phase spectrum, the second spectrum information containing the human ear perception domain audio can be obtained; that is, the above step 502 can be implemented by steps 521 to 523 (not shown in the figure):

[0133] Step 521 , performing complex domain conversion on the target audio and the target human ear perception domain audio to obtain a time-frequency spectrum of the target audio in the complex domain and a time-frequency spectrum of the target human ear perception domain audio in the complex domain.

[0134] Among them, the target audio and the target human ear perception domain audio both have left channel audio and right channel audio when saved. The time spectrum is the spectrum of the audio data in the complex domain. The time spectrum of the target audio in the complex domain includes: the time spectrum of the left channel in the complex domain and the time spectrum of the right channel in the complex domain of the target audio. The time spectrum of the target human ear perception domain audio in the complex domain includes: the time spectrum of the left channel in the complex domain and the time spectrum of the right channel in the complex domain of the target human ear perception domain audio.

[0135] In some possible implementations, the left and right channels of the target audio and the left and right channels of the target human ear perception domain audio are short-time Fourier transformed to achieve complex domain conversion of the target audio and the target human ear perception domain audio. For example, a length of 512 points and a jump length of 50% (256 points) are used, and a Hamming window with a length of 512 points is used to perform a short-time Fourier transform on the audio data.

[0136] Step 522: extract the amplitude spectrum and phase spectrum of the time-frequency spectrum.

[0137] Among them, the amplitude spectrum and phase spectrum are extracted from the time-frequency spectrum in the complex domain of the left and right channels respectively, so as to obtain the target audio left channel amplitude spectrum, target audio right channel amplitude spectrum, target audio left channel phase spectrum, target audio right channel phase spectrum, as well as the left channel amplitude spectrum of the sound to be evaluated in the human ear perception domain, the right channel amplitude spectrum of the target human ear perception domain audio, the left channel phase spectrum of the target human ear perception domain audio, and the right channel phase spectrum of the target human ear perception domain audio.

[0138] Step 523: stack the amplitude spectrum and phase spectrum corresponding to the target audio, and the amplitude spectrum and phase spectrum corresponding to the target human ear perception domain audio to obtain the second frequency spectrum information.

[0139] Among them, according to the target audio left channel amplitude spectrum, the target audio right channel amplitude spectrum, the target audio left channel phase spectrum, the target audio right channel phase spectrum, the target human ear perception domain audio left channel amplitude spectrum, the target human ear perception domain audio right channel amplitude spectrum, the target human ear perception domain audio left channel phase spectrum, the target human ear perception domain audio right channel phase spectrum, this order is stacked on the channel latitude to obtain a matrix composed of amplitude spectrum and phase spectrum, that is, the second spectrum information. In this way, by performing complex domain conversion on the original time domain target audio and the target human ear perception domain audio respectively, the respective time spectrum can be obtained; and the phase spectrum and amplitude spectrum extracted from the time spectrum are stacked to obtain the second spectrum information that can reflect both the phase spectrum and amplitude spectrum of the original time domain and the phase spectrum and amplitude spectrum of the human ear perception domain. In this way, the second spectrum information fully introduces the phase spectrum and amplitude spectrum of the human ear perception domain, which can be beneficial to the subsequent audio evaluation to combine the human ear perception domain, thereby improving the correlation between the evaluation results and the main evaluation of the human ear.

[0140] Step 503: input the first spectrum information and the second spectrum information into the evaluation network, so as to use the evaluation network to perform evaluation processing on the original audio and the audio in the human ear perception domain.

[0141] After the first spectrum information and the second spectrum information are obtained, the first spectrum information and the second spectrum information are processed through an evaluation network to determine difference information between the reference audio and the target audio.

[0142] The first spectrum information and the second spectrum information are split according to channels through the evaluation network to split into spectrum information of the original time domain (i.e., phase spectrum and amplitude spectrum of the original time domain) and spectrum information of the human ear perception domain (phase spectrum and amplitude spectrum of the human ear perception domain); then, feature extraction is performed on the phase spectrum and amplitude spectrum of the original time domain respectively through the first feature extraction layer to obtain the first spectrum feature; feature extraction is performed on the phase spectrum and amplitude spectrum of the human ear perception domain through the second feature extraction layer to obtain the second spectrum feature.

[0143] Since the stacking order of the amplitude spectrum and the phase spectrum in the first spectrum information is: reference audio left channel amplitude spectrum, reference audio right channel amplitude spectrum, reference audio left channel phase spectrum, reference audio right channel phase spectrum, reference human ear perception domain audio left channel amplitude spectrum, reference human ear perception domain audio right channel amplitude spectrum, reference human ear perception domain audio left channel phase spectrum, reference human ear perception domain audio right channel phase spectrum; the stacking order of the amplitude spectrum and the phase spectrum in the second spectrum information is: target audio left channel amplitude spectrum, target audio right channel amplitude spectrum, target audio left channel phase spectrum, target audio right channel phase spectrum, target human ear perception domain audio left channel amplitude spectrum, target human ear perception domain audio right channel amplitude spectrum, target human ear perception domain audio left channel phase spectrum, target human ear perception domain audio right channel phase spectrum; therefore, in the evaluation network, the first 4 channels in the first spectrum information and the second spectrum information (latitude: [8, f, 257])0 are used as the spectrum information of the original time domain, and the last 4 channels are used as the spectrum information of the human ear perception domain. In this way, after the first spectrum information and the second spectrum information are split, they are input into the corresponding feature extraction layer, which can make the features output by the feature extraction layer more accurate.

[0144] In one embodiment, the training process of the evaluation network is as follows: Figure 6 As shown, it can be achieved through steps 601 to 604:

[0145] Step 601, obtaining sample audio.

[0146] The sample audio includes at least three samples, such as an anchor sample, a positive sample and a negative sample.

[0147] In some possible implementations, the sample audio is constructed in a triplet manner to make the sample audio richer, that is, the above step 601 can be implemented by the following process:

[0148] First, at least three preset audios are obtained from a preset audio database.

[0149] The preset audio database includes multiple audio data with audio quality labels. At least three preset audios are randomly selected from the preset audio database, and energy is normalized to the same signal-to-noise ratio. For example, three audio data are randomly sampled from the preset audio database, and the energy of the three audio data is normalized to -25 decibels (dB).

[0150] Secondly, according to different preset signal-to-noise ratios, preset noise is added to the at least three preset audios to obtain at least three processed audios.

[0151] The preset noise may be a piece of noise data randomly sampled from a noise library. Three signal-to-noise ratios are sampled within a preset signal-to-noise ratio range, for example, three signal-to-noise ratios of different sizes are sampled within a signal-to-noise ratio range of [-20dB, +30dB] to obtain three different preset signal-to-noise ratios. Afterwards, the preset noise is added to each day's audio data according to the size of the preset signal-to-noise ratio, so that the signal-to-noise ratio of the processed audio data after the noise is added is the preset signal-to-noise ratio.

[0152] Again, among the at least three processed audios, a positive sample, an anchor sample with a maximum signal-to-noise ratio, and a negative sample with a minimum signal-to-noise ratio are determined.

[0153] Among them, taking three processed audio data as an example, the three processed audio data are arranged from large to small according to the signal-to-noise ratio, the processed audio data with the largest signal-to-noise ratio is used as the anchor sample, the processed audio data with the smallest signal-to-noise ratio is used as the negative sample, and the remaining processed audio data is used as the positive sample.

[0154] Finally, the anchor sample, the positive sample and the negative sample are determined as the sample audio.

[0155] In this way, after obtaining anchor samples, positive samples and negative samples according to different signal-to-noise ratios, the anchor samples, positive samples and negative samples are formed into triplets as sample audio, which enables the evaluation network to be trained to learn feature representations with better discrimination, making samples of the same category closer and samples of different categories more dispersed.

[0156] Step 602: Perform human ear perception processing on the sample audio to obtain sample audio in the human ear perception domain.

[0157] Among them, the anchor samples, positive samples and negative samples in the sample audio are respectively converted from the original time domain to the human ear perception domain to simulate the perception and processing process of the human ear on these sample data, and the human ear perception domain sample audio is obtained, that is, the anchor samples of the human ear perception domain, the positive samples of the human ear perception domain and the negative samples of the human ear perception domain.

[0158] Step 603: Use the to-be-trained evaluation network to perform evaluation processing on the sample audio and the sample audio in the human ear perception domain to obtain an objective distortion quality evaluation loss and a spatial quality loss of the sample audio.

[0159] Among them, after the sample spectrum information is obtained by performing time-frequency transformation on the sample audio and the sample audio in the human ear perception domain, it is input into the evaluation network to be trained. The objective distortion quality evaluation loss of the sample audio is used to represent the difference between the objective distortion quality evaluation predicted by the sample audio and the true value of the sample audio. The spatial quality loss is used to represent the difference between the spatial quality predicted by the sample audio and the true value of the sample audio.

[0160] For example, by performing short-time Fourier transform on the sample audio and the sample audio in the human ear perception domain, and extracting the amplitude spectrum and phase spectrum from the obtained time-frequency in the complex domain, the sample spectrum information is obtained; thus, the sample spectrum information includes: the amplitude spectrum and phase spectrum of the sample audio in the original time domain, and the amplitude spectrum and phase spectrum of the sample audio in the human ear perception domain. The amplitude spectrum and phase spectrum of the sample audio in the original time domain, and the amplitude spectrum and phase spectrum of the sample audio in the human ear perception domain are input into the network to be trained and evaluated.

[0161] Step 604: According to the objective distortion quality evaluation loss and the spatial quality loss, the network parameters of the evaluation network to be trained are adjusted to obtain an evaluation network.

[0162] The network parameters of the evaluation network to be trained include: weights, learning rates and other parameters of the evaluation network to be trained. The network parameters of the evaluation network to be trained are iterated cyclically by combining the objective distortion quality evaluation loss and the spatial quality loss, so that the function values ​​corresponding to the output objective distortion quality evaluation loss and the spatial quality loss converge, thereby obtaining the evaluation network.

[0163] In an embodiment of the present application, during the training process of the evaluation network to be trained, by converting the sample audio to the human ear perception domain, the sample spectrum information can be integrated with the audio data of the human ear perception domain. Finally, the objective distortion quality evaluation loss and spatial quality loss of the sample audio are determined through the sample spectrum information, and based on the objective distortion quality evaluation loss and the spatial quality loss, the network parameters of the evaluation network to be trained are adjusted to obtain the evaluation network. In this way, since the sample spectrum information comprehensively considers the sample audio of the human ear perception domain and the sample audio of the original time domain, the evaluation network to be trained can learn the characteristics of the human ear perception domain, thereby improving the performance of the evaluation network.

[0164] In some possible implementations, the objective distortion quality evaluation loss and the spatial quality loss are combined as the overall loss to adjust the network parameters of the evaluation network to be trained, that is, the above step 604 can be implemented by the following steps 641 and 642 (not shown in the figure):

[0165] Step 641: determine the overall loss based on the objective distortion quality assessment loss and the spatial quality loss.

[0166] Among them, the objective distortion quality evaluation loss and the spatial quality loss are summed and then averaged to obtain the overall loss.

[0167] Step 642: Use the overall loss to adjust the network parameters of the evaluation network to be trained to obtain the evaluation network.

[0168] Among them, the network parameters of the evaluation network to be trained are iterated cyclically through the overall loss so that the total loss output by the evaluation network converges. In this way, since the sample audio in the human ear perception domain is learned in the spatial loss, the evaluation network to be trained is trained by taking the average of the objective distortion quality evaluation loss and the spatial quality loss as the overall loss, so that the evaluation network can fully learn the perceptual characteristics of the human ear perception domain for audio data, thereby making the evaluation results of the target audio predicted by the evaluation network more in line with the perceptual characteristics of the human ear.

[0169] In one embodiment, the evaluation network to be trained includes: an objective distortion quality evaluation subnetwork and a spatial quality evaluation subnetwork, and the objective distortion quality evaluation loss and the spatial quality loss are determined by performing feature extraction, time domain aggregation and other evaluation processing on the sample spectrum information of the anchor samples, positive samples and negative samples, that is, the above step 603 can be implemented by the following steps 631 to 633:

[0170] Step 631: Use the objective distortion quality evaluation sub-network in the to-be-trained evaluation network to perform evaluation processing on the anchor sample, the positive sample, and the negative sample to obtain the objective distortion quality evaluation loss of the sample audio.

[0171] The first sample spectrum information of the anchor sample, the first sample spectrum information of the positive sample and the first sample spectrum information of the negative sample are input into the objective distortion quality assessment sub-network, and the input spectrum information is processed by the first feature extraction layer, the first time domain aggregation layer and the objective distortion quality assessment head in the objective distortion quality assessment sub-network to obtain the objective distortion quality assessment loss of the sample audio. In this way, the objective distortion quality assessment loss is independent of the human ear perception domain and can accurately learn the sample features of the original time domain.

[0172] Step 632: Use the spatial quality evaluation sub-network in the evaluation network to be trained to perform evaluation processing on the anchor samples in the human ear perception domain.

[0173] Among them, the second sample spectrum information of the anchor sample in the human ear perception domain is input into the spatial quality assessment subnetwork, and the input spectrum information is processed through the second feature extraction layer and the second time domain aggregation layer to obtain the output of the second time domain aggregation layer.

[0174] Step 633: Determine the spatial quality loss based on the evaluation processing result of the anchor sample in the human ear perception field and the evaluation processing result of the anchor sample.

[0175] Among them, the output of the second time-domain aggregation layer and the evaluation processing result of the anchor sample output by the objective distortion quality evaluation sub-network are weighted and summed, and then input into the spatial quality head to obtain the spatial quality loss. In this way, the spatial quality loss is obtained by inputting only the anchor samples in the human ear perception domain into the spatial quality head of the evaluation network to be trained; in this way, the spatial quality loss is determined by using the sample aggregation features of the anchor samples with a large signal-to-noise ratio, which can improve the accuracy of the evaluation network to be trained in predicting the spatial quality.

[0176] In some possible implementations, the objective distortion quality evaluation loss is determined by analyzing the depth feature distance between the anchor sample and the positive sample, and the depth feature distance between the positive sample and the negative sample. That is, the above step 631 can be implemented by the following process:

[0177] First, a first sample depth feature distance between the sample feature of the original time domain corresponding to the anchor sample and the sample feature of the original time domain corresponding to the positive sample is determined.

[0178] The first sample depth feature distance may be obtained by determining the 1-norm of the difference between the sample feature in the original time domain and the sample feature in the original time domain corresponding to the positive sample.

[0179] Secondly, a second sample depth feature distance between the sample feature of the original time domain corresponding to the anchor sample and the sample feature of the original time domain corresponding to the negative sample is determined.

[0180] The second sample depth feature distance may be obtained by determining the 1-norm of the difference between the sample feature of the original time domain corresponding to the anchor sample and the sample feature of the original time domain corresponding to the negative sample.

[0181] Finally, the objective distortion quality assessment loss is determined based on the first sample depth feature distance and the gap between the first sample depth feature distance.

[0182] Among them, after determining the difference between the first sample depth feature distance and the first sample depth feature distance, the difference is summed with the preset spacing parameter, and the maximum value between the summation result and 0 is used as the objective distortion quality evaluation loss. In order to prevent the network from outputting a spacing parameter with all 0 results, the initial value of the preset spacing parameter is 0.5, and gradually increases to 1.5 with the increase of training arguments. Setting the value of the preset spacing parameter in this way can make the gap between different data larger and larger during the learning process, thereby achieving better clustering effect. In this way, by taking the maximum value between the gap between the first sample depth feature distance and the second sample depth feature distance and the constant value 0 as the objective distortion quality evaluation loss, the objective distortion quality evaluation loss can more fully learn the differences between different samples, thereby improving the clustering effect of the evaluation network to be trained.

[0183] In some possible implementations, the spatial quality loss can be obtained by classifying and predicting the azimuth angles of the anchor samples, which can be achieved by the following process:

[0184] First, according to the evaluation processing result of the anchor sample in the human ear perception domain, the prediction probability distribution of the predicted azimuth angle of the anchor sample in the human ear perception domain in a preset angle interval set is predicted.

[0185] After obtaining the characteristics of the human ear perception domain corresponding to the anchor sample, the horizontal angle and elevation angle of the characteristics of the human ear perception domain of the anchor sample in the training evaluation network are predicted in polar coordinates, thereby obtaining the predicted azimuth of the anchor sample. And the predicted azimuth angles falling into the preset angle interval set are classified to obtain the predicted probability distribution of the predicted azimuth angle in each preset angle interval.

[0186] Secondly, according to the true azimuth of the evaluation processing result of the anchor sample and the predicted azimuth, the angle difference information is determined.

[0187] Among them, the angle difference information is the difference between the polar coordinates representing the true azimuth and the polar coordinates representing the predicted azimuth, which can reflect the angle difference between the vector corresponding to the true azimuth and the vector corresponding to the predicted azimuth. Since the azimuth includes the elevation angle and the horizontal angle, the angle difference information can be determined by the true horizontal angle of the sample feature in the original time domain and the predicted horizontal angle in the predicted azimuth, as well as the true elevation angle of the sample feature in the original time domain and the predicted elevation angle in the predicted azimuth. For example, the great spherical distance between the true value of the sample feature in the original time domain and the predicted value of the feature in the human ear perception domain is determined.

[0188] Again, distribution difference information is determined based on the true value distribution of the true value azimuth in a preset angle interval set and the predicted probability distribution.

[0189] Among them, after obtaining the predicted probability distribution of the predicted azimuth in each preset angle interval, the average earth moving distance between the true value distribution and the predicted probability distribution is determined to represent the gap between the two probability densities, that is, to obtain distribution difference information.

[0190] Finally, the spatial quality loss is determined according to the angle difference information and the distribution difference information.

[0191] The angle difference information and the distribution difference information are summed to obtain the spatial quality loss. In this way, by determining the predicted probability distribution of the predicted azimuth of the anchor sample in the preset angle interval set, the distribution difference information between the true value distribution and the predicted probability distribution can be analyzed. Therefore, the angle difference information between the true value azimuth and the predicted azimuth is combined with the distribution difference information, and the characteristics of the human ear perception domain can be introduced into the spatial quality loss, so that the spatial quality loss can learn the characteristics of the human ear perception domain, and then the evaluation network can learn the characteristics of the human ear perception domain, and improve the evaluation results output by the evaluation network to be more in line with the human ear perception characteristics.

[0192] In one embodiment, the audio data evaluation method provided in the embodiment of the present application obtains spatial audio in the human ear perception domain as auxiliary information by performing human ear perception processing on the original spatial audio, which is used to assist in the spatial quality prediction of the spatial audio, so as to improve the correlation between the model evaluation results and human subjective perception. The original spatial audio and the spatial audio in the human ear perception domain are first subjected to short-time Fourier transform respectively, the amplitude spectrum and the phase spectrum are extracted, and the stacked amplitude / phase spectrum matrix is ​​used as the input of the neural network. Afterwards, the amplitude / phase spectrum matrix of the spatial audio in the original and human ear perception domains is simultaneously input into the deep neural network for feature extraction and time domain aggregation, and the corresponding embedding layer is calculated. The embedding layer of the original spatial audio is directly input into the objective distortion quality evaluation task head for objective distortion quality evaluation prediction, because the objective distortion quality evaluation task is an objective task that is unrelated to the human ear perception characteristics. Accordingly, for the spatial quality task head related to the human ear perception characteristics, we use the embedding layer of the spatial audio in the human ear perception domain to assist the embedding layer of the original spatial audio in a learnable weighted manner to predict the spatial quality, so as to obtain a prediction result that is more in line with the human subjective perception characteristics. Finally, we calculate the depth feature distance between the reference audio and the target audio embedding layer to obtain the quality score of the spatial audio, including the objective distortion quality evaluation score, the spatial quality score, and the overall quality score. The following is a detailed description of each part:

[0193] Through multiple filtering operations, the human ear's perception of sound is simulated to map the original spatial audio to the human ear's perception domain, such as Figure 7 As shown, this can be achieved by following the steps below:

[0194] Step 701 , input reference audio 71 and target audio 72 .

[0195] Step 702: filter the input spatial audio to obtain filtered audio.

[0196] Here, a second-order Butterworth bandpass filter with a bandpass frequency range of [500 Hz, 2000 Hz] is used to filter the input spatial audio to simulate the frequency response of the human outer ear and middle ear and the perception and processing of sound.

[0197] Step 703, filtering the filtered audio to simulate the perception and processing of sound by the human basilar membrane.

[0198] A set of fourth-order Gammatone filter groups is used to filter the filtered audio outputted from step 702 to simulate the perception and processing of sound by the human basilar membrane. The sampling rate of the Gammatone filter group is equivalent to the sampling rate of the spatial audio, the lowest cutoff frequency is 50 Hz, the highest cutoff frequency is 14000 Hz, the base frequency is 700 Hz, the number of Gammatone filters per equivalent rectangular bandwidth is 1, the order is 4, and the bandwidth factor is 1.

[0199] Step 704, processing the output of step 703 to simulate the cochlear compression process of the human cochlea on sound.

[0200] Here, the output of step 703 is processed using a transient compression with an exponent of 0.4 to simulate the cochlear compression processing process of the human cochlea on sound.

[0201] Step 705, low-pass filtering the output of step 704 to simulate the electromechanical transmission process of human cochlear hair cells.

[0202] The output of step 704 is filtered using a half-wave correction and a fifth-order Butterworth low-pass filter with a cutoff frequency of 770 Hz to simulate the electromechanical transmission process of human cochlear hair cells. The output of step 705 is the spatial audio mapped to the human ear perception domain, thereby obtaining the reference audio 73 and the target audio 74 in the human ear perception domain.

[0203] After obtaining the reference audio 73 in the human ear perception domain and the target audio 74 in the human ear perception domain, the two-tuples of <reference audio, reference audio in the human ear perception domain> and <target audio, target audio in the human ear perception domain> are respectively subjected to short-time Fourier transform to extract the amplitude spectrum and the phase spectrum, which are stacked to form an amplitude / phase spectrum matrix as the input of the deep neural network, as follows Figure 8 As shown:

[0204] Step 801 , performing short-time Fourier transform on the input original audio to extract the time spectrum, and performing short-time Fourier transform on the audio in the human ear perception domain to extract the time spectrum.

[0205] The input spatial audio (i.e., reference audio 71, reference audio 73 in the human ear perception domain, target audio 72, and target audio 74 in the human ear perception domain) is subjected to short-time Fourier transform using a length of 512 points, a jump length of 50% (256 points), and a Hamming window of 512 points to obtain a complex time-frequency spectrum 811 and a complex time-frequency spectrum in the human ear perception domain 812. The short-time Fourier transform uses a length of 512 points, a jump length of 50% (256 points), and a Hamming window of 512 points to reduce spectrum leakage.

[0206] Step 802 , extracting the amplitude spectrum and the phase spectrum from the complex time-frequency spectrum 811 ; and extracting the amplitude spectrum and the phase spectrum from the complex time-frequency spectrum 812 in the human ear perception domain.

[0207] Step 803: stack the amplitude spectrum and the phase spectrum in the channel dimension to obtain an input matrix.

[0208] For <reference audio, reference audio in human ear perception domain>, we stack the amplitude spectrum of the left ear of the reference audio, the amplitude spectrum of the right ear of the reference audio, the phase spectrum of the left ear of the reference audio, the phase spectrum of the right ear of the reference audio, the amplitude spectrum of the left ear of the reference audio in the human ear perception domain, the amplitude spectrum of the right ear of the reference audio in the human ear perception domain, the amplitude spectrum of the left ear of the reference audio in the human ear perception domain, the phase spectrum of the left ear of the reference audio in the human ear perception domain, the phase spectrum of the right ear of the reference audio in the human ear perception domain] in the channel dimension to obtain an input matrix of dimension [c=8,f,T=257], i.e., reference audio amplitude spectrum / phase spectrum 813. Where c represents 8 different channels, f represents frame (frame, proportional to the duration of the input audio), and T represents time resolution (frequency), which is half the number of points in the short-time Fourier transform. For <target audio, target audio in human ear perception domain>, the amplitude and phase spectra are stacked in the same stacking order as the reference audio to obtain an input matrix of dimension [8,f,257], i.e., target audio amplitude spectrum / phase spectrum 814.

[0209] In step 804, the input matrix is ​​input to the neural network to determine the quality of the target audio relative to the reference audio.

[0210] In the embodiment of the present application, the calculated amplitude / phase spectrum matrix of the reference audio and the target audio is input into the deep neural network calculation embedding layer. The training process, loss function and prediction process of the deep neural network are respectively introduced below:

[0211] Regarding the training process of the neural network, the embodiment of the present application is based on a multi-task learning framework and uses a triplet approach to train the neural network, such as Fig. 9 As shown:

[0212] First, the stacked magnitude and phase spectrum matrices are input.

[0213] Among them, three spatial audios are randomly sampled from the spatial audio data set and the energy is normalized to -25dB, and one noise data is randomly sampled from the noise data set. Three signal-to-noise ratios are uniformly sampled on [-20dB, +30dB] and the signal-to-noise ratio A>P>N is guaranteed to meet the condition of |AP|<|AN| in the triple training. After that, the three different spatial audios with energy normalized to the same distribution are denoised with the same noise at three different signal-to-noise ratio levels A, P, and N to construct a <anchor sample, positive sample, negative sample> triple. After that, the anchor sample a (anchor), the positive sample p (positive), and the negative sample n (negative) are processed by human ear perception and short-time Fourier transform, and the amplitude / phase spectrum matrix is ​​extracted and input into the neural network to calculate the embedding layer. That is, the amplitude / phase spectrum matrix 91 of the anchor sample a, the amplitude / phase spectrum matrix 93 of the negative sample n, and the amplitude / phase spectrum matrix 92 of the positive sample p are input into the neural network to calculate the embedding layer through the input module 94. The first four channels (0 to 3) input the amplitude and phase spectrum matrix 95 of the original audio, and the last four channels (4 to 7) input the amplitude and phase spectrum 96 of the human ear perception domain.

[0214] Secondly, the amplitude / phase spectrum matrices of the anchor sample, negative sample, and positive sample are respectively passed through the feature extraction layer 901, the time domain aggregation layer 902, the objective distortion quality evaluation head 903, and the spatial quality head 904 to calculate the embedding layer and the prediction results. And the objective distortion quality evaluation loss 905 and the spatial quality loss 906 are respectively determined according to the embedding layer and the prediction results, and the overall quality loss 907 is calculated according to the objective distortion quality evaluation loss 905 and the spatial quality loss 906 for back propagation.

[0215] In a deep neural network, feature extraction layers are used to learn the features of a signal while maintaining its temporal structure. The feature extraction layer consists of 6 layers of Inception-style feature extraction blocks stacked together with increasing number of channels. After the 6th feature extraction block, 1*1 convolutions are used on the channels to compress the channel dimension and on the frequency dimension to compress the frequency dimension. The input dimension of the feature extraction block is [B, 4, f, 257] and the output dimension is [B, f, 64]. Features are extracted for both the original spatial audio and the spatial audio in the human ear perception domain, and both use feature extraction layers with shared parameters. The output of the feature extraction block is used as the input to the time domain aggregation layer. In the feature extraction layer, the number of inputs, outputs and convolution kernels used are shown in Table 1, where the “parameter column 1” (such as [[1], [2, 4], [1, 2], [3, 1]) represents [[the number of output channels of the first 1*1 convolution] [the number of output channels of the second 1*1 convolution, the number of channels of 3*3 convolution] [the number of output channels of the third 1*1 convolution, the number of channels of 5*5 convolution] [the maximum pooling kernel size, the maximum number of pooling output channels]] respectively.

[0216] Table 1 Parameter settings in the feature extraction layer structure

[0217]

[0218] The structure of the feature extraction layer is as follows: Fig.10 As shown in the feature extraction layer 1000 in , it includes: a pooling layer 1001 with a step size of 4, an aggregation layer 1002, and 4 batch normalization and multiple convolution layers.

[0219] In the deep neural network, the temporal aggregation layer is used to learn long-term dependencies from the frame-level representation generated by the feature extraction layer. The temporal aggregation block is composed of 4 layers of temporal convolutional networks, each of which consists of two 1*3 convolution kernels, weight normalization and residual modules. Causal convolution is used to extract temporal information, and dilated convolution is used to continuously expand the receptive field. In the four-layer temporal convolutional network, the number of input channels first increases from 64 to 128 and 256, and then decreases from 256 to 128 and 64 and outputs. Therefore, the input and output dimensions of the temporal aggregation layer are [B, f, 64]. The original spatial audio and the spatial audio in the human ear perception domain are temporally aggregated respectively, and the two use the temporal aggregation layer with shared parameters. The output of the temporal aggregation layer will be used as the input of the objective distortion quality assessment head and the spatial quality head. The structure is shown in Table 2:

[0220] Table 2 Time domain aggregation layer structure

[0221]

[0222] Here, the structure of the time domain aggregation layer is as follows Fig.10As shown in the time domain aggregation layer 1010 in FIG. 1 , it includes: two 1*3 convolution kernels and two weight normalizations 1011 and 1012 .

[0223] In a deep neural network, an objective distortion quality measurement task head is used to learn an objective distortion quality measurement representation of spatial audio from the embedding layer of the temporal aggregation layer. Fig.10 As shown in the objective distortion quality assessment head 1020 in , the objective distortion quality assessment head 1020 consists of two layers of stacked 1*3 convolution and weight normalization 1021 and 1022, with a step size of 1 and a padding of 1 to ensure that the input and output dimensions remain unchanged, followed by two fully connected layers that do not change the channel dimension. Since the objective distortion quality assessment task is unrelated to human perception characteristics, the objective distortion quality assessment head directly uses the time domain aggregation layer embedding layer of the original spatial audio as input. The input and output dimensions are [B, f, 64], and the structure is shown in Table 3:

[0224] Table 3 Objective distortion quality evaluation task head structure

[0225]

[0226] In the neural network, the spatial quality task head is used to learn the spatial quality representation of spatial audio from the embedding layer of the time domain aggregation layer. For the prediction task of spatial quality, that is, the prediction task of azimuth, the interval [0°, 360°] is divided into 50 equally spaced sectors and regarded as a 50-classification task. The spatial quality task head is as follows Fig.10 As shown in the spatial quality head 1030 in , it consists of two layers of stacked 1*3 convolution and weight normalization 1031 and 1032, as well as two layers of fully connected layers and softmax activation function. Its input dimension is [B, f, 64] and its output dimension is [B, f, 50], which represents the probability distribution of the azimuth angle of the spatial audio in each frame in a certain sector. Since the spatial quality task head is strongly related to human subjective perception, the time domain aggregation layer embedding layer of the spatial audio in the human ear perception domain is used to assist the prediction of the spatial quality task head. Specifically, a learnable weight is used to weight the time domain aggregation layer embedding layer of the original spatial audio and the time domain aggregation layer embedding layer of the spatial audio in the human ear perception domain, and the input is given to the spatial quality task head for prediction. The perceptual characteristics of the human ear and humans are used to assist in the objective distortion quality evaluation prediction of the spatial audio, as shown in Table 4:

[0227] Table 4 Space quality task header structure

[0228]

[0229] Next, calculate the objective distortion quality evaluation loss, that is, the triplet loss function, as shown in formula (1):

[0230] loss_lq=max{0,D1(X a ,X p )-D1(X a ,X n )+δ} (1);

[0231] Among them, D1(X a ,X p ) is the deep feature distance of the deep feature stack of the full feature activation stack between the positive sample and the anchor sample. D1(X a ,X n ) is the deep feature distance of the deep feature stack of the full feature activation stack between the positive sample and the negative sample, as shown in formula (2):

[0232]

[0233] Here, T l ,C l ,B l are the temporal resolution, frequency band, and number of channels, respectively. In the model training, the deep feature stack involved in the objective distortion quality assessment loss is the feature extraction layer, the temporal aggregation layer, and the embedding layer output by the objective distortion quality assessment task head. δ is the spacing parameter to prevent the network from outputting all zero results. It is initially 0.5 and gradually increases to 1.5 as the number of training rounds increases.

[0234] Here, the objective distortion quality assessment task head is as follows Fig.10 As shown in the objective distortion quality assessment head 1020 in FIG, it includes: two 1*3 convolution kernels and two weight normalization layers 1021 and 1022. The objective distortion quality assessment task head is as follows Fig.10 As shown in the spatial quality head 1030, it includes: two 1*3 convolution kernels and two weight normalization layers 1031 and 1032.

[0235] Again, calculate the spatial quality loss. Here, the spatial positioning task is regarded as a classification task, and the angle of [0°, 360°] is mapped to 50 equidistant sectors such as [0, 49]. The haversine distance that reflects continuous angle changes and the earth moving distance (EMD distance) that focuses on inter-class relationships are used as loss functions. Since excessive noise may affect the prediction of the objective distortion quality assessment head, the anchor spatial audio with the largest signal-to-noise ratio (lowest noise) is used to calculate the spatial quality loss. The label of the anchor sample is a, and the output of the objective distortion quality assessment head in the human ear perception domain is The loss function is shown in formula (3):

[0236]

[0237] Wherein, loss_haversine represents the big ball distance, or the half-haversine distance, as shown in formula (4):

[0238]

[0239] in, Represent the true and predicted horizontal angles, respectively, based on a and It is calculated from the median value of the sector where it is located; θ1 and θ2 represent the real and predicted elevation angles, respectively. Since the embodiment of the present application does not perform elevation angle prediction, it is considered that θ1=θ2=real elevation angle.

[0240] Here, loss_emd represents the average distance of the earth's movement, as shown in formula (5):

[0241]

[0242] Where P n Yes n The cumulative probability density distribution function, and p n is a probability distribution of a normal distribution calculated based on the correct sector subscript v in a, as shown in formula (6), is the probability distribution of network predictions The probability density distribution function of is:

[0243]

[0244] Finally, the overall loss is calculated as shown in formula (7):

[0245] loss=(loss_lq+loss_sq) / 2 (7);

[0246] The network is trained by backpropagating the loss so that it can predict objective quality attributes such as spatial quality and objective distortion quality assessment of spatial audio with the assistance of human ear perception.

[0247] In the embodiment of the present application, for the prediction process of the neural network, the amplitude / phase spectrum matrix of the input reference audio and target audio is calculated to obtain the corresponding calculation results, that is, to obtain the corresponding embedding layer for the subsequent calculation of the deep feature distance and objective score. Fig.11The network structure of the spatial audio objective evaluation model of the human ear perception characteristics shown in FIG. processes the input reference audio and target audio amplitude / phase spectrum matrices to obtain the embedding layers of the reference audio and target audio. The amplitude / phase spectrum 1101 of the reference audio and the amplitude / phase spectrum 1102 of the target audio are respectively input to the network architecture through the input module 1103. Here, the dimension of the input amplitude / phase spectrum matrix is ​​[8, f, 257], the first 4 channels [4, f, 257] are the amplitude / phase spectrum matrix 1104 of the original spatial audio, and the last 4 channels [4, f, 257] are the amplitude / phase spectrum matrix 1105 of the spatial audio in the human ear perception domain. The amplitude / phase spectrum matrix 1104 of the original spatial audio is feature extracted on the first 4 channels by the feature extraction layer 1106, and the extracted features are used as the embedding layer of the feature extraction layer and input to the time domain aggregation layer 1109. The feature extraction layer 1106 extracts features from the amplitude / phase spectrum matrix 1105 of the spatial audio in the human ear perception domain on the last four channels, and inputs the extracted features into the time domain aggregation layer 1109. The time domain aggregation layer 1109 performs convolution, weight normalization, and residual processing on the amplitude / phase spectrum matrix 1104 of the original spatial audio and the amplitude / phase spectrum matrix 1105 of the spatial audio in the human ear perception domain, respectively, and uses the two output results as the input of the objective distortion quality assessment head 1107, respectively, to obtain the output embedding layer 1110. The output results of the amplitude / phase spectrum matrix 1104 of the original spatial audio in the time domain aggregation layer 1109 and the output results of the amplitude / phase spectrum matrix 1105 of the spatial audio in the human ear perception domain in the time domain aggregation layer 1109 are weighted and combined, and the weighted output results are used as the input of the spatial quality header 1108 to obtain the output embedding layer 1110; the output results of the amplitude / phase spectrum matrix 1104 of the original spatial audio in the time domain aggregation layer 1109 and the output results of the amplitude / phase spectrum matrix 1105 of the spatial audio in the human ear perception domain in the time domain aggregation layer 1109 are spliced, and the spliced ​​output results are used as the input of the spatial quality header 1108 to obtain the output embedding layer 1110; thereby finally obtaining the reference audio embedding layer 1111 and the target audio embedding layer 1112.

[0248] In other embodiments, the input of the objective distortion quality assessment head 1107 may be the output result of the amplitude / phase spectrum matrix 1104 of the original spatial audio in the time domain aggregation layer 1109, or may be the output result of the amplitude / phase spectrum matrix 1104 of the original spatial audio in the time domain aggregation layer 1109 and the output result of the amplitude / phase spectrum matrix 1105 of the spatial audio in the human ear perception domain in the time domain aggregation layer 1109. For example, the output results of the amplitude / phase spectrum matrix 1104 of the original spatial audio in the time domain aggregation layer 1109 and the output results of the amplitude / phase spectrum matrix 1105 of the spatial audio in the human ear perception domain in the time domain aggregation layer 1109 are weighted and combined and then input into the objective distortion quality assessment head 1107; or, the output results of the amplitude / phase spectrum matrix 1104 of the original spatial audio in the time domain aggregation layer 1109 and the output results of the amplitude / phase spectrum matrix 1105 of the spatial audio in the human ear perception domain in the time domain aggregation layer 1109 are spliced ​​and then input into the objective distortion quality assessment head 1107 to further improve the accuracy of the output results of the objective distortion quality assessment head 1107.

[0249] Among them, whether it is the reference audio / target audio or the original spatial audio / spatial audio in the human ear perception domain, the same feature extraction layer, time domain aggregation layer, and multi-task head processing are used, which can be achieved through Fig.12 The process shown is implemented as:

[0250] First, the input amplitude / phase spectrum is split according to the channel into the original audio amplitude / phase spectrum and the human ear perception domain amplitude / phase spectrum.

[0251] Here, the input reference audio 1202 and target audio 1203 are processed by human ear perception processing module 1201 to obtain audio 1204 in the human ear perception domain. Afterwards, the reference audio 1202, the target audio 1203, the reference audio 1205 in the human ear perception domain, and the target audio 1206 in the human ear perception domain are input to the time-frequency transformation module 1207. The input audio is transformed by the time-frequency transformation module 1207 to obtain a complex time-frequency spectrum 1220 and a complex time-frequency spectrum 1221 in the human ear perception domain, and the amplitude spectrum and phase spectrum 1222 of these time-frequency spectrums are extracted; afterward, the extracted amplitude spectrum and phase spectrum are stacked in the channel dimension to obtain the input 1223 of the neural network, that is, the reference audio amplitude and phase spectrum matrix 1208, and the target audio amplitude and phase spectrum matrix 1209.

[0252] Secondly, the reference audio amplitude and phase spectrum matrix 1208 and the target audio amplitude and phase spectrum matrix 1209 are input into the deep neural network module 1210, the input phase spectrum and amplitude spectrum are divided into the original audio amplitude and phase spectrum 1224 and the audio amplitude and phase spectrum 1225 in the human ear perception domain, and are input into the same feature extraction layer 1230 to calculate the feature extraction embedding layer.

[0253] Again, in the deep neural network module 1210, the original audio feature extraction embedding layer and the human ear perception domain audio feature extraction embedding layer are input into the same time domain aggregation layer 1231 to calculate the time domain aggregation embedding layer.

[0254] Again, the original audio time domain aggregation embedding layer is directly input into the objective distortion quality assessment head 1232 to determine the objective distortion quality assessment task head embedding layer (objective distortion quality assessment has nothing to do with human ear perception).

[0255] Again, the original audio time domain aggregation embedding layer and the human ear perception domain audio time domain aggregation embedding layer are weighted combined with a learnable weight α, and input into the spatial quality head 1233 to calculate the spatial quality task head embedding layer (spatial quality is a task related to human ear perception, and is predicted using an audio auxiliary network in the human ear perception domain).

[0256] Finally: return the original audio feature extraction embedding layer ([B, f, 64]), the weighted combined time domain aggregation embedding layer ([B, f, 64]), the objective distortion quality evaluation embedding layer ([B, f, 64]), and the spatial quality embedding layer output ([B, f, 50]) to obtain the final output embedding layer 1234, including: the reference audio embedding layer and the target audio embedding layer, and input the calculated deep feature distance as the scoring module 1211 to calculate the subsequent deep feature distance as the objective score of the spatial audio, thereby obtaining the objective distortion quality evaluation 1212, the overall quality 1213 and the spatial quality 1214. The same network is used to calculate two different sets of embedding layers for the reference audio and the target audio.

[0257] After obtaining the embedding layers of the target audio and reference audio, the overall quality, objective distortion quality evaluation, and spatial quality score of the target audio relative to the reference audio are obtained by calculating their depth feature distance, such as Fig.13 As shown:

[0258] First, the depth feature distance 1303 between the feature extraction layer and the time domain aggregation layer embedding layer shared between the reference audio embedding layer 1301 and the target audio embedding layer 1302 is calculated as the overall quality score 1304 of the spatial audio.

[0259] Secondly, the deep feature distance 1305 between the feature extraction layer, time domain aggregation layer, and objective distortion quality assessment task head embedding layer of the reference audio, target audio is calculated as the objective distortion quality assessment score 1306 of the spatial audio.

[0260] Again, the deep feature distance 1307 between the feature extraction layer, time domain aggregation layer, and spatial quality task head embedding layer of the reference audio, the target audio is calculated as the spatial quality score 1308 of the spatial audio.

[0261] The depth feature distance is shown in formula (8):

[0262]

[0263] Finally, the overall quality score, objective distortion quality evaluation score, and spatial quality score are output as the evaluation results of "objective evaluation method of spatial audio based on human ear perception characteristics".

[0264] The audio data evaluation method proposed in the embodiment of the present application can perform objective evaluation of spatial audio well, has better objective properties, and its correlation with human subjective scores in a human subjective evaluation data set exceeds the baseline indicators DPLM and SAQAM.

[0265] For objective testing, common area test, clustering test and monotonicity test were performed respectively:

[0266] Common area test: In the common area test, two sets of data are constructed. The spatial audio pairs in the first set of data have different content and the same quality attributes (LQ or SQ), and the spatial audio pairs in the second set of data have different content and different quality attributes. The quality attribute scores (LQ or SQ) of the two sets of audio pairs are respectively calculated, and the standard normalized normal distribution parameters (i.e., mean and variance) of the two sets of data are calculated using the maximum likelihood estimation method, and the standard normalized normal distribution curves of the two sets of data are drawn. The area of ​​the common area (i.e., the overlapping area) of the two sets of curves is the desired area. The smaller the common area, the more robust the model is to content changes and the stronger the content invariance.

[0267] Clustering test: In the clustering test, the retrieval performance of the model for quality attributes (LQ or SQ) is verified. Specifically, according to different quality attributes, 10 different quality level groups are generated, and each quality level group consists of 100 spatial audios with the same quality attributes but mismatched content. The quality attributes of all 1000 spatial audios are calculated, and the k (=10 / 25) audios with the best scores are selected, and the probability that they belong to the quality level group with the highest quality attributes is calculated. The larger the probability, the stronger the clustering and retrieval characteristics of the network, indicating that the embedding layer in the model does capture quality attributes such as objective distortion quality evaluation and spatial positioning attributes.

[0268] Monotonicity test: In the monotonicity test, we verify the monotonic behavior of the model with increasing perturbation level (LQ or SQ). In the monotonicity test, a series of spatial audio with increasing perturbation levels are generated and the corresponding quality attribute scores are calculated. Finally, the Spearman correlation coefficient between the quality attribute score (LQ or SQ) and the increasing perturbation level (SNR or deflection angle) is calculated. The higher the correlation coefficient, the more monotonic the neural network is to the perturbation.

[0269] At the same time, in order to verify the correlation between the "objective spatial quality evaluation method based on human ear perception characteristics" proposed in the embodiment of the present application and human subjective perception, the spatial audio human subjective evaluation data set "Bitrate Compression Ambisonic" was used for subjective testing.

[0270] The purpose of this subjective test dataset is to study the impact of bit rate compression on Ambisonic audio. It includes 8 groups of 88 spatial audios, including six audio types: pink noise, human voice, castanets, bells, electronic music (2 groups), and original music (2 groups). The subjective score of this dataset is the MUSHAR score, which is a score of the degree of audio timbre distortion.

[0271] In the embodiment of the present application, an objective test is performed on the subjective test set, and the Spearman correlation coefficient between the objective test score and the subjective score of humans in the subjective test set is calculated. The larger the correlation coefficient, the higher the correlation between the model provided by the embodiment of the present application and human subjective perception. Moreover, the experimental results prove that the model provided by the embodiment of the present application has better subjective intuition and has a stronger correlation with human subjective perception, which also proves the usefulness and superiority of the embodiment of the present application in using the human ear perception characteristics to assist in the objective evaluation of spatial audio.

[0272] In an embodiment of the present application, the input spatial audio is first processed for human ear perception to obtain spatial audio mapped to the human ear perception domain, which is used to assist in the objective evaluation of spatial audio, thereby solving the problem that the objective evaluation algorithm cannot reasonably take into account the subjective and objective attributes of spatial audio, and increasing the correlation between the evaluation results and human subjective perception. By introducing spatial audio in the human ear perception domain as auxiliary information, and guiding the embedding layer in the neural network to focus on the human ear perception attributes of spatial audio through learnable weight weighting, the correlation between the subjective evaluation results of the model and the human subjective score can be enhanced. In addition, a series of subjective and objective evaluations were conducted on the proposed model. The experimental results show that the model proposed in the embodiment of the present application has achieved better overall performance, better objective characteristics, and a stronger correlation with human subjective perception, whether it is a subjective test or an objective test, which also proves the effectiveness of introducing human ear perception processing in the embodiment of the present application.

[0273] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indications of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequentially, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps. Based on the same inventive concept, the embodiment of the present application also provides an audio data evaluation device for implementing the above-mentioned audio data evaluation method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific definition in the one or more audio data evaluation device embodiments provided below can refer to the definition of the audio data evaluation method in the above text, and will not be repeated here.

[0274] In one embodiment, Fig.14As shown, an audio data evaluation device 1400 is provided, including: an acquisition module 1401, a processing module 1402, an evaluation module 1403 and a determination module 1404, wherein: the acquisition module 1401 is used to acquire original audio, the original audio includes reference audio and target audio to be evaluated; the processing module 1402 is used to perform human ear perception processing on the original audio to obtain human ear perception domain audio, the human ear perception domain audio includes reference human ear perception domain audio corresponding to the reference audio and target human ear perception domain audio corresponding to the target audio; the evaluation module 1403 is used to use an evaluation network to perform evaluation processing on the original audio and the human ear perception domain audio; the determination module 1404 is used to determine the difference information between the reference audio and the target audio according to the result of the evaluation processing, and determine the evaluation result of the target audio according to the difference information.

[0275] In one embodiment, the evaluation module 1403 is further used to: use the objective distortion quality evaluation subnetwork to at least perform objective evaluation processing on the original audio to obtain multiple reference objective evaluation features corresponding to the reference audio and multiple target objective evaluation features corresponding to the target audio; use the spatial quality evaluation subnetwork to perform spatial evaluation processing on the audio in the human ear perception domain, and fuse the evaluation features obtained by the objective evaluation processing during the spatial evaluation processing to obtain multiple reference spatial evaluation features corresponding to the reference audio and multiple target spatial evaluation features corresponding to the target audio.

[0276] In one embodiment, the objective distortion quality assessment subnetwork includes a cascaded first feature extraction layer, a first time domain aggregation layer and an objective distortion quality assessment head, and the assessment module 1403 is further used to: use the first feature extraction layer to perform feature extraction processing on the original audio to obtain a first spectrum feature, and the first spectrum feature includes a first reference spectrum feature corresponding to the reference audio and a first target spectrum feature corresponding to the target audio; use the first time domain aggregation layer to perform time domain aggregation processing on the first spectrum feature to obtain a first aggregation feature, and the first aggregation feature includes a first reference aggregation feature corresponding to the reference audio and a first target spectrum feature corresponding to the target audio. a target aggregate feature; using the objective distortion quality assessment head to perform objective distortion quality assessment identification processing on the first aggregate feature to obtain an objective distortion quality assessment identification feature, wherein the objective distortion quality assessment identification feature includes a reference objective distortion quality assessment identification feature corresponding to the reference audio and a target objective distortion quality assessment identification feature corresponding to the target audio; using the first reference spectrum feature, the first reference aggregate feature and the reference objective distortion quality assessment identification feature as the multiple reference objective evaluation features; using the first target spectrum feature, the first target aggregate feature and the target objective distortion quality assessment identification feature as the multiple target objective evaluation features.

[0277] In one embodiment, the spatial quality evaluation subnetwork includes a cascaded second feature extraction layer, a second time domain aggregation layer and a spatial quality head, and the evaluation module 1403 is further used to: use the second feature extraction layer to perform feature extraction processing on the human ear perception domain audio to obtain a second spectrum feature, the second spectrum feature includes a second reference spectrum feature corresponding to the reference audio and a second target spectrum feature corresponding to the target audio; use the second time domain aggregation layer to perform time domain aggregation processing on the second spectrum feature to obtain a second aggregation feature, the second aggregation feature includes a second reference aggregation feature corresponding to the reference audio and a second target aggregation feature corresponding to the target audio; and combine the second aggregation feature with the first The aggregated features are weightedly fused to obtain fused features, wherein the fused features include reference fused features corresponding to the reference audio and target fused features corresponding to the target audio; the fused features are spatially identified using the spatial quality head to obtain spatial quality identification features, wherein the spatial quality identification features include reference spatial quality identification features corresponding to the reference audio and target spatial quality identification features corresponding to the target audio; the second reference spectrum features, the second reference aggregated features and the reference spatial quality identification features are used as the multiple reference spatial evaluation features; the second target spectrum features, the second target aggregated features and the target spatial quality identification features are used as the multiple target spatial evaluation features.

[0278] In one embodiment, the evaluation module 1403 is also used to perform at least one level of convolution processing on the second spectral feature in the second time domain aggregation layer according to multiple preset channel numbers that increase step by step to obtain a first convolution feature; perform at least one level of convolution processing on the first convolution feature according to the multiple preset channel numbers that decrease step by step to obtain a second convolution feature; and perform residual processing on the second convolution feature to obtain the second aggregation feature.

[0279] In one embodiment, the determination module 1404 is further used to determine the overall feature distance between the reference audio and the target audio based on the first spectral feature, the second spectral feature, the first aggregate feature and the second aggregate feature; determine the objective feature distance between the reference audio and the target audio based on the first spectral feature, the first aggregate feature and the objective distortion quality assessment identification feature; determine the spatial feature distance between the reference audio and the target audio based on the second spectral feature, the second aggregate feature and the spatial quality identification feature.

[0280] In one embodiment, the evaluation module 1403 is also used to perform time-frequency transformation on the reference audio and the reference human ear perception domain audio to obtain first spectrum information; perform time-frequency transformation on the target audio and the target human ear perception domain audio respectively to obtain second spectrum information; input the first spectrum information and the second spectrum information into the evaluation network, so as to use the evaluation network to evaluate the original audio and the human ear perception domain audio.

[0281] In one embodiment, the evaluation module 1403 is also used to perform complex domain conversion on the target audio and the target human ear perception domain audio to obtain a time-frequency spectrum of the target audio in the complex domain and a time-frequency spectrum of the target human ear perception domain audio in the complex domain; extract the amplitude spectrum and phase spectrum of the time-frequency spectrum; stack the amplitude spectrum and phase spectrum corresponding to the target audio, and the amplitude spectrum and phase spectrum corresponding to the target human ear perception domain audio to obtain the second frequency spectrum information.

[0282] In one of the embodiments, the device further includes: a sample acquisition module for acquiring sample audio; a perception processing module for performing human ear perception processing on the sample audio to obtain sample audio in the human ear perception domain; an evaluation processing module for performing evaluation processing on the sample audio and the sample audio in the human ear perception domain using a to-be-trained evaluation network to obtain an objective distortion quality evaluation loss and a spatial quality loss of the sample audio; an adjustment module for adjusting the network parameters of the to-be-trained evaluation network according to the objective distortion quality evaluation loss and the spatial quality loss to obtain the evaluation network. In one of the embodiments, the sample acquisition module is further used to acquire at least three preset audios in a preset audio database; add preset noise to the at least three preset audios according to different preset signal-to-noise ratios to obtain at least three processed audios; determine a positive sample, an anchor sample with the largest signal-to-noise ratio, and a negative sample with the smallest signal-to-noise ratio in the at least three processed audios; and determine the anchor sample, the positive sample, and the negative sample as the sample audio.

[0283] In one of the embodiments, the sample audio in the human ear perception domain includes: anchor samples in the human ear perception domain, positive samples in the human ear perception domain and negative samples in the human ear perception domain. The evaluation and processing module is further used to use the objective distortion quality evaluation subnetwork in the evaluation network to be trained to evaluate the anchor samples, the positive samples and the negative samples to obtain the objective distortion quality evaluation loss of the sample audio; use the spatial quality evaluation subnetwork in the evaluation network to be trained to evaluate the anchor samples in the human ear perception domain; and determine the spatial quality loss based on the evaluation processing results of the anchor samples in the human ear perception domain and the evaluation processing results of the anchor samples.

[0284] In one of the embodiments, the evaluation processing module is used to predict the predicted probability distribution of the predicted azimuth of the anchor sample in the human ear perception domain in a preset angle interval set based on the evaluation processing result of the anchor sample in the human ear perception domain; determine the angle difference information based on the true value azimuth of the evaluation processing result of the anchor sample and the predicted azimuth; determine the distribution difference information based on the true value distribution of the true value azimuth in a preset angle interval set and the predicted probability distribution; determine the spatial quality loss based on the angle difference information and the distribution difference information. In one of the embodiments, the adjustment module is also used to determine the overall loss based on the objective distortion quality evaluation loss and the spatial quality loss; use the overall loss to adjust the network parameters of the evaluation network to be trained to obtain the evaluation network.

[0285] Each module in the above-mentioned audio data evaluation device can be implemented in whole or in part by software, hardware or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in an electronic device in the form of hardware, or can be stored in a memory in an electronic device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0286] In one embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as follows: Fig.15 As shown. The electronic device includes a collection device, a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the collection device, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is used to exchange information between the processor and the external device. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an audio data evaluation method is implemented. The display unit of the electronic device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the electronic device casing, or an external keyboard, touchpad or mouse.

[0287] Those skilled in the art will understand that Fig.15 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In one embodiment, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor performs the steps in the above-mentioned method embodiments when executing the computer program. In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and the computer program implements the steps in the above-mentioned method embodiments when executed by the processor. In one embodiment, a computer program product is provided, including a computer program, and the computer program implements the steps in the above-mentioned method embodiments when executed by the processor.

[0288] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions. A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it may include the processes of the embodiments of the above-mentioned methods. Among them, any reference to a memory, a database or other medium used in the embodiments provided in this application may include at least one of a non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, data processing logic devices based on quantum computing, etc., but are not limited thereto. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above-described embodiments only express several implementation methods of the present application, and the description is relatively specific and detailed, but it cannot be understood as a limitation on the scope of the patent of this application.It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these modifications and improvements are all within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. An audio data evaluation method, characterized in that: The method comprises: Acquire original audio, where the original audio includes reference audio and target audio to be evaluated; Performing human ear perception processing on the original audio to obtain human ear perception domain audio, wherein the human ear perception domain audio includes reference human ear perception domain audio corresponding to the reference audio and target human ear perception domain audio corresponding to the target audio; Using an evaluation network to perform evaluation processing on the original audio and the audio in the human ear perception domain; Difference information between the reference audio and the target audio is determined according to the result of the evaluation process, and the evaluation result of the target audio is determined according to the difference information.

2. The method according to claim 1, characterized in that: The evaluation network includes an objective distortion quality evaluation subnetwork and a spatial quality evaluation subnetwork. The evaluation network is used to evaluate the original audio and the audio in the human ear perception domain, including: Using the objective distortion quality assessment subnetwork to at least perform objective assessment processing on the original audio to obtain a plurality of reference objective assessment features corresponding to the reference audio and a plurality of target objective assessment features corresponding to the target audio; The spatial quality evaluation subnetwork is used to perform spatial evaluation processing on the audio in the human ear perception domain, and during the spatial evaluation processing, the evaluation features obtained by the objective evaluation processing are fused to obtain multiple reference spatial evaluation features corresponding to the reference audio and multiple target spatial evaluation features corresponding to the target audio.

3. The method according to claim 2, characterized in that The objective distortion quality evaluation sub-network includes a cascaded first feature extraction layer, a first time domain aggregation layer, and an objective distortion quality evaluation head. The objective distortion quality evaluation sub-network is used to perform objective evaluation processing on the original audio, including: Using the first feature extraction layer to perform feature extraction processing on the original audio to obtain a first spectrum feature, where the first spectrum feature includes a first reference spectrum feature corresponding to the reference audio and a first target spectrum feature corresponding to the target audio; Performing time-domain aggregation processing on the first spectrum feature by using the first time-domain aggregation layer to obtain a first aggregation feature, where the first aggregation feature includes a first reference aggregation feature corresponding to the reference audio and a first target aggregation feature corresponding to the target audio; Using the objective distortion quality assessment head to perform objective distortion quality assessment identification processing on the first aggregated feature to obtain an objective distortion quality assessment identification feature, wherein the objective distortion quality assessment identification feature includes a reference objective distortion quality assessment identification feature corresponding to the reference audio and a target objective distortion quality assessment identification feature corresponding to the target audio; using the first reference spectrum feature, the first reference aggregation feature, and the reference objective distortion quality assessment identification feature as the multiple reference objective assessment features; The first target spectrum feature, the first target aggregation feature and the target objective distortion quality assessment identification feature are used as the multiple target objective assessment features.

4. The method according to claim 3, characterized in that The spatial quality evaluation subnetwork includes a cascaded second feature extraction layer, a second time domain aggregation layer, and a spatial quality head. The spatial quality evaluation subnetwork is used to perform spatial evaluation processing on the audio in the human ear perception domain, and the evaluation features obtained by the objective evaluation processing are integrated during the spatial evaluation processing, including: Using the second feature extraction layer to perform feature extraction processing on the audio in the human ear perception domain to obtain a second spectrum feature, where the second spectrum feature includes a second reference spectrum feature corresponding to the reference audio and a second target spectrum feature corresponding to the target audio; Performing time-domain aggregation processing on the second spectrum features by using the second time-domain aggregation layer to obtain second aggregate features, where the second aggregate features include a second reference aggregate feature corresponding to the reference audio and a second target aggregate feature corresponding to the target audio; Performing a weighted fusion process on the second aggregated feature and the first aggregated feature to obtain a fused feature, wherein the fused feature includes a reference fused feature corresponding to the reference audio and a target fused feature corresponding to the target audio; Using the spatial quality head to perform spatial quality identification processing on the fusion feature to obtain a spatial quality identification feature, wherein the spatial quality identification feature includes a reference spatial quality identification feature corresponding to the reference audio and a target spatial quality identification feature corresponding to the target audio; using the second reference spectrum feature, the second reference aggregation feature and the reference space quality identification feature as the multiple reference space evaluation features; The second target spectrum feature, the second target aggregation feature and the target space quality identification feature are used as the multiple target space evaluation features.

5. The method according to claim 4, characterized in that The using the second time domain aggregation layer to perform time domain aggregation processing on the second spectrum feature to obtain a second aggregation feature includes: In the second time-domain aggregation layer, at least one level of convolution processing is performed on the second spectrum feature according to a plurality of preset channel numbers that increase step by step to obtain a first convolution feature; According to the plurality of preset channel numbers that decrease step by step, performing the at least one level of convolution processing on the first convolution feature to obtain a second convolution feature; Perform residual processing on the second convolutional features to obtain the second aggregated features.

6. The method according to claim 4, characterized in that The difference information includes at least one of an overall feature distance, an objective feature distance, and a spatial feature distance, and the determining the difference information between the reference audio and the target audio according to the result of the evaluation process includes: Determine the overall feature distance between the reference audio and the target audio according to the first spectrum feature, the second spectrum feature, the first aggregate feature, and the second aggregate feature; Determining the objective feature distance between the reference audio and the target audio according to the first spectrum feature, the first aggregate feature, and the objective distortion quality assessment identification feature; The spatial feature distance between the reference audio and the target audio is determined according to the second spectral feature, the second aggregation feature, and the spatial quality identification feature.

7. The method according to claim 1, characterized in that The using the evaluation network to perform evaluation processing on the original audio and the audio in the human ear perception domain includes: Performing time-frequency transformation on the reference audio and the reference human ear perception domain audio to obtain first spectrum information; Performing time-frequency transformation on the target audio and the target human ear perception domain audio respectively to obtain second spectrum information; The first spectrum information and the second spectrum information are input into the evaluation network, so as to use the evaluation network to perform evaluation processing on the original audio and the audio in the human ear perception domain.

8. The method according to claim 7, characterized in that The step of performing time-frequency transformation on the target audio and the target human ear perception domain audio to obtain second spectrum information includes: Performing complex domain conversion on the target audio and the target human ear perception domain audio to obtain a time-frequency spectrum of the target audio in the complex domain and a time-frequency spectrum of the target human ear perception domain audio in the complex domain; Extracting the amplitude spectrum and the phase spectrum of the time-frequency spectrum; The amplitude spectrum and phase spectrum corresponding to the target audio and the amplitude spectrum and phase spectrum corresponding to the target human ear perception domain audio are stacked to obtain the second frequency spectrum information.

9. The method according to claim 1, characterized in that: The method further comprises: Get sample audio; Performing human ear perception processing on the sample audio to obtain sample audio in the human ear perception domain; Using the to-be-trained evaluation network to perform evaluation processing on the sample audio and the sample audio in the human ear perception domain to obtain an objective distortion quality evaluation loss and a spatial quality loss of the sample audio; According to the objective distortion quality evaluation loss and the spatial quality loss, the network parameters of the evaluation network to be trained are adjusted to obtain the evaluation network.

10. The method according to claim 9, characterized in that The step of obtaining the sample audio comprises: Obtain at least three preset audios from a preset audio database; adding preset noise to the at least three preset audios according to different preset signal-to-noise ratios to obtain at least three processed audios; Determine, from the at least three processed audios, a positive sample, an anchor sample with a maximum signal-to-noise ratio, and a negative sample with a minimum signal-to-noise ratio; The anchor sample, the positive sample and the negative sample are determined as the sample audio.

11. The method according to claim 10, characterized in that The sample audio in the human ear perception domain includes: anchor samples in the human ear perception domain, positive samples in the human ear perception domain, and negative samples in the human ear perception domain. The sample audio and the sample audio in the human ear perception domain are evaluated and processed by the evaluation network to be trained to obtain the objective distortion quality evaluation loss and spatial quality loss of the sample audio, including: Using the objective distortion quality evaluation sub-network in the evaluation network to be trained to evaluate the anchor sample, the positive sample and the negative sample, to obtain the objective distortion quality evaluation loss of the sample audio; Using the spatial quality evaluation sub-network in the evaluation network to be trained to perform evaluation processing on the anchor samples in the human ear perception domain; The spatial quality loss is determined based on the evaluation processing result of the anchor sample in the human ear perception field and the evaluation processing result of the anchor sample.

12. The method according to claim 10, characterized in that The determining the spatial quality loss based on the evaluation processing result of the anchor sample in the human ear perception field and the evaluation processing result of the anchor sample comprises: According to the evaluation processing result of the anchor sample in the human ear perception domain, predicting the predicted probability distribution of the predicted azimuth of the anchor sample in the human ear perception domain in a preset angle interval set; Determine angle difference information according to the true azimuth of the evaluation processing result of the anchor sample and the predicted azimuth; Determining distribution difference information according to the true value distribution of the true value azimuth in a preset angle interval set and the predicted probability distribution; The spatial quality loss is determined according to the angle difference information and the distribution difference information.

13. The method according to claim 9, characterized in that The step of adjusting the network parameters of the evaluation network to be trained according to the objective distortion quality evaluation loss and the spatial quality loss to obtain the evaluation network comprises: Determining an overall loss according to the objective distortion quality evaluation loss and the spatial quality loss; The overall loss is used to adjust the network parameters of the evaluation network to be trained to obtain the evaluation network.

14. An audio data evaluation device, characterized in that: The device comprises: An acquisition module, used to acquire original audio, wherein the original audio includes reference audio and target audio to be evaluated; A processing module, configured to perform human ear perception processing on the original audio to obtain human ear perception domain audio, wherein the human ear perception domain audio includes a reference human ear perception domain audio corresponding to the reference audio and a target human ear perception domain audio corresponding to the target audio; An evaluation module, used for performing evaluation processing on the original audio and the audio in the human ear perception domain using an evaluation network; The determination module is used to determine the difference information between the reference audio and the target audio according to the result of the evaluation process, and determine the evaluation result of the target audio according to the difference information.

15. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 13 are implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

Images