Audio data processing method and device, storage medium and electronic equipment

Abstract

The invention relates to an audio data processing method and device, a storage medium and electronic equipment, and relates to the technical field of computers, and the method comprises the steps: obtaining first original audio data and second original audio data, extracting first to-be-compared audio data and second to-be-compared audio data from the first original audio data and the second original audio data; inputting the first to-be-compared audio data and the second to-be-compared audio data into a preset audio data processing model to obtain a tone quality difference result between the first to-be-compared audio data and the second to-be-compared audio data; and determining whether the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data according to the tone quality difference result. According to the invention, the accuracy of the sound quality comparison result is improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and more specifically, the embodiments of the present disclosure relate to a method for processing audio data, an apparatus for processing audio data, a computer-readable storage medium, and an electronic device. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] In some relevant audio data sound quality comparison schemes, the sound quality of two encoded compressed materials of the same song can be directly compared; however, considering the possibility of false sound quality, if only the simple encoding format / encoding bit rate of the audio data is compared during the sound quality comparison, the accuracy of the sound quality comparison result will be low. Summary of the invention

[0004] However, in the prior art, on the one hand, directly judging the sound quality of audio data by the encoding format / encoding bit rate of the audio data has the problem of low accuracy of the sound quality comparison result; on the other hand, since the encoding format and encoding bit rate of all audio data need to be compared, the comparison efficiency is low.

[0005] To this end, an improved audio data processing method is urgently needed, which can obtain first original audio data and second original audio data, and extract first audio data to be compared and second audio data to be compared from the first original audio data and the second original audio data; then input the first audio data to be compared and the second audio data to be compared into a preset audio data processing model to obtain the sound quality difference result between the first audio data to be compared and the second audio data to be compared; finally, determine whether the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data according to the sound quality difference result, so as to improve the audio quality comparison efficiency on the basis of improving the accuracy of the audio quality comparison result.

[0006] In this context, embodiments of the present disclosure are intended to provide a method for processing audio data, an apparatus for processing audio data, a computer-readable storage medium, and an electronic device.

[0007] According to one aspect of the present disclosure, there is provided a method for processing audio data, comprising:

[0008] Acquire first original audio data and second original audio data, and extract first audio data to be compared and second audio data to be compared from the first original audio data and the second original audio data;

[0009] Inputting the first audio data to be compared and the second audio data to be compared into a preset audio data processing model to obtain a sound quality difference result between the first audio data to be compared and the second audio data to be compared;

[0010] Determine whether a first audio quality of the first original audio data is higher than a second audio quality of the second original audio data according to the sound quality difference result.

[0011] In an exemplary embodiment of the present disclosure, extracting first audio data to be compared and second audio data to be compared from the first original audio data and the second original audio data includes:

[0012] Extracting first audio fingerprint data of the first original audio data and second audio fingerprint data of the second original audio data; wherein the first original audio data and the second original audio data are audio data of different bit rates corresponding to the same song;

[0013] Performing audio alignment processing on the first original audio data and the second original audio data according to the first audio fingerprint data and the second audio fingerprint data to obtain a first audio alignment result and a second audio alignment result;

[0014] The first audio data to be compared is intercepted from the first audio alignment result, and the second audio data to be compared is intercepted from the second audio alignment result; wherein the first audio position of the first audio data to be compared in the first audio alignment result is the same as the second audio position of the second audio data to be compared in the second audio alignment result.

[0015] In an exemplary embodiment of the present disclosure, extracting first audio fingerprint data of first original audio data includes:

[0016] Performing a short-time Fourier transform on the first original audio data to obtain a first original spectrogram, and traversing the first original spectrogram to extract a first local peak point from the first original spectrogram;

[0017] The first audio fingerprint data is constructed according to a first time series position of the first local peak point in the first original spectrogram and a first frequency band sequence number of the first local peak point.

[0018] In an exemplary embodiment of the present disclosure, performing audio alignment processing on first original audio data and second original audio data according to the first audio fingerprint data and the second audio fingerprint data to obtain a first audio alignment result and a second audio alignment result includes:

[0019] Traversing a first original spectrogram corresponding to the first original audio data, searching the first original spectrogram for a position having the most overlap with a second local peak point in the second audio fingerprint data as an audio alignment position of the first original audio data and the second original audio data; and / or

[0020] Traversing a second original spectrogram corresponding to the second original audio data, searching the second original spectrogram for a position having the most overlap with a first local peak point in the first audio fingerprint data as an audio alignment position of the first original audio data and the second original audio data;

[0021] Audio alignment processing is performed on the first original audio data and the second original audio data based on the audio alignment position to obtain a first audio alignment result and a second audio alignment result.

[0022] In an exemplary embodiment of the present disclosure, the preset audio data processing model includes a sound quality feature extraction model, a feature combination layer, and a sound quality score difference prediction model;

[0023] The first audio data to be compared and the second audio data to be compared are input into a preset audio data processing model to obtain a sound quality difference result between the first audio data to be compared and the second audio data to be compared, including:

[0024] Inputting the first audio data to be compared and the second audio data to be compared into the sound quality feature extraction model to obtain a first audio feature corresponding to the first audio data to be compared and a second audio feature corresponding to the second audio data to be compared;

[0025] The first audio feature and the second audio feature are combined based on the feature combination layer to obtain a sound quality combination feature, and the sound quality combination feature is input into a sound quality score difference prediction model to obtain a sound quality score difference result between the first audio data to be compared and the second audio data to be compared.

[0026] In an exemplary embodiment of the present disclosure, the preset sound quality feature extraction model includes a first feature extraction model and a second feature extraction model, and the first feature extraction model and the second feature extraction model share model parameters;

[0027] The first audio data to be compared and the second audio data to be compared are input into the sound quality feature extraction model to obtain a first audio feature corresponding to the first audio data to be compared and a second audio feature corresponding to the second audio data to be compared, including:

[0028] Performing a short-time Fourier transform on the first audio data to be compared to obtain a first spectrogram to be compared, and performing a short-time Fourier transform on the second audio data to be compared to obtain a second spectrogram to be compared;

[0029] Inputting the first spectrogram to be compared into the first feature extraction model to obtain a first audio feature corresponding to the first audio data to be compared;

[0030] The second spectrogram to be compared is input into the second feature extraction model to obtain a second audio feature corresponding to the second audio data to be compared.

[0031] In an exemplary embodiment of the present disclosure, the first feature extraction model includes a first embedding mapping layer, a position encoding module, a feature encoding module, a feature decoding module, a linear transformation layer and a Softmax layer;

[0032] The step of inputting the first spectrogram to be compared into the first feature extraction model to obtain a first audio feature corresponding to the first audio data to be compared includes:

[0033] Performing embedding mapping processing on the first to-be-compared sound spectrogram based on the first embedding mapping layer to obtain a first embedding vector, and performing encoding processing on the first to-be-compared sound spectrogram based on the position encoding module to obtain a first position vector;

[0034] Superimposing the first embedding vector and the first position vector to obtain a first input vector, and encoding the first input vector based on a feature encoding module to obtain a first encoding result;

[0035] The first encoding result is decoded based on the feature decoding module to obtain a first encoding matrix, the first encoding matrix is ​​transformed based on the linear transformation layer to obtain a first logical matrix, and the first logical matrix is ​​mapped based on the Softmax layer to obtain a first audio feature corresponding to the first audio data to be compared.

[0036] In an exemplary embodiment of the present disclosure, the feature encoding module includes a first multi-head self-attention module, a first residual connection and normalization module, a first feedforward neural network, and a second residual connection and normalization module;

[0037] The first input vector is encoded based on the feature encoding module to obtain a first encoding result, including:

[0038] Calculating a first attention mechanism of the first input vector based on a first multi-head self-attention module, and performing residual connection and normalization processing on the first attention mechanism based on a first residual connection and normalization module to obtain a first normalization processing result;

[0039] The first normalization processing result is linearized based on the first feedforward neural network to obtain a first linearization processing result, and the first linearization processing result is residually connected and normalized based on the second residual connection and normalization module to obtain a first encoding result.

[0040] In an exemplary embodiment of the present disclosure, calculating a first attention mechanism of the first input vector based on a first multi-head self-attention module includes:

[0041] Performing linear fusion on the first input vector to obtain a first query vector, a first key vector, and a first value vector, and calculating a first outer product vector of the first query vector and the first key vector to obtain a first similarity between the first query vector and the first key vector;

[0042] The first similarity is normalized to obtain a first weight matrix, and a second outer product vector of the first weight matrix and the first value vector is calculated to obtain the first attention mechanism.

[0043] In an exemplary embodiment of the present disclosure, combining the first audio feature and the second audio feature based on the feature combination layer to obtain a sound quality combination feature includes:

[0044] Calculating a difference between the first audio feature and the second audio feature to obtain a feature difference calculation result, and determining the sound quality combination feature according to the feature difference calculation result; and / or

[0045] Combining the feature difference calculation result, the first audio feature and the second audio feature to obtain the sound quality combination feature; and / or

[0046] The first audio feature and the second audio feature are combined to obtain the sound quality combination feature.

[0047] In an exemplary embodiment of the present disclosure, determining whether a first audio quality of first original audio data is higher than a second audio quality of second original audio data according to the sound quality difference result includes:

[0048] If the sound quality difference result is greater than or equal to a preset threshold, determining that the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data;

[0049] If the sound quality score difference result is less than a preset threshold, determining that the first audio quality of the first original audio data is lower than the second audio quality of the second original audio data;

[0050] Among them, the larger the value of the sound quality difference result, the higher the first audio quality of the first original audio data; the smaller the value of the sound quality difference result, the higher the second audio quality of the second original audio data.

[0051] In an exemplary embodiment of the present disclosure, the method for processing audio data further includes:

[0052] If the first audio quality is higher than the second audio quality, using the first original audio data as target audio data of a song corresponding to the original audio data;

[0053] If the second audio quality is higher than the first audio quality, the second original audio data is used as target audio data of the song corresponding to the original audio data.

[0054] In an exemplary embodiment of the present disclosure, the preset audio data processing model is obtained in the following manner:

[0055] Acquire audio data of lossless sound quality of historical songs, and encode and compress the audio data of lossless sound quality at different bit rates based on a preset encoder to obtain an audio encoding result;

[0056] Randomly pairing the audio encoding result and the audio data with lossless sound quality to obtain multiple audio pairing results, and determining actual sound quality differences of the audio pairing results;

[0057] Inputting each of the audio pairing results into the neural network model to be trained to obtain a predicted sound quality score difference, and constructing a target loss function according to the actual sound quality score difference and the predicted sound quality score difference;

[0058] Based on the target loss function, the parameters of the neural network model to be trained are adjusted to obtain the preset audio data processing model.

[0059] According to one aspect of the present disclosure, there is provided an audio data processing device, comprising:

[0060] An audio data extraction module, used to obtain the first original audio data and the second original audio data, and extract the first audio data to be compared and the second audio data to be compared from the first original audio data and the second original audio data;

[0061] A sound quality difference result determination module, used for inputting the first audio data to be compared and the second audio data to be compared into a preset audio data processing model to obtain a sound quality difference result between the first audio data to be compared and the second audio data to be compared;

[0062] The audio quality determination module is used to determine the first audio quality of the first audio data to be compared and the second audio quality of the second audio data to be compared according to the sound quality difference result.

[0063] In an exemplary embodiment of the present disclosure, extracting first audio data to be compared and second audio data to be compared from first original audio data and second original audio data includes:

[0064] Extracting first audio fingerprint data of the first original audio data and second audio fingerprint data of the second original audio data; wherein the first original audio data and the second original audio data are audio data of different bit rates corresponding to the same song;

[0065] Performing audio alignment processing on the first original audio data and the second original audio data according to the first audio fingerprint data and the second audio fingerprint data to obtain a first audio alignment result and a second audio alignment result;

[0066] The first audio data to be compared is intercepted from the first audio alignment result, and the second audio data to be compared is intercepted from the second audio alignment result; wherein the first audio position of the first audio data to be compared in the first audio alignment result is the same as the second audio position of the second audio data to be compared in the second audio alignment result.

[0067] In an exemplary embodiment of the present disclosure, extracting first audio fingerprint data of first original audio data includes:

[0068] Performing a short-time Fourier transform on the first original audio data to obtain a first original spectrogram, and traversing the first original spectrogram to extract a first local peak point from the first original spectrogram;

[0069] The first audio fingerprint data is constructed according to a first time series position of the first local peak point in the first original spectrogram and a first frequency band sequence number of the first local peak point.

[0070] In an exemplary embodiment of the present disclosure, performing audio alignment processing on first original audio data and second original audio data according to the first audio fingerprint data and the second audio fingerprint data to obtain a first audio alignment result and a second audio alignment result includes:

[0071] Traversing a first original spectrogram corresponding to the first original audio data, searching the first original spectrogram for a position having the most overlap with a second local peak point in the second audio fingerprint data as an audio alignment position of the first original audio data and the second original audio data; and / or

[0072] Traversing a second original spectrogram corresponding to the second original audio data, searching the second original spectrogram for a position having the most overlap with a first local peak point in the first audio fingerprint data as an audio alignment position of the first original audio data and the second original audio data;

[0073] Audio alignment processing is performed on the first original audio data and the second original audio data based on the audio alignment position to obtain a first audio alignment result and a second audio alignment result.

[0074] In an exemplary embodiment of the present disclosure, the preset audio data processing model includes a sound quality feature extraction model, a feature combination layer, and a sound quality score difference prediction model;

[0075] The first audio data to be compared and the second audio data to be compared are input into a preset audio data processing model to obtain a sound quality difference result between the first audio data to be compared and the second audio data to be compared, including:

[0076] Inputting the first audio data to be compared and the second audio data to be compared into the sound quality feature extraction model to obtain a first audio feature corresponding to the first audio data to be compared and a second audio feature corresponding to the second audio data to be compared;

[0077] The first audio feature and the second audio feature are combined based on the feature combination layer to obtain a sound quality combination feature, and the sound quality combination feature is input into a sound quality score difference prediction model to obtain a sound quality score difference result between the first audio data to be compared and the second audio data to be compared.

[0078] In an exemplary embodiment of the present disclosure, the preset sound quality feature extraction model includes a first feature extraction model and a second feature extraction model, and the first feature extraction model and the second feature extraction model share model parameters;

[0079] The first audio data to be compared and the second audio data to be compared are input into the sound quality feature extraction model to obtain a first audio feature corresponding to the first audio data to be compared and a second audio feature corresponding to the second audio data to be compared, including:

[0080] Performing a short-time Fourier transform on the first audio data to be compared to obtain a first spectrogram to be compared, and performing a short-time Fourier transform on the second audio data to be compared to obtain a second spectrogram to be compared;

[0081] Inputting the first spectrogram to be compared into the first feature extraction model to obtain a first audio feature corresponding to the first audio data to be compared;

[0082] The second spectrogram to be compared is input into the second feature extraction model to obtain a second audio feature corresponding to the second audio data to be compared.

[0083] In an exemplary embodiment of the present disclosure, the first feature extraction model includes a first embedding mapping layer, a position encoding module, a feature encoding module, a feature decoding module, a linear transformation layer and a Softmax layer;

[0084] The step of inputting the first spectrogram to be compared into the first feature extraction model to obtain a first audio feature corresponding to the first audio data to be compared includes:

[0085] Performing embedding mapping processing on the first to-be-compared sound spectrogram based on the first embedding mapping layer to obtain a first embedding vector, and performing encoding processing on the first to-be-compared sound spectrogram based on the position encoding module to obtain a first position vector;

[0086] Superimposing the first embedding vector and the first position vector to obtain a first input vector, and encoding the first input vector based on a feature encoding module to obtain a first encoding result;

[0087] The first encoding result is decoded based on the feature decoding module to obtain a first encoding matrix, the first encoding matrix is ​​transformed based on the linear transformation layer to obtain a first logical matrix, and the first logical matrix is ​​mapped based on the Softmax layer to obtain a first audio feature corresponding to the first audio data to be compared.

[0088] In an exemplary embodiment of the present disclosure, the feature encoding module includes a first multi-head self-attention module, a first residual connection and normalization module, a first feedforward neural network, and a second residual connection and normalization module;

[0089] The first input vector is encoded based on the feature encoding module to obtain a first encoding result, including:

[0090] Calculating a first attention mechanism of the first input vector based on a first multi-head self-attention module, and performing residual connection and normalization processing on the first attention mechanism based on a first residual connection and normalization module to obtain a first normalization processing result;

[0091] The first normalization processing result is linearized based on the first feedforward neural network to obtain a first linearization processing result, and the first linearization processing result is residually connected and normalized based on the second residual connection and normalization module to obtain a first encoding result.

[0092] In an exemplary embodiment of the present disclosure, calculating a first attention mechanism of the first input vector based on a first multi-head self-attention module includes:

[0093] Performing linear fusion on the first input vector to obtain a first query vector, a first key vector, and a first value vector, and calculating a first outer product vector of the first query vector and the first key vector to obtain a first similarity between the first query vector and the first key vector;

[0094] The first similarity is normalized to obtain a first weight matrix, and a second outer product vector of the first weight matrix and the first value vector is calculated to obtain the first attention mechanism.

[0095] In an exemplary embodiment of the present disclosure, combining the first audio feature and the second audio feature based on the feature combination layer to obtain a sound quality combination feature includes:

[0096] Calculating a difference between the first audio feature and the second audio feature to obtain a feature difference calculation result, and determining the sound quality combination feature according to the feature difference calculation result; and / or

[0097] Combining the feature difference calculation result, the first audio feature and the second audio feature to obtain the sound quality combination feature; and / or

[0098] The first audio feature and the second audio feature are combined to obtain the sound quality combination feature.

[0099] In an exemplary embodiment of the present disclosure, determining whether a first audio quality of first original audio data is higher than a second audio quality of second original audio data according to the sound quality difference result includes:

[0100] If the sound quality difference result is greater than or equal to a preset threshold, determining that the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data;

[0101] If the sound quality score difference result is less than a preset threshold, determining that the first audio quality of the first original audio data is lower than the second audio quality of the second original audio data;

[0102] Among them, the larger the value of the sound quality difference result, the higher the first audio quality of the first original audio data; the smaller the value of the sound quality difference result, the higher the second audio quality of the second original audio data.

[0103] In an exemplary embodiment of the present disclosure, the audio data processing device further includes:

[0104] a first target audio data determination module, configured to use the first original audio data as target audio data of a song corresponding to the original audio data if the first audio quality is higher than the second audio quality;

[0105] A second target audio data determination module is used to use the second original audio data as target audio data of a song corresponding to the original audio data if the second audio quality is higher than the first audio quality.

[0106] In an exemplary embodiment of the present disclosure, the preset audio data processing model is obtained in the following manner:

[0107] Acquire audio data of lossless sound quality of historical songs, and encode and compress the audio data of lossless sound quality at different bit rates based on a preset encoder to obtain an audio encoding result;

[0108] Randomly pairing the audio encoding result and the audio data with lossless sound quality to obtain multiple audio pairing results, and determining actual sound quality differences of the audio pairing results;

[0109] Inputting each of the audio pairing results into the neural network model to be trained to obtain a predicted sound quality score difference, and constructing a target loss function according to the actual sound quality score difference and the predicted sound quality score difference;

[0110] Based on the target loss function, the parameters of the neural network model to be trained are adjusted to obtain the preset audio data processing model.

[0111] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for processing audio data described in any one of the above is implemented.

[0112] According to one aspect of the present disclosure, there is provided an electronic device, including:

[0113] Processor; and

[0114] A memory, configured to store executable instructions of the processor;

[0115] The processor is configured to execute any one of the above-mentioned methods for processing audio data by executing the executable instructions.

[0116] According to the audio data processing method and the audio data processing device of the embodiment of the present disclosure, the first original audio data and the second original audio data can be obtained, and the first audio data to be compared and the second audio data to be compared can be extracted from the first original audio data and the second original audio data; then the first audio data to be compared and the second audio data to be compared are input into a preset audio data processing model to obtain the sound quality difference result between the first audio data to be compared and the second audio data to be compared; according to the sound quality difference result, it is determined whether the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data, without comparing the encoding format and the encoding bit rate of the audio data, thereby significantly reducing the problem of low accuracy of the sound quality comparison result caused by only comparing the encoding format and the encoding bit rate of the audio data, and reducing the problem of low comparison efficiency caused by the need to compare all encoding formats and encoding bit rates, bringing a better experience to users. BRIEF DESCRIPTION OF THE DRAWINGS

[0117] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0118] Figure 1 A flowchart schematically shows a method for processing audio data according to an exemplary embodiment of the present disclosure;

[0119] Figure 2 Schematically shows an architecture diagram of a twin neural network according to an embodiment of the present disclosure;

[0120] Figure 3 A schematic diagram of an example structure of an audio data processing model according to an example embodiment of the present disclosure is shown;

[0121] Figure 4 A schematic diagram of a structural example of a feature extraction model according to an exemplary embodiment of the present disclosure is shown;

[0122] Figure 5 A schematic diagram of a structural example of a feature coding module according to an exemplary embodiment of the present disclosure is shown;

[0123] Figure 6 A schematic diagram of a structure example of a deep convolutional neural network according to an exemplary embodiment of the present disclosure is shown;

[0124] Figure 7 A structural example diagram of a sound quality score difference prediction model according to an exemplary embodiment of the present disclosure is schematically shown;

[0125] Figure 8 A flowchart of a training process of an audio data processing model according to an exemplary embodiment of the present disclosure is schematically shown;

[0126] Fig. 9 An example diagram schematically shows an audio pairing result obtained according to an example embodiment of the present disclosure;

[0127] Fig.10 Schematically shows an example diagram of an original spectrogram obtained according to an example embodiment of the present disclosure;

[0128] Fig.11 Schematically shows an example diagram of a local peak point obtained according to an example embodiment of the present disclosure;

[0129] Fig.12 A scene example diagram schematically shows a matching process of an audio alignment position according to an exemplary embodiment of the present disclosure;

[0130] Fig.13 A scene example diagram schematically shows another audio alignment position matching process according to an example embodiment of the present disclosure;

[0131] Fig.14 A block diagram schematically shows a device for processing audio data according to an exemplary embodiment of the present disclosure;

[0132] Fig.15 Schematically illustrates a computer-readable storage medium for storing a processing method for audio data according to an example embodiment of the present disclosure;

[0133] Fig.16 An electronic device for implementing a method for processing audio data according to an exemplary embodiment of the present disclosure is schematically shown.

[0134] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0135] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0136] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0137] According to an embodiment of the present disclosure, a method for processing audio data, an apparatus for processing audio data, a computer-readable storage medium, and an electronic device are proposed.

[0138] In this document, any number of elements in the drawings is used for illustration rather than limitation, and any naming is used only for distinction and does not have any limiting meaning.

[0139] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure. SUMMARY OF THE INVENTION

[0141] The applicant has found that high sound quality can bring users a more immersive listening experience, so sound quality is one of the key concerns of all music streaming platforms. In the process of actual application, in order to ensure the sound quality of the audio materials in the music library, the following methods are usually adopted to achieve it: the first implementation method is to use more advanced audio compression coding technology to provide users with more audio details at the same bit rate; the second implementation method is to increase the coverage and proportion of high-quality materials in the music library; for example, when a song in the music library has only 128kbps MP3 format audio, you can purchase lossless or 320kbps MP3 format audio data from the song publisher or musician; the third implementation method is to detect fake sound quality songs in the music library; in the process of actual application, the sound quality of fake sound quality songs is low, but it takes up more storage space and transmission bandwidth, bringing additional losses to the platform; and if such songs are perceived by users, they will reduce the user's auditory experience, and thus cause a negative word-of-mouth impact on the song playback platform.

[0142] Furthermore, in the second and third implementations described above, in order to improve the sound quality of the audio materials in the music library, it is usually necessary to compare the sound quality of two encoded compressed materials of the same song; however, considering the possibility of false sound quality, when comparing the sound quality, only comparing the encoding formats / encoding bit rates through a simple comparison method may lead to wrong judgments, so it is more reasonable to analyze which is better from the actual signal level.

[0143] For example, in the second implementation method recorded above, when an audio material with higher sound quality (only from the encoding level) is obtained from the outside, it is necessary to compare it with the existing material in the library to determine whether the sound quality at the actual signal level is higher. If so, the material in the library can be replaced and upgraded; for example, in the third implementation method recorded above, versions of the same song with different encoders / bit rates are compared. When it is found that version A is better than version B in terms of encoding format / encoding bit rate, but version A is significantly worse than version B at the actual signal level, it can be determined that version A is fake sound quality.

[0144] In some related technical solutions, the audio quality can be judged based on the following method: first, extract the audio spectrum, treat it as an input image, and send it to the neural network; then, use the neural network model to predict the voice quality score of the song. The so-called voice quality score is mainly determined based on the overtone volume and overtone distribution. However, this method has the following defects: the above scheme is to score the file of a single song. For the input song, a voice quality score is predicted, and the quality of audio data at different bit rates cannot be compared at the same time; further, if the quality of audio data at different bit rates can be predicted by multiple scoring methods based on the neural network model, the difference between the two materials cannot be fully referenced in the prediction and scoring process, and the effect is greatly reduced.

[0145] Based on this, the example embodiment of the present disclosure proposes a method for comparing audio data. Based on the audio data comparison method proposed in the example embodiment of the present disclosure, the problem of comparing the quality of different versions of materials of the same song can be solved. In addition, the main purpose of the audio data comparison method proposed in the example embodiment of the present disclosure is to differentiate the sound quality between different versions of materials, and to judge the difference in sound quality between the two by the size of the difference; therefore, the predicted absolute score of a single audio data is not important in this scenario. In the case of a comparative reference, predicting the score difference is an easier machine learning problem for the model.

[0146] It is necessary to further explain here that, since the singer's voice quality score and the sound quality recorded in the example embodiment of the present disclosure are concepts of two different dimensions, there are completely different approaches when preparing training data. Specifically, in the general preparation of training data, songs by well-known singers can be roughly regarded as high-quality songs, and songs sung by users can be regarded as low-quality songs. Although this is reasonable to a certain extent, it is too rough; based on this, the example embodiment of the present disclosure will use a fully automated way to construct training data, covering a variety of audio encoders and a variety of bit rates, and can produce a large amount of training data relatively accurately.

[0147] Furthermore, the audio data processing method described in the example embodiments of the present disclosure can extract sound quality features from audio materials and predict sound quality differences for different encoded and compressed versions of the same song, determine the sound quality difference between the two songs at the actual signal level, and provide sound quality reference information for subsequent business use. In the actual application process, first, it is necessary to compress the lossless material with multiple encoders and multiple bit rates to construct a sound quality difference pair for training the audio data processing model; secondly, before comparing the sound quality difference, the two input audio materials can also be aligned on the time axis using audio fingerprint technology. The alignment link can reduce the difficulty of subsequent sound quality feature extraction and sound quality difference prediction, and obtain a more accurate difference prediction.

[0148] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0149] Exemplary Methods

[0150] The exemplary embodiment of the present disclosure first provides a method for processing audio data, which can be run on a terminal device, a server, a server cluster or a cloud server, etc. Of course, those skilled in the art can also run the method of the present disclosure on other platforms as required, and this exemplary embodiment does not specifically limit this. Figure 1 As shown, the method for generating audio data may include the following steps:

[0151] Step S110. Acquire first original audio data and second original audio data, and extract first audio data to be compared and second audio data to be compared from the first original audio data and the second original audio data;

[0152] Step S120. Input the first audio data to be compared and the second audio data to be compared into a preset audio data processing model to obtain a sound quality difference result between the first audio data to be compared and the second audio data to be compared;

[0153] Step S130: Determine whether the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data according to the sound quality difference result.

[0154] In the above-mentioned audio data processing method, the first original audio data and the second original audio data can be obtained, and the first audio data to be compared and the second audio data to be compared can be extracted from the first original audio data and the second original audio data; then the first audio data to be compared and the second audio data to be compared are input into a preset audio data processing model to obtain the sound quality difference result between the first audio data to be compared and the second audio data to be compared; according to the sound quality difference result, it is determined whether the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data, without comparing the encoding format and the encoding bit rate of the audio data, thereby significantly reducing the problem of low accuracy of the sound quality comparison result caused by only comparing the encoding format and the encoding bit rate of the audio data, and reducing the problem of low comparison efficiency caused by the need to compare all encoding formats and encoding bit rates, bringing a better experience to users.

[0155] Hereinafter, the method for processing audio data described in the exemplary embodiment of the present disclosure will be explained and illustrated in detail with reference to the accompanying drawings.

[0156] First, the terms involved in the exemplary embodiments of the present disclosure are explained and illustrated.

[0157] Digital audio: Sound signals in the physical world come from the vibration of the sound source and the transmission of sound waves. In order to collect, store, transmit and process sound signals in computers, they are digitized. Specifically, continuous sound signals are sampled at a certain frequency (i.e., sampling rate) in time to obtain a discrete sequence of values, and then these discrete values ​​are binary-encoded to obtain the final digital audio signal; in actual application, digital audio technology allows audio to be stored, transmitted and processed, while also allowing editing and effect processing, such as equalization, reverberation, etc.; and digital audio has been widely used in modern audio production, music recording, film production, and broadcasting.

[0158] Audio compression coding: Original digital audio usually takes up a lot of storage space and network transmission bandwidth; specifically, taking common song audio as an example, if the sampling rate is 48000Hz, the data channel is 2 channels, and the bit rate is 16 bits, the data volume per second is 48000*2*16 / 8=192,000 bytes, then the audio file of a 4-minute song needs 40MB in size, which brings great inconvenience to storage and transmission. For this reason, related companies have developed various audio compression coding technologies to compress and encode original digital audio, which are mainly divided into two categories: lossless compression and lossy compression.

[0159] Lossless compression: During the lossless compression process, the signal will not suffer any loss. By decoding the compressed audio, the original digital audio signal can be completely reconstructed. However, lossless compression usually has a limited degree of compression and is only suitable for scenarios with extremely high requirements for audio quality. Among them, common audio lossless compression algorithms may include but are not limited to Free Lossless Audio Codec (FLAC), Apple Lossless Audio Codec (ALAC), etc.

[0160] Lossy compression: In the process of lossy compression, the details of the audio signal are discarded in order to pursue a high compression rate. Therefore, if lossy compression is used for compression, the original digital audio signal cannot be completely reconstructed after decoding the compressed audio; specifically, in the actual compression process, the more signal details are discarded, the higher the compression rate; accordingly, the greater the signal loss. Among them, common audio lossy compression includes MP3, AAC, Vorbis, etc.; currently, mainstream music streaming platforms use lossy compression technology to encode audio to facilitate providing it to end users.

[0161] Audio bit rate: also known as audio bit rate, refers to the amount of data per second in an audio stream; specifically, taking 128kbps bit rate as an example, kb stands for kilobits, ps (per second) stands for second, then 128kbps means that the amount of data in one second is 128 kilobits; in lossy compression, you can set a target compression bit rate, a high bit rate means more details are retained, a low degree of compression, and a large amount of data; a low bit rate means fewer details are retained, a high degree of compression, and a small amount of data; for example, taking MP3 as an example, an MP3 with a bit rate of 320kbps can be considered to be close to the original audio, and most people will not have a significant perception, while the loss of detail in the 128kbps MP3 signal is more serious, and most people can hear the difference.

[0162] Sound quality: Sound quality in a broad sense refers to the listening quality of audio, which is the subjective feeling of the audience and is directly related to the audio content, acquisition and production, compression coding, sound effects, playback equipment and the user's auditory system. Sound quality in a narrow sense mainly refers to the degree of damage to audio details during the compression coding process; for example, in music streaming platforms, the concepts of lossless sound quality and lossy sound quality are often mentioned; for example, a song encoded in FLAC will be considered to have better sound quality than a 320kbps MP3, and a 320kbps MP3 is better than a 256kbps MP3. At the same time, the sound quality involved in the example embodiments of the present disclosure is all sound quality in a narrow sense, that is, the sound quality changes caused by compression coding.

[0163] Fake sound quality: Under normal circumstances, the lossless version of the same song has higher sound quality than the lossy version; under the same lossy encoding format, the high bit rate has higher sound quality than the low bit rate. However, due to production negligence or subjective malice, the following situations may sometimes occur: 1) Lossy audio is sent to encoders such as FLAC and ALAC to generate audio in flac / alac format; therefore, on the surface, the audio is in lossless format; however, at the signal level, it is actually in lossy format; 2) Low-bit-rate lossy audio is used as input, and a high bit rate is specified for compression encoding to generate high-bit-rate audio; therefore, on the surface, the audio is high-bit-rate lossy audio; however, at the signal level, there is actually no new audio detail, and it is still low-bit-rate lossy. Therefore, in the above two situations, the audio material produced is considered to have good sound quality only from the perspective of the audio format, but from the perspective of the signal level, the sound quality is actually poor. This situation is fake sound quality.

[0164] Embedding: A common concept in the field of deep learning, which refers to mapping the input data into a vector of fixed dimension through the network for representation.

[0165] Siamese Network: A concept in deep neural networks. The basic definition of a Siamese network is that the structures of the neural networks network_1 and network_2 are the same, and the network weight parameters are shared. Therefore, they can be understood as the same network. In practical applications, Siamese networks are widely used in deep metric learning. For example, element 1 and element 2 can be mapped to embedding vectors through network_1 and network_2, respectively, for subsequent use. However, the network_1 and network_2 recorded here are actually the same network, and the network weight parameters are shared. For the specific network architecture diagram of the Siamese network, please refer to Figure 2 shown.

[0166] Next, the audio data processing model involved in the exemplary embodiment of the present disclosure is explained and illustrated. Figure 3As shown, the audio data processing model may include a first input layer 310, a sound quality feature extraction model 320, a feature combination layer 330, a sound quality score difference prediction model 340 and a first output layer 350; further, the audio feature extraction model recorded here includes a first sound quality feature extraction model and a second sound quality feature extraction model, the first sound quality feature extraction model and the second sound quality feature extraction model are twin network models, and the first feature extraction model and the second feature extraction model share model parameters; that is, the model structure and model parameters of the first feature extraction model and the second feature extraction model are completely consistent; further, the first feature extraction model and the second feature extraction model recorded here may include a deep convolutional neural network, or may include a Resnet residual convolutional network or a Transformer network, etc.; in the process of actual application, it can be selected according to actual needs, and this example does not impose special restrictions on this. Further, the sound quality score difference prediction model recorded here can be a decision tree model, a multi-layer perceptron or a long short-term memory network, etc.; in the process of actual application, it can be selected according to actual needs, and this example does not impose special restrictions on this.

[0167] In an exemplary embodiment, referring to Figure 4 As shown, taking the first feature extraction model and the second feature extraction model as Transformer networks as an example, the specific model structure may include a first embedding mapping layer 410, a position encoding module 420, a feature encoding module 430, a feature decoding module 440, a linear transformation layer 450 and a Softmax layer 460; at the same time, referring to Figure 5 As shown, the feature encoding module recorded here may include a first multi-head self-attention module 501, a first residual connection and normalization module 502, a first feedforward neural network 503, and a second residual connection and normalization module 504; wherein, the specific functions that each functional module needs to perform in the audio data processing process will be described in detail later, and no further description will be given here.

[0168] In an exemplary embodiment, referring to Figure 6As shown, taking the first feature extraction model and the second feature extraction model as a deep convolutional neural network as an example, the specific model structure may include a first convolutional pooling layer 610, a second convolutional pooling layer 620, a third convolutional pooling layer 630, a fourth convolutional pooling layer 640, a fifth convolutional pooling layer 650 and a fully connected layer 660. Among them, the first convolutional pooling layer may include 2 convolutional layers + 1 pooling layer, the second convolutional pooling layer may include 2 convolutional layers + 1 pooling layer, the third convolutional pooling layer may include 3 convolutional layers + 1 pooling layer, the fourth convolutional pooling layer may include 3 convolutional layers + 1 pooling layer, the fifth convolutional pooling layer may include 3 convolutional layers + 1 pooling layer, and the fully connected layer may include 3 Dense layers. At the same time, in the actual application process, if the pooling layer recorded here is GlobalPooling, then after the first feature extraction model and the second feature extraction model, t seconds of audio is mapped to a fixed-dimensional vector e; if Global Pooling is not used, it is mapped to a fixed-dimensional vector sequence [e1, e2, ...e n ]; In the actual application process, you can select a specific pooling layer according to your actual needs. This example does not impose any special restrictions on this.

[0169] In an exemplary embodiment, referring to Figure 7 As shown, taking the sound quality score difference prediction model as a multi-layer perceptron as an example, the specific model structure may include an input layer (Input Layer) 710, a first hidden layer (Hidden Layer1) 720, a second hidden layer (Hidden Layer2) 730, and an output layer (Output Layer) 740, etc.

[0170] The following will explain and illustrate the specific training process of the audio data processing model. Figure 8 As shown, the specific training process of the audio data processing model may include the following steps:

[0171] Step S810, obtaining audio data of lossless sound quality of historical songs, and encoding and compressing the audio data of lossless sound quality at different bit rates based on a preset encoder to obtain an audio encoding result.

[0172] In this example embodiment, first, lossless song audio materials (i.e., audio data with lossless sound quality) of a certain number of songs can be exported from the music library; wherein, the audio data with lossless sound quality recorded here refers to audio data that has not been encoded or compressed in any way; at the same time, the historical songs recorded here may include a variety of different music genres (for example, rock, manifold, lyrical, classic, retro, etc.) and songs in different languages ​​(such as Chinese, English, French, Japanese, German, etc.); secondly, the audio data with lossless sound quality is encoded and compressed to obtain an audio encoding result; specifically, the encoders recorded here may include but are not limited to MP3, AAC, Vorbis, etc., and the bit rates recorded here may include but are not limited to 64kbps, 128kbps, 256kbps, 320kbps or 640kbps, etc., and this example does not impose any special restrictions on this.

[0173] Step S820: randomly pair the audio encoding result and the audio data with lossless sound quality to obtain multiple audio pairing results, and determine the actual sound quality difference of each audio pairing result.

[0174] Specifically, in the process of random pairing, audio data with lossless sound quality and audio encoding results can be paired, or the audio encoding results can be paired to obtain a sound quality Pair pair (that is, an audio pairing result); wherein the obtained audio pairing results may include audio encoding results with the same encoding method and the same bit rate, audio encoding results with the same encoding method and different bit rates, audio encoding results with different encoding methods and the same bit rate, audio encoding results with different encoding methods and different bit rates, etc., and this example does not impose any special restrictions on this; it should be noted here that, since the random pairing can also be audio data with lossless sound quality, the audio encoding results with the same encoding method and the same bit rate can also include two audio data with lossless sound quality, etc.; further, the two audio materials included in the generated audio pairing result can be represented by A1 and A2, and the obtained audio pairing result can refer to Fig. 9 shown.

[0175] It should also be noted here that, in the process of constructing the audio pairing results, the starting point is the lossless audio material of each historical song, and then P sound quality pair pairs (that is, the audio pairing results) are obtained; therefore, after encoding and compressing all lossless song materials, a set of sound quality pair pairs covering different encoders and different bit rates can be obtained, which will be used for subsequent sound quality comparison model training, thereby achieving the purpose of improving the accuracy of the obtained audio data processing model.

[0176] In addition, in order to meet the needs of subsequent model training, it is also necessary to know the actual sound quality difference of the pair; therefore, in the data construction stage (that is, in the process of generating audio pairing results), the corresponding sound quality difference Score(A1)-Score(A2) can be obtained by looking up the table according to the encoders and encoding bit rates used by the two materials; for example, the corresponding Score(A1) and Score(A2) can be obtained by looking up the table respectively, and then Score(A1)-Score(A2) can be calculated to get the actual sound quality difference; wherein, the table can be determined by expert knowledge, and for any two encoders and bit rate combinations, it is mapped to a corresponding sound quality difference; at the same time, the actual sound quality difference can be scaled to a reasonable range, for example, taking the 5-point system as an example, the sound quality difference range is [-5,5]; wherein, a negative number indicates that the sound quality of A1 is worse than that of A2, and a positive number indicates that the sound quality of A1 is better than that of A2. The larger the absolute value of the difference, the greater the difference in sound quality between the two.

[0177] Step S830, input each of the audio pairing results into the neural network model to be trained to obtain a predicted sound quality score difference, and construct a target loss function based on the actual sound quality score difference and the predicted sound quality score difference.

[0178] Specifically, the model training process described herein can be performed offline; that is, in the offline training stage, the two audio materials contained in the sound quality pair (that is, the audio pairing result) can be input to obtain the predicted sound quality score difference; wherein, the obtained predicted sound quality score difference is: S_diff = Score(A1) - Score(A2); it should be noted here that, since the two audio materials are generated from the same lossless material in the training stage, they are aligned in time, and there is no need to perform audio fingerprint alignment. After obtaining the audio pairing result, it can be directly input into the neural network model to be trained to obtain the corresponding predicted sound quality score difference.

[0179] Furthermore, after obtaining the predicted sound quality score difference, the target loss function can be constructed according to the predicted sound quality score difference and the actual sound quality score difference; wherein, the target loss function recorded here can adopt L2 (mean square error loss function), or can adopt cross entropy loss function or other loss functions, and can be selected according to actual needs in the actual application process, and this example does not impose any special restrictions on this.

[0180] Furthermore, before the audio pairing result is input into the neural network model to be trained, the audio data included in the audio pairing result needs to be preprocessed; the specific processing process may include the following two aspects: on the one hand, a time domain audio signal of a fixed length of t seconds is intercepted from the audio as the input of this link; specifically, in the interception process, A1 and A2 need to be intercepted from the same position to ensure that the two are time-aligned; among them, a typical approach is: randomly sampling a starting position in [0,dur-t], and starting from this position, intercepting a t-second segment; another typical approach is: starting from 0 seconds, intercepting a t-second segment every δ seconds, this approach can intercept multiple t seconds, which can be used as multiple training or test samples; another aspect is to perform short-time Fourier Transform (STFT) analysis on the audio, for example, using 2048-point FFT, 512-point hop size, and Hanning window to perform STFT spectrum analysis to obtain a spectrum diagram. After this step, the t-second time domain signal can be transformed into a fixed-size two-dimensional spectrogram, with the horizontal axis being the number of frames and the vertical axis being the number of frequency bands. That is, the data finally input into the neural network model to be trained is the two-dimensional spectrogram of the two audio data in the audio pairing result.

[0181] In an exemplary embodiment, in the process of predicting the sound quality score difference, after passing through the audio feature network (i.e., the feature extraction model), the sound quality feature vectors (or vector sequences) corresponding to A1 and A2 can be obtained; then, the two are combined at the feature combination layer so as to be sent to the subsequent sound quality score difference prediction model; at the same time, since Score(A1)-Score(A2)≠Score(A2)-Score(A1), this means that the two have a relative order relationship, so this relative order relationship should be reflected in the sound quality feature combination module; further, in order to reflect the order relationship between A1 and A2, a possible approach is to perform difference processing on the sound quality features of the two, and the specific processing process can be shown in the following formula (1):

[0182] e_diff = e(A1) - e(A2); Formula (1)

[0183] Then, e_diff is used as the output of the feature combination and sent to the sound quality score difference prediction model; of course, e_diff can also be directly concatenated with e(A1) and e(A2) and sent to the sound quality score difference prediction model; further, another possible approach is to directly concatenate e(A1) and (A2) and send them to the sound quality score difference prediction model, but this method requires the sound quality score difference prediction model to learn the relative order relationship between the two during the training process, which will increase the difficulty of model learning to a certain extent.

[0184] In an exemplary embodiment, since the sound quality features have been combined in the feature combination layer, the input in the sound quality score difference prediction model is the spliced ​​and combined feature vector or the spliced ​​and combined feature vector sequence; on this basis, the final sound quality score difference can be predicted (that is, the predicted sound quality score difference is determined); at the same time, since the specific determination process of predicting the sound quality score difference is a typical regression problem, a variety of common neural network structures can be selected; in the actual application process, if the input is a feature vector, a simple feedforward neural network structure such as MultiLayer Perceptron can be used; if the input is a feature vector sequence, a recursive neural network such as Long Short-Term Memory Network (LSTM) can be used. Finally, a floating point value is output as the predicted sound quality score difference. Furthermore, after obtaining the predicted sound quality score difference, the corresponding target loss function can be constructed; specifically, in the construction stage of the target loss function, the expert score difference corresponding to the data set pair (i.e., the actual sound quality score difference groundtruth) can be taken as the target, and the regression loss function Loss can be used as the target loss function to train the entire neural network; wherein, the Loss recorded here can select common loss functions such as L1 and L2; wherein, taking the target loss function as the L2 loss function as an example, the target loss function can be shown as the following formula (2):

[0185]

[0186] Step S840: Adjust the parameters of the neural network model to be trained based on the target loss function to obtain the preset audio data processing model.

[0187] Specifically, after obtaining the target loss function, the network weight parameters of the sound quality feature extraction model and the sound quality score difference prediction model can be updated based on the target loss function until convergence. At the same time, after the parameter adjustment is completed, the neural network model can also be tested based on the test set, and after the test is completed, the audio data processing model can be obtained.

[0188] The following will be combined Figure 2-Figure 9 right Figure 1 The method for processing the audio data shown in the figure is further explained and illustrated. Specifically:

[0189] In step S110 , first original audio data and second original audio data are acquired, and first audio data to be compared and second audio data to be compared are extracted from the first original audio data and the second original audio data.

[0190] In this example embodiment, first, the first original audio data and the second original audio data are obtained; specifically, the first original audio data and the second original audio data recorded here refer to two copies of audio data of a certain song with different encoding methods and encoding bit rates, and the first original audio data and the second original audio data can be obtained from the server corresponding to the application; secondly, the first audio data to be compared and the second audio data to be compared are extracted from the first original audio data and the second original audio data; specifically, they can be obtained in the following manner: extracting the first audio fingerprint data of the first original audio data and the second audio fingerprint data of the second original audio data; wherein the first original audio data and the second original audio data are audio data of different bit rates corresponding to the same song; performing audio alignment processing on the first original audio data and the second original audio data according to the first audio fingerprint data and the second audio fingerprint data to obtain a first audio alignment result and a second audio alignment result; intercepting the first audio data to be compared from the first audio alignment result, and intercepting the second audio data to be compared from the second audio alignment result; wherein the first audio position of the first audio data to be compared in the first audio alignment result is the same as the second audio position of the second audio data to be compared in the second audio alignment result.

[0191] In an exemplary embodiment, extracting the first audio fingerprint data of the first original audio data can be achieved in the following manner: performing a short-time Fourier transform on the first original audio data to obtain a first original spectrogram, traversing the first original spectrogram, and extracting a first local peak point from the first original spectrogram; constructing the first audio fingerprint data according to a first time series position of the first local peak point in the first original spectrogram and a first frequency band sequence number of the first local peak point.

[0192] In an exemplary embodiment, extracting the second audio fingerprint data of the second original audio data can be achieved in the following manner: performing a short-time Fourier transform on the second original audio data to obtain a second original spectrogram, traversing the second original spectrogram, and extracting a second local peak point from the second original spectrogram; constructing the second audio fingerprint data according to a second time series position of the second local peak point in the second original spectrogram and a second frequency band number of the second local peak point.

[0193] In an exemplary embodiment, audio alignment processing is performed on first original audio data and second original audio data according to first audio fingerprint data and second audio fingerprint data to obtain a first audio alignment result and a second audio alignment result, including: traversing a first original sound spectrogram corresponding to the first original audio data, and searching the first original sound spectrogram for a position having the most overlapping part with a second local peak point in the second audio fingerprint data as the audio alignment position of the first original audio data and the second original audio data; and / or traversing a second original sound spectrogram corresponding to the second original audio data, and searching the second original sound spectrogram for a position having the most overlapping part with a first local peak point in the first audio fingerprint data as the audio alignment position of the first original audio data and the second original audio data; and performing audio alignment processing on the first original audio data and the second original audio data based on the audio alignment position to obtain a first audio alignment result and a second audio alignment result.

[0194] The specific extraction process of the first audio data to be compared and the second audio data to be compared will be further explained and illustrated below. Specifically, in the actual application process, since there may be a time misalignment between the first original audio data and the second original audio data, before extracting the first audio data to be compared and the second audio data to be compared, audio alignment processing is first required; wherein, the time misalignment between the first original audio data and the second original audio data is mainly reflected in: assuming that there is a song x in the song library with a bit rate of 320kbps MP3, which is a material that was put into the library many years ago; and at this stage, the corresponding lossless audio of the song x is purchased from the publisher; however, due to further production reasons, there may be more or less silence of several seconds at the beginning and end of the film, resulting in a time offset with the material in the library; therefore, in this case, if the sound quality comparison is performed directly, the accuracy of the sound quality comparison will be affected due to the time misalignment of the two materials. To this end, before performing a sound quality comparison, the audio fingerprint technology can be used to time-align the two audio materials (ie, the first original audio data and the second original audio data); after alignment, the audio clips are intercepted according to the original process for sound quality comparison.

[0195] In an example embodiment, audio fingerprint technology is a typical music retrieval technology, and its basic principle is: for the audio of each song in the music library, the fingerprint information representing the song is extracted from the audio through signal processing or machine learning technology. The fingerprint information can usually be stored as a numerical value, a numerical vector or a numerical vector sequence, etc., and then the fingerprint information of these songs is assembled into an efficient index library. When used, for the query audio to be retrieved, the fingerprint information can be extracted from the query audio, and then the search matching (i.e., fingerprint comparison) is performed in the index library to find the song corresponding to the recording to be retrieved and the position of the query audio in the target song. It should be supplemented here that the usual audio fingerprint is used for music retrieval scenarios such as identifying songs by listening to them. Since it can not only find the corresponding song but also accurately return the position of the query audio in the target song, it can also be used to align the two audio materials A1 and A2.

[0196] In an exemplary embodiment, in the process of aligning the first original audio data and the second original audio data, it is first necessary to detect the energy maximum point in the audio spectrogram as a feature point to construct an audio fingerprint. Specifically, firstly, a short time Fourier transform (STFT) is performed on the audio signal (that is, the first original audio data and the second original audio data) to convert the one-dimensional time domain audio signal into a two-dimensional spectrogram to obtain the first original spectrogram and the second original spectrogram; wherein, the reference Fig.10 As shown, in the first original raw spectrum and the second original spectrogram, the horizontal axis is the time axis, with audio frames as units; the vertical axis is the frequency axis, with frequency band numbers as units; and the value of a point p(t,f) in the spectrogram represents the signal strength of the f-band at the t-frame moment. Secondly, after completing the short-time Fourier transform, the local peak point can be found in the two-dimensional spectrogram, that is: if the value of a point in the spectrogram is greater than all the points around it (such as a rectangular area around it), then the point is a local peak point; the local peak point obtained can refer to Fig.11 As shown; then, based on all the peak points found in the spectrogram, a fingerprint is constructed; for example, the time frame t and the frequency band number f of the peak point can be directly used as the fingerprint corresponding to the current peak point; further, in the alignment process, the position where the peak points overlap the most and the number of overlapping peak points can be found for alignment; at the same time, the position recorded here is the time position of the alignment, and the overlapping peak points are the scores of the fingerprint search; the specific matching process can be referred to Fig.12 as well as Fig.13As shown; it is also necessary to explain here that, during the alignment process, the first original spectrogram of A1 can be scanned from front to back based on the second local peak point in the second audio fingerprint data to find the position with the most overlap with the peak point fingerprint of A2, which is the alignment position of the two; or, the second original spectrogram of A2 can be scanned from front to back based on the first local peak point in the first audio fingerprint data to find the position with the most overlap with the peak point fingerprint of A1, which is the alignment position of the two; further, when the first original audio data and the second original audio data are aligned, t seconds segments can be intercepted as needed, and at this time, it can be guaranteed that the two segments intercepted from A1 and A2 are time-aligned.

[0197] In an exemplary embodiment, in the process of extracting the first audio data to be compared and the second audio data to be compared from the first original audio data and the second original audio data, it is necessary to intercept a time domain audio signal of a fixed length of t seconds from the first original audio data and the second original audio data as the input of this link; when intercepting, it is necessary to ensure that A1 and A2 are intercepted from the same position to ensure that the two are time-aligned; for example, a starting position can be randomly sampled in [0,dur-t], and a t-second segment can be intercepted from this position; for another example, it is also possible to start from 0 seconds and intercept t-second segments every δ seconds. This approach can intercept multiple t seconds to obtain the first audio data to be compared and the second audio data to be compared. It should also be noted here that one copy of the first audio data to be compared and the second audio data to be compared can be extracted from the first original audio data and the second original audio data, or multiple copies of the first audio data to be compared and the second audio data to be compared can be extracted, and this example does not impose any special restrictions on this.

[0198] In step S120, the first audio data to be compared and the second audio data to be compared are input into a preset audio data processing model to obtain a sound quality difference result between the first audio data to be compared and the second audio data to be compared.

[0199] Specifically, the specific determination process of the sound quality score difference result can be achieved in the following way: input the first audio data to be compared and the second audio data to be compared into the sound quality feature extraction model to obtain the first audio feature corresponding to the first audio data to be compared and the second audio feature corresponding to the second audio data to be compared; combine the first audio feature and the second audio feature based on the feature combination layer to obtain the sound quality combination feature, and input the sound quality combination feature into the sound quality score difference prediction model to obtain the sound quality score difference result between the first audio data to be compared and the second audio data to be compared. That is, in the actual application process, in order to obtain the sound quality score difference result, three processes such as audio feature extraction, audio feature combination and sound quality score difference prediction need to be performed. Below, the specific implementation process of audio feature extraction, audio feature combination and sound quality score difference prediction will be further explained and illustrated respectively. Specifically:

[0200] In an exemplary embodiment, the specific extraction process of the first audio feature and the second audio feature can be achieved in the following manner: performing a short-time Fourier transform on the first audio data to be compared to obtain a first spectrogram to be compared, and performing a short-time Fourier transform on the second audio data to be compared to obtain a second spectrogram to be compared; inputting the first spectrogram to be compared into the first feature extraction model to obtain a first audio feature corresponding to the first audio data to be compared; and inputting the second spectrogram to be compared into the second feature extraction model to obtain a second audio feature corresponding to the second audio data to be compared. Specifically, the specific determination process of the spectrogram to be compared can be achieved in the following manner: short-time Fourier Transform (STFT) analysis is performed on the audio data to be compared. For example, a 2048-point FFT, a 512-point hop size, and a Hanning window can be used to perform STFT spectrum analysis to obtain a spectrogram to be compared; at the same time, after short-time Fourier transform, the time domain signal of t seconds (that is, the first audio data to be compared and the second audio data to be compared) can be transformed into a two-dimensional spectrum diagram of a fixed size (that is, the first spectrogram to be compared and the second spectrogram to be compared); wherein, in the obtained first spectrogram to be compared and the second spectrogram to be compared, the horizontal axis can be the number of frames, and the vertical axis can be the number of frequency bands.

[0201] In an exemplary embodiment, after obtaining the first spectrogram to be compared and the second spectrogram to be compared, the first spectrogram to be compared can be input into the first feature extraction model to obtain the first audio feature corresponding to the first audio data to be compared, and the second spectrogram to be compared can be input into the second feature extraction model to obtain the second audio feature corresponding to the second audio data to be compared. It should be supplemented here that since the first feature extraction model and the second feature extraction model are twin network models, and the two have the same model structure and share model parameters; therefore, the specific extraction process of the first audio feature and the second audio feature is consistent.

[0202] Here, taking the extraction process of the first audio feature as an example and the first feature extraction model as the Transformer model as an example, the specific extraction process is further explained and illustrated. Specifically, the specific extraction process of the first audio feature can be implemented in the following manner: embedding and mapping the first spectrogram to be compared based on the first embedding mapping layer to obtain a first embedding vector, and encoding the first spectrogram to be compared based on the position encoding module to obtain a first position vector; superimposing the first embedding vector and the first position vector to obtain a first input vector, and encoding the first input vector based on the feature encoding module to obtain a first encoding result; decoding the first encoding result based on the feature decoding module to obtain a first encoding matrix, transforming the first encoding matrix based on the linear transformation layer to obtain a first logical matrix, and mapping the first logical matrix based on the Softmax layer to obtain a first audio feature corresponding to the first audio data to be compared. Among them, encoding the first input vector based on the feature encoding module to obtain the first encoding result can be achieved in the following ways: calculating the first attention mechanism of the first input vector based on the first multi-head self-attention module, and performing residual connection and normalization processing on the first attention mechanism based on the first residual connection and normalization module to obtain the first normalization processing result; linearizing the first normalization processing result based on the first feedforward neural network to obtain the first linearization processing result, and performing residual connection and normalization processing on the first linearization processing result based on the second residual connection and normalization module to obtain the first encoding result. Further, calculating the first attention mechanism of the first input vector based on the first multi-head self-attention module can be achieved in the following ways: linearly fusing the first input vector to obtain the first query vector, the first key vector and the first value vector, and calculating the first outer product vector of the first query vector and the first key vector to obtain the first similarity between the first query vector and the first key vector; normalizing the first similarity to obtain the first weight matrix, and calculating the second outer product vector of the first weight matrix and the first value vector to obtain the first attention mechanism.

[0203] Here, taking the extraction process of the first audio feature as an example and the first feature extraction model as a deep convolutional neural network model as an example, the specific extraction process is further explained and illustrated. Specifically, the specific extraction process of the first audio feature can be implemented in the following way: based on the first convolutional pooling layer, the second convolutional pooling layer, the third convolutional pooling layer, the fourth convolutional pooling layer and the fifth convolutional pooling layer, the first spectrogram to be compared is convolutionally pooled in sequence; then, the first spectrogram to be compared after convolutional pooling is classified based on the fully connected layer to obtain the first audio feature; at the same time, since the pooling layer involved in the deep convolutional neural network model can include global pooling or local pooling; therefore, if the pooling layer used here is global pooling, the first audio feature obtained is a fixed-dimensional vector e; if Global Pooling is not used, the first audio feature obtained is a fixed-dimensional vector sequence [e1,e2,...e n ]; In the actual application process, you can select it according to your actual needs, and this example does not impose any special restrictions on this.

[0204] In an exemplary embodiment, the first audio feature and the second audio feature are combined based on the feature combination layer to obtain a sound quality combination feature, which can be achieved in the following ways: one way is to calculate the difference between the first audio feature and the second audio feature to obtain a feature difference calculation result, and determine the sound quality combination feature based on the feature difference calculation result; another way of implementation is to splice the feature difference calculation result, the first audio feature and the second audio feature to obtain the sound quality combination feature; and yet another way of implementation is to splice the first audio feature and the second audio feature to obtain the sound quality combination feature.

[0205] In an exemplary embodiment, the sound quality combination feature is input into the sound quality score difference prediction model to obtain the sound quality score difference result between the first audio data to be compared and the second audio data to be compared, which can be achieved in the following manner: Specifically, taking the sound quality score difference prediction model as a multi-layer perceptron as an example, the sound quality combination feature can be forward propagated based on the hidden layer included in the multi-layer perceptron to obtain the corresponding sound quality score difference result. The specific expression form of the obtained sound quality score difference result is: S_diff = Score (A1) - Score (A2).

[0206] In step S130, it is determined whether the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data according to the sound quality difference result.

[0207] Specifically, the specific determination process of the first audio quality and the second audio quality is: if the sound quality difference result is greater than or equal to a preset threshold, it is determined that the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data; if the sound quality difference result is less than the preset threshold, it is determined that the first audio quality of the first original audio data is lower than the second audio quality of the second original audio data; wherein, the larger the value of the sound quality difference result, the higher the first audio quality of the first original audio data; the smaller the value of the sound quality difference result, the higher the second audio quality of the second original audio data.

[0208] Finally, after obtaining the first audio quality and the second audio quality, the audio data processing method further includes: if the first audio quality is higher than the second audio quality, the first original audio data is used as the target audio data of the song corresponding to the original audio data; if the second audio quality is higher than the first audio quality, the second original audio data is used as the target audio data of the song corresponding to the original audio data. At the same time, after the target audio data is determined, the low-quality audio data can be deleted. Based on this method, the user experience of the user in the process of listening to songs can be further improved.

[0209] So far, the audio data processing method recorded in the exemplary embodiment of the present disclosure has been fully realized. Based on the above-mentioned content, it can be known that the audio data processing method recorded in the exemplary embodiment of the present disclosure can automatically construct a training data set without too much manual participation to complete the training of the sound quality comparison model; at the same time, when used online, for two audio materials of the same song, after completing the audio fingerprint alignment, they can be sent to the sound quality comparison model to predict the sound quality difference between the two; and the sound quality difference reflects the sound quality of the two audio materials and the degree of sound quality difference at the signal level, which can be used for music library management, false sound quality song detection and other scenarios, bringing great convenience to music library management. Further, the prediction of the sound quality difference recorded in the exemplary embodiment of the present disclosure is a judgment of the sound quality of the two inputs and the degree of sound quality difference. It is easier to learn than directly predicting the absolute value of the sound quality, and it also provides richer information such as the degree of difference than just predicting the sound quality, which is convenient for selective processing according to the degree of sound quality difference in subsequent applications.

[0210] Exemplary Devices

[0211] After introducing the method for processing audio data according to the exemplary embodiment of the present disclosure, Fig.14 An audio data processing apparatus according to an exemplary embodiment of the present disclosure is explained and described.

[0212] Specifically, refer to Fig.14 As shown, the audio data processing device may include an audio data extraction module 1410, a sound quality difference result determination module 1420, and an audio quality determination module 1430. Among them:

[0213] The audio data extraction module 1410 may be used to obtain the first original audio data and the second original audio data, and extract the first audio data to be compared and the second audio data to be compared from the first original audio data and the second original audio data;

[0214] The sound quality difference result determination module 1420 may be used to input the first audio data to be compared and the second audio data to be compared into a preset audio data processing model to obtain a sound quality difference result between the first audio data to be compared and the second audio data to be compared;

[0215] The audio quality determination module 1430 may be configured to determine a first audio quality of the first audio data to be compared and a second audio quality of the second audio data to be compared according to the sound quality difference result.

[0216] In an exemplary embodiment of the present disclosure, extracting first audio data to be compared and second audio data to be compared from first original audio data and second original audio data includes: extracting first audio fingerprint data of the first original audio data and second audio fingerprint data of the second original audio data; wherein the first original audio data and the second original audio data are audio data of different bit rates corresponding to the same song; performing audio alignment processing on the first original audio data and the second original audio data according to the first audio fingerprint data and the second audio fingerprint data to obtain a first audio alignment result and a second audio alignment result; intercepting the first audio data to be compared from the first audio alignment result, and intercepting the second audio data to be compared from the second audio alignment result; wherein a first audio position of the first audio data to be compared in the first audio alignment result is the same as a second audio position of the second audio data to be compared in the second audio alignment result.

[0217] In an exemplary embodiment of the present disclosure, extracting first audio fingerprint data of first original audio data includes: performing a short-time Fourier transform on the first original audio data to obtain a first original spectrogram, traversing the first original spectrogram, and extracting a first local peak point from the first original spectrogram; constructing the first audio fingerprint data according to a first time series position of the first local peak point in the first original spectrogram and a first frequency band sequence number of the first local peak point.

[0218] In an exemplary embodiment of the present disclosure, audio alignment processing is performed on first original audio data and second original audio data according to the first audio fingerprint data and the second audio fingerprint data to obtain a first audio alignment result and a second audio alignment result, including: traversing a first original sound spectrogram corresponding to the first original audio data, and searching the first original sound spectrogram for a position having the most overlapping part with a second local peak point in the second audio fingerprint data as the audio alignment position of the first original audio data and the second original audio data; and / or traversing a second original sound spectrogram corresponding to the second original audio data, and searching the second original sound spectrogram for a position having the most overlapping part with a first local peak point in the first audio fingerprint data as the audio alignment position of the first original audio data and the second original audio data; performing audio alignment processing on the first original audio data and the second original audio data based on the audio alignment position to obtain a first audio alignment result and a second audio alignment result.

[0219] In an exemplary embodiment of the present disclosure, the preset audio data processing model includes a sound quality feature extraction model, a feature combination layer and a sound quality score difference prediction model; wherein, the first audio data to be compared and the second audio data to be compared are input into the preset audio data processing model to obtain the sound quality score difference result between the first audio data to be compared and the second audio data to be compared, including: inputting the first audio data to be compared and the second audio data to be compared into the sound quality feature extraction model to obtain the first audio feature corresponding to the first audio data to be compared and the second audio feature corresponding to the second audio data to be compared; combining the first audio feature and the second audio feature based on the feature combination layer to obtain the sound quality combination feature, and inputting the sound quality combination feature into the sound quality score difference prediction model to obtain the sound quality score difference result between the first audio data to be compared and the second audio data to be compared.

[0220] In an exemplary embodiment of the present disclosure, the preset sound quality feature extraction model includes a first feature extraction model and a second feature extraction model, and the first feature extraction model and the second feature extraction model share model parameters; wherein, the first audio data to be compared and the second audio data to be compared are input into the sound quality feature extraction model to obtain a first audio feature corresponding to the first audio data to be compared and a second audio feature corresponding to the second audio data to be compared, including: performing a short-time Fourier transform on the first audio data to be compared to obtain a first spectrogram to be compared, and performing a short-time Fourier transform on the second audio data to be compared to obtain a second spectrogram to be compared; inputting the first spectrogram to be compared into the first feature extraction model to obtain a first audio feature corresponding to the first audio data to be compared; and inputting the second spectrogram to be compared into the second feature extraction model to obtain a second audio feature corresponding to the second audio data to be compared.

[0221] In an exemplary embodiment of the present disclosure, the first feature extraction model includes a first embedding mapping layer, a position encoding module, a feature encoding module, a feature decoding module, a linear transformation layer and a Softmax layer; wherein, the first spectrogram to be compared is input into the first feature extraction model to obtain a first audio feature corresponding to the first audio data to be compared, including: embedding and mapping the first spectrogram to be compared based on the first embedding mapping layer to obtain a first embedding vector, and encoding the first spectrogram to be compared based on the position encoding module to obtain a first position vector; superimposing the first embedding vector and the first position vector to obtain a first input vector, and encoding the first input vector based on the feature encoding module to obtain a first encoding result; decoding the first encoding result based on the feature decoding module to obtain a first encoding matrix, converting the first encoding matrix based on the linear transformation layer to obtain a first logical matrix, and mapping the first logical matrix based on the Softmax layer to obtain a first audio feature corresponding to the first audio data to be compared.

[0222] In an exemplary embodiment of the present disclosure, the feature encoding module includes a first multi-head self-attention module, a first residual connection and normalization module, a first feedforward neural network, and a second residual connection and normalization module; wherein, encoding the first input vector based on the feature encoding module to obtain a first encoding result includes: calculating the first attention mechanism of the first input vector based on the first multi-head self-attention module, and performing residual connection and normalization processing on the first attention mechanism based on the first residual connection and normalization module to obtain a first normalization processing result; linearizing the first normalization processing result based on the first feedforward neural network to obtain a first linearization processing result, and performing residual connection and normalization processing on the first linearization processing result based on the second residual connection and normalization module to obtain a first encoding result.

[0223] In an exemplary embodiment of the present disclosure, a first attention mechanism for calculating the first input vector is calculated based on a first multi-head self-attention module, including: linearly fusing the first input vector to obtain a first query vector, a first key vector and a first value vector, and calculating a first outer product vector of the first query vector and the first key vector to obtain a first similarity between the first query vector and the first key vector; normalizing the first similarity to obtain a first weight matrix, and calculating a second outer product vector of the first weight matrix and the first value vector to obtain the first attention mechanism.

[0224] In an exemplary embodiment of the present disclosure, the first audio feature and the second audio feature are combined based on the feature combination layer to obtain a sound quality combination feature, including: calculating the difference between the first audio feature and the second audio feature to obtain a feature difference calculation result, and determining the sound quality combination feature based on the feature difference calculation result; and / or splicing the feature difference calculation result, the first audio feature and the second audio feature to obtain the sound quality combination feature; and / or splicing the first audio feature and the second audio feature to obtain the sound quality combination feature.

[0225] In an exemplary embodiment of the present disclosure, determining whether the first audio quality of first original audio data is higher than the second audio quality of second original audio data based on a sound quality difference result includes: if the sound quality difference result is greater than or equal to a preset threshold, determining that the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data; if the sound quality difference result is less than the preset threshold, determining that the first audio quality of the first original audio data is lower than the second audio quality of the second original audio data; wherein, the larger the value of the sound quality difference result, the higher the first audio quality of the first original audio data; the smaller the value of the sound quality difference result, the higher the second audio quality of the second original audio data.

[0226] In an exemplary embodiment of the present disclosure, the audio data processing device further includes:

[0227] a first target audio data determination module, configured to use the first original audio data as target audio data of a song corresponding to the original audio data if the first audio quality is higher than the second audio quality;

[0228] A second target audio data determination module is used to use the second original audio data as target audio data of a song corresponding to the original audio data if the second audio quality is higher than the first audio quality.

[0229] In an exemplary embodiment of the present disclosure, the preset audio data processing model is obtained in the following manner:

[0230] Acquire audio data of lossless sound quality of historical songs, and encode and compress the audio data of lossless sound quality at different bit rates based on a preset encoder to obtain an audio encoding result;

[0231] Randomly pairing the audio encoding result and the audio data with lossless sound quality to obtain multiple audio pairing results, and determining actual sound quality differences of the audio pairing results;

[0232] Inputting each of the audio pairing results into the neural network model to be trained to obtain a predicted sound quality score difference, and constructing a target loss function according to the actual sound quality score difference and the predicted sound quality score difference;

[0233] Based on the target loss function, the parameters of the neural network model to be trained are adjusted to obtain the preset audio data processing model.

[0234] Exemplary Storage Media

[0235] After introducing the audio data processing method and the audio data processing device according to the exemplary embodiment of the present disclosure, next, referring to Fig.15 A storage medium according to an exemplary embodiment of the present disclosure is described.

[0236] refer to Fig.15 As shown, a program product 1500 for implementing the above method according to an embodiment of the present disclosure is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto.

[0237] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0238] Computer readable signal media may include a data signal propagated in baseband or as a part of a carrier wave, wherein a readable program code is carried. The data signal of this propagation may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium. The program code for performing the disclosed operation may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also including conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed completely on a user computing device, partially on a user device, partially on a remote computing device, or completely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN).

[0239] Exemplary Electronic Devices

[0240] After introducing the storage medium of the exemplary embodiment of the present disclosure, next, reference is made to Fig.16 An electronic device according to an exemplary embodiment of the present disclosure is described.

[0241] Fig.16The electronic device 1600 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0242] like Fig.16 As shown, the electronic device 1600 is in the form of a general computing device. The components of the electronic device 1600 may include, but are not limited to: the at least one processing unit 1610, the at least one storage unit 1620, a bus 1630 connecting different system components (including the storage unit 1620 and the processing unit 1610), and a display unit 1640.

[0243] The storage unit 1620 stores program codes, which can be executed by the processing unit 1610, so that the processing unit 1610 performs the steps described in the above “Exemplary Method” section of this specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 1610 can perform the following steps: Figure 1 Steps S110-S130 shown in FIG.

[0244] The storage unit 1620 may include a volatile storage unit, such as a random access storage unit (RAM) 16201 and / or a cache storage unit 16202 , and may further include a read-only storage unit (ROM) 16203 .

[0245] The storage unit 1620 may also include a program / utility 16204 having a set (at least one) of program modules 16205, such program modules 16205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0246] The bus 1630 may include a data bus, an address bus, and a control bus.

[0247] The electronic device 1600 may also communicate with one or more external devices 1700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.) via an input / output (I / O) interface 1650. Furthermore, the electronic device 1600 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1660. As shown, the network adapter 1660 communicates with other modules of the electronic device 1600 via a bus 1630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0248] It should be noted that although several modules or submodules of the pop-up window processing device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be concretized in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules for concretization.

[0249] In addition, although the operations of the disclosed method are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0250] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. A method for processing audio data, characterized in that: include: Acquire first original audio data and second original audio data, and extract first audio data to be compared and second audio data to be compared from the first original audio data and the second original audio data; Inputting the first audio data to be compared and the second audio data to be compared into a preset audio data processing model to obtain a sound quality difference result between the first audio data to be compared and the second audio data to be compared; Determine whether a first audio quality of the first original audio data is higher than a second audio quality of the second original audio data according to the sound quality difference result.

2. The method for processing audio data according to claim 1, characterized in that: Extracting first audio data to be compared and second audio data to be compared from the first original audio data and the second original audio data includes: Extracting first audio fingerprint data of the first original audio data and second audio fingerprint data of the second original audio data; wherein the first original audio data and the second original audio data are audio data of different bit rates corresponding to the same song; Performing audio alignment processing on the first original audio data and the second original audio data according to the first audio fingerprint data and the second audio fingerprint data to obtain a first audio alignment result and a second audio alignment result; The first audio data to be compared is intercepted from the first audio alignment result, and the second audio data to be compared is intercepted from the second audio alignment result; wherein the first audio position of the first audio data to be compared in the first audio alignment result is the same as the second audio position of the second audio data to be compared in the second audio alignment result.

3. The method for processing audio data according to claim 2, characterized in that: Extracting first audio fingerprint data from the first original audio data includes: Performing a short-time Fourier transform on the first original audio data to obtain a first original spectrogram, and traversing the first original spectrogram to extract a first local peak point from the first original spectrogram; The first audio fingerprint data is constructed according to a first time series position of the first local peak point in the first original spectrogram and a first frequency band sequence number of the first local peak point.

4. The method for processing audio data according to claim 2, characterized in that: Performing audio alignment processing on the first original audio data and the second original audio data according to the first audio fingerprint data and the second audio fingerprint data to obtain a first audio alignment result and a second audio alignment result, including: Traversing a first original spectrogram corresponding to the first original audio data, searching the first original spectrogram for a position having the most overlap with a second local peak point in the second audio fingerprint data as an audio alignment position of the first original audio data and the second original audio data; and / or Traversing a second original spectrogram corresponding to the second original audio data, searching the second original spectrogram for a position having the most overlap with a first local peak point in the first audio fingerprint data as an audio alignment position of the first original audio data and the second original audio data; Audio alignment processing is performed on the first original audio data and the second original audio data based on the audio alignment position to obtain a first audio alignment result and a second audio alignment result.

5. The method for processing audio data according to claim 1, characterized in that: The preset audio data processing model includes a sound quality feature extraction model, a feature combination layer and a sound quality score difference prediction model; The first audio data to be compared and the second audio data to be compared are input into a preset audio data processing model to obtain a sound quality difference result between the first audio data to be compared and the second audio data to be compared, including: Inputting the first audio data to be compared and the second audio data to be compared into the sound quality feature extraction model to obtain a first audio feature corresponding to the first audio data to be compared and a second audio feature corresponding to the second audio data to be compared; The first audio feature and the second audio feature are combined based on the feature combination layer to obtain a sound quality combination feature, and the sound quality combination feature is input into a sound quality score difference prediction model to obtain a sound quality score difference result between the first audio data to be compared and the second audio data to be compared.

6. The method for processing audio data according to claim 5, characterized in that: The preset sound quality feature extraction model includes a first feature extraction model and a second feature extraction model, and the first feature extraction model and the second feature extraction model share model parameters; The first audio data to be compared and the second audio data to be compared are input into the sound quality feature extraction model to obtain a first audio feature corresponding to the first audio data to be compared and a second audio feature corresponding to the second audio data to be compared, including: Performing a short-time Fourier transform on the first audio data to be compared to obtain a first spectrogram to be compared, and performing a short-time Fourier transform on the second audio data to be compared to obtain a second spectrogram to be compared; Inputting the first spectrogram to be compared into the first feature extraction model to obtain a first audio feature corresponding to the first audio data to be compared; The second spectrogram to be compared is input into the second feature extraction model to obtain a second audio feature corresponding to the second audio data to be compared.

7. The method for processing audio data according to claim 6, characterized in that: The first feature extraction model includes a first embedding mapping layer, a position encoding module, a feature encoding module, a feature decoding module, a linear transformation layer and a Softmax layer; The step of inputting the first spectrogram to be compared into the first feature extraction model to obtain a first audio feature corresponding to the first audio data to be compared includes: Performing embedding mapping processing on the first to-be-compared sound spectrogram based on the first embedding mapping layer to obtain a first embedding vector, and performing encoding processing on the first to-be-compared sound spectrogram based on the position encoding module to obtain a first position vector; Superimposing the first embedding vector and the first position vector to obtain a first input vector, and encoding the first input vector based on a feature encoding module to obtain a first encoding result; The first encoding result is decoded based on the feature decoding module to obtain a first encoding matrix, the first encoding matrix is ​​transformed based on the linear transformation layer to obtain a first logical matrix, and the first logical matrix is ​​mapped based on the Softmax layer to obtain a first audio feature corresponding to the first audio data to be compared.

8. An audio data processing device, characterized in that: include: An audio data extraction module, used to obtain the first original audio data and the second original audio data, and extract the first audio data to be compared and the second audio data to be compared from the first original audio data and the second original audio data; A sound quality difference result determination module, used for inputting the first audio data to be compared and the second audio data to be compared into a preset audio data processing model to obtain a sound quality difference result between the first audio data to be compared and the second audio data to be compared; The audio quality determination module is used to determine the first audio quality of the first audio data to be compared and the second audio quality of the second audio data to be compared according to the sound quality difference result.

9. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method for processing audio data according to any one of claims 1 to 7.

10. An electronic device comprising: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the method for processing audio data according to any one of claims 1 to 7 by executing the executable instructions.

Images