Method and apparatus for processing audio data, storage medium, and electronic device

Abstract

The present disclosure relates to a processing method and device of audio data, a storage medium and an electronic device, and relates to the technical field of computers. The method comprises: obtaining first original audio data and second original audio data, and extracting first to-be-compared audio data and second to-be-compared audio data from the first original audio data and the second original audio data; inputting the first to-be-compared audio data and the second to-be-compared audio data into a preset audio data processing model to obtain a sound quality difference result between the first to-be-compared audio data and the second to-be-compared audio data; and determining whether a first audio quality of the first original audio data is higher than a second audio quality of the second original audio data according to the sound quality difference result. The present disclosure improves the accuracy of the sound quality comparison result.

Description

Technical Field

[0001] The embodiments of this disclosure relate to the field of computer technology, and more specifically, the embodiments of this disclosure relate to an audio data processing method, an audio data processing apparatus, a computer-readable storage medium, and an electronic device. Background Technology

[0002] This section is intended to provide background or context for the embodiments of this disclosure as set forth in the claims. The description herein is not intended to be a prior art simply because it is included in this section.

[0003] In some audio data quality comparison schemes, the sound quality of two encoded compressed materials of the same song can be directly compared; however, considering the possibility of false sound quality, if only the encoding format / bitrate of the audio data is compared, the accuracy of the sound quality comparison results will be low. Summary of the Invention

[0004] However, in the existing technology, on the one hand, judging the sound quality of audio data directly by the encoding format / encoding bitrate of the audio data has the problem of low accuracy of the sound quality comparison results; on the other hand, since it is necessary to compare the encoding format and encoding bitrate of all audio data, the comparison efficiency is low.

[0005] Therefore, there is a great need for an improved audio data processing method. This method involves acquiring first and second original audio data, extracting first and second audio data to be compared from these two sets of data, and then inputting them into a preset audio data processing model to obtain the sound quality difference between them. Finally, based on the sound quality difference, it is determined whether the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data. This improves both the accuracy and efficiency of audio quality comparison.

[0006] In this context, embodiments of the present disclosure are intended to provide an audio data processing method, an audio data processing apparatus, a computer-readable storage medium, and an electronic device.

[0007] According to one aspect of this disclosure, a method for processing audio data is provided, comprising:

[0008] Acquire first raw audio data and second raw audio data, and extract first audio data to be compared and second audio data to be compared from the first raw audio data and second raw audio data;

[0009] The first audio data to be compared and the second audio data to be compared are input into a preset audio data processing model to obtain the sound quality difference result between the first audio data to be compared and the second audio data to be compared.

[0010] Based on the sound quality difference result, determine whether the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data.

[0011] In one exemplary embodiment of this disclosure, extracting first audio data to be compared and second audio data to be compared from the first original audio data and the second original audio data includes:

[0012] Extract the first audio fingerprint data of the first original audio data and the second audio fingerprint data of the second original audio data; wherein, the first original audio data and the second original audio data are audio data of different bit rates corresponding to the same song;

[0013] Based on the first audio fingerprint data and the second audio fingerprint data, audio alignment processing is performed on the first original audio data and the second original audio data to obtain the first audio alignment result and the second audio alignment result.

[0014] The first audio data to be compared is extracted from the first audio alignment result, and the second audio data to be compared is extracted from the second audio alignment result; wherein the first audio position of the first audio data to be compared in the first audio alignment result is the same as the second audio position of the second audio data to be compared in the second audio alignment result.

[0015] In one exemplary embodiment of this disclosure, extracting first audio fingerprint data from first raw audio data includes:

[0016] Perform a short-time Fourier transform on the first original audio data to obtain a first original spectrogram, and traverse the first original spectrogram to extract the first local peak point from the first original spectrogram.

[0017] The first audio fingerprint data is constructed based on the first time sequence position of the first local peak point in the first original spectrogram and the first frequency band number of the first local peak point.

[0018] In one exemplary embodiment of this disclosure, audio alignment processing is performed on first original audio data and second original audio data based on the first audio fingerprint data and the second audio fingerprint data to obtain a first audio alignment result and a second audio alignment result, including:

[0019] Traverse the first original spectrogram corresponding to the first original audio data, and find the position in the first original spectrogram that has the most overlap with the second local peak point in the second audio fingerprint data, using this position as the audio alignment position for the first original audio data and the second original audio data; and / or

[0020] Traverse the second original spectrogram corresponding to the second original audio data, and find the position in the second original spectrogram that has the most overlap with the first local peak point in the first audio fingerprint data, and use it as the audio alignment position of the first original audio data and the second original audio data.

[0021] Based on the audio alignment position, the first original audio data and the second original audio data are subjected to audio alignment processing to obtain the first audio alignment result and the second audio alignment result.

[0022] In one exemplary embodiment of this disclosure, the preset audio data processing model includes a sound quality feature extraction model, a feature combination layer, and a sound quality difference prediction model;

[0023] Specifically, the first and second audio data to be compared are input into a preset audio data processing model to obtain the sound quality difference result between the first and second audio data to be compared, including:

[0024] The first audio data to be compared and the second audio data to be compared are input into the sound quality feature extraction model to obtain the first audio feature corresponding to the first audio data to be compared and the second audio feature corresponding to the second audio data to be compared.

[0025] The first audio feature and the second audio feature are combined based on the feature combination layer to obtain the sound quality combination feature. The sound quality combination feature is then input into the sound quality difference prediction model to obtain the sound quality difference result between the first audio data to be compared and the second audio data to be compared.

[0026] In an exemplary embodiment of this disclosure, the preset sound quality feature extraction model includes a first feature extraction model and a second feature extraction model, wherein the first feature extraction model and the second feature extraction model share model parameters;

[0027] Specifically, the first and second audio data to be compared are input into the sound quality feature extraction model to obtain a first audio feature corresponding to the first audio data to be compared and a second audio feature corresponding to the second audio data to be compared, including:

[0028] The first audio data to be compared is subjected to a short-time Fourier transform to obtain a first spectrogram of the comparison, and the second audio data to be compared is subjected to a short-time Fourier transform to obtain a second spectrogram of the comparison.

[0029] The first spectrogram to be compared is input into the first feature extraction model to obtain the first audio feature corresponding to the first audio data to be compared;

[0030] The second spectrogram to be compared is input into the second feature extraction model to obtain the second audio feature corresponding to the second audio data to be compared.

[0031] In one exemplary embodiment of this disclosure, the first feature extraction model includes a first embedding mapping layer, a position encoding module, a feature encoding module, a feature decoding module, a linear transformation layer, and a Softmax layer;

[0032] Specifically, inputting the first spectrogram to be compared into the first feature extraction model to obtain the first audio feature corresponding to the first audio data to be compared includes:

[0033] The first spectrogram to be compared is embedded and mapped based on the first embedding mapping layer to obtain a first embedding vector, and the first spectrogram to be compared is encoded based on the position encoding module to obtain a first position vector.

[0034] The first embedding vector and the first position vector are superimposed to obtain the first input vector, and the first input vector is encoded based on the feature encoding module to obtain the first encoding result;

[0035] The first encoding result is decoded by the feature decoding module to obtain the first encoding matrix. The first encoding matrix is ​​then transformed by the linear transformation layer to obtain the first logical matrix. Finally, the first logical matrix is ​​mapped by the Softmax layer to obtain the first audio feature corresponding to the first audio data to be compared.

[0036] In one exemplary embodiment of this disclosure, the feature encoding module includes a first multi-head self-attention module, a first residual connection and normalization module, a first feedforward neural network, and a second residual connection and normalization module;

[0037] The first input vector is encoded using a feature encoding module to obtain a first encoding result, including:

[0038] The first attention mechanism is calculated based on the first multi-head self-attention module, and the first attention mechanism is subjected to residual connection and normalization processing based on the first residual connection and normalization module to obtain the first normalization processing result.

[0039] The first normalization result is linearized based on the first feedforward neural network to obtain the first linearized result. The first linearized result is then subjected to residual connection and normalization based on the second residual connection and normalization module to obtain the first encoding result.

[0040] In one exemplary embodiment of this disclosure, a first attention mechanism for calculating the first input vector based on a first multi-head self-attention module includes:

[0041] The first input vector is linearly fused to obtain a first query vector, a first key vector, and a first value vector. The first outer product vector of the first query vector and the first key vector is calculated to obtain the first similarity between the first query vector and the first key vector.

[0042] The first similarity is normalized to obtain the first weight matrix, and the second outer product vector of the first weight matrix and the first value vector is calculated to obtain the first attention mechanism.

[0043] In one exemplary embodiment of this disclosure, the first audio feature and the second audio feature are combined based on the feature combination layer to obtain a sound quality combination feature, including:

[0044] Calculate the difference between the first audio feature and the second audio feature to obtain the feature difference calculation result, and determine the sound quality combination feature based on the feature difference calculation result; and / or

[0045] The feature difference calculation result, the first audio feature, and the second audio feature are combined to obtain the sound quality combination feature; and / or

[0046] The first audio feature and the second audio feature are combined to obtain the sound quality combination feature.

[0047] In one exemplary embodiment of this disclosure, determining whether the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data based on the audio quality difference result includes:

[0048] If the sound quality difference result is greater than or equal to a preset threshold, then it is determined that the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data.

[0049] If the sound quality difference result is less than a preset threshold, then it is determined that the first audio quality of the first original audio data is lower than the second audio quality of the second original audio data.

[0050] The larger the value of the sound quality difference result, the higher the first audio quality of the first original audio data; the smaller the value of the sound quality difference result, the higher the second audio quality of the second original audio data.

[0051] In one exemplary embodiment of this disclosure, the method for processing the audio data further includes:

[0052] If the first audio quality is higher than the second audio quality, then the first original audio data is used as the target audio data of the song corresponding to the original audio data.

[0053] If the second audio quality is higher than the first audio quality, then the second original audio data is used as the target audio data of the song corresponding to the original audio data.

[0054] In one exemplary embodiment of this disclosure, the preset audio data processing model is obtained in the following manner:

[0055] The lossless audio data of historical songs is acquired, and the lossless audio data is encoded and compressed at different bit rates based on a preset encoder to obtain the audio encoding result.

[0056] The audio encoding result and the lossless audio data are randomly paired to obtain multiple audio pairing results, and the actual audio quality difference of each audio pairing result is determined.

[0057] The audio pairing results are input into the neural network model to be trained to obtain the predicted sound quality difference, and a target loss function is constructed based on the actual sound quality difference and the predicted sound quality difference.

[0058] The parameters of the neural network model to be trained are adjusted based on the target loss function to obtain the preset audio data processing model.

[0059] According to one aspect of this disclosure, an audio data processing apparatus is provided, comprising:

[0060] The audio data extraction module is used to acquire first raw audio data and second raw audio data, and extract first audio data to be compared and second audio data to be compared from the first raw audio data and second raw audio data;

[0061] The audio quality difference result determination module is used to input the first audio data to be compared and the second audio data to be compared into a preset audio data processing model to obtain the audio quality difference result between the first audio data to be compared and the second audio data to be compared.

[0062] An audio quality determination module is used to determine the first audio quality of the first audio data to be compared and the second audio quality of the second audio data to be compared based on the audio quality difference result.

[0063] In one exemplary embodiment of this disclosure, extracting first audio data to be compared and second audio data to be compared from first original audio data and second original audio data includes:

[0064] Extract the first audio fingerprint data of the first original audio data and the second audio fingerprint data of the second original audio data; wherein, the first original audio data and the second original audio data are audio data of different bit rates corresponding to the same song;

[0065] Based on the first audio fingerprint data and the second audio fingerprint data, audio alignment processing is performed on the first original audio data and the second original audio data to obtain the first audio alignment result and the second audio alignment result.

[0066] The first audio data to be compared is extracted from the first audio alignment result, and the second audio data to be compared is extracted from the second audio alignment result; wherein the first audio position of the first audio data to be compared in the first audio alignment result is the same as the second audio position of the second audio data to be compared in the second audio alignment result.

[0067] In one exemplary embodiment of this disclosure, extracting first audio fingerprint data from first raw audio data includes:

[0068] Perform a short-time Fourier transform on the first original audio data to obtain a first original spectrogram, and traverse the first original spectrogram to extract the first local peak point from the first original spectrogram.

[0069] The first audio fingerprint data is constructed based on the first time sequence position of the first local peak point in the first original spectrogram and the first frequency band number of the first local peak point.

[0070] In one exemplary embodiment of this disclosure, audio alignment processing is performed on first original audio data and second original audio data based on the first audio fingerprint data and the second audio fingerprint data to obtain a first audio alignment result and a second audio alignment result, including:

[0071] Traverse the first original spectrogram corresponding to the first original audio data, and find the position in the first original spectrogram that has the most overlap with the second local peak point in the second audio fingerprint data, using this position as the audio alignment position for the first original audio data and the second original audio data; and / or

[0072] Traverse the second original spectrogram corresponding to the second original audio data, and find the position in the second original spectrogram that has the most overlap with the first local peak point in the first audio fingerprint data, and use it as the audio alignment position of the first original audio data and the second original audio data.

[0073] Based on the audio alignment position, the first original audio data and the second original audio data are subjected to audio alignment processing to obtain the first audio alignment result and the second audio alignment result.

[0074] In one exemplary embodiment of this disclosure, the preset audio data processing model includes a sound quality feature extraction model, a feature combination layer, and a sound quality difference prediction model;

[0075] Specifically, the first and second audio data to be compared are input into a preset audio data processing model to obtain the sound quality difference result between the first and second audio data to be compared, including:

[0076] The first audio data to be compared and the second audio data to be compared are input into the sound quality feature extraction model to obtain the first audio feature corresponding to the first audio data to be compared and the second audio feature corresponding to the second audio data to be compared.

[0077] The first audio feature and the second audio feature are combined based on the feature combination layer to obtain the sound quality combination feature. The sound quality combination feature is then input into the sound quality difference prediction model to obtain the sound quality difference result between the first audio data to be compared and the second audio data to be compared.

[0078] In an exemplary embodiment of this disclosure, the preset sound quality feature extraction model includes a first feature extraction model and a second feature extraction model, wherein the first feature extraction model and the second feature extraction model share model parameters;

[0079] Specifically, the first and second audio data to be compared are input into the sound quality feature extraction model to obtain a first audio feature corresponding to the first audio data to be compared and a second audio feature corresponding to the second audio data to be compared, including:

[0080] The first audio data to be compared is subjected to a short-time Fourier transform to obtain a first spectrogram of the comparison, and the second audio data to be compared is subjected to a short-time Fourier transform to obtain a second spectrogram of the comparison.

[0081] The first spectrogram to be compared is input into the first feature extraction model to obtain the first audio feature corresponding to the first audio data to be compared;

[0082] The second spectrogram to be compared is input into the second feature extraction model to obtain the second audio feature corresponding to the second audio data to be compared.

[0083] In one exemplary embodiment of this disclosure, the first feature extraction model includes a first embedding mapping layer, a position encoding module, a feature encoding module, a feature decoding module, a linear transformation layer, and a Softmax layer;

[0084] Specifically, inputting the first spectrogram to be compared into the first feature extraction model to obtain the first audio feature corresponding to the first audio data to be compared includes:

[0085] The first spectrogram to be compared is embedded and mapped based on the first embedding mapping layer to obtain a first embedding vector, and the first spectrogram to be compared is encoded based on the position encoding module to obtain a first position vector.

[0086] The first embedding vector and the first position vector are superimposed to obtain the first input vector, and the first input vector is encoded based on the feature encoding module to obtain the first encoding result;

[0087] The first encoding result is decoded by the feature decoding module to obtain the first encoding matrix. The first encoding matrix is ​​then transformed by the linear transformation layer to obtain the first logical matrix. Finally, the first logical matrix is ​​mapped by the Softmax layer to obtain the first audio feature corresponding to the first audio data to be compared.

[0088] In one exemplary embodiment of this disclosure, the feature encoding module includes a first multi-head self-attention module, a first residual connection and normalization module, a first feedforward neural network, and a second residual connection and normalization module;

[0089] The first input vector is encoded using a feature encoding module to obtain a first encoding result, including:

[0090] The first attention mechanism is calculated based on the first multi-head self-attention module, and the first attention mechanism is subjected to residual connection and normalization processing based on the first residual connection and normalization module to obtain the first normalization processing result.

[0091] The first normalization result is linearized based on the first feedforward neural network to obtain the first linearized result. The first linearized result is then subjected to residual connection and normalization based on the second residual connection and normalization module to obtain the first encoding result.

[0092] In one exemplary embodiment of this disclosure, a first attention mechanism for calculating the first input vector based on a first multi-head self-attention module includes:

[0093] The first input vector is linearly fused to obtain a first query vector, a first key vector, and a first value vector. The first outer product vector of the first query vector and the first key vector is calculated to obtain the first similarity between the first query vector and the first key vector.

[0094] The first similarity is normalized to obtain the first weight matrix, and the second outer product vector of the first weight matrix and the first value vector is calculated to obtain the first attention mechanism.

[0095] In an exemplary embodiment of this disclosure, the first audio feature and the second audio feature are combined based on the feature combination layer to obtain a sound quality combination feature, including:

[0096] Calculate the difference between the first audio feature and the second audio feature to obtain the feature difference calculation result, and determine the sound quality combination feature based on the feature difference calculation result; and / or

[0097] The feature difference calculation result, the first audio feature, and the second audio feature are combined to obtain the sound quality combination feature; and / or

[0098] The first audio feature and the second audio feature are combined to obtain the sound quality combination feature.

[0099] In one exemplary embodiment of this disclosure, determining whether the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data based on the audio quality difference result includes:

[0100] If the sound quality difference result is greater than or equal to a preset threshold, then it is determined that the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data.

[0101] If the sound quality difference result is less than a preset threshold, then it is determined that the first audio quality of the first original audio data is lower than the second audio quality of the second original audio data.

[0102] The larger the value of the sound quality difference result, the higher the first audio quality of the first original audio data; the smaller the value of the sound quality difference result, the higher the second audio quality of the second original audio data.

[0103] In one exemplary embodiment of this disclosure, the audio data processing apparatus further includes:

[0104] The first target audio data determination module is used to determine the first original audio data as the target audio data of the song corresponding to the original audio data if the first audio quality is higher than the second audio quality.

[0105] The second target audio data determination module is used to determine the second original audio data as the target audio data of the song corresponding to the original audio data if the second audio quality is higher than the first audio quality.

[0106] In one exemplary embodiment of this disclosure, the preset audio data processing model is obtained in the following manner:

[0107] The lossless audio data of historical songs is acquired, and the lossless audio data is encoded and compressed at different bit rates based on a preset encoder to obtain the audio encoding result.

[0108] The audio encoding result and the lossless audio data are randomly paired to obtain multiple audio pairing results, and the actual audio quality difference of each audio pairing result is determined.

[0109] The audio pairing results are input into the neural network model to be trained to obtain the predicted sound quality difference, and a target loss function is constructed based on the actual sound quality difference and the predicted sound quality difference.

[0110] The parameters of the neural network model to be trained are adjusted based on the target loss function to obtain the preset audio data processing model.

[0111] According to one aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the audio data processing method described in any of the preceding claims.

[0112] According to one aspect of this disclosure, an electronic device is provided, comprising:

[0113] Processor; and

[0114] Memory for storing the executable instructions of the processor;

[0115] The processor is configured to execute the audio data processing method described above by executing the executable instructions.

[0116] According to the audio data processing method and apparatus of this disclosure, first original audio data and second original audio data can be acquired, and first audio data to be compared and second audio data to be compared can be extracted from the first original audio data and second original audio data. Then, the first audio data to be compared and second audio data to be compared are input into a preset audio data processing model to obtain the sound quality difference result between the first audio data to be compared and the second audio data to be compared. Based on the sound quality difference result, it is determined whether the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data, without having to compare the encoding format and encoding bitrate of the audio data. This significantly reduces the problem of low accuracy of sound quality comparison results caused by only comparing the encoding format and encoding bitrate of the audio data, and also reduces the problem of low comparison efficiency caused by having to compare all encoding formats and encoding bitrates, thus bringing a better experience to the user. Attached Figure Description

[0117] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which:

[0118] Figure 1 A flowchart illustrating an audio data processing method according to an exemplary embodiment of the present disclosure is shown schematically.

[0119] Figure 2 An architectural diagram of a twin neural network according to an exemplary embodiment of the present disclosure is shown schematically;

[0120] Figure 3 A schematic diagram illustrating the structure of an audio data processing model according to an exemplary embodiment of the present disclosure is shown.

[0121] Figure 4 A schematic diagram illustrating the structure of a feature extraction model according to an exemplary embodiment of the present disclosure is shown.

[0122] Figure 5 A schematic diagram illustrating the structure of a feature encoding module according to an exemplary embodiment of the present disclosure is shown.

[0123] Figure 6 A schematic diagram illustrating an example structure of a deep convolutional neural network according to an exemplary embodiment of the present disclosure is shown.

[0124] Figure 7 A schematic diagram illustrating the structure of a sound quality difference prediction model according to an exemplary embodiment of the present disclosure is shown.

[0125] Figure 8 A flowchart illustrating the training process of an audio data processing model according to an exemplary embodiment of the present disclosure is shown schematically.

[0126] Figure 9 An example diagram illustrating an audio pairing result obtained according to an exemplary embodiment of the present disclosure is shown schematically.

[0127] Figure 10 An example diagram of a raw spectrogram obtained according to an exemplary embodiment of the present disclosure is shown schematically;

[0128] Figure 11 An example diagram illustrating a local peak point obtained according to an exemplary embodiment of the present disclosure is shown schematically;

[0129] Figure 12 A scenario example diagram illustrating an audio alignment position matching process according to an exemplary embodiment of the present disclosure is shown.

[0130] Figure 13 A scenario example diagram schematically illustrates another audio alignment position matching process according to an exemplary embodiment of this disclosure;

[0131] Figure 14 A block diagram of an audio data processing apparatus according to an exemplary embodiment of the present disclosure is shown schematically;

[0132] Figure 15 A computer-readable storage medium for a processing method for storing audio data according to an exemplary embodiment of the present disclosure is illustrated schematically;

[0133] Figure 16 An electronic device for implementing a method for processing audio data is illustrated schematically according to an example embodiment of the present disclosure.

[0134] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0135] The principles and spirit of this disclosure will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0136] Those skilled in the art will recognize that embodiments of this disclosure can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0137] According to embodiments of this disclosure, an audio data processing method, an audio data processing apparatus, a computer-readable storage medium, and an electronic device are proposed.

[0138] In this document, any number of elements in the accompanying figures is for illustrative purposes and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0139] The principles and spirit of this disclosure will be explained in detail below with reference to several representative embodiments. Invention Overview

[0141] The applicant has found that high-quality audio can provide users with a more immersive listening experience, making audio quality a key focus for all music streaming platforms. In practical applications, to ensure the audio quality of the music library's audio materials, the following methods are typically employed: The first method is to use more advanced audio compression encoding technology to provide users with more audio details at the same bitrate; the second method is to increase the coverage and proportion of high-quality audio materials in the music library; for example, if a song in the library only has a 128kbps MP3 audio file, lossless or 320kbps MP3 audio data can be purchased from the song's publisher or musician; the third method is to detect songs with low-quality audio in the music library. In practice, these low-quality songs not only have low audio quality but also consume more storage space and transmission bandwidth, causing additional losses to the platform; furthermore, if users perceive these songs, it will reduce their auditory experience, thus negatively impacting the reputation of the music streaming platform.

[0142] Furthermore, in the second and third implementation methods described above, in order to improve the sound quality of the audio materials in the music library, it is usually necessary to compare the sound quality of the two encoded compressed materials of the same song. However, considering the possibility of false sound quality, when comparing sound quality, simply comparing the encoding format / encoding bitrate may lead to incorrect judgments. Therefore, analyzing which is better from the perspective of actual signal is a more reasonable approach.

[0143] For example, in the second implementation described above, when an audio material with higher sound quality (from the encoding level only) is obtained from the outside, it needs to be compared with the existing material in the library to determine whether the sound quality is higher at the actual signal level. If so, the material in the library can be replaced or upgraded. As another example, in the third implementation described above, when different encoder / bitrate versions of the same song are compared, if it is found that version A is better than version B in terms of encoding format / encoding bitrate, but version A is significantly worse than version B at the actual signal level, version A can be determined to be fake sound quality.

[0144] In some related technical solutions, audio quality can be judged in the following way: First, the audio spectrum is extracted and used as an input image, which is then fed into a neural network. Then, the neural network model predicts the vocal quality score of the song, which is mainly based on the volume and distribution of overtones. However, this method has the following drawbacks: The above solution scores a single song file, predicting a single vocal quality score for one input song, and cannot simultaneously compare the quality of audio data at different bitrates. Furthermore, if the quality of audio data at different bitrates can be predicted by multiple scoring methods based on a neural network model, the differences between the two materials cannot be fully considered during the prediction and scoring process, resulting in a significant reduction in effectiveness.

[0145] Based on this, the present disclosure provides an audio data comparison method, which can solve the problem of comparing the quality of different versions of the same song. Furthermore, the main purpose of the audio data comparison method proposed in the present disclosure is to differentiate the sound quality between different versions of the material, and to determine the magnitude of the sound quality difference between them by the size of the difference. Therefore, the absolute score predicted for a single audio data point is not important in this scenario; with a comparative reference, predicting the score difference is a more easily solved machine learning problem for the model.

[0146] It is further important to note that since the singer's vocal quality score and the sound quality described in the exemplary embodiments of this disclosure are two different concepts, there are drastically different approaches to preparing training data. Specifically, in the general process of preparing training data, songs by well-known singers can be roughly considered as high-quality songs, while songs sung by users can be considered as low-quality songs. Although this is reasonable to some extent, it is too crude. Based on this, the exemplary embodiments of this disclosure will use a fully automated method to construct training data, covering multiple audio encoders and multiple bitrates, which can produce a large amount of training data with relatively high accuracy.

[0147] Furthermore, the audio data processing method described in the exemplary embodiments of this disclosure can extract sound quality features from different encoded and compressed versions of the same song and predict sound quality differences, thereby determining the sound quality differences between the two songs at the actual signal level and providing sound quality reference information for subsequent business applications. In practical applications, firstly, it is necessary to construct sound quality difference pairs by encoding and compressing the lossless material with multiple encoders and multiple bitrates for training the audio data processing model; secondly, before comparing sound quality differences, audio fingerprinting technology can be used to align the two input audio materials on the time axis. The alignment process can reduce the difficulty of subsequent sound quality feature extraction and sound quality difference prediction, resulting in more accurate difference prediction.

[0148] After introducing the basic principles of this disclosure, various non-limiting embodiments of this disclosure will be described in detail below.

[0149] Exemplary methods

[0150] This disclosure first provides an audio data processing method, which can run on terminal devices, servers, server clusters, or cloud servers, etc. Of course, those skilled in the art can also run the method of this disclosure on other platforms as needed, and this exemplary embodiment does not impose any special limitations on this. Specifically, refer to... Figure 1 As shown, the method for generating this audio data may include the following steps:

[0151] Step S110. Obtain the first raw audio data and the second raw audio data, and extract the first audio data to be compared and the second audio data to be compared from the first raw audio data and the second raw audio data;

[0152] Step S120. Input the first audio data to be compared and the second audio data to be compared into a preset audio data processing model to obtain the sound quality difference result between the first audio data to be compared and the second audio data to be compared.

[0153] Step S130. Determine whether the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data based on the audio quality difference result.

[0154] In the above-mentioned audio data processing method, first original audio data and second original audio data can be obtained, and first audio data to be compared and second audio data to be compared can be extracted from the first original audio data and second original audio data. Then, the first audio data to be compared and second audio data to be compared are input into a preset audio data processing model to obtain the sound quality difference result between the first audio data to be compared and the second audio data to be compared. Based on the sound quality difference result, it is determined whether the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data, without having to compare the encoding format and encoding bitrate of the audio data. This significantly reduces the problem of low accuracy of sound quality comparison results caused by only comparing the encoding format and encoding bitrate of the audio data, and also reduces the problem of low comparison efficiency caused by having to compare all encoding formats and encoding bitrates, thus bringing a better experience to the user.

[0155] The following will provide a detailed explanation and description of the audio data processing method described in the exemplary embodiments of this disclosure, in conjunction with the accompanying drawings.

[0156] First, the terms used in the exemplary embodiments of this disclosure will be explained and described.

[0157] Digital audio: Sound signals in the physical world originate from the vibration of sound sources and the transmission of sound waves. To acquire, store, transmit, and process sound signals in a computer, they are digitized. Specifically, continuous sound signals are sampled at a certain frequency (i.e., sampling rate) to obtain discrete numerical sequences. These discrete values ​​are then binary-coded to obtain the final digital audio signal. In practical applications, digital audio technology enables audio to be stored, transmitted, and processed, while also allowing for editing and effects processing, such as equalization and reverb. Furthermore, digital audio is widely used in modern audio production, music recording, film production, and broadcasting.

[0158] Audio compression coding: Raw digital audio typically occupies a large amount of storage space and network transmission bandwidth. Specifically, taking a common song as an example, if the sampling rate is 48000Hz, the data channel is 2 channels, and the bit rate is 16 bits, the data volume per second is 48000 * 2 * 16 / 8 = 192,000 bytes. Therefore, a 4-minute song would require a 40MB audio file, causing significant inconvenience for storage and transmission. To address this, relevant companies have developed various audio compression coding technologies to compress and encode raw digital audio, mainly divided into two categories: lossless compression and lossy compression.

[0159] Lossless compression: During lossless compression, the signal is not lost in any way. Decoding the compressed audio can completely reconstruct the original digital audio signal. However, lossless compression usually has a limited degree of compression and is only suitable for scenarios with extremely high audio quality requirements. Common lossless audio compression algorithms include, but are not limited to, Free Lossless Audio Codec (FLAC) and Apple Lossless Audio Codec (ALAC).

[0160] Lossy compression: In lossy compression, details of the audio signal are discarded in pursuit of a high compression ratio. Therefore, if lossy compression is used, the original digital audio signal cannot be completely reconstructed after decoding the compressed audio. Specifically, in actual compression, the more signal details are discarded, the higher the compression ratio; correspondingly, the greater the signal loss. Common lossy audio compression formats include MP3, AAC, and Vorbis. Currently, mainstream music streaming platforms extensively use lossy compression technology to encode audio for convenient delivery to end users.

[0161] Audio bitrate, also known as audio bit rate, refers to the amount of data per second in an audio stream. Specifically, taking a bitrate of 128kbps as an example, where kb stands for kilobits and ps (per second) stands for per second, then 128kbps means that the amount of data per second is 128 kilobits. In lossy compression, a target compression bitrate can be set. A high bitrate means that more details are preserved, the degree of compression is low, and the amount of data is large; a low bitrate means that less details are preserved, the degree of compression is high, and the amount of data is small. For example, taking MP3 as an example, an MP3 with a bitrate of 320kbps can be considered close to the original audio, and most people will not have a significant perception of it, while the MP3 signal with a bitrate of 128kbps has a more serious loss of detail, and most people can hear the difference.

[0162] Sound quality: In a broad sense, sound quality refers to the perceived quality of audio, a subjective feeling of the listener, and is directly related to the audio content, acquisition and production, compression encoding, sound effects, playback device, and the user's auditory system. In a narrow sense, sound quality mainly refers to the degree of loss of audio details during the compression encoding process; for example, in music streaming platforms, the concepts of lossless sound quality and lossy sound quality are often mentioned; for example, a song encoded in FLAC is considered to have better sound quality than 320kbps MP3, and 320kbps MP3 is considered better than 256kbps MP3. Furthermore, the sound quality involved in the exemplary embodiments of this disclosure refers to sound quality in the narrow sense, that is, the sound quality changes caused by compression encoding.

[0163] False audio quality: Normally, the lossless version of the same song has higher audio quality than the lossy version; under the same lossy encoding format, a higher bitrate has higher audio quality than a lower bitrate. However, due to negligence or malicious intent, the following situations may occur: 1) Lossy audio is fed into encoders such as FLAC and ALAC to generate FLAC / ALAC format audio; therefore, superficially, the audio appears to be lossless; however, at the signal level, it is actually a lossy format; 2) Low-bitrate lossy audio is used as input, and high-bitrate compression encoding is specified to generate high-bitrate audio; therefore, superficially, the audio appears to be high-bitrate lossy audio; however, at the signal level, no new audio details are added, and it remains low-bitrate lossy. Therefore, in the above two situations, the produced audio material appears to have good audio quality only from the perspective of the audio format, but actually has poor audio quality from the signal level. This situation is called false audio quality.

[0164] Embedding: A commonly used concept in the field of deep learning, it refers to mapping input data into a fixed-dimensional vector through a network.

[0165] Siamese Network: A concept in deep neural networks. The basic definition of a Siamese network is that network_1 and network_2 have identical structures and share network weights; therefore, they can essentially be understood as the same network. In practical applications, Siamese networks are widely used in deep metric learning. For example, elements 1 and 2 can be mapped to embedding vectors via network_1 and network_2 respectively for subsequent use. However, network_1 and network_2, as described here, are actually the same network with shared weights. A detailed network architecture diagram of a Siamese network can be found in [reference needed]. Figure 2 As shown.

[0166] Secondly, the audio data processing model involved in the exemplary embodiments of this disclosure will be explained and described. Specifically, refer to... Figure 3As shown, the audio data processing model may include a first input layer 310, a sound quality feature extraction model 320, a feature combination layer 330, a sound quality difference prediction model 340, and a first output layer 350. Further, the audio feature extraction model described here includes a first sound quality feature extraction model and a second sound quality feature extraction model, both of which are Siamese network models, sharing model parameters; that is, the model structure and model parameters of the first and second feature extraction models are completely identical. Furthermore, the first and second feature extraction models described here may include deep convolutional neural networks, or ResNet residual convolutional networks, or Transformer networks, etc.; in practical applications, the appropriate model can be selected according to actual needs, and this example does not impose any special restrictions. Furthermore, the sound quality difference prediction model described here may be a decision tree model, a multilayer perceptron, or a long short-term memory network, etc.; in practical applications, the appropriate model can be selected according to actual needs, and this example does not impose any special restrictions.

[0167] In one example embodiment, reference is made to... Figure 4 As shown, taking the first feature extraction model and the second feature extraction model as examples of Transformer networks, the specific model structure may include a first embedding mapping layer 410, a position encoding module 420, a feature encoding module 430, a feature decoding module 440, a linear transformation layer 450, and a Softmax layer 460; meanwhile, referring to Figure 5 As shown, the feature encoding module described here may include a first multi-head self-attention module 501, a first residual connection and normalization module 502, a first feedforward neural network 503, and a second residual connection and normalization module 504; the specific functions that each functional module needs to perform in the audio data processing process will be described in detail later, and will not be elaborated further here.

[0168] In one example embodiment, reference is made to... Figure 6As shown, taking the first and second feature extraction models as deep convolutional neural networks as examples, the specific model structure may include a first convolutional pooling layer 610, a second convolutional pooling layer 620, a third convolutional pooling layer 630, a fourth convolutional pooling layer 640, a fifth convolutional pooling layer 650, and a fully connected layer 660. Specifically, the first convolutional pooling layer may include 2 convolutional layers + 1 pooling layer, the second convolutional pooling layer may include 2 convolutional layers + 1 pooling layer, the third convolutional pooling layer may include 3 convolutional layers + 1 pooling layer, the fourth convolutional pooling layer may include 3 convolutional layers + 1 pooling layer, and the fifth convolutional pooling layer may include 3 convolutional layers + 1 pooling layer. The fully connected layer may include 3 Dense layers. Meanwhile, in practical applications, if the pooling layer described here is GlobalPooling, then after the first feature extraction model and the second feature extraction model, the audio of t seconds is mapped to a fixed-dimensional vector e; if GlobalPooling is not used, it is mapped to a fixed-dimensional vector sequence [e1, e2, ... e]. n In practical applications, you can choose the specific pooling layer according to your actual needs. This example does not impose any special restrictions on this.

[0169] In one example embodiment, reference is made to... Figure 7 As shown, taking the sound quality difference prediction model as a multilayer perceptron as an example, the specific model structure may include an input layer 710, a first hidden layer 720, a second hidden layer 730, and an output layer 740, etc.

[0170] The following will explain and illustrate the specific training process of the audio data processing model. For details, please refer to... Figure 8 As shown, the specific training process of the audio data processing model may include the following steps:

[0171] Step S810: Obtain lossless audio data of historical songs, and encode and compress the lossless audio data at different bit rates based on a preset encoder to obtain audio encoding results.

[0172] In this example embodiment, firstly, a certain number of lossless audio materials (i.e., lossless audio data) can be exported from the music library; wherein, the lossless audio data described herein refers to audio data that has not undergone any encoding or compression; at the same time, the historical songs described herein may include various different music genres (e.g., rock, pop, ballad, classical, retro, etc.) and songs in different languages ​​(e.g., Chinese, English, French, Japanese, German, etc.); secondly, the lossless audio data is encoded and compressed to obtain the audio encoding result; specifically, the encoder described herein may include, but is not limited to, MP3, AAC, Vorbis, etc., and the bitrate described herein may include, but is not limited to, 64kbps, 128kbps, 256kbps, 320kbps, or 640kbps, etc., and this example does not impose any special restrictions on this.

[0173] Step S820: Randomly pair the audio encoding result and the lossless audio data to obtain multiple audio pairing results, and determine the actual audio quality difference of each audio pairing result.

[0174] Specifically, in the random pairing process, lossless audio data and audio encoding results can be paired, or audio encoding results can be paired to obtain audio quality pairs (i.e., audio pairing results). The resulting audio pairing results can include audio encoding results with the same encoding method and bitrate, audio encoding results with the same encoding method but different bitrates, audio encoding results with different encoding methods but the same bitrate, audio encoding results with different encoding methods and different bitrates, etc. This example does not impose any special restrictions on this. It should be noted that since random pairing can also involve lossless audio data, audio encoding results with the same encoding method and bitrate can also include two lossless audio data sets, etc. Furthermore, the two audio materials included in the generated audio pairing results can be represented by A1 and A2, and the obtained audio pairing results can be referenced... Figure 9 As shown.

[0175] It should also be noted that, since the process of constructing the audio pairing results starts from the lossless audio materials of each historical song, resulting in P audio quality pairs (i.e., audio pairing results), after encoding and compressing all the lossless song materials, a set of audio quality pairs covering different encoders and bitrates can be obtained. This set will be used for subsequent audio quality comparison model training, thereby improving the accuracy of the obtained audio data processing model.

[0176] Furthermore, to meet the needs of subsequent model training, it is also necessary to know the actual sound quality difference of the pair. Therefore, in the data construction stage (i.e., in the process of generating audio pairing results), the corresponding sound quality difference Score(A1)-Score(A2) can be obtained by looking up a table based on the encoders and bitrates used by the two materials. For example, the corresponding Score(A1) and Score(A2) can be obtained by looking up the table separately, and then Score(A1)-Score(A2) can be calculated to obtain the actual sound quality difference. The table can be determined by expert knowledge, mapping any combination of encoders and bitrates to a corresponding sound quality difference. At the same time, the actual sound quality difference can be scaled to a reasonable range. For example, taking a 5-point scale, the range of sound quality difference is [-5, 5]. Negative numbers indicate that the sound quality of A1 is worse than that of A2, and positive numbers indicate that the sound quality of A1 is better than that of A2. The larger the absolute value of the difference, the greater the difference in sound quality between the two.

[0177] Step S830: Input the audio pairing results into the neural network model to be trained to obtain the predicted sound quality difference, and construct the target loss function based on the actual sound quality difference and the predicted sound quality difference.

[0178] Specifically, the model training process described here can be performed offline. That is, during the offline training phase, the two audio materials contained in the audio quality pair (i.e., the audio pairing result) can be fed in to obtain the predicted audio quality difference. The obtained predicted audio quality difference is: S_diff = Score(A1) - Score(A2). It should be noted that since the two audio materials are generated from the same lossless material during the training phase, they are already aligned in time, and there is no need to perform audio fingerprint alignment. After obtaining the audio pairing result, it can be directly input into the neural network model to be trained to obtain the corresponding predicted audio quality difference.

[0179] Furthermore, after obtaining the predicted sound quality difference, a target loss function can be constructed based on the predicted sound quality difference and the actual sound quality difference. The target loss function described here can be L2 (mean squared error loss function), cross-entropy loss function, or other loss functions. In actual application, you can choose according to the actual needs. This example does not impose any special restrictions on this.

[0180] Furthermore, before inputting the audio pairing results into the neural network model to be trained, the audio data included in the audio pairing results needs to be preprocessed. The specific processing can include the following two aspects: First, a fixed-length t-second time-domain audio signal is extracted from the audio as the input of this step. Specifically, during the extraction process, A1 and A2 need to be extracted from the same position to ensure that the two are time-aligned. One typical approach is to randomly sample a starting position in [0, dur-t] and extract a t-second segment from that position. Another typical approach is to extract a t-second segment every δ seconds starting from 0 seconds. This approach can extract multiple t-second segments, which can be used as multiple training or test samples. The other aspect is to perform short-time Fourier Transform (STFT) analysis on the audio. For example, using a 2048-point FFT, a 512-point hop size, and a Hanning window to perform STFT spectral analysis to obtain a spectrogram. After this step, the t-second time-domain signal can be transformed into a fixed-size two-dimensional spectrogram, with the horizontal axis representing the number of frames and the vertical axis representing the number of frequency bands. In other words, the data ultimately input into the neural network model to be trained is a two-dimensional spectrogram of the two audio data points from the audio pairing result.

[0181] In one example embodiment, during the prediction of sound quality difference, after passing through the audio feature network (i.e., the feature extraction model), the sound quality feature vectors (or vector sequences) corresponding to A1 and A2 can be obtained; then, the two are combined in the feature combination layer so as to be fed into the subsequent sound quality difference prediction model; at the same time, since Score(A1)-Score(A2)≠Score(A2)-Score(A1), this means that the two have a relative order relationship, so this relative order relationship should be reflected in the sound quality feature combination module; furthermore, in order to reflect the order relationship between A1 and A2, one possible approach is to perform difference processing on the sound quality features of the two, and the specific processing process can be shown in the following formula (1):

[0182] e_diff = e(A1) - e(A2); Formula (1)

[0183] Then, e_diff is used as the output of the feature combination and fed into the sound quality difference prediction model. Of course, e_diff can also be directly concatenated with e(A1) and e(A2) and fed into the sound quality difference prediction model. Furthermore, another possible approach is to directly concatenate e(A1) and (A2) and feed them into the sound quality difference prediction model. However, this approach requires the sound quality difference prediction model to learn the relative order relationship between the two during the training process, which will increase the difficulty of model learning to some extent.

[0184] In one example embodiment, since the sound quality features have already been combined in the feature combination layer, the input to the sound quality difference prediction model is either a concatenated feature vector or a sequence of concatenated feature vectors. Based on this, the final sound quality difference can be predicted (i.e., the predicted sound quality difference is determined). Furthermore, since the specific determination process of the predicted sound quality difference is a typical regression problem, various common neural network structures can be selected. In practical applications, if the input is a feature vector, a simple feedforward neural network structure such as the MultiLayer Perceptron can be used; if the input is a sequence of feature vectors, a recurrent neural network such as the Long Short-Term Memory (LSTM) can be used, and so on. Finally, a floating-point value is output as the predicted sound quality difference. Furthermore, after obtaining the predicted sound quality difference, the corresponding target loss function can be constructed. Specifically, in the construction stage of the target loss function, the expert difference (i.e., the actual sound quality difference ground truth) corresponding to the dataset pair can be used as the target, and the regression loss function Loss can be used as the target loss function to train the entire neural network. Here, the Loss recorded here can be common loss functions such as L1 and L2. Taking the target loss function as the L2 loss function as an example, the target loss function can be shown in the following formula (2):

[0185]

[0186] Step S840: Adjust the parameters of the neural network model to be trained based on the target loss function to obtain the preset audio data processing model.

[0187] Specifically, after obtaining the target loss function, the network weight parameters of the sound quality feature extraction model and the sound quality difference prediction model can be updated based on this target loss function until convergence. Simultaneously, after parameter adjustment, the neural network model can be tested on a test set, and the audio data processing model can be obtained after the test.

[0188] The following will combine Figures 2-9 right Figure 1 The audio data processing method shown will be further explained and illustrated. Specifically:

[0189] In step S110, first raw audio data and second raw audio data are acquired, and first audio data to be compared and second audio data to be compared are extracted from the first raw audio data and second raw audio data.

[0190] In this example embodiment, firstly, first original audio data and second original audio data are acquired; specifically, the first original audio data and second original audio data mentioned here refer to two audio data sets of a certain song with different encoding methods and encoding bitrates, which can be obtained from the server corresponding to the application; secondly, first audio data to be compared and second audio data to be compared are extracted from the first original audio data and second original audio data; specifically, this can be obtained as follows: extracting first audio fingerprint data of the first original audio data and second audio fingerprint data of the second original audio data; wherein, the first original audio data and second original audio data are audio data with different bitrates corresponding to the same song; performing audio alignment processing on the first original audio data and second original audio data according to the first audio fingerprint data and second audio fingerprint data to obtain a first audio alignment result and a second audio alignment result; extracting the first audio data to be compared from the first audio alignment result, and extracting the second audio data to be compared from the second audio alignment result; wherein, the first audio position of the first audio data to be compared in the first audio alignment result is the same as the second audio position of the second audio data to be compared in the second audio alignment result.

[0191] In an exemplary embodiment, the first audio fingerprint data of the first original audio data can be extracted by performing a short-time Fourier transform on the first original audio data to obtain a first original spectrogram, and traversing the first original spectrogram to extract a first local peak point from the first original spectrogram; and constructing the first audio fingerprint data based on the first time sequence position of the first local peak point in the first original spectrogram and the first frequency band number of the first local peak point.

[0192] In one exemplary embodiment, the extraction of second audio fingerprint data from the second original audio data can be achieved by performing a short-time Fourier transform on the second original audio data to obtain a second original spectrogram, and traversing the second original spectrogram to extract a second local peak point from the second original spectrogram; and constructing the second audio fingerprint data based on the second time sequence position of the second local peak point in the second original spectrogram and the second frequency band number of the second local peak point.

[0193] In one exemplary embodiment, audio alignment processing is performed on first original audio data and second original audio data based on first audio fingerprint data and second audio fingerprint data to obtain first audio alignment result and second audio alignment result. This includes: traversing the first original spectrogram corresponding to the first original audio data, finding the position in the first original spectrogram that has the most overlapping part with the second local peak point in the second audio fingerprint data, and using this position as the audio alignment position of the first original audio data and the second original audio data; and / or traversing the second original spectrogram corresponding to the second original audio data, finding the position in the second original spectrogram that has the most overlapping part with the first local peak point in the first audio fingerprint data, and using this position as the audio alignment position of the first original audio data and the second original audio data; and performing audio alignment processing on the first original audio data and the second original audio data based on the audio alignment position to obtain first audio alignment result and second audio alignment result.

[0194] The following will further explain and illustrate the specific extraction process of the first and second audio data to be compared. Specifically, in practical applications, due to potential temporal misalignment between the first and second original audio data, audio alignment processing is required before extracting them. This temporal misalignment mainly manifests as follows: Assume the song library already contains a 320kbps MP3 of song x, material added many years ago; currently, lossless audio of song x has been purchased from the publisher; however, due to further production, there may be a few seconds of silence at the beginning or end, causing a temporal offset from the material in the library. Therefore, in this case, directly comparing audio quality will affect the accuracy of the comparison due to the temporal misalignment between the two materials. Therefore, before comparing audio quality, audio fingerprinting technology can be used to time-align the two audio materials (i.e., the first original audio data and the second original audio data); after alignment, audio segments are extracted and compared according to the original process.

[0195] In one example embodiment, audio fingerprinting is a typical music retrieval technology. Its basic principle is as follows: for the audio of each song in the music library, fingerprint information representing the song is extracted from the audio using signal processing or machine learning techniques. This fingerprint information can typically be stored as a numerical value, a numerical vector, or a sequence of numerical vectors, etc. Then, these song fingerprints are assembled into an efficient index. In use, for the query audio to be retrieved, fingerprint information can be extracted from the query audio, and then a search and matching (i.e., fingerprint comparison) is performed in the index to find the song corresponding to the retrieval recording and the position of the query audio within the target song. It should be noted that audio fingerprinting is typically used in music retrieval scenarios such as song recognition. Because it can not only find the corresponding song but also accurately return the position of the query audio within the target song, it can also be used to align two audio materials, A1 and A2.

[0196] In one example embodiment, during the alignment of the first and second original audio data, it is first necessary to detect energy maxima points in the audio spectrogram as feature points to construct an audio fingerprint. Specifically, the audio signals (i.e., the first and second original audio data) are first subjected to a Short-Time Fourier Transform (STFT) to convert the one-dimensional time-domain audio signal into a two-dimensional spectrogram, thereby obtaining the first and second original spectrograms; wherein, reference... Figure 10 As shown, in the obtained first and second original spectrograms, the horizontal axis is the time axis, in audio frames; the vertical axis is the frequency axis, in frequency band numbers; and the value of a point p(t,f) in the spectrogram represents the signal strength of frequency band f at time t. Secondly, after performing the short-time Fourier transform, local peaks can be found in the two-dimensional spectrogram. That is, if the value of a point in the spectrogram is greater than that of all surrounding points (e.g., a surrounding rectangular area), then that point is a local peak. The obtained local peaks can be referenced... Figure 11 As shown; then, based on all the peak points found in the spectrogram, a fingerprint is constructed; for example, the time frame t and frequency band number f of the peak point can be directly used as the fingerprint corresponding to the current peak point; furthermore, during the alignment process, the position with the most overlapping peak points and the number of overlapping peak points can be found for alignment; at the same time, the position recorded here is the alignment time position, and the overlapping peak points are the fingerprint search score; the specific matching process can be referred to Figure 12 as well as Figure 13As shown; it should also be noted that during the alignment process, the first original spectrogram of A1 can be scanned from front to back based on the second local peak point in the second audio fingerprint data to find the position where the peak point fingerprint of A2 overlaps the most, which is the alignment position between the two; or, the second original spectrogram of A2 can be scanned from front to back based on the first local peak point in the first audio fingerprint data to find the position where the peak point fingerprint of A1 overlaps the most, which is the alignment position between the two; furthermore, after the first original audio data and the second original audio data are aligned, a t-second segment can be extracted as needed, which ensures that the two segments extracted from A1 and A2 are time-aligned.

[0197] In one example embodiment, during the extraction of the first and second audio data to be compared from the first and second original audio data, a fixed-length t-second time-domain audio signal needs to be extracted from the first and second original audio data as input for this step. During extraction, it is necessary to ensure that A1 and A2 are extracted from the same position to guarantee time alignment. For example, a starting position can be randomly sampled in [0, dur-t], and a t-second segment can be extracted from that position. Alternatively, a t-second segment can be extracted every δ seconds starting from 0 seconds. This approach can extract multiple t-second segments to obtain the first and second audio data to be compared. It should also be noted that one or more sets of the first and second audio data to be compared can be extracted from the first and second original audio data; this example does not impose any special limitations on this.

[0198] In step S120, the first audio data to be compared and the second audio data to be compared are input into a preset audio data processing model to obtain the sound quality difference result between the first audio data to be compared and the second audio data to be compared.

[0199] Specifically, the process of determining the audio quality difference result can be implemented as follows: Input the first and second audio data to be compared into the audio quality feature extraction model to obtain the first audio feature corresponding to the first audio data and the second audio feature corresponding to the second audio data; combine the first and second audio features based on the feature combination layer to obtain the audio quality combination feature, and input the audio quality combination feature into the audio quality difference prediction model to obtain the audio quality difference result between the first and second audio data. That is, in practical applications, to obtain the audio quality difference result, three processes are required: audio feature extraction, audio feature combination, and audio quality difference prediction. The specific implementation processes of audio feature extraction, audio feature combination, and audio quality difference prediction will be further explained and illustrated below. Specifically:

[0200] In one exemplary embodiment, the specific extraction process of the first audio feature and the second audio feature can be implemented as follows: performing a short-time Fourier transform on the first audio data to be compared to obtain a first spectrogram to be compared, and performing a short-time Fourier transform on the second audio data to be compared to obtain a second spectrogram to be compared; inputting the first spectrogram to be compared into the first feature extraction model to obtain a first audio feature corresponding to the first audio data to be compared; and inputting the second spectrogram to be compared into the second feature extraction model to obtain a second audio feature corresponding to the second audio data to be compared. Specifically, the determination of the spectrogram to be compared can be achieved as follows: Short-time Fourier Transform (STFT) analysis is performed on the audio data to be compared. For example, a 2048-point FFT, a 512-point hop size, and a Hanning window can be used for STFT spectrum analysis to obtain the spectrogram to be compared. Simultaneously, after the STFT, the time-domain signal of t seconds (i.e., the first and second audio data to be compared) can be transformed into a fixed-size two-dimensional spectrogram (i.e., the first and second spectrograms to be compared). In the obtained first and second spectrograms to be compared, the horizontal axis can represent the number of frames, and the vertical axis can represent the number of frequency bands.

[0201] In one example embodiment, after obtaining the first and second spectrograms to be compared, the first spectrogram is input into a first feature extraction model to obtain a first audio feature corresponding to the first audio data to be compared, and the second spectrogram is input into a second feature extraction model to obtain a second audio feature corresponding to the second audio data to be compared. It should be noted that since the first and second feature extraction models are Siamese network models, and both have the same model structure and share model parameters, the specific extraction processes for the first and second audio features are consistent.

[0202] This section takes the extraction process of the first audio feature as an example, and uses the Transformer model as the first feature extraction model to further explain and illustrate the specific extraction process. Specifically, the extraction process of the first audio feature can be implemented as follows: The first spectrogram to be compared is embedded and mapped based on the first embedding mapping layer to obtain a first embedding vector; the first spectrogram to be compared is encoded based on the position encoding module to obtain a first position vector; the first embedding vector and the first position vector are superimposed to obtain a first input vector; the first input vector is encoded based on the feature encoding module to obtain a first encoding result; the first encoding result is decoded based on the feature decoding module to obtain a first encoding matrix; the first encoding matrix is ​​transformed based on the linear transformation layer to obtain a first logical matrix; and the first logical matrix is ​​mapped based on the Softmax layer to obtain the first audio feature corresponding to the first audio data to be compared. The encoding of the first input vector based on the feature encoding module to obtain the first encoding result can be achieved as follows: A first attention mechanism is calculated based on the first multi-head self-attention module, and the first attention mechanism is subjected to residual connection and normalization processing based on the first residual connection and normalization module to obtain a first normalization processing result; the first normalization processing result is linearized based on the first feedforward neural network to obtain a first linearization processing result, and the first linearization processing result is subjected to residual connection and normalization processing based on the second residual connection and normalization module to obtain the first encoding result. Further, the calculation of the first attention mechanism based on the first multi-head self-attention module can be achieved as follows: The first input vector is linearly fused to obtain a first query vector, a first key vector, and a first value vector; the first outer product vector of the first query vector and the first key vector is calculated to obtain a first similarity between the first query vector and the first key vector; the first similarity is normalized to obtain a first weight matrix; and the second outer product vector of the first weight matrix and the first value vector is calculated to obtain the first attention mechanism.

[0203] This section takes the extraction process of the first audio feature as an example, and uses a deep convolutional neural network model as the first feature extraction model to further explain and illustrate the specific extraction process. Specifically, the extraction process of the first audio feature can be implemented as follows: The first spectrogram to be compared is sequentially processed by convolutional pooling layers 1, 2, 3, 4, and 5; then, the first spectrogram to be compared after convolutional pooling is classified based on a fully connected layer to obtain the first audio feature. Meanwhile, since the pooling layers involved in the deep convolutional neural network model can include global pooling or local pooling, if the pooling layer used here is global pooling, the obtained first audio feature is a fixed-dimensional vector e; if global pooling is not used, the obtained first audio feature is a fixed-dimensional vector sequence [e1, e2, ... e]. n In practical applications, you can choose according to your actual needs; this example does not impose any special restrictions on this.

[0204] In one exemplary embodiment, the combination of the first audio feature and the second audio feature based on the feature combination layer to obtain the sound quality combination feature can be achieved in the following ways: one way is to calculate the difference between the first audio feature and the second audio feature to obtain the feature difference calculation result, and determine the sound quality combination feature based on the feature difference calculation result; another way is to combine the feature difference calculation result, the first audio feature, and the second audio feature to obtain the sound quality combination feature; yet another way is to combine the first audio feature and the second audio feature to obtain the sound quality combination feature.

[0205] In one example embodiment, the audio quality combination features are input into the audio quality difference prediction model to obtain the audio quality difference result between the first audio data to be compared and the second audio data to be compared. This can be achieved as follows: Specifically, taking a multilayer perceptron as the audio quality difference prediction model, the audio quality combination features can be forward-propagated based on the hidden layers included in the multilayer perceptron to obtain the corresponding audio quality difference result. The specific expression of the obtained audio quality difference result is: S_diff = Score(A1) - Score(A2).

[0206] In step S130, it is determined whether the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data based on the audio quality difference result.

[0207] Specifically, the process for determining the first audio quality and the second audio quality is as follows: if the audio quality difference result is greater than or equal to a preset threshold, then the first audio quality of the first original audio data is determined to be higher than the second audio quality of the second original audio data; if the audio quality difference result is less than the preset threshold, then the first audio quality of the first original audio data is determined to be lower than the second audio quality of the second original audio data; wherein, the larger the value of the audio quality difference result, the higher the first audio quality of the first original audio data; the smaller the value of the audio quality difference result, the higher the second audio quality of the second original audio data.

[0208] Finally, after obtaining the first and second audio quality, the audio data processing method further includes: if the first audio quality is higher than the second audio quality, then the first original audio data is used as the target audio data for the song corresponding to the original audio data; if the second audio quality is higher than the first audio quality, then the second original audio data is used as the target audio data for the song corresponding to the original audio data. Simultaneously, after the target audio data is determined, low-quality audio data can be deleted. Based on this method, the user experience during music listening can be further improved.

[0209] Thus, the audio data processing method described in the exemplary embodiments of this disclosure has been fully implemented. Based on the foregoing description, it can be understood that the audio data processing method described in the exemplary embodiments of this disclosure can automatically construct a training dataset and complete the training of the sound quality comparison model without much manual intervention. Furthermore, when used online, for two audio materials of the same song, after audio fingerprint alignment, they can be fed into the sound quality comparison model to predict the sound quality difference between the two. This sound quality difference reflects the quality of the two audio materials and the degree of their sound quality difference at the signal level, which can be used in scenarios such as music library management and fake sound quality song detection, bringing great convenience to music library management. Further, the sound quality difference prediction described in the exemplary embodiments of this disclosure is a judgment of the quality of two inputs and the degree of sound quality difference. This is easier to learn than directly predicting absolute sound quality values, and it provides richer information, such as the degree of difference, than simply predicting the quality of the sound. This facilitates selective processing based on the degree of sound quality difference in subsequent applications.

[0210] Exemplary device

[0211] After introducing the audio data processing method according to exemplary embodiments of this disclosure, the following will refer to... Figure 14 The audio data processing apparatus of exemplary embodiments of the present disclosure will be explained and described.

[0212] For details, please refer to Figure 14 As shown, the audio data processing device may include an audio data extraction module 1410, a sound quality difference result determination module 1420, and an audio quality determination module 1430. Wherein:

[0213] The audio data extraction module 1410 can be used to acquire first raw audio data and second raw audio data, and extract first audio data to be compared and second audio data to be compared from the first raw audio data and second raw audio data.

[0214] The sound quality difference result determination module 1420 can be used to input the first audio data to be compared and the second audio data to be compared into a preset audio data processing model to obtain the sound quality difference result between the first audio data to be compared and the second audio data to be compared.

[0215] The audio quality determination module 1430 can be used to determine the first audio quality of the first audio data to be compared and the second audio quality of the second audio data to be compared based on the audio quality difference result.

[0216] In one exemplary embodiment of this disclosure, extracting first audio data to be compared and second audio data to be compared from first original audio data and second original audio data includes: extracting first audio fingerprint data of the first original audio data and second audio fingerprint data of the second original audio data; wherein the first original audio data and the second original audio data are audio data with different bitrates corresponding to the same song; performing audio alignment processing on the first original audio data and the second original audio data according to the first audio fingerprint data and the second audio fingerprint data to obtain a first audio alignment result and a second audio alignment result; extracting the first audio data to be compared from the first audio alignment result and extracting the second audio data to be compared from the second audio alignment result; wherein the first audio position of the first audio data to be compared in the first audio alignment result is the same as the second audio position of the second audio data to be compared in the second audio alignment result.

[0217] In one exemplary embodiment of this disclosure, extracting first audio fingerprint data from first original audio data includes: performing a short-time Fourier transform on the first original audio data to obtain a first original spectrogram, and traversing the first original spectrogram to extract a first local peak point from the first original spectrogram; and constructing the first audio fingerprint data based on the first time sequence position of the first local peak point in the first original spectrogram and the first frequency band number of the first local peak point.

[0218] In one exemplary embodiment of this disclosure, audio alignment processing is performed on first original audio data and second original audio data based on the first audio fingerprint data and the second audio fingerprint data to obtain a first audio alignment result and a second audio alignment result. This includes: traversing a first original spectrogram corresponding to the first original audio data, finding the position in the first original spectrogram that has the most overlapping portion with a second local peak point in the second audio fingerprint data, and using this position as the audio alignment position for the first original audio data and the second original audio data; and / or traversing a second original spectrogram corresponding to the second original audio data, finding the position in the second original spectrogram that has the most overlapping portion with a first local peak point in the first audio fingerprint data, and using this position as the audio alignment position for the first original audio data and the second original audio data; and performing audio alignment processing on the first original audio data and the second original audio data based on the audio alignment position to obtain the first audio alignment result and the second audio alignment result.

[0219] In one exemplary embodiment of this disclosure, the preset audio data processing model includes a sound quality feature extraction model, a feature combination layer, and a sound quality difference prediction model. The process of inputting first and second audio data to be compared into the preset audio data processing model to obtain a sound quality difference result between the first and second audio data includes: inputting the first and second audio data to be compared into the sound quality feature extraction model to obtain a first audio feature corresponding to the first audio data and a second audio feature corresponding to the second audio data; combining the first and second audio features based on the feature combination layer to obtain a sound quality combination feature; and inputting the sound quality combination feature into the sound quality difference prediction model to obtain a sound quality difference result between the first and second audio data.

[0220] In an exemplary embodiment of this disclosure, the preset audio quality feature extraction model includes a first feature extraction model and a second feature extraction model, which share model parameters. The process of inputting first and second audio data to be compared into the audio quality feature extraction model to obtain a first audio feature corresponding to the first audio data and a second audio feature corresponding to the second audio data includes: performing a short-time Fourier transform on the first audio data to be compared to obtain a first audio spectrogram, and performing a short-time Fourier transform on the second audio data to obtain a second audio spectrogram; inputting the first audio spectrogram into the first feature extraction model to obtain the first audio feature corresponding to the first audio data; and inputting the second audio spectrogram into the second feature extraction model to obtain the second audio feature corresponding to the second audio data.

[0221] In an exemplary embodiment of this disclosure, the first feature extraction model includes a first embedding mapping layer, a position encoding module, a feature encoding module, a feature decoding module, a linear transformation layer, and a Softmax layer. The process of inputting the first spectrogram to be compared into the first feature extraction model to obtain a first audio feature corresponding to the first audio data to be compared includes: performing embedding mapping processing on the first spectrogram to be compared based on the first embedding mapping layer to obtain a first embedding vector; and encoding the first spectrogram to be compared based on the position encoding module to obtain a first position vector; superimposing the first embedding vector and the first position vector to obtain a first input vector; and encoding the first input vector based on the feature encoding module to obtain a first encoding result; decoding the first encoding result based on the feature decoding module to obtain a first encoding matrix; transforming the first encoding matrix based on the linear transformation layer to obtain a first logical matrix; and mapping the first logical matrix based on the Softmax layer to obtain the first audio feature corresponding to the first audio data to be compared.

[0222] In one exemplary embodiment of this disclosure, the feature encoding module includes a first multi-head self-attention module, a first residual connection and normalization module, a first feedforward neural network, and a second residual connection and normalization module. The process of encoding a first input vector based on the feature encoding module to obtain a first encoding result includes: calculating a first attention mechanism for the first input vector based on the first multi-head self-attention module, and performing residual connection and normalization processing on the first attention mechanism based on the first residual connection and normalization module to obtain a first normalization processing result; performing linearization processing on the first normalization processing result based on the first feedforward neural network to obtain a first linearization processing result, and performing residual connection and normalization processing on the first linearization processing result based on the second residual connection and normalization module to obtain the first encoding result.

[0223] In an exemplary embodiment of this disclosure, a first attention mechanism based on a first multi-head self-attention module to calculate the first input vector includes: linearly fusing the first input vector to obtain a first query vector, a first key vector, and a first value vector; calculating a first outer product vector of the first query vector and the first key vector to obtain a first similarity between the first query vector and the first key vector; normalizing the first similarity to obtain a first weight matrix; and calculating a second outer product vector of the first weight matrix and the first value vector to obtain the first attention mechanism.

[0224] In one exemplary embodiment of this disclosure, combining the first audio feature and the second audio feature based on the feature combination layer to obtain a sound quality combination feature includes: calculating the difference between the first audio feature and the second audio feature to obtain a feature difference calculation result, and determining the sound quality combination feature based on the feature difference calculation result; and / or combining the feature difference calculation result, the first audio feature, and the second audio feature to obtain the sound quality combination feature; and / or combining the first audio feature and the second audio feature to obtain the sound quality combination feature.

[0225] In one exemplary embodiment of this disclosure, determining whether the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data based on the audio quality difference result includes: if the audio quality difference result is greater than or equal to a preset threshold, then determining that the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data; if the audio quality difference result is less than the preset threshold, then determining that the first audio quality of the first original audio data is lower than the second audio quality of the second original audio data; wherein, the larger the value of the audio quality difference result, the higher the first audio quality of the first original audio data; the smaller the value of the audio quality difference result, the higher the second audio quality of the second original audio data.

[0226] In one exemplary embodiment of this disclosure, the audio data processing apparatus further includes:

[0227] The first target audio data determination module is used to determine the first original audio data as the target audio data of the song corresponding to the original audio data if the first audio quality is higher than the second audio quality.

[0228] The second target audio data determination module is used to determine the second original audio data as the target audio data of the song corresponding to the original audio data if the second audio quality is higher than the first audio quality.

[0229] In one exemplary embodiment of this disclosure, the preset audio data processing model is obtained in the following manner:

[0230] The lossless audio data of historical songs is acquired, and the lossless audio data is encoded and compressed at different bit rates based on a preset encoder to obtain the audio encoding result.

[0231] The audio encoding result and the lossless audio data are randomly paired to obtain multiple audio pairing results, and the actual audio quality difference of each audio pairing result is determined.

[0232] The audio pairing results are input into the neural network model to be trained to obtain the predicted sound quality difference, and a target loss function is constructed based on the actual sound quality difference and the predicted sound quality difference.

[0233] The parameters of the neural network model to be trained are adjusted based on the target loss function to obtain the preset audio data processing model.

[0234] Exemplary storage media

[0235] After introducing the audio data processing method and audio data processing apparatus according to exemplary embodiments of the present disclosure, the following will refer to... Figure 15 The storage medium of the exemplary embodiments of this disclosure will be described.

[0236] refer to Figure 15 As shown, a program product 1500 for implementing the above-described method according to an embodiment of the present disclosure is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto.

[0237] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0238] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable signal medium may also be any readable medium other than a readable storage medium. Program code for performing the operations of this disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages—such as Java, C++, etc.—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code may execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device may be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN).

[0239] Exemplary electronic devices

[0240] Having described the storage medium of exemplary embodiments of this disclosure, the following references are made. Figure 16 An electronic device according to an exemplary embodiment of the present disclosure will be described.

[0241] Figure 16The electronic device 1600 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0242] like Figure 16 As shown, the electronic device 1600 is manifested in the form of a general-purpose computing device. The components of the electronic device 1600 may include, but are not limited to: at least one processing unit 1610, at least one storage unit 1620, a bus 1630 connecting different system components (including storage unit 1620 and processing unit 1610), and a display unit 1640.

[0243] The storage unit 1620 stores program code that can be executed by the processing unit 1610, causing the processing unit 1610 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 1610 can perform actions such as... Figure 1 Steps S110-S130 are shown in the diagram.

[0244] Storage unit 1620 may include volatile storage units, such as random access memory (RAM) 16201 and / or cache memory 16202, and may further include read-only memory (ROM) 16203.

[0245] Storage unit 1620 may also include a program / utility 16204 having a set (at least one) program module 16205, such program module 16205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0246] The bus 1630 may include a data bus, an address bus, and a control bus.

[0247] Electronic device 1600 can also communicate with one or more external devices 1700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.) via input / output (I / O) interface 1650. Furthermore, electronic device 1600 can also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 1660. As shown in the figure, network adapter 1660 communicates with other modules of electronic device 1600 via bus 1630. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with electronic device 1600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0248] It should be noted that although several modules or sub-modules of the pop-up window handling device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0249] Furthermore, although the operations of the methods disclosed herein are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0250] While the spirit and principles of this disclosure have been described with reference to several specific embodiments, it should be understood that this disclosure is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for convenience of expression. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method for processing audio data, characterized in that, include: Acquire first raw audio data and second raw audio data, and extract first audio data to be compared and second audio data to be compared from the first raw audio data and second raw audio data; The first audio data to be compared and the second audio data to be compared are input into a preset audio data processing model to obtain the sound quality difference result between the first audio data to be compared and the second audio data to be compared; the preset audio data processing model includes a sound quality feature extraction model, a feature combination layer and a sound quality difference prediction model. Based on the sound quality difference result, determine whether the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data; The sound quality difference result is determined as follows: the first audio data to be compared and the second audio data to be compared are input into the sound quality feature extraction model to obtain the first audio feature corresponding to the first audio data to be compared and the second audio feature corresponding to the second audio data to be compared. The first audio feature and the second audio feature are combined based on the feature combination layer to obtain the sound quality combination feature. The sound quality combination feature is then input into the sound quality difference prediction model to obtain the sound quality difference result between the first audio data to be compared and the second audio data to be compared.

2. The method for processing audio data according to claim 1, characterized in that, Extracting the first audio data to be compared and the second audio data to be compared from the first original audio data and the second original audio data includes: Extract the first audio fingerprint data of the first original audio data and the second audio fingerprint data of the second original audio data; wherein, the first original audio data and the second original audio data are audio data of different bit rates corresponding to the same song; Based on the first audio fingerprint data and the second audio fingerprint data, audio alignment processing is performed on the first original audio data and the second original audio data to obtain the first audio alignment result and the second audio alignment result. The first audio data to be compared is extracted from the first audio alignment result, and the second audio data to be compared is extracted from the second audio alignment result; wherein the first audio position of the first audio data to be compared in the first audio alignment result is the same as the second audio position of the second audio data to be compared in the second audio alignment result.

3. The audio data processing method according to claim 2, characterized in that, Extracting the first audio fingerprint data from the first original audio data includes: Perform a short-time Fourier transform on the first original audio data to obtain a first original spectrogram, and traverse the first original spectrogram to extract the first local peak point from the first original spectrogram. The first audio fingerprint data is constructed based on the first time sequence position of the first local peak point in the first original spectrogram and the first frequency band number of the first local peak point.

4. The method for processing audio data according to claim 2, characterized in that, Based on the first audio fingerprint data and the second audio fingerprint data, audio alignment processing is performed on the first original audio data and the second original audio data to obtain the first audio alignment result and the second audio alignment result, including: Traverse the first original spectrogram corresponding to the first original audio data, and find the position in the first original spectrogram that has the most overlap with the second local peak point in the second audio fingerprint data, using this position as the audio alignment position for the first original audio data and the second original audio data; and / or Traverse the second original spectrogram corresponding to the second original audio data, and find the position in the second original spectrogram that has the most overlap with the first local peak point in the first audio fingerprint data, and use it as the audio alignment position of the first original audio data and the second original audio data. Based on the audio alignment position, the first original audio data and the second original audio data are subjected to audio alignment processing to obtain the first audio alignment result and the second audio alignment result.

5. The method for processing audio data according to claim 1, characterized in that, The preset sound quality feature extraction model includes a first feature extraction model and a second feature extraction model, and the first feature extraction model and the second feature extraction model share model parameters; Specifically, the first and second audio data to be compared are input into the sound quality feature extraction model to obtain a first audio feature corresponding to the first audio data to be compared and a second audio feature corresponding to the second audio data to be compared, including: The first audio data to be compared is subjected to a short-time Fourier transform to obtain a first spectrogram of the comparison, and the second audio data to be compared is subjected to a short-time Fourier transform to obtain a second spectrogram of the comparison. The first spectrogram to be compared is input into the first feature extraction model to obtain the first audio feature corresponding to the first audio data to be compared; The second spectrogram to be compared is input into the second feature extraction model to obtain the second audio feature corresponding to the second audio data to be compared.

6. The method for processing audio data according to claim 5, characterized in that, The first feature extraction model includes a first embedding mapping layer, a position encoding module, a feature encoding module, a feature decoding module, a linear transformation layer, and a Softmax layer; Specifically, inputting the first spectrogram to be compared into the first feature extraction model to obtain the first audio feature corresponding to the first audio data to be compared includes: The first spectrogram to be compared is embedded and mapped based on the first embedding mapping layer to obtain a first embedding vector, and the first spectrogram to be compared is encoded based on the position encoding module to obtain a first position vector. The first embedding vector and the first position vector are superimposed to obtain the first input vector, and the first input vector is encoded based on the feature encoding module to obtain the first encoding result; The first encoding result is decoded by the feature decoding module to obtain the first encoding matrix. The first encoding matrix is ​​then transformed by the linear transformation layer to obtain the first logical matrix. Finally, the first logical matrix is ​​mapped by the Softmax layer to obtain the first audio feature corresponding to the first audio data to be compared.

7. The method for processing audio data according to claim 6, characterized in that, The feature encoding module includes a first multi-head self-attention module, a first residual connection and normalization module, a first feedforward neural network, and a second residual connection and normalization module; The first input vector is encoded using a feature encoding module to obtain a first encoding result, including: The first attention mechanism is calculated based on the first multi-head self-attention module, and the first attention mechanism is subjected to residual connection and normalization processing based on the first residual connection and normalization module to obtain the first normalization processing result. The first normalization result is linearized based on the first feedforward neural network to obtain the first linearized result. The first linearized result is then subjected to residual connection and normalization based on the second residual connection and normalization module to obtain the first encoding result.

8. The method for processing audio data according to claim 7, characterized in that, The first attention mechanism, which calculates the first input vector based on the first multi-head self-attention module, includes: The first input vector is linearly fused to obtain a first query vector, a first key vector, and a first value vector. The first outer product vector of the first query vector and the first key vector is calculated to obtain the first similarity between the first query vector and the first key vector. The first similarity is normalized to obtain the first weight matrix, and the second outer product vector of the first weight matrix and the first value vector is calculated to obtain the first attention mechanism.

9. The method for processing audio data according to claim 1, characterized in that, The first audio feature and the second audio feature are combined based on the feature combination layer to obtain sound quality combination features, including: Calculate the difference between the first audio feature and the second audio feature to obtain the feature difference calculation result, and determine the sound quality combination feature based on the feature difference calculation result; and / or The feature difference calculation result, the first audio feature, and the second audio feature are combined to obtain the sound quality combination feature; and / or The first audio feature and the second audio feature are combined to obtain the sound quality combination feature.

10. The method for processing audio data according to claim 1, characterized in that, Determining whether the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data based on the audio quality difference results includes: If the sound quality difference result is greater than or equal to a preset threshold, then it is determined that the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data. If the sound quality difference result is less than a preset threshold, then it is determined that the first audio quality of the first original audio data is lower than the second audio quality of the second original audio data. The larger the value of the sound quality difference result, the higher the first audio quality of the first original audio data; the smaller the value of the sound quality difference result, the higher the second audio quality of the second original audio data.

11. The method for processing audio data according to claim 10, characterized in that, The method for processing the audio data also includes: If the first audio quality is higher than the second audio quality, then the first original audio data is used as the target audio data of the song corresponding to the original audio data. If the second audio quality is higher than the first audio quality, then the second original audio data is used as the target audio data of the song corresponding to the original audio data.

12. The method for processing audio data according to claim 1, characterized in that, The preset audio data processing model is obtained in the following way: The lossless audio data of historical songs is acquired, and the lossless audio data is encoded and compressed at different bit rates based on a preset encoder to obtain the audio encoding result. The audio encoding result and the lossless audio data are randomly paired to obtain multiple audio pairing results, and the actual audio quality difference of each audio pairing result is determined. The audio pairing results are input into the neural network model to be trained to obtain the predicted sound quality difference, and a target loss function is constructed based on the actual sound quality difference and the predicted sound quality difference. The parameters of the neural network model to be trained are adjusted based on the target loss function to obtain the preset audio data processing model.

13. An audio data processing apparatus, characterized in that, include: The audio data extraction module is used to acquire first raw audio data and second raw audio data, and extract first audio data to be compared and second audio data to be compared from the first raw audio data and second raw audio data; The audio quality difference determination module is used to input the first audio data to be compared and the second audio data to be compared into a preset audio data processing model to obtain the audio quality difference result between the first audio data to be compared and the second audio data to be compared; the preset audio data processing model includes an audio quality feature extraction model, a feature combination layer and an audio quality difference prediction model. An audio quality determination module is used to determine the first audio quality of the first audio data to be compared and the second audio quality of the second audio data to be compared based on the audio quality difference result. The sound quality difference result is determined as follows: the first audio data to be compared and the second audio data to be compared are input into the sound quality feature extraction model to obtain the first audio feature corresponding to the first audio data to be compared and the second audio feature corresponding to the second audio data to be compared. The first audio feature and the second audio feature are combined based on the feature combination layer to obtain the sound quality combination feature. The sound quality combination feature is then input into the sound quality difference prediction model to obtain the sound quality difference result between the first audio data to be compared and the second audio data to be compared.

14. The audio data processing apparatus according to claim 13, characterized in that, Extracting the first and second audio data to be compared from the first and second original audio data includes: Extract the first audio fingerprint data of the first original audio data and the second audio fingerprint data of the second original audio data; wherein, the first original audio data and the second original audio data are audio data of different bit rates corresponding to the same song; Based on the first audio fingerprint data and the second audio fingerprint data, audio alignment processing is performed on the first original audio data and the second original audio data to obtain the first audio alignment result and the second audio alignment result. The first audio data to be compared is extracted from the first audio alignment result, and the second audio data to be compared is extracted from the second audio alignment result; wherein the first audio position of the first audio data to be compared in the first audio alignment result is the same as the second audio position of the second audio data to be compared in the second audio alignment result.

15. The audio data processing apparatus according to claim 14, characterized in that, Extracting the first audio fingerprint data from the first original audio data includes: Perform a short-time Fourier transform on the first original audio data to obtain a first original spectrogram, and traverse the first original spectrogram to extract the first local peak point from the first original spectrogram. The first audio fingerprint data is constructed based on the first time sequence position of the first local peak point in the first original spectrogram and the first frequency band number of the first local peak point.

16. The audio data processing apparatus according to claim 14, characterized in that, Based on the first audio fingerprint data and the second audio fingerprint data, audio alignment processing is performed on the first original audio data and the second original audio data to obtain the first audio alignment result and the second audio alignment result, including: Traverse the first original spectrogram corresponding to the first original audio data, and find the position in the first original spectrogram that has the most overlap with the second local peak point in the second audio fingerprint data, using this position as the audio alignment position for the first original audio data and the second original audio data; and / or Traverse the second original spectrogram corresponding to the second original audio data, and find the position in the second original spectrogram that has the most overlap with the first local peak point in the first audio fingerprint data, and use it as the audio alignment position of the first original audio data and the second original audio data. Based on the audio alignment position, the first original audio data and the second original audio data are subjected to audio alignment processing to obtain the first audio alignment result and the second audio alignment result.

17. The audio data processing apparatus according to claim 13, characterized in that, The preset sound quality feature extraction model includes a first feature extraction model and a second feature extraction model, and the first feature extraction model and the second feature extraction model share model parameters; Specifically, the first and second audio data to be compared are input into the sound quality feature extraction model to obtain a first audio feature corresponding to the first audio data to be compared and a second audio feature corresponding to the second audio data to be compared, including: The first audio data to be compared is subjected to a short-time Fourier transform to obtain a first spectrogram of the comparison, and the second audio data to be compared is subjected to a short-time Fourier transform to obtain a second spectrogram of the comparison. The first spectrogram to be compared is input into the first feature extraction model to obtain the first audio feature corresponding to the first audio data to be compared; The second spectrogram to be compared is input into the second feature extraction model to obtain the second audio feature corresponding to the second audio data to be compared.

18. The audio data processing apparatus according to claim 17, characterized in that, The first feature extraction model includes a first embedding mapping layer, a position encoding module, a feature encoding module, a feature decoding module, a linear transformation layer, and a Softmax layer; Specifically, inputting the first spectrogram to be compared into the first feature extraction model to obtain the first audio feature corresponding to the first audio data to be compared includes: The first spectrogram to be compared is embedded and mapped based on the first embedding mapping layer to obtain a first embedding vector, and the first spectrogram to be compared is encoded based on the position encoding module to obtain a first position vector. The first embedding vector and the first position vector are superimposed to obtain the first input vector, and the first input vector is encoded based on the feature encoding module to obtain the first encoding result; The first encoding result is decoded by the feature decoding module to obtain the first encoding matrix. The first encoding matrix is ​​then transformed by the linear transformation layer to obtain the first logical matrix. Finally, the first logical matrix is ​​mapped by the Softmax layer to obtain the first audio feature corresponding to the first audio data to be compared.

19. The audio data processing apparatus according to claim 18, characterized in that, The feature encoding module includes a first multi-head self-attention module, a first residual connection and normalization module, a first feedforward neural network, and a second residual connection and normalization module; The first input vector is encoded using a feature encoding module to obtain a first encoding result, including: The first attention mechanism is calculated based on the first multi-head self-attention module, and the first attention mechanism is subjected to residual connection and normalization processing based on the first residual connection and normalization module to obtain the first normalization processing result. The first normalization result is linearized based on the first feedforward neural network to obtain the first linearized result. The first linearized result is then subjected to residual connection and normalization based on the second residual connection and normalization module to obtain the first encoding result.

20. The audio data processing apparatus according to claim 18, characterized in that, The first attention mechanism, which calculates the first input vector based on the first multi-head self-attention module, includes: The first input vector is linearly fused to obtain a first query vector, a first key vector, and a first value vector. The first outer product vector of the first query vector and the first key vector is calculated to obtain the first similarity between the first query vector and the first key vector. The first similarity is normalized to obtain the first weight matrix, and the second outer product vector of the first weight matrix and the first value vector is calculated to obtain the first attention mechanism.

21. The audio data processing apparatus according to claim 13, characterized in that, The first audio feature and the second audio feature are combined based on the feature combination layer to obtain sound quality combination features, including: Calculate the difference between the first audio feature and the second audio feature to obtain the feature difference calculation result, and determine the sound quality combination feature based on the feature difference calculation result; and / or The feature difference calculation result, the first audio feature, and the second audio feature are combined to obtain the sound quality combination feature; and / or The first audio feature and the second audio feature are combined to obtain the sound quality combination feature.

22. The audio data processing apparatus according to claim 13, characterized in that, Determining whether the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data based on the audio quality difference results includes: If the sound quality difference result is greater than or equal to a preset threshold, then it is determined that the first audio quality of the first original audio data is higher than the second audio quality of the second original audio data. If the sound quality difference result is less than a preset threshold, then it is determined that the first audio quality of the first original audio data is lower than the second audio quality of the second original audio data. The larger the value of the sound quality difference result, the higher the first audio quality of the first original audio data; the smaller the value of the sound quality difference result, the higher the second audio quality of the second original audio data.

23. The audio data processing apparatus according to claim 22, characterized in that, The audio data processing device further includes: The first target audio data determination module is used to determine the first original audio data as the target audio data of the song corresponding to the original audio data if the first audio quality is higher than the second audio quality. The second target audio data determination module is used to determine the second original audio data as the target audio data of the song corresponding to the original audio data if the second audio quality is higher than the first audio quality.

24. The audio data processing apparatus according to claim 13, characterized in that, The preset audio data processing model is obtained in the following way: The lossless audio data of historical songs is acquired, and the lossless audio data is encoded and compressed at different bit rates based on a preset encoder to obtain the audio encoding result. The audio encoding result and the lossless audio data are randomly paired to obtain multiple audio pairing results, and the actual audio quality difference of each audio pairing result is determined. The audio pairing results are input into the neural network model to be trained to obtain the predicted sound quality difference, and a target loss function is constructed based on the actual sound quality difference and the predicted sound quality difference. The parameters of the neural network model to be trained are adjusted based on the target loss function to obtain the preset audio data processing model.

25. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for processing audio data according to any one of claims 1-12.

26. An electronic device, comprising: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to perform the audio data processing method according to any one of claims 1-12 by executing the executable instructions.

Images