Music recommendation method, system, device and storage medium based on audio similarity

Abstract

The present invention discloses a music recommendation method, system, device and storage medium based on audio similarity, including: obtaining a first audio file of historical music; converting the first audio file into a first spectrogram; converting the first spectrogram into a first chromaticity gradient map, inputting the first chromaticity gradient map into a trained chromaticity feature extraction model to obtain a first chromaticity gradient matrix; obtaining a second audio file of music to be selected, converting the second audio file into a second spectrogram; converting the second spectrogram into a second chromaticity gradient map, inputting the second chromaticity gradient map into a trained chromaticity feature extraction model to obtain a second chromaticity gradient matrix; calculating the similarity between the first chromaticity gradient matrix and the second chromaticity gradient matrix, and determining the recommended music according to the similarity. The embodiment of the present invention can find and recommend music with high similarity according to the essential attributes of music, better meet user needs, improve user experience, and can be widely used in the field of information processing technology.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a music recommendation method, system, device and storage medium based on audio similarity. Background Art

[0002] With the development of the Internet, digital music has become the mainstream consumption content. However, the massive amount of music data far exceeds the audience's tolerance. Looking for personal favorite music in the massive music library may cause information fatigue for users. In order to improve the user experience, music recommendation systems came into being. A good recommendation system can establish a close relationship with users and create stickiness to the website, thereby effectively improving the user experience of the website and creating economic value.

[0003] In the related art, there are many music recommendation algorithms, such as collaborative filtering based on items, which calculates the similarity between items and recommends them through users with common preferences; such as recommending music to users based on the similarity between tags, etc. Since each piece of music has a unique style, if the recommendation is not based on the inherent attributes of the music, it may not meet the needs or expectations of users. Summary of the invention

[0004] In view of this, the purpose of the embodiments of the present invention is to provide a music recommendation method, system, device and storage medium based on audio similarity, which can find and recommend music with high similarity according to the essential attributes of the music, better meet user needs and improve user experience.

[0005] In a first aspect, an embodiment of the present invention provides a music recommendation method based on audio similarity, comprising:

[0006] Acquire a first audio file of historical music; and convert the first audio file into a first spectrogram;

[0007] Converting the first spectrogram into a first chromaticity gradient map, and inputting the first chromaticity gradient map into a trained chromaticity feature extraction model to obtain a first chromaticity gradient matrix;

[0008] Acquire a second audio file of the music to be selected, and convert the second audio file into a second spectrogram;

[0009] Converting the second spectrogram into a second chromaticity gradient map, and inputting the second chromaticity gradient map into a trained chromaticity feature extraction model to obtain a second chromaticity gradient matrix;

[0010] The similarity between the first chromaticity gradient matrix and the second chromaticity gradient matrix is ​​calculated, and the recommended music is determined according to the similarity.

[0011] Optionally, the training method of the chromaticity feature extraction model is as follows:

[0012] Obtaining a sample file of a chromaticity diagram, and marking a chromaticity gradient matrix of the sample file of the chromaticity diagram;

[0013] Dividing the chromaticity diagram sample data into a training set and a test set; the chromaticity diagram sample data includes the sample file and the chromaticity gradient matrix;

[0014] In combination with the attention mechanism, the learning model is trained according to the training set, and the prediction accuracy of the learning model is verified through the test set, and the learning model with the highest prediction accuracy is determined as the chromaticity feature extraction model.

[0015] Optionally, the learning model includes a convolutional neural network, which includes a first convolutional layer, a maximum pooling layer and a second convolutional layer connected in sequence, and the first convolutional layer includes a horizontal filter and a vertical filter.

[0016] Optionally, converting the first audio file into a first spectrogram specifically includes:

[0017] Performing frame division and windowing on the first audio file to obtain a plurality of frame audio files;

[0018] Performing fast Fourier transform on the plurality of frames of audio files to obtain corresponding plurality of frames of frequency spectrum information graphs;

[0019] Stacking the plurality of frames of spectrum information graphs in sequence to obtain a linear spectrogram;

[0020] The linear spectrogram is filtered and logarithm is taken to obtain a first spectrogram.

[0021] Optionally, converting the first spectrogram into a first chromaticity gradient map specifically includes:

[0022] Obtaining the chromaticity of each pixel in the first spectrogram;

[0023] The chromaticity difference between adjacent pixels is calculated according to the chromaticity of each pixel to obtain a first chromaticity gradient map.

[0024] Optionally, the calculating the similarity between the first chromaticity gradient matrix and the second chromaticity gradient matrix specifically includes:

[0025] Calculating the Pearson correlation coefficient between the first chromaticity gradient matrix and the second chromaticity gradient matrix;

[0026] The Pearson correlation coefficient is used as the similarity between the first chromaticity gradient matrix and the second chromaticity gradient matrix.

[0027] Optionally, determining the recommended music according to the similarity specifically includes:

[0028] When the similarity is greater than or equal to a preset threshold, the music to be selected is determined as recommended music;

[0029] When the similarity is less than a preset threshold or is a negative value, the music to be selected is determined as abandoned music.

[0030] In a second aspect, an embodiment of the present invention provides a music recommendation system based on audio similarity, comprising:

[0031] A first acquisition module, configured to acquire a first audio file of historical music; and convert the first audio file into a first spectrogram;

[0032] A first prediction module, configured to convert the first spectrogram into a first chromaticity gradient map, and input the first chromaticity gradient map into a trained chromaticity feature extraction model to obtain a first chromaticity gradient matrix;

[0033] A second acquisition module, used for acquiring a second audio file of the music to be selected, and converting the second audio file into a second spectrogram;

[0034] A second prediction module, used for converting the second spectrogram into a second chromaticity gradient map, and inputting the second chromaticity gradient map into a trained chromaticity feature extraction model to obtain a second chromaticity gradient matrix;

[0035] The recommendation module is used to calculate the similarity between the first chromaticity gradient matrix and the second chromaticity gradient matrix, and determine the recommended music according to the similarity.

[0036] In a third aspect, an embodiment of the present invention provides a music recommendation device based on audio similarity, comprising:

[0037] at least one processor;

[0038] at least one memory for storing at least one program;

[0039] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0040] In a fourth aspect, an embodiment of the present invention provides a storage medium, in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to perform the above method.

[0041] The implementation of the embodiments of the present invention includes the following beneficial effects: the embodiments of the present invention converts a first audio file of historical music into a first spectrogram, and obtains a first chromaticity gradient matrix from the first spectrogram through a chromaticity feature extraction model; converts a second audio file of the music to be selected into a second spectrogram, and obtains a second chromaticity gradient matrix from the second spectrogram through a chromaticity feature extraction model; determines recommended music based on the similarity between the first chromaticity gradient matrix and the second chromaticity gradient matrix; calculates the similarity of music and recommended music by extracting the chromaticity gradient matrix from the audio file of the music and extracting significant features from the chromaticity gradient matrix, thereby finding and recommending music with high similarity based on the essential attributes of the music, better meeting user needs and improving user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 1 is a schematic diagram of the steps of a music recommendation method based on audio similarity provided by an embodiment of the present invention;

[0043] Figure 2 It is a schematic diagram of a process flow of converting an audio file into a spectrogram provided by an embodiment of the present invention;

[0044] Figure 3 is an example of a spectrogram provided by an embodiment of the present invention;

[0045] Figure 4 is a structural block diagram of a music recommendation system based on audio similarity provided by an embodiment of the present invention;

[0046] Figure 5 It is a structural block diagram of a music recommendation device based on audio similarity provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0048] like Figure 1 As shown, an embodiment of the present invention provides a music recommendation method based on audio similarity, which includes the following steps.

[0049] S100: Acquire a first audio file of historical music; and convert the first audio file into a first spectrogram.

[0050] It should be noted that the audio file is converted into a spectrogram through a discrete Fourier transform (DFT), which is a visual description of the music. The discrete Fourier transform is a discrete form of the Fourier transform in both the time domain and the frequency domain, which transforms the sampling of the time domain signal into the sampling of the discrete time Fourier transform frequency domain. In form, the sequences at both ends of the transformation (in the time domain and the frequency domain) are of finite length, but in fact both sets of sequences should be considered as the principal value sequences of discrete periodic signals. Even if a discrete Fourier transform is performed on a discrete signal of finite length, it should be considered as a periodic signal after periodic extension and then transformed.

[0051] Optionally, see Figure 2 , converting the first audio file into a first spectrogram specifically includes:

[0052] S110, performing frame division and windowing on the first audio file A1 to obtain a plurality of frame audio files A2;

[0053] S120, performing fast Fourier transform on the plurality of frames of audio files A2 to obtain corresponding plurality of frames of frequency spectrum information graphs B1;

[0054] S130, stacking the plurality of frame frequency spectrum information graphs B1 in sequence to obtain a linear spectrogram B2;

[0055] S140, filtering the linear spectrogram B2 and taking its logarithm to obtain a first spectrogram.

[0056] It should be noted that the audio is converted into a spectrogram, that is, a frequency-time change graph, but the law of frequency change over time cannot be obtained; through discrete Fourier transform, the spectrum graph is turned into an easy-to-process spectrogram, which is convenient for the subsequent generation of a chromaticity gradient graph. The transformation process is as follows: first, the original audio signal is framed and windowed to obtain many frames; then a fast Fourier transform is performed on each frame. The function of the Fourier transform is to convert the time domain into a frequency domain signal, and the spectrum information of each frame is collected to form a spectrogram; then the Mel filter is applied to the linear spectrogram, and then the logarithm is taken to obtain a logarithmic Mel spectrogram.

[0057] S200: Convert the first spectrogram into a first chromaticity gradient map, and input the first chromaticity gradient map into a trained chromaticity feature extraction model to obtain a first chromaticity gradient matrix.

[0058] Those skilled in the art will understand that the amount, speed and time span of the beat change have an important impact on the listener. Since the spectrogram is transformed from the audio file, there is a great correlation between the chromaticity distribution of the spectrogram and the beat, etc. The chromaticity change of the spectrogram can reveal the rhythm or beat change of the audio. Figure 3In the spectrogram, different colors represent different musical characteristics such as frequency, loudness, amplitude and overtones. Elements such as instruments and styles can be learned from them through deep learning. These contents can summarize the characteristics and essence of the song to a certain extent, thereby inferring the reasons for the audience's preference for music.

[0059] For the chromaticity changes between the pixels of the spectrogram, the size of the change can be measured by the chromaticity difference, and the amplitude of the change can be composed of the chromaticity change values ​​of each row and column to form a chromaticity difference map, thereby calculating the chromaticity change gradient, and at the same time, the gradient map representation of the chromaticity of the spectrogram can be realized.

[0060] Optionally, converting the first spectrogram into a first chromaticity gradient map specifically includes:

[0061] S210, obtaining the chromaticity of each pixel in the first spectrogram;

[0062] S220 , calculating the chromaticity difference between adjacent pixels according to the chromaticity of each pixel to obtain a first chromaticity gradient map.

[0063] It should be noted that the size of the gradient represents the speed of the music's beat changes, and combined with the original chromaticity to determine the changes in the music's frequency, amplitude and other characteristics, so as to more accurately portray the essential attributes of the music. The speed and amount of rhythm changes can be represented by the chromaticity gradient. The regional chromaticity of the original sound spectrum represents the amplitude and frequency of the sound at that moment. The similarity between songs depends not only on the speed of chromaticity changes, but also on the original pitch, pitch and frequency. Therefore, the gradient combined with the two end values ​​can determine the rhythm direction of the music and can be used to judge the similarity between music.

[0064] Optionally, the training method of the chromaticity feature extraction model is as follows:

[0065] S010, obtaining a sample file of a chromaticity diagram, and marking a chromaticity gradient matrix of the sample file of the chromaticity diagram;

[0066] S020, dividing the chromaticity diagram sample data into a training set and a test set; the chromaticity diagram sample data includes the sample file and the chromaticity gradient matrix;

[0067] S030. In combination with the attention mechanism, the learning model is trained according to the training set, and the prediction accuracy of the learning model is verified through the test set, and the learning model with the highest prediction accuracy is determined as the chromaticity feature extraction model.

[0068] Optionally, the learning model includes a convolutional neural network, which includes a first convolutional layer, a maximum pooling layer and a second convolutional layer connected in sequence, and the first convolutional layer includes a horizontal filter and a vertical filter.

[0069] It should be noted that the convolutional neural network can also include other optimization layers. The convolutional neural network is trained in combination with the attention mechanism, so that the convolutional neural network focuses on the part where the melody changes the fastest and the content changes the most, and enhances the feature representation and proportion of such positions.

[0070] A horizontal filter is used to capture the features on the timbre envelope, and a vertical filter is used to capture the features on the time energy envelope. In order to maintain the invariance of the fundamental pitch, the maximum pooling layer collects the maximum values ​​across the frequency axis, connects the captured time, timbre and other features in series within the channel, and then summarizes the features in the 1D convolution layer to generate the prediction results of the song.

[0071] S300: Obtain a second audio file of the music to be selected, and convert the second audio file into a second spectrogram.

[0072] Optionally, converting the second audio file into a second spectrogram specifically includes:

[0073] S310, performing frame division and windowing on the second audio file to obtain a plurality of audio file frames;

[0074] S320, performing fast Fourier transform on the plurality of frames of audio files to obtain corresponding plurality of frames of frequency spectrum information graphs;

[0075] S330, stacking the plurality of frames of frequency spectrum information graphs in order to obtain a linear spectrogram;

[0076] S340: Filter the linear spectrogram and take its logarithm to obtain a second spectrogram.

[0077] It should be noted that the method for obtaining the second spectrogram is the same as the method for obtaining the first spectrogram.

[0078] S400: Convert the second spectrogram into a second chromaticity gradient map, and input the second chromaticity gradient map into a trained chromaticity feature extraction model to obtain a second chromaticity gradient matrix.

[0079] Optionally, converting the second spectrogram into a second chromaticity gradient map specifically includes:

[0080] S410, obtaining the chromaticity of each pixel in the second spectrogram;

[0081] S420, calculating the chromaticity difference between adjacent pixels according to the chromaticity of each pixel to obtain a second chromaticity gradient map.

[0082] It should be noted that the method for acquiring the second chromaticity gradient map is the same as the method for acquiring the first chromaticity gradient map.

[0083] S500, calculating the similarity between the first chromaticity gradient matrix and the second chromaticity gradient matrix, and determining recommended music according to the similarity.

[0084] It should be noted that the first chromaticity gradient matrix or the second chromaticity gradient matrix represents the characteristic vector corresponding to the number of beat changes, the change frequency or the change amplitude corresponding to the audio file.

[0085] Optionally, the calculating the similarity between the first chromaticity gradient matrix and the second chromaticity gradient matrix specifically includes:

[0086] S510, calculating the Pearson correlation coefficient between the first chromaticity gradient matrix and the second chromaticity gradient matrix;

[0087] S520. Use the Pearson correlation coefficient as the similarity between the first chromaticity gradient matrix and the second chromaticity gradient matrix.

[0088] Specifically, the similarity calculation formula is as follows:

[0089]

[0090] Among them, corr(x,y) represents the Pearson correlation coefficient, (x i ,y i ) represents the corresponding dimensions of the two feature vectors of the music to be selected, Represents the corresponding dimensions of the two feature vectors of historical music.

[0091] It should be noted that the larger the Pearson correlation coefficient is, the higher the similarity between the historical music and the music to be selected is. If the Pearson correlation coefficient is negative, it means that the audio features are negatively correlated and are not recommended.

[0092] Optionally, determining the recommended music according to the similarity specifically includes:

[0093] S530: when the similarity is greater than or equal to a preset threshold, determining the music to be selected as recommended music;

[0094] S540: When the similarity is less than a preset threshold or is a negative value, the music to be selected is determined as abandoned music.

[0095] It should be noted that the preset threshold value may be set by a user or according to an empirical value. The preset threshold value is greater than 0, and this embodiment does not impose any specific limitation thereto.

[0096] The implementation of the embodiments of the present invention includes the following beneficial effects: the embodiments of the present invention converts a first audio file of historical music into a first spectrogram, and obtains a first chromaticity gradient matrix from the first spectrogram through a chromaticity feature extraction model; converts a second audio file of the music to be selected into a second spectrogram, and obtains a second chromaticity gradient matrix from the second spectrogram through a chromaticity feature extraction model; determines recommended music based on the similarity between the first chromaticity gradient matrix and the second chromaticity gradient matrix; calculates the similarity of music and recommended music by extracting the chromaticity gradient matrix from the audio file of the music and extracting significant features from the chromaticity gradient matrix, thereby finding and recommending music with high similarity based on the essential attributes of the music, better meeting user needs and improving user experience.

[0097] like Figure 4 As shown, an embodiment of the present invention provides a music recommendation system based on audio similarity, including:

[0098] A first acquisition module, configured to acquire a first audio file of historical music; and convert the first audio file into a first spectrogram;

[0099] A first prediction module, configured to convert the first spectrogram into a first chromaticity gradient map, and input the first chromaticity gradient map into a trained chromaticity feature extraction model to obtain a first chromaticity gradient matrix;

[0100] A second acquisition module, used for acquiring a second audio file of the music to be selected, and converting the second audio file into a second spectrogram;

[0101] A second prediction module, used for converting the second spectrogram into a second chromaticity gradient map, and inputting the second chromaticity gradient map into a trained chromaticity feature extraction model to obtain a second chromaticity gradient matrix;

[0102] The recommendation module is used to calculate the similarity between the first chromaticity gradient matrix and the second chromaticity gradient matrix, and determine the recommended music according to the similarity.

[0103] It can be seen that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0104] like Figure 5 As shown, an embodiment of the present invention provides a music recommendation device based on audio similarity, comprising:

[0105] at least one processor;

[0106] at least one memory for storing at least one program;

[0107] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0108] It can be seen that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0109] In addition, the embodiment of the present application also discloses a computer program product or a computer program, and the computer program product or the computer program is stored in a computer-readable storage medium. The processor of the computer device can read the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device performs the above method. Similarly, the contents in the above method embodiment are all applicable to the storage medium embodiment, and the functions specifically implemented by the storage medium embodiment are the same as those in the above method embodiment, and the beneficial effects achieved are also the same as those achieved by the above method embodiment.

[0110] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A music recommendation method based on audio similarity, characterized in that: include: Get the first audio file of historical music; and converting the first audio file into a first spectrogram; Converting the first spectrogram into a first chromaticity gradient map, and inputting the first chromaticity gradient map into a trained chromaticity feature extraction model to obtain a first chromaticity gradient matrix; Obtaining a second audio file of the music to be selected, and converting the second audio file into a second spectrogram; Converting the second spectrogram into a second chromaticity gradient map, and inputting the second chromaticity gradient map into a trained chromaticity feature extraction model to obtain a second chromaticity gradient matrix; The similarity between the first chromaticity gradient matrix and the second chromaticity gradient matrix is ​​calculated, and the recommended music is determined according to the similarity.

2. The method according to claim 1, characterized in that The training method of the chromaticity feature extraction model is as follows: Obtaining a sample file of a chromaticity diagram, and marking a chromaticity gradient matrix of the sample file of the chromaticity diagram; Dividing the chromaticity diagram sample data into a training set and a test set; the chromaticity diagram sample data includes the sample file and the chromaticity gradient matrix; In combination with the attention mechanism, the learning model is trained according to the training set, and the prediction accuracy of the learning model is verified through the test set, and the learning model with the highest prediction accuracy is determined as the chromaticity feature extraction model.

3. The method according to claim 2, characterized in that The learning model includes a convolutional neural network, which includes a first convolutional layer, a maximum pooling layer and a second convolutional layer connected in sequence, and the first convolutional layer includes a horizontal filter and a vertical filter.

4. The method according to claim 1, characterized in that: The converting the first audio file into a first spectrogram specifically includes: Performing frame division and windowing on the first audio file to obtain a plurality of frame audio files; Performing fast Fourier transform on the plurality of frames of audio files to obtain corresponding plurality of frames of frequency spectrum information graphs; Stacking the plurality of frames of spectrum information graphs in sequence to obtain a linear spectrogram; The linear spectrogram is filtered and logarithm is taken to obtain a first spectrogram.

5. The method according to claim 1, characterized in that The converting the first sound spectrogram into a first chromaticity gradient map specifically includes: Obtaining the chromaticity of each pixel in the first spectrogram; The chromaticity difference between adjacent pixels is calculated according to the chromaticity of each pixel to obtain a first chromaticity gradient map.

6. The method according to any one of claims 1 to 5, characterized in that: The calculating the similarity between the first chromaticity gradient matrix and the second chromaticity gradient matrix specifically includes: Calculating the Pearson correlation coefficient between the first chromaticity gradient matrix and the second chromaticity gradient matrix; The Pearson correlation coefficient is used as the similarity between the first chromaticity gradient matrix and the second chromaticity gradient matrix.

7. The method according to any one of claim 6, characterized in that: Determining the recommended music according to the similarity specifically includes: When the similarity is greater than or equal to a preset threshold, the music to be selected is determined as recommended music; When the similarity is less than a preset threshold or is a negative value, the music to be selected is determined as abandoned music.

8. A music recommendation system based on audio similarity, characterized in that: include: A first acquisition module, used for acquiring a first audio file of historical music; and converting the first audio file into a first spectrogram; A first prediction module, configured to convert the first spectrogram into a first chromaticity gradient map, and input the first chromaticity gradient map into a trained chromaticity feature extraction model to obtain a first chromaticity gradient matrix; A second acquisition module, used for acquiring a second audio file of the music to be selected, and converting the second audio file into a second spectrogram; A second prediction module, used for converting the second spectrogram into a second chromaticity gradient map, and inputting the second chromaticity gradient map into a trained chromaticity feature extraction model to obtain a second chromaticity gradient matrix; The recommendation module is used to calculate the similarity between the first chromaticity gradient matrix and the second chromaticity gradient matrix, and determine the recommended music according to the similarity.

9. A music recommendation device based on audio similarity, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 7 when executed by the processor.

Images